AI crawler checker: are they allowed in right now?
Example: what a result looks like
Example · example.com- AllowedOAI-SearchBotFetches pages that ChatGPT shows and links in its answers.OpenAI · Blocking it removes you from answers
- AllowedPerplexityBotIndexes pages so Perplexity can cite them.Perplexity · Blocking it removes you from answers
- AllowedGPTBotCrawls pages to train OpenAI's models.OpenAI · Blocking it affects training only · Naming it dropped 2
*rules for it — /wp-admin/, /checkout/ are now open to this crawler. - AllowedClaudeBotCollects pages that may be used to train Anthropic's models. Claude's answers fetch with Claude-SearchBot and Claude-User.Anthropic · Blocking it affects training only
- BlockedCCBotCommon Crawl's public archive. Not an engine — its corpus is an input to many models.Common Crawl · Blocking it keeps you out of bulk datasets only
Two mistakes this catches that a generator cannot
The narrow group that loosened everything
A site has User-agent: * with a careful set of disallows. Someone adds User-agent: GPTBot with Disallow: /. GPTBot is now blocked — and every other rule in the wildcard group no longer applies to it, because a crawler that is named reads only the groups that name it. That is usually not what anyone intended, and it is invisible unless you evaluate the file the way a crawler does. This checker says, on the crawler's row, which paths that opened or closed; the robots.txt validator names the group and rule behind any path you test; the AI robots.txt generator rewrites only the AI groups and leaves the rest of the file alone. The rules both follow are Google's own parser's — how we checked that, and what we got wrong.
The directives that do less than they look like
Noindex: in robots.txt was never a supported directive and stopped working in Google in 2019. Crawl-delay: is ignored by Google too, and where a crawler does honour it — Anthropic documents it for ClaudeBot — it only slows that crawler down. Both appear in files all the time, usually added by someone who believed they were blocking something. This lists them.
If everything is allowed and you are still not recommended
Then the problem is not access — it is that there is nothing on the site worth quoting for the question being asked. That is the common case, and it is what the report is for — your first one free: 10 prompts a buyer types, across three engines, and the names of whoever got recommended instead.
Questions
- Why does this matter more than the rest of the list?
- Because it is the one finding that makes everything else pointless. If an engine's crawler cannot fetch your pages, no amount of schema, new pages or llms.txt will get you named in its answers. It is the first thing worth checking on any site you did not build yourself.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot crawls pages to train OpenAI's models. OAI-SearchBot fetches the pages ChatGPT shows and links when it answers. Blocking GPTBot keeps you out of training; blocking OAI-SearchBot takes you out of the answers. A lot of sites blocked the second while meaning to block the first.
- Does blocking Google-Extended hurt my Google rankings?
- No. Google says it does not affect inclusion or ranking in Google Search, and AI Overviews and AI Mode go by Googlebot. What it does decide is whether Gemini may train on your pages and ground its answers in them, so blocking it can cost you citations in the Gemini app.
- My robots.txt has a wildcard group and a GPTBot group. Which applies?
- The GPTBot one, and only that one. A crawler reads the groups that name it and ignores the wildcard group completely — so adding a narrow group for one bot silently exempts it from every rule you wrote for everyone. If GPTBot is named in two groups, the two are read as one. This tool applies the same rules, and tells you which group matched.
- I have no robots.txt at all. Is that bad?
- Not for this. An absent robots.txt means nothing is restricted, which is the permissive case. You may still want one for other reasons — a Sitemap line, keeping crawlers out of a staging path — but its absence is not blocking anyone.
- Should I block AI crawlers?
- That is a real decision and it depends on what you sell. If your business is publishing, there is an argument for keeping training crawlers out. If your business is getting hired by people who ask an assistant for a recommendation, blocking the crawlers that feed answers is closing your own front door. Tell the two apart before you decide — each row in the result says which.
See who the engines recommend instead
The prompts your client loses, who wins them, and the tasks that would change it.