robots.txt validator: find the rules that do nothing
Almost every robots.txt parses. That is not the same as doing what its author meant. Load or paste one, then test a real path against Googlebot, Bingbot and every AI crawler.
Pasted text is parsed in your browser and never sent anywhere.
Four things that are usually wrong
Disallow:with nothing after it blocks nothing. It is the opposite ofDisallow: /, and a single missing character is the difference.- A rule before the first
User-agentline belongs to no group and is skipped entirely. NoindexandCrawl-delaylook like they are working. Google reads neither, and Crawl-delay only slows the few crawlers that honour it — ClaudeBot is one — without ever keeping them out.- Blocking a page does not remove it from results. It prevents the crawl that would have found your noindex tag.
How the verdict is checked
Every verdict this tool gives, but for one rule of Google's own, is held to Google's open-source parser: 72 cases recorded from its binary, and a test that fails the day the two disagree. The rules it had to learn, and the four large sites the old version got wrong, are in how Google parses robots.txt.
Then ask the question that is actually about your business
A clean robots.txt means the engines can read you. It says nothing about whether they name you. Check the AI crawlers specifically, or run your first report — 10 prompts, free — and see who is being recommended instead.
FAQ
Questions
- Which rule wins when two match?
- The longest one. Disallow: /blog and Allow: /blog/public both match /blog/public/post — the Allow is longer, so the page is crawlable. On an exact-length tie, Allow wins. This is the rule RFC 9309 sets and Google documents; the AI operators say they follow the RFC.
- Do consecutive User-agent lines make one group or several?
- One. Three User-agent lines followed by a single Disallow apply that rule to all three agents. Parsers that get this wrong report a file as far more permissive than it is.
- Why does adding a group for one crawler change what it can reach?
- Because a crawler that finds its own name obeys only the groups with that name — two of them are read as one — and ignores the wildcard group entirely. Adding User-agent: GPTBot with one rule means every rule you wrote under User-agent: * no longer applies to GPTBot.
- Does Noindex work in robots.txt?
- No, and it never did as a supported directive. Use a meta robots tag on the page, or an X-Robots-Tag response header. This tool flags Noindex lines because they are common and give people false confidence.
- What about Crawl-delay?
- Ignored by Google. Some crawlers honour it — Anthropic documents it for ClaudeBot — but it only slows a crawler down; it never keeps one out. It is flagged rather than treated as an error. What it does not do is end a group: a group ends only after an Allow or Disallow line, so User-agent: Googlebot with nothing but a Crawl-delay under it, then User-agent: GPTBot and Disallow: /, is one group — and Googlebot is shut out of the whole site. Google's own parser reads it that way. This tool flags the line where the two join.
- Is blocking a page in robots.txt the same as removing it from search?
- No — and it can be the opposite. A blocked page cannot be read, so a crawler cannot see a noindex tag on it, and the URL can still surface with no description. If you want a page out of results, let it be crawled and mark it noindex.
See who the engines recommend instead
The prompts your client loses, who wins them, and the tasks that would change it.