Question 1

Which rule wins when two match?

Accepted Answer

The longest one. Disallow: /blog and Allow: /blog/public both match /blog/public/post — the Allow is longer, so the page is crawlable. On an exact-length tie, Allow wins. This is the rule Google documents and the major AI crawlers follow.

Question 2

Do consecutive User-agent lines make one group or several?

Accepted Answer

One. Three User-agent lines followed by a single Disallow apply that rule to all three agents. Parsers that get this wrong report a file as far more permissive than it is.

Question 3

Why does adding a group for one crawler change what it can reach?

Accepted Answer

Because a crawler obeys exactly one group — the most specific one whose name matches it — and ignores the wildcard group entirely. Adding User-agent: GPTBot with one rule means every rule you wrote under User-agent: * no longer applies to GPTBot.

Question 4

Does Noindex work in robots.txt?

Accepted Answer

No, and it never did as a supported directive. Use a meta robots tag on the page, or an X-Robots-Tag response header. This tool flags Noindex lines because they are common and give people false confidence.

Question 5

What about Crawl-delay?

Accepted Answer

Ignored by Google and by every AI crawler listed here. Some other crawlers honour it. It is flagged rather than treated as an error.

Question 6

Is blocking a page in robots.txt the same as removing it from search?

Accepted Answer

No — and it can be the opposite. A blocked page cannot be read, so a crawler cannot see a noindex tag on it, and the URL can still surface with no description. If you want a page out of results, let it be crawled and mark it noindex.

Find the robots.txt rules that do nothing

Four things that are usually wrong

Then ask the question that is actually about your business

Questions

See who the engines recommend instead