Are the AI crawlers allowed in right now?
Two mistakes this catches that a generator cannot
The narrow group that loosened everything
A site has User-agent: * with a careful set of disallows. Someone adds User-agent: GPTBot with Disallow: /. GPTBot is now blocked — and every other rule in the wildcard group no longer applies to it, because a crawler reads exactly one group. That is usually not what anyone intended, and it is invisible unless you evaluate the file the way a crawler does.
The directives that were never supported
Noindex: in robots.txt was never a supported directive and stopped working in Google in 2019. Neither does Crawl-delay: for Google or for any of the AI crawlers. Both appear in files all the time, usually added by someone who believed they were doing something. This lists them.
If everything is allowed and you are still not recommended
Then the problem is not access — it is that there is nothing on the site worth quoting for the question being asked. That is the common case, and it is what the free report is for: 12 real buyer questions across three engines, and the names of whoever got recommended instead.
Questions
See who the engines recommend instead
12 buyer questions across ChatGPT, Perplexity and Gemini. You get the questions your client loses, who won them, and what is missing from the site. No score, no signup wall.