Decide which AI crawlers get to read you
Optional. Start from the file this site already has: skip it and you get a fresh file, but on a site you inherited the rules already in there are the ones worth keeping, and this is how they survive the paste.
21 allowed · 0 blocked
Fetches pages for answers
Block these and you are not in that engine's answers.
Answers and training
One agent doing both jobs, so blocking it costs you the answer surface too.
Training only
Block these to keep your pages out of model training. Answer surfaces are unaffected.
Collected in bulk, then resold
Archives and data brokers. No answer surface of their own — their output is somebody else's training set, which is also why blocking them is the one decision here you cannot check up on later.
Optional, and worth adding while you are in here.
# AI crawler rules — https://passcite.com/tools/ai-robots-txt-generator
User-agent: *
Allow: /
Nothing blocked — which for most businesses is the right answer. This file says so explicitly, which is worth having: it means the next person to inherit the site can see the decision was made rather than skipped.
The mistake this is built to prevent
Through 2023 and 2024 a lot of sites added User-agent: GPTBot / Disallow: / because they did not want their writing used for training. Reasonable. Then answer engines arrived, and the crawler that fetches pages for ChatGPT’s answers turned out to be a different agent — OAI-SearchBot — and several sites had blocked that one too, in the same sweep, without meaning to.
It is not a one-off. Applebot is Apple’s search crawler and Applebot-Extended is the training flag; block the first when you meant the second and you have left Siri and Spotlight to keep your pages out of a model. That is why the list above is grouped by consequence rather than by company. “Keep my work out of training” and “stay out of the answers” are different decisions and deserve to look like it.
Why this hands you a file and not a snippet
Every other builder of this kind ends with a block of text to paste at the bottom of a robots.txt it has never read. Two things go wrong with that, and both are silent.
The first is the one worth understanding, because it looks like the safe option. Those snippets write a group for every crawler they name — including the ones being allowed, as User-agent: GPTBot followed by an empty Disallow:. It reads as a line that does nothing. What it actually does is give GPTBot a group of its own, and a crawler with a named group stops reading the * group completely. If your file said Disallow: /wp-admin/ under the wildcard, that rule has just stopped applying to every bot in the snippet.
The second is duplication. Paste an AI section onto a file that got one last year and both are now in there, with the older and usually stricter one first. Nobody re-reads a robots.txt.
So this reads your live file, rewrites only the groups that name AI crawlers, and leaves the rest byte for byte — your wildcard rules, your Googlebot group, your sitemap lines, your comments. The panel under the output counts what it carried over, because the point of the receipt is that you should not have to take our word for it.
What is deliberately not on the list
FacebookBot, which some lists of “AI crawlers” include to reach eighteen. It renders the thumbnail when somebody shares your URL. Blocking it costs a link preview and buys you nothing in AI — which is the exact shape of mistake the grouping above exists to prevent, so putting it here to make the list look longer would be arguing against ourselves.
Then check what the server is actually serving
An editor that says it saved the file and a URL that returns it are different claims. Read the live file back after you publish, or paste a file into the validator to find rules that contradict each other or do nothing at all.
Questions
See who the engines recommend instead
12 buyer questions across ChatGPT, Perplexity and Gemini. You get the questions your client loses, who won them, and what is missing from the site. No score, no signup wall.