What happens to the User-agent: * group?

If you put a domain in the box, nothing: the wildcard group and every other rule in that file come back byte for byte, and only the groups that name an AI crawler are rewritten. If the wildcard group says Disallow: / and you untick a crawler, it gets a group of its own that repeats the wildcard group's other rules — so letting it in does not also hand it /wp-admin/. If you skip the domain and build a fresh file, the "Keep everyone else allowed" box adds an explicit User-agent: * with Allow: /; turn it off only if you mean to write the wildcard rules yourself, because a fresh file replaces whatever is live.

What do the four groups mean?

Fetches pages for answers: the crawler reads pages so an engine can show and cite them, so blocking it takes you out of that engine's answers. Answers and training: one agent doing both jobs, so blocking it costs the answer surface too. Training only: the crawler collects pages to train a model, and no answer surface moves. Collected in bulk, then resold: an archive or a data broker with no surface of its own, whose corpus is sold or given on to other people's training. Blocking anything in the last two has no effect on whether you appear in answers, which is the distinction most advice on this skips.

Can I paste this at the end of my existing robots.txt?

No — it is a whole file, and that is deliberate. The usual snippet appends a named group for every crawler it names, including the ones it is allowing, as a `User-agent: GPTBot` with an empty `Disallow:`. That looks like a no-op and is the reverse of one: a crawler that matches a named group stops reading the `*` group entirely, so your own `Disallow: /wp-admin/` quietly stops applying to every bot in the snippet. Put the domain in the box instead and you get your file back with only the AI groups changed.

Why is there no box for Googlebot?

Because Googlebot is Google Search's crawler, and AI Overviews and AI Mode cite only pages Google has indexed: a rule that keeps it out takes you out of those too. This tool only rewrites groups that name one of the crawlers listed above. A Googlebot group in your live file comes back untouched and is counted in the "carried over" line under the output; a group Googlebot shares with GPTBot keeps its rules for Googlebot and loses only the GPTBot line. The Google control that is on the list is Google-Extended, which governs Gemini training and grounding and has no effect on Search ranking.

Where does robots.txt go?

At the root of the domain, answering at example.com/robots.txt. Some platforms — Squarespace, for one — do not let you edit it at all, and on those this file is information rather than an action.

Will blocking a crawler remove content it already has?

No. robots.txt controls future fetching. It is not a deletion request, and pages already collected are not affected by it.

AI robots.txt generator: choose which crawlers get in

22 crawlers, sorted by what blocking them actually costs you. Paste a domain and the list fills in with what that site blocks today — then you get the same file back, with the AI rules rewritten and everything else in it untouched.

Optional. Start from the file this site already has: skip it and you get a fresh file, but on a site you inherited the rules already in there are the ones worth keeping, and this is how they survive the paste.

robots.txt

# AI crawler rules — https://passcite.com/tools/ai-robots-txt-generator

User-agent: *
Allow: /

Nothing blocked — which for most businesses is the right answer. This file says so explicitly, which is worth having: it means the next person to inherit the site can see the decision was made rather than skipped.

The mistake this is built to prevent

Through 2023 and 2024 a lot of sites added User-agent: GPTBot / Disallow: / because they did not want their writing used for training. Reasonable. Then answer engines arrived, and the crawler that fetches pages for ChatGPT’s answers turned out to be a different agent — OAI-SearchBot — and several sites had blocked that one too, in the same sweep, without meaning to.

It is not a one-off. Applebot is Apple’s search crawler and Applebot-Extended is the training flag; block the first when you meant the second and you have left Siri and Spotlight to keep your pages out of a model. That is why the list above is grouped by consequence rather than by company. “Keep my work out of training” and “stay out of the answers” are different decisions and deserve to look like it.

Why this hands you a file and not a snippet

Every other builder of this kind ends with a block of text to paste at the bottom of a robots.txt it has never read. Two things go wrong with that, and both are silent.

The first is the one worth understanding, because it looks like the safe option. Those snippets write a group for every crawler they name — including the ones being allowed, as User-agent: GPTBot followed by an empty Disallow:. It reads as a line that does nothing. What it actually does is give GPTBot a group of its own, and a crawler with a named group stops reading the * group completely. If your file said Disallow: /wp-admin/ under the wildcard, that rule has just stopped applying to every bot in the snippet.

The second is duplication. Paste an AI section onto a file that got one last year and both are now in there. Crawlers that follow the standard read the two as one group, and what wins comes down to spelling: last year’s Disallow: / beats an empty Disallow: and loses to Allow: /. Nobody re-reads a robots.txt.

So this reads your live file, rewrites only the groups that name AI crawlers, and leaves the rest byte for byte — your wildcard rules, your Googlebot group, your sitemap lines, your comments. The panel under the output counts what it carried over, because the point of the receipt is that you should not have to take our word for it.

What is deliberately not on the list

facebookexternalhit, which renders the preview when somebody shares your URL. Blocking it costs the preview and buys you nothing in AI. Nor FacebookBot, which some lists of “AI crawlers” still include to reach eighteen: Meta no longer lists it among its crawlers. Meta's AI crawlers are the three named above.

Then check what the server is actually serving

An editor that says it saved the file and a URL that returns it are different claims. Read the live file back after you publish, or paste a file into the validator to find rules that contradict each other or do nothing at all.

FAQ

Questions

What happens to the User-agent: * group?: If you put a domain in the box, nothing: the wildcard group and every other rule in that file come back byte for byte, and only the groups that name an AI crawler are rewritten. If the wildcard group says Disallow: / and you untick a crawler, it gets a group of its own that repeats the wildcard group's other rules — so letting it in does not also hand it /wp-admin/. If you skip the domain and build a fresh file, the "Keep everyone else allowed" box adds an explicit User-agent: * with Allow: /; turn it off only if you mean to write the wildcard rules yourself, because a fresh file replaces whatever is live.
What do the four groups mean?: Fetches pages for answers: the crawler reads pages so an engine can show and cite them, so blocking it takes you out of that engine's answers. Answers and training: one agent doing both jobs, so blocking it costs the answer surface too. Training only: the crawler collects pages to train a model, and no answer surface moves. Collected in bulk, then resold: an archive or a data broker with no surface of its own, whose corpus is sold or given on to other people's training. Blocking anything in the last two has no effect on whether you appear in answers, which is the distinction most advice on this skips.
Can I paste this at the end of my existing robots.txt?: No — it is a whole file, and that is deliberate. The usual snippet appends a named group for every crawler it names, including the ones it is allowing, as a User-agent: GPTBot with an empty Disallow:. That looks like a no-op and is the reverse of one: a crawler that matches a named group stops reading the * group entirely, so your own Disallow: /wp-admin/ quietly stops applying to every bot in the snippet. Put the domain in the box instead and you get your file back with only the AI groups changed.
Why is there no box for Googlebot?: Because Googlebot is Google Search's crawler, and AI Overviews and AI Mode cite only pages Google has indexed: a rule that keeps it out takes you out of those too. This tool only rewrites groups that name one of the crawlers listed above. A Googlebot group in your live file comes back untouched and is counted in the "carried over" line under the output; a group Googlebot shares with GPTBot keeps its rules for Googlebot and loses only the GPTBot line. The Google control that is on the list is Google-Extended, which governs Gemini training and grounding and has no effect on Search ranking.
Where does robots.txt go?: At the root of the domain, answering at example.com/robots.txt. Some platforms — Squarespace, for one — do not let you edit it at all, and on those this file is information rather than an action.
Will blocking a crawler remove content it already has?: No. robots.txt controls future fetching. It is not a deletion request, and pages already collected are not affected by it.

See who the engines recommend instead

The prompts your client loses, who wins them, and the tasks that would change it.