passcite
Start free

WordPress and AI search: what crawlers meet, and what to change

The platform we publish into, and the one where an SEO plugin already serves most of what AI crawlers read. What a WordPress site usually lacks is the page that answers the buyer's question.
AI crawlers

What AI crawlers meet on WordPress

WordPress serves a robots.txt that is not a file: three lines that keep crawlers out of /wp-admin/, and a Sitemap line once the site is public. A real robots.txt in the web root replaces it, and SEO plugins write one or the other — Yoast's file editor writes a real file, while Rank Math edits only the virtual one, so a real file has to go before Rank Math's rules show. Two settings change what crawlers are told without touching the file. 'Discourage search engines' (Settings → Reading) now adds a noindex tag, not a Disallow. On WordPress.com, 'Prevent third-party sharing' (Settings → Reading → Site visibility) adds known AI crawlers to the disallow list. Most often, though, the block is in front of WordPress: a CDN or firewall that refuses AI crawlers before robots.txt is read — Cloudflare has done so by default for AI training crawlers on domains added since July 2025.

robots.txt

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
The file WordPress core serves when the web root has none.

What we read on 246 WordPress sites

WordPress sites, most of them marketing agencies' own, read on the way an AI crawler reads them: robots.txt first, then the homepage as it arrives, before any JavaScript runs.

  • 238 of 246

    have JSON-LD on the homepage

  • 89 of 246

    serve an /llms.txt

  • 1 of 240

    robots.txt files block a crawler that fetches for answers

  • 1,360

    words on the median homepage before any JavaScript runs

On WordPress

What you can change, and where

The changes a report asks for on the site itself, and llms.txt, which it does not.

  • Page titles & meta descriptions

    Possible: Yes
    Who: You, with steps
    Where: Per page through the SEO plugin the site runs (Yoast, Rank Math, SEOPress); core has no meta description field. On WordPress.com, Jetpack's SEO tools on Premium and up.

    See what is live: Website metadata checker

  • JSON-LD schema

    Possible: Yes
    Who: You, with steps
    Where: The SEO plugin's graph covers the basics on every page; anything specific goes in the plugin's schema settings, the theme or a block. Core adds none.

    See what is live: Schema markup validator

  • robots.txt

    Possible: Yes
    Who: You, with steps
    Where: The virtual file until a real one is uploaded to the web root. Yoast's file editor (Yoast SEO → Tools → File editor) writes a real one; Rank Math edits only the virtual one.

    See what is live: robots.txt validator

  • A plain-text file like /llms.txt

    Possible: Yes
    Who: You, with steps
    Where: Yoast: Settings → Site features → AI tools → llms.txt. Rank Math: the LLMS Txt module. Or a file in the web root. WordPress.com's free plan has no plugins, so no file.

    See what is live: llms.txt validator

  • New pages (FAQ, comparison, use case)

    Possible: Yes
    Who: We write and publish
    Where: The normal page editor.

  • A new section on an existing page

    Possible: Yes
    Who: We write; whoever runs the site puts it up
    Where: The page it goes on, in the normal editor, beside the section it follows.
Checklist

Getting a WordPress site ready to be cited

  1. 01

    Read /robots.txt as it is served

    Open example.com/robots.txt itself. A virtual file, a real one and a plugin's editor can disagree, and the served one is what crawlers obey.

  2. 02

    Check the layer in front of WordPress

    A CDN or firewall can refuse GPTBot or ClaudeBot before robots.txt is ever read, and nothing in robots.txt shows it. Check the bot settings of whatever sits in front of the site.

  3. 03

    Tell the answer crawlers from the training ones

    OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch the pages answers cite; GPTBot and ClaudeBot collect for training only. Blocking those two keeps you out of training, not out of answers — but Google-Extended also decides whether Gemini grounds its answers in your pages.

  4. 04

    Turn on the plugin's llms.txt

    Yoast and Rank Math each write one from the site's pages. Pick the pages a buyer should read first.

  5. 05

    Link the new page from the page it is about

    A comparison page linked only from the sitemap gets read late.

Check it yourself

Free, no account. Each reads what your site serves today, the way the crawlers do.

Publishing

What we publish, and what stays yours

An approved new page goes straight onto the site as a post, through an application password you create in WordPress, and one click takes it back down — to a draft, never deleted. It works on a self-hosted site, or on WordPress.com on a paid plan once the site's hosting features are on; the free plan has no application passwords. A section for a page that already exists is never published for you: it is copied in by whoever runs the site. Listings, reviews and site files stay tasks with steps either way.

Passcite on WordPress

  • The reportLive

    It reads public pages, so it works the same on WordPress as anywhere.

  • Pages for your approvalLive

    Written from the site's own words for the prompts it loses.

  • Publishing into WordPressLive

    New pages, once you approve them.

  • ProofLive

    A week after a page goes live, its prompts are asked again.

FAQ

Questions

Does WordPress block AI crawlers by default?
No. Core's robots.txt keeps crawlers out of /wp-admin/ and nothing else. What turns AI crawlers away is usually in front of WordPress — a CDN or firewall — or, on WordPress.com, the 'Prevent third-party sharing' setting.
Do I need a plugin for llms.txt on WordPress?
Core does not write one. Yoast and Rank Math generate it from a setting, or you can upload a file to the web root yourself.
Will my SEO plugin's schema clash with the pages Passcite publishes?
No. A page goes up as a post with its text, and the plugin writes its markup as it does for every post.
Do I need to give Passcite access to my WordPress site?
Not for the report: it reads public pages, the same as any visitor. Publishing into WordPress — a self-hosted site, or WordPress.com on a paid plan with its hosting features on — needs an application password you create there, and you can revoke it at any time.
What does Passcite cost?
The first report is free on a free account: one brand, 10 prompts, three engines, no card. Plans start at $49 a month for 150 credits — a prompt asked on three engines is one, an article 10.
Keep reading

See what the engines say about your WordPress site

The prompts it loses, who was named instead, and what the site never answers.

10 free credits — your first report, 10 prompts across ChatGPT, Claude and Google AI Mode. A free account, no card. About 3 minutes.