passcite
Start free

Webflow and AI search: what crawlers meet, and what to change

Webflow serves clean HTML and gives you every file and field an AI crawler reads. Its trap is a toggle that blocks the crawlers answering questions along with the ones training models.
AI crawlers

What AI crawlers meet on Webflow

A Webflow site's robots.txt holds a Sitemap line and nothing else until you write rules, so every AI crawler reads it. Site settings → SEO → Indexing has the controls: a robots.txt field, and two toggles that write rules for groups of bots — 'Search engine crawlers' (Googlebot, Bingbot, OAI-SearchBot) and 'AI bots' (GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, CCBot and others). Both allow by default. Turning 'AI bots' off also blocks ChatGPT-User and Claude-User, the fetches an assistant makes while it answers someone, so the toggle that sounds like 'no training' takes you out of live answers too. The staging address (*.webflow.io) can be indexed until 'Staging indexing' is turned off.

robots.txt

Sitemap: https://example.com/sitemap.xml
The whole file on a Webflow site with no rules written.

What we read on 35 Webflow sites

Webflow sites, most of them agencies' and software companies' own, read on the way an AI crawler reads them: robots.txt first, then the homepage as it arrives, before any JavaScript runs.

  • 17 of 35

    serve an /llms.txt

  • 27 of 35

    have JSON-LD on the homepage, every block added by the site

  • 0 of 34

    robots.txt files block a crawler that fetches for answers

  • 1,357

    words on the median homepage before any JavaScript runs

On Webflow

What you can change, and where

The changes a report asks for on the site itself, and llms.txt, which it does not.

  • Page titles & meta descriptions

    Possible: Yes
    Who: You, with steps
    Where: Pages panel → page settings → SEO settings. On a CMS template, fields build the title and description for every item.

    See what is live: Website metadata checker

  • JSON-LD schema

    Possible: Yes
    Who: You, with steps
    Where: Page settings → Schema markup: paste it or have Webflow AI write it; on a collection template it can read the item's fields (not reference fields). Paid site plan or workspace.

    See what is live: Schema markup validator

  • robots.txt

    Possible: Yes
    Who: You, with steps
    Where: Site settings → SEO → Indexing: the robots.txt field and the two crawler toggles. Paid site plan or workspace.

    See what is live: robots.txt validator

  • A plain-text file like /llms.txt

    Possible: Yes
    Who: You, with steps
    Where: Site settings → SEO → LLMs.txt → Upload file (UTF-8, under 100 KB), then publish. Served on the custom domain only.

    See what is live: llms.txt validator

  • New pages (FAQ, comparison, use case)

    Possible: Yes
    Who: We write and publish
    Where: An item in a CMS collection — the blog, usually — laid out by the collection's template; that is how we publish them too.

  • A new section on an existing page

    Possible: Yes
    Who: We write; whoever runs the site puts it up
    Where: The page it goes on, in the normal editor, beside the section it follows.
Checklist

Getting a Webflow site ready to be cited

  1. 01

    Check both crawler toggles

    Site settings → SEO → Indexing. 'AI bots' off blocks the fetches assistants make while they answer, not only training.

  2. 02

    Turn off staging indexing

    Otherwise the *.webflow.io copy can be crawled beside your domain.

  3. 03

    Put schema on the collection template

    One Schema markup field on the template covers every item, read from its fields.

  4. 04

    Upload llms.txt, then read it on the domain

    It is not served on webflow.io, so open the custom domain's /llms.txt after publishing.

  5. 05

    Publish, then run the report again

    Webflow publishes instantly, which makes it one of the best platforms for a clean before and after.

Check it yourself

Free, no account. Each reads what your site serves today, the way the crawlers do.

Publishing

What we publish, and what stays yours

An approved new page goes into one of the site's CMS collections — the blog, usually — through an API token you create in the site's settings, and the collection's template lays it out. One click takes it back down: unpublished, kept in the collection as a draft, never deleted. A section for a page that already exists is never published for you: it is copied in by whoever runs the site. Listings, reviews and site files stay tasks with steps either way.

Passcite on Webflow

  • The reportLive

    It reads public pages, so it works the same on Webflow as anywhere.

  • Pages for your approvalLive

    Written from the site's own words for the prompts it loses.

  • Publishing into WebflowLive

    New pages, once you approve them.

  • ProofLive

    A week after a page goes live, its prompts are asked again.

FAQ

Questions

Does Webflow block AI crawlers?
Not by default. The 'AI bots' toggle in Site settings → SEO → Indexing blocks GPTBot, ClaudeBot, PerplexityBot and others when you turn it off — and with them ChatGPT-User and Claude-User, which fetch pages while an assistant answers.
Does Webflow write an llms.txt?
No. You upload one at Site settings → SEO → LLMs.txt, and it is served at the custom domain's root, not on webflow.io.
Does Webflow add structured data?
Not by itself. Page settings → Schema markup takes JSON-LD you paste or Webflow AI writes, and on a collection template it can read the item's fields. It needs a paid site plan or workspace.
Do I need to give Passcite access to my Webflow site?
Not for the report: it reads public pages, the same as any visitor. Publishing into Webflow needs an API token you create in the site's settings, and you can delete it there at any time.
What does Passcite cost?
The first report is free on a free account: one brand, 10 prompts, three engines, no card. Plans start at $49 a month for 150 credits — a prompt asked on three engines is one, an article 10.
Other platforms

See what the engines say about your Webflow site

The prompts it loses, who was named instead, and what the site never answers.

10 free credits — your first report, 10 prompts across ChatGPT, Claude and Google AI Mode. A free account, no card. About 3 minutes.