passcite
Start free

BigCommerce and AI search: what crawlers meet, and what to change

BigCommerce lets every AI crawler in, but slowly: a group in every Stencil store's robots.txt, out of sight in the settings, asks 42 of them to wait ten seconds between pages.
AI crawlers

What AI crawlers meet on BigCommerce

The robots.txt you edit under Settings → Website → Search Engine Robots is not the whole file. BigCommerce adds rules you cannot see or change there: filter URLs (_bc_fsnf) are disallowed under User-agent: *, and a group of 42 'known high-volume and AI crawlers' — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others — gets Crawl-delay: 10 and is kept only from account, cart, checkout and admin pages. Three things follow. A crawler that obeys the delay reads at most 8,640 URLs a day. Because those crawlers have a group of their own, a Disallow under User-agent: * never reaches them — the filter rule included, so filter URLs stay open to them and count against those 8,640. And to keep one of them out of a path, give it its own group, which BigCommerce merges with its rules crawler by crawler. Claude-SearchBot and Claude-User are not in the group, so they read under *. A Catalyst storefront serves its own robots.txt, without the group.

robots.txt

User-agent: *
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
…
Disallow: /*?_bc_fsnf=1*
Disallow: /*&_bc_fsnf=1*

User-agent: AI2Bot
User-agent: Amazonbot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: Google-Extended
User-agent: OAI-SearchBot
User-agent: PerplexityBot
…
Crawl-delay: 10
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /checkout
Disallow: /admin/
Abridged from a Stencil store's /robots.txt. The second group names 42 crawlers.

What we read on 24 BigCommerce sites

BigCommerce stores, from the platform's theme demos to live shops, read on the way an AI crawler reads them: robots.txt first, then the homepage as it arrives, before any JavaScript runs.

  • 20 of 21

    robots.txt files we could read (of the 24 stores) carry the group with Crawl-delay: 10 — the other is a Catalyst storefront

  • 0 of 24

    serve an /llms.txt

  • 5 of 24

    have JSON-LD on the homepage

  • 585

    words on the median homepage before any JavaScript runs

On BigCommerce

What you can change, and where

The changes a report asks for on the site itself, and llms.txt, which it does not.

  • Page titles & meta descriptions

    Possible: Yes
    Who: You, with steps
    Where: Home page: Settings → Website → Search Engine Optimization. Each product, category, web page and blog post has its own SEO fields; in bulk, the product CSV. There are no title templates.

    See what is live: Website metadata checker

  • JSON-LD schema

    Possible: Yes
    Who: You, with steps
    Where: Cornerstone writes Product, Offer and BreadcrumbList. More goes in Storefront → Script Manager, which places a script on storefront pages, checkout or all pages; one page needs a condition in the script.

    See what is live: Schema markup validator

  • robots.txt

    Possible: Yes
    Who: You, with steps
    Where: Settings → Website → Search Engine Robots. What you save merges with the hidden filter and AI-crawler rules, which show only at /robots.txt.

    See what is live: robots.txt validator

  • A plain-text file like /llms.txt

    Possible: Partly
    Who: You, with steps
    Where: Storefront → Web Pages → a page that contains raw HTML, with llms.txt as its URL — BigCommerce's own workaround. A Catalyst storefront can serve the file directly.

    See what is live: llms.txt validator

  • New pages (FAQ, comparison, use case)

    Possible: Yes
    Who: We write; whoever runs the site puts it up
    Where: Storefront → Web Pages, or a post in the store's blog.

  • A new section on an existing page

    Possible: Yes
    Who: We write; whoever runs the site puts it up
    Where: The page it goes on, in the normal editor, beside the section it follows.
Checklist

Getting a BigCommerce site ready to be cited

  1. 01

    Read the live /robots.txt, not the settings box

    The AI-crawler group and the filter rules appear only in the file the store serves.

  2. 02

    Leave the AI group alone unless you mean to block

    It already lets the crawlers behind ChatGPT, Claude and Perplexity read every product. To keep one out, give it a group of its own: rules under User-agent: * do not reach it.

  3. 03

    Write the category copy the report asks for

    Give each category page two or three paragraphs on who the category is for and how to choose within it: the comparison a buyer asks an assistant for.

  4. 04

    Add schema through Script Manager

    A script can be switched off in one click; a theme edit cannot. Product schema is there already — add Organization and FAQ where the report names them.

  5. 05

    Publish llms.txt as a web page

    Storefront → Web Pages → 'Contain raw HTML entered in the text area below', page URL llms.txt. Then open the live address to see what it serves.

Check it yourself

Free, no account. Each reads what your site serves today, the way the crawlers do.

Publishing

Why we don’t publish to BigCommerce

We publish into WordPress and Webflow, not into BigCommerce. An approved page is copied — rich text for the editor, HTML or Markdown — and pasted in, by you or by the site owner from a link you send them; once either of you gives us its address, we look for it on the site. The table above says where in the control panel each of the other changes lives.

Passcite on BigCommerce

  • The reportLive

    It reads public pages, so it works the same on BigCommerce as anywhere.

  • Pages for your approvalLive

    Written from the site's own words for the prompts it loses.

  • Publishing into BigCommerceCopied in

    Rich text, HTML or Markdown, looked for on the site once it is up.

  • ProofLive

    A week after a page goes live, its prompts are asked again.

FAQ

Questions

Does BigCommerce block AI crawlers?
No, but it slows them. A group in every Stencil store's robots.txt names 42 crawlers, GPTBot and ClaudeBot among them, with Crawl-delay: 10, and keeps them only from account, cart, checkout and admin pages. Not every crawler honours Crawl-delay; one that does reads at most 8,640 URLs a day.
Why don't my robots.txt rules apply to GPTBot on BigCommerce?
GPTBot is named in BigCommerce's own group, and a crawler that finds a group naming it ignores User-agent: * entirely. Give GPTBot a group of your own; BigCommerce merges the rules you save with its own, crawler by crawler.
Can I add an llms.txt to a BigCommerce store?
Not as a file. BigCommerce's workaround is a web page that contains raw HTML, with llms.txt as its URL. A Catalyst storefront can serve the file directly.
Do I need to give Passcite access to my BigCommerce site?
No. The report reads public pages, the same as any visitor, and nothing we do connects to your BigCommerce account.
What does Passcite cost?
The first report is free on a free account: one brand, 10 prompts, three engines, no card. Plans start at $49 a month for 150 credits — a prompt asked on three engines is one, an article 10.
Other platforms

See what the engines say about your BigCommerce site

The prompts it loses, who was named instead, and what the site never answers.

10 free credits — your first report, 10 prompts across ChatGPT, Claude and Google AI Mode. A free account, no card. About 3 minutes.