passcite
Start free

GitHub Pages and AI search: what crawlers meet, and what to change

Static files and nothing in the way: every change is a commit. The catch is where robots.txt has to live.
AI crawlers

What AI crawlers meet on GitHub Pages

GitHub Pages serves no robots.txt unless you add one, so nothing is blocked. A robots.txt only counts at the root of the host: on a project site (owner.github.io/repo) a file at /repo/robots.txt is ignored, and control needs the owner.github.io repository or a custom domain. On Jekyll, the jekyll-sitemap plugin writes sitemap.xml and, when there is no robots.txt, one with just the Sitemap line. Jekyll leaves out folders whose names start with a dot or an underscore — .well-known included — unless the config lists them under include.

robots.txt

Sitemap: https://example.com/sitemap.xml
What jekyll-sitemap writes when the site has no robots.txt.
On GitHub Pages

What you can change, and where

The changes a report asks for on the site itself, and llms.txt, which it does not.

  • Page titles & meta descriptions

    Possible: Yes
    Who: You, with steps
    Where: Front matter or the layout's head; on Jekyll, jekyll-seo-tag writes them.

    See what is live: Website metadata checker

  • JSON-LD schema

    Possible: Yes
    Who: You, with steps
    Where: jekyll-seo-tag adds it, or an include in the layout.

    See what is live: Schema markup validator

  • robots.txt

    Possible: Yes
    Who: You, with steps
    Where: A file at the host's root: on a project site, that means the owner.github.io repository or a custom domain.

    See what is live: robots.txt validator

  • A plain-text file like /llms.txt

    Possible: Yes
    Who: You, with steps
    Where: A file in the repository. A project site publishes at /repo-name/, and llms.txt v2 defines exactly that case: a file at a subpath covers the pages under it — GitHub Pages is the example the spec cites.

    See what is live: llms.txt validator

  • New pages (FAQ, comparison, use case)

    Possible: Yes
    Who: We write; whoever runs the site puts it up
    Where: The normal page editor.

  • A new section on an existing page

    Possible: Yes
    Who: We write; whoever runs the site puts it up
    Where: The page it goes on, in the normal editor, beside the section it follows.
Checklist

Getting a GitHub Pages site ready to be cited

  1. 01

    Put robots.txt where crawlers read it

    The host's root. A project site's own folder does not count.

  2. 02

    Add jekyll-seo-tag

    It is not on by default; once in the config it writes the title, description, canonical, Open Graph and JSON-LD.

  3. 03

    List dot-folders under include

    Jekyll skips .well-known and anything else that starts with a dot or an underscore.

  4. 04

    Set canonical tags deliberately

    There are no server-side redirects, so the canonical tag is the strongest consolidation signal you have.

Check it yourself

Free, no account. Each reads what your site serves today, the way the crawlers do.

Publishing

Why we don’t publish to GitHub Pages

The changes are commits, and a pull request is already the review step every article here waits for. An approved page comes as Markdown or HTML to commit; once it is deployed and you give us its address, we look for it on the site.

Passcite on GitHub Pages

  • The reportLive

    It reads public pages, so it works the same on GitHub Pages as anywhere.

  • Pages for your approvalLive

    Written from the site's own words for the prompts it loses.

  • Publishing into GitHub PagesCopied in

    Rich text, HTML or Markdown, looked for on the site once it is up.

  • ProofLive

    A week after a page goes live, its prompts are asked again.

FAQ

Questions

Where does robots.txt go on GitHub Pages?
At the root of the host. On a project site, owner.github.io/repo, crawlers ignore /repo/robots.txt; use the owner.github.io repository or a custom domain.
Can a GitHub Pages project site have an llms.txt?
Yes. llms.txt v2 lets a file at a subpath cover the pages under it, and the spec's own example is a GitHub Pages project site.
Can I redirect duplicate URLs on GitHub Pages?
Not with a 301; it serves static files. jekyll-redirect-from writes meta-refresh pages, and a canonical tag on each page is the stronger signal.
Do I need to give Passcite access to my GitHub Pages site?
No. The report reads public pages, the same as any visitor, and nothing we do connects to your GitHub Pages account.
What does Passcite cost?
The first report is free on a free account: one brand, 10 prompts, three engines, no card. Plans start at $49 a month for 150 credits — a prompt asked on three engines is one, an article 10.
Other platforms

See what the engines say about your GitHub Pages site

The prompts it loses, who was named instead, and what the site never answers.

10 free credits — your first report, 10 prompts across ChatGPT, Claude and Google AI Mode. A free account, no card. About 3 minutes.