Sitemap generator: what you have, against what is linked
Two things at once: it looks for the sitemap the site already publishes, and it follows links from the page you give it — same host only, up to 50 pages or 20 seconds, whichever comes first. 6 at a time, which is brisk without being rude to the site.
The file is rarely the answer
Most sites that need a sitemap already have one. WordPress with Yoast or Rank Math publishes an index of several; Shopify and Squarespace publish one and regenerate it on every change. A tool that crawls 50 pages and hands you XML without ever looking for that file is offering a downgrade in the shape of a deliverable — and a sitemap that omits pages is not neutral, it is a statement that the omitted ones do not exist.
So the first thing this does is go and find it. What it hands back when one is there is not a replacement file: it is the list of pages the crawl reached by following links that your sitemap does not mention. That list is usually short, and when it is not, it is usually a plugin that quietly stopped including a post type.
Orphan pages are the other direction
The comparison runs both ways, and the other direction is worth reading too: open the crawl list next to the pages you believe exist. Anything on your list and not in the crawl is orphaned — reachable only if you already know the URL.
It happens to exactly the pages that matter here. Someone writes a comparison page or a detailed FAQ, publishes it, and links it from nowhere. It is in the CMS, it is in the sitemap the plugin generates, and no crawler following links will ever arrive at it with any weight behind it.
Then check whether it can be read at all
A sitemap gets a page found. The AI crawler checker tells you whether the engines are allowed to fetch it, and the metadata checker tells you whether there is any text in it before JavaScript runs.
Questions
- Why 50 pages?
- Because past that the honest answer is that your platform already generates one. A truncated sitemap is worse than no sitemap: it tells a crawler that the pages it left out do not exist, and it looks complete while doing it. If you hit the limit, use your CMS's own sitemap.
- It says my site already has a sitemap. Should I replace it?
- Almost certainly not, and that is why the tool looks for it first. A sitemap generated by your CMS knows about every post type and updates itself; this one knows about 50 pages reachable by following links. Replacing a 400-URL file with a 50-URL file tells crawlers the other 350 pages are not worth having. Use the comparison — the list of linked pages missing from your sitemap — and leave the file alone.
- Where does it look for the existing one?
- The Sitemap: lines in your robots.txt first, because that is the only place a site states it rather than us guessing. Then /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml. It will only fetch a sitemap on the same domain you typed. If yours lives somewhere else, the robots.txt line is what points anything at it — including us.
- Where does the file go?
- At the root of your domain, at /sitemap.xml, and referenced from robots.txt with a Sitemap: line. Submit it in Google Search Console and Bing Webmaster Tools too — the robots.txt line is discovery, the submission is a nudge.
- Does a sitemap help with AI search?
- Indirectly and modestly. It helps crawlers find pages; it does nothing to make a page worth citing. If you are here because an engine is not recommending your client, this is not the thing that will change that.
- Some pages are missing.
- The crawl follows links. A page nobody links to will not be found — which is itself worth knowing, because a page nothing links to is a page crawlers struggle to reach as well.
See who the engines recommend instead
The prompts your client loses, who wins them, and the tasks that would change it.