Segmented XML Sitemaps Turn Indexing Into Measurable Data

Most sites submit one sitemap index, or worse, one flat sitemap.xml with every URL in it. Then someone opens the Pages report in Search Console, sees "412,318 not indexed," and has no way to tell whether the problem is the paginated archive, the tag pages, the 2019 blog posts, or the money pages.

That is a self-inflicted measurement failure. Search Console lets you filter indexing status by an individual submitted sitemap. That filter is the only free, site-wide, per-segment indexation instrument Google gives you — and its resolution is entirely determined by how you chose to split your sitemap files. Sitemaps are a submission mechanism, but they are also a measurement schema.

Why the filter matters more than the sitemap itself

Google has been consistent that sitemaps help with discovery and do not guarantee indexing or affect ranking. If your site is small and well linked, submitting a sitemap changes very little about crawling. So the discovery argument is weak for most established sites.

The reporting argument is not. In Search Console's Pages report you can scope the entire breakdown — indexed, Crawled — currently not indexed, Discovered — currently not indexed, Duplicate, Google chose different canonical, soft 404 — to a single sitemap file you submitted. Every file you submit is a cohort you can measure. Every file you don't submit separately is a cohort you have permanently blurred.

One important detail: the filter works on sitemaps submitted in the Search Console UI (or via the API), not merely referenced by a Sitemap: line in robots.txt. Reference them in robots.txt for discovery, but submit them explicitly if you want the per-file reporting.

How to choose segment boundaries

Segment along the dimensions where you would actually take different action. Three schemes cover most sites, and they compose — a sitemap index can point to sitemaps built on all three.

Google documents a limit of 50,000 URLs or 50MB uncompressed per sitemap file, with up to 50,000 sitemaps per index file. Those are ceilings, not targets. For measurement, smaller is better: a 50,000-URL file gives you one number, while ten 5,000-URL files give you ten. Files in the 2,000–10,000 URL range are a reasonable balance between resolution and maintenance overhead.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemaps/product-active-01.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemaps/product-active-02.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemaps/product-discontinued.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemaps/category.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemaps/editorial-2025.xml</loc></sitemap>
</sitemapindex>

Submitting the index file alone is enough for Google to fetch the children, and in most cases the children appear individually in the Sitemaps report and become filterable. If a child sitemap doesn't show up as its own row, submit it directly.

The rule that makes the data trustworthy: one URL, one sitemap

Nothing in the sitemaps protocol stops you from listing a URL in two files. For measurement purposes, don't. Once a URL is in both sitemap-blog.xml and sitemap-priority.xml, its status is counted in both segments and your denominators stop meaning anything.

Two more hygiene rules that follow from the same logic:

• Only canonical, indexable URLs go in sitemaps. A noindex page listed in a sitemap will show up as an excluded row and pollute the segment's rate. Google has said sitemaps should contain the canonical version of each URL.
• Keep lastmod honest. If your CMS stamps today's date on every URL on every build, lastmod becomes noise and Google is known to discount it. Reflect actual content change, or omit the field.

What to compute once the segments exist

For each sitemap, the headline metric is the indexation rate: indexed URLs divided by submitted URLs. The number itself is less interesting than its shape across segments and over time.

1. Rate by template. If products sit at 94% and category pages at 41%, you have a template problem, not a site problem. Look at what the 59% resolve to — duplicate canonicals usually mean faceted or paginated variants collapsing together.
2. Reason mix, not just the rate. Discovered — currently not indexed concentrated in one segment points to crawl scheduling and capacity. Crawled — currently not indexed points at quality or duplication assessment. Duplicate, Google chose different canonical points at your URL and canonical design.
3. Cohort curves. With date-based sitemaps, plot indexation rate of each cohort by age. A healthy site shows new cohorts climbing steeply in the first week or two and plateauing. A flattening curve across successive cohorts is an early warning that shows up long before traffic moves.
4. Segment drift. Track the rate weekly. A single segment sliding from 90% to 70% over a month is a far sharper signal than the site-level total, which averages the drop into invisibility.

One caveat on precision: Search Console's Pages data lags by a couple of days and the counts are Google's own rounding of a moving target. Treat differences of a few percent as noise and multi-point moves as signal. If you need per-URL truth for a specific list, the URL Inspection API returns live coverage state for individual URLs, subject to a per-property daily quota in the low thousands — enough to spot-check a segment, not to audit a large catalog.

What this cannot tell you

Sitemap segmentation measures indexing, not demand or quality. A segment at 98% indexed can still get zero clicks. It also cannot force indexing: if a template is genuinely thin, splitting its sitemap only makes the failure legible faster.

It is also blind to URLs you never submitted. If Google is indexing parameterized variants you didn't list, those never appear in any segment. Cross-check with the unfiltered Pages report and with server logs — the gap between "all known pages" and the union of your sitemaps is itself a useful number.

What to do this week

• Split your existing sitemap by template first. That is usually a change to one generation script and gives the largest interpretive gain.
• Submit each child sitemap in Search Console explicitly, not only via robots.txt, and keep the index file referenced in robots.txt.
• De-duplicate: assert that every URL appears in exactly one file, and add that assertion as a test in your build.
• Add date-cohort sitemaps for new content going forward, and record the indexed/submitted pair per sitemap weekly into a spreadsheet or table. The time series is worthless until you start collecting it, and Search Console won't collect it for you.
• Before you blame the algorithm for a traffic drop, check whether one segment's indexation rate moved first.