Three numbers describe your site's relationship with a search engine: how many URLs you publish, how many get crawled, and how many get indexed. The ratios between them diagnose more problems than any individual metric.
Getting the three numbers
Published. The count of canonical, indexable URLs you intend to have in the index. From your sitemap, or from a crawl of your own site with non-canonical and noindexed URLs excluded.
Crawled. From server logs, filtered to verified search engine bots, deduplicated by URL, over a fixed window — 30 days works.
Indexed. From Search Console's Page Indexing report, the "Indexed" count. Not from a site: query, which returns a rough estimate that varies between runs and should not be used for anything quantitative.
Reading the ratios
Crawled ÷ Published below ~0.8. Google is not reaching a meaningful share of your pages. Almost always an internal linking or architecture problem: orphaned pages, content buried too deep, pagination that does not expose deeper items, or navigation that only exists after JavaScript renders.
Indexed ÷ Crawled below ~0.7. Google is reaching your pages and declining to index them. This is a quality or duplication judgment, not a technical one, and it is the harder problem. Check the Page Indexing report's exclusion reasons — "Crawled – currently not indexed" and "Discovered – currently not indexed" are the two that matter, and they mean different things.
Crawled substantially exceeding Published. Index bloat. Google is spending crawl on URLs you did not intend to exist.
Where the bloat comes from
The generators are predictable:
- Faceted navigation. Filter combinations multiply combinatorially. Three filters with ten options each produce a thousand URLs from one category page.
- Internal search results pages. If they are linked and crawlable, they generate unbounded URL space.
- Session IDs and tracking parameters in links. Each variant is a distinct URL.
- Pagination without limits. Archives that paginate indefinitely, including empty pages past the end.
- Staging and development environments. A crawlable staging subdomain that duplicates production is the most damaging version of this, because it also creates genuine duplicate content.
- Tag and category archives in CMSs that generate them automatically for every tag ever used, including single-post tags.
Fixing it, in order of effectiveness
Stop linking to them. By far the most effective and the most commonly skipped. Google finds URLs primarily through links. If your faceted navigation does not emit crawlable links to the combinations you do not want indexed — using buttons that fire JavaScript rather than anchor tags, for example — the combinatorial space never gets discovered.
Return the right status. Pages that should not exist should return 404 or 410, not 200 with an empty result. A soft 404 — an empty results page returning 200 — is worse than a real 404, because Google has to spend judgment deciding what it is.
Canonical tags for genuine duplicates. Correct when the alternate URL is a legitimate variant of a canonical page. Not a fix for pages that should not exist at all — a canonical is a hint, and Google may ignore it.
noindex for pages that must exist but should not be indexed. Internal search results, thin tag archives, filter pages that users need. Note that noindex requires the page to be crawlable to be seen, which is why the next item is often misapplied.
robots.txt disallow for crawl-level exclusion. Prevents crawling, which prevents Google from seeing a noindex on the same URL. Use it for genuinely large URL spaces you want excluded from crawl entirely — parameter patterns, internal search paths — and understand that a disallowed URL can still appear in results if it is linked from elsewhere.
The noindex versus robots.txt confusion is the most common technical SEO error in this area: applying both to the same URL means Google never crawls it, never sees the noindex, and may index it anyway based on external signals.
Why bloat costs you
The mechanism is not mystical. Crawl capacity spent on worthless URLs is capacity not spent on valuable ones. On a small site with abundant crawl budget, bloat is mostly cosmetic. On a large site, it is the difference between new content being discovered in hours and being discovered in weeks.
There is a second effect worth naming carefully: a site where a large fraction of crawled URLs are thin or duplicative gives a search engine a worse overall impression of quality. Google has described site-wide quality assessment in general terms without publishing a mechanism, so the honest framing is that this is a reasonable inference rather than a documented rule.
The cadence
Compute the three numbers quarterly. The ratios move slowly and predictably in a healthy site, and a step change in any of them is a specific event with a specific cause — a template change, a CMS upgrade, a new filter feature — that you can find and address while it is still recent.