Faceted Navigation Creates More URLs Than You Think

A catalog with 4,000 products can easily expose tens of millions of crawlable URLs. Nobody decides to do this. It happens because filters compose, and composition is multiplicative while your product count is additive.

The damage is not a penalty. Google's large-site crawl budget guidance names faceted navigation as a common source of low-value URLs, and the mechanism is straightforward: crawl capacity is finite, so requests spent re-fetching ?color=blue&size=m&sort=price&page=7 are requests not spent discovering your new products or picking up your updated category copy. The symptom shows up later as slow indexing of pages you care about, not as a ranking drop on the pages you don't.

Why the URL count explodes faster than the facet count

Count the state space, not the links in the sidebar. Each facet with k selectable values contributes k+1 states per URL — every value plus "not applied." With single-select facets, the combinations for one category are the product of those states.

Six facets with five values each is 6⁶ = 46,656 combinations. Not six facets' worth of pages. Multiply by 20 categories and you are at 933,120 before anything else happens. Then the multipliers stack:

• Sort order. Four sort options quadruple whatever you had.
• Pagination. A 60-page result set on a filtered view multiplies again.
• Parameter order. If your application does not normalize order, a combination with j applied facets can be reached through j! distinct URL strings. Google treats each unique URL as a distinct URL; it has no rule that says query parameters are order-insensitive.
• Multi-select. If a facet accepts several values at once, its state count stops being k+1 and becomes 2^k.
• Continuous ranges. A price slider that writes min and max into the URL creates an effectively unbounded space. This is the one that turns a large crawl space into an infinite one.

The practical test: crawl your own site with a crawler configured to follow parameters, cap it at a few hundred thousand URLs, and see whether it terminates. If it does not, you have an unbounded space, and no amount of rel=canonical will fix a crawler that never runs out of new URLs to fetch.

What Google actually does with these URLs

Some documented behavior worth being precise about:

• rel=canonical is a hint, not a directive. Search Console's "Duplicate, Google chose a different canonical" state exists because Google overrides it regularly.
• Both noindex and rel=canonical require the URL to be crawled. If you block a URL in robots.txt, neither signal is ever read.
• A robots.txt-blocked URL can still be indexed with no content if enough links point at it — Search Console reports this as "Indexed, though blocked by robots.txt."
• Google retired Search Console's URL Parameters tool in 2022. There is no longer a place to declare "this parameter does not change content." Normalization is now your job, server-side.

The reasonable inference — not documented, but consistent with what crawl logs show on large catalogs — is that Google learns patterns and throttles low-value parameter spaces on its own, eventually. You do not control the timeline, and "eventually" on a site with an unbounded crawl space can be long enough to delay indexing of new inventory.

Which facets deserve to be indexable

A facet combination earns indexation when it satisfies all four of these, not three:

1. Real query demand for the phrase the page would target ("waterproof hiking boots," not "boots sorted by price descending").
2. Enough stable inventory that the page will not be empty next month. Thin or zero-result filter pages are a liability.
3. Differentiated content — a distinct H1, title, and ideally a paragraph of copy, not just a re-sorted grid.
4. A stable URL you are willing to keep and redirect if the taxonomy changes.

In practice this means a whitelist: category plus one approved facet, occasionally category plus two for a handful of high-demand combinations. Everything else is functional UI, not a landing page.

What each control mechanism actually does

The decision rule follows from the table. Bounded and potentially valuable (a facet you might promote later, or one that attracts external links): keep it crawlable, self-referencing or canonicalized, and let Google consolidate. Bounded but worthless (sort orders, view toggles): normalize away with a 301 to the canonical form. Unbounded (price ranges, multi-select stacks, session or tracking parameters): block the pattern in robots.txt and stop emitting hrefs to it.

# robots.txt — block unbounded combinations, keep single-facet pages crawlable
Disallow: /*?*price_min=
Disallow: /*?*sort=
Disallow: /*&*color=      # any color param that is not first = multi-facet

That last line only works if your application enforces a canonical parameter order. Which is the point: the rules are cheap once normalization exists, and unmaintainable without it.

Normalize the URL grammar before anything else

Define one canonical string form and 301 everything else to it:

• Fixed alphabetical parameter order.
• Lowercase parameter names and values.
• Drop empty parameters (?color=), drop defaults (?sort=relevance, ?page=1).
• Deduplicate repeated keys.
• Strip known tracking parameters server-side or at the CDN edge.

This single change often removes the majority of a crawl space, because most of the bulk is permutations and defaults rather than genuinely distinct filter states. Pagination stays crawlable with self-referencing canonicals — Google announced in 2019 that it no longer uses rel=next/prev for indexing, so paginated pages need to be reachable by ordinary links.

How to measure whether it is working

Server logs are the only source that shows what crawlers actually requested. Verify Googlebot by reverse DNS before counting anything, then bucket verified requests by URL pattern:

# share of verified Googlebot requests that carry a query string
awk '{n++; if ($7 ~ /\?/) q++} END {printf "%.1f%%\n", 100*q/n}' googlebot-verified.log

Track three numbers monthly:

• Parameterized share of crawl requests. Should fall after normalization ships.
• Discovery vs. refresh split in Search Console's Crawl Stats report. A high discovery share on a stable catalog means the crawler is still finding new junk URLs.
• Time from publish to first crawl for new products. This is the business-relevant outcome; crawl-waste percentages are a means to it.

In the Pages report, watch "Crawled — currently not indexed" and "Duplicate, Google chose a different canonical." Rising counts on filter URLs mean your canonical hints are being read and rejected, which is your cue to stop hinting and start blocking or removing the URLs entirely.

What to do first

1. Crawl your own faceted templates with parameters followed and a hard URL cap. Note whether it terminates. That tells you bounded vs. unbounded.
2. Ship parameter normalization with 301s: fixed order, no defaults, no empties. Measure the parameterized share of Googlebot requests before and after.
3. Write the whitelist of indexable facet pages using the four criteria above. Everything on it gets unique titles, internal links, and sitemap entries.
4. For unbounded patterns, remove the hrefs and block the pattern in robots.txt. Do not rely on noindex for a space that is infinite — you would be paying crawl budget forever to deliver the instruction.
5. Re-check discovery-to-first-crawl time on new products in 30 and 60 days. That is the metric that justifies the work.