A category page with five filters at eight options each has 32,768 possible states. Add sorting and pagination and the number gets worse. If each state is a crawlable URL, you have built an effectively infinite site on top of a finite catalog.
This is the most common source of severe crawl waste in e-commerce and directory sites, and it is entirely preventable at design time.
Decide which combinations deserve to be pages
Not every filter state is worthless. Some correspond to real search demand and should be indexable landing pages with their own titles and copy.
The test is search demand plus inventory. "Women's waterproof hiking boots" is a query people type and a set with enough products to justify a page. "Women's waterproof hiking boots, size 7.5, blue, under $200, sorted by newest" is neither.
Practically, this produces three tiers:
Tier one — indexable landing pages. One or two filters deep, matching real queries, with sufficient inventory. Given unique titles, descriptions, and ideally some intro copy. Linked from navigation. Included in the sitemap.
Tier two — crawlable but not indexed. Combinations users need but nobody searches for. Reachable, functional, marked noindex, follow so link equity still flows through them.
Tier three — not crawlable at all. The combinatorial tail. Users can reach these by interacting; crawlers cannot, because no crawlable link to them exists.
That third tier is where the actual solution lives, and it is the one most implementations skip.
Making tier three structurally invisible
The reliable method is not to emit links to those states. Concretely:
- Filter controls that update results via JavaScript without producing an
<a href>per combination. - Filter state in the URL fragment (
#) rather than the query string, since fragments are not sent to the server and are not treated as distinct URLs. - Or a
POST-based filter form, which crawlers do not submit.
Any of these means the combinatorial space is never discovered, which is far more robust than discovering it and then trying to suppress it.
The counter-argument is that filter states then cannot be shared or bookmarked. That is a genuine trade-off. The usual resolution is to make tier-one combinations real, shareable URLs and accept that the deep tail is a session-level UI state.
When links already exist
For sites where crawlable filter links are already deployed and rewriting the front end is not immediate:
Canonical tags to the parent category. Filter states declare the unfiltered category as canonical. A hint, not a directive — Google may disregard it, particularly when the filtered page's content differs substantially.
noindex, follow. Stronger for keeping pages out of the index. Requires crawling to be seen, so it reduces index bloat but not crawl waste.
robots.txt disallow on parameter patterns. The only option that actually reduces crawl. Blocks discovery of anything matching the pattern.
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*color=
Disallow: /*&price_min=Two cautions. Disallowed pages cannot have their noindex read, so do not apply both to the same URL. And disallowed URLs can still be indexed from external links, appearing as bare URLs with no snippet.
Consistent parameter ordering. If your application emits ?color=blue&size=7 and ?size=7&color=blue for the same state, you have doubled the URL space for free. Normalize the order server-side.
Pagination
Google no longer uses rel="next" and rel="prev" for indexing. Current guidance is straightforward:
- Each paginated page gets a self-referencing canonical, not a canonical to page one. Page two is not a duplicate of page one.
- Paginated pages should be crawlable so products deeper in the set get discovered.
- Do not
noindexpaginated pages if they are the only path to deeper items — you block the crawl path to your own catalog. - "View all" pages are fine if performance allows, and can be canonical if they genuinely contain everything.
Diagnosing what you have
Server logs answer this directly. Extract crawled URLs, count distinct query strings, and count requests hitting parameterized URLs versus clean ones. If a large share of bot requests carry parameters, you have found where your crawl budget goes.
Then compare against your sitemap. Parameterized URLs being crawled heavily while sitemap URLs go untouched is the specific pattern that means new products are not being discovered because filter combinations consumed the budget.
That comparison takes ten minutes and, on a large catalog site, frequently explains a discovery problem the team has been attributing to something else entirely.