How Googlebot Discovers Item 4,000 in a Paginated List

Most pagination advice is still written as if there were a markup tag that tells Google "these pages form a series." There isn't one that Google uses. Google announced in 2019 that it had not used rel="next" and rel="prev" for years, and its guidance since has been to treat paginated pages as ordinary pages that link to each other.

That changes the nature of the problem. Once the series markup is gone, pagination is purely a question of reachability: how many links does Googlebot have to follow to arrive at the URL for item 4,000, and how much crawl priority does a URL that deep inherit? Everything else — canonicals, noindex, infinite scroll, sort order — matters only insofar as it changes that answer.

Why the retirement of rel=next/prev matters more than people think

When the markup was believed to work, the mental model was that Google assembled the series, understood page 7 as part of a set, and consolidated signals across it. Under the current model, page 7 of your category is just a URL with some links on it. It gets crawled on its own merits, it can rank on its own merits, and it passes links like any other page.

Google has documented that it discovers URLs primarily through links and sitemaps, and that crawl scheduling is priority-based rather than exhaustive. Google has not published the weights in that scheduler. The reasonable inference — consistent with what you can observe in any large site's access logs — is that URLs that are many links from your entry points and receive few internal links get crawled infrequently, and the item URLs that hang off them are discovered slowly or not at all.

Keeping rel="next"/rel="prev" in your markup does no harm. It just won't fix a depth problem.

The arithmetic of pagination depth

Take a listing with 4,000 items at 20 per page: 200 paginated pages. The maximum click depth of the deepest page depends entirely on the largest jump your pagination UI offers.

These are arithmetic, not measurements: with a maximum jump of k, and shortcuts to both ends, the worst-case depth is roughly N / 2k. The lesson is that the fix is not "add more links," it's "increase the maximum jump." A window of ±3 feels generous to users and barely dents crawl depth.

The opposite extreme — rendering all 200 numbered links on every page — flattens depth to 1 but spreads internal link weight thinly and makes every paginated page look nearly identical in its link graph. For long series, coarse anchors (every 10th page) plus a local window is the better trade: depth collapses to a handful of hops without 200 links in the footer of every page.

What infinite scroll changes

Googlebot does not scroll. Google's own JavaScript SEO guidance for infinite scroll is to back it with crawlable, individually addressable paginated URLs and to update the URL with the History API as the user scrolls. If your next batch of items only loads on a scroll event and never produces a link in the DOM, those items have no discovery path from the listing at all.

The pattern that satisfies both audiences is to render real anchors and enhance them:

<div id="results"><!-- items 1-20 --></div>
<nav class="pagination">
  <a href="/shoes?page=2" rel="next">Next</a>
  <a href="/shoes?page=10">10</a>
  <a href="/shoes?page=200">200</a>
</nav>

JavaScript can intercept the click, fetch the batch, and push the new URL. Googlebot, which fetches rather than clicks, follows the href.

The canonical and noindex mistakes that slow discovery

Two widespread patterns actively suppress the crawl path you just built.

• Canonicalizing page 2…N to page 1. These pages are not duplicates of page 1 — they contain different items. Google may ignore the canonical, but when it accepts it, the duplicate URL loses its own crawl priority and gets refetched less often. The links on it are still seen, but seen less frequently. Deep item discovery slows accordingly.
• noindex on page 2…N. Google representatives have said that pages kept out of the index long term tend to be crawled less, and that Google may eventually stop following links from them. If your paginated pages are the only path to item pages, that is the worst place to apply noindex.

The safe default is boring: paginated pages are index, follow, each with a self-referencing canonical, each with a distinct title where it makes sense. If you don't want them competing in search, improve the hub page rather than suppressing the series.

One more, less-discussed failure: unstable sort order. If your default listing is "newest first" and you publish frequently, every item shifts pages over time. Googlebot re-fetches page 40 and finds an entirely different set of items than last time. That produces churn with no gain in coverage. Stable alternatives — date-sliced archives (/2024/03/), alphabetical or ID-range hubs — give each item a fixed address in the crawl graph.

How to measure whether deep pages are actually reached

Don't argue about depth in the abstract. Three measurements settle it.

1. Crawl your own site with a depth cap. Start at the homepage, limit to depth 5, and compute the share of known item URLs reached. That single number — item coverage at depth ≤ 5 — is the one to track before and after a pagination change.
2. Bucket Googlebot hits by page number in your logs. Extract the page parameter and count verified Googlebot requests per bucket:

grep 'Googlebot' access.log \
  | grep -oE 'page=[0-9]+' \
  | awk -F= '{b=int($2/10)*10; c[b]++} END {for (i in c) print i, c[i]}' \
  | sort -n

A steep decay after the first few buckets tells you the deep tail is effectively uncrawled, regardless of what your pagination UI claims to link.

1. Inspect a sample of deep item URLs. Pull 50 item URLs that only appear late in the series and check them via the URL Inspection API. A cluster of Discovered – currently not indexed is a crawl-priority symptom, not a content-quality symptom, and the fix is structural.

What to do

• Count your series length. Under ~20 pages, render all page numbers and stop worrying.
• Over that, add coarse anchors so the maximum jump is at least N/20, and verify the resulting depth with a capped crawl rather than by eye.
• Make every paginated page self-canonical and indexable-and-followable; remove page-1 canonicals and blanket noindex from the series.
• If your listing sorts by recency, add a stable secondary access path — date, alphabet, or ID range — so item URLs keep a fixed position in the link graph.
• Keep item URLs in sitemaps, but treat sitemaps as a discovery aid, not a substitute for internal links. URLs with no internal links tend to be crawled slowly no matter how they were found.
• Re-run the depth-capped crawl and the log bucketing a month after the change. If item coverage at depth ≤ 5 hasn't moved, the pagination change didn't do what you assumed.