Infinite Scroll That Search Engines Can Actually Crawl

Infinite scroll is a fine user experience and a terrible crawler experience, and the reason is mechanical, not mysterious. A crawler fetches a URL, renders it once, and records what it got. It does not scroll. Everything your page loads in response to a scroll event simply never happens during a crawl.

The result is predictable: a category with 800 products where search engines can see 24 of them, and every item past the first viewport-load reachable only through the sitemap — discovered, maybe, but stripped of the internal links that tell a search engine it matters.

Why crawlers never see your second batch

Google has documented that Googlebot does not interact with pages the way a user does — it does not click buttons and it does not scroll through content, though it renders with a large viewport that can trigger some lazy-loading implementations. That caveat is narrower than it sounds. Content loaded by an IntersectionObserver watching for viewport entry may fire during rendering; content loaded by a listener waiting for actual scroll events, or by a "Load more" button waiting for a click, will not.

Relying on the tall-viewport behavior is fragile even where it works: it is a rendering implementation detail, it varies with how much content you load per batch, and other crawlers — Bing, and the AI crawlers now fetching your pages — each have their own, generally less capable, rendering behavior. The reasonable engineering position is to assume no crawler scrolls, ever.

The paginated-URL fallback pattern

The fix Google has long recommended is not to remove infinite scroll but to back it with real pagination. Every batch of content corresponds to an ordinary, fetchable URL:

/products/           (items 1-24)
/products/?page=2    (items 25-48)
/products/?page=3    (items 49-72)

Each of those URLs must return its items as server-rendered HTML — or at minimum, HTML that renders without user interaction — and must contain plain <a href> links to the adjacent pages. A crawler that lands anywhere in the sequence can walk the whole thing with nothing but URL fetches, which is the only operation you can count on every crawler performing.

Component pages should each carry a self-referencing canonical. The common mistake is canonicalizing every page to page 1, which tells search engines the deeper pages are duplicates and quietly asks them to ignore the very URLs you built the fallback to expose. Page 2 is not a duplicate of page 1; it contains different items.

One historical note: rel="next" and rel="prev" annotations are optional. Google stated in 2019 that it no longer uses them as an indexing signal and had not for some time. They do no harm and other consumers may read them, but they are not the mechanism that makes this pattern work. The links in the HTML are.

Stitching the two together with the History API

The paginated URLs are the crawler's path. The History API is what lets human visitors keep the seamless scroll while staying on that same path.

As the user scrolls and batch N loads, call history.replaceState() (or pushState(), if you want each batch in the back-button stack) to update the address bar to the corresponding page URL:

// after appending items for page 3
history.replaceState({page: 3}, "", "/products/?page=3");

This buys you three things. The URL in the address bar always identifies what the user is looking at, so a shared or bookmarked link reproduces the user's actual position — served by the same paginated URL the crawler uses. Analytics record which depth users actually reach. And there is exactly one URL space; the crawlable version and the scrollable version are the same resource, not a parallel structure that drifts out of sync.

The inverse requirement matters just as much: loading /products/?page=3 directly must work, rendering that page's items server-side, with the infinite scroll continuing from there in both directions or at least forward.

Where implementations actually break

• "Load more" buttons. A button needs a click, and crawlers do not click. The fix is the same as for scroll: make the button a real <a href="?page=2"> that JavaScript intercepts and enhances. Without JavaScript it navigates; with it, it appends.
• Fragment URLs. Batches addressed as #page=2 are not separate URLs to a crawler; fragments are not sent to the server. Use path or query parameters.
• Overlapping or shifting batches. If the item list is re-sorted between requests, page boundaries move, items duplicate across pages or fall between them, and the paginated view stops being a stable partition of the inventory. Pin the sort order for the paginated sequence.
• Blocked APIs. If batches load from an endpoint disallowed in robots.txt, rendering-based crawlers cannot fetch them even when the trigger fires. The server-rendered fallback makes this moot, which is another argument for it.

How to verify you got it right

Fetch a deep page with a plain HTTP client — curl, not a browser — and confirm the items and the prev/next links are present in the raw HTML. Then use Search Console's URL Inspection on the same URL and read the rendered HTML Google reports. Finally, check your server logs for Googlebot hits on ?page= URLs; a working implementation shows crawls walking the sequence within days or weeks, depending on your crawl frequency. Silence in the logs means discovery is not happening, whatever the rendering test said.

What to do

Inventory every scroll-triggered or click-triggered content loader on your site and ask one question of each: does an equivalent plain URL return this content as HTML with anchor links onward? Where the answer is no, build the paginated fallback first, wire the History API second, and treat the scroll behavior as an enhancement layered over a crawlable page structure — not the structure itself. Then verify in the raw HTML and the logs, because this is a pattern where the user-facing site can look perfect while the crawlable site is missing almost everything.