Most teams treat the CDN as a speed layer: same HTML, delivered faster. It isn't. A CDN is a decision layer that picks which stored response to hand back, and the rule it uses — the cache key — is usually configured by someone optimizing for hit rate, not for what Googlebot indexes.
When the cache key is narrower than the set of responses your origin can produce, the edge will happily serve one variant to everyone. Sometimes the variant Googlebot gets is a geo-redirect page, a logged-out shell, a bot-challenge interstitial, or last week's HTML. None of that shows up in a normal browser test, because your browser gets a different cached object than the crawler does.
What a cache key actually is
A cache key is the string a CDN builds from the request to look up a stored response. By default on most CDNs it's roughly:
scheme + host + path + (query string, sometimes normalized or stripped)Notice what's not in there: request headers, cookies, client geography, user agent, Accept-Language. If your origin varies the HTML on any of those inputs but they aren't part of the key, the cache is storing one response for many logically distinct requests. First request wins; everyone else gets that copy until it expires.
The HTTP-standard fix is the Vary response header, which tells shared caches "this response depends on these request headers, store variants separately." Most CDNs honor Vary for a limited set of headers, and many let you add key components directly in configuration instead:
# Fastly VCL sketch
sub vcl_hash {
hash_data(req.url);
hash_data(req.http.host);
hash_data(req.http.X-Device-Class); # normalized to mobile|desktop at the edge
hash_data(req.http.X-Geo-Country);
}The practical rule: every input your origin uses to change the HTML must appear either in Vary or in the CDN's cache key. If it appears in neither, you have a variant-leak.
The four failure modes that change what gets indexed
1. Geo-personalized HTML under a shared key. Google has documented that Googlebot crawls primarily from US-based IP addresses, with some locale-distributed crawling. If your origin swaps currency, language, or the whole page based on IP, and country isn't in the cache key, then whichever variant got cached first is what the crawler receives — and whichever variant the crawler receives can be what gets indexed for every market. The nastier version is a 302 to /us/ cached under the bare path: now your canonical landing page is a redirect in Google's eyes.
2. Cookie- or session-dependent HTML. Googlebot generally does not carry cookies across requests, so it always looks like a brand-new anonymous visitor. If your origin renders a different shell for anonymous users — a consent wall, a "choose your store" screen, an empty personalized module — that's the version in the index. If instead a logged-in response gets cached under a key that ignores the session cookie, the crawler can receive account UI and, occasionally, private content. That's a security bug before it's an SEO bug.
3. Mobile and desktop HTML under one key. Google indexes with the smartphone Googlebot. If you use dynamic serving — different HTML from the same URL depending on user agent — and the cache key doesn't distinguish device classes, Googlebot can be served the desktop variant. Google's dynamic-serving guidance has long recommended Vary: User-Agent for exactly this reason.
4. Cached error and challenge pages. Bot-mitigation interstitials, rate-limit pages, and maintenance notices are often returned with a 200 status and short bodies. If the edge caches one of those under the normal key, every subsequent crawl gets it. A cached 200 containing a challenge page is a strong candidate for Google to treat as a soft 404, and it will stay that way until TTL expiry or a purge.
Why Vary: User-Agent is a blunt instrument
Vary: User-Agent is correct HTTP, but user agent strings are nearly unique per browser build. Telling a shared cache to key on the full UA string fragments your cache into thousands of near-identical objects and collapses hit rate — the exact outcome the CDN was bought to prevent.
The better pattern is to normalize at the edge: parse the UA once, reduce it to a small enumerated value (mobile, desktop, bot), set that as a custom header, and put that in the cache key. You get correct variant separation with a handful of objects per URL instead of thousands. If you must emit Vary: User-Agent for downstream proxies you don't control, pair it with an edge-level normalized key so your own CDN stays efficient.
And the simplest option is often best: if you can serve one responsive HTML document to everyone, you delete the entire class of problem.
How to find out which variant Google actually got
Don't reason about this from config. Fetch and diff.
- Compare the rendered HTML in Search Console's URL Inspection tool against what you get in a browser. The "view crawled page" HTML is the single most direct evidence of what Google received.
- Diff origin against edge. Request the URL with the CDN bypassed (direct to origin, host header set) and through the CDN. Differences are cache-layer artifacts by definition.
- Vary one input at a time. Same URL, different
Accept-Language; same URL, mobile vs desktop UA; same URL from a non-US exit node. If the body changes, that input must be in the key. - Read the cache headers on every response. They tell you whether you're looking at a fresh object or a stored one.
| Header | What it tells you |
|---|---|
x-cache / cf-cache-status | HIT, MISS, EXPIRED, BYPASS — vendor-specific but decisive |
age | Seconds this object has been sitting at the edge |
cache-control (s-maxage) | How long shared caches may keep it, separate from browser TTL |
vary | Which request headers the origin claims matter |
set-cookie | A response that sets cookies should almost never be shared-cached |
One caution: testing your own site with a Googlebot user agent string is fine, but many origins verify crawlers by reverse DNS, so a spoofed UA may not reproduce the path Googlebot actually takes. Where verification exists, test both with and without it.
Cache TTL is part of your time-to-index
Edge TTL stacks on top of crawl interval. If HTML sits at the edge with s-maxage=86400 and Googlebot crawls the URL a few hours after a publish, it gets yesterday's document — and the freshly updated version may not be fetched again for days. Revalidation with ETag or Last-Modified reduces bandwidth but doesn't help if the edge considers the object fresh and never asks the origin at all.
The related deploy bug: purging static assets but not HTML. Googlebot then renders stale HTML that references hashed JS and CSS filenames that no longer exist, producing resource 404s during rendering and, in bad cases, a page that renders empty.
What to do
- Inventory the inputs your origin uses to vary HTML — geo, language, cookie, device, A/B bucket, feature flag. For each one, confirm it appears in the CDN cache key or in
Vary. Anything on neither list is an active variant-leak. - Normalize high-cardinality inputs into small enumerated values at the edge rather than keying on raw header strings.
- Never shared-cache a response that carries
Set-Cookieor renders account state. Mark thoseprivate, no-storeat the origin. - Exclude bot-mitigation and maintenance responses from caching explicitly, and return
503withRetry-Afterrather than a200containing an interstitial. - Add HTML purge to your deploy pipeline, ordered after asset upload, not before.
- Add a header-diff check to monitoring: fetch a sample of templates with mobile and desktop UAs, two
Accept-Languagevalues, and a non-US egress, and alert on unexpected body differences. This catches a bad cache-key change in minutes instead of at the next index-coverage drop.