Most debugging of "Google is seeing the wrong page" stops at the origin. You curl the URL with a Googlebot user agent, the HTML looks right, and you move on. But in a CDN-fronted stack, the origin is often not what answered Googlebot. An edge node did, from a cached object that may have been produced by a completely different request — a different device class, a different country, a different bot, or a version of the page from four days ago.
The variable that decides this is the cache key: the string your CDN uses to decide whether two requests are "the same request." Almost every wrong-variant bug in front of a CDN is a cache key bug.
What is actually in a default cache key
Out of the box, most CDNs build the cache key from scheme, host, path, and query string. Some normalize or strip query parameters. What is usually not in the key: user agent, cookies, Accept-Language, client IP, country, or any custom header your origin uses to branch.
That means any response whose content depends on a request attribute outside the key is a landmine. The first request to warm a cold object wins, and everyone else — including Googlebot — gets that object until it expires.
The HTTP spec's answer to this is Vary. The origin says Vary: User-Agent and the cache is supposed to store a separate object per distinct user agent. Google's dynamic serving documentation does recommend Vary: User-Agent when you serve different HTML to mobile and desktop at the same URL. But Vary works badly in practice for two reasons:
- Cardinality. User agent strings are effectively unique per browser build.
Vary: User-Agentcan fragment one object into thousands, collapsing your hit rate and pushing load back to origin. - CDNs do not all honor it. Cloudflare documents that its cache ignores
Varyfor most content, supporting it forAccept-Encodingand, separately, a Vary-for-images feature. Other CDNs honor it but normalize the header first.
So Vary is a signal to shared caches generally, not a reliable instruction to your specific edge. If correctness depends on it, you need to verify your provider's behavior rather than assume.
The four ways Googlebot ends up with the wrong object
1. A cached bot challenge. Bot management returns an interstitial or JS challenge to a suspicious request. If that response is cacheable and the key does not distinguish it, the challenge HTML becomes the object for that URL. Googlebot then crawls a page whose body is a spinner and a <noscript> block. Google is likely to classify that as a soft 404 or thin duplicate rather than a crawl error, so it will not show up as an error row — it shows up as pages silently falling out of the index.
2. A cached geo redirect or country splash. Google has documented that Googlebot crawls predominantly from US IP addresses, with geo-distributed crawling for some sites. If your edge redirects by country and the redirect is cacheable, a US-origin crawl can mint a cached 302 that every other region then receives — or the reverse, where a cached non-US splash page gets served to Googlebot.
3. Personalized or session HTML. A response carrying Set-Cookie, a logged-in header, or a consent-dependent layout gets stored because the cookie is not in the key. Googlebot crawls someone's cart state.
4. Stale HTML under stale-while-revalidate. This one is not a wrong variant, it is a wrong time. With a long stale-while-revalidate or stale-if-error window, the edge can serve a copy minutes, hours, or days old while refreshing in the background. For news, pricing, or availability markup, the version Googlebot indexes may predate the change you are trying to get indexed — and the dates and structured data in that HTML will be stale too.
| Symptom | Likely cache mechanism |
|---|---|
| Rendered HTML in URL Inspection is an interstitial | Challenge response cached under the normal key |
| Live test passes, crawled page fails | Live test bypasses Google's copy but not necessarily your edge object; or the object expired between the two |
| Pages become "Duplicate without user-selected canonical" after a CDN change | One variant now answers for many URLs |
| Structured data missing in the field, present at origin | Stale or variant object at edge |
lastmod and on-page date disagree with what's indexed | Long stale-while-revalidate window on HTML |
How to detect it from your own logs
The test is not "what does the origin return." It is "what did the edge return to this specific request, from which cached object, and how old was it." Every major CDN can emit that. Log these fields at the edge and join them to your bot-verification pipeline:
timestamp, url, status, cache_status, cache_key, age,
user_agent, client_country, resp_content_length, resp_set_cookie
Then run three checks:
- Cache status split by requester class. Compare HIT/MISS/EXPIRED/BYPASS rates for verified Googlebot versus human traffic on HTML. A Googlebot HIT rate near 100% with a high
Agep95 means Googlebot is almost never seeing fresh HTML. - Content-length distribution per URL. For a single URL, plot response byte size over time across requests. A bimodal distribution on HTML is the fingerprint of two variants sharing one key. A sudden cliff down to a few kilobytes is the challenge-page signature.
Set-Cookieon cacheable responses. Any HTML response that is bothcache_status=HIT-eligible and carriesSet-Cookieis a bug regardless of whether Google has noticed yet.
A fourth check that catches the stale case: compare the Age header on Googlebot HTML hits against your publishing cadence. If median Age exceeds your typical time-to-update for a template, your crawl is structurally behind your content.
What to change
- Move variance into the key, not into
Vary. Compute a normalized flag at the edge —device=mobile|desktop,botclass=search|other|none,country-group=eu|na|apac— and append it to the cache key. Low cardinality, deterministic, and independent of whether your CDN honorsVary. Keep sendingVary: User-Agentfor downstream caches if you serve device-specific HTML, but do not rely on it. - Make challenge and error responses uncacheable. Explicitly set
Cache-Control: private, no-storeon anything bot management, WAF, or rate limiting generates. - Never cache responses with
Set-Cookieunless you strip the header deliberately. - Bound staleness on HTML. A short TTL with revalidation is usually safer for indexable HTML than a long
stale-while-revalidatewindow. Reserve the long windows for static assets. - Alert on the signals above, not on origin health. Origin health checks pass happily while the edge serves garbage.
When you next see a page that renders correctly for you and wrongly for Google, pull the edge log line for the actual Googlebot request before you touch the application. In a cached stack, the HTML Google indexed had a specific cache_key and a specific Age, and those two values usually contain the whole answer.