WriteMySEO / Blog / Your CDN Cache Key Decides What Googlebot Sees
SEO technology

Your CDN Cache Key Decides What Googlebot Sees

How edge cache keys, Vary headers, and stale-while-revalidate cause Googlebot to crawl the wrong HTML variant — and how to detect it in your logs.

Most debugging of "Google is seeing the wrong page" stops at the origin. You curl the URL with a Googlebot user agent, the HTML looks right, and you move on. But in a CDN-fronted stack, the origin is often not what answered Googlebot. An edge node did, from a cached object that may have been produced by a completely different request — a different device class, a different country, a different bot, or a version of the page from four days ago.

The variable that decides this is the cache key: the string your CDN uses to decide whether two requests are "the same request." Almost every wrong-variant bug in front of a CDN is a cache key bug.

What is actually in a default cache key

Out of the box, most CDNs build the cache key from scheme, host, path, and query string. Some normalize or strip query parameters. What is usually not in the key: user agent, cookies, Accept-Language, client IP, country, or any custom header your origin uses to branch.

That means any response whose content depends on a request attribute outside the key is a landmine. The first request to warm a cold object wins, and everyone else — including Googlebot — gets that object until it expires.

The HTTP spec's answer to this is Vary. The origin says Vary: User-Agent and the cache is supposed to store a separate object per distinct user agent. Google's dynamic serving documentation does recommend Vary: User-Agent when you serve different HTML to mobile and desktop at the same URL. But Vary works badly in practice for two reasons:

So Vary is a signal to shared caches generally, not a reliable instruction to your specific edge. If correctness depends on it, you need to verify your provider's behavior rather than assume.

The four ways Googlebot ends up with the wrong object

1. A cached bot challenge. Bot management returns an interstitial or JS challenge to a suspicious request. If that response is cacheable and the key does not distinguish it, the challenge HTML becomes the object for that URL. Googlebot then crawls a page whose body is a spinner and a <noscript> block. Google is likely to classify that as a soft 404 or thin duplicate rather than a crawl error, so it will not show up as an error row — it shows up as pages silently falling out of the index.

2. A cached geo redirect or country splash. Google has documented that Googlebot crawls predominantly from US IP addresses, with geo-distributed crawling for some sites. If your edge redirects by country and the redirect is cacheable, a US-origin crawl can mint a cached 302 that every other region then receives — or the reverse, where a cached non-US splash page gets served to Googlebot.

3. Personalized or session HTML. A response carrying Set-Cookie, a logged-in header, or a consent-dependent layout gets stored because the cookie is not in the key. Googlebot crawls someone's cart state.

4. Stale HTML under stale-while-revalidate. This one is not a wrong variant, it is a wrong time. With a long stale-while-revalidate or stale-if-error window, the edge can serve a copy minutes, hours, or days old while refreshing in the background. For news, pricing, or availability markup, the version Googlebot indexes may predate the change you are trying to get indexed — and the dates and structured data in that HTML will be stale too.

SymptomLikely cache mechanism
Rendered HTML in URL Inspection is an interstitialChallenge response cached under the normal key
Live test passes, crawled page failsLive test bypasses Google's copy but not necessarily your edge object; or the object expired between the two
Pages become "Duplicate without user-selected canonical" after a CDN changeOne variant now answers for many URLs
Structured data missing in the field, present at originStale or variant object at edge
lastmod and on-page date disagree with what's indexedLong stale-while-revalidate window on HTML

How to detect it from your own logs

The test is not "what does the origin return." It is "what did the edge return to this specific request, from which cached object, and how old was it." Every major CDN can emit that. Log these fields at the edge and join them to your bot-verification pipeline:

timestamp, url, status, cache_status, cache_key, age,
user_agent, client_country, resp_content_length, resp_set_cookie

Then run three checks:

  1. Cache status split by requester class. Compare HIT/MISS/EXPIRED/BYPASS rates for verified Googlebot versus human traffic on HTML. A Googlebot HIT rate near 100% with a high Age p95 means Googlebot is almost never seeing fresh HTML.
  2. Content-length distribution per URL. For a single URL, plot response byte size over time across requests. A bimodal distribution on HTML is the fingerprint of two variants sharing one key. A sudden cliff down to a few kilobytes is the challenge-page signature.
  3. Set-Cookie on cacheable responses. Any HTML response that is both cache_status=HIT-eligible and carries Set-Cookie is a bug regardless of whether Google has noticed yet.

A fourth check that catches the stale case: compare the Age header on Googlebot HTML hits against your publishing cadence. If median Age exceeds your typical time-to-update for a template, your crawl is structurally behind your content.

What to change

When you next see a page that renders correctly for you and wrongly for Google, pull the edge log line for the actual Googlebot request before you touch the application. In a cached stack, the HTML Google indexed had a specific cache_key and a specific Age, and those two values usually contain the whole answer.

CDNcachingGooglebotlog files

WriteMySEO produces marketing content, not legal, medical, financial, or compliance advice. Figures cited reflect publicly reported industry data at time of writing and shift over time.

Get started

We write this well about your industry, every month.

AI-drafted, human-reviewed SEO content on a flat subscription. Blog posts, metadata, schema, and internal links, shipped on a monthly rhythm.

See plans

More from the blog