How ETags and Last-Modified Change What Googlebot Crawls

Most crawl budget advice is about subtraction: block parameters, prune faceted URLs, kill the calendar. That work matters, but it ignores the cheaper lever. Googlebot will happily tell your server which version of a URL it already has, and let your server answer "nothing changed" in a few hundred bytes instead of re-sending 90KB of HTML.

Google supports conditional requests and has publicly encouraged sites to use HTTP caching headers, noting that adoption is low. In practice, a large share of sites answer every crawl of every URL with a full 200 and a full body, including URLs that have not changed in three years. That is a self-inflicted tax on a resource you are otherwise fighting to conserve.

What a conditional request actually looks like

When your server returns a response with Last-Modified or ETag, those are validators: opaque tokens identifying the version of the resource. Googlebot can store them and send them back on a later fetch.

A first fetch:

GET /guides/pricing HTTP/1.1

HTTP/1.1 200 OK
Last-Modified: Tue, 04 Mar 2025 09:12:00 GMT
ETag: "a1f3-6c2"
Content-Length: 91240

A later fetch, conditionally:

GET /guides/pricing HTTP/1.1
If-Modified-Since: Tue, 04 Mar 2025 09:12:00 GMT
If-None-Match: "a1f3-6c2"

HTTP/1.1 304 Not Modified
ETag: "a1f3-6c2"

A 304 carries no body. Google's crawl budget documentation explicitly says Googlebot supports If-Modified-Since and If-None-Match and that returning 304 when content has not changed reduces load. If-None-Match takes precedence when both are present, and weak ETags (W/"...") are perfectly usable here — weak comparison is what If-None-Match uses.

Why 304s matter for crawling but not for ranking

Be precise about the causal chain, because this is where people oversell it.

Google has documented that crawl rate is bounded by crawl capacity — how much your host can take without degrading — and crawl demand, which is driven by perceived value and change rate. Serving 304s reduces bytes and, if implemented well, origin CPU. On a host that is genuinely capacity-limited, freeing that capacity leaves room for more requests. That is the mechanism Google itself points to when recommending caching.

What is not documented is any ranking benefit. A 304 is not a quality signal. And the freshness story is more subtle than it looks: it is a reasonable inference that a URL which consistently returns 304 gives Google evidence it is stable, and that a URL which changes every fetch invites more frequent recrawls. But you should not treat "my 304 rate went up" as a KPI in itself. The KPI is crawl coverage of URLs you care about, and total bytes served to crawlers.

Where validator implementations quietly break

Almost every site I have looked at that "has ETags on" is never actually returning 304s to Googlebot. The usual causes:

• Compression mangles the ETag. Historically some Apache configurations appended a -gzip suffix to ETags when compressing, and some server builds strip or weaken the ETag when gzip or Brotli is applied. If the token differs between the fetch that generated it and the fetch that validates it, you never match.
• The validator is derived from server-local state. ETags computed from inode numbers or file mtimes differ across nodes in a load-balanced fleet. Requests round-robin, validators never match, every fetch is a 200. Hash the response content or the underlying record version instead.
• Last-Modified is set to now. Dynamic frameworks that stamp the current time defeat the whole mechanism. Last-Modified should reflect the last change to the content — the post's updated timestamp, the latest change among the entities rendered on the page.
• The 304 is generated too late to save anything. Many frameworks build the full HTML, hash it, compare, then send a 304. You save bandwidth but not CPU or database time. That is still worth having, but if crawl load is your problem, compute the validator from cheap metadata before rendering.
• A CDN or proxy strips or rewrites the headers. Check both directions: does the edge forward If-None-Match to origin, and does it preserve the origin's validators when it caches? Some configurations normalise away conditional headers entirely.
• 304s served for content that did change. The dangerous failure. If your validator is tied to the article record but you just shipped a template change that altered navigation, canonical tags, or structured data sitewide, every URL still validates and Google keeps its stale copy. Any global template or config version needs to be part of the validator input.

How to measure your actual 304 rate

This is a log question, and it takes ten minutes. Filter to verified Googlebot requests — reverse-DNS verified, not just user-agent string matched — then break status and bytes down by URL class.

awk '$0 ~ /Googlebot/ {print $9, $10}' access.log \
  | sort | uniq -c | sort -rn

Then build a table per section, because the right target differs by content type:

If your archived-content classes show a 304 share near zero, you have a bug, not a strategy.

Cache-Control is not a substitute for validators

Cache-Control: max-age is a freshness directive; validators are a revalidation mechanism. They solve different problems, and Googlebot is not a browser cache. Google has said its rendering service caches subresources aggressively using its own heuristics rather than strictly honouring your headers, which is why you cannot reliably force a JS bundle to refresh by shortening its max-age. Content-hashed filenames (app.9f2c1b.js) solve that properly: the URL changes when the bytes change.

For HTML, long max-age values buy you little with crawlers and can hurt users behind intermediaries. Validators are the safer lever there.

What to do this week

1. Pull a week of verified-Googlebot logs. Compute status distribution and total bytes by URL class. That is your baseline.
2. Pick your largest stable class — archive, docs, old blog posts. Confirm whether responses carry ETag or Last-Modified at all, then replay a request with If-None-Match using curl -H and see whether you get a 304.
3. Fix the validator source so it is deterministic across nodes and derived from content state plus a template/deploy version.
4. Verify the edge passes conditional headers through and preserves validators.
5. Move static assets to content-hashed filenames with long max-age, and stop trying to manage them with short TTLs.
6. Re-measure bytes served to crawlers after two weeks. Bytes, not rankings, is the outcome this work owns.

If your host is not capacity-constrained and your index coverage is fine, this is housekeeping rather than an emergency. If Search Console shows crawl-rate limiting, average response times climbing under crawler load, or large sections going weeks between fetches, it is one of the highest-leverage fixes available — and it does not require deleting a single URL.