Every few months someone opens a log file, sees Googlebot requesting /product/discontinued-2019-model for the hundredth time, and concludes something is broken. Usually nothing is broken. Google keeps a long memory of URLs it has seen, and a 404 does not erase that memory — it only changes how often the URL gets revisited.
The useful question isn't "how do I stop this?" It's "is the recrawl rate decaying the way it should, and is any of it signal I should act on?" Both are measurable from your own logs.
Why a dead URL stays in the crawl frontier
Google's crawl scheduler works from a set of known URLs, not from a set of pages that currently exist. A URL enters that set when it's discovered — a link, a sitemap entry, a redirect target, a canonical reference, a Referer in someone else's logs — and discovery is not reversible. There's no event that means "forget this string."
What a 404 changes is the scheduled interval. The reasonable inference from observed behavior is that each URL carries an estimated likelihood of useful change, and repeated non-200 responses push that estimate down, lengthening the gap between requests. Google documents the general principle in its crawl budget guide: crawl demand is driven by perceived value and change rate, and URLs that consistently return nothing useful get crawled less.
It lengthens; it doesn't go to zero. There's a sound engineering reason for that. A single 404 is weak evidence. Deployments break, origin servers misfire, a CDN rule misroutes a path for twenty minutes. A crawler that permanently dropped URLs on the first 404 would delete real pages every time someone shipped a bad release on a Friday. Occasional re-verification is cheap insurance, and Google has said publicly that seeing 404s crawled is normal and not a ranking problem.
Does returning 410 make it stop?
Mostly no. Google has stated that 404 and 410 are treated very similarly, with 410 sometimes processed marginally faster for removal from the index. "Marginally faster" is the honest summary — think in terms of days versus weeks on the drop, not in terms of the URL disappearing from the crawl frontier.
So the decision criteria for 410 are narrow:
- Use 410 when you are certain the resource is permanently gone and you want the fastest clean removal — expired listings, deleted user accounts, retired SKUs with no successor.
- Use 404 when you're unsure, when a URL pattern might come back, or when your stack makes 410 awkward. The cost of choosing wrong here is small.
- Don't redirect a dead URL to a loosely related page just to avoid the 404. Google is good at classifying those as soft 404s, which gets you the removal anyway plus a confusing report in Search Console.
- Never block dead URLs in robots.txt. That prevents Google from seeing the 404 at all. The URL stays known, stays unresolved, and can keep appearing in search results as a URL-only entry.
What re-promotes a dead URL back into frequent crawling
If your 404 recrawl rate is flat instead of decaying, something is still telling Google the URL matters. The usual culprits:
| Signal | Effect | Where to look |
|---|---|---|
| Internal links still pointing at the URL | Strongest and most common cause | Your own crawl, filtered to links with non-200 targets |
| Sitemap inclusion | Explicit "this is a current URL" claim | Sitemap diff against live URL set |
| Redirect chains ending at the dead URL | Keeps the endpoint discoverable | Log join on redirect targets |
rel=canonical on live pages | Points indexing at a URL that doesn't exist | Rendered HTML audit |
| External links | You can't remove these; decay is slower | Referring-page data |
| Paginated or faceted variants | Generates fresh unseen dead URLs continuously | Parameter analysis in logs |
The first four are entirely in your control and are worth fixing before you blame the crawler. Leaving a dead URL in a sitemap is the single most effective way to keep Googlebot checking it indefinitely.
How to measure the decay curve in your own logs
The metric that matters is recrawls per dead URL over time since death, not raw 404 counts. Raw counts move whenever you delete a batch of pages and tell you nothing about scheduler behavior.
Cohort the URLs by the first date they returned 404 or 410, then count verified-Googlebot hits per week since that date:
WITH death AS (
SELECT url, min(day) AS died
FROM logs
WHERE bot = 'googlebot' AND status IN (404, 410)
GROUP BY url
)
SELECT date_diff('week', d.died, l.day) AS weeks_since_death,
count(*) AS hits,
count(DISTINCT l.url) AS live_cohort,
count(*) * 1.0 / count(DISTINCT l.url) AS hits_per_url
FROM logs l JOIN death d USING (url)
WHERE l.bot = 'googlebot' AND l.day >= d.died
GROUP BY 1 ORDER BY 1;
Plot hits_per_url against weeks_since_death. A healthy curve falls steeply for the first few weeks and then flattens into a long, low tail — occasional re-verification. An unhealthy curve plateaus high, which almost always means the URLs are still linked or still in a sitemap.
Two refinements make this far more actionable:
- Split the cohort by whether the URL is still internally linked. The gap between the two curves is the crawl cost of your stale links, in requests.
- Verify Googlebot by reverse DNS before counting. Spoofed user agents hitting old URLs are common and will distort the tail badly.
Don't try to do this from Search Console's "Not found (404)" report. It shows a capped sample of example URLs, with no per-URL request history, so you can't build a decay curve from it.
When dead-URL crawling is actually a problem
Use the share of Googlebot requests, not the absolute count. On most sites, a few percent of crawl requests ending in 404 is unremarkable. The threshold where it starts to matter is roughly when dead URLs take a double-digit share of Googlebot's daily requests and you have evidence of slow discovery for new content — new pages taking much longer to appear in logs than they used to.
The mechanism to worry about isn't a fixed "budget" being spent down. It's that generated dead URLs — session parameters, infinite facets, expired filter combinations — produce an unbounded supply of new, never-before-seen URLs. Each one is a first-time crawl at full scheduling priority, so the decay curve never helps you. That is a crawl problem. Ten thousand old product pages 404ing on a slow decay curve is not.
What to do
- Build the cohort query above and look at the shape before changing anything.
- Crawl your own site and fix internal links, canonicals, and redirect targets that point at non-200 URLs. This is the highest-yield fix.
- Remove dead URLs from sitemaps on the same deploy that removes the pages.
- Return 410 for URLs you're certain are permanently gone; keep 404 everywhere else and stop worrying about the difference.
- Audit for generated dead URLs separately — those need a pattern-level fix at the link source, not a status code decision.
- Re-run the cohort query a month later. If the curve steepened, the links were the problem; if it didn't, external links are driving it and you can safely ignore the tail.