WriteMySEO / Blog / Googlebot Keeps Recrawling URLs You Deleted Years Ago
SEO technology

Googlebot Keeps Recrawling URLs You Deleted Years Ago

Why dead URLs stay in Google's crawl frontier, what a 410 actually changes, and how to measure the 404 recrawl decay curve in your own server logs.

Every few months someone opens a log file, sees Googlebot requesting /product/discontinued-2019-model for the hundredth time, and concludes something is broken. Usually nothing is broken. Google keeps a long memory of URLs it has seen, and a 404 does not erase that memory — it only changes how often the URL gets revisited.

The useful question isn't "how do I stop this?" It's "is the recrawl rate decaying the way it should, and is any of it signal I should act on?" Both are measurable from your own logs.

Why a dead URL stays in the crawl frontier

Google's crawl scheduler works from a set of known URLs, not from a set of pages that currently exist. A URL enters that set when it's discovered — a link, a sitemap entry, a redirect target, a canonical reference, a Referer in someone else's logs — and discovery is not reversible. There's no event that means "forget this string."

What a 404 changes is the scheduled interval. The reasonable inference from observed behavior is that each URL carries an estimated likelihood of useful change, and repeated non-200 responses push that estimate down, lengthening the gap between requests. Google documents the general principle in its crawl budget guide: crawl demand is driven by perceived value and change rate, and URLs that consistently return nothing useful get crawled less.

It lengthens; it doesn't go to zero. There's a sound engineering reason for that. A single 404 is weak evidence. Deployments break, origin servers misfire, a CDN rule misroutes a path for twenty minutes. A crawler that permanently dropped URLs on the first 404 would delete real pages every time someone shipped a bad release on a Friday. Occasional re-verification is cheap insurance, and Google has said publicly that seeing 404s crawled is normal and not a ranking problem.

Does returning 410 make it stop?

Mostly no. Google has stated that 404 and 410 are treated very similarly, with 410 sometimes processed marginally faster for removal from the index. "Marginally faster" is the honest summary — think in terms of days versus weeks on the drop, not in terms of the URL disappearing from the crawl frontier.

So the decision criteria for 410 are narrow:

What re-promotes a dead URL back into frequent crawling

If your 404 recrawl rate is flat instead of decaying, something is still telling Google the URL matters. The usual culprits:

SignalEffectWhere to look
Internal links still pointing at the URLStrongest and most common causeYour own crawl, filtered to links with non-200 targets
Sitemap inclusionExplicit "this is a current URL" claimSitemap diff against live URL set
Redirect chains ending at the dead URLKeeps the endpoint discoverableLog join on redirect targets
rel=canonical on live pagesPoints indexing at a URL that doesn't existRendered HTML audit
External linksYou can't remove these; decay is slowerReferring-page data
Paginated or faceted variantsGenerates fresh unseen dead URLs continuouslyParameter analysis in logs

The first four are entirely in your control and are worth fixing before you blame the crawler. Leaving a dead URL in a sitemap is the single most effective way to keep Googlebot checking it indefinitely.

How to measure the decay curve in your own logs

The metric that matters is recrawls per dead URL over time since death, not raw 404 counts. Raw counts move whenever you delete a batch of pages and tell you nothing about scheduler behavior.

Cohort the URLs by the first date they returned 404 or 410, then count verified-Googlebot hits per week since that date:

WITH death AS (
  SELECT url, min(day) AS died
  FROM logs
  WHERE bot = 'googlebot' AND status IN (404, 410)
  GROUP BY url
)
SELECT date_diff('week', d.died, l.day)      AS weeks_since_death,
       count(*)                               AS hits,
       count(DISTINCT l.url)                  AS live_cohort,
       count(*) * 1.0 / count(DISTINCT l.url) AS hits_per_url
FROM logs l JOIN death d USING (url)
WHERE l.bot = 'googlebot' AND l.day >= d.died
GROUP BY 1 ORDER BY 1;

Plot hits_per_url against weeks_since_death. A healthy curve falls steeply for the first few weeks and then flattens into a long, low tail — occasional re-verification. An unhealthy curve plateaus high, which almost always means the URLs are still linked or still in a sitemap.

Two refinements make this far more actionable:

  1. Split the cohort by whether the URL is still internally linked. The gap between the two curves is the crawl cost of your stale links, in requests.
  2. Verify Googlebot by reverse DNS before counting. Spoofed user agents hitting old URLs are common and will distort the tail badly.

Don't try to do this from Search Console's "Not found (404)" report. It shows a capped sample of example URLs, with no per-URL request history, so you can't build a decay curve from it.

When dead-URL crawling is actually a problem

Use the share of Googlebot requests, not the absolute count. On most sites, a few percent of crawl requests ending in 404 is unremarkable. The threshold where it starts to matter is roughly when dead URLs take a double-digit share of Googlebot's daily requests and you have evidence of slow discovery for new content — new pages taking much longer to appear in logs than they used to.

The mechanism to worry about isn't a fixed "budget" being spent down. It's that generated dead URLs — session parameters, infinite facets, expired filter combinations — produce an unbounded supply of new, never-before-seen URLs. Each one is a first-time crawl at full scheduling priority, so the decay curve never helps you. That is a crawl problem. Ten thousand old product pages 404ing on a slow decay curve is not.

What to do

  1. Build the cohort query above and look at the shape before changing anything.
  2. Crawl your own site and fix internal links, canonicals, and redirect targets that point at non-200 URLs. This is the highest-yield fix.
  3. Remove dead URLs from sitemaps on the same deploy that removes the pages.
  4. Return 410 for URLs you're certain are permanently gone; keep 404 everywhere else and stop worrying about the difference.
  5. Audit for generated dead URLs separately — those need a pattern-level fix at the link source, not a status code decision.
  6. Re-run the cohort query a month later. If the curve steepened, the links were the problem; if it didn't, external links are driving it and you can safely ignore the tail.
crawlinglog filesstatus codescrawl budget

WriteMySEO produces marketing content, not legal, medical, financial, or compliance advice. Figures cited reflect publicly reported industry data at time of writing and shift over time.

Get started

We write this well about your industry, every month.

AI-drafted, human-reviewed SEO content on a flat subscription. Blog posts, metadata, schema, and internal links, shipped on a monthly rhythm.

See plans

More from the blog