410 vs 404: How Google Actually Retires a URL

Two beliefs about retiring URLs are common and both are wrong. The first is that a 410 wipes a page out of the index immediately while a 404 leaves it lingering. The second is that once a page returns an error, Googlebot stops asking for it and you get the crawl budget back.

Neither matches observed behavior. Status codes tell Google what kind of failure this is, which changes the retry schedule and the indexing verdict — but the drop out of the index is a crawl-scheduling outcome, not an instant delete, and dead URLs get re-requested for a long time afterward. If you understand the scheduling side, you can pick the right code and, more importantly, set expectations you can actually measure.

What each status code tells Googlebot

Google's Search Central documentation on HTTP status codes is unusually explicit about this. The short version:

The row people get wrong most often is 403. Blocking Googlebot at the firewall or behind a login returns 403 or 401, and Google treats those as "this page is gone" for indexing purposes — not as "come back later." A bot-mitigation rule that starts challenging Googlebot looks identical to mass deletion from Google's side.

Why 410 is only marginally faster than 404

Google's John Mueller has said repeatedly that 404 and 410 are treated almost the same, with 410 processed slightly faster because it skips the grace period Google applies to 404s in case the error is a mistake. "Slightly faster" means hours-to-days of difference in when the URL is dropped, not weeks.

The reason is architectural. Neither code triggers a deletion event. Both mark the fetch result as an error during the next crawl, and the URL leaves the index on the next indexing pass for that URL. If Googlebot recrawls an unimportant URL every six weeks, your 410 does nothing at all until week six. The status code controls what happens when the crawl happens; it does not control when the crawl happens.

This is why "we switched to 410 and nothing changed" is such a common report. The bottleneck was scheduling, not semantics.

503 is a promise, not a hiding place

A 503 with a Retry-After header tells Google the resource is temporarily unavailable. Google documents that short-lived 5xx responses cause it to retry and, importantly, to slow down crawling of the site — the server is signalling distress, so Googlebot backs off.

Two consequences people underestimate:

• Prolonged 5xx is removal with extra steps. If the errors persist, the pages drop out anyway, and you have spent the intervening weeks suppressing crawl rate across the whole host.
• robots.txt 5xx is a site-wide handbrake. Google documents that an unreachable robots.txt (5xx or network failure) causes it to pause crawling of the site; after roughly 30 days of failures it falls back to the last cached copy, and if none exists it assumes no crawl restrictions. A CDN rule that returns 503 for /robots.txt under load is far more damaging than the same rule on a content page.

Use 503 for maintenance windows measured in hours. Do not use it to "pause" a section for a quarter.

Dead URLs keep getting crawled for months

Googlebot does not forget URLs. Erroring URLs move to a long, decaying retry schedule, and URLs that once had internal links or external links keep getting re-requested long after they 404. In server logs it is normal to see requests for URLs deleted a year earlier.

The reasonable inference — consistent with how a frontier-based crawler must work — is that Google keeps a URL in its known set with a next-crawl timestamp that backs off on repeated failure rather than evicting entries outright. That is why retiring 50,000 URLs does not immediately free 50,000 crawl requests. What does reduce the requests is removing the paths that keep re-feeding discovery: internal links, sitemap entries, canonical and hreflang annotations pointing at dead URLs, and paginated or faceted link graphs that regenerate them.

How to measure removal time instead of guessing

Stop asking "how long does deindexing take" in general and measure it for your site, which is the only number that matters. Build a cohort:

1. On retirement day, write the full list of retired URLs to a table with the retirement date and the status code you served.
2. Poll a random sample of that cohort through the URL Inspection API on a fixed schedule — daily for the first two weeks, then weekly. Google documents a quota of 2,000 inspections per property per day, so sample rather than inspect everything.
3. Record inspectionResult.indexStatusResult.verdict, coverageState, lastCrawlTime, and pageFetchState on each poll.
4. Define removal per URL as the first poll where coverageState stops reporting the URL as indexed, and compute the distribution of days-to-removal — median and 90th percentile, not just the mean.

GET https://searchconsole.googleapis.com/v1/urlInspection/index:inspect
{
  "inspectionUrl": "https://example.com/retired/widget-2019",
  "siteUrl": "sc-domain:example.com"
}
→ indexStatusResult.coverageState: "Not found (404)"
→ indexStatusResult.lastCrawlTime: "2024-..."

Cross-reference lastCrawlTime with your own logs. If the gap between retirement and removal is mostly explained by the gap between retirement and first recrawl, your lever is discovery and importance, not the status code. If Google recrawled quickly and the URL still showed as indexed for weeks, the lever is reprocessing latency and you simply wait.

Segmenting the cohort by prior impressions or internal link count usually shows what you would expect: high-value URLs are recrawled and removed fast, and the long tail takes far longer. Report the 90th percentile to stakeholders so nobody panics at week three.

What to do

• Use 410 when you are certain, 404 when you are not. The difference is small; the cost of being wrong with 410 on a page you restore is also small. Do not spend a sprint on this choice.
• Audit for accidental 403/401 on Googlebot. Verify by reverse DNS, not user agent, and check your bot-mitigation logs for challenges served to verified Googlebot.
• Reserve 503 for hours, with Retry-After. Never serve 5xx for /robots.txt; serve the last known good file from cache if your origin is down.
• Cut discovery paths at the same time as the status change. Remove internal links, drop the URLs from sitemaps, and fix canonicals and hreflang that still reference them. This, not the status code, is what reduces wasted crawling.
• If a URL must disappear from results today, use the Search Console removals tool for the temporary hide — Google documents it as lasting about six months — and serve 404/410 underneath so the removal becomes permanent.
• Instrument the cohort before you delete anything. A measured median and 90th percentile for your own site ends the recurring argument about whether deindexing is "stuck."