Most crawl budget advice treats Googlebot's request rate as a number Google assigns you, like a quota in a billing plan. It isn't. It's the output of a control loop that adjusts continuously based on how your server responds, bounded above by how much Google actually wants to crawl.
That distinction matters because it changes the diagnosis. If your crawl rate is low, there are two completely different causes with opposite fixes, and the usual advice — "improve crawl budget" — doesn't tell you which one you have.
Crawl rate is a controller, not a quota
Google's own crawl budget documentation splits crawling into two factors: the crawl capacity limit and crawl demand.
The capacity limit is Google's estimate of how many concurrent connections and how much request frequency your host can absorb without degrading. Google documents that this limit rises when the site responds quickly and without errors, and falls when responses get slow or the server starts returning server errors. It is adaptive — it moves on its own, in both directions, without you touching anything.
Crawl demand is how much Google wants to crawl: how many URLs it knows about, how popular they are, how often they change, and how stale its copies have become.
Actual crawling is roughly the minimum of the two. Google has been explicit that making your server faster does not automatically produce more crawling — if demand is the binding constraint, extra capacity sits unused. This is the single most common misreading of crawl stats: teams spend a quarter on infrastructure and see a flat crawl curve, because they were never capacity-limited in the first place.
Google also retired the manual crawl rate limiter in Search Console, which was the last piece of UI that encouraged thinking about crawl rate as a setting. The remaining levers are indirect: your responses, and your URL inventory.
What actually moves the capacity limit
The inputs are response-level, not page-level. Googlebot does not care whether a URL is valuable when deciding how hard to push your host; it cares what happened on the wire.
| Server behavior | Effect on capacity | What it looks like in logs |
|---|---|---|
| Fast 200s (low TTFB) | Capacity can rise, if demand exists | Stable or growing requests/day, flat response time |
| Slow 200s | Capacity falls | Response time climbs, then requests/day falls days later |
| 5xx responses | Capacity falls sharply | Error spike, then a crawl trough |
| 429 responses | Capacity falls; Google treats it as an explicit back-off signal | Same shape as 5xx |
| Connection timeouts / resets | Worst case — no useful response at all | Gaps in the log, fetch errors in Crawl Stats |
| Fast 404s and 410s | Cheap; costs little capacity | High volume, minimal response time impact |
| Unreachable robots.txt (5xx) | Google's robots.txt documentation describes pausing crawling of the host, falling back to a cached copy, and only after a prolonged failure treating the site as unrestricted | Near-total crawl stop with no obvious page-level cause |
Two things follow from this table that people get wrong.
A fast error is not expensive. Serving 404s quickly is not a crawl budget problem in the capacity sense. The cost of junk URLs is almost entirely demand-side: Google spends discovery and refresh attention on URLs that will never earn anything.
A slow 200 is worse than a fast 500 for rate. A 200 that takes eight seconds ties up a connection and feeds the latency signal. If you must fail, fail fast and with the right code.
How to tell if you're capacity-limited or demand-limited
This is a measurable question, not a judgment call. Build a daily series from your logs for verified Googlebot requests only, then compare the shape of two curves: requests per day and median (plus p90) response time.
- Capacity-limited: response time degrades first, crawl volume falls afterward with a lag of roughly days. The two curves are anti-correlated. Error rate often moves with them.
- Demand-limited: crawl volume is flat or declining while response time is flat and low, and your server is nowhere near saturated. Pushing latency from 400ms to 200ms will not move it.
The Crawl Stats report in Search Console gives you a coarse version of the same comparison — total requests, average response time, and host status — plus one field logs can't produce: the Discovery vs. Refresh split under "By purpose." If refresh dominates and discovery is a thin slice while you're publishing heavily, that's a demand and discovery problem, not a server problem.
A minimal daily rollup from a combined-format log:
awk '$0 ~ /Googlebot/ {
split($4, d, ":"); day = substr(d[1], 2)
n[day]++; status[day "," $9]++; t[day] += $NF
} END {
for (k in n) printf "%s\treq=%d\tavg_ms=%.0f\n", k, n[k], (t[k]/n[k])*1000
}' access.log | sortReplace the user-agent match with a reverse-DNS-verified IP list before you trust any of it; the raw UA string is trivially spoofed.
Why recovery looks slower than the drop
The pattern practitioners report repeatedly — and what you will usually see in your own logs — is that crawl volume collapses within a day or two of an incident and takes considerably longer to climb back. Google has not published the controller's parameters, so treat the mechanism as inference rather than documented behavior: a system that protects origins has every reason to back off aggressively and probe back upward cautiously.
The practical consequence is firm regardless of the exact algorithm. A four-hour outage can cost you more than four hours of crawling. When you plan maintenance, serving a fast 503 with a Retry-After header is the documented correct move — but Google's guidance also warns that prolonged 5xx responses eventually risk URLs being dropped from the index, so a 503 is a short-window tool, not a parking strategy.
The infrastructure that breaks the model
The capacity limit is tracked per host, which produces a few counterintuitive situations:
- Shared hosting. Google has noted that sites sharing server infrastructure can effectively share crawl capacity. Your neighbor's traffic spike is your crawl trough.
- CDNs. If the CDN serves Googlebot from cache, your origin logs show almost nothing and your origin latency is irrelevant to the controller. Measure at the edge, or you'll be tuning a server Googlebot rarely touches.
- Subdomains and separate hostnames.
www,m., and an API hostname are distinct hosts for capacity purposes, even when they're the same box behind the scenes. - WAF and bot-management rules. Challenge pages and rate limiters returning 429 or 403 to Googlebot feed the back-off signal directly. This is one of the most common causes of unexplained crawl collapse, and it never shows up in application logs.
What to do with this
- Pull 90 days of verified-Googlebot requests and plot requests/day against median and p90 response time. Decide which regime you're in before changing anything.
- If you're capacity-limited, attack the slowest responses for Googlebot specifically — bot traffic often skips the caches that make human traffic look fine.
- If you're demand-limited, stop buying servers. Work on internal links, sitemap segmentation, and removing the low-value URL inventory that absorbs refresh crawling.
- Audit your WAF and rate-limiter rules for anything that can return 429 or 403 to verified Googlebot ranges. Alert on it.
- Alert on Googlebot-specific 5xx rate and p90 latency, separately from overall site health, so an incident that only affects crawling is visible.
- During planned downtime, serve a fast 503 with
Retry-After, keep the window short, and confirm robots.txt itself still returns 200 or 404 — never 5xx.