Most people open the Crawl Stats report expecting a sampled, slightly lossy version of their server logs. It isn't. It's a different dataset, produced by a different system, measuring a different set of events. When the two disagree — and they always do — the gap is usually information, not error.
The practical consequence: if you diff total requests and the numbers are off by 30%, that does not mean one source is broken. It usually means you have a CDN, or unverified bot traffic in your logs, or both. Knowing which direction the skew runs tells you what's actually happening at your edge.
What is Crawl Stats actually counting?
Crawl Stats is Google's own outbound request log, aggregated and filtered down to your hosts. It records the request from the client side, which means it includes events that never produced a line in your access log:
- Requests that failed before reaching origin. DNS resolution failures, refused connections, TLS handshake errors, and timeouts are all crawl requests from Google's perspective. Your origin never saw them.
- robots.txt fetches. Google refetches robots.txt constantly. Those requests count.
- Each hop of a redirect chain. A 301 followed by a 200 is two crawl requests, not one.
- Non-Googlebot Google clients. The report covers Google's crawlers and fetchers generally, broken out by type — smartphone and desktop Googlebot, Image, Video, AdsBot, and the on-demand fetcher used by URL Inspection. If someone on your team hits "Test live URL" fifty times, that shows up.
Google also classifies each request by purpose — discovery (first fetch of a new URL) and refresh (recrawl of a known URL) — and by response code and file type. The purpose split is the single most valuable field in the report, because you cannot derive it from logs. Nothing in an HTTP request tells you whether Google thought this URL was new.
The report is limited to roughly the last 90 days, and it's reported at host level. A URL-prefix property for example.com/blog/ shows crawl data for the whole host, not just that directory. You cannot scope Crawl Stats to a path.
What are your origin logs actually counting?
Whatever reached the process doing the logging. That's a narrower set than you think:
- Edge cache hits never arrive. If your CDN serves Googlebot a cached HTML response, origin logs show nothing. On a well-cached site this alone can account for the majority of the gap.
- WAF and rate-limit rejections are typically logged in a separate security log, not the access log. A 403 served at the edge is invisible in both your app log and, often, your analysis pipeline.
- Spoofed Googlebot is included. User-agent strings are free to type. Unless you've verified requests by reverse DNS or against Google's published IP ranges, your "Googlebot" count is inflated by scrapers.
- Logs may be sampled or rotated. At high volume, sampled logging and aggressive rotation are common and quietly discard data.
Which direction does the gap run?
| Event | Crawl Stats | Origin access log |
|---|---|---|
| CDN cache hit | Counted | Missing |
| DNS / connection failure | Counted (as failed) | Missing |
| robots.txt fetch | Counted | Present (if it reaches origin) |
| Each redirect hop | Counted separately | Present |
| URL Inspection live test | Counted | Present |
| Spoofed Googlebot UA | Not counted | Counted unless verified |
| Requests to other hosts on the domain | Counted (domain property) | Depends on which hosts you log |
So: a site behind a CDN with unverified logs has two errors pointing in opposite directions, which occasionally cancel out and make you think everything matches. Verify Googlebot first, then compare.
Why is Crawl Stats' average response time higher than yours?
Because it's measured from Google's side of the network. Google's figure reflects the time to retrieve the page content including DNS, connection setup, and transit; your origin's request_time typically starts when the request arrives and stops when the last byte is handed to the socket. The Google number will essentially always be larger, and the delta is your network path plus any edge processing.
That makes Crawl Stats the right source for the question "is Google experiencing my site as slow?" and the wrong source for "is my application slow?" If Crawl Stats response time climbs while origin timings are flat, look at TLS termination, edge config, or a geographic routing change — not your database.
The host status section works the same way. It tracks three independent checks: robots.txt fetch, DNS resolution, and server connectivity. Failures in the first two produce zero origin log lines by definition. If host status shows problems and your logs look clean, that's consistent, not contradictory.
How do you reconcile them properly?
- Verify before you count. Filter log lines to requests whose IP reverse-resolves into Google's crawler domains, or that match Google's published crawler IP ranges. Everything downstream depends on this.
- Add edge logs. Pull CDN logs and union them with origin logs. Without this step, cached hits are permanently missing and no reconciliation is possible.
- Align time zones. Search Console buckets days in Pacific Time. UTC log days are offset by seven or eight hours, which produces spurious day-over-day wobble on any chart you overlay.
- Strip Google's non-crawl fetchers from the log side — or keep them and compare against the matching Googlebot-type filter in Crawl Stats. Don't mix.
- Compare shapes, not totals. Response-code mix, file-type mix, and week-over-week trend lines should agree closely once steps 1–4 are done. Absolute request counts rarely will, and chasing the last 5% is not a good use of a day.
Use each dataset for what it can answer
Use Crawl Stats for host-level health (robots.txt availability, DNS, connectivity), for Google-side latency, for the discovery-versus-refresh split, and for confirming that a crawl-rate change is global rather than an artifact of your logging.
Use verified logs for anything URL-level: which templates consume crawl budget, which parameter combinations are being fetched, whether your new section is getting hit at all, how deep Googlebot goes into pagination. Crawl Stats shows only example URLs per bucket; logs show all of them.
When the two disagree, work the list above in order. The reconciliation almost always terminates at a CDN cache layer nobody added to the log pipeline.