Every other SEO data source is an inference. Search Console tells you what Google reported, crawlers tell you what your site looks like to a simulated bot, and rank trackers tell you what a query returned from one location at one moment. Server logs tell you what actually happened: which bot requested which URL, when, and what your server returned.
Getting the data
Access logs live wherever your stack puts them — /var/log/nginx/access.log on a typical Nginx box, an S3 bucket if you are behind CloudFront, the Logs Explorer on Google Cloud, or a Cloudflare Logpush destination. If you are on a managed platform that does not expose raw logs, this analysis is not available to you, which is worth knowing before you plan around it.
You want at minimum: timestamp, requested path, status code, user agent, and response size. A week of data is enough to start; a month is better for seeing patterns.
Verify the bot before you trust the label
User agent strings are trivially spoofed, and a meaningful share of traffic claiming to be Googlebot is not. Google publishes IP ranges, and reverse DNS verification is the traditional check: resolve the IP to a hostname, confirm it ends in googlebot.com or google.com, then forward-resolve that hostname back to the same IP.
Skipping verification does not just add noise — it actively misleads, because scrapers impersonating Googlebot often crawl very differently from the real thing, and their pattern will contaminate every conclusion you draw.
The questions worth asking
Which URLs get crawled most? Sort request counts by path. The distribution is nearly always more lopsided than anyone expects, and the top of the list frequently contains URLs nobody intended to be important — a paginated archive, a faceted filter combination, a legacy redirect chain.
awk '$9 ~ /^2/ {print $7}' access.log | sort | uniq -c | sort -rn | head -50What share of crawl hits non-200 responses? Every request that returns a 404, 301, or 500 is crawl capacity spent on nothing. A high redirect share usually means internal links point at old URLs — fix the links, not just the redirects.
Which important URLs are not being crawled at all? This is the question no other tool answers. Take your sitemap, take your crawled-URL list from logs, and diff them. URLs in the sitemap that no bot has requested in thirty days are either unreachable through your internal link graph or being actively deprioritized.
How deep does crawl reach? Bucket requests by URL path depth. If crawl volume collapses past the third level, your architecture is burying content, and no amount of on-page work will surface it.
What crawl budget actually is
Google describes crawl budget as the product of two things: crawl capacity (how much your server can take without degrading) and crawl demand (how much Google wants your content). Most sites are not capacity-limited. Most sites are demand-limited, and demand is a function of perceived quality and update frequency.
This matters because the common advice — "improve crawl budget by speeding up the server" — only helps sites that are actually hitting capacity limits. You can tell from the logs: if crawl rate drops when response times rise, you are capacity-limited. If crawl rate is flat regardless of response time, you are demand-limited, and the fix is content and link equity, not infrastructure.
The patterns that indicate a problem
Parameter explosion. Thousands of unique URLs differing only in query strings, all being crawled. Faceted navigation is the usual cause. The fix is some combination of canonical tags, robots.txt disallow rules for parameter patterns, and — most effectively — not linking to the combinations you do not want crawled.
Crawl concentrated on stale content. If Google spends most of its budget on content you updated years ago and rarely touches your new work, your internal linking is pointing backward. New content needs links from pages that already get crawled.
Sudden drops. A step change in crawl volume usually has a specific cause — a robots.txt change, a server error spike, a DNS problem, a sudden increase in response time. The log timestamp tells you exactly when it started, which usually tells you what changed.
Scale expectations
For a site under a few thousand URLs, this analysis takes an afternoon with awk and a spreadsheet, and you probably need to do it once a year unless something breaks. For sites in the hundreds of thousands of URLs, it justifies dedicated tooling and a recurring cadence.
The value is not proportional to site size, though. Small sites with a crawl problem tend to have a severe one — an entire section unreachable, a canonical loop, a staging subdomain competing with production — and those are exactly the problems that no amount of content work will overcome and that only the logs will reveal.