Separating Real Googlebot From Spoofed Hits in Logs

Most log file analysis starts with a filter like user_agent LIKE '%Googlebot%'. That string is self-reported. Anyone can send it, and plenty of scrapers do, because sites that would block a generic Python client often wave Googlebot through.

The error runs in both directions. You count requests that were never Google, and you count requests that were Google but had nothing to do with indexing your pages for Search. Both distort the numbers you use to argue about crawl budget, and the second one quietly corrupts time-to-index measurements.

Why user-agent filtering fails in both directions

False positives. The User-Agent header is attacker-controlled. Scrapers, vulnerability scanners, price-monitoring bots, and plenty of commercial SEO crawlers in their default configuration all send Googlebot strings. These clients don't behave like Googlebot: they hammer faceted URLs, follow every parameter combination, and ignore robots.txt. That behavior is exactly what shows up in a "crawl waste" report.

False negatives — or rather, false attributions. Google operates many crawlers, and only some of them are building the Search index. Google-InspectionTool fetches the page when you run URL Inspection or the Rich Results Test. GoogleOther serves internal, non-Search use cases. Google-Extended governs Gemini and Vertex AI grounding use, not indexing. AdsBot checks landing page quality for ads. If you bucket all of these as "Googlebot crawl," your crawl-frequency chart partly measures your own QA activity.

How reverse DNS verification actually works

Google documents a two-step check, and both steps matter:

1. Reverse DNS lookup on the client IP. The PTR record should resolve to a hostname under googlebot.com, google.com, or googleusercontent.com.
2. Forward DNS lookup on that hostname. It must resolve back to the original IP.

Step one alone is worthless, because whoever controls an IP block usually controls its PTR records and can publish anything. Step two closes the loop: a spoofer can claim the hostname crawl-66-249-66-1.googlebot.com, but they cannot make Google's authoritative DNS point that name at their server.

# Step 1: reverse
dig +short -x 66.249.66.1
# -> crawl-66-249-66-1.googlebot.com.

# Step 2: forward-confirm
dig +short crawl-66-249-66-1.googlebot.com
# must return 66.249.66.1

The cost is two DNS round trips. Run it on unique IPs per day, not per log row, cache the verdict, and the cost collapses to something trivial even for large logs.

When to use the published IP range files instead

Google publishes machine-readable IP range lists as JSON, which lets you verify offline with no DNS dependency. The crawler documentation splits them into buckets:

Two caveats. First, ranges change; fetch the files on a schedule rather than hardcoding CIDRs into a query. Second, a range match proves the request came from Google, not the request was Search crawling. Classification requires IP verdict and user-agent string together — the UA tells you which crawler, the IP tells you whether to believe it.

The crawlers that fail verification for legitimate reasons

Google documents that some user-triggered fetchers do not run from Google's own address space. A fetch kicked off by a user action can originate from infrastructure outside the published ranges, which means a strict "not in googlebot.json → spoof" rule will mislabel real Google traffic.

The practical consequence: don't collapse verification into a boolean. Keep at least five classes:

• Verified Googlebot (Search crawl)
• Verified Google, non-Search crawler (InspectionTool, GoogleOther, AdsBot, Extended)
• Claims Google UA, fails verification
• Known other bot (Bingbot, AI crawlers, monitoring)
• Everything else

How misattribution changes your conclusions

Crawl waste is overstated. Spoofed traffic concentrates on exactly the URL patterns people flag as waste — sort parameters, deep pagination, internal search results — because scrapers enumerate rather than follow signals. Blocking those paths in robots.txt to "save crawl budget" then saves nothing Google was spending, and it may cost you discovery paths that real Googlebot was using.

Crawl rate time series get noisy. A two-day scraping burst looks identical to a demand-driven crawl increase. If you correlate crawl rate with a publishing change or a server speed fix, unverified traffic is pure confounder.

Status code mix gets skewed. Scrapers generate 404s at rates real crawlers don't. A report showing "18% of Googlebot requests return 404" often dissolves to a normal number after verification.

Index coverage analysis gets circular. If Google-InspectionTool hits count as crawl, then every URL you manually inspected appears to have been crawled by Google — and any time-to-index or crawl-to-index analysis built on that column is measuring your own clicks.

Building the pipeline

The implementation that scales is a daily verdict table keyed on IP, joined back to raw log rows:

import ipaddress, json

ranges = [ipaddress.ip_network(p["ipv6Prefix"] if "ipv6Prefix" in p
           else p["ipv4Prefix"]) for p in json.load(open("googlebot.json"))["prefixes"]]

def in_googlebot_ranges(ip: str) -> bool:
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in ranges)

For volume, load the prefixes into a CIDR trie rather than looping. Handle IPv6 — a meaningful share of Googlebot traffic arrives over it, and a pipeline that silently drops non-IPv4 addresses will report a crawl collapse that never happened.

One more failure mode: if your logs come from behind a CDN or load balancer, the remote_addr field may be the edge node, not the client. Every verification check then fails. Pull the real client IP from X-Forwarded-For (leftmost trusted hop) or your CDN's dedicated header before you verify anything.

What to do next

1. Compute the share of Googlebot-UA requests in the last 30 days that fail forward-confirmed verification. Until you have that number, treat every crawl conclusion you've drawn as provisional.
2. Store the verdict as a column on the log table, not as a filter applied at query time, so historical analyses are reproducible after range files change.
3. Split verified Google traffic by crawler name and exclude Google-InspectionTool and GoogleOther from anything you call "crawl coverage."
4. Don't block on the user-agent string. It blocks no real scraper. Rate-limit or challenge unverified clients claiming a Googlebot UA, and leave verified ones alone.
5. Re-run the crawl waste report on verified traffic only, then decide what to disallow.