Separating Real Googlebot From Fakes in Your Logs

Your log analysis says Googlebot requested 400,000 URLs last month. Some meaningful share of those requests were not Google. A user-agent string is a header the client chooses; anyone can send Googlebot/2.1 and most servers will happily log it as Google.

This matters because crawl analysis is the one SEO dataset you own end to end, and it is the dataset most people never validate. If you are sizing crawl budget, diagnosing discovery problems, or building a case that your server is throttling Google, unverified logs will point you at the wrong problem.

Why user-agent filtering fails

Scrapers imitate Googlebot deliberately, because a lot of infrastructure treats it as privileged traffic. Sites whitelist it past rate limits, past bot challenges, past consent walls, and sometimes past paywalls. Copying the string costs nothing.

The distortion is rarely uniform. Spoofed traffic clusters — on product detail pages, on search result URLs, on paginated archives, on whatever a competitor or a price aggregator wants. So the error does not cancel out in aggregate. It shows up as a fake pattern: "Google is hammering our faceted URLs" when Google is not. You then spend a sprint on crawl controls that solve nothing.

The two verification methods Google documents

Google documents exactly two ways to confirm a request came from its infrastructure.

Forward-confirmed reverse DNS. Do a reverse lookup on the client IP. The PTR record must resolve to a hostname under googlebot.com, google.com, or googleusercontent.com. Then do a forward lookup on that hostname and confirm it resolves back to the original IP. Both directions are required — reverse DNS alone can be poisoned by whoever controls the PTR record for that IP block.

# illustrative
$ dig +short -x 66.249.66.1
crawl-66-249-66-1.googlebot.com.
$ dig +short crawl-66-249-66-1.googlebot.com
66.249.66.1   # matches the original IP -> verified

Published IP range files. Google publishes JSON lists of the prefixes its crawlers use, split by crawler class: googlebot.json for the main indexing crawlers, special-crawlers.json for things like AdsBot and APIs-Google, and user-triggered-fetchers.json for fetches initiated by a person or product. Match the client IP against the CIDR blocks.

import ipaddress, json, urllib.request

URL = "https://developers.google.com/static/search/apis/ipranges/googlebot.json"
data = json.load(urllib.request.urlopen(URL))
NETS = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"])
        for p in data["prefixes"]]

def is_googlebot(ip: str) -> bool:
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in NETS)

For batch log processing, prefer the CIDR match. It is a local comparison against a few hundred prefixes, so it scales to hundreds of millions of log lines. Reverse DNS requires a network round trip per unique IP, which is fine if you dedupe IPs first and cache results, and unworkable if you do it per request. Refresh the JSON files daily; the prefixes change.

Which Google token means what

Verification tells you the request is really Google. It does not tell you what Google was doing. Different tokens have different rules, and lumping them together is the second most common log-analysis error.

Two things there trip people up. First, Google documents that user-triggered fetchers may ignore robots.txt, because a person explicitly asked for the fetch. Seeing Google-InspectionTool hit a disallowed URL is expected behavior, not a bug. Second, Google-Extended never appears in your logs as a user agent. It exists only so you can allow or disallow use of your content for Gemini and related model training in robots.txt. Searching logs for it and finding nothing proves nothing.

Everyone else has their own scheme

• Bing supports the same forward-confirmed reverse DNS pattern (hostnames under search.msn.com) and also publishes a JSON IP list for bingbot.
• Apple publishes IP ranges for Applebot, and Applebot's documented behavior includes respecting Googlebot rules as a fallback when no Applebot group exists in robots.txt.
• OpenAI publishes separate IP range files for GPTBot (training), OAI-SearchBot (search index), and ChatGPT-User (user-triggered fetch). Treating those three as one line in a dashboard hides the only distinction that matters for policy.
• Perplexity publishes IP ranges, but its crawling behavior is contested: Cloudflare published research in 2025 alleging fetches from undeclared user agents and rotating addresses. Perplexity disputed the characterization. The practical takeaway is not to pick a side but to treat allowlist-by-IP as the only enforceable mechanism, and to expect a residual band of AI-adjacent traffic you cannot attribute.

Where a vendor publishes neither IP ranges nor a verifiable DNS scheme, you cannot verify their crawler at all. Say that in your reporting rather than implying precision you do not have.

What changes once you verify

Expect three shifts, in direction if not magnitude:

1. Total "Googlebot" volume drops, and the drop is concentrated in specific URL patterns rather than spread evenly.
2. Crawl-rate conclusions get less dramatic. Apparent spikes often turn out to be a single unverified ASN.
3. Your response-time picture changes. Verified Googlebot requests and scraper requests hit different caches and different origin paths, so the p95 you report to engineering is a different number.

One caution: verification is for analysis and access control, not for content decisions. Serving different HTML to verified Googlebot than to users is cloaking. Rate-limiting or blocking unverified traffic that claims to be a crawler is fine and defensible; changing what verified crawlers see is not.

What to do this week

1. Add a verified_bot column to your log pipeline, populated by CIDR match against the published range files, refreshed daily.
2. Keep the raw claimed user agent alongside it. The gap between claimed and verified is itself a security signal worth charting.
3. Split reporting by token class: indexing crawlers, ads crawlers, user-triggered fetchers, AI crawlers. Never sum them into one "bot traffic" line.
4. Re-run last quarter's crawl-budget analysis with verification on before you present any of its conclusions again.
5. For unverified requests claiming a major crawler identity, apply challenges or rate limits at the edge — and log the decision so you can measure what you dropped.