What Blocking GPTBot and Google-Extended Actually Does

Most "block AI bots" advice treats the problem as a single switch. It isn't. There are at least four distinct operations an AI vendor can perform on your content, they are controlled by different tokens, and several of them are not controlled by robots.txt at all.

The practical consequence: teams regularly block a training crawler, feel protected, and stay fully quotable in the product they were worried about. Others block a retrieval agent and quietly remove themselves from citations in an assistant that was sending them traffic.

What are the four operations you might actually want to control?

Separate these before you touch a config file:

1. Bulk crawling for model training. A crawler harvests pages into a corpus used in pretraining or fine-tuning. Effect on you is diffuse and permanent-ish: once weights are trained, withdrawal does nothing retroactively.
2. Building a retrieval index for an assistant. A separate crawler builds a searchable index the assistant queries at answer time. This is what determines whether you can be cited.
3. Live retrieval-time fetching. The assistant fetches a URL while answering, often because the user pasted a link or the system decided it needs fresh content. Usually a different user-agent token, sometimes not a crawler at all in the robots.txt sense.
4. Grounding an answer from an existing search index. The model never fetches you directly. It reads a snippet from an index built by a conventional search crawler. This is how AI Overviews work.

Operations 1–3 are crawler behaviors and respond to robots.txt. Operation 4 does not, because no AI-specific crawler ever visited you.

Which token controls which operation?

Vendor tokens change, so treat this as a map of the categories rather than a permanent list — verify against each vendor's current documentation before deploying.

The pattern is consistent: vendors that offer a training opt-out give it a separate token from the crawler that keeps their search or assistant product working. That separation is deliberate, and it is the whole game. Google-Extended is not even a crawler — it is a control token with no traffic of its own, which is why you will never see it in your logs.

Why can't robots.txt remove you from AI Overviews?

Because AI Overviews are built on the Search index. Google has been explicit that its AI features in Search use Googlebot and the standard Search infrastructure, and that Google-Extended does not affect them. There is no token that removes you from AI Overviews while leaving you in the ten blue links.

What you do have is the preview controls. Google has documented that AI experiences in Search respect nosnippet, max-snippet, and data-nosnippet. So:

<meta name="robots" content="nosnippet">

will keep your text out of an AI Overview — and out of your regular search snippet, and out of most rich result treatments. data-nosnippet on specific elements is the surgical version:

<p data-nosnippet>Proprietary pricing detail we do not want quoted.</p>

The trade-off is real and it is the decision criterion: if the snippet is how users qualify themselves before clicking, suppressing it costs more than the extraction does. Most publishers should not use nosnippet sitewide. Use it on the specific content that has extraction risk.

Does blocking the training crawler stop you appearing in answers?

No, and this is the most common misread. Three reasons:

• Retrieval beats training for factual answers. An assistant answering "what does X cost" is far more likely to be reading a live or recently indexed page than reciting weights. Blocking the training crawler leaves the retrieval path untouched.
• Third-party corpora carry your content anyway. Common Crawl archives are widely used. Blocking CCBot today does not unpublish the snapshots that already exist.
• Syndication and aggregation launder the block. If your content is quoted, syndicated, or scraped onto domains that permit crawling, it enters the corpus through them.

The honest framing: robots.txt controls are a distribution decision, not a copyright enforcement mechanism.

How do you verify any of this is being honored?

robots.txt is voluntary. Compliance is a claim, and you should measure it.

• Match user-agent against published IP ranges. OpenAI, Anthropic, and Google publish IP ranges or support reverse DNS verification. A request claiming GPTBot from an unlisted address is a forgery and should be treated as one.
• Log the token, not just "bot: yes/no". Aggregate hits per AI user-agent per day, per directory. If you disallow a token and hits continue from verified IPs, you have an evidence trail worth escalating.
• Watch for undeclared agents. Cloudflare published research in 2025 alleging Perplexity used undisclosed user-agents to reach content that blocked its declared crawler; Perplexity disputed the characterization. Regardless of who is right there, the class of behavior exists, and only WAF-level fingerprinting catches it.
• Remember noindex is irrelevant here. A crawler that is not building a search index has no reason to honor an indexing directive. Only Disallow (crawl) and blocking (enforcement) apply.

What to actually do

1. Decide per-operation, not per-vendor. Most sites want: visible in assistant search results, excluded from bulk training corpora. That means allowing the search-index tokens and disallowing the training tokens — the opposite of a blanket block.
2. Write the file explicitly, one group per token, and comment why. Future-you will not remember which token was training.
3. Use data-nosnippet for the specific passages you do not want quoted verbatim, and resist sitewide nosnippet unless you have measured that snippets are not earning clicks.
4. Add AI user-agents as a dimension in your log analysis so you can answer "did the block work" with data instead of assumption.
5. If you need enforcement rather than a request, do it at the WAF or CDN with IP verification. robots.txt is a sign, not a lock.
6. Re-check vendor documentation quarterly. Tokens split, get renamed, and gain new purposes faster than any blog post can track.