A lot of sites added this block in 2024 and told their stakeholders they had opted out of AI:
User-agent: Google-Extended
Disallow: /It does not do what most of those stakeholders think. Google has documented that Google-Extended controls whether your content is used to improve Gemini apps and the Vertex AI generative APIs. It does not control AI Overviews or AI Mode, both of which are Search features served from the Search index. If your goal was to disappear from the AI box at the top of the SERP, this directive accomplishes nothing toward it.
The confusion is structural, not careless. Google ships three different kinds of robots tokens — real crawlers, user-triggered fetchers, and pure usage tokens that never make an HTTP request — and they all look identical in a robots.txt file. Only one of those three categories shows up in your logs.
What Google-Extended actually is
Google-Extended is not a user agent that fetches anything. Google's crawler documentation is explicit that it is a standalone product token used for usage control, not for crawling. Content still arrives via Googlebot; the token decides whether that already-fetched content may be used downstream for Gemini app grounding and model improvement.
The practical consequence: you will never see a log line with Google-Extended in the user-agent string, no matter how your robots.txt is configured. If you asked your ops team to "confirm in the logs that Google-Extended is blocked," you gave them an impossible task. The only thing you can verify is that the directive is present and syntactically correct — and that Googlebot, which is a different token, is still allowed.
Apple uses the same pattern with Applebot-Extended: Applebot crawls, Applebot-Extended governs whether the crawled content trains Apple's foundation models. Blocking the extended token has no effect on Siri or Spotlight surfacing your pages.
Which control actually affects AI Overviews
AI Overviews and AI Mode are Search features. They are governed by Search-level controls — the same ones that have governed featured snippets for years. Google's documentation on AI features states that the standard preview controls apply: nosnippet, max-snippet, and data-nosnippet restrict what can appear in AI-generated responses, and noindex removes the page from Search entirely, AI features included.
That is the whole menu, and it comes with an unavoidable trade-off. There is no AI-only opt-out inside Search. If you set nosnippet, you lose your regular text snippet too, and the blue-link result degrades to a title and URL. The decision criterion is simple to state and uncomfortable to answer: is the traffic you believe you are losing to AI Overviews larger than the traffic you would lose by shipping snippet-less results across the whole query set? For most publishers the answer is no, which is why the opt-out is rarely used despite being loudly discussed.
data-nosnippet is the surgical version. It marks specific spans inside the HTML, so you can keep a snippet while fencing off the paragraph that answers the question outright:
<p>The failure mode usually looks like this.</p>
<div data-nosnippet>
<p>The exact remediation steps, which we'd rather you click through for.</p>
</div>This is documented behavior for snippets. Whether it meaningfully reduces AI Overview inclusion in practice is harder to establish, because the model may synthesize an answer from the remaining visible text. Treat it as a constraint on quoted material, not a guaranteed exclusion.
The directive map
| Control | Blue-link snippet | AI Overviews / AI Mode | Gemini app grounding | Shows in logs |
|---|---|---|---|---|
Disallow: / for Googlebot | Removed from Search | Not crawled, may still be referenced | Not crawled | Yes (absence of hits) |
noindex | Removed from Search | Excluded | Excluded | Yes (page still fetched) |
nosnippet | No snippet shown | Preview text restricted | Unchanged by this control | Yes (page still fetched) |
max-snippet:50 | Capped at 50 chars | Preview length capped | Unchanged by this control | Yes |
Disallow: / for Google-Extended | No effect | No effect | Blocked | Never |
The last two rows are where the mental model usually breaks. Snippet controls touch Search surfaces; Google-Extended touches non-Search products. They are orthogonal.
Why the same confusion exists for every other AI vendor
Once you see the three-category structure, the other vendors' tokens sort themselves. OpenAI documents GPTBot for model training, OAI-SearchBot for inclusion in ChatGPT search results, and ChatGPT-User for fetches triggered by a user action in a conversation. Anthropic documents a similar split between its training crawler and its search and user-triggered fetchers. Perplexity documents PerplexityBot for indexing alongside a separate user-triggered token.
The mechanism that matters: blocking a training crawler does not remove you from the corresponding search product, and blocking a search crawler does remove you from it. Sites that blanket-blocked GPTBot in 2023 did not thereby exit ChatGPT's browsing results; sites that later blocked OAI-SearchBot did. If your policy is "don't train on us, but do cite us," you need to block exactly one token per vendor and allow the other. A wildcard User-agent: * block gets this wrong in both directions.
User-triggered fetchers are a third case worth deciding separately. These fire when a person pastes your URL into a chat. Blocking them means a user who explicitly asked to read your page gets an error instead.
You cannot measure any of this in your logs
Because Google-Extended never requests anything, and because AI Overviews are served from the existing index rather than a distinct live fetch with a distinct user agent, log analysis gives you no read on AI surface inclusion. Search Console does not break out AI Overview impressions as a separate dimension either; Google has said AI feature data is folded into the aggregate Performance totals rather than reported separately.
That leaves SERP-side observation — a fixed query panel, checked on a schedule, recording whether an AI block appears and whether your domain is cited. It is sampling, not census, and it is the only measurement available. Build it before you change directives, not after, or you will have no baseline to compare against.
What to do
- Audit what you actually blocked. Grep your robots.txt for
Google-Extended,GPTBot,OAI-SearchBot,Applebot-Extended,CCBot, and any wildcard rules that catch them. Write down, per token, which surface it governs. - Separate the two policy questions. "May models train on us" and "may AI search surfaces cite us" are different decisions with different tokens. Decide each on its own merits.
- Stop promising AI Overview exclusion via robots.txt. The only in-Search levers are
noindex,nosnippet,max-snippet, anddata-nosnippet, and the first three cost you blue-link presentation. - Use
data-nosnippetfor specific answer blocks if you want partial protection without losing the whole snippet — and verify the rendered HTML actually contains the attribute after your framework hydrates. - Stand up SERP-side sampling for the queries you care about, since neither logs nor Search Console will isolate AI surfaces for you.