The AI Crawlers in Your robots.txt, and What Blocking Each One Costs

The AI crawler user-agent list has grown quickly, and the default advice — block them all, or allow them all — is wrong in both directions, because they do different things.

The distinction that matters

Some crawlers collect training data. Some fetch pages at the moment a user asks a question, to ground an answer with a citation. Some do both under one name. Blocking the first costs you nothing directly. Blocking the second removes you from AI-mediated answers entirely.

The main agents

Two of these need particular care.

Google-Extended is not Googlebot. It controls whether your content is used for Gemini training and grounding. Blocking it does not affect Google Search ranking or indexing — Google has been explicit about this. Blocking Googlebot does affect search, catastrophically. They are separate tokens and confusing them is the single most expensive mistake in this area.

Applebot-Extended similarly governs AI training use, while Applebot handles Siri and Spotlight search.

The asymmetry

Blocking training crawlers costs little that is measurable. Your content will not be used to improve a future model. Whether that matters to you is a licensing and principle question, not an SEO one.

Blocking search and live-fetch crawlers costs visibility directly. If OAI-SearchBot or Claude-User cannot fetch your page, you cannot be cited in answers those systems produce. As AI interfaces take a growing share of research behavior, that is a real exclusion.

So the configuration most businesses actually want is not "block all AI" or "allow all AI." It is:

# Training crawlers — opt out of dataset collection
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search and live-fetch crawlers — stay eligible for citation
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Whether you want that particular split depends on how you value training use, but the structural point holds: decide per purpose, not per company.

Things worth knowing before you rely on this

robots.txt is voluntary. Well-behaved crawlers honor it. Others do not, and the list of operators who have been credibly reported to ignore it is not empty. If you need enforcement rather than a request, that is a WAF or bot-management problem — Cloudflare, Fastly, and similar services offer AI crawler controls that block at the network layer.

Blocking does not remove what was already collected. A Disallow added today has no effect on training runs completed last year.

The agent list changes. New crawlers appear regularly. A robots.txt written eighteen months ago is missing several. Review it periodically, and check your server logs for user agents you do not recognize.

Directive precedence is by specificity, not order. Within a matching group, Google follows the most specific rule for the path. A common failure is a broad Disallow: / in a wildcard group combined with a specific Allow in another group, applied in a way the author did not intend. Test with a robots.txt tester rather than reasoning about it.

The strategic question

There is a real tension and it deserves to be stated plainly rather than resolved by slogan. Allowing training crawlers contributes your work to systems that may answer questions your content would otherwise have been visited for. Blocking them protects that, and forfeits nothing measurable today.

Allowing search and live-fetch crawlers is a much clearer call for most businesses: those systems cite sources, drive some referral traffic, and increasingly mediate how people research purchases. Excluding yourself from that is a significant decision that should be made deliberately, not inherited from a copy-pasted robots.txt.

The one recommendation that applies universally: go read your current file. A large share of sites are blocking something they did not mean to block, or allowing something they assumed they had blocked.