Allow the AI crawlers that can send you readers; make a deliberate choice about the ones that only take. Concretely: let the search and user-fetch bots through (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User), decide on the training-only crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) according to how you feel about training, and remember that robots.txt is a request — enforcement lives at your reverse proxy. For a site whose business is being found, the default below allows almost everything and blocks one notorious rule-ignorer mostly on principle.
The inventory, mid-2026
| User-agent | Operator | Feeds | Blocking costs you |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Nothing immediate; possible absence from future model knowledge |
| OAI-SearchBot | OpenAI | ChatGPT search index | Citations in ChatGPT answers |
| ChatGPT-User | OpenAI | Live fetches for a user's question | Being readable at the moment someone asks about you |
| ClaudeBot | Anthropic | Model training | Same trade as GPTBot |
| Claude-User | Anthropic | Live fetches for a user | Claude reading and citing you on request |
| PerplexityBot | Perplexity | Perplexity's search index | Perplexity citations |
| CCBot | Common Crawl | Open crawl dataset | Presence in many downstream models and research corpora |
| Google-Extended | Gemini training (a control token, not a crawler) | Nothing in Search or AI Overviews | |
| Applebot-Extended | Apple | Apple foundation-model training opt-out | Nothing in Siri/Spotlight search |
| Bytespider | ByteDance | Training | Little — and it's widely reported to ignore robots.txt anyway |
Verify current user-agent strings and published IP ranges against OpenAI's bot documentation and the operators' equivalents; names and behaviours have changed more than once and will again.
The trade-offs nobody states plainly
You cannot opt out of AI Overviews and stay in Google Search. AI Overviews are built from Googlebot's ordinary crawl. Google-Extended only controls Gemini training; blocking it changes nothing about Search. The only lever against Overviews is nosnippet/max-snippet, which degrades your regular results too. Price it accordingly.
Blocking GPTBot does not remove you from ChatGPT answers. Search citations flow through OAI-SearchBot and ChatGPT-User; GPTBot is the training pipeline. Sites that blocked GPTBot in 2023 expecting to vanish from ChatGPT got the opposite of what they wanted: still summarised, no longer accurately.
CCBot is the odd one out. Common Crawl feeds academic research and dozens of smaller models, not one vendor. Blocking it is the broadest possible opt-out for the least targeted reason. I leave it open.
Citation traffic is small but real. On this site, AI-referred sessions are low single digits as a share of traffic, arriving with high intent. That's the revenue side of the ledger when you weigh training opt-outs; the measurement setup is in how to get cited by AI search.
A sane default policy
# robots.txt — AI crawler policy
# Search-index and user-fetch bots: allow. They produce citations.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-User
Allow: /
# Training-only bots: a judgement call. This site allows;
# flip these to "Disallow: /" if your content is the product.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: CCBot
Allow: /
# Ignores the rules anyway; blocked here for the record,
# enforced at the proxy.
User-agent: Bytespider
Disallow: /The judgement call is real, and it flips with your content model. A directory monetised by being discovered should allow nearly everything. A photographer or a paywalled publication is handing over the product itself, and blocking every trainer is coherent. The mistake is not picking either side; it's copying someone else's robots.txt without knowing which side they were on.
Two syntax gotchas that quietly invert policies like this one. First, group matching is exclusive: a bot that matches User-agent: GPTBot ignores your User-agent: * rules entirely, so any path exclusions in the wildcard group must be repeated in every named group that should share them. Second, precedence between Allow and Disallow goes to the longest matching rule, not the first one. When in doubt, test a URL against your file with Google's open-source robots.txt parser rather than reasoning it out at 11pm.
robots.txt is not enforcement
Compliant operators honour the file; the ones you actually worry about don't, and anyone can spoof a Googlebot user-agent string. Three layers of real enforcement, in escalating order of effort: verify claimed bots against their published IP ranges (OpenAI, Google, and Apple all publish JSON lists); rate-limit aggressive agents at Caddy or whatever fronts your stack; and subscribe the proxy to CrowdSec, whose community blocklists include the scrapers that rotate user-agents to dodge per-name rules. Worth doing for load reasons alone — AI crawlers now account for a sizeable fraction of all bot traffic, and an unthrottled scraper hammering 54,000 URLs is a hosting bill, not a philosophy question.
What I'd do
Publish an explicit AI-crawler section in robots.txt this week, even if every rule is Allow — an explicit policy beats a default you never chose. Allow the search and user-fetch bots unconditionally. Allow the trainers if discovery is your business model; block them if content is your product. Block Bytespider, then back the whole file with IP verification and rate limits at the proxy, because a policy without enforcement is a suggestion. Revisit the inventory twice a year — this list has never stayed still for twelve months.