Robots.txt looks like a single on/off switch for AI. It isn’t. Training, search and user-requested retrieval have different purposes and controls. Review each provider’s rules before changing access: a search restriction can affect visibility, but a robots file alone does not establish that every AI mention disappears.
The file that can’t say no
Robots.txt is a request, not a lock. RFC 9309 is explicit that the rules “are not a form of access authorization” — anything sitting behind a Disallow line is still reachable by anyone who fetches the URL directly. The spec also sets the failure modes that matter in practice: if the file 4xxs, a crawler “MAY access any resources”; if it 5xxs, the crawler “MUST assume complete disallow”. Most sites never test either case. And most sites don’t have a file worth testing: as of July 2025, only 37% of the top 10,000 domains had a robots.txt at all, and just 7.8% of those disallowed GPTBot. Robots.txt is a declaration of intent. Your CDN or WAF is the enforcement layer, if you have one.
Three lanes of AI bot, not one
Several providers publish separate user agents for these jobs. Some crawlers serve more than one purpose, so the categories below are a guide, not a universal separation.
Training crawlers
OpenAI separates GPTBot’s training use from OAI-SearchBot search access; Anthropic describes ClaudeBot separately from its search and user-fetch bots. Do not assume every provider makes the same separation. In particular, Google-Extended also governs certain Gemini grounding uses, discussed below. AIEO reports a Disallow on GPTBot, ClaudeBot or Google-Extended as a warning under its chosen rubric, rather than treating all three as equivalent search failures.
Search and answer indexers
Search access is worth checking, but the documented effects differ. OpenAI says sites opting out of OAI-SearchBot are excluded from ChatGPT search answers, although navigational links can remain. Anthropic says disabling Claude-SearchBot prevents its search indexing and may reduce visibility or accuracy. Perplexity recommends allowing PerplexityBot and its published IP ranges. None of these checks measures whether your site is actually cited for a particular question.
User-requested fetchers
ChatGPT-User, Perplexity-User and Google-Agent fetch a page because a person asked the assistant to look at it right now. OpenAI’s own docs say plainly: “because these actions are initiated by a user, robots.txt rules may not apply.” Google says its user-triggered fetchers “generally ignore robots.txt rules”. Perplexity also says its user fetcher generally ignores robots.txt. Claude-User differs: Anthropic says its bots honour the file, and disabling Claude-User prevents that retrieval route. Use the provider-specific rule rather than assuming a common policy.
Control tokens that aren’t crawlers
Google-Extended and Applebot-Extended never touch your logs. Google is explicit that blocking Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” — it only governs Gemini training and grounding, and crawling uses existing Google user-agent strings. Apple says the same of Applebot-Extended: it “does not crawl webpages” and disallowing it doesn’t remove you from Siri or Spotlight results. Bing documents separate meta-tag controls: NOARCHIVE excludes linking in Chat/Copilot and training use, while NOCACHE still permits URL, title and snippet use. If both are present, Bing treats them as NOCACHE; they are not interchangeable opt-outs.
What the numbers say
Training now accounts for nearly 80% of all AI crawler traffic, and Cloudflare’s data shows how lopsided the exchange is: between January and July 2025, Anthropic’s crawl-to-referral ratio ran from 286,930:1 down to 38,066:1, OpenAI sat around 1,100:1, and Perplexity was 195:1 — meaning thousands of pages get crawled for every one visit a site actually receives back. User-triggered crawling, by contrast, grew more than 15x across 2025 as assistants started fetching pages live on request, according to Cloudflare’s 2025 Radar review. The same report found GPTBot, ClaudeBot and CCBot are now the three most fully-disallowed user agents across the top 10,000 domains — site owners are already voting with their robots.txt files, just not always distinguishing the lanes.
What UK sites actually do
Checked live on 5 September 2026, the pattern is inconsistent even among sites that clearly thought about it. The BBC disallows more than 30 AI tokens outright, including OAI-SearchBot, ChatGPT-User and Perplexity-User — the search and user-fetch lanes, not just training — which records restrictions on those retrieval routes, not a measured absence from AI answers. The Guardian is more surgical: it blocks Claude-SearchBot, Claude-User and PerplexityBot but leaves GPTBot and OAI-SearchBot open, and carries an RSL License: line pointing to its own licensing terms. The FT runs a similar setup with a Google-Extended Allow carved out for a syndication parameter. Gov.uk only throttles meta-externalagent on its search path, because — per its own comment in the file — the bot “results in timeouts from Vertex”. NHS, John Lewis, Tesco and the Telegraph carry no AI rules at all. And several of these sites, BBC included, still list dead tokens like anthropic-ai and Claude-Web that vendors retired years ago.
Cloudflare has opinions about your robots.txt
Cloudflare has started writing AI rules on your behalf. Its managed robots.txt is available on every plan, including Free, and is prepended to your existing file when enabled: it Disallows eight named training bots and injects a Content-Signal line — as of August 2026, search=yes, ai-train=no, use=reference — across more than 3.8 million domains. The use field (immediate, reference or full) is new and deliberately leaves ai-input unset. Cloudflare announced a 15 September 2026 change to its defaults: new domains — and, according to Cloudflare’s own announcement, existing Free customers who have not changed their settings — were expected to get the Training and Agent categories blocked on pages carrying ads, while Search stayed open. Cloudflare warned that “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training” — a real risk of losing ordinary search visibility by accident. The announced date has passed, but the announcement does not verify any particular site’s current settings. Check your Cloudflare Security settings and the rules actually served by your domain.
In our recorded 5 September 2026 check, aieo.fyi’s robots.txt carried Cloudflare’s managed Disallow: / for GPTBot, ClaudeBot and Google-Extended sitting directly above our own Allow: / for those same tokens — while our llms.txt separately stated “training permitted”. RFC 9309 recommends preferring an equivalent Allow rule, but that does not prove a WAF permits the request. The v2 scan reports this conflicting policy pattern. That recorded observation does not establish today’s deployed setting or that the setting has since been changed.
Policy patterns to adapt
Open for business (most SMEs): allow everything, with explicit Allow lines for OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User, plus an optional Content-Signal: search=yes, ai-input=yes, ai-train=yes if your CDN supports it.
Search access with selected training opt-outs: choose provider-specific training controls, such as GPTBot and ClaudeBot, while allowing the corresponding search routes you want. Review Google-Extended separately because its scope includes Gemini grounding as well as training. A Content-Signal preference can express your policy, but does not itself enforce access or guarantee citation.
Restrict automated access: set the relevant crawler preferences and use technical access controls where needed. User-fetcher compliance differs, so robots.txt alone is not a reliable barrier.
What to do
- Run
curl -s https://yourdomain/robots.txt | grep -iE 'content-signal|license|gptbot|searchbot|claude|perplexity'and check what’s actually being served, not what’s in your repo. - If you’re on Cloudflare, check Security → Settings for the managed robots.txt toggle and the settings now applied to your zone; do not infer them from the announced 15 September default.
- Decide which uses you want, then review the provider-specific controls for training, search and user-requested retrieval.
- Delete legacy tokens (
anthropic-ai,Claude-Web,cohere-ai) — they do nothing. - Check user-fetcher behaviour per provider: Claude-User honours robots.txt; other providers describe different exceptions.
Technical access controls and copyright permissions are separate questions. The UK government’s March 2026 report says a broad copyright exception with opt-out is no longer its preferred approach and that further evidence and alternatives will be considered. That does not settle the legal effect of every licensing or rights-reservation signal.
Want to see which lane your own robots.txt is actually blocking? Run the free scan at /scan/.