How to block AI crawlers in robots.txt: every user agent, what blocking costs and three files to copy

The short answer

The major AI companies each run separate user agents for model training, for their search index and for pages a user asks them to open. You block or allow each one by naming it in robots.txt: User-agent: GPTBot followed by Disallow: / asks OpenAI's training crawler to stay away.

Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent) has no documented effect on search. Blocking a search crawler or user agent (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) takes you out of that assistant's cited answers. robots.txt is a request, not a lock: compliant crawlers follow it, some user-triggered fetchers say they may not, and only a CDN or firewall rule actually refuses a request. If you want customers to find you, allow search crawlers and user agents, and decide on training separately.

Should you block GPTBot? Decide by purpose

So should you block GPTBot? Block it if you don't want OpenAI training on your future content; OpenAI says GPTBot and OAI-SearchBot are independent, so ChatGPT search can still show you. Leave it allowed if you have no objection: no vendor documents a benefit or a cost in search either way.

Two exceptions break the clean split. Google and Microsoft run one crawler for search and their AI features (Googlebot, bingbot), and Apple's Applebot both indexes for search and may feed training. For those, the training opt-out is a separate token or a page-level tag, covered below.

Every AI crawler's user agent, from each vendor's documentation

We read each vendor's own page on 17 October 2026. "robots.txt" says what the vendor itself commits to.

User-agent strings are easy to fake. OpenAI, Anthropic, Perplexity, Apple, Amazon, DuckDuckGo and Common Crawl publish IP ranges or reverse-DNS names so you can tell a real crawler from an impostor in your logs.

Copilot and Bing: no separate AI token

Microsoft has no AI-only crawler to block. Bing's Webmaster Guidelines say Bing and Copilot share the same crawling and indexing, list blocking Bingbot among the things to avoid, and move the AI controls to page level: NOARCHIVE keeps content out of Copilot responses and grounding results, and NOCACHE limits Copilot to the URL, title and snippet. Bing's 2023 announcement adds that NOARCHIVE content isn't used to train Microsoft's generative AI models and still appears in search results. Cloudflare reports that Microsoft plans to honour a robots.txt "no training" preference from early 2027; Microsoft hasn't documented that itself.

Google: Google-Extended doesn't touch AI Overviews

Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI, not Google Search. AI Overviews and AI Mode draw on Googlebot's index. To keep a site out of them, Google offers the Search generative AI control in Search Console (rolled out to all sites by 31 August 2026; Google says it doesn't affect training or ranking elsewhere in Search), plus nosnippet and noindex on pages.

How robots.txt matching works (RFC 9309)

The Robots Exclusion Protocol has been an IETF standard since September 2022, as RFC 9309. The rules that catch people out:

• A named group replaces the * group. A crawler obeys the groups that name its token, matched in any letter case and merged if there are several. Only when no group names it does it fall back to *. If your * group disallows /admin/ and you add User-agent: GPTBot with Allow: /, GPTBot may now crawl /admin/. Repeat shared rules in every named group.
• One group can name several crawlers. Stack User-agent lines above one set of rules, as in the files below.
• The longest matching rule wins. With Disallow: /blog/ and Allow: /blog/public/, a page under /blog/public/ is allowed. When an Allow and a Disallow match equally, Allow should win.
• An erroring robots.txt blocks everything. If the file answers with a 5xx server error, compliant crawlers must assume complete disallow; a 4xx lets them crawl anything.
• Each host has its own file. shop.example.com needs its own robots.txt; Anthropic and Amazon both say so.
• Changes aren't instant. The RFC says crawlers shouldn't use a cached copy for more than 24 hours. OpenAI and Amazon quote about 24 hours, Meta up to 24, DuckDuckGo 72.
• It isn't access control. The RFC says these rules are not a form of access authorization, and listing a path in robots.txt makes it public.

Crawl-delay isn't part of the standard. Anthropic and Common Crawl honour it; Apple and Amazon say they don't.

robots.txt is a request: Cloudflare's AI crawler controls

robots.txt states a preference. A crawler that ignores it, or a user-triggered fetcher whose vendor says robots.txt may not apply, still gets your page unless something at the edge refuses it. Anthropic warns the other way too: blocking its IP addresses can stop it reading your robots.txt, so for its bots the documented opt-out is robots.txt itself.

Cloudflare, which says it handles traffic for 20% of the web, has changed its defaults several times:

• September 2024: a one-click option to block AI crawlers. Cloudflare's July 2025 press release says more than one million customers had turned it on.
• 1 July 2025: every new domain is asked at sign-up whether to allow AI crawlers, and pay per crawl launched: a crawler either signals intent to pay or gets HTTP 402 with a price, and Cloudflare acts as merchant of record. Its documentation still calls it a closed beta.
• 1 July 2026: separate Search, Agent and Training controls for every plan, Free included (changelog).
• August 2026: Bot Preference Sync writes those choices into the top of your robots.txt, on by default for new customers.
• 15 September 2026: a new "Disallow AI Training" setting, and a change with real search consequences. Cloudflare's announcement says Block and "Block on pages with ads" now apply to mixed-use crawlers, including Applebot, Bingbot and Googlebot, so either setting on Training also stops them crawling for search.

That last change is the one to check today. On Cloudflare, "block GPTBot" means setting Training to Disallow AI Training: Cloudflare says it blocks training-only crawlers such as OpenAI's, Anthropic's, Meta's and Amazon's at the edge, publishes the preference in robots.txt and keeps Googlebot, Bingbot and Applebot crawling for search. Choosing Block on Training now stops Googlebot entirely. Cloudflare reports that fewer than 1% of its sites block Search bots, while 17% block training in some way.

Three robots.txt files to copy

Replace the sitemap URL, and keep any rules you already have for private paths in every group (see the first matching rule above).

1. Allow every crawler

For most businesses that want to be found and cited.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

2. Allow search and user fetches, block training

Search crawlers and user agents fall under * and stay allowed; the training crawlers and control tokens are refused.

# Training crawlers and training control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# Everyone else, including search crawlers and user fetchers
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Two judgement calls in that list. Amazon says Amazonbot improves its products and may train its models; Amzn-SearchBot, its search crawler, stays allowed. Common Crawl's page doesn't mention AI, but its archive is open to anyone, so include CCBot if you don't want your pages in a public dataset. Bing has no training token: add NOARCHIVE to pages you want kept out of Microsoft's training and Copilot.

3. Block AI crawlers, keep Google and Bing search

This refuses training, search and user agents from the AI companies while leaving Googlebot, bingbot and Applebot alone.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Know what it costs: you leave ChatGPT search, Claude, Perplexity, Meta AI and DuckDuckGo's AI answers. It doesn't take you out of Google's AI Overviews or Copilot, which use Googlebot's and Bing's indexes; those need the Search Console setting and NOARCHIVE. And the fetchers whose vendors say robots.txt may not apply (ChatGPT-User, Perplexity-User, meta-externalfetcher, Google-Agent) need an edge rule if you mean it.

What large publishers do: 10 robots.txt files we read

On 17 October 2026 we downloaded the robots.txt of ten large US and UK news publishers (The New York Times, The Washington Post, CNN, The Guardian, Reuters, AP, BBC, The Wall Street Journal, Forbes and The Atlantic) and evaluated each token for the home page under RFC 9309.

The pattern follows the trade-off: wide blocks on training, while Google's, Bing's and Apple's search crawlers are never refused. Treat it as what a generic client receives: sites can serve different files to different clients (Reddit served us one that refuses every crawler), and publishers' licensing deals don't show in robots.txt.

How to test your robots.txt

1. See what each crawler gets. Our free AI visibility checker evaluates 26 AI crawlers under RFC 9309, names the line that decides each one, and tests your home page and up to 50 pages from your sitemap. It also validates your llms.txt.
2. Check that the file answers 200. A robots.txt that returns a server error blocks compliant crawlers from everything.
3. Check the edge. Read your CDN's bot settings (on Cloudflare: Block versus Disallow AI Training) and any firewall rules that match user agents.
4. Read your logs. Look for the user agents above and verify their IPs against the vendors' published lists before trusting the name.
5. Recheck after a day. OpenAI and Amazon quote about 24 hours for a change to apply; DuckDuckGo quotes 72.

Being allowed isn't being cited. What gets a page into AI answers once crawlers can reach it is covered in how to show up in AI answers, and the rest of the agent-facing setup in what an agent-ready website needs.

What to do next

Run the AI visibility checker on your domain and compare its crawler table with the policy you meant to have. If you want us to set the rules, test them at your CDN and track where AI answers cite you, that's part of our AI search optimization work, with audits from USD 690 on our pricing page.