--- title: "Block AI crawlers in robots.txt: user agents and trade-offs" description: "Which AI crawlers to block in robots.txt and what it costs: 28 user agents from vendor docs, Cloudflare's new defaults, and three robots.txt files to copy." url: "https://dardo.studio/en/blog/block-ai-crawlers-robots-txt/" language: "en" translations: {"es":"https://dardo.studio/es/blog/bloquear-rastreadores-de-ia-robots-txt/","pt":"https://dardo.studio/pt/blog/bloquear-robos-de-ia-robots-txt/"} updated: "2026-10-17T23:23:57.800Z" --- [Blog](https://dardo.studio/en/blog/) # How to block AI crawlers in robots.txt: every user agent, what blocking costs and three files to copy Block AI training crawlers like GPTBot without leaving ChatGPT or Google search: 28 user agents from vendor docs, RFC 9309 rules and three robots.txt files. By [Nicolás Cerón](https://dardo.studio/en/nicolas-ceron/) ·October 17, 2026 ![A misty rail yard at night where trains wait on branching tracks below a single crimson semaphore signal arm raised over the junction.](https://dardo.studio/_astro/01M562V51CG0KFYH6CYMRYCGCQ_Z1ls8n4.webp) ## Key takeaways - OpenAI, Anthropic, Perplexity, Meta, Amazon and Mistral run separate user agents for training, for search and for user-triggered fetches, so you can block training without leaving their search answers. - Google-Extended and Applebot-Extended are control tokens, not crawlers: disallowing them opts out of Gemini and Apple model training without affecting Google Search or Apple search. - Under RFC 9309 a group that names a crawler replaces the \* group for that crawler, the longest matching rule wins, and a robots.txt that returns a server error blocks compliant crawlers from everything. - Since 15 September 2026, Cloudflare's Block setting for AI training also stops Googlebot, Bingbot and Applebot; its Disallow AI Training setting keeps them crawling for search. - Of ten large US and UK news publishers we checked on 17 October 2026, all ten block ClaudeBot and CCBot, six block GPTBot, three block OAI-SearchBot and none blocks Googlebot or bingbot. ## The short answer The major AI companies each run separate user agents for model training, for their search index and for pages a user asks them to open. You block or allow each one by naming it in robots.txt: `User-agent: GPTBot` followed by `Disallow: /` asks OpenAI's training crawler to stay away. Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent) has no documented effect on search. Blocking a search crawler or user agent (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) takes you out of that assistant's cited answers. robots.txt is a request, not a lock: compliant crawlers follow it, some user-triggered fetchers say they may not, and only a CDN or firewall rule actually refuses a request. If you want customers to find you, allow search crawlers and user agents, and decide on training separately. ## Should you block GPTBot? Decide by purpose | Purpose | Examples | What blocking does, per the vendors | | -------------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, MistralAI-Training | Asks that your content be left out of training. No vendor says it affects search | | Search index | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Applebot, meta-webindexer | Removes you from that company's search answers and citations | | User-triggered fetch | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User | Stops the assistant opening your page when a person asks, where the vendor honours robots.txt at all | So should you block GPTBot? Block it if you don't want OpenAI training on your future content; OpenAI says GPTBot and OAI-SearchBot are independent, so ChatGPT search can still show you. Leave it allowed if you have no objection: no vendor documents a benefit or a cost in search either way. Two exceptions break the clean split. Google and Microsoft run one crawler for search and their AI features (Googlebot, bingbot), and Apple's Applebot both indexes for search and may feed training. For those, the training opt-out is a separate token or a page-level tag, covered below. ## Every AI crawler's user agent, from each vendor's documentation We read each vendor's own page on 17 October 2026. "robots.txt" says what the vendor itself commits to. | User agent | Vendor | What it does | robots.txt, per the vendor | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | GPTBot | [OpenAI](https://developers.openai.com/api/docs/bots) | Collects content that may train its models | Disallowing it means content shouldn't be used for training | | OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Opted-out sites aren't shown in search answers; changes take about 24 hours | | ChatGPT-User | OpenAI | Visits pages during a user's chat or custom GPT | Rules may not apply; not used to decide search inclusion | | OAI-AdsBot | OpenAI | Checks landing pages of ads submitted to ChatGPT | Visits only submitted ad pages; not used for training | | ClaudeBot | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Collects content that could contribute to training | Honoured; supports Crawl-delay | | Claude-SearchBot | Anthropic | Indexes content for Claude's search results | Honoured; blocking may reduce your visibility | | Claude-User | Anthropic | Fetches pages when a user asks Claude | Honoured; blocking stops those retrievals | | Googlebot | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) | Google Search, including AI Overviews and AI Mode | Followed | | Google-Extended | Google | Control token for Gemini training and grounding in Gemini Apps and Vertex AI | No crawler of its own; doesn't affect Search inclusion or ranking | | GoogleOther | Google | Generic crawler for product teams, such as one-off research crawls | Followed; affects no specific product | | Google-Agent | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) | Agents on Google's infrastructure acting on a user's request | User-triggered fetchers generally ignore robots.txt | | Applebot | [Apple](https://support.apple.com/en-us/119829) | Spotlight, Siri and Safari search; may also help train Apple's models | Followed; without an Applebot group it follows Googlebot's rules | | Applebot-Extended | Apple | Control token for training Apple's foundation models | Doesn't crawl; disallowed pages stay in search | | PerplexityBot | [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) | Perplexity search results; not used for foundation models | Allow it to appear in results | | Perplexity-User | Perplexity | Visits pages to answer a user's question | Generally ignores robots.txt | | meta-externalagent | [Meta](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) | Training foundation models and indexing for Meta products | Disallow it by name; robots.txt cached up to 24 hours | | meta-webindexer | Meta | Meta AI search results | Allowing it helps Meta AI cite and link you | | meta-externalfetcher | Meta | Fetches links at a user's request | May bypass robots.txt | | Amazonbot | [Amazon](https://developer.amazon.com/amazonbot) | Improves Amazon products; may train Amazon AI models | Follows RFC 9309; changes take about 24 hours | | Amzn-SearchBot | Amazon | Search experiences such as Alexa; not training | Unnamed, it follows your rules for other search bots | | Amzn-User | Amazon | Live fetches for Alexa users | May not follow all robots.txt directives | | bingbot | [Microsoft](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) | Bing search; Copilot relies on the same index | Followed; Bing advises not to block it | | MistralAI-Training | [Mistral AI](https://docs.mistral.ai/robots) | Builds training datasets; not search | Mistral says you can disallow it | | MistralAI-Index | Mistral AI | Indexes for Mistral search in its Vibe assistant; not training | Token documented | | MistralAI-User | Mistral AI | Visits pages a Vibe user asks about | Mistral says it governs which sites user requests reach | | DuckAssistBot | [DuckDuckGo](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) | Real-time fetches for AI-assisted answers; not training | Opt-out applies after 72 hours; organic rankings unaffected | | CCBot | [Common Crawl](https://commoncrawl.org/ccbot) | Builds an open web archive anyone can download | Checks robots.txt; obeys Crawl-delay | | Bytespider | [ByteDance](https://zhanzhang.toutiao.com/docs/intro/26899) | Crawler of Toutiao Search (page in Chinese) | Documents its robots.txt matching; says nothing about AI | User-agent strings are easy to fake. OpenAI, Anthropic, Perplexity, Apple, Amazon, DuckDuckGo and Common Crawl publish IP ranges or reverse-DNS names so you can tell a real crawler from an impostor in your logs. ### Copilot and Bing: no separate AI token Microsoft has no AI-only crawler to block. Bing's [Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a) say Bing and Copilot share the same crawling and indexing, list blocking Bingbot among the things to avoid, and move the AI controls to page level: `NOARCHIVE` keeps content out of Copilot responses and grounding results, and `NOCACHE` limits Copilot to the URL, title and snippet. Bing's [2023 announcement](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat) adds that NOARCHIVE content isn't used to train Microsoft's generative AI models and still appears in search results. Cloudflare reports that Microsoft plans to honour a robots.txt "no training" preference from early 2027; Microsoft hasn't documented that itself. ### Google: Google-Extended doesn't touch AI Overviews Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI, not Google Search. AI Overviews and AI Mode draw on Googlebot's index. To keep a site out of them, Google offers the [Search generative AI control](https://support.google.com/webmasters/answer/16908024) in Search Console (rolled out to all sites by 31 August 2026; Google says it doesn't affect training or ranking elsewhere in Search), plus `nosnippet` and `noindex` on pages. ## How robots.txt matching works (RFC 9309) The Robots Exclusion Protocol has been an IETF standard since September 2022, as [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309). The rules that catch people out: - **A named group replaces the** `*****` **group.** A crawler obeys the groups that name its token, matched in any letter case and merged if there are several. Only when no group names it does it fall back to `*`. If your `*` group disallows `/admin/` and you add `User-agent: GPTBot` with `Allow: /`, GPTBot may now crawl `/admin/`. Repeat shared rules in every named group. - **One group can name several crawlers.** Stack `User-agent` lines above one set of rules, as in the files below. - **The longest matching rule wins.** With `Disallow: /blog/` and `Allow: /blog/public/`, a page under `/blog/public/` is allowed. When an Allow and a Disallow match equally, Allow should win. - **An erroring robots.txt blocks everything.** If the file answers with a 5xx server error, compliant crawlers must assume complete disallow; a 4xx lets them crawl anything. - **Each host has its own file.** `shop.example.com` needs its own robots.txt; Anthropic and Amazon both say so. - **Changes aren't instant.** The RFC says crawlers shouldn't use a cached copy for more than 24 hours. OpenAI and Amazon quote about 24 hours, Meta up to 24, DuckDuckGo 72. - **It isn't access control.** The RFC says these rules are not a form of access authorization, and listing a path in robots.txt makes it public. `Crawl-delay` isn't part of the standard. Anthropic and Common Crawl honour it; Apple and Amazon say they don't. ## robots.txt is a request: Cloudflare's AI crawler controls robots.txt states a preference. A crawler that ignores it, or a user-triggered fetcher whose vendor says robots.txt may not apply, still gets your page unless something at the edge refuses it. Anthropic warns the other way too: blocking its IP addresses can stop it reading your robots.txt, so for its bots the documented opt-out is robots.txt itself. Cloudflare, which says it handles traffic for 20% of the web, has changed its defaults several times: - **September 2024:** a one-click option to block AI crawlers. Cloudflare's [July 2025 press release](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) says more than one million customers had turned it on. - **1 July 2025:** every new domain is asked at sign-up whether to allow AI crawlers, and [pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) launched: a crawler either signals intent to pay or gets HTTP 402 with a price, and Cloudflare acts as merchant of record. Its [documentation](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) still calls it a closed beta. - **1 July 2026:** separate Search, Agent and Training controls for every plan, Free included ([changelog](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)). - **August 2026:** [Bot Preference Sync](https://blog.cloudflare.com/bot-preference-sync/) writes those choices into the top of your robots.txt, on by default for new customers. - **15 September 2026:** a new "Disallow AI Training" setting, and a change with real search consequences. Cloudflare's [announcement](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) says Block and "Block on pages with ads" now apply to mixed-use crawlers, including Applebot, Bingbot and Googlebot, so either setting on Training also stops them crawling for search. That last change is the one to check today. On Cloudflare, "block GPTBot" means setting Training to **Disallow AI Training**: Cloudflare says it blocks training-only crawlers such as OpenAI's, Anthropic's, Meta's and Amazon's at the edge, publishes the preference in robots.txt and keeps Googlebot, Bingbot and Applebot crawling for search. Choosing **Block** on Training now stops Googlebot entirely. Cloudflare reports that fewer than 1% of its sites block Search bots, while 17% block training in some way. ## Three robots.txt files to copy Replace the sitemap URL, and keep any rules you already have for private paths in every group (see the first matching rule above). ### 1\. Allow every crawler For most businesses that want to be found and cited. ```txt User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` ### 2\. Allow search and user fetches, block training Search crawlers and user agents fall under `*` and stay allowed; the training crawlers and control tokens are refused. ```txt # Training crawlers and training control tokens User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot Disallow: / # Everyone else, including search crawlers and user fetchers User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` Two judgement calls in that list. Amazon says Amazonbot improves its products and may train its models; Amzn-SearchBot, its search crawler, stays allowed. Common Crawl's page doesn't mention AI, but its archive is open to anyone, so include CCBot if you don't want your pages in a public dataset. Bing has no training token: add `NOARCHIVE` to pages you want kept out of Microsoft's training and Copilot. ### 3\. Block AI crawlers, keep Google and Bing search This refuses training, search and user agents from the AI companies while leaving Googlebot, bingbot and Applebot alone. ```txt User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: MistralAI-Training User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: DuckAssistBot User-agent: CCBot User-agent: Bytespider Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` Know what it costs: you leave ChatGPT search, Claude, Perplexity, Meta AI and DuckDuckGo's AI answers. It doesn't take you out of Google's AI Overviews or Copilot, which use Googlebot's and Bing's indexes; those need the Search Console setting and `NOARCHIVE`. And the fetchers whose vendors say robots.txt may not apply (ChatGPT-User, Perplexity-User, meta-externalfetcher, Google-Agent) need an edge rule if you mean it. ## What large publishers do: 10 robots.txt files we read On 17 October 2026 we downloaded the robots.txt of ten large US and UK news publishers (The New York Times, The Washington Post, CNN, The Guardian, Reuters, AP, BBC, The Wall Street Journal, Forbes and The Atlantic) and evaluated each token for the home page under RFC 9309. | User agent | Blocked by | Purpose | | -------------------------------------------------- | ---------- | --------------------------------------- | | ClaudeBot, PerplexityBot, Applebot-Extended, CCBot | 10 of 10 | Mostly training (PerplexityBot: search) | | Bytespider, Amazonbot | 9 of 10 | Search / training | | meta-externalagent | 8 of 10 | Training | | Claude-SearchBot, Claude-User | 7 of 10 | Search / user fetch | | GPTBot | 6 of 10 | Training | | Google-Extended | 5 of 10 | Training control | | OAI-SearchBot, ChatGPT-User | 3 of 10 | Search / user fetch | | Googlebot, bingbot, Applebot | 0 of 10 | Search | The pattern follows the trade-off: wide blocks on training, while Google's, Bing's and Apple's search crawlers are never refused. Treat it as what a generic client receives: sites can serve different files to different clients (Reddit served us one that refuses every crawler), and publishers' licensing deals don't show in robots.txt. ## How to test your robots.txt 1. **See what each crawler gets.** Our free [AI visibility checker](https://dardo.studio/en/tools/ai-visibility-checker/) evaluates 26 AI crawlers under RFC 9309, names the line that decides each one, and tests your home page and up to 50 pages from your sitemap. It also validates your [llms.txt](https://dardo.studio/en/blog/llms-txt/). 2. **Check that the file answers 200.** A robots.txt that returns a server error blocks compliant crawlers from everything. 3. **Check the edge.** Read your CDN's bot settings (on Cloudflare: Block versus Disallow AI Training) and any firewall rules that match user agents. 4. **Read your logs.** Look for the user agents above and verify their IPs against the vendors' published lists before trusting the name. 5. **Recheck after a day.** OpenAI and Amazon quote about 24 hours for a change to apply; DuckDuckGo quotes 72. Being allowed isn't being cited. What gets a page into AI answers once crawlers can reach it is covered in [how to show up in AI answers](https://dardo.studio/en/blog/appear-in-ai-answers/), and the rest of the agent-facing setup in [what an agent-ready website needs](https://dardo.studio/en/blog/agent-ready-website/). ## What to do next Run the [AI visibility checker](https://dardo.studio/en/tools/ai-visibility-checker/) on your domain and compare its crawler table with the policy you meant to have. If you want us to set the rules, test them at your CDN and track where AI answers cite you, that's part of our [AI search optimization](https://dardo.studio/en/services/ai-search-optimization/) work, with audits from USD 690 on our [pricing](https://dardo.studio/en/pricing/) page. ## Common questions **Should I block GPTBot?** Block it if you don't want OpenAI to use your future content for training. OpenAI says GPTBot and OAI-SearchBot are independent, so blocking GPTBot doesn't remove you from ChatGPT search; blocking OAI-SearchBot does. **Does blocking Google-Extended remove my site from AI Overviews?** No. Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI, and Google says it doesn't affect inclusion or ranking in Google Search. AI Overviews and AI Mode use Googlebot's index; the controls for them are Search Console's Search generative AI setting, nosnippet and noindex. **Do AI crawlers have to obey robots.txt?** No. RFC 9309 says its rules are not a form of access authorization, and some vendors say their user-triggered fetchers may ignore robots.txt. To actually refuse requests you need a CDN, firewall or bot-management rule. **How do I block GPTBot on Cloudflare?** In your zone's security settings, set the Training behaviour to Disallow AI Training. Cloudflare says that blocks training-only crawlers such as GPTBot at the edge and writes the preference into robots.txt while keeping Googlebot, Bingbot and Applebot crawling for search. Choosing Block for Training also stops those search crawlers. **How long does a robots.txt change take to apply?** RFC 9309 says crawlers shouldn't use a cached copy for more than 24 hours. OpenAI and Amazon quote about 24 hours, Meta up to 24 hours and DuckDuckGo 72 hours for DuckAssistBot. **How do I stop Microsoft Copilot from using my content?** Bing has no separate AI crawler token. Its guidelines say the NOARCHIVE robots meta tag keeps content out of Copilot responses and grounding results, and NOCACHE limits Copilot to the URL, title and snippet. Blocking bingbot in robots.txt also stops Bing search from crawling you. ## Sources & further reading Sources behind this page, with further detail from the original publishers. - [IETF: RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) · rfc-editor.org - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) · developers.openai.com - [Anthropic (Claude Help Center): Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) · support.claude.com - [Google Crawling Infrastructure: Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) · developers.google.com - [Google Crawling Infrastructure: Google's user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) · developers.google.com - [Search Console Help: Search generative AI control](https://support.google.com/webmasters/answer/16908024) · support.google.com - [Apple: About Applebot](https://support.apple.com/en-us/119829) · support.apple.com - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) · docs.perplexity.ai - [Meta for Developers: Meta web crawlers](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) · developers.facebook.com - [Amazon: About Amazonbot](https://developer.amazon.com/amazonbot) · developer.amazon.com - [Bing Webmaster Tools: Which crawlers does Bing use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) · bing.com - [Bing Webmaster Tools: Bing Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a) · bing.com - [Bing Webmaster Blog: New options to control usage of content in Bing Chat (September 2023)](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat) · blogs.bing.com - [Mistral AI: Mistral crawlers](https://docs.mistral.ai/robots) · docs.mistral.ai - [DuckDuckGo: DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) · duckduckgo.com - [Common Crawl: CCBot](https://commoncrawl.org/ccbot) · commoncrawl.org - [Toutiao Search webmaster platform: About Bytespider (in Chinese)](https://zhanzhang.toutiao.com/docs/intro/26899) · zhanzhang.toutiao.com - [Cloudflare: Press release, Cloudflare just changed how AI crawlers scrape the internet-at-large (1 July 2025)](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) · cloudflare.com - [Cloudflare blog: Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) · blog.cloudflare.com - [Cloudflare docs: What is pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) · developers.cloudflare.com - [Cloudflare changelog: New options to manage AI traffic (1 July 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) · developers.cloudflare.com - [Cloudflare blog: Bot Preference Sync](https://blog.cloudflare.com/bot-preference-sync/) · blog.cloudflare.com - [Cloudflare blog: Have it both ways, stay discoverable in search while disallowing AI training (15 September 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) · blog.cloudflare.com [Nicolás Cerón](https://dardo.studio/en/nicolas-ceron/) Nicolás Cerón is the founder of Dardo, a brand, web design and development studio in Bogotá, Colombia. ## Related articles - [**llms.txt: what it is, who actually reads it, an example to copy and how to validate it**llms.txt is a Markdown guide to your site for AI agents. Google Search ignores it; coding agents use it. The format, a template and how to validate it.](https://dardo.studio/en/blog/llms-txt/) - [**How to show up in AI answers in 2026: what ChatGPT, Perplexity, Google and Copilot actually document**There is no switch for appearing in ChatGPT, Perplexity, Google's AI Overviews or Copilot. Here is what each platform documents about being cited, and how to measure it without guessing.](https://dardo.studio/en/blog/appear-in-ai-answers/) - [**What an agent-ready website needs: notes from building dardo.studio**We made dardo.studio readable and callable by AI agents, one layer at a time. Here is what each layer does, who documents reading it today, and which ones we would build again.](https://dardo.studio/en/blog/agent-ready-website/) - [**How much does SEO cost? US agency prices, audits, hourly rates and what a small business pays per month**US agencies that publish SEO prices start at $299 to $5,000 a month; Clutch puts US firms at $100 to $149 an hour. What each price buys, and what to avoid.](https://dardo.studio/en/blog/seo-cost/) ## Keep exploring. - [Service · **AI search optimization (GEO): get cited by ChatGPT and Google AI**](https://dardo.studio/en/services/ai-search-optimization/) - [Your project · **Start a conversation**](https://dardo.studio/en/contact/)