--- title: "AI Visibility Checker & llms.txt Validator (Free) — Dardo" description: "Check which of 26 AI crawlers your robots.txt allows (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot…), validate your llms.txt and get a fix for each issue." url: "https://dardo.studio/en/tools/ai-visibility-checker/" language: "en" translations: {"es":"https://dardo.studio/es/herramientas/verificador-visibilidad-ia/","pt":"https://dardo.studio/pt/ferramentas/verificador-visibilidade-ia/"} updated: "2026-10-15" --- Free tools # AI visibility checker and llms.txt validator: can ChatGPT, Claude, Perplexity and Google read your site? An AI visibility checker reads the files AI crawlers read first. Enter a domain and this one fetches its robots.txt, llms.txt, llms-full.txt, home page and sitemap, then shows which of 26 AI crawlers may read the site, whether its llms.txt follows the llmstxt.org format and what its structured data tells machines, with a fix for each finding. It can't tell you whether ChatGPT or Google will cite you. It tells you whether something on your site stands in the way. By Dardo editorial team · Updated Oct 15, 2026 · 6 min read [Get a proposal](https://dardo.studio/en/contact/?service=ai-search-optimization) We read only public files: your home page, robots.txt, llms.txt, llms-full.txt and sitemap, and up to 15 links your llms.txt lists. Nothing is stored. **Form fields** - Your site's address ## Key takeaways - Allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) if you want to be cited; training crawlers are a separate decision. - Blocking Google-Extended doesn't keep you out of AI Overviews; noindex, nosnippet and Search Console's generative AI setting do. - llms.txt is optional and Google Search ignores it. If you publish one, keep it valid and its links working. ## How to use the checker 1. **Enter your domain.** The checker follows redirects to the address your home page actually answers at (http to https, example.com to www.example.com) and reads the files there. 2. **Read the crawler table.** Each row is one crawler, named the way its company documents it, with what it does and whether your robots.txt lets it in. The deciding rule is the line of your file that applies to your home page. 3. **Work through the findings.** "To fix" items stop crawlers or break a file, "To review" items deserve a look, and notes explain a choice you've made. Templates and robots.txt lines can be copied straight from the report. 4. **Keep the report.** Download it as Markdown, copy it, or send it to us from the form at the end. ## Search crawlers, user agents and training crawlers OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon and Mistral give their crawlers separate robots.txt tokens, so you can decide by purpose: - **Search crawlers** (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and others) build the indexes that AI answers cite. [OpenAI says](https://developers.openai.com/api/docs/bots) sites that block OAI-SearchBot aren't shown in ChatGPT search answers, and [Anthropic says](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) blocking Claude-SearchBot may reduce your visibility in its search results. If you want to be cited, allow them. - **User agents** (ChatGPT-User, Claude-User, Perplexity-User and others) fetch a page when someone asks an assistant about it. OpenAI and [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) say robots.txt may not apply to these requests, because a person asked for them; Anthropic says its bots honour robots.txt. - **Training crawlers** (GPTBot, ClaudeBot, meta-externalagent and others), plus the tokens Google-Extended and Applebot-Extended, control whether your content is used to train models. OpenAI says GPTBot and OAI-SearchBot are independent, and [Google says](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) Google-Extended doesn't affect inclusion or ranking in Google Search. Blocking them is a business decision, not an error. The checker matches each token the way the robots.txt standard ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)) says: the groups that name the crawler, in any letter case, else the \* group; then the longest matching rule wins, and Allow wins a tie. Where a company says its crawler follows another's rules (Apple's Applebot follows Googlebot's), the checker does too. It tests your home page and up to 50 pages spread across your sitemap. ## What llms.txt is, and what it isn't llms.txt is a [proposal published at llmstxt.org](https://llmstxt.org/) in September 2024 and revised in August 2026: a Markdown file that gives language models a short guide to a site. In order, it has an H1 with the site's name (the only required part), a blockquote summary, optional notes without headings, and H2 sections that list links as "- \[name\](url): notes". The checker validates each part and tests up to 15 of the links. Version 2 lets a file sit at any path; the checker reads the one at your site's root. Keep its weight in perspective. Google's guide to its generative AI features says you don't need AI text files to appear in Google Search, because [Google Search ignores them](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), and none of the AI companies in the table documents its crawlers reading other sites' llms.txt. It's cheap to publish and useful to agents and developer tools, but it isn't a ranking factor. llms-full.txt, a single file with a site's full text, isn't part of the proposal at all. ## What the check can't see - **Firewalls and bot protection.** robots.txt states your preferences; a CDN rule can refuse a crawler that robots.txt allows. The checker identifies itself as DardoAIVisibilityChecker, so if your site refuses it, it may refuse AI crawlers too, but a successful check doesn't prove that crawlers get through. Check your CDN's bot settings and logs. - **Google's own setting.** Search Console has a [Search generative AI control](https://support.google.com/webmasters/answer/16908024) that can exclude a site from AI Overviews and AI Mode. It isn't in robots.txt, so check it in Search Console. - **Indexing and citations.** Being allowed isn't being indexed or cited. To measure where AI answers cite you, see [how to show up in AI answers](https://dardo.studio/en/blog/appear-in-ai-answers/). - **JavaScript.** The checker reads the HTML your server sends, without running scripts, and counts its words. ## Questions before you choose **Does blocking GPTBot remove my site from ChatGPT?** Not from ChatGPT search. OpenAI says GPTBot (training) and OAI-SearchBot (search) are independent: a site can allow OAI-SearchBot to appear in ChatGPT search answers and block GPTBot to keep its content out of training. Blocking OAI-SearchBot is what keeps a site out of those answers. **Does Google use llms.txt?** Not for Search. Google's guide to its generative AI features says Google Search ignores AI text files such as llms.txt: publishing one neither helps nor harms your visibility in Google Search, including AI Overviews and AI Mode. **Which robots.txt rule applies when several match?** Under RFC 9309, a crawler follows the groups that name it (matched in any letter case, and combined if there are several); if none does, it follows the \* group. Within those rules the longest matching path wins, and Allow wins a tie. The checker shows the rule that decides your home page. **My robots.txt allows everything. Can AI crawlers still be blocked?** Yes. robots.txt is a request, not a lock: a CDN, firewall or bot-protection rule can refuse a crawler before it reads anything. Cloudflare, for example, lets sites block selected AI crawlers at its edge. Check those settings and your server logs too. **What's the difference between llms.txt and llms-full.txt?** llms.txt is a short guide with links, defined by the llmstxt.org proposal. llms-full.txt is a convention some documentation sites use for their full text in one file; the proposal doesn't define it. Neither is required, and a marketing site can skip llms-full.txt. **Do you store the sites I check?** No. Our server fetches the site's public files when you press the button and sends the results to your browser; nothing is saved. Checks are limited per connection to prevent abuse. ## Sources & further reading Sources behind this page, with further detail from the original publishers. - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) · developers.openai.com - [Anthropic (Claude Help Center): Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) · support.claude.com - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) · docs.perplexity.ai - [Google Crawling Infrastructure: Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) · developers.google.com - [Google Search Central: Optimizing for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) · developers.google.com - [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) · developers.google.com - [Search Console Help: Search generative AI control](https://support.google.com/webmasters/answer/16908024) · support.google.com - [Apple: About Applebot](https://support.apple.com/en-us/119829) · support.apple.com - [Meta: Meta web crawlers](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) · developers.facebook.com - [Amazon: Amazonbot](https://developer.amazon.com/amazonbot) · developer.amazon.com - [Mistral AI: Robots and user agents](https://docs.mistral.ai/robots) · docs.mistral.ai - [DuckDuckGo: DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) · duckduckgo.com - [Bing Webmaster Tools: Which crawlers does Bing use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) · bing.com - [You.com: YouBot](https://you.com/docs/youbot) · you.com - [Common Crawl: CCBot](https://commoncrawl.org/ccbot) · commoncrawl.org - [ByteDance, Toutiao Search webmaster platform: about Bytespider (in Chinese)](https://zhanzhang.toutiao.com/docs/intro/26899) · zhanzhang.toutiao.com - [IETF RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) · rfc-editor.org - [llmstxt.org: The /llms.txt file (version 2)](https://llmstxt.org/) · llmstxt.org - [Cloudflare: robots.txt setting and Content Signals](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) · developers.cloudflare.com - [Google Search Central: Organization structured data](https://developers.google.com/search/docs/appearance/structured-data/organization) · developers.google.com - [Google Search Central: Site names in Google Search](https://developers.google.com/search/docs/appearance/site-names) · developers.google.com - [sitemaps.org: Sitemaps XML format](https://www.sitemaps.org/protocol.html) · sitemaps.org [Dardo](https://dardo.studio/) · [Free tools](https://dardo.studio/en/tools/) · AI visibility checker ## From a check to being cited See what the platforms document, fix what the check found, then measure it. [01 · **How to show up in AI answers** · What ChatGPT, Perplexity, Google and Copilot document, and how to measure citations.](https://dardo.studio/en/blog/appear-in-ai-answers/) · [02 · **AI search optimization** · An audit, the fixes, and monthly tracking of where AI answers cite you.](https://dardo.studio/en/services/ai-search-optimization/) · [03 · **Ask for an AI visibility audit** · Send your report and we'll reply with a proposal; audits start at USD 690.](https://dardo.studio/en/contact/?service=ai-search-optimization) ## Keep exploring. - [Service · **AI search optimization (GEO): get cited by ChatGPT and Google AI**](https://dardo.studio/en/services/ai-search-optimization/) - [Service · **Technical SEO for new and custom-built websites**](https://dardo.studio/en/services/seo/)