Dardo
ENESPT
ENESPT

Free tools

AI visibility checker and llms.txt validator: can ChatGPT, Claude, Perplexity and Google read your site?

An AI visibility checker reads the files AI crawlers read first. Enter a domain and this one fetches its robots.txt, llms.txt, llms-full.txt, home page and sitemap, then shows which of 26 AI crawlers may read the site, whether its llms.txt follows the llmstxt.org format and what its structured data tells machines, with a fix for each finding. It can't tell you whether ChatGPT or Google will cite you. It tells you whether something on your site stands in the way.

By Dardo editorial team6 min read
Get a proposal

We read only public files: your home page, robots.txt, llms.txt, llms-full.txt and sitemap, and up to 15 links your llms.txt lists. Nothing is stored.

On this page

How to use the checker

  1. Enter your domain. The checker follows redirects to the address your home page actually answers at (http to https, example.com to www.example.com) and reads the files there.
  2. Read the crawler table. Each row is one crawler, named the way its company documents it, with what it does and whether your robots.txt lets it in. The deciding rule is the line of your file that applies to your home page.
  3. Work through the findings. "To fix" items stop crawlers or break a file, "To review" items deserve a look, and notes explain a choice you've made. Templates and robots.txt lines can be copied straight from the report.
  4. Keep the report. Download it as Markdown, copy it, or send it to us from the form at the end.

Search crawlers, user agents and training crawlers

OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon and Mistral give their crawlers separate robots.txt tokens, so you can decide by purpose:

  • Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and others) build the indexes that AI answers cite. OpenAI says sites that block OAI-SearchBot aren't shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce your visibility in its search results. If you want to be cited, allow them.
  • User agents (ChatGPT-User, Claude-User, Perplexity-User and others) fetch a page when someone asks an assistant about it. OpenAI and Perplexity say robots.txt may not apply to these requests, because a person asked for them; Anthropic says its bots honour robots.txt.
  • Training crawlers (GPTBot, ClaudeBot, meta-externalagent and others), plus the tokens Google-Extended and Applebot-Extended, control whether your content is used to train models. OpenAI says GPTBot and OAI-SearchBot are independent, and Google says Google-Extended doesn't affect inclusion or ranking in Google Search. Blocking them is a business decision, not an error.

The checker matches each token the way the robots.txt standard (RFC 9309) says: the groups that name the crawler, in any letter case, else the * group; then the longest matching rule wins, and Allow wins a tie. Where a company says its crawler follows another's rules (Apple's Applebot follows Googlebot's), the checker does too. It tests your home page and up to 50 pages spread across your sitemap.

What llms.txt is, and what it isn't

llms.txt is a proposal published at llmstxt.org in September 2024 and revised in August 2026: a Markdown file that gives language models a short guide to a site. In order, it has an H1 with the site's name (the only required part), a blockquote summary, optional notes without headings, and H2 sections that list links as "- [name](url): notes". The checker validates each part and tests up to 15 of the links. Version 2 lets a file sit at any path; the checker reads the one at your site's root.

Keep its weight in perspective. Google's guide to its generative AI features says you don't need AI text files to appear in Google Search, because Google Search ignores them, and none of the AI companies in the table documents its crawlers reading other sites' llms.txt. It's cheap to publish and useful to agents and developer tools, but it isn't a ranking factor. llms-full.txt, a single file with a site's full text, isn't part of the proposal at all.

What the check can't see

  • Firewalls and bot protection. robots.txt states your preferences; a CDN rule can refuse a crawler that robots.txt allows. The checker identifies itself as DardoAIVisibilityChecker, so if your site refuses it, it may refuse AI crawlers too, but a successful check doesn't prove that crawlers get through. Check your CDN's bot settings and logs.
  • Google's own setting. Search Console has a Search generative AI control that can exclude a site from AI Overviews and AI Mode. It isn't in robots.txt, so check it in Search Console.
  • Indexing and citations. Being allowed isn't being indexed or cited. To measure where AI answers cite you, see how to show up in AI answers.
  • JavaScript. The checker reads the HTML your server sends, without running scripts, and counts its words.

Questions before you choose

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. OpenAI says GPTBot (training) and OAI-SearchBot (search) are independent: a site can allow OAI-SearchBot to appear in ChatGPT search answers and block GPTBot to keep its content out of training. Blocking OAI-SearchBot is what keeps a site out of those answers.

Does Google use llms.txt?

Not for Search. Google's guide to its generative AI features says Google Search ignores AI text files such as llms.txt: publishing one neither helps nor harms your visibility in Google Search, including AI Overviews and AI Mode.

Which robots.txt rule applies when several match?

Under RFC 9309, a crawler follows the groups that name it (matched in any letter case, and combined if there are several); if none does, it follows the * group. Within those rules the longest matching path wins, and Allow wins a tie. The checker shows the rule that decides your home page.

My robots.txt allows everything. Can AI crawlers still be blocked?

Yes. robots.txt is a request, not a lock: a CDN, firewall or bot-protection rule can refuse a crawler before it reads anything. Cloudflare, for example, lets sites block selected AI crawlers at its edge. Check those settings and your server logs too.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a short guide with links, defined by the llmstxt.org proposal. llms-full.txt is a convention some documentation sites use for their full text in one file; the proposal doesn't define it. Neither is required, and a marketing site can skip llms-full.txt.

Do you store the sites I check?

No. Our server fetches the site's public files when you press the button and sends the results to your browser; nothing is saved. Checks are limited per connection to prevent abuse.

Sources & further reading

Sources behind this page, with further detail from the original publishers.

From a check to being cited

See what the platforms document, fix what the check found, then measure it.