How to use the llms.txt validator
- Enter your site's address. The validator reads /llms.txt and /llms-full.txt at the root of wherever your home page ends up after redirects, so a site that sends visitors from http to https, or to its www address, is checked where it actually lives.
- Read the llms.txt findings first. Each is marked to fix, to review, a note or OK, and every problem comes with its fix. If you have no llms.txt, or the address returns a web page, the report drafts a starting file: your site's name as the H1, your meta description as the summary and up to eight pages from your sitemap.
- Then read the rest: the table of AI crawlers your robots.txt allows or blocks, your home page's indexing and structured data, and your sitemap.
- Fix, publish and check again. Keep the report as a Markdown file or on your clipboard, or hand it to us through the contact form at the end; nothing is sent until you press Send.
What the validator checks
Each row is one check, in the order the report shows them. The rules come from version 2 of the llms.txt proposal.
| Check | What passes | Common error | Fix |
|---|---|---|---|
| The file answers | Status 200 with a text file at /llms.txt, or a 404 if you don't publish one | A single-page app answers every address, /llms.txt included, with its home page and a 200 | Serve the Markdown file, or return 404 at that address |
| Content type | Served as text/plain or text/markdown | Served as text/html or another type | Set the Content-Type header to text/plain or text/markdown with charset=utf-8 |
| H1 | One H1, on the first line, with the site's or project's name | No H1 at all (the file starts with an H2), text before the H1, or comment lines starting with a hash that Markdown reads as extra H1s | Keep one H1 at the top; delete the comments or turn extra H1s into H2 sections |
| Summary | A blockquote right after the H1 | Missing, or placed after a paragraph, where it no longer counts as the summary | One or two sentences: what you do, for whom and where |
| H2 sections | At least one H2 section, each with a list of links | No H2 sections, empty sections, or H3 headings the format doesn't use | Group pages under H2 headings; turn H3s into H2s or into text above the first H2 |
| List items | Each item is a hyphen, a Markdown link and, optionally, a colon and a note | Bare addresses or sentences without a link under an H2 | Link each item to the page it describes; move facts into the text above the first H2 |
| Links | The first 15 links answer without an error | Pages renamed or removed after the file was written | Update or remove those links, or generate the file from the same source as your pages |
| llms-full.txt | Absent (it's optional), or a text file that starts with an H1 | The address returns your home page as HTML | Return 404 if you don't publish one; start it with an H1 if you do |
A missing llms.txt is only a note, because the file is optional. Links that take more than 8 seconds are listed as unanswered rather than broken, and links after the 15th aren't tested.
The llms.txt format in short
llms.txt is a plain Markdown file, and the proposal fixes the order of its parts so that a script can parse it, not only a model. First an H1 with the site's or project's name, the only required part. Then a blockquote with a short summary. Then, optionally, paragraphs or lists without headings. Then H2 sections, each a list of links. A byte-order mark at the very start is allowed.
A minimal file that passes every check needs five lines, with blank lines between the parts:
- An H1 with your site's name.
- A blockquote with one or two sentences on what you do, for whom and where.
- An H2 naming a group of pages, such as Services.
- A list item for one page: a hyphen, the page's name in square brackets, its full address in parentheses, then a colon and a few words on what the page covers.
- A second item in the same form, for example your contact page.
Version 2, published in August 2026, kept that format. It defined files at other paths (a file covers the pages under its path, and the most specific one applies), accepted page.md as well as page.html.md for Markdown copies of pages, added link relations so a page can point to its Markdown copy and to the llms.txt that covers it, and dropped the special meaning of an H2 named Optional, which is still the usual home for links an agent can skip. The choices behind our own file are in what an agent-ready website needs.
llms.txt vs robots.txt vs sitemap.xml
The three files answer different questions, and none replaces the others.
| File | What it does | Who reads it | Status |
|---|---|---|---|
| robots.txt | Says which paths crawlers may fetch | Search engine and AI crawlers, before they crawl | IETF Proposed Standard (RFC 9309) |
| sitemap.xml | Lists the pages you want crawled, with optional details such as the date of the last change | Search engines | sitemaps.org protocol |
| llms.txt | Gives agents a short, curated guide to the pages that matter | Agents that fetch it when they need information about a site | Informal proposal, version 2 |
robots.txt grants or refuses access; llms.txt grants nothing. Allow and Disallow lines written into llms.txt have no effect, and listing a page in it doesn't unblock that page in robots.txt. A sitemap lists every page you want indexed, while the proposal describes llms.txt as a curated overview small enough for a model's context. Which AI crawlers to let in is a separate decision, covered in our guide to blocking or allowing AI crawlers.
What llms.txt does, and doesn't do, for AI search
A valid llms.txt helps the agents that read it. No AI company documents that it changes rankings or citations.
- Google Search ignores it. Google's guide to its generative AI features says AI text files aren't needed to appear in Search, and that keeping one neither helps nor harms your visibility or rankings.
- AI crawler documentation doesn't mention reading it. The crawler pages of OpenAI, Anthropic and Perplexity, which we reread on 22 October 2026, describe GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and their user-triggered fetchers without saying they read other sites' llms.txt. OpenAI's and Perplexity's pages only point readers to their own documentation index.
- Agents use it. Version 2 expects an agent to read or search the file for what it needs and then follow the links, and the proposal says documentation is where it's used most: think of a coding assistant told to read your API docs.
- Lighthouse checks for it. Chrome's experimental agentic browsing audits flag a server error at /llms.txt and mark a 404 as not applicable (Lighthouse audit). That's an agent-readiness check, not a search signal.
Citations in ChatGPT, Claude, Perplexity and Google's AI features come from pages their search crawlers index, so crawlable pages and a robots.txt that lets those crawlers in do more for them; see how to show up in AI answers. A file also only points to pages: if you want agents to look up live data or take actions, that's the job of an MCP server. Publish an llms.txt when agents need accurate facts about your product, or when your platform writes it for you.
What the check can't see
- Who reads your file. No AI company reports which agents fetched it or what they did with it. Your server logs show requests to /llms.txt, which is the closest evidence you'll get.
- Files at other paths. Version 2 allows files such as /docs/llms.txt; the validator reads only the one at your site's root.
- Whether the facts are right. It checks structure, not truth. Compare the summary and notes with your pages, because an agent will repeat what the file says.
- What the links lead to. It tests whether the first 15 links answer, not whether they point to Markdown copies or to the right pages, and it doesn't read the link relations in your pages.
- The whole of llms-full.txt. It reads the start of the file to confirm it's text that begins with an H1, not everything in it.
- Bot protection. If a firewall or bot rule refuses our checker, the report shows the error; a clean result doesn't prove that AI agents get past the same rules.
If you'd rather have your llms.txt, robots.txt and structured data audited and fixed together, that's part of our AI search optimization work.
Questions before you choose
Do I need an llms.txt?
No. It's optional, and Google says Search ignores it. Publish one if agents need accurate facts about your product, such as documentation, plans or limits, or if your platform generates it for you. If you don't publish one, make /llms.txt return a 404 rather than your home page.
Why does the validator say my llms.txt is a web page?
Because that address returned HTML. Many sites, single-page apps in particular, answer any unknown address with their home page and a 200 status, so /llms.txt seems to exist but holds HTML. Serve a real Markdown file there, or return 404.
Can I block or allow AI crawlers in llms.txt?
No. llms.txt is a guide, not a set of rules: Allow and Disallow lines in it have no effect. Crawler access is set in robots.txt, and a page blocked there stays blocked even if llms.txt links to it.
Will a valid llms.txt get my site cited by ChatGPT or Google?
Nothing documents that. Google says Search ignores the file, and the crawler pages of OpenAI, Anthropic and Perplexity don't say their crawlers read other sites' llms.txt. Citations come from pages their search crawlers index. A valid file does help the agents that read it.
Is llms.txt an official standard?
No. Jeremy Howard published it on 3 September 2024 and revised it as version 2 in August 2026; it lives in a public GitHub repository, and neither the IETF nor the W3C has adopted it. robots.txt, by contrast, is an IETF Proposed Standard (RFC 9309).
Does the validator check llms.txt files at other paths, like /docs/llms.txt?
No. It reads /llms.txt and /llms-full.txt at the root of the address your home page answers at. Version 2 lets a file cover the pages under its own path; to check one of those, open it and apply the same rules: one H1, a summary and lists of links under H2 headings.
Do you store the sites I check?
No. The files are fetched when you run the check, the results go straight to your browser and nothing is kept on our side. There's a per-connection limit on checks to prevent abuse.
Sources & further reading
Sources behind this page, with further detail from the original publishers.
- llmstxt.org: The /llms.txt file (version 2)llmstxt.org
- llmstxt.org: Changes in version 2 (August 2026)llmstxt.org
- Answer.AI on GitHub: llms-txt repositorygithub.com
- Google Search Central: Optimizing for generative AI features on Google Searchdevelopers.google.com
- Chrome for Developers: Lighthouse llms.txt auditdeveloper.chrome.com
- Chrome for Developers: Lighthouse agentic browsing scoringdeveloper.chrome.com
- OpenAI: Overview of OpenAI crawlersdevelopers.openai.com
- Anthropic (Claude Help Center): Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
- Perplexity: Perplexity crawlersdocs.perplexity.ai
- IETF RFC 9309: Robots Exclusion Protocolrfc-editor.org
- sitemaps.org: Sitemaps XML formatsitemaps.org