llms.txt: what it is, who actually reads it, an example to copy and how to validate it

The short answer

llms.txt is a Markdown file at /llms.txt that gives AI agents a short, curated guide to your site: your name as an H1, a one-paragraph summary in a blockquote, and H2 sections that list the pages worth reading. Jeremy Howard proposed it in September 2024 and published version 2 in August 2026.

Google says Google Search ignores it, and none of the crawler documentation we read on 17 October 2026 (OpenAI, Anthropic, Perplexity, Apple, Amazon, Mistral, DuckDuckGo) says those crawlers read it. Its real readers are agents working at answer time, mostly coding assistants reading documentation. Publish one if agents need accurate facts about your product and it costs you an hour, or if your platform writes it for you. Don't expect rankings or citations from it.

What llms.txt is, and what changed in version 2

Jeremy Howard published the llms.txt proposal on 3 September 2024. The reasoning: web pages wrap their content in navigation, scripts and banners, and a whole site doesn't fit in a model's context window, so a site can hand agents a short index of what matters, with links to clean Markdown versions of its pages. The proposal expected the file to be used at inference time, when an agent looks something up for a person, more than for training.

Version 2, dated August 2026 on the proposal's changes page, kept the format and changed four things:

• Any path, not only the root. A file at /docs/llms.txt covers the pages under /docs/, and agents should use the most specific file that applies.
• Discovery through link relations. A page can point to its Markdown copy with rel="alternate" type="text/markdown" and to the llms.txt that covers it with rel="describedby", either as HTML <link> elements or as an HTTP Link header.
• Two Markdown URL forms. Append .md (page.html.md) or replace the extension (page.md). Addresses that end in a slash use index.md.
• "Optional" lost its special meaning. It is still the usual name for links an agent can skip, but no tool is expected to drop that section automatically any more.

It remains an informal proposal maintained in a GitHub repository, not an IETF or W3C standard.

llms-full.txt is not part of it

llms-full.txt, one file with a site's entire text, is a convention that documentation platforms adopted; the proposal doesn't define it. Mintlify, for example, serves both files for the docs sites it hosts. Size is the catch: the llms-full.txt of Cloudflare's developer documentation was about 69 MB uncompressed when we downloaded it on 17 October 2026, far more than any model reads at once. Ours is 0.8 MB. For a marketing site, a good llms.txt plus Markdown copies of the key pages does the same job.

The format, part by part

The proposal fixes the order of the parts so that a script, not only a model, can parse the file:

A file may start with a byte-order mark. Everything after the first H2 should be link lists; one stray # line turns into a second H1 and breaks the structure for any parser.

An llms.txt example you can copy

Replace the bracketed parts. Write the summary with the facts agents most often get wrong about you: what you sell, to whom, where, how you price, and who you are not.

# [Company name]

> [Company] makes [product] for [customer]. Based in [city, country]; serves [markets]. Pricing: [starting price or how you quote]. Contact: [email or contact page URL].

Key facts:

- Founded [year]; team of [number].
- Works in [languages].
- What we don't do: [the requests you turn down].
- Not affiliated with [similarly named businesses].

## Product

- [Product overview](https://example.com/product/index.md): what it does and who it is for
- [Pricing](https://example.com/pricing/index.md): plans, limits and what each includes
- [Security](https://example.com/security/index.md): certifications, data location, subprocessors

## Docs

- [Quickstart](https://example.com/docs/quickstart/index.md): set up and first result
- [API reference](https://example.com/docs/api/index.md): endpoints, authentication, rate limits

## Company

- [About](https://example.com/about/index.md): team, location, how we work
- [Contact](https://example.com/contact/index.md): sales, support and press

## Optional

- [Changelog](https://example.com/changelog/index.md): release notes
- [Blog](https://example.com/blog/index.md): guides and announcements

A few rules of thumb:

• Link Markdown copies if you have them. Version 2 asks that links point to content an agent can read cleanly. HTML pages work if that's all you have, and most marketing sites in our sample link HTML.
• Keep it short. The proposal wants the file to fit in context, with the detail behind the links. Mintlify splits its generated indexes into sub-files once they pass 100,000 characters.
• Serve it as text. text/plain or text/markdown, in UTF-8, so names with accents survive.

Who actually reads llms.txt (checked 17 October 2026)

Google: ignored by Search, checked by Lighthouse

Google's guide to its generative AI features, last updated 10 July 2026, lists llms.txt among the things you can ignore for Google Search. You don't need AI text files to appear in Search or its AI features, and keeping one for other services neither helps nor harms your rankings, because Search ignores it. That settles the "llms.txt Google" question for SEO.

Two other parts of Google treat the file as real. Chrome's Lighthouse checks for it in its Agentic Browsing category, which is experimental, needs Chrome 150 or later and gives a pass ratio instead of a score (scoring). And Google's Gemini API documentation publishes one at ai.google.dev. Neither is a Search signal.

AI companies publish one; their crawler pages don't mention it

OpenAI, Anthropic and Perplexity each publish an llms.txt for their developer documentation: developers.openai.com, platform.claude.com and docs.perplexity.ai. Their crawler pages, which describe GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers, say nothing about reading other sites' files. We found no first-party statement that any assistant's search index uses llms.txt.

So when ChatGPT, Claude or Perplexity cites one of your pages, a search crawler indexed that page; the llms.txt played no documented part. What does read the file is an agent someone points at your site: a coding assistant told to read your API docs, or a browser agent asked to compare your plans. The proposal says software documentation is where it's used most, and the biggest files in our sample are documentation.

Who publishes one: 15 files we read on 17 October 2026

We downloaded each file and counted its parts. "Links to Markdown" counts links to .md pages or to other llms.txt files.

What the sample shows:

• Documentation links Markdown; marketing sites link HTML. Every link in Anthropic's and Google's Gemini files points to a Markdown copy. Cloudflare's developer file links to 112 product-level llms.txt files, the nested pattern version 2 describes. HubSpot, Wix and Framer link only HTML pages.
• The blockquote is the most skipped part. Anthropic's, Stripe's docs and the Gemini API files have none. Stripe's docs file uses its free-text section for instructions to coding agents, such as checking the npm registry for the current package version instead of trusting memory. That was the most practical use of the file we saw.
• Plenty of large sites have none. anthropic.com, google.com, developers.google.com, apple.com, nytimes.com and figma.com answered 404. openai.com answered our script with a 403, so we can't tell.
• Some files ignore the format. rdstation.com's file uses # as a comment marker, which Markdown reads as 59 separate H1 headings and no links. hotmart.com's starts with an H2, so it has no H1 at all.

Our own file: dardo.studio/llms.txt

Our file at https://dardo.studio/llms.txt is generated at every deploy from the same CMS data as the pages, so it can't drift from them. On 17 October 2026 it was 28.6 KB: 161 lines, 13 H2 sections and 103 links, and it grows with every article we publish. Spanish and Portuguese versions sit at /es/llms.txt and /pt/llms.txt, next to an llms-full.txt.

The blockquote says what we do, where, and how to reach us. A list of studio facts follows (Bogotá, founded 2026, a team of three, our published starting prices), including a line naming similarly named businesses that aren't us. Then come sections for each service group, our work, the free tools and the blog. The design choices behind it are in what an agent-ready website needs.

It is served with a noindex header:

$ curl -sI https://dardo.studio/llms.txt
HTTP/2 200
content-type: text/plain; charset=utf-8
x-robots-tag: noindex

Google's guide notes that it may crawl and index many kinds of files besides HTML. A plain-text index ranking in place of the page it describes helps nobody, so noindex keeps it out of search results while any agent can still fetch it. Don't disallow the file in robots.txt for the same purpose: Google's noindex documentation explains that a crawler blocked by robots.txt never sees the noindex rule.

Since 17 October 2026 every page points to the file with rel="describedby", next to the rel="alternate" type="text/markdown" link to its own Markdown copy at index.md. One version 2 suggestion we skip on purpose: our llms.txt links go to the HTML pages, not to their Markdown copies, because an assistant cites the address it read, and we'd rather people land on the page than on raw Markdown.

How to validate an llms.txt

1. It answers 200, as text. Run curl -sI https://example.com/llms.txt and look for status 200 and text/plain or text/markdown with UTF-8. The most common failure is a single-page app that answers every unknown path with its home page and a 200, so /llms.txt exists but is HTML.
2. The structure parses. Exactly one H1, and it comes first; a blockquote; no headings before the first H2; H2 sections whose items read - [name](url) with optional : notes.
3. The links work. Each one answers 200, ideally with Markdown.
4. The facts are current. Prices, plans and locations match your pages today. An agent will repeat what the file says.
5. An agent understands it. The proposal suggests giving an assistant only your llms.txt and asking it questions. Ask what you sell, what it costs and how to reach you, and compare the answers with your pages.

Our free AI visibility checker and llms.txt validator does the first three steps in one pass. It parses the H1, summary, sections and list items, flags HTML served as llms.txt and unexpected content types, tests up to 15 of the linked URLs, and drafts a starting file from your sitemap if you don't have one. It also reads your robots.txt for 26 AI crawlers; what those rules should say is in our guide to blocking or allowing AI crawlers.

llms.txt generators

• WordPress. Yoast SEO generates the file for free, and AIOSEO has a generator.
• Wix. Every site gets one automatically. Once you edit it, Wix stops updating it until you reset it to the default.
• Documentation platforms. Mintlify generates llms.txt and llms-full.txt, and also serves the file at /.well-known/llms.txt. The proposal lists GitBook and Docusaurus and VitePress plugins too.
• Custom sites. Generate it at build time from the same content as your pages, as we do. A hand-written file is fine for a small site if someone owns updating it.

A generator picks pages; you still write the summary. An automatic list of your latest blog posts is not a guide to your business.

When it's worth the hour, and when it isn't

llms.txt doesn't replace crawlable pages, a sitemap or robots.txt. The citations in ChatGPT, Claude, Perplexity and Google's AI features come from pages their search crawlers index, which is the work described in how to show up in AI answers.

What to do next

Run the AI visibility checker on your domain: it validates your llms.txt, or drafts one, and shows which AI crawlers your robots.txt lets in. If you'd rather have the whole picture audited and fixed, that is our AI search optimization work; audits start at USD 690 (see pricing).