--- title: "llms.txt explained: format, example and who reads it — Dardo" description: "What llms.txt is, the exact format, a template to copy and who reads it: Google Search ignores it, AI crawler docs don't mention it. 15 real files compared." url: "https://dardo.studio/en/blog/llms-txt/" language: "en" translations: {"es":"https://dardo.studio/es/blog/que-es-llms-txt/","pt":"https://dardo.studio/pt/blog/o-que-e-llms-txt/"} updated: "2026-10-17T23:44:00.677Z" --- [Blog](https://dardo.studio/en/blog/) # llms.txt: what it is, who actually reads it, an example to copy and how to validate it llms.txt is a Markdown guide to your site for AI agents. Google Search ignores it; coding agents use it. The format, a template and how to validate it. By [Nicolás Cerón](https://dardo.studio/en/nicolas-ceron/) ·October 17, 2026 ![A night-time glasshouse with branching paths between palms, and a crimson folded map lying on a lantern-lit lectern at the entrance.](https://dardo.studio/_astro/01M562SXC97Z6PZWG30QJ48P8N_1SLFmW.webp) ## Key takeaways - llms.txt is a Markdown file at /llms.txt with an H1 name, a blockquote summary and H2 lists of links, proposed by Jeremy Howard on 3 September 2024 and revised as version 2 in August 2026. - Google's AI optimization guide says Google Search ignores llms.txt, so publishing one neither helps nor harms rankings, AI Overviews or AI Mode. - The crawler documentation of OpenAI, Anthropic, Perplexity, Apple, Amazon, Mistral and DuckDuckGo, read on 17 October 2026, doesn't say their crawlers read other sites' llms.txt. - The documented use is agents reading at answer time, mostly coding assistants following documentation; of 15 files we compared, the documentation sites linked Markdown copies and the marketing sites mostly linked HTML. - Worth an hour for developer docs and product sites whose facts agents get wrong; not a substitute for crawlable pages, a sitemap or robots.txt. ## The short answer llms.txt is a Markdown file at `/llms.txt` that gives AI agents a short, curated guide to your site: your name as an H1, a one-paragraph summary in a blockquote, and H2 sections that list the pages worth reading. Jeremy Howard proposed it in September 2024 and published version 2 in August 2026. Google says Google Search ignores it, and none of the crawler documentation we read on 17 October 2026 (OpenAI, Anthropic, Perplexity, Apple, Amazon, Mistral, DuckDuckGo) says those crawlers read it. Its real readers are agents working at answer time, mostly coding assistants reading documentation. Publish one if agents need accurate facts about your product and it costs you an hour, or if your platform writes it for you. Don't expect rankings or citations from it. ## What llms.txt is, and what changed in version 2 Jeremy Howard published the [llms.txt proposal](https://llmstxt.org/) on 3 September 2024. The reasoning: web pages wrap their content in navigation, scripts and banners, and a whole site doesn't fit in a model's context window, so a site can hand agents a short index of what matters, with links to clean Markdown versions of its pages. The proposal expected the file to be used at inference time, when an agent looks something up for a person, more than for training. Version 2, dated August 2026 on the proposal's [changes page](https://llmstxt.org/changes.html), kept the format and changed four things: - **Any path, not only the root.** A file at `/docs/llms.txt` covers the pages under `/docs/`, and agents should use the most specific file that applies. - **Discovery through link relations.** A page can point to its Markdown copy with `rel="alternate" type="text/markdown"` and to the llms.txt that covers it with `rel="describedby"`, either as HTML `` elements or as an HTTP `Link` header. - **Two Markdown URL forms.** Append `.md` (`page.html.md`) or replace the extension (`page.md`). Addresses that end in a slash use `index.md`. - **"Optional" lost its special meaning.** It is still the usual name for links an agent can skip, but no tool is expected to drop that section automatically any more. It remains an informal proposal maintained in a [GitHub repository](https://github.com/AnswerDotAI/llms-txt), not an IETF or W3C standard. ### llms-full.txt is not part of it `llms-full.txt`, one file with a site's entire text, is a convention that documentation platforms adopted; the proposal doesn't define it. [Mintlify](https://www.mintlify.com/docs/ai/llmstxt), for example, serves both files for the docs sites it hosts. Size is the catch: the `llms-full.txt` of Cloudflare's developer documentation was about 69 MB uncompressed when we downloaded it on 17 October 2026, far more than any model reads at once. Ours is 0.8 MB. For a marketing site, a good llms.txt plus Markdown copies of the key pages does the same job. ## The format, part by part The proposal fixes the order of the parts so that a script, not only a model, can parse the file: | Part | Required? | What goes in it | Mistake we found | | ------------------- | --------------------------- | ----------------------------------------------------------------- | --------------------------------------------- | | H1 (\# Name) | Yes, the only required part | The site or project name | Using # as a comment marker, as in robots.txt | | Blockquote (\> …) | Recommended | One or two sentences an agent needs to understand the rest | Left out, or a slogan instead of facts | | Free text | Optional | Paragraphs or lists without headings: key facts, notes for agents | A heading here, which starts a new section | | H2 sections (\## …) | Optional, any number | Link lists: \- \[name\](url): notes | Bare URLs, or prose instead of a list | | \## Optional | Optional | Secondary links an agent can skip | The most important pages filed here | A file may start with a byte-order mark. Everything after the first H2 should be link lists; one stray `#` line turns into a second H1 and breaks the structure for any parser. ## An llms.txt example you can copy Replace the bracketed parts. Write the summary with the facts agents most often get wrong about you: what you sell, to whom, where, how you price, and who you are not. ```markdown # [Company name] > [Company] makes [product] for [customer]. Based in [city, country]; serves [markets]. Pricing: [starting price or how you quote]. Contact: [email or contact page URL]. Key facts: - Founded [year]; team of [number]. - Works in [languages]. - What we don't do: [the requests you turn down]. - Not affiliated with [similarly named businesses]. ## Product - [Product overview](https://example.com/product/index.md): what it does and who it is for - [Pricing](https://example.com/pricing/index.md): plans, limits and what each includes - [Security](https://example.com/security/index.md): certifications, data location, subprocessors ## Docs - [Quickstart](https://example.com/docs/quickstart/index.md): set up and first result - [API reference](https://example.com/docs/api/index.md): endpoints, authentication, rate limits ## Company - [About](https://example.com/about/index.md): team, location, how we work - [Contact](https://example.com/contact/index.md): sales, support and press ## Optional - [Changelog](https://example.com/changelog/index.md): release notes - [Blog](https://example.com/blog/index.md): guides and announcements ``` A few rules of thumb: - **Link Markdown copies if you have them.** Version 2 asks that links point to content an agent can read cleanly. HTML pages work if that's all you have, and most marketing sites in our sample link HTML. - **Keep it short.** The proposal wants the file to fit in context, with the detail behind the links. Mintlify splits its generated indexes into sub-files once they pass 100,000 characters. - **Serve it as text.** `text/plain` or `text/markdown`, in UTF-8, so names with accents survive. ## Who actually reads llms.txt (checked 17 October 2026) | Who | What their own documentation says | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Google Search | Doesn't use it: keeping one neither helps nor harms visibility in Search, AI Overviews or AI Mode ([guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)) | | Chrome Lighthouse | Its experimental Agentic Browsing audits check for /llms.txt: a server error is flagged, a 404 counts as not applicable ([audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)) | | OpenAI crawlers | No mention of llms.txt ([crawler page](https://developers.openai.com/api/docs/bots)) | | Anthropic crawlers | No mention ([crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) | | Perplexity crawlers | No mention for crawling; the page points readers to Perplexity's own llms.txt ([crawler page](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) | | Wix, Yoast, Mintlify | Generate a file for their users ([Wix](https://support.wix.com/en/article/understanding-your-sites-llmstxt-file), [Yoast](https://yoast.com/features/llms-txt/)) | ### Google: ignored by Search, checked by Lighthouse Google's guide to its generative AI features, last updated 10 July 2026, lists llms.txt among the things you can ignore for Google Search. You don't need AI text files to appear in Search or its AI features, and keeping one for other services neither helps nor harms your rankings, because Search ignores it. That settles the "llms.txt Google" question for SEO. Two other parts of Google treat the file as real. Chrome's Lighthouse checks for it in its Agentic Browsing category, which is experimental, needs Chrome 150 or later and gives a pass ratio instead of a score ([scoring](https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring)). And Google's Gemini API documentation publishes one at [ai.google.dev](https://ai.google.dev/gemini-api/docs/llms.txt). Neither is a Search signal. ### AI companies publish one; their crawler pages don't mention it OpenAI, Anthropic and Perplexity each publish an llms.txt for their developer documentation: [developers.openai.com](https://developers.openai.com/llms.txt), [platform.claude.com](https://platform.claude.com/llms.txt) and [docs.perplexity.ai](https://docs.perplexity.ai/llms.txt). Their crawler pages, which describe GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers, say nothing about reading other sites' files. We found no first-party statement that any assistant's search index uses llms.txt. So when ChatGPT, Claude or Perplexity cites one of your pages, a search crawler indexed that page; the llms.txt played no documented part. What does read the file is an agent someone points at your site: a coding assistant told to read your API docs, or a browser agent asked to compare your plans. The proposal says software documentation is where it's used most, and the biggest files in our sample are documentation. ## Who publishes one: 15 files we read on 17 October 2026 We downloaded each file and counted its parts. "Links to Markdown" counts links to `.md` pages or to other llms.txt files. | Site | Size | H2 sections | Links | Links to Markdown | Summary blockquote | | ------------------------------------ | ----- | ----------- | ----- | ----------------- | ------------------------- | | platform.claude.com (Anthropic docs) | 86 KB | 4 | 784 | 784 | No | | docs.stripe.com | 98 KB | 27 | 527 | 516 | No | | stripe.com | 70 KB | 31 | 308 | 25 | Yes | | ai.google.dev (Gemini API docs) | 38 KB | 1 | 222 | 222 | No | | docs.perplexity.ai | 38 KB | 2 | 165 | 163 | Yes | | developers.cloudflare.com | 17 KB | 9 | 112 | 112 | Yes, after a line of text | | www.cloudflare.com | 17 KB | 13 | 90 | 83 | Yes | | developers.openai.com | 6 KB | 4 | 40 | 31 | Yes | | vercel.com | 5 KB | 9 | 27 | 22 | Yes | | www.wix.com | 37 KB | 22 | 232 | 0 | Yes | | www.hubspot.com | 36 KB | 29 | 225 | 0 | Yes | | webflow.com | 13 KB | 8 | 76 | 1 | Yes | | www.shopify.com | 13 KB | 12 | 74 | 1 | Yes | | www.framer.com | 10 KB | 9 | 60 | 0 | Yes | | dardo.studio | 29 KB | 13 | 103 | 3 | Yes | What the sample shows: - **Documentation links Markdown; marketing sites link HTML.** Every link in Anthropic's and Google's Gemini files points to a Markdown copy. Cloudflare's developer file links to 112 product-level llms.txt files, the nested pattern version 2 describes. HubSpot, Wix and Framer link only HTML pages. - **The blockquote is the most skipped part.** Anthropic's, Stripe's docs and the Gemini API files have none. Stripe's docs file uses its free-text section for instructions to coding agents, such as checking the npm registry for the current package version instead of trusting memory. That was the most practical use of the file we saw. - **Plenty of large sites have none.** anthropic.com, google.com, developers.google.com, apple.com, nytimes.com and figma.com answered 404. openai.com answered our script with a 403, so we can't tell. - **Some files ignore the format.** rdstation.com's file uses `#` as a comment marker, which Markdown reads as 59 separate H1 headings and no links. hotmart.com's starts with an H2, so it has no H1 at all. ## Our own file: dardo.studio/llms.txt Our file at `https://dardo.studio/llms.txt` is generated at every deploy from the same CMS data as the pages, so it can't drift from them. On 17 October 2026 it was 28.6 KB: 161 lines, 13 H2 sections and 103 links, and it grows with every article we publish. Spanish and Portuguese versions sit at `/es/llms.txt` and `/pt/llms.txt`, next to an `llms-full.txt`. The blockquote says what we do, where, and how to reach us. A list of studio facts follows (Bogotá, founded 2026, a team of three, our published starting prices), including a line naming similarly named businesses that aren't us. Then come sections for each service group, our work, the free tools and the blog. The design choices behind it are in [what an agent-ready website needs](https://dardo.studio/en/blog/agent-ready-website/). It is served with a noindex header: ```txt $ curl -sI https://dardo.studio/llms.txt HTTP/2 200 content-type: text/plain; charset=utf-8 x-robots-tag: noindex ``` Google's guide notes that it may crawl and index many kinds of files besides HTML. A plain-text index ranking in place of the page it describes helps nobody, so `noindex` keeps it out of search results while any agent can still fetch it. Don't disallow the file in robots.txt for the same purpose: Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) explains that a crawler blocked by robots.txt never sees the noindex rule. Since 17 October 2026 every page points to the file with `rel="describedby"`, next to the `rel="alternate" type="text/markdown"` link to its own Markdown copy at `index.md`. One version 2 suggestion we skip on purpose: our llms.txt links go to the HTML pages, not to their Markdown copies, because an assistant cites the address it read, and we'd rather people land on the page than on raw Markdown. ## How to validate an llms.txt 1. **It answers 200, as text.** Run `curl -sI https://example.com/llms.txt` and look for status 200 and `text/plain` or `text/markdown` with UTF-8. The most common failure is a single-page app that answers every unknown path with its home page and a 200, so `/llms.txt` exists but is HTML. 2. **The structure parses.** Exactly one H1, and it comes first; a blockquote; no headings before the first H2; H2 sections whose items read `- [name](url)` with optional `: notes`. 3. **The links work.** Each one answers 200, ideally with Markdown. 4. **The facts are current.** Prices, plans and locations match your pages today. An agent will repeat what the file says. 5. **An agent understands it.** The proposal suggests giving an assistant only your llms.txt and asking it questions. Ask what you sell, what it costs and how to reach you, and compare the answers with your pages. Our free [AI visibility checker and llms.txt validator](https://dardo.studio/en/tools/ai-visibility-checker/) does the first three steps in one pass. It parses the H1, summary, sections and list items, flags HTML served as llms.txt and unexpected content types, tests up to 15 of the linked URLs, and drafts a starting file from your sitemap if you don't have one. It also reads your robots.txt for 26 AI crawlers; what those rules should say is in our guide to [blocking or allowing AI crawlers](https://dardo.studio/en/blog/block-ai-crawlers-robots-txt/). ## llms.txt generators - **WordPress.** Yoast SEO generates the file for free, and [AIOSEO](https://aioseo.com/features/llms-txt/) has a generator. - **Wix.** Every site gets one automatically. Once you edit it, Wix stops updating it until you reset it to the default. - **Documentation platforms.** Mintlify generates llms.txt and llms-full.txt, and also serves the file at `/.well-known/llms.txt`. The proposal lists [GitBook](https://www.gitbook.com/blog/what-is-llms-txt) and Docusaurus and VitePress plugins too. - **Custom sites.** Generate it at build time from the same content as your pages, as we do. A hand-written file is fine for a small site if someone owns updating it. A generator picks pages; you still write the summary. An automatic list of your latest blog posts is not a guide to your business. ## When it's worth the hour, and when it isn't | Your situation | Verdict | Why | | ------------------------------------------ | ---------------------------- | ------------------------------------------------- | | API or developer documentation | Yes, with Markdown copies | Coding agents are the documented heavy users | | SaaS or product site with plans and limits | Yes, kept short | Put the facts agents misstate in the summary | | Your platform generates it | Keep it, rewrite the summary | It costs nothing; check what it lists | | A ten-page brochure site | Optional | Titles, crawlable text and robots.txt matter more | | You want Google rankings or AI Overviews | No | Google says Search ignores it | | Your content only renders with JavaScript | Fix that first | llms.txt doesn't make pages crawlable | llms.txt doesn't replace crawlable pages, a sitemap or robots.txt. The citations in ChatGPT, Claude, Perplexity and Google's AI features come from pages their search crawlers index, which is the work described in [how to show up in AI answers](https://dardo.studio/en/blog/appear-in-ai-answers/). ## What to do next Run the [AI visibility checker](https://dardo.studio/en/tools/ai-visibility-checker/) on your domain: it validates your llms.txt, or drafts one, and shows which AI crawlers your robots.txt lets in. If you'd rather have the whole picture audited and fixed, that is our [AI search optimization](https://dardo.studio/en/services/ai-search-optimization/) work; audits start at USD 690 (see [pricing](https://dardo.studio/en/pricing/)). ## Common questions **Is llms.txt an official standard?** No. It is an informal proposal by Jeremy Howard, first published on 3 September 2024 and revised as version 2 in August 2026, maintained in a public GitHub repository. No standards body such as the IETF or W3C has adopted it. **Does Google use llms.txt?** Not for Search. Google's guide to its generative AI features says Google Search ignores llms.txt, so having one neither helps nor harms visibility in Search, AI Overviews or AI Mode. Chrome's Lighthouse does check for the file in an experimental agentic browsing audit, which is not a ranking signal. **Do ChatGPT, Claude or Perplexity read llms.txt?** Their crawler documentation doesn't say so. OpenAI, Anthropic and Perplexity publish llms.txt files for their own developer docs, but their pages about GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot don't mention reading other sites' files. Citations in their answers come from pages their search crawlers index. **What is the difference between llms.txt and robots.txt?** robots.txt tells crawlers which paths they may fetch. llms.txt doesn't grant or deny anything: it is a Markdown guide that tells an agent what your site is and which pages to read. Allow and disallow rules written into llms.txt have no effect. **Do I need llms-full.txt as well?** Usually not. llms-full.txt, a single file with a site's full text, is a convention some documentation platforms use; the llms.txt proposal doesn't define it. For a marketing site, a short llms.txt with links to Markdown copies of key pages does the same job. **Should llms.txt be noindexed?** It can be. Google may index text files, and a noindex header keeps the file out of search results while agents can still fetch it. Don't block it in robots.txt instead, because a crawler that can't fetch the file never sees the noindex rule. **Where do I put llms.txt?** At the root of your site, at /llms.txt. Version 2 of the proposal also allows files at subpaths such as /docs/llms.txt, each covering the pages under its path, with the most specific file applying. ## Sources & further reading Sources behind this page, with further detail from the original publishers. - [llmstxt.org: The /llms.txt file, v2](https://llmstxt.org/) · llmstxt.org - [llmstxt.org: Changes (v2, August 2026)](https://llmstxt.org/changes.html) · llmstxt.org - [Answer.AI on GitHub: llms-txt repository](https://github.com/AnswerDotAI/llms-txt) · github.com - [Google Search Central: Optimizing for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) · developers.google.com - [Google Search Central: Block Search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) · developers.google.com - [Chrome for Developers: Lighthouse llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) · developer.chrome.com - [Chrome for Developers: Lighthouse agentic browsing scoring](https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring) · developer.chrome.com - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) · developers.openai.com - [Anthropic (Claude Help Center): Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) · support.claude.com - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) · docs.perplexity.ai - [Mintlify: llms.txt](https://www.mintlify.com/docs/ai/llmstxt) · mintlify.com - [Wix Help Center: Understanding your site's llms.txt file](https://support.wix.com/en/article/understanding-your-sites-llmstxt-file) · support.wix.com - [Yoast: llms.txt feature](https://yoast.com/features/llms-txt/) · yoast.com - [AIOSEO: llms.txt generator](https://aioseo.com/features/llms-txt/) · aioseo.com - [GitBook: What is llms.txt](https://www.gitbook.com/blog/what-is-llms-txt) · gitbook.com - [Anthropic: developer documentation llms.txt](https://platform.claude.com/llms.txt) · platform.claude.com - [OpenAI: developer documentation llms.txt](https://developers.openai.com/llms.txt) · developers.openai.com - [Google: Gemini API documentation llms.txt](https://ai.google.dev/gemini-api/docs/llms.txt) · ai.google.dev - [Perplexity: documentation llms.txt](https://docs.perplexity.ai/llms.txt) · docs.perplexity.ai [Nicolás Cerón](https://dardo.studio/en/nicolas-ceron/) Nicolás Cerón is the founder of Dardo, a brand, web design and development studio in Bogotá, Colombia. ## Related articles - [**What an agent-ready website needs: notes from building dardo.studio**We made dardo.studio readable and callable by AI agents, one layer at a time. Here is what each layer does, who documents reading it today, and which ones we would build again.](https://dardo.studio/en/blog/agent-ready-website/) - [**How to show up in AI answers in 2026: what ChatGPT, Perplexity, Google and Copilot actually document**There is no switch for appearing in ChatGPT, Perplexity, Google's AI Overviews or Copilot. Here is what each platform documents about being cited, and how to measure it without guessing.](https://dardo.studio/en/blog/appear-in-ai-answers/) - [**How to block AI crawlers in robots.txt: every user agent, what blocking costs and three files to copy**Block AI training crawlers like GPTBot without leaving ChatGPT or Google search: 28 user agents from vendor docs, RFC 9309 rules and three robots.txt files.](https://dardo.studio/en/blog/block-ai-crawlers-robots-txt/) - [**How much does SEO cost? US agency prices, audits, hourly rates and what a small business pays per month**US agencies that publish SEO prices start at $299 to $5,000 a month; Clutch puts US firms at $100 to $149 an hour. What each price buys, and what to avoid.](https://dardo.studio/en/blog/seo-cost/) ## Keep exploring. - [Service · **AI search optimization (GEO): get cited by ChatGPT and Google AI**](https://dardo.studio/en/services/ai-search-optimization/) - [Your project · **Start a conversation**](https://dardo.studio/en/contact/)