llms.txt: What It Is, Who Reads It, and Why Ship One Anyway

Search “llms.txt” and you find two camps, both wrong in opposite directions. One sells it as the mandatory new ticket to AI visibility — generate yours today or vanish from ChatGPT. The other dismisses it as dead on arrival because Google said it does not read the file. Neither camp states the actual position, which is more interesting than either pitch: llms.txt is a sensible, well-designed proposal that, as of now, no major answer engine has confirmed consuming — and it is still worth shipping, because it costs about an hour and the downside is zero.

This guide is the honest version: what the file actually is, what it is not, what the evidence says about who reads it, and why the low-cost case survives the hype collapsing. It sits under our AI search pillar, next to the disciplines that demonstrably do move AI visibility — answer engine optimization chief among them.

What llms.txt actually is

llms.txt is a proposed standard: a Markdown file served at the root of your site — /llms.txt — that gives language models a curated, noise-free map of your most important content. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and the spec lives at llmstxt.org. The reasoning is sound. An LLM reading your site at inference time fights two constraints: a context window too small to ingest everything, and HTML that buries the substance under navigation, cookie banners, and script. A hand-curated Markdown index solves both — here are our essential pages, one line each on why they matter, in a format a model parses natively.

The format is deliberately simple: an H1 with the site name, a blockquote summary of what the site is, then sections of links with short descriptions. The spec adds two optional companions — /llms-full.txt, which inlines the full content of key pages into one large file, and Markdown mirrors of individual pages (append .md to a URL, get the clean-text version). Documentation platforms took to it first: Mintlify ships llms.txt for the docs sites it hosts, and developer-facing companies — Anthropic included — publish their own. That adoption is real. What it signifies is where the honesty has to start.

What llms.txt is not

Most confusion around the file comes from mapping it onto standards it superficially resembles. It is not robots.txt — it grants and restricts nothing, and a crawler that wants to ignore it simply does. It is not sitemap.xml — a sitemap enumerates every URL for indexing machinery; llms.txt curates a shortlist for a reader with a small attention budget. And it is not a ranking signal of any kind: no search engine has ever said the file’s presence affects how a site ranks or how often it is cited.

That third row is the entire debate. The file is well-designed and cheap to produce — but a standard is only a standard once the other side of the protocol shows up. So far, the publishing side has shown up in force and the consuming side has not.

The evidence, honestly: who actually reads it

Here is the state of play, stated plainly. Google has said it does not use llms.txt — John Mueller publicly compared it to the keywords meta tag, a site’s self-description that no engine trusts enough to consume. OpenAI, Anthropic, and Perplexity publish crawler documentation covering robots.txt directives; none of them documents fetching or honoring llms.txt for their search or answer products. And practitioners who have checked server logs report the same pattern: the major AI crawlers request pages, sometimes aggressively, and largely do not request /llms.txt at all.

Adoption statistics muddy this, so separate the two sides of the market. Thousands of sites publishing llms.txt is a supply-side fact — it proves the file is easy to ship and the idea is attractive. It proves nothing about consumption. Anthropic publishing an llms.txt for its own docs does not mean Claude fetches yours; Mintlify generating the file for every docs site it hosts does not mean an answer engine reads any of them. The demand side — an engine saying “we fetch this file and it changes what we cite” — is the claim that matters, and it has not been made by anyone who runs an answer engine at scale.

Thousands of sites publish llms.txt. No major engine has confirmed reading one. A standard with only a supply side is a proposal — and it is fine to treat it as exactly that.

One live detail makes the point better than any argument. Run a SERP check on the query “llms.txt” itself and Google answers it with an AI Overview above the results — a query about a file for AI answers, resolved by an AI answer, sourced from the ordinary ranked web. The engines are already synthesizing answers about llms.txt using everything except llms.txt. That is the zero-click mechanic in miniature, and a reminder of where citation decisions actually come from today: the indexed, ranked, quotable page — not the manifest file beside it.

The low-cost case for shipping one anyway

Given all that, the honest recommendation is still: ship one. Not because the hype is secretly right, but because the bet is asymmetric in a way almost nothing in SEO is. The cost is roughly an hour and a small static file. The downside is zero — no engine penalizes the file, no user sees it, nothing breaks. The upside has three parts, in descending order of certainty.

First, the certain part: writing it forces content clarity. To produce a good llms.txt you must name your ten or twenty most important pages and describe each in one honest line. Most sites cannot do this without discovering something — pages that overlap, pages nobody can summarize because they lack a point, an architecture with no answer to “what should a stranger read first?” That editorial audit pays for the hour by itself, whether or not any machine ever fetches the result.

Second, the real-today part: agents at inference time. Crawlers ignoring the file does not mean nothing reads it. A developer pointing a coding agent at your docs, a researcher handing your site to an assistant, an agentic tool told to evaluate your product — these one-off, directed fetches are exactly the consumer the spec was designed for, and they exist now. For documentation-heavy sites this is a genuine, if narrow, audience.

Third, the speculative part: insurance. If any major engine ever announces support, the sites with the file already in place win a news cycle’s head start for free. Weight this at close to nothing — priced honestly, it is a lottery ticket that came free with work you already benefited from.

How to write a good llms.txt in about an hour

Skip the generators — the value is in the curation, and curation is the part a generator cannot do. The format is plain Markdown: an H1 with the site name, a one-or-two-sentence blockquote stating what the site is and who it is for, then a few H2 sections of links, each link followed by a one-line description. Three rules make the difference between a useful file and a token gesture.

Curate ruthlessly. Ten to twenty-five links, not your sitemap. The file exists because a model’s attention is scarce; a 400-line llms.txt has already failed its own premise. Describe honestly. Each line should say what a page contains, not sell it — “pricing and plan comparison” beats “discover our flexible solutions.” A model deciding what to fetch next needs information scent, not adjectives. Keep it true. A stale llms.txt that points at renamed pages is worse than none. Generate it from your route structure or content registry at build time if you can; on a Next.js site that is one small route handler reading the same source of truth as your sitemap.

Where the real AI-search work is

The danger with llms.txt is not shipping it — it is mistaking it for a strategy. If your goal is being cited by answer engines, the levers with actual evidence behind them are the unglamorous ones: answer-first page structure a model can lift verbatim, clean entity and schema signals, real authority, and pages that directly answer the question-shaped queries already visible in your Search Console. That is the discipline our AEO guide walks through step by step, and none of it is replaced by a manifest file. Engines cite pages they indexed, ranked, and could quote cleanly. A site with a perfect llms.txt and buried answers loses to a site with no llms.txt and a quotable first paragraph — every time, on every engine anyone has measured.

So sequence it honestly: llms.txt goes at the end of the AI-search checklist, not the start. It is the cheapest item on the list and also the least consequential — do it in the hour it deserves, then spend the rest of the quarter on the work that determines whether you are the source inside the answer.

Ship it, log it, expect nothing

One last discipline, because it is the house rule for everything we ship: treat llms.txt like any other change — as a tracked task with an honest expectation attached. In this case the honest expectation is no measurable movement. Do not let anyone — vendor, agency, or your own optimism — attribute a traffic or citation change to a file with no confirmed consumers. Log the task, note that the expected direct impact is nil and the real deliverable was the clarity audit, and check your server logs in a quarter to see whether anything fetched it. If the standard ever gains a confirmed consumer, that log entry is your before/after baseline. That is the closed-loop habit: every change carries its expected impact up front, so nobody gets to claim credit after the fact for movement they did not cause.

If you want to see where your site actually stands with AI search — which queries already surface you, where you are seen but never chosen, and which pages a model could quote today — start with a free SEO report on your real Search Console data. It will not mention llms.txt much, and now you know why: the file is worth an hour, and the answers are worth the quarter. If you would rather talk it through, book a call.