llms.txt
llms.txt is a Markdown file placed at the root of a website that states what the site is and indexes its main pages, so that an AI agent can navigate it without having to parse the whole site.
It is a proposed convention, not a ratified standard, and that distinction matters when deciding how much to invest in it. Writing one costs an hour. Treating it as a guarantee that models will read you would be a mistake.
What is llms.txt for?
An agent arriving at a website has to work out what the site is and which pages matter, from HTML built for human eyes. llms.txt answers both questions directly, in Markdown, in one request: a title, a one-paragraph summary and a curated list of links with a line explaining each.
The value is curation rather than completeness. A sitemap lists every URL and treats them as equals; llms.txt says which handful you would want quoted, which is a different and more useful piece of information.
How is it different from robots.txt and sitemap.xml?
Three files at the root of a site, three different jobs. They are often confused because they share a location, and having one does not replace the others.
- robots.txt says what a crawler may and may not fetch. It is permission, settled by RFC 9309.
- sitemap.xml lists every URL that exists, with dates. It is inventory.
- llms.txt says what the site is and which pages are worth reading. It is orientation, and it is the only one written for a reader rather than a machine parser.
Does llms.txt actually work?
Honest answer: no major engine has publicly committed to reading it, and there is no public measurement showing it changes how often a site gets cited. What is documented is the convention itself and a growing number of sites that publish one.
It is scored in the Citability Index for a narrower reason: a site that publishes a current, accurate llms.txt has almost always thought about the rest. Its weight may change, and the methodology says so in writing, because a convention that does not consolidate should not keep its points.
For contrast, the one finding this field has actually measured is the GEO study's: citing sources and supplying data raised visibility in answers by up to 40%. There is nothing comparable behind llms.txt, and saying so is more useful than pretending otherwise.
Where does this come from?
One proposal and two published standards. Reading the three together is what makes the difference between the files obvious, which is where most of the confusion lives.
- llmstxt.org, the proposal
The original specification: what the file contains, in what order and with what Markdown structure. It is a de facto convention rather than a ratified standard, and it says so itself.
- RFC 9309, the Robots Exclusion Protocol
The settled standard for robots.txt, and the contrast that explains what llms.txt is not: one grants permission, the other offers orientation.
- OpenAI crawler documentation
What an engine does publish about how it reads sites. Useful for seeing what is actually committed to and what is still convention.
Frequently asked questions about llms.txt
Is llms.txt an official standard?
- No. It is a proposed convention published at llmstxt.org and adopted voluntarily. No search or answer engine has publicly committed to reading it, which is why any tool that promises results from publishing one is overselling it.
Do models actually read llms.txt?
- There is no public evidence that the major engines fetch it as a matter of course. Publishing one is cheap and does no harm, and it forces a useful exercise: deciding in one page what your site is and which parts you would want quoted.