llms.txt explained.
llms.txt is a plain-text file at the root of a website that gives AI systems a curated map of the site: what it is, which pages matter and where the machine-readable versions live. It was proposed by Jeremy Howard of Answer.AI in September 2024 and has since been adopted by documentation sites, software companies and a growing number of marketing sites. Here is what it is, what it is not, and how to write one.
What llms.txt is
Web pages are built for browsers: navigation, scripts, adverts, cookie banners. A language model reading a page has to dig the substance out of all that. llms.txt sidesteps the problem by offering a short Markdown document at /llms.txt that says, in plain language, what the site is and links to the pages worth reading, ideally in clean Markdown versions. Some sites also publish /llms-full.txt, a single file containing the full text of the important pages.
What it is not
- It is not robots.txt. robots.txt tells crawlers what they may not fetch. llms.txt tells models what is worth reading. One is a fence, the other is a guide.
- It is not a sitemap. A sitemap lists every URL for indexing. llms.txt is curated and annotated, and short on purpose.
- It is not a ranking factor. No search engine has said it uses the file for ranking, and Google has said it does not. Its value is in how AI tools that fetch pages on demand read your site, and in the discipline of deciding what your site is actually about.
Who reads it
Adoption by the large engines is partial and changes month to month, so we do not claim any of them uses it in a specific way. What is observable: AI coding assistants and documentation tools use it heavily, agents that fetch pages at answer time can use it as an index, and it costs almost nothing to publish. Treat it as a low-cost bet with a real upside, not as a guarantee.
The format
The proposed structure is deliberately simple:
- An H1 with the site or organisation name.
- A blockquote with a one-paragraph summary of what the site is and who it is for.
- Optional paragraphs with any context a reader needs: what you sell, where you operate, what to trust the site for.
- H2 sections, each a list of links with a short description after each. Common sections are Services, Case studies, Pricing, Contact.
- An optional section headed Optional for pages that can be skipped if the reader is short on context.
Ours is at 121group.io/llms.txt if you want a working example.
What to put in yours
- The one-sentence description of the business you would give a new employee.
- Your service pages, each with a one-line description in the buyer's words.
- Your strongest proof: case studies with names and numbers, published pricing, credentials.
- Contact details and locations, stated plainly.
- Nothing you would not want quoted. The file is public.
Three mistakes we see
- Listing every URL. The file becomes a sitemap and loses its value as a guide. Fifteen to forty links is typical.
- Marketing copy in the descriptions. 'Award-winning, results-driven' tells a model nothing. 'Google Ads management for medical centres in Victoria' tells it everything.
- Blocking the crawlers anyway. An llms.txt file behind a robots.txt that disallows GPTBot and ClaudeBot is a menu for a locked restaurant. Check both.
Is it worth doing
Yes, at the cost involved: an hour to write, a minute to publish, and it forces a useful conversation about what the site is for. It belongs inside a broader GEO programme rather than standing alone; the file helps a model read the site, but schema, entities and evidence are what make it worth citing. We include it in every llms.txt implementation, along with the crawler-access check that usually turns out to matter more.
Want an llms.txt written for your site, with the crawler access and schema checked at the same time?
See the llms.txt service