What is llms.txt?

What it is for

The argument behind it is reasonable. A language model reading a website has a limited context window and encounters a lot of noise: navigation, cookie banners, boilerplate, duplicated template text. A curated file that says here is what this site is, here are the pages that matter, here is what each one covers, makes the site cheaper and more accurate to read.

It is deliberately similar in spirit to robots.txt, which tells crawlers what they may access, and to a sitemap, which tells them what exists. The difference is that llms.txt is meant to tell them what is worth reading and what it means.

The format is markdown rather than XML, on the reasoning that models read markdown well and humans can maintain it.

The part that gets left out of most articles about it

Adoption is the whole question, and it is usually skipped.

A proposal only works if the systems it addresses honour it. As of now, the major AI companies have not committed to reading llms.txt in the way search engines committed to robots.txt decades ago. A number of developer-tooling companies publish one. That is not the same as the engines consuming it.

So the honest position is this. Publishing llms.txt is cheap, harmless, and may help if adoption arrives. Treating it as a lever that will change how AI describes your company today is not supported by anything observable.

If you see a service charging meaningfully for llms.txt implementation, the work involved is writing a short markdown file. Price it accordingly.

What one looks like

A heading with the site name. A short blockquote summarising what the site is. Then sections of markdown links, each with a one-line description of what the page covers, grouped so the important material is first and optional material is under a section marked optional.

That is the whole format. It is intentionally simple.

What to do with it

1. Publish one if you have a content-heavy site. It takes an hour and it costs nothing to maintain if you keep it short.
2. Write the descriptions for a reader who knows nothing about you. The value is in the one-line summaries, not in the links.
3. Do not pay a meaningful sum for it, and do not let it displace work that is evidenced.
4. Keep it current. A stale guide is worse than none, because it misdescribes.
5. Check back on adoption periodically rather than assuming either way.

If you run the company rather than the marketing

This is a small item that occasionally appears in proposals with a price attached that does not match the work.

llms.txt is a text file. Writing a good one for a normal company website is an hour, maybe two if the descriptions are written carefully. It may turn out to matter and it may not, because the companies it is addressed to have not committed to reading it.

Publish one. Spend an hour on it. Then put your attention on the things that are evidenced: whether the engines describe you correctly, and whether anybody other than you has written about your company.

If a line item for AI file standards appears in a scope of work at a few thousand, ask what else is in it.