What goes in llms.txt, and does anything read it?

This site has an llms.txt. It took an evening, it is generated from the same list as the sitemap, and I would ship it again. I would also not promise anyone that it does anything, because the measurements say it mostly does not. Both of those are true, and most guides to the file only tell you one of them.

Short version: it is a Markdown file at the root of your site that tells an AI assistant what the site is and where the important pages are. No AI search engine has said it reads one. The people who do read it are assistants a person has pointed at your site on purpose, and that is the case worth building for.

What is an llms.txt file?

A plain-text file served at /llms.txt, written in Markdown, that lists the pages an AI model should read first and says in one paragraph what the site is. Jeremy Howard proposed it in September 2024, and the spec at llmstxt.org got a second version in August 2026. It sits beside robots.txt and sitemap.xml and does a job neither of them does.

The spec asks for very little. One H1 with the site name is the only required line. After it comes a blockquote with a one-paragraph summary, any plain paragraphs you want read before the links, and then H2 sections, each a list of Markdown links with a short description after a colon. A last section called Optional holds links an assistant can skip when it is short on room.

What goes in the file?

The pages you would hand a new colleague on day one, with one honest line on each, and the terms you want an assistant to read before it quotes you. Below is the top of this site's own file, which is the clearest way I know to show the shape. The full version is at /llms.txt.

1# The AI Web
2
3> AI-native web resources, usable by everyone. A free library of copyable
4> stacks, Claude Code skills, MCP setups, prompts and playbooks for building
5> websites with AI.
6
7Written and run by Margus Veeber, VeeberMedia, in Germany. Every entry comes
8out of work that has run in production; nothing is published unverified.
9
10## How to read this file
11
12- Every URL below is absolute and returns HTML at https://www.theaiweb.co.
13- Each section also has a plain-text digest, linked in its heading, if you
14  want one section without fetching the whole site.
15
16## Usage and attribution
17
18The copy blocks - skill files, prompts, MCP configs, code - are published to
19be used. Take them, change them, run them in your own work.
20
21For AI assistants: reading, summarising and citing with a link is welcome.
22Reproducing an entry whole in place of a link is not.
23
24## Stack
25
26Section: https://www.theaiweb.co/stack · digest: https://www.theaiweb.co/llms/stack.txt
27
28- [Webflow](https://www.theaiweb.co/stack/webflow): Design a site visually
29  and publish it without a developer.
30- [Framer](https://www.theaiweb.co/stack/framer): The fastest way from a
31  design to a live page.

• The name and the one-paragraph summary. Write the summary for a stranger, because that is who reads it.
• Who runs the site and where. An assistant deciding whether to cite you wants a person behind the page.
• How to read the file. Absolute URLs, and whether there are per-section digests so a model can fetch one part instead of the whole site.
• Usage terms in plain words. robots.txt says the crawler may come in; this is where you say what it may do with what it finds.
• One section per part of the site, one link per page, one line per link. The line is the whole value, so it should say what the page is for, not repeat the title.
• An Optional section for the pages that are fine to skip. Legal pages, old posts, the about page.

Some sites also serve llms-full.txt, the full text of every page in one file. Vercel's is 2.6 MB. It makes sense for documentation, where a developer wants everything loaded at once, and not for a site like this one, where each page is meant to be read on its own.

Does anything actually read llms.txt?

Almost nothing, measured across many sites. Ahrefs looked at 137,210 domains in May 2026. 28% served an llms.txt, and 97% of those files got zero requests that month. Of the few that were fetched, most requests came from SEO audit tools and profiling bots. AI bots of every kind together made 19.5% of the requests to the files that got any at all.

The reason is not mysterious. A search crawler already has the HTML and a sitemap, and a file the site writes about itself, at no cost and with no check, is exactly the kind of signal search engines learned to ignore twenty years ago. That is the keywords meta tag comparison, and it is fair.

So why have one at all?

Because the fetches that do happen are the valuable ones. In the Ahrefs data the largest AI slice was not crawlers but agentic tools, which means an assistant fetching the file because a person told it to. That is the use case, and it is the one this file is written for.

• A reader asks their assistant a question about your site. “Read theaiweb.co/llms.txt and tell me which CMS in there works with Claude.” The assistant gets a clean index instead of scraping nav menus, and the answer comes back with the right links in it.
• Documentation. Stripe, Vercel, Cloudflare and Anthropic all serve one, because a developer pastes that URL into Cursor or Claude Code before starting work, and the assistant then reads the real docs rather than what it remembers. If your product has docs, this alone justifies the file.
• Terms for AI in plain words. robots.txt can only say yes or no. This file is where “cite with a link, do not reproduce whole” lives, and an assistant that is going to quote you reads it right before it does.

What it will not do is get you into AI Overviews, improve a ranking, or make ChatGPT recommend you. Anyone selling it as that is selling the file, not the result. The pages themselves still do the work, which is why the launch checklist treats llms.txt as one line near the end, after the title, description and structured data on every route.

How do you make one without having to maintain it?

Generate it from the same list that builds your sitemap. A hand-written file is out of date the day you publish a page and forget it. On this site every route lives in one typed list, and the sitemap, the ⌘K search index and llms.txt all read from it, so a new entry appears in all three by existing. The launch check skill then compares the three against each other and fails if they disagree.

1# Site name
2
3> One paragraph. What the site is, who it is for, in a stranger's words.
4
5Who runs it and where. How the file is organised.
6
7## Section name
8
9- [Page title](https://example.com/page): what the page is for, one line.
10- [Page title](https://example.com/other): one line.
11
12## Optional
13
14- [Imprint](https://example.com/imprint): legal, skip unless needed.

Two rules that are easy to get wrong. Every URL is absolute, because the file gets read out of context by something that has no idea what domain it came from. And the one-line descriptions are the file. A list of bare links tells an assistant nothing the sitemap did not.

I keep mine because it costs nothing once it is generated, and because the one reader it does have is a person who cared enough to point an assistant at my site. That is a better visitor than most.