Assistants and answers
A file proposing which of your pages a machine should read first. It is a community convention rather than a standard, and nobody is obliged to fetch it.
Also called LLMs.txt · AI readme
The idea is sound and cheap: a curated map at a fixed address, in Markdown, naming the pages worth reading and what each is for, so a machine with a small budget spends it well. Ours lists the writing, the plain-text version of every piece, the datasets and the contact address.
Who honours it is a different matter, and most published answers to that question are not evidence. There is no RFC and no formal standard. The vendor guides contradict one another on which operators fetch it and cite each other rather than any operator statement. The one clear position on the record is Google’s, stated by Gary Illyes at Google’s own Search Central event and repeated in the mythbusting section of its guidance on generative-AI features: Google does not crawl or use the file, and normal search work is what its AI surfaces run on. Everything else circulating about adoption is inference, and repeating it here would be repeating a claim we cannot check.
So we instrumented it instead. This site records every fetch of its own llms.txt and its robots.txt, at the edge, by declared agent — the first version of that instrument excluded both along with every stylesheet, which meant we were publishing on the subject while unable to answer it from our own logs. The comparison between the two files is the finding worth waiting for: one of them is honoured by everybody.
It ranks nothing and grants nothing. At best it is a hint an operator may choose to take, which makes it worth the twenty minutes it costs to generate and not worth a line item. A proposal that prices llms.txt as a deliverable is pricing a file, and the honest question to put back is which operator is expected to read it and where they said so.
Knowing the definition is not the same as being able to check the figure. These are the procedures that do the second thing.
- Testing a promise about AI answers
- “We will get you cited by ChatGPT and the other assistants.” · 20 minutes, 5 questions.
The definitions are the easy part. Whether the figure on your dashboard was computed this way is a different question, and usually the more expensive one.