Documentation got a second audience
Nobody sensible disputes the change. A developer evaluating three content platforms no longer opens three websites and reads three feature pages. They ask an assistant, get two suggestions, and are integrating against one of them before lunch. Somewhere in that sequence your documentation was read by something without eyes.
That changes what documentation is for. It is no longer only the manual for people who have already chosen you. It is now part of how the choosing happens, and it has to serve both jobs at once. If you are still measuring docs on ticket deflection alone, you are measuring the least interesting thing they do.
So far the conventional wisdom holds. Then it concludes that the answer is to get crawled more, and that is where it comes apart.
Reading is not referring
Cloudflare publishes the ratio between requests an AI crawler makes and referrals attributed to that platform. For the 28 days to 21 July 2026, compiled from Cloudflare Radar, it looked like this.
Mistral's crawler took roughly 3,389 pages for every referral it sent. Anthropic's ClaudeBot, 2,237. Perplexity, 225. OpenAI's GPTBot, 217. Microsoft, 35. Google, 4.6. DuckDuckGo, 2.5.
Two qualifications before anybody quotes those at a board. The metric is an aggregate across each operator's whole estate, not a trace of a particular page being read and later producing a visit, so it does not measure what your documentation specifically earned. And Cloudflare say plainly that referrals from Claude's native app arrive without a Referer header, that they expect the same of other providers' native apps, and that their figures may therefore overstate the imbalance by an amount nobody can currently size.
Take both seriously and the ratios are softer than they look. The shape still holds. A search engine's arrangement with you is legible: it reads pages in order to send people to them, and the exchange rate is single digits. A model is not running that trade, and no plausible correction for uncounted native-app traffic closes a gap of three orders of magnitude.
The direction of travel is worth noting too. ClaudeBot sat at roughly 23,951 to one in the first quarter of 2026 and 2,237 by July. GPTBot went from about 1,276 to 217 across the same period. That is referral behaviour improving, which is not the same thing as citations becoming more frequent, and the two get confused constantly.
But you are making the decision now, against today's numbers. And today, "agents will crawl my documentation and send me traffic" is not a plan.
A search engine reads your pages in order to send you people. A model is not running that trade.
llms.txt is not what you were told it is
The standard remedy is llms.txt: a plain markdown file listing what matters on your site, so a model can read you cheaply.
Google settled the search half of the question on 15 June 2026. Their guidance now says you do not need to create machine-readable files, AI text files, markup or markdown to appear in Google Search including its generative capabilities, because Search does not use them. They add that keeping such files for other systems is fine. No effect either way, stated by the largest crawler on the web.
Beyond Google, no major provider has publicly committed to reading llms.txt in production either. It remains a community convention.
The comparison people reach for is robots.txt, and it does not work the way it usually gets told. robots.txt was one person's proposal in 1994 and it was not standardised until RFC 9309 in September 2022, twenty-eight years later. It spent almost its entire life exactly where llms.txt is now: a convention with nothing formal behind it. What carried it was never a standards body. It was that the crawlers who mattered chose to honour it, and the standard arrived afterwards to describe what was already happening. The standardisation was the symptom. llms.txt has not had the thing that causes it.
So should you write one? That depends on something more specific than the advice usually asks. Three different things are being run together: a crawler discovering your file by convention, a coding assistant fetching it because its tooling looks there, and a developer explicitly pointing an agent at it. The third is reliable. The second happens with some tools and not others, and you can test which. The first is what the search-marketing advice is quietly promising, and there is no evidence for it.
So publish one when you can name the tools that will consume it, and then check that they do. Supabase, Zapier and Modal all publish one, and none of them is doing it for rankings. If you cannot name the consumer, you are writing a file for an audience you have assumed.
Being recommended is not being chosen
Two moments are being collapsed into one, and they behave nothing alike.
The first is recommendation. It happens inside model weights and retrieval indexes, formed over months out of everything that has been crawled. You have very little control over it, the feedback loop runs in training cycles, and it is where the lopsided crawl ratios go. It is the AI equivalent of brand: you influence it slowly and you cannot audit it.
The second is selection, and it happens in an editor, at the moment somebody is actually integrating. The developer has it down to two. Their agent fetches your documentation, reads it against the task in hand, and either produces working code or produces something that fails and gets blamed on you. That is where you are chosen or discarded. It takes seconds, and documentation quality is one of the few variables in it you can actually move.
llms.txt earns its place in the second moment rather than the first. Coding assistants do fetch it, and they are genuinely token constrained.
Nearly all the advice being written about this is aimed at the first moment, because the people writing it come from search marketing and the first moment looks like search marketing. The conversion is in the second moment, and it looks like engineering.
There is a consequence worth sitting with. A product can gain real adoption through AI-assisted development while its documentation traffic stays flat, because the contribution happens somewhere your analytics does not reach.
Recommendation is brand. Selection is an integration that either worked or it didn't.
What an agent needs that a human does not
The differences are not subtle once you go looking for them.
It needs pages organised by problem rather than by endpoint. Generated reference tells an agent what an endpoint does. It never tells it which endpoint solves the problem, because that was not the question the generator was answering. The page that converts is the one titled with a task.
It needs working code rather than prose about code. An agent will lift the example. If the example is a fragment embedded in a paragraph explaining the example, it will lift the fragment and invent the rest.
It needs small addressable units, because tokens are a budget. A four-thousand-word guide that answers the question in paragraph nineteen costs more than it returns. Structure so that paragraph nineteen has an address of its own.
Most of all it needs to be current, and unambiguous about what it is current for. A stale page is worse than a missing one. A missing page sends the agent looking elsewhere. A stale page makes it write confident code against an API you removed two releases ago.
And everything an agent does with your documentation is parsing, so consistency beats elegance. Real headings, real code fences, semantic markup, one canonical version of each fact.
The usual failure is version drift
The most common way this goes wrong is not a missing page. It is the right page for the wrong version, and you can reproduce it in ten minutes against your own product.
Ask an assistant to wire a frontend up to a Strapi 5 API and watch where it goes for documentation. Strapi 4 and Strapi 5 return different response shapes: the nested data.attributes wrapper is gone in 5, and documentId replaced the numeric id as the identifier you carry around. Both versions are documented thoroughly. Both are indexed. If the agent lands on the version 4 material it writes entirely reasonable code that reaches for a field which is not there, and the developer meets an error with your product's name on it rather than yours.
Point the same agent at version-specific documentation, with the version in the URL and working examples inside it, and the integration completes.
Nothing in that sequence is exotic and almost none of it is about AI. It is the ordinary cost of keeping two live versions of the truth, arriving at a reader with no way to tell which one it is holding. A human skims the page and notices the version banner. An agent frequently does not.
Where page-centric content gets expensive
Here is the bit that gets skipped, and it is the bit I get called about. It needs a concession first, because this is exactly where the argument usually overreaches.
An agent does not need markdown. A well-formed HTML page with semantic headings, real code blocks and a stable URL is perfectly readable, and plenty of documentation sites serve precisely that and do fine. Nor do you need a headless CMS to produce alternative formats: static site generators, build pipelines and HTML-to-markdown conversion all get you there. Anyone telling you these requirements can be met exactly one way is selling you the one way.
The problem is not capability. It is maintenance.
Almost every team I say this to agrees with it, then finds acting on it costly, because their content is not content. It is pages. The words live inside a template, the structure is implied by visual hierarchy rather than stored anywhere, and the same fact is pasted into four places.
That is survivable with one output format. It gets expensive at three, and expensive again every time the requirements move, because each format is a transformation somebody builds and then has to keep correct. Version drift is the same problem wearing a different hat: the fact lives in several places and nothing enforces agreement between them.
There is a quick test. Take your most important documentation page and try to get it as markdown, at a stable URL, without opening a browser. If you cannot, that is not fatal. It tells you roughly what the next format is going to cost you.
Structured content is the boring answer: the page as one rendering of a model rather than the thing itself, addressable at the component level, with markdown, JSON and HTML as outputs of one source. It buys you a single place where each fact lives. That was always the argument for it. What changed is only that the number of outputs went up.
In this order, because the order matters
Audit what an agent can actually read, before you buy anything. Fetch your own documentation the way an agent would, without a browser, and look at what comes back. It costs an afternoon and it is the step nobody regrets.
Then fix currency before format. Stale content actively harms you and it is the cheaper problem. Correct every page describing behaviour you have since changed, and if you support more than one major version, make sure a reader can tell which is which without inferring it.
Only then touch the architecture, and touch it once. If your content is structured, the next format is a serialisation job. If it is not, it is a transformation to build and then keep correct, and there will be another one after it.
Decide in advance how you will know whether it worked, because the traffic numbers are not going to tell you. Run the same integration task against your documentation on a schedule and record whether the agent finishes it. Time how long a new developer takes to a first working request. Watch support for failures that arrived out of generated code. None of those is clean attribution and every one of them is closer to the thing you care about than a referral count.
Untangling content stuck in page templates is most of what I get hired for. The audit is an afternoon and you do not need me for it, so do that part yourself first.
And be honest about the ratio while you work. You are not doing this because agents are about to send you a flood of traffic. Today they are not. You are doing it because when somebody's assistant is halfway through an integration against your product, you want it to have what it needs. That is the whole return, and it will never show up cleanly in analytics.