Some platforms generate llms.txt by default, but publishing a file does not establish that anyone uses it. Ahrefs found that 97% of the files in its sample received no requests during May 2026. Whether maintaining one is useful depends on its intended readers and content.
The file nobody asked for
Most UK sites checking for llms.txt on 5 September 2026 don’t have one. gov.uk, bbc.co.uk, nhs.uk, theguardian.com, ons.gov.uk, hsbc.co.uk, johnlewis.com and monzo.com all return 404. Two that do exist are worse than useless: rightmove.co.uk serves an HTML “we couldn’t find that page” as a 200, and arm.com’s first link is malformed, reading https://https://www.arm.com/.... Meanwhile allbirds.co.uk and huel.com have working files they never wrote — Shopify generated them automatically.
Where this file came from
Jeremy Howard of Answer.AI proposed llms.txt on 3 September 2024, solving a problem he had building coding-docs tools: HTML is noisy and context windows are finite, so give a model a curated Markdown index instead. The spec is short. An H1 is the only required element; an optional blockquote summary follows, then H2 sections of - [name](url): note links, plus an “Optional” section for anything skippable. A v2 landed in August 2026, adding Link rel="alternate" type="text/markdown" and rel="describedby" relations and subpath scoping, so /docs/llms.txt can cover everything under /docs/. It isn’t robots.txt (which sets permissions) or sitemap.xml (which lists everything with no curation) — it’s a hand-picked map.
The proposal is maintained in the Answer.AI repository. Its activity and adoption do not by themselves demonstrate a search benefit.
Adoption is real but small, and uneven
Three independent measurements agree on direction, disagree on scale. Rankability found top-1,000 site adoption rose from 0.3% in June 2025 to 8.7% in June 2026. HTTP Archive analysis by Casey Burridge put the top-10,000 figure at 5.61% in June 2026, up 5.4x from 1.04% a year earlier — roughly 39,000 sites in the top million. SE Ranking’s survey of 300,000 domains found only 10.13% overall, and counter-intuitively, big sites (100,001+ monthly visits) were less likely to have one (8.27%) than mid-traffic sites (10.54%).
The skew by category is stark. Rankability’s breakdown shows technology sites at 36.4% adoption against 0% for government and 0% for social media. And most of the growth isn’t organic: Burridge found Shopify stores at 78.1% adoption in the top 10,000, dwarfing WordPress (8.7%) and Drupal (1.5%) — that’s a platform shipping the file by default, not merchants deciding to write one.
Does anyone actually read it?
The evidence splits cleanly by which kind of reader you mean.
The case against
Google’s John Mueller said on Bluesky on 17 June 2025: “FWIW no AI system currently uses llms.txt… none of them fetch the llms.txt file.” Gary Illyes told Search Central Live APAC on 23 July 2025 that Google “doesn’t support LLMs.txt and isn’t planning to.” An llms.txt briefly appeared on Google’s own Search Central docs in December 2025 and was pulled; asked if that was an endorsement, Mueller replied “to be direct, no” on 20 January 2026.
The numbers back the scepticism up. SE Ranking’s regression model found that removing llms.txt as a factor actually improved its prediction of AI citations. And the sharpest data point: Ahrefs analysed 137,210 domains using its analytics products in May 2026 and found 97% of llms.txt files received zero requests in the study period. Of the requests that did land, 96% were bots of any kind, and AI retrieval bots specifically made up just 1.1% — Slackbot fetched llms.txt more often than PerplexityBot did. No AI bot in the sample even probed for a missing file. Ahrefs’ verdict: “the cons outweigh the pros right now.”
The case for — it’s a different reader
The counter-evidence isn’t about search crawlers at all. Chrome Lighthouse 13.3.0, released 7 May 2026, made the experimental “Agentic Browsing” category part of its default configuration, including an llms.txt audit alongside WebMCP checks. Anthropic, OpenAI and Gemini publish documentation files of their own. This demonstrates a supply of agent-oriented documentation; it does not establish how often their consumer search products read other sites’ files.
A separate single-site log report from April 2026 recorded requests using Accept: text/markdown. Those are content-negotiation requests, not necessarily visits to llms.txt. The report does not connect them to downstream citations or referrals, and a user-agent label alone is not proof of the requester’s identity. These observations concern a different behaviour from Ahrefs’ index-file study; they do not overturn its result.
Shopify has taken this furthest. Its auto-generated files for Allbirds and Huel aren’t a search aid at all — they’re agent instructions, pointing at shop.app/SKILL.md and documenting Universal Commerce Protocol endpoints (GET /.well-known/ucp, POST /api/ucp/mcp) with tools like search_catalog and create_checkout, explicitly noting that “checkout requires human approval.”
What this means in practice
Think of it as three layers: llms.txt is the map, your Markdown pages are the territory, and WebMCP is the hands an agent uses to act. Lighthouse’s Agentic Browsing category now audits the first and the third side by side. Cloudflare’s Markdown for Agents, shipped 12 February 2026, makes the territory cheap to produce — eligible enabled zones can serve text/markdown on request without maintaining separate .md files by hand. Current availability lists Pro, Business, Enterprise and SSL for SaaS.
None of this establishes llms.txt as a ranking lever. Google says Search ignores it. It can provide a useful map for agents that actually consume it. Lighthouse treats a missing file returning 404 as Not Applicable; errors or invalid content in a supplied file are different outcomes. Passing that audit does not prove customer use.
What to do
- Write one only if you have something worth curating — docs, an API, a pricing page, policies. A blank template copied from a generator earns nothing.
- Use the proposal’s H1 and curate a concise set of useful links. There is no required 20–50-link minimum; the right size depends on the content.
- Serve it as
text/plainortext/markdown, never HTML disguised as a 200 (rightmove.co.uk’s soft-404 fails Lighthouse’s own check). - Check the links resolve. Our recorded arm.com example had a doubled
https://https://prefix; that dated observation does not establish its current state. - Regenerate it when you deploy. A stale file that lists pages you’ve since removed is worse than no file at all.
AIEO checks for a valid, non-empty llms.txt in its AI Discoverability group, but reports it as informative rather than scored — the evidence above is exactly why. Run the free scan to see where your site actually stands.