Does llms.txt work in 2026? What the studies and Google say
Ahrefs found 97% of llms.txt files got zero requests in May 2026, and SISTRIX saw no citation effect. Here is the evidence and what to fix first instead.
, 6 min read
For getting cited by ChatGPT, Perplexity, Google AI Mode or AI Overviews, the evidence as of November 2026 says llms.txt does not work: almost no AI search bot requests it, and the largest citation study to test it found no measurable effect. It is cheap and harmless to keep, and it may help coding agents read your docs, but it should not be the first thing you fix.
The rest of this page walks through the evidence, explains why big platforms ship the file anyway, and lists what is better supported.
What llms.txt is
llms.txt is a Markdown file, usually at /llms.txt, that summarizes a site and links to its most useful pages. Jeremy Howard proposed it on September 3, 2024 as "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website." It is a proposal, not a standard that any AI search engine has said it follows.
The evidence
Ahrefs, June 2026: almost nobody fetches the file
Ahrefs published a study on June 15, 2026 covering 137,210 domains in its web analytics product. About 28% had an llms.txt file, and roughly 38,000 had a valid one. In May 2026, 97% of those files got zero requests.
The 3% that were requested received about 22,000 requests in total. AI retrieval bots, the ones that fetch pages to answer a user's question (such as OAI-SearchBot and PerplexityBot), made 233 of them. SEO audit tools made more requests than any AI category, and GPTBot, OpenAI's training crawler, was the largest single AI fetcher.
Ahrefs adds two fair caveats. Its customers skew technical, so 28% adoption is an upper bound. And a fetch is not proof the content was used, so the authors call every figure "a ceiling on actual llms.txt consumption." Their conclusion was that the file is "largely decoration" for ChatGPT, Perplexity and AI Overviews visibility, with coding agents as the most plausible audience.
SISTRIX, October 2026: no measurable citation effect
SISTRIX published a citation study on October 22, 2026 that analysed 96,418 URLs from 17,380 domains, comparing pages cited by ChatGPT, AI Mode and AI Overviews with uncited pages. On llms.txt its verdict was: "As things stand, there is no measurable effect on AI citations."
Cited URLs were slightly less likely to come from domains with an llms.txt file (11.7% versus 14.0% across all domains). SISTRIX is careful here: that comparison was at domain level, those domains differ in many other ways, and it does not show the file causes harm. The study is also correlational, covers only the German-speaking DACH market, and did not include Perplexity, Claude or the Gemini app.
Google: Search does not use it
Google's AI optimization guide (first published May 2026) is direct. You "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It also says creating one will "neither harm nor help" your visibility in Google Search, and lists llms.txt among the "AEO/GEO hacks" you can ignore.
That covers AI Overviews and AI Mode. OpenAI, Perplexity and Anthropic have not, as far as we could find, said their search features read llms.txt either.
Why Shopify and Yoast ship it anyway
If it does not move citations, why do large platforms offer it?
- Shopify documented on May 28, 2026 that every store serves a default
/agents.md, and that/llms.txtand/llms-full.txtpoint to the same content by default. Merchants can override each one with a Liquid template. - Yoast says llms.txt is "built into Yoast, free for everyone", including the free plugin. You turn it on in the site feature settings and it is generated automatically.
A reasonable reading: generating the file costs these platforms almost nothing, customers ask for it, and AI agents (not AI search) may use it more over time. That is a sensible product decision. It is not evidence that the file earns citations, and neither company claims it does.
One practical upshot: if you run Shopify or Yoast, you may already have the file, which is another reason not to pay someone to make one.
Is there any case for keeping one?
Yes, a narrow one. If you sell a developer product with API docs, coding agents are the most likely readers. Ahrefs found that Claude-Code requested files more often than any AI assistant or retrieval bot in its sample. A short, accurate llms.txt that links to your docs costs an hour.
Keep it truthful and in sync with the site. Ahrefs also flags prompt-injection risk when agents trust these files, which is a reason to keep yours plain and factual. If you want to know whether yours is used, check your server logs for requests to /llms.txt before and after you publish it.
What to do instead
The levers with stronger evidence are less novel. None of them guarantees a citation, and most AEO research is correlational, but these at least address how AI search engines actually reach and read your pages.
| Fix | Why it matters | Evidence |
|---|---|---|
| Let AI search crawlers in | Blocked bots cannot fetch your pages | Documented by OpenAI and Anthropic |
| Check your CDN and firewall | Cloudflare settings can block bots your robots.txt allows | Documented by Cloudflare |
| Serve content as HTML | Several AI crawlers do not run JavaScript | One large dataset, from 2024 |
| Put the answer first | Citations cluster near the top of pages | Correlational |
Remove stray noindex |
Pages excluded from search indexes are unlikely to be retrieved | Standard search behaviour |
1. Allow the crawlers that feed AI answers
OpenAI's crawler documentation states that "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers" (they can still appear as navigational links). Training is a separate control: blocking GPTBot does not remove you from ChatGPT search. Anthropic documents ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests) as separate tokens, and Perplexity documents PerplexityBot for search results.
If your robots.txt has a broad Disallow, name the search bots explicitly. A crawler follows the group that names it rather than the User-agent: * group:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
2. Check Cloudflare, not only robots.txt
Since September 15, 2026, Cloudflare's AI Crawl Control has three controls per domain: Search, Training and Agent. For Training, choose "Disallow AI Training" if you want to opt out of training. Choosing "Block" also blocks mixed-use crawlers such as Googlebot, Bingbot and Applebot, which removes the site from search. Cloudflare's own figures put sites blocking search bots at under 1%, so this is uncommon, but it is worth a two-minute check.
3. Make key pages readable without JavaScript
Vercel and MERJ found in December 2024 that none of the major AI crawlers they tracked, including GPTBot, ClaudeBot and PerplexityBot, render JavaScript. OpenAI's and Anthropic's crawlers fetch JavaScript files but do not execute them. The data is now two years old, but if your pricing or product pages only appear after client-side rendering, some AI crawlers may see an empty shell. Server-side rendering or static generation fixes this.
4. Answer the question in the first lines
Kevin Indig's February 2026 analysis of 18,012 ChatGPT citations, reported by Search Engine Land, found 44.2% came from the first 30% of a page. That is correlational, and we have not seen the paywalled original, but it matches common sense: state the answer plainly, then explain.
Check your site
The free scan at domyaeo.com checks your robots.txt rules for 14 AI and search crawlers, Cloudflare Content-Signal, how your CDN or firewall responds to AI crawler user agents (a hint, not proof), JavaScript-only rendering, noindex, and whether key pages open with a direct answer. Those are the checks with better evidence behind them than llms.txt.