llms.txt is a proposed convention: a Markdown file at your site root that gives large language models a curated, clean summary of your site and links to its most useful content. It was proposed by Jeremy Howard in 2024 and has been adopted by a number of developer-tooling and documentation sites.
It is worth understanding precisely, because it sits in a category where marketing enthusiasm has run well ahead of demonstrated effect.
The problem it addresses
When a language model retrieves a web page, it gets HTML built for browsers: navigation, cookie banners, analytics, ads, and the actual content interleaved. Extracting the substance costs tokens and introduces errors. Context windows are finite, and much of what a page contains is chrome rather than content.
llms.txt proposes that sites publish a Markdown file — plain, structured, no chrome — that describes the site and links to its key resources. A companion convention, llms-full.txt, contains the full content rather than just links.
The format is deliberately simple: an H1 with the site name, a blockquote summary, then H2 sections containing lists of links with short descriptions.
# WriteMySEO
> AI-drafted, human-reviewed SEO content for specific industries,
> delivered on a flat monthly subscription.
## Services
- [Pricing](https://writemyseo.com/#pricing): Three flat monthly plans.
- [Industries](https://writemyseo.com/#industries): Industry-specific approaches.
## Blog
- [SEO data and technology](https://writemyseo.com/blog/): Daily analysis.The honest status
No major AI provider has publicly confirmed that llms.txt influences retrieval, ranking, or citation. Adoption on the publishing side has grown; confirmed consumption on the model side has not been demonstrated. Analyses of server logs by several SEO practitioners have found little evidence of AI crawlers requesting the file at meaningful rates.
That is the accurate picture, and anyone selling llms.txt implementation as an AI search ranking tactic is ahead of the evidence.
Why implement it anyway
The cost is close to zero and the downside is nil. Beyond that, there are two defensible reasons.
Option value. If consumption does materialize, sites that already publish a well-formed file are ready. The file takes an hour to write and roughly no time to maintain if you generate it from the same source as your sitemap.
It forces a useful exercise. Writing a clean, structured summary of what your site is and which pages matter is clarifying. Teams that do it frequently discover their own information architecture is harder to describe than they assumed — which is a finding about the site, not about the file.
What actually influences AI retrieval today
Since this is the underlying question people are asking when they ask about llms.txt, it is worth answering directly. Based on how current retrieval-augmented systems are documented to work, the things that demonstrably matter are:
- Being crawlable by the AI crawlers you want to be read by. If
GPTBot,ClaudeBot,PerplexityBot, orGoogle-Extendedare blocked inrobots.txt, nothing else you do matters for those systems. - Clean, semantic HTML. Content in the initial response, real heading hierarchy, no critical text locked behind interaction.
- Structured data. Explicit facts beat inferred ones.
- Self-contained, well-scoped sections. Retrieval systems chunk documents. A section that makes sense in isolation survives chunking; one that depends on three paragraphs of earlier context does not.
- Specific, verifiable claims. Content that states checkable facts is more citable than content that gestures at generalities.
- Being corroborated elsewhere. Facts that appear consistently across multiple independent sources are more likely to be reproduced confidently.
Every item on that list also improves conventional SEO, which is a useful property: it means you can pursue AI visibility without betting on any particular unproven convention.
The recommendation
Publish an llms.txt. Keep it short, keep it accurate, generate it from your existing content inventory so it does not rot. Spend fifteen minutes on it.
Then spend the rest of your time on the crawlability, structure, and specificity items above, because those have a mechanism behind them today rather than a proposal that may acquire one later.