Do you need an llms.txt file?
llms.txt is the most over-recommended file in AI search. It is not useless, but almost every article telling you to add it overstates what it does, so here is the state of it as plainly as we can put it.
What is llms.txt and do I need it?
llms.txt is a proposed plain-text file that lists a site's most important content for large language models. Most major AI companies have not committed to reading it in production, and Google has stated its search and AI features do not use it. It has real value for developer documentation and agent tooling, and little measurable effect on B2B search visibility.
What the file is supposed to do
The proposal is reasonable in principle: give models a clean, curated map of your site so they use your best content rather than guessing from navigation and boilerplate. It sits at your root, like robots.txt, and lists key pages with short descriptions.
Who actually reads it
This is where the recommendation and the reality diverge. Google has stated directly that Search does not use it and that no special machine-readable file is needed to appear in its AI features. OpenAI's crawler documentation governs its bots through robots.txt and does not reference it.
Where it does get used is agent and developer tooling: coding assistants and agent frameworks consume it, and documentation sites benefit accordingly. Adoption across the wider web sits at roughly one site in ten, and studies have not found a correlation between having the file and being cited more often.
What to do instead
If you have thirty minutes, spend it on the things with evidence behind them:
- Get your robots.txt right. This one is genuinely read by every major engine. Make sure the search-time retrievers can reach you.
- Restructure your top pages for extraction. A direct answer in the opening lines under a question-shaped heading does more than any file at your root.
- Build brand mentions. Slower, harder, and the strongest correlate with AI visibility.
If you publish developer documentation, add llms.txt. It is cheap and it helps in that specific context. Just do not expect it to change your B2B search visibility, and be sceptical of anyone selling it as an AI search strategy.
Key takeaways
- Google has said its search and AI features do not use llms.txt
- Adoption is around one site in ten with no demonstrated citation lift
- It has genuine value for developer docs and agent tooling
- robots.txt, page structure, and brand mentions carry the actual weight
Frequently asked questions
Will llms.txt hurt my site?
No. It is a plain text file that engines ignoring it simply do not fetch. The cost is the time spent building and maintaining it, and the opportunity cost of believing it did something.
Should I add it anyway just in case?
If you have developer documentation, yes. If you are a B2B services or SaaS marketing site, put the effort into robots.txt correctness and page structure first and revisit if adoption changes.
What is the difference between llms.txt and robots.txt?
robots.txt is an established standard that every major crawler reads and obeys, controlling access. llms.txt is a proposal that curates content for models, and support for it is limited and inconsistent.
How do I control AI crawlers properly?
Through robots.txt, naming the specific user agents. Training crawlers and search-time retrievers are separate agents, so you can allow citation while opting out of training if you choose.
Want to be the company AI engines recommend?
We run AEO and GEO alongside outbound so buyers find you before the cold email lands. Book a call and we will show you where you stand today.
Book a strategy call →