Is llms.txt like robots.txt?
They do different jobs. llms.txt guides agents to the material you want them to read. robots.txt states crawl permissions for bots.
ANSWERS
What is llms.txt and does my site need one?
llms.txt is an optional proposed Markdown guide for agents — summaries and links — that can sit at your site root or a relevant subpath. Documentation tools have adopted it. Publishing it, or seeing requests for it, does not establish an AI-search ranking benefit. Use it when it helps agents navigate maintained factual material. ye5's GEO Web audit checks website access and readability.
What the file contains
llms.txt is a Markdown file that can live at your site root or at a relevant subpath. It carries summaries of what you do and links to key pages, in plain language. The proposal came from the open-source community in 2024 as a robots.txt-style convention, but for content rather than permissions.
Documentation tools adopted it; AI search has not promised anything
Documentation tools have adopted the format, so the file has real use in that world. Publication or file requests do not establish an AI-search ranking benefit. Google says no AI text file is required for its AI Search features. Crawler requests show interest, not outcomes (see: What is ClaudeBot?). Fetched is not the same as used.
When llms.txt is worth publishing
Publish it when it helps agents navigate maintained factual material. Prioritize reachable, useful pages first, then keep the file current as those pages change. There is a second benefit in the writing itself. Stating what you offer, in one page, in plain language is the exercise that improves how models read the rest of your site.
If you add one
- —Generate it from your site's data; don't hand-write a second copy of your marketing.
- —State products and factual claims — the things you want models to get right.
- —Keep it consistent with visible pages; contradictions are worse than absence.
- —Keep it current; a stale guide points agents at material you no longer maintain.
- —Remember the split: llms.txt suggests what to read, robots.txt sets crawl permissions.
Whether machines can reach and read your site is measurable (see: Should I block AI crawlers from my site?). ye5's GEO Web audit checks website access and readability across your templates, with llms.txt as one optional item among them.
FAQ
Does Google read it?
Google says no AI text file is required for its AI Search features. Publish llms.txt when it helps agents navigate maintained factual material, not for a promised ranking payoff.