Your developer can add an llms.txt file this afternoon. The AI crawlers you would add it for will not ask for it. That gap — between a file that takes an hour to publish and a file nothing ever fetches — is where a lot of this year's AI-search budget went.
The file itself is a reasonable idea. llms.txt sits at your site's root and lists your important pages in Markdown, so a language model gets a clean map of your content instead of parsing your navigation. The proposal is sound engineering. Its problem is that it is a convention nobody on the reading side ever agreed to.
Who actually fetches llms.txt?
Ahrefs answered that on 15 June 2026 with server data rather than opinion. Across 137,210 domains carrying May 2026 traffic in its own analytics, 28% published a valid llms.txt — and 97% of those files received zero requests for the entire month. Of the requests the remaining 3% did get, AI retrieval bots, the crawlers that fetch a page to answer a live question, accounted for 1.1%. Audit tools and general web crawlers outnumbered them, and 12% of all the traffic came from tools whose job is to validate llms.txt files.
The finding that settles the argument is the one about files that do not exist. Ahrefs looked at requests for llms.txt addresses that returned nothing at all, and the AI-bot share of those was zero. No AI bot speculatively asks. If nothing probes for the file, publishing one cannot change what any crawler does — it can only be found by a tool you told where to look.
A smaller check points the same way. One site published an llms.txt and then read 14 days of its own nginx logs across 18 named AI crawler families: those crawlers fetched robots.txt 723 times and llms.txt zero times, while making 151 requests to real content pages. One site is not a census. It is a useful shape, though — the crawlers were there, reading pages, and reading the file that has been standard since 1994.
So why do large sites publish one?
Because it does one job well: it is an index for a tool you point at it. OpenAI's own developer documentation opens by telling the reader that the complete documentation index is at its llms.txt, and that appending .md to any page address returns Markdown. That is an engineer deliberately handing a coding assistant a map. It is a documentation convenience, and if you sell developer tools it may well be worth maintaining. It is not a search-visibility lever, and Google says so without hedging: you do not need to create new machine readable files, AI text files, or markup to appear in its AI features, and Google Search ignores them, so the file neither helps nor hurts your ranking.
Harmless is not the same as useful
The file will not damage your visibility, because the engines ignore it. The cost is the belief that travels with it — that the AI-readiness work is now done. Keep it as documentation for tools you point at it, and spend the hour on crawler access instead.
What actually gates an AI citation?
Access, first, and it is finer-grained than most people assume. OpenAI documents four separate crawlers, all controlled from robots.txt: OAI-SearchBot surfaces sites in ChatGPT's search features, GPTBot crawls content for model training, ChatGPT-User handles fetches a person asked for inside a conversation, and OAI-AdsBot checks pages submitted as ads. Different jobs, different user agents. The one that decides whether ChatGPT can cite you in a live answer is OAI-SearchBot — not GPTBot. A business that blocked "the OpenAI bot" to keep its writing out of training may have blocked itself out of answers instead. Those are opposite outcomes from two lines in the same file.
Then the ordinary things, which Google lists without inventing a single new file: be indexed and eligible for a snippet, allow crawling in robots.txt and at your hosting layer, keep important content as text rather than sealed inside images, make structured data match what a visitor actually sees, keep your Google Business Profile current, and verify the site in Search Console. None of that is new. That is the point. The work that earns a mention in an AI answer is the work that earned a ranking, done well enough that a machine can lift one paragraph out of context and still be right.
- Open your robots.txt and read it line by line.
- Allow the search crawlers by name, OAI-SearchBot first.
- Decide training access separately from search access.
- Check the firewall and bot rules too, not only the file.
- Grep a week of server logs for each crawler's user agent.
- Put a direct answer in the first paragraph of every page.
- Keep prices, hours and service names as text, not images.
A file nothing fetches is not an optimization. It is a line on an invoice.
How do you know it worked?
Two checks, and both have to be able to fail. First, access: fetch your own robots.txt, match it against the crawler names you care about, then search a week of server logs for each of those user agents. A rule you believe allows a crawler is a claim about intent; a request from that crawler sitting in your log is a claim about state. If a crawler you want never appears, the work is blocked at the door and nothing on the page matters yet.
Second, the citation itself. Write down five questions a customer would ask that you should be the answer to, put them to the assistants you care about, and record whether you are named and linked. Run the same five every month. It is a crude measure and it wobbles between runs, and it is still the only one tied to the outcome you are paying for. A report that only confirms a file exists cannot fail, which is exactly why it is not evidence.
Find out what is blocking the citation
We write these guides from real client work — GEO, SEO, and builds we've shipped. If a piece raised a question about your own site, we're happy to look at it.
- Straight answers, no retainer required
- We reply in your language — EN · فارسی
- Based in Canada, working worldwide



