Skip to main content
Book a call
Last updated: ⁨November 20, 2026⁩

Nobody Reads Your llms.txt. Check Your robots.txt

FROM THE TEAM
Hamed Azimi
Hamed Azimi
Founder, NexusWave
Let's build something together.

We write these guides from real client work — GEO, SEO, and builds we've shipped. If a piece raised a question about your own site, we're happy to look at it.

  • Straight answers, no retainer required
  • We reply in your language — EN · فارسی
  • Based in Canada, working worldwide
Start a conversation ←See what we do ←

Your developer can add an llms.txt file this afternoon. The AI crawlers you would add it for will not ask for it. That gap — between a file that takes an hour to publish and a file nothing ever fetches — is where a lot of this year's AI-search budget went.

The file itself is a reasonable idea. llms.txt sits at your site's root and lists your important pages in Markdown, so a language model gets a clean map of your content instead of parsing your navigation. The proposal is sound engineering. Its problem is that it is a convention nobody on the reading side ever agreed to.

Who actually fetches llms.txt?

Ahrefs answered that on 15 June 2026 with server data rather than opinion. Across 137,210 domains carrying May 2026 traffic in its own analytics, 28% published a valid llms.txt — and 97% of those files received zero requests for the entire month. Of the requests the remaining 3% did get, AI retrieval bots, the crawlers that fetch a page to answer a live question, accounted for 1.1%. Audit tools and general web crawlers outnumbered them, and 12% of all the traffic came from tools whose job is to validate llms.txt files.

The finding that settles the argument is the one about files that do not exist. Ahrefs looked at requests for llms.txt addresses that returned nothing at all, and the AI-bot share of those was zero. No AI bot speculatively asks. If nothing probes for the file, publishing one cannot change what any crawler does — it can only be found by a tool you told where to look.

A smaller check points the same way. One site published an llms.txt and then read 14 days of its own nginx logs across 18 named AI crawler families: those crawlers fetched robots.txt 723 times and llms.txt zero times, while making 151 requests to real content pages. One site is not a census. It is a useful shape, though — the crawlers were there, reading pages, and reading the file that has been standard since 1994.

137,210
domains in the Ahrefs llms.txt study (May 2026 traffic)
97
percent of published files with zero requests that month
1.1
percent of those requests that came from AI retrieval bots
0
requests AI bots made for files that were not there

So why do large sites publish one?

Because it does one job well: it is an index for a tool you point at it. OpenAI's own developer documentation opens by telling the reader that the complete documentation index is at its llms.txt, and that appending .md to any page address returns Markdown. That is an engineer deliberately handing a coding assistant a map. It is a documentation convenience, and if you sell developer tools it may well be worth maintaining. It is not a search-visibility lever, and Google says so without hedging: you do not need to create new machine readable files, AI text files, or markup to appear in its AI features, and Google Search ignores them, so the file neither helps nor hurts your ranking.

Harmless is not the same as useful

The file will not damage your visibility, because the engines ignore it. The cost is the belief that travels with it — that the AI-readiness work is now done. Keep it as documentation for tools you point at it, and spend the hour on crawler access instead.

What actually gates an AI citation?

Access, first, and it is finer-grained than most people assume. OpenAI documents four separate crawlers, all controlled from robots.txt: OAI-SearchBot surfaces sites in ChatGPT's search features, GPTBot crawls content for model training, ChatGPT-User handles fetches a person asked for inside a conversation, and OAI-AdsBot checks pages submitted as ads. Different jobs, different user agents. The one that decides whether ChatGPT can cite you in a live answer is OAI-SearchBot — not GPTBot. A business that blocked "the OpenAI bot" to keep its writing out of training may have blocked itself out of answers instead. Those are opposite outcomes from two lines in the same file.

Then the ordinary things, which Google lists without inventing a single new file: be indexed and eligible for a snippet, allow crawling in robots.txt and at your hosting layer, keep important content as text rather than sealed inside images, make structured data match what a visitor actually sees, keep your Google Business Profile current, and verify the site in Search Console. None of that is new. That is the point. The work that earns a mention in an AI answer is the work that earned a ranking, done well enough that a machine can lift one paragraph out of context and still be right.

  1. Open your robots.txt and read it line by line.
  2. Allow the search crawlers by name, OAI-SearchBot first.
  3. Decide training access separately from search access.
  4. Check the firewall and bot rules too, not only the file.
  5. Grep a week of server logs for each crawler's user agent.
  6. Put a direct answer in the first paragraph of every page.
  7. Keep prices, hours and service names as text, not images.
A file nothing fetches is not an optimization. It is a line on an invoice.
Hamed Azimi, Founder, NexusWave Technologies

How do you know it worked?

Two checks, and both have to be able to fail. First, access: fetch your own robots.txt, match it against the crawler names you care about, then search a week of server logs for each of those user agents. A rule you believe allows a crawler is a claim about intent; a request from that crawler sitting in your log is a claim about state. If a crawler you want never appears, the work is blocked at the door and nothing on the page matters yet.

Second, the citation itself. Write down five questions a customer would ask that you should be the answer to, put them to the assistants you care about, and record whether you are named and linked. Run the same five every month. It is a crude measure and it wobbles between runs, and it is still the only one tied to the outcome you are paying for. A report that only confirms a file exists cannot fail, which is exactly why it is not evidence.

Find out what is blocking the citation

We verify AI-crawler access, score your content structure, inventory your structured data, review entity clarity and freshness signals, add a competitor AI-visibility snapshot, and hand you a prioritized fix list, for a fixed 499 dollars.
Hamed Azimi

Hamed Azimi

Founder, NexusWave

Hamed Azimi is the founder of NexusWave Technologies, a Vancouver-based AI-native digital agency and product studio.

FROM THE TEAM
Hamed Azimi
Hamed Azimi
Founder, NexusWave
Let's build something together.

We write these guides from real client work — GEO, SEO, and builds we've shipped. If a piece raised a question about your own site, we're happy to look at it.

  • Straight answers, no retainer required
  • We reply in your language — EN · فارسی
  • Based in Canada, working worldwide
Start a conversation ←See what we do ←
The NexusWave brief
One useful email a month — GEO findings & build notes, no noise. Double opt-in.
FAQ

Frequently asked questions

Should I delete the llms.txt I already have?

No. It costs nothing to serve and the engines ignore it, so removing it buys you nothing either. Keep it if a tool of yours reads it, let it go stale otherwise, and stop counting it as AI-search work on anyone's invoice.

If I block GPTBot, do I vanish from ChatGPT answers?

Not by itself. GPTBot crawls for model training; OAI-SearchBot is the crawler behind ChatGPT's search features. Blocking the training crawler while allowing the search crawler is a coherent position, and it is the one most businesses actually want. Check both lines, because one broad rule aimed at OpenAI usually catches both.

Is there any file that does help?

robots.txt, and only in the sense that it decides who gets in. Past access, the lever is the page itself: a direct answer near the top, structured data that matches the visible content, and material that stays current. There is no file you can add that substitutes for being worth quoting.

RELATED READING
All articles →
GEO & AI Search
How AI Assistants Decide Which Businesses to Recommend
Inside the signals AI engines weigh before they name a business in an answer — and the moves that earn a citation.
Hamed Azimi
Hamed AzimiFounder, NexusWave
⁨June 20, 2026⁩⁨3 min⁩4:01
GEO & AI Search
What Is GEO? The Complete Guide for Business Owners
How businesses get cited in AI answers from ChatGPT, Perplexity, and Google AI Overviews — what it is and how to start.
Hamed Azimi
Hamed AzimiFounder, NexusWave
⁨June 12, 2026⁩⁨2 min⁩5:20
Paid Ads & Social Media
ChatGPT Ads Dropped Max Bids. Prove Your Tracking Works
ChatGPT Ads replaced max bids with cost targets on 9 November. A target is an average, not a ceiling, so your conversion tracking is now the brake.
Hamed Azimi
Hamed AzimiFounder, NexusWave
⁨November 17, 2026⁩⁨4 min⁩6:52
GET STARTED
Tell us what you’re building.
Share your goals and you’ll get an itemized written proposal within one business day — in English, Persian, or Arabic. No lock-in, no pressure.
Skip the form — book directly
A 30-minute intro call, synced live to our calendar via Skejel.
Pick a time
START THE CONVERSATION1-day reply
⁨Budget range: *⁩
Posts directly into Zoho CRM. Page URL, locale & UTM parameters are captured automatically for attribution.
Prefer chat? The SalesIQ bubble in the corner reaches a human — the bot only handles the first hello.
5.0 · Google & Clutch1 business-day replyEN · FA · AR