llms.txt and what AI crawlers actually read

llms.txt is a simple idea borrowed from robots.txt. You place a Markdown file at your domain root that points language models to your most important pages in clean, readable form. The pitch is that instead of crawling cluttered HTML, a model reads a tidy summary you wrote yourself.

What the file actually is

It is a plain Markdown file at /llms.txt. At the top, a short description of what your site is. Below it, grouped links to the pages you most want understood: docs, key product pages, pricing, comparisons. Optionally a fuller version that inlines the content itself, so a model does not even need to follow the links.

# YourBrand

> One honest sentence about what the product does and for whom.

## Core pages
- [Pricing](https://example.com/pricing): Plans from $X/mo, what each includes.
- [Product](https://example.com/product): What it does, in plain language.

## Guides
- [Guide title](https://example.com/guides/one): One-line summary a model can trust.

What it does, honestly

At the time of writing, no major answer engine has committed to reading llms.txt as a ranking input. It is a proposal with growing adoption, not a standard the big models obey. Treat it as low-cost hygiene: it cannot hurt, it makes your site easier to parse, and it positions you well if support arrives.

• Do add it if you have real docs or a content-heavy site. It is an afternoon of work.
• Do keep it honest and current. A stale map is worse than no map.
• Do not expect it to move answers on its own today. It is not a growth lever yet.

Know your crawlers

The AI traffic hitting your site is not one thing, and the robots.txt decisions are different for each kind. Three groups matter. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data for future models. Retrieval bots (OAI-SearchBot, PerplexityBot) fetch pages live to answer a user's question right now. And classic search crawlers (Googlebot, Bingbot) still matter because answer engines lean on search indexes to decide what to retrieve.

Blocking a training crawler keeps you out of the next model version's memory. Blocking a retrieval bot keeps you out of today's cited answers, which is usually the opposite of what a brand wants. The most common self-inflicted wound we see is a WAF or bot-protection preset that silently 403s every non-Google crawler — the site looks fine in a browser while every AI engine sees a wall. If you want to be recommended, verify each of these bots gets a 200 from your key pages.

What actually moves answers now

The files models genuinely read are the boring ones: a robots.txt that does not block them, a complete sitemap, fast server-rendered HTML, and structured data on the pages that matter. Those are doing the heavy lifting while llms.txt is still optional.

Ship llms.txt because it is cheap and tidy, not because it is the lever. The lever is still clear, quotable, crawlable pages.

A sane order of operations

Make sure you are crawlable and rendered, add structured data where it earns its keep, then add llms.txt as the finishing touch. Doing it in that order means the work that actually counts is done before you reach for the file that might count later.

Questions, answered

Do AI engines actually read llms.txt?

Not as a committed standard. No major engine has confirmed llms.txt as an input to its answers. Adoption is growing and the cost is one afternoon, so it is sensible hygiene — but crawlable HTML, an accurate sitemap, and structured data are what demonstrably feed answers today.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they may not fetch; llms.txt suggests what language models should read first, in clean Markdown. One is an enforcement file with two decades of support, the other is a young proposal with no enforcement and partial adoption. You need robots.txt; llms.txt is optional polish.

Should I block AI crawlers like GPTBot?

Only if you genuinely do not want to appear in AI answers. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) removes you from future model knowledge; blocking retrieval bots (OAI-SearchBot, PerplexityBot) removes you from today's cited answers. For most brands the goal is the opposite: verify all of them get a 200.

Where does llms.txt go and what format is it?

A plain Markdown file served at your domain root: yoursite.com/llms.txt. It starts with a one-line description of the site, then grouped links with one-sentence annotations for the pages you most want models to understand — product, pricing, docs, and key guides.