LLMs.txt in 2026: Does Your Website Actually Need It?

Every few years the SEO industry adopts a file that promises to tell machines what matters. Meta keywords did it in the nineties. llms.txt is doing it now, and the parallel is close enough to be worth taking seriously — a plain-text, self-declared, unverified statement of what you'd like machines to think about your site.

That comparison is not quite fair, and I'll explain where it breaks down. But the burden of proof sits with the people selling it, and right now the primary documentation runs against them.

This article covers what the file is, how it differs from the two files that genuinely matter, what Google has published about it, and — the part that actually affects your traffic — how to set AI crawler policy deliberately rather than by copying someone's 2023 gist.

1. What llms.txt actually is

The proposal is straightforward: a Markdown file at /llms.txt containing a curated, human-written summary of your site — what it is, what the important pages are, and how they relate. The stated motivation is that language models work with limited context windows, and a hand-written index is more useful to them than a raw crawl of navigation, cookie banners and boilerplate.

A typical file looks like this:

# Company Name

> One-paragraph description of what this organisation does.

## Products
- [Product A](https://example.com/a): what it is and who it's for
- [Product B](https://example.com/b): what it is and who it's for

## Documentation
- [Getting started](https://example.com/docs/start)
- [API reference](https://example.com/docs/api)

As an idea it is reasonable. The problem is not the format — it is that a file only does something if somebody reads it, and the reading is the part nobody has committed to.

2. robots.txt vs sitemap.xml vs llms.txt

These get discussed as though they were three versions of the same idea. They are not remotely comparable in status.

Read that last row carefully, because it contains the whole strategic point. A mistake in robots.txt can remove you from an entire platform's answers. A mistake in llms.txt costs nothing, because nothing depends on it. Attention should follow consequence.

3. Google's published position

Google's guidance on optimising for generative AI features contains an explicit mythbusting section, and this file is named in it. The position has three parts:

• Google Search does not use llms.txt or similar files.
• You do not need to create new machine-readable files, AI-specific text files, or Markdown versions of your pages to appear in generative AI features.
• Publishing one neither helps nor harms your visibility or rankings in Google Search.

That last clause is the one to hold onto in an argument. It is not "we don't recommend it" or "it's low priority." It is a statement that the file is inert with respect to Google.

Google's stated position on what does matter is consistent and unglamorous: be indexed, be eligible for a snippet, have unique and useful content, keep pages technically accessible, and don't block the crawlers you want to be read by.

What about the other providers?

The honest summary is that nobody has committed publicly. OpenAI, Anthropic and Perplexity all document their crawlers and all respect robots.txt; none of them documents llms.txt as a retrieval input. Some developer tools and documentation platforms do read it, deliberately, because a human pointed them at it — that is a real use case, and it is a narrow one.

So: not vapourware, not a standard. A file with genuine adoption in developer tooling and no adoption in the systems most people are publishing it for.

4. The crawler decisions that do matter

Here is the substitution I'd make. The hour you were going to spend writing llms.txt is better spent auditing the file that has actual enforcement behind it.

The most common failure I find in audits is a robots.txt copied from a 2023 blog post at the height of the "block the AI scrapers" wave. Those posts blocked broadly. Since then the crawler landscape has split into distinct purposes, and a blanket block now costs visibility on platforms the business actively wants to appear in.

5. Training vs search crawling

The distinction that makes a coherent policy possible: training crawlers collect content to build models; search crawlers collect content so it can be retrieved and cited in answers. They are separate user-agents and separate decisions.

Which means the position most businesses actually want is available:

# Decline training, keep search visibility
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Stay retrievable and citable
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Whether you want that posture is a business decision with a real trade-off. Training exclusion protects content you consider proprietary; it also removes you from the latent knowledge a model carries between retrievals. My own site allows both, because visibility is the goal and the content is published to be read — but I'd argue the other way for a publisher whose archive is the product.

6. Should you publish one?

My answer is a qualified yes, with the reasoning stated honestly so nobody mistakes it for a ranking recommendation.

Reasons to publish

• It costs an hour. If you already have a clear picture of your site, writing it is transcription.
• Some tools genuinely read it — particularly developer-facing ones pointed at documentation.
• Option value. If adoption grows, you're already there. If it doesn't, you've lost an hour.
• The exercise is clarifying. Writing one paragraph on what your business is and listing the pages that genuinely matter surfaces uncomfortable questions about your information architecture. That has been the main benefit for me.

Reasons not to bother

• Google ignores it. If Google is your visibility strategy, this file is not part of it.
• It's another artefact to maintain. A stale llms.txt describing last year's services is worse than none.
• Opportunity cost. An hour on rendering, crawler policy or one genuinely original page returns more.

I publish one at /llms.txt. I do not count it as part of my AI visibility work, and I would not include it in a client proposal as a deliverable that affects rankings.

If you do publish one

7. Frequently asked questions

Where I'd land

Publish llms.txt if you want to. It is cheap, it is harmless, and the act of writing it will probably improve how you describe your own business. Just don't file it under AI visibility, and don't pay anyone for it.

Then go and read your robots.txt line by line. One of those two files can remove your business from an entire platform's answers, and it isn't the new one.

Work with me

Crawler policy set deliberately, not inherited from a 2023 blog post

I'm Jayant Solanki — an SEO, GEO and automation strategist working with eCommerce, local-service and global brands. I'd rather fix the file that has consequences than sell you the one that doesn't.

A crawler and visibility engagement typically covers:

• robots.txt reviewed against the current AI user-agent list
• Training vs search-crawling policy decided per platform, with the trade-offs stated
• WAF and bot rules checked for blocks that catch legitimate agents
• Server-log analysis of which crawlers actually reach you, and how often
• Rendering checks — because a crawler that arrives and finds an empty page is the same as one that never came

Book a technical review → See how I work

Ranked #1 for "metal buildings" · +30% YoY organic traffic · +60% inbound calls for a UAE local-service brand

Sources

• Google Search Central, Optimizing your website for generative AI features on Google Search
• Google Search Central, Google crawlers and user-agents
• Google Search Central, Introduction to robots.txt
• IETF, RFC 9309 — Robots Exclusion Protocol
• OpenAI, OpenAI crawlers
• Anthropic, Anthropic web crawlers
• Perplexity, PerplexityBot documentation
• llmstxt.org — the original proposal

Jayant Solanki

AI-Ready SEO, GEO & AIO strategist based in Indore, India, working with eCommerce, local service and global brands across India, the UAE and the US. Ranked a client #1 for a 45,000/month commercial keyword; builds the measurement and automation behind the work.

More about Jayant →