There's a new file quietly showing up at the root of sites like Anthropic, Stripe, and Zapier: /llms.txt. It's the robots.txt of the AI era — except instead of telling crawlers what to stay out of, it hands language models a curated map of your best content and says: read this first.
We added one to this site, we score it in our own free SEO checker, and we watch AI referral traffic land in our analytics every month. So here's the honest version of the llms.txt story: what it is, what it isn't yet, and why the 15 minutes it takes to write one is probably worth it anyway.
What llms.txt Actually Is
llms.txt is a proposed standard introduced in September 2024 by Jeremy Howard, co-founder of Answer.AI (and before that, fast.ai). The spec lives at llmstxt.org, and the idea is disarmingly simple: language models have limited context windows and can't crawl your whole site at inference time, so give them one small, structured file that says what your site is and where the important parts are.
The format is plain markdown, deliberately — markdown is the one document format every LLM reads natively. An H1 for the site name, a blockquote summary, then H2 sections full of links with one-line descriptions. That's the whole spec.
The framing that clicks for most people: robots.txt is about blocking machine readers. llms.txt is about briefing them.
llms.txt vs robots.txt vs sitemap.xml
These three root files get conflated constantly. They solve different problems for different readers:
| File | Written for | Purpose | Format | Curated? |
|---|---|---|---|---|
| robots.txt | Search + AI crawlers | Permissions — what crawlers may NOT access | Plain-text directives | No — rules, not content |
| sitemap.xml | Search engine indexers | Exhaustive list of every URL you want indexed | XML | No — everything, no descriptions |
| llms.txt | Large language models | Curated map of your MOST important content | Markdown (LLM-readable) | Yes — hand-picked links + descriptions |
A fence, a phone book, and a welcome brochure. You want all three.
The Honest State of Adoption in 2026
Here's the part most llms.txt articles skip: no major AI provider has officially committed to consuming llms.txt as a retrieval or ranking input as of mid-2026. OpenAI, Anthropic, and Google haven't announced that their crawlers or models treat the file specially. Anyone telling you llms.txt is a confirmed ranking factor is selling something.
And yet adoption keeps growing anyway. Anthropic publishes one for its docs. Stripe publishes one. Zapier publishes one. Hundreds of developer-tool and docs sites have followed, because the cost-benefit math is lopsided: the file costs nothing to serve, takes minutes to write, and if any AI system starts consuming it — officially or through training data — early adopters are already positioned.
The demand side is what makes this more than a fad. AI referral traffic is real: on our own site we see visitors arriving from Gemini, Claude, ChatGPT, and Perplexity every month, and that channel is growing while traditional Google click-through rates fall. When AI assistants describe your site to a user, you want them working from your curated brief — not from whatever fragment of a 2023 crawl happens to surface. We cover the broader shift in our SEO vs AEO guide.
A Minimal llms.txt Example
The full spec is at llmstxt.org, but a valid, useful file is this small:
# Value Add VC
> Free venture capital tools, real-time tech and VC news,
> and data-driven analysis of startups, IPOs, and AI markets.
> Built by Trace Cohen for founders, VCs, and operators.
## Tools
- [SEO Score Checker](https://valueaddvc.com/seo-score): Free
23-signal SEO and AI-readiness audit for any URL
- [IPO Tracker](https://valueaddvc.com/ipo-tracker): Live tracker
of every major tech IPO with valuations and performance
## News & Analysis
- [Value Add Pulse](https://valueaddvc.com/pulse): Real-time
tech and venture capital news with analyst commentary
## Optional
- [Blog archive](https://valueaddvc.com/blog): 900+ posts on
startups, AI, and venture capitalH1 name, blockquote summary, H2 sections, one-line descriptions after each colon. The "Optional" section is spec-reserved for links an AI can skip when context is tight.
How to Write Yours in 15 Minutes
1. Create the file
Make a plain markdown file. It will be served at yourdomain.com/llms.txt — in Next.js, drop it in /public; on most hosts, the site root.
2. Name and summarize
H1 with your site name, then a blockquote: one paragraph on what the site is and who it's for. This is the highest-value real estate in the file.
3. Pick your 10–30 best pages
Not everything — the pages you'd want an AI to read before describing you: docs, flagship tools, cornerstone guides, pricing, about.
4. Group them under H2 sections
Sections like 'Tools', 'Guides', 'Data'. Each link gets a one-line description after a colon — that description is what the model actually uses.
5. Add an 'Optional' section
The spec reserves an H2 named 'Optional' for links an AI can skip when its context window is tight. Put long-tail material here.
6. Verify it serves
Load yourdomain.com/llms.txt in a browser (raw markdown is correct — no HTML needed), then confirm it with a checker.
How to Check Your llms.txt (Free)
The quick check is manual: load yourdomain.com/llms.txt in a browser. Raw markdown is the correct behavior — it's a file for machines, not a styled page.
But llms.txt is one signal among many, and it rarely fails alone. Our free SEO Score checker scans any URL for 23 ranking signals in one pass — llms.txt presence, indexability, title and meta quality, schema markup, canonical tags, internal links, and sitemaps — and gives you a scored report with specific fixes. It's one of the only free checkers that scores llms.txt at all.
Check your site's AI-readiness in 10 seconds
Run any URL through the free checker — llms.txt, schema, indexability, and 20 more signals, scored with fixes.
Run the free SEO Score check →The Bottom Line
llms.txt is a bet, not a guarantee. The standard is young, unconsumed by the major providers, and might be superseded. But the bet costs 15 minutes, the downside is zero, and the trend it rides — AI assistants becoming a real referral channel — is already visible in any site's analytics.
Write the file, serve it, verify it, and move on. It's the cheapest hedge in SEO right now.
Check your own site's llms.txt, schema, and 21 other signals with the free SEO Score checker at Value Add VC.
Latest from the Pulse
Get VC data most people never see
— 100% free
Weekly benchmarks, valuations, and fund data. Join 5,000+ investors. No spam.