lms.txt is a simple Markdown file you place at your domain root at yourdomain.com/llms.txt, that tells AI crawlers which pages on your site actually matter. So what is llms.txt, exactly? Think of it as a curated map for AI systems, the way robots.txt is a map for search engine bots. Except the two work very differently, and confusing one for the other is costing websites AI visibility right now.
The spec was proposed on September 3, 2024 by Jeremy Howard, founder of FastAI and Answer.AI. By May 2026, it’s been adopted by Anthropic, Stripe, Cursor, Cloudflare, Vercel, Mintlify, Supabase, and LangGraph. And roughly 10% of all websites have one. The other 90% don’t know what they’re missing yet.
Why does this file exist?
Search engines and AI crawlers handle your website completely differently.
Google’s bot indexes your pages over time, builds a persistent database, and retrieves from it when someone searches. AI crawlers don’t work that way. They access content on demand, at the exact moment a user asks a question. They work within tight context windows. And they struggle with JavaScript-heavy pages, cluttered HTML, tracking scripts, and navigation menus that eat up space without adding information.
The result: a lot of your best content never makes it into an AI’s answer, because the model couldn’t cleanly parse it or ran out of context window before reaching the good stuff. This is the same reason content fails to surface in traditional search and it’s even worse with AI.
The file solves this by giving AI systems a clean, human-curated shortlist: your most important pages, what each covers, no clutter. It’s the foundation of what’s now called generative engine optimization (GEO), optimizing not for Google’s index, but for how AI systems find and cite your content.
May 2026 fact-
By May 2026, the spec has been adopted by Anthropic, Stripe, Cursor, Cloudflare, Vercel, Mintlify, Supabase, and LangGraph. A study of 300,000 domains by SE Ranking found 10.13% adoption across the web overall. Growing fast in developer and SaaS sectors. Financial services and healthcare are barely moving.
What does an llms.txt file actually look like?
It’s a Markdown file. Four parts, nothing fancy.
A single H1 with your brand name. A blockquote: a one-paragraph summary of what your site does. Then H2 sections for each content category, each with bullet-point links and a one-line description per page. That’s the whole spec.
No special syntax. No configuration. A developer can build one in a couple of hours. There’s also a companion: llms-full.txt, a complete text export of your entire site in one Markdown document. Used mainly for developer docs and structured data ingestion. Most sites only need the standard version.
Real example-
In December 2025, Google quietly added the file across many of its developer and documentation properties, which looked like a clear signal the standard had arrived. Then pulled it within 24 hours. Search Engine Land confirmed Google’s John Mueller called it an accidental sitewide CMS update. Google has not officially endorsed the standard as of May 2026. But the near-miss triggered a fresh wave of implementations anyway.
Who’s actually using it in 2026?
Adoption follows a classic diffusion curve. Early movers were tech-native companies: cybersecurity firms, developer tool companies, blockchain projects. By mid-2025 it was routine among developer-facing SaaS. By Q1 2026 it had spread into mainstream SaaS, publishing, and some consumer brands.
Financial services, healthcare, and legal sectors are largely sitting it out. Compliance teams want a formal standard before approving publication. And building authority with AI platforms is still a newer concept than building it with Google.
As of April 2026, there’s still no formal IETF RFC. The spec is community-managed at llmstxt.org. That’s slowing enterprise adoption. But core syntax is stable enough that a well-formed file works across every major AI platform that supports it.
“The file is a low-cost, low-yield bet with clear optionality. Ship it. Don’t expect your AI traffic to spike overnight.”
Does llms.txt for SEO actually move the needle?
This is where the honest answer gets complicated.
Search Engine Land tracked 10 sites across finance, B2B SaaS, ecommerce, insurance, and pet care for 90 days before and after implementation. They measured AI crawl frequency and traffic from ChatGPT, Claude, Perplexity, and Gemini.
8 out of 10 sites saw no measurable change. 1 declined 19.7% in AI traffic. 2 saw increases of 12.5% and 25%, but both gains were attributed to other work: a PR campaign, restructured product pages with better comparison tables. The file itself wasn’t the driver.
Study data (January 2026)-
ALLMO.ai analyzed 120 AI-cited websites across multiple industries. Sites with the file showed no statistically significant difference in AI citation frequency compared to those without it. The file appeared in less than 1% of cited sources, despite roughly 10% of websites having one. Content quality and structure still do the heavy lifting.
So why are serious companies shipping it anyway? Because the implementation cost is half a day, the IDE and agent ecosystem already reads it actively, and the downside of not having one when a major LLM starts fully respecting it is real. It’s optionality, not guaranteed ROI.
llms.txt vs robots.txt: different jobs entirely
robots.txt controls access: which URLs AI crawlers are allowed to visit. The file adds editorial curation on top: which pages are most worth reading, with context about what each contains. They coexist. You need both. And if you block AI crawlers in your robots.txt, don’t include those URLs here either.
Should you add llms.txt to your site?
If you run a content site, SaaS product, or developer tool: yes. If you’re already thinking about how AI systems find and use your content, this is the next logical step. Start at the official spec and you can have a working file live by end of day.
If you run a small local business or personal blog: wait. A well-structured page with clear headings will do more for your AI visibility right now.
The biggest outstanding question is Google. If Google formally supports the standard for AI Overviews, adoption goes from 10% to near-universal almost overnight. That decision hasn’t been made as of May 2026. But the moment it is, you’ll want to already be in place.
Half a day of work now. Potentially significant leverage later.
Frequently Asked Questions:
What is llms.txt and who created it?
llms.txt is a Markdown file placed at your domain root that tells AI systems which pages on your site are most important. It was proposed by Jeremy Howard, founder of FastAI and Answer.AI, on September 3, 2024. The official spec is maintained at llmstxt.org.
Does llms.txt actually improve your AI search visibility?
Current data says not reliably on its own. An SE Ranking study of 300,000 domains and a Search Engine Land 90-day tracking study both found no statistically significant improvement in AI citations from having the file. Content quality remains the primary driver. That said, the cost of implementation is low and the upside is real if major AI platforms increase their support for the standard.
What’s the difference between llms.txt and robots.txt?
robots.txt controls which URLs AI crawlers can access. llms.txt adds editorial guidance on top: it tells AI systems which pages are most worth reading and provides a short description of each. They serve different purposes and you need both. If a URL is blocked in robots.txt, don’t include it in your llms.txt file.
