llms.txt is a Markdown file at /llms.txt on your website that tells AI models what your site is about, which pages matter most, and where to find them. It takes 5 minutes to create. Whether you need one depends on your content type: documentation-heavy sites benefit most, marketing sites less so.
The file is a convention, not a web standard. No AI search engine officially reads it as a retrieval input. But Anthropic, Stripe, Cloudflare, and thousands of developer docs sites use one, and Google quietly added their own despite John Mueller calling it "unnecessary."
How llms.txt Works
llms.txt is a Markdown file at your domain root (yourdomain.com/llms.txt) that lists your most important pages with one-line descriptions. Think of it as a curated sitemap written in plain language that AI models can read directly. The only required element is an H1 heading with your site name. Everything else, page links organized by topic, a short summary, an optional llms-full.txt file that concatenates your key content into one document, is optional but useful.
The format takes 30 minutes to create. Here is how to build one for your site:
- Create a Markdown file with your company name as the H1 heading.
- Add a one-line summary as a blockquote explaining what your product does.
- List your 10 to 20 most important pages organized under H2 headings by topic (Documentation, Key Resources, etc.), each with a link and a one-sentence description.
- Host the file at your domain root:
yourdomain.com/llms.txt. - Optionally create
llms-full.txtthat concatenates the actual content of those pages into a single file. This is useful for coding agents that can ingest an entire documentation set in one context window.
Should You Adopt llms.txt?
The decision depends on what kind of content you publish and who is consuming it. A survey of 300,000 domains found a 10.13% adoption rate, skewing heavily toward developer tools and documentation sites. Marketing sites and content publishers have adopted at much lower rates, because the benefits are different for each.
Adopt it if your product has developer documentation, API references, or technical guides. If your users interact with your content through coding agents like Cursor or Windsurf, llms.txt gives those tools a clean overview of your docs without parsing HTML. If you use a documentation platform like Mintlify or GitBook, the file is generated automatically with zero effort.
Skip it if your site is primarily marketing pages or blog content and your goal is to get cited in AI search results. llms.txt does not influence how ChatGPT, Perplexity, or other AI search engines retrieve and rank content during web search. For AI search visibility, your time is better spent on content structure and third-party presence.
The cost-benefit calculation: creating a basic llms.txt takes 30 minutes. The downside risk of having one is near zero. If you are unsure, create it and move on to the work that actually moves AI search visibility.
The Case Against llms.txt
Check your server logs for requests to /llms.txt from AI crawlers (GPTBot, ClaudeBot, PerplexityBot). You will likely find minimal or no activity. That is the strongest argument against investing time in the format: the bots that power AI search engines are crawling your actual pages, not reading your llms.txt summary.
Google's John Mueller made this point explicitly in June 2025, comparing llms.txt to the deprecated keywords meta tag. In May 2026, Google went further: their official AI optimization guide explicitly listed llms.txt as one of several "AEO hacks" that do not work, alongside content chunking for AI, rewriting specifically for AI systems, and pursuing inauthentic mentions. No AI search engine has committed to using it as a first-class retrieval signal. If your goal is to get cited in AI search results, llms.txt is not the path. Focus on the content structure and freshness signals that AI search engines actually use in their retrieval pipelines.
The Case For llms.txt
The counterargument is that llms.txt serves a different use case than Mueller's criticism addresses. It is not primarily about getting cited in AI search results. It is about making your content accessible to AI coding agents, developer workflows, and context-window-based interactions where a user pastes a URL directly into a conversation.
Where llms.txt demonstrably helps
AI coding agents. Tools like Cursor, Windsurf, and GitHub Copilot can consume llms.txt to understand a project's documentation structure. When a developer working with your API pastes your llms.txt URL into their coding agent, the agent gets a clean overview of available documentation without parsing HTML.
Direct URL pasting. When users paste an llms.txt URL directly into ChatGPT, Claude, Gemini, Perplexity, or Grok, these tools read the file and use it effectively. The models parse Markdown natively, so llms.txt provides a cleaner input than an HTML page.
Documentation-heavy sites. If your product relies on extensive documentation (APIs, SDKs, integration guides), llms.txt helps AI models understand the structure of your docs without crawling every page.
Reduced hallucination. Early adopters report that providing context via llms.txt and llms-full.txt reduces hallucination in AI-generated answers about their products, because the model has access to authoritative source material rather than relying on training data or web search snippets.
Google added an llms.txt file to its Search Central documentation portal despite Mueller's public criticism. However, Google's May 2026 AI optimization guide explicitly lists llms.txt among approaches that do not influence AI search visibility, suggesting the file serves Google's developer documentation use case rather than validating llms.txt as an AI search signal.
How to Create an llms.txt File
The easiest way to create an llms.txt file is to ask ChatGPT to do it for you. Give it this prompt:
"Go to [yourdomain.com] and create an llms.txt file for my website. Include a one-line description of the company, then list the 10-20 most important pages with one-sentence descriptions for each. Format it as Markdown with an H1 heading of the company name."
ChatGPT can access your website, read your pages, and generate the file in seconds. Copy the output, save it as llms.txt, and upload it to your domain root. Done.
If you prefer to do it manually, here is a basic template:
Basic template
# Your Company Name
> Brief description of your company and what your product does.
## Documentation
- [Getting Started Guide](https://yourdomain.com/docs/getting-started): How to set up and configure the product
- [API Reference](https://yourdomain.com/docs/api): Complete API documentation with endpoints and examples
- [Integration Guide](https://yourdomain.com/docs/integrations): How to connect with third-party tools
## Key Resources
- [Pricing](https://yourdomain.com/pricing): Current plans and pricing
- [Changelog](https://yourdomain.com/changelog): Recent product updates and releases
- [Blog](https://yourdomain.com/blog): Industry analysis and product insights
Implementation guidelines
- Keep descriptions to one sentence per link
- Prioritize your 10 to 20 most important pages, not your entire sitemap
- Update the file when you add or remove significant content
- If Google indexes your llms.txt, consider adding a noindex meta tag or X-Robots-Tag header as Google recommends
- Optionally create an llms-full.txt that concatenates the actual content of your linked pages
What llms.txt Does Not Replace
llms.txt is not a substitute for the structural content optimization that drives AI search citations. AI search engines like ChatGPT, Perplexity, and Gemini already have the ability to access and crawl your website directly. They do not need llms.txt to find your content. Their crawlers, retrieval systems, and ranking algorithms operate independently of llms.txt. The file is a convenience for AI agents and coding tools that want a curated summary, not a requirement for AI search visibility.
Think of llms.txt as a convenience layer for AI agents and direct model interactions, not as an AI search optimization tool. The two objectives (AI agent accessibility and AI search visibility) require different approaches, and conflating them leads to wasted effort.
Frequently Asked Questions
Does llms.txt help with AI search rankings?
No. As of May 2026, no major AI search engine (ChatGPT, Perplexity, Gemini, Claude, Grok) uses llms.txt as a retrieval signal for generating search answers. Google's official AI optimization guide (May 2026) explicitly lists llms.txt among approaches that do not work for AI search visibility. AI search engines use their own crawlers and retrieval pipelines. llms.txt helps with direct model interactions and AI coding agents, not with search visibility.
Is llms.txt an official web standard?
llms.txt is a community-proposed convention, not a ratified IETF or W3C standard. It was proposed by Jeremy Howard of Answer.AI in 2024 and has gained adoption through community momentum and platform support (particularly Mintlify), but it has no formal standards body backing.
Should I noindex my llms.txt file?
Google recommends noindexing llms.txt files to avoid cluttering their search index. Since llms.txt is designed for AI models, not search engine results pages, adding an X-Robots-Tag: noindex header or a meta robots noindex directive prevents unnecessary indexing without affecting the file's utility for AI agents.
How often should I update llms.txt?
Update llms.txt when you add or remove significant content from your site, such as new documentation sections, deprecated features, or major product changes. Monthly reviews are reasonable. Unlike blog content, which benefits from 30-day freshness signals, llms.txt changes infrequently because it describes your site's structure rather than time-sensitive information.
Can llms.txt hurt my site?
There are no known negative effects of having an llms.txt file. The worst case is that AI systems ignore it, which is the current default behavior for most AI search engine crawlers. If Google indexes the file and it shows up in search results, noindexing it resolves that issue.
Updated for April 2026: Replaced spec documentation with a step-by-step creation guide, replaced adoption list with a decision framework, shortened technical criticism sections.