llms.txt is a proposed convention for publishing an AI-readable guide to a website. The file usually lives at /llms.txt, uses Markdown-like formatting, and points readers to the most important pages on a site. Some sites also publish /llms-full.txt, a longer file that includes more complete page context.
The practical idea is simple: if an AI assistant, retrieval system, developer agent, or crawler wants a compact map of your site, llms.txt can provide one.
The careful part is just as important: llms.txt is not a proven ranking factor. It does not replace robots.txt, XML sitemaps, schema, internal links, clean HTML, or strong editorial quality. As of this draft, major LLM providers have not publicly confirmed a universal benefit for sites that publish it.
llms.txt in one minute
An llms.txt file is usually:
- Hosted at the root of a domain.
- Written in Markdown or Markdown-like text.
- Organized by sections.
- Limited to important URLs.
- Descriptive enough that an AI system can understand why each URL matters.
A simple version might include:
# Example Site
> Practical guides and reviews for AI tools.
## Core resources
- [AI visibility tools](https://example.com/reviews/best-ai-visibility-tools-2026): Compare tools that monitor how brands appear in AI answers.
- [LLM evaluation tools](https://example.com/reviews/best-llm-evaluation-tools-2026): Compare testing and evaluation platforms for LLM applications.
## Optional
- [About](https://example.com/about): Company background and editorial standards.llms.txt vs robots.txt vs sitemap
| File | Main job | Audience | Does it control access? |
|---|---|---|---|
| robots.txt | Tells compliant crawlers which paths they may crawl | Search and web crawlers | Yes, for compliant crawlers |
| XML sitemap | Lists discoverable URLs and metadata for search engines | Search engines | No |
| llms.txt | Curates important content in an AI-readable text format | LLM tools, agents, retrieval workflows, and humans | No |
| llms-full.txt | Provides longer Markdown-style context | LLM tools and retrieval workflows | No |
The easiest mental model: robots.txt is a rule file, sitemap.xml is a URL discovery file, and llms.txt is a curated reading guide.
Does llms.txt help AI search?
It may help in specific AI-readable workflows, especially where tools intentionally check for it. It can also help your own team clarify which pages matter most.
But cautious teams should avoid claims like:
- "llms.txt guarantees ChatGPT visibility."
- "llms.txt makes LLMs rank your site higher."
- "AI crawlers must obey llms.txt."
- "Publishing llms.txt replaces technical SEO."
Better wording:
- "llms.txt gives AI-oriented tools a curated map of important public pages."
- "The convention is emerging, and adoption is not yet universal."
- "It is a low-cost readiness step, not a guaranteed ranking lever."
Who should consider it first?
Documentation-heavy companies should consider it first because their content is naturally structured and often used by developer agents. SaaS companies with clear docs, API references, changelogs, and support pages are also strong fits.
Review publishers, AI tool directories, and knowledge hubs can use it to point AI-readable systems toward category hubs, comparison pages, methodology pages, and evergreen resources.
Ecommerce sites should be selective. A useful file points to category pages, buying guides, support policies, and evergreen educational content. It should not list every product variant.
Small brochure sites can still publish a short file, but the benefit may be mostly operational clarity unless AI tools in their market actively use it.
What should go into llms.txt?
Start with pages that explain what your site is, what users can do there, and where the highest-value public information lives.
Good candidates:
- Home or product overview pages.
- Documentation entry points.
- API references.
- Category hubs.
- Methodology pages.
- Comparison and review hubs.
- Pricing, security, and trust pages.
- Key tutorials and resource pages.
Poor candidates:
- Login, checkout, cart, and account pages.
- Duplicate archives and tag pages.
- Thin autogenerated pages.
- Outdated announcements.
- Private or gated pages.
- Parameterized URLs.
Implementation checklist
1. Decide whether the file should cover the full site, docs only, or a specific product area. 2. Draft a short description of the site. 3. Group links by audience or intent. 4. Add one-sentence descriptions for each link. 5. Validate that every URL returns 200. 6. Keep the file short enough to be useful. 7. Add llms-full.txt only when you can maintain the larger version. 8. Recheck the file after major IA, docs, or product changes.
What ClawNewbie should do
For a site like ClawNewbie, llms.txt should point to the strongest AI tool and AI knowledge hub assets:
- AI visibility tools.
- LLM evaluation tools.
- LLM observability tools.
- LLM gateway tools.
- AI web scraping and crawler tools.
- AI agent frameworks and platforms.
- Editorial methodology and resource templates.
That would create a cleaner bridge between the site's AI search, AI crawler, and LLM infrastructure coverage. It would not guarantee AI answer inclusion, but it would make the site's preferred entry points clearer.
FAQ
Is llms.txt official?
It is best described as a proposed and emerging convention. It has visible adoption, especially around documentation and developer sites, but it is not a universal web standard like XML sitemaps.
Can llms.txt hurt SEO?
A well-curated public file should not hurt SEO. The risk is operational: listing outdated, private, duplicate, or low-quality pages can create a poor machine-readable map.
Do I need both llms.txt and llms-full.txt?
No. Start with llms.txt. Add llms-full.txt when you want to provide a larger context bundle and have a workflow to refresh it.
How often should I update it?
Update it after major navigation changes, new category hubs, docs migrations, product launches, or important evergreen page refreshes.
---
Publisher caveat
- Maintain distinction between robots.txt, XML sitemap, llms.txt, and llms-full.txt.
- Avoid saying major LLM providers rely on the file unless Publisher adds new direct evidence.
- If quoting adoption counts from third-party pages, reverify date and source first.
Use llms.txt as a curated reading guide for important public pages, not as a replacement for technical SEO.
Before publishing, validate every URL and remove private, duplicate, gated, and low-quality pages.