Updated 2026-07-06

What is llms.txt?

TL;DR

llms.txt is a proposed plain-text file at /llms.txt that lists a site's key pages and what they're about, designed as a markdown index for LLM crawlers — the AI equivalent of what robots.txt is for crawler access rules. It is not a standard; adoption is early and its effect on citations is unproven.

Definition

llms.txt sits at the root path /llms.txt and follows a lightweight markdown convention: a heading with the project or site name, a short description, and a list of links to the pages the site owner considers most important for an LLM to know about. The intent is to give model and answer-engine crawlers a curated map of the site in a format they parse natively, rather than leaving them to discover structure from HTML.

llms.txt vs robots.txt

They solve different problems and don't replace each other. robots.txt tells crawlers what they may and may not fetch — it's an access rule. llms.txt tells crawlers which pages matter most and what they contain — it's a content hint. A site can have both: robots.txt gates access; llms.txt prioritizes and labels.

llms.txt vs sitemap.xml

Both list pages, but for different consumers. sitemap.xml is XML structured for search engine indexers and is widely supported. llms.txt is markdown structured for LLM crawlers and is far newer. A sitemap tells an indexer "these URLs exist"; an llms.txt tells an LLM "these pages matter and here's what each is about." PilotCite generates both — the sitemap from the page registry, and the llms.txt from the same source.

What llms.txt does and doesn't do

What it does: gives LLM crawlers a plain-text, curated index with descriptions, in a format that requires no parsing beyond markdown. What it doesn't do: guarantee inclusion in answers, substitute for crawlable and citable pages, or work as a ranking signal in any documented way. It's a helpful nudge, not a mechanism. The real work — answer-first content, entity consistency, crawlability — still determines whether you get cited.

Key facts

llms.txt is a proposed markdown index for LLM crawlers, not an adopted standard.
It lists and describes key pages; robots.txt gates access and sitemap.xml lists URLs.
Adoption is early and its effect on citations is unproven.
PilotCite generates llms.txt from the same page registry as its sitemap.