llms.txt Validator
Checks title, summary, sections, links, optional entries, and source-line findings in pasted or published files.
Web / SEOllms.txt in Practice: Curating a Useful Entry Point
llms.txt is a compact reading guide for language-model agents, not a new access-control layer and not a sitemap with fashionable branding. Its value comes from editorial restraint: name the pages that explain the site, keep the file coherent with published policy, and make every link earn its context cost.
A convention with deliberately narrow ambitions
The useful mental model is a README at the root of a public site. Formally, the current proposal requires an initial H1. An optional blockquote summary and ordinary Markdown prose or lists may follow before any H2. H2 sections conventionally contain Markdown link-list items. A compact, link-centric file is an editorial best practice because it reduces discovery guesswork; it is not a rule that makes valid pre-H2 prose invalid.
The file does not grant permission, compel retrieval, improve ranking by itself, or guarantee that an answer cites your site. Operators choose whether to fetch or honor it. It cannot retract content already collected, bypass authentication, or replace accurate pages. Think of publication as an offer: “If you need a compact map of this site, start here.” The llms.txt Validator can remove structural reasons to reject that offer, but no validator can make an external system accept it.
llms.txt and llms-full.txt solve different budgets
The short file should help an agent decide what to load next. Put the canonical overview, major product or documentation sections, high-value reference pages, and a modest Optional section there. Optional means exactly that: background, secondary references, and long-form material an agent may skip when context is scarce. A link list containing every localized page, filter state, changelog entry, and legal duplicate has ceased to curate. It makes the consumer spend tokens reconstructing the priorities you declined to express.
llms-full.txt is the companion for sites that can usefully provide a larger, consolidated body of documentation. It serves a reader prepared to ingest more context; it should not be a blind concatenation of rendered HTML, menus, cookie banners, and repeated footers. Keep stable headings, canonical terminology, explicit scope and limitations, and enough source structure to locate a claim. The short file may link to the full one, but do not make the large document the only route to basic orientation. A constrained agent should still succeed after reading only llms.txt.
Keep the three public maps consistent
robots.txt, sitemap.xml, and llms.txt answer separate questions. Robots states access policy for compliant crawlers. A sitemap enumerates indexable URLs. The language-model file recommends what to read first. None substitutes for another, yet contradictions are operational bugs. Recommending a page that your relevant crawler group blocks is self-defeating. Listing redirected, removed, or non-canonical URLs teaches consumers stale routes. Omitting a whole public section from the sitemap while declaring it essential in llms.txt may be intentional, but it deserves an explanation rather than accidental drift.
Make the files derive from the same canonical route inventory while preserving their different selection rules. Check language alternates, trailing slashes, redirects, and case exactly as served. Use the AI Crawler robots.txt Checker when policy is in doubt: it evaluates pasted rules or a public robots URL for a chosen path, shows the matched group, winning directive, and source line, and separates search, training, and user-triggered roles. Remember that compliant-crawler policy is still instruction, not enforcement.
Validate the draft and the wire separately
Draft locally, then validate the pasted Markdown before publication. Separate formal validity from editorial quality: require the initial H1, preserve optional summary and project details before the first H2, and parse H2 resource sections without pretending that every prose block is a link. Common quality problems are empty resource sections, relative targets that resolve somewhere unintended, duplicate canonicals, stale anchors, and link labels that no longer describe their destinations. A formally valid file can still be unhelpful; a validator cannot decide whether twenty nearly identical links are useful.
After deployment, fetch the public /llms.txt and validate that response rather than trusting the repository artifact. Redirect handling, a wrong content variant, CDN caching, an HTML error page returned with status 200, or a build that published yesterday's file all live between source and wire. The validator's URL mode uses a guarded same-origin helper for public HTTP(S) targets ending in /llms.txt, with restricted ports, no redirects, timeouts, and bounded size. That is a focused deployment check, not a general crawler simulation.
Design for how agents actually spend context
An agent that reads the file still makes economic choices. It may load only the first relevant section, follow two links, summarize the result, and stop. Descriptive link labels and short notes therefore matter more than volume. Name the artifact and the decision it supports: “API authentication and error model” beats “Developer docs.” Put prerequisites before specialist references. Avoid claims that exist only in the inventory; the linked page remains the authoritative content and must be understandable when opened alone.
Treat changes as contract maintenance. When a route is renamed, update the sitemap, redirects, navigation, llms.txt, and any fuller document together. When a new website function becomes callable through an agent-facing interface, expose its human documentation as well; the WebMCP Tool Validator can inspect tool metadata, but it does not repair a missing explanatory page. Review access logs cautiously if you want evidence of adoption: fetching the file shows interest, not that a model used, trusted, or cited its contents.