AI Crawler robots.txt Checker
Pair AI-readable route inventory with crawler access rules for the same site.
Web / SEOMaking Your Site Agent-Readable: robots.txt, llms.txt, Agent Cards, and WebMCP
A growing share of your traffic reads metadata before content: AI crawlers checking robots policy, agents looking for llms.txt, runtimes probing for tool manifests. Publishing this layer well is cheap; publishing it dishonestly is worse than not publishing it at all.
Crawler policy first: robots.txt in the AI era
The oldest file on this list suddenly has new readers. Training crawlers, search-preview fetchers, and user-triggered agent browsers each identify with their own user agents, and RFC 9309 matching rules decide what your directives actually permit — longest-path matching, group merging, and the wildcard semantics that most hand-written files get subtly wrong. A policy you believe blocks AI training may match nothing, and a broad block may be starving the user-triggered fetchers you actually want.
Decide the policy deliberately — training crawlers, search crawlers, and on-demand agent fetchers are different populations with different value exchanges — and then verify the file says what you decided. The AI Crawler robots.txt Checker evaluates your rules against the known AI and search crawler population with RFC 9309 matching evidence, which beats squinting at wildcard rules every time.
llms.txt: navigation for readers without a DOM
Where robots.txt says whether to read, llms.txt says what is worth reading: a Markdown file at the site root listing your important routes with one-line descriptions, so an agent with a small context budget lands on substance instead of crawling navigation chrome. The format is deliberately modest — a title, a short summary, link sections with descriptions — and its value stands or falls with honesty and freshness. Stale entries and dead links teach agents the same lesson broken sitemaps taught crawlers.
Treat it like the sitemap's editorial sibling: generated from the same route inventory that builds your sitemap, updated by the same deploy that changes routes, and validated rather than assumed correct — the llms.txt Validator checks structure, sections, and link health. Keep parity with your canonical URLs, and resist padding: a curated twenty-line file serves an agent better than an exhaustive dump.
Agent cards: identity you can be held to
The A2A ecosystem moves discovery from pages to services: an agent card at /.well-known/agent-card.json declares identity, skills, endpoints, transports, and authentication requirements, and consuming agents parse it mechanically. The card is a contract — every field is something another system may rely on, which cuts both ways. Skill descriptions are read by selecting models, so vague descriptions are functionally broken links; declared auth schemes must match the deployment exactly, or every consumer pays a confusing failed round-trip.
The integrity rule matters more than any field: never advertise what does not exist. No OAuth metadata without a real protected resource behind it, no skills the deployment cannot serve, no internal hostnames or secrets in a file that is public by definition. Validate the card as served — from the live well-known URL, where redirects, caches, and content-type rewrites live — with the A2A Agent Card Validator.
WebMCP: tools in the page itself
WebMCP closes the loop by letting a page register callable tools with the browser's agent runtime — name, description, JSON-schema parameters, and an execution path — so an agent can run your site's functionality directly instead of scraping it. The craft transfers directly from API design: precise names, descriptions written for a model that must choose between tools, parameters that validate strictly, and honest read-only annotations where they apply.
Before shipping, lint the contracts with the WebMCP Tool Validator — names, descriptions, parameter schemas, labels, and the security hints agents increasingly check. And round out the layer with the boring, load-bearing files: a current security.txt Validator/Generator so findings reach you, and consistent canonicals so every reader — human, crawler, or agent — converges on the same URLs. Agent-readability is not one file; it is the discipline of keeping all of them telling the same true story.