Inputs
Example
example.com/robots.txt -> GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and user-triggered agents with matched rules.
AI Crawler robots.txt Checker
Check RFC 9309 robots.txt matching for documented or custom crawler product tokens with merged groups, transport evidence, and registry provenance.
Use it before publishing robots.txt changes for AI crawlers, search bots, training crawlers, or user-triggered agents.
What an AI crawler report actually tells you
robots.txt has quietly become the place where a site states its relationship with machine visitors, and that audience has changed. Alongside classic search crawlers there are now training crawlers collecting corpus data, retrieval bots answering live user questions, and user-triggered agents fetching a single page on someone's behalf. Each category reads the same file, but blocking one is a completely different decision from blocking another — cutting off search indexing costs visibility, while cutting off training crawlers is a licensing stance.
This checker evaluates robots.txt with RFC 9309 matching: it selects the most specific matching product token, merges every group carrying that same token, and uses the wildcard group only when no specific token matches. Path comparison is case-sensitive and octet-normalized; the longest rule wins, with Allow winning an equal-length tie. Every decision keeps the matched groups, winning directive and source line.
From pasted draft to publishable policy
Start with pasted content, not a live fetch. The paste path analyses the file locally in the browser, which makes it the right place for drafts that are not published yet — a policy change can be rehearsed before any crawler sees it. Add a concrete path such as / or /tools/ to test the rules against the URL shapes that matter, and pass a custom crawler list when the default set does not include the bot you care about.
Read the three categories separately before publishing. Search, training and user-triggered agents have different operational meaning, but those labels never alter RFC matching. The registry exposes its source URLs and update date, while custom product tokens are tested independently without an inferred category. A live URL fetch then reports redirects, status, media type and size separately from the parsed policy.
Matching semantics and their boundaries
Pasted robots.txt never leaves the browser; the URL mode sends only a public /robots.txt address to the same-origin helper after an explicit request. The parser accepts UTF-8 with an optional BOM and at least 500 KiB, preserves Sitemap and unknown fields separately, and applies RFC 9309 redirect, 4xx-unavailable and temporary 5xx/unreachable semantics without inventing a durable allow or block from a temporary fetch failure. robots.txt remains guidance for compliant crawlers, not access control.
Frequently asked questions
Which crawlers are in the default set?
The dated registry covers documented tokens from OpenAI, Google, Anthropic, Perplexity and Common Crawl, with source links shown as metadata. Search, training and user-triggered labels are explanatory only; any custom product token can be passed and is evaluated independently.
Does blocking a training crawler remove my content from existing models?
No. robots.txt controls future access by compliant crawlers. Content collected before the block, or gathered by operators that ignore the convention, is outside what this file can influence.
Why does the report separate Search, Training, and User-triggered categories?
Because they answer different business questions. Blocking search indexing trades away visibility, blocking training is a data-licensing position, and blocking user-triggered agents affects people asking an assistant about your site right now.
Why does a rule I expected to match not apply?
A crawler selects the most specific matching product token and merges all groups with that same token. Nonmatching groups, including a wildcard group when a specific token matched, are ignored. Within the merged rules, the longest octet match wins and Allow wins an equal-length tie.
Can I test rules for a path that is not deployed yet?
Yes. Path evaluation is string matching against the pasted or fetched rules, so a planned URL layout can be tested before the first page exists.
Is a missing robots.txt a problem?
A missing file simply means everything is allowed for compliant crawlers. That is a legitimate policy — it only becomes a problem when it is an accident rather than a decision, which is exactly what a deliberate check makes visible.