Robots.txt generator & validator
Verified against current specsResults appear here.
What the output means
The tester shows which group applies to the selected crawler, which rule matched, and the resulting Allow or Disallow decision. It is a Googlebot-oriented simulation built on RFC 9309; other crawlers may behave differently.
How to use this tool
- Write or paste a robots.txt, or start from a template.
- Enter the user-agent and a URL path to test.
- Read the matched group, rule, and final decision.
- Export the file as UTF-8 robots.txt.
How the check works
The parser resolves groups by user-agent, supports * and $ wildcards, percent-encoding, and comments. When Allow and Disallow both match, the longest rule wins; on equal length, Allow wins. A missing match defaults to Allow. /robots.txt is always implicitly allowed.
Worked example
Rules: Disallow: /private/ and Allow: /private/public/. Testing /private/public/page.html is Allowed (longer Allow wins). Testing /private/account.html is Blocked.
What this tool cannot tell you
robots.txt is not a security or privacy control. It does not guarantee a URL is removed from search results, and Google does not treat crawl-delay or noindex as valid robots.txt directives.
Common questions
Does robots.txt block indexing?
No. It only controls crawling. To keep a page out of results, use noindex and keep it crawlable.
Why is my Allow not working?
The most specific (longest) matching rule wins. Check rule order and wildcard placement.
Official references
- RFC 9309 - Robots Exclusion Protocol — verified 2026-11-01
- Google: Robots.txt intro — verified 2026-11-01
Related tools
Was this tool helpful? Tell us what to improve.
