Home/Free Tools/robots.txt Tester

robots.txt Tester

Test any path on any site against your robots.txt — for Googlebot, Bingbot, GPTBot, or any custom user-agent. Follows Google's official longest-match-wins rule so you see exactly which line allowed or blocked the crawl.

1 · Load robots.txt
Enter a domain — we'll fetch its /robots.txt. Or paste your own below.
2 · robots.txt content
3 · Test a URL
How rule matching works
• Longest match wins. If both Allow: /blog/private/draft and Disallow: /blog/ match, the longer Allow wins.
• On a tie, Allow beats Disallow. This is Google's documented behaviour.
• Wildcards. * matches any sequence, $ anchors to end of URL.
• User-agent matching. We pick the most specific UA block — a Googlebot block overrides the * block for Googlebot.
• No match = Allowed. Absence of any rule means the crawler is free to crawl.