// free tool //
Robots.txt Tester
Paste any page URL to see whether Googlebot, Bingbot and the main AI crawlers may fetch it - with the exact robots.txt line that decides it. Uses Google's own matching rules. Free, no account needed.
How robots.txt rules are matched
robots.txt is read in groups. Each crawler obeys only the group that names its own user agent - Googlebot follows a User-agent: Googlebot group if there is one and ignores User-agent: * entirely. Inside that group the rule with the longest matching path wins, and when an Allow and a Disallow tie, Allow wins. * matches anything and a trailing $ anchors the end of the URL.
This tester applies exactly those rules to the URL you enter, for Googlebot, Bingbot and the AI crawlers GPTBot, ClaudeBot, PerplexityBot and CCBot. It shows the deciding line, highlights it in the file, lists the sitemaps the file declares, and warns about the mistakes that hurt most: blocking the whole site, blocking CSS and JavaScript Google needs to render pages, or a robots.txt that returns a server error - which Google treats as “block everything”.
Remember that robots.txt controls crawling, not indexing. To keep a page out of search results, let it be crawled and add a noindex - you can confirm that with our Noindex Checker.
One wrong line can hide a whole site. Know the day it changes.
MyKavo watches the pages that matter and alerts you when indexing signals change - with before-and-after evidence and a severity on every alert.
- Free forever plan
- No credit card
- Monitoring in minutes
// questions //
Frequently asked questions
How do I test if a URL is blocked by robots.txt?
Enter the full URL of the page. The tester fetches the site's robots.txt, finds the group each crawler obeys and applies Google's matching rules: the longest matching rule wins, allow wins a tie, and * and $ work as wildcards. It shows the result for Googlebot, Bingbot and the main AI crawlers, with the exact line that decided it.
Which rule wins when Allow and Disallow both match?
The rule with the longest matching path wins. If an Allow and a Disallow match with paths of the same length, Allow wins. A crawler also only obeys one group: the one naming its own user agent, or the * group if no group names it.
What happens if robots.txt is missing or returns an error?
A missing robots.txt (404) means nothing is blocked. A server error (5xx) is different: Google treats the whole site as blocked until the file can be fetched again, which is why an outage that only hits robots.txt can still hurt crawling.
Can I block AI crawlers like GPTBot and ClaudeBot with robots.txt?
Yes, the major AI companies publish crawler names that respect robots.txt, such as GPTBot for OpenAI and ClaudeBot for Anthropic. This tester shows whether each is currently allowed on the URL you check, so you can confirm a block is written correctly.