Subu’s Robots.txt Interrogator
Enter a domain. I will fetch the robots.txt file, judge its syntax, and let you test if specific bots are being blocked from crawling.
File Status
Pending
Sitemaps Declared
0
Rules Found
0
Audit Log
Can Google Crawl This URL?
Test a specific URL path against the rules fetched above.