Robots.txt tester
Test and validate robots.txt files for crawlers
Your robots.txt
Test a Path
Note: This tool parses robots.txt files and tests whether specific URL paths would be allowed or blocked for different user agents. It follows the basic robots.txt specification, checking Allow and Disallow directives with path prefix matching.
About this tool
Use it to
- Test if URLs are blocked by robots.txt rules
- Validate robots.txt syntax and structure
- Check crawler access for different user agents
- Debug SEO issues related to robots.txt
- Test Admin Pages
- Verify that /admin/ and /private/ paths are properly blocked from all crawlers using Disallow rules.
- Googlebot Access
- Test specific URLs to ensure Googlebot can access your important pages and is only blocked from temporary content.
Related tools
- Favicon tester
Check what favicons a website has and what devices they support
- Sitemap.xml validator
Validate and analyze XML sitemap files
- JSON-LD Validator
Validate and test JSON-LD structured data for SEO
- SEO Checker
Analyze SEO elements of any webpage including meta tags, headings, and more