All toolsSitemap & Robots.txt Checker
Make sure both files agree
CrawlOut fetches your robots.txt and every sitemap it references, validates both against the real specs, spot-checks a sample of listed URLs for reachability, and flags any page your sitemap tells crawlers to index while robots.txt tells them not to fetch.
robots.txt + sitemap.xml
robots.txt found
Sitemap validated
No blocked URLs indexed
No Conflictssitemap ↔ robots.txt
Fully Crawlablelive URL sample
Checks Both Filesrobots.txt and every sitemap it references, together.
Live ReachabilitySamples real URLs from your sitemap and fetches them.
Conflict DetectionFlags pages your sitemap lists but robots.txt blocks.
Full Spec ValidationChecked against the real robots.txt and sitemap specs.
Learn more
Understanding robots.txt & sitemap health
A few common questions about how these two files work together, and what CrawlOut checks for.
Build a robots.txt