CrawlOut checks robots.txt, live HTTP accessibility, indexing directives and a sample of real pages against every documented crawler from Google, OpenAI and Anthropic — search, training and live-request bots evaluated separately, with full evidence.
A few common questions about search, training and live-request crawlers, and how robots.txt controls them.