AI crawler audit
See how your website responds to AI crawlers.
What it checks
- robots.txt, crawler by crawler
- Responses to test requests
- Training versus answering
Questions
Which crawlers are checked?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Bytespider and Amazonbot. Google-Extended and Applebot-Extended are checked as robots directives only; they do not have separate crawler user agents.
Does this tell me whether AI crawlers have visited?
No. Only server or edge logs can establish actual visits. This checks detected robots rules and responses to requests from Rankveo. It does not authenticate as an engine's crawler. A workspace can ingest supported edge logs to help you inspect visits.
Will the audit trip my bot protection?
It can. The audit sends about a dozen requests from Rankveo, a few at a time, using different crawler user agents. Bot protection may treat these differently from verified crawlers, so a challenge is a result for our probe, not proof that the real engine is blocked.