AI crawler audit
See how your website responds to AI crawlers.
About this check
What's included
robots.txt, crawler by crawler
Each listed AI user agent is checked against the detected robots.txt rules for your homepage, including wildcard groups and the matching rule.
Responses to test requests
Rankveo requests your homepage using crawler user-agent strings and reports HTTP errors, restrictions and rate limits. Verified engine IPs can receive different responses.
Training versus answering
Training, search and user-requested agents have different roles. Review each policy separately; a successful probe does not guarantee an AI citation.
Common questions
Which crawlers are checked?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, meta-externalagent, Bytespider and Amazonbot. Google-Extended and Applebot-Extended are checked as robots directives only; they do not have separate crawler user agents.
Does this tell me whether AI crawlers have visited?
No. Only server or edge logs can establish actual visits. This checks detected robots rules and responses to requests from Rankveo. It does not authenticate as an engine's crawler. A workspace can ingest supported edge logs to help you inspect visits.
Will the audit trip my bot protection?
It can. The audit sends about a dozen requests from Rankveo, a few at a time, using different crawler user agents. Bot protection may treat these differently from verified crawlers, so a challenge is a result for our probe, not proof that the real engine is blocked.