Why this signal affects whether AI cites you.
AI systems can only cite content their crawlers are actually allowed to fetch. Many sites unknowingly block the exact bots that feed AI answers — either through an overly aggressive robots.txt, a security or WAF rule, or a blanket 'block all bots' setting inherited from an SEO plugin. The result is invisible: the page ranks fine in Google, looks healthy in analytics, and is completely unreachable by the assistant a customer is asking. Crawler access is the foundational, layer-one factor in the AI Visibility Stack because nothing downstream — schema, structured answers, authority — can matter if the bot never reaches the page. Unlike most optimizations, this one is binary and immediate: an allow rule either exists or it does not, and flipping it can take a page from uncitable to citable in a single deploy. The nuance is that 'allow everything' is not the goal either; publishers reasonably gate training crawlers while permitting the retrieval and search crawlers that drive live citations. Getting this right means making a deliberate, per-bot decision rather than inheriting a default — and auditing it regularly, because new AI user-agents appear faster than most robots.txt files are updated. Treat the allow list as living infrastructure: review it whenever you ship a new AI feature, change CDN or hosting providers, or notice that a platform you care about has launched its own crawler under a new user-agent string.
How to get this right.
- An explicit robots.txt Allow directive for GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot rather than a blanket disallow.
- A WAF or Cloudflare rule audited so it does not silently challenge or block known AI retrieval user-agents.
- A deliberate split that permits live retrieval and search crawlers while gating training-only crawlers via Google-Extended.
Common mistakes.
- Inheriting a 'block all bots' default from an SEO or security plugin and never auditing which AI crawlers it catches.
- Blocking retrieval crawlers that drive live citations when you only meant to opt out of model training.
You now know the signal. See your score.
This page covers what AI Crawler Access is and how to get it right. AIVZ measures it on your actual pages — across six AI platforms, weighted and prioritized against all 93 factors — and hands you the exact fixes in priority order.
Scan your site against all 93 factorsFrequently asked.
Should I allow every AI crawler?
- Not necessarily. A common approach permits retrieval and search crawlers that drive live citations while gating training-only crawlers. The key is making a deliberate per-bot decision instead of inheriting a blanket block.
My page ranks on Google — isn't it already accessible to AI?
- Not always. Google ranking and AI-crawler access are separate. Bots like GPTBot or PerplexityBot can be blocked even when Googlebot is allowed, leaving the page invisible to those assistants.
Sources
Signals that work together.
Last updated · See changelog
The fastest way to evaluate fit.
The free scan is the canonical demo of what AIVZ does. Run a scan on your own site, your competitor's, or a prospect's.