Bot Access Control
AI Bot Rules
Explicit allow rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot) in robots.txt.
What are AI Bot Rules?
AI Bot Rules are sections in robots.txt that explicitly define access rules for AI crawlers from major platforms. Each AI company uses its own User-Agent.
Current AI User-Agents:
| User-Agent | Platform |
|---|---|
GPTBot, ChatGPT-User, OAI-SearchBot | OpenAI (ChatGPT) |
ClaudeBot, Claude-Web, anthropic-ai | Anthropic (Claude) |
PerplexityBot, Perplexity-User | Perplexity AI |
Google-Extended | Google AI Overview / Gemini |
Applebot-Extended | Apple AI |
YandexAdditional | Yandex |
Why does a site need AI Bot Rules?
By default, AI bots follow the general User-agent: * rule. Explicit sections provide three advantages:
- Guarantee access even when
*is restricted or closed - Allow giving AI bots specific permissions (e.g.,
/blog/is open,/api/private/is closed) - Signal intent — an explicit trust signal for AI platforms
How to configure AI Bot Rules?
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: YandexAdditional
Allow: /Add these sections to robots.txt after the main User-agent: * block.
How do we check AI Bot Rules?
The check depends on robots_txt — data is taken from the cache of the first check.
The scanner looks in the file for sections matching known AI User-Agents: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, YandexAdditional, CCBot, Bytespider, FacebookBot.
Gradient result:
- pass (1.0) — 3 or more AI-specific sections found
- warning (0.5) — 1–2 AI-specific sections found
- fail (0.0) — only
User-agent: *with no AI-specific sections, or nothing at all
If robots.txt is unavailable or invalid — the check receives status skipped.