# ai-bot-rules

## What are AI Bot Rules?

**AI Bot Rules** are sections in [robots.txt](/glossary/robots-txt) that explicitly define access rules for AI crawlers from major platforms. Each AI company uses its own User-Agent.

Current AI User-Agents:

| User-Agent | Platform |
|---|---|
| `GPTBot`, `ChatGPT-User`, `OAI-SearchBot` | OpenAI (ChatGPT) |
| `ClaudeBot`, `Claude-Web`, `anthropic-ai` | Anthropic (Claude) |
| `PerplexityBot`, `Perplexity-User` | Perplexity AI |
| `Google-Extended` | Google AI Overview / Gemini |
| `Applebot-Extended` | Apple AI |
| `YandexAdditional` | Yandex |

## Why does a site need AI Bot Rules?

By default, AI bots follow the general `User-agent: *` rule. Explicit sections provide three advantages:

1. Guarantee access even when `*` is restricted or closed
2. Allow giving AI bots specific permissions (e.g., `/blog/` is open, `/api/private/` is closed)
3. Signal intent — an explicit trust signal for AI platforms

## How to configure AI Bot Rules?

```
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: YandexAdditional
Allow: /
```

Add these sections to `robots.txt` after the main `User-agent: *` block.

## How do we check AI Bot Rules?

The check depends on `robots_txt` — data is taken from the cache of the first check.

The scanner looks in the file for sections matching known AI User-Agents: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, YandexAdditional, CCBot, Bytespider, FacebookBot.

Gradient result:
- **pass (1.0)** — 3 or more AI-specific sections found
- **warning (0.5)** — 1–2 AI-specific sections found
- **fail (0.0)** — only `User-agent: *` with no AI-specific sections, or nothing at all

If `robots.txt` is unavailable or invalid — the check receives status `skipped`.

[← All glossary terms](/en/glossary)