Comparison
AI bot rules vs Content Signals: access or usage
AI bot rules decide whether a bot may crawl; Content Signals decide whether content may be used for training/search/answers. Different questions.
In short
Two different questions in one robots.txt: AI bot rules are about access
(may it crawl), Content Signals are about usage (may it train, search, use
in AI answers). Not a choice — both.
What each is
- AI bot rules —
User-agentsections for specific AI crawlers (GPTBot, ClaudeBot…):Allow/Disallow. Control over crawling. - Content Signals — a
Content-Signaldirective (search,ai-input,ai-train). Control over content usage.
Side by side
| AI bot rules | Content Signals | |
|---|---|---|
| Question | May it crawl? | May it be used? |
| Mechanism | User-agent + Allow/Disallow | Content-Signal: ai-train=… |
| Granularity | By bot and path | By usage type |
Key difference
Access ≠ permission. You can allow a bot (Allow) but forbid training on
the content (ai-train=no). AI bot rules govern who comes in; Content Signals
govern what may be done with what they read.
Do both
Yes, and it gives nuance: “allow GPTBot for search and AI answers, but not for model training.” Visibility in GEO without feeding content to training datasets.
How to verify
A scan parses both the AI sections and the Content-Signal from robots.txt — both
sides in one report.