# web-bot-auth

## What is Web Bot Auth?

**Web Bot Auth** is a profile of the RFC 9421 (HTTP Message Signatures) standard for cryptographic identification of AI bots and crawlers. The bot signs each HTTP request with an Ed25519 private key, and the site verifies the signature using the public key from the `/.well-known/http-message-signatures-directory`.

This is fundamentally different from User-Agent checking: a User-Agent header can be spoofed; a cryptographic signature cannot.

The initiative is driven by Cloudflare and is formalized in two active IETF drafts: the protocol draft (how to sign) and the directory draft (where to publish keys). RFC 9421 was published in February 2024 as a Proposed Standard and is already stable.

## Why does a site need Web Bot Auth?

The User-Agent in robots.txt is today's only tool for distinguishing bots. It is unreliable: any scraper can claim to be "ClaudeBot" or "GPTBot."

Web Bot Auth solves three problems:

1. **Identity verification** — confirm that a request genuinely comes from OpenAI's GPTBot, not a scraper with a spoofed header
2. **Granular access control** — allow verified bots to reach paid or sensitive content while blocking unverified ones
3. **Rate-limiting by identity** — throttle a specific agent rather than by IP (bots' IPs change frequently)

As of 2026: Google publishes keys for its AI browsing agent at `agent.bot.goog`. Cloudflare has integrated Web Bot Auth into its Verified Bots program.

## How does Web Bot Auth work?

The mechanism in three steps:

**1. The bot generates an Ed25519 key pair** and publishes the public key in JWKS format:
```
GET https://bot.example.com/.well-known/http-message-signatures-directory
→ { "keys": [{ "kty": "OKP", "crv": "Ed25519", "x": "..." }] }
```

**2. The bot signs the request** with three additional headers:
```http
Signature-Agent: <https://bot.example.com/.well-known/http-message-signatures-directory>
Signature-Input: sig1=("@method" "@path" "@authority");keyid="key-1";created=1716900000;expires=1716900300;tag="web-bot-auth";alg="ed25519"
Signature: sig1=:Base64EncodedSignatureBytes==:
```

**3. The site verifies** — downloads the public key from the bot's directory and checks the signature per RFC 9421.

The `tag="web-bot-auth"` parameter distinguishes bot signatures from other uses of RFC 9421.

## How to publish a Web Bot Auth directory?

**For crawlers and agents** (you want sites to be able to verify your bot):

Generate an Ed25519 key pair and publish the public key:

```bash
# Generate key (Node.js)
node -e "
const { generateKeyPairSync } = require('crypto');
const { publicKey, privateKey } = generateKeyPairSync('ed25519');
console.log(publicKey.export({ type: 'spki', format: 'pem' }));
"
```

Publish the JWKS at `/.well-known/http-message-signatures-directory`:

```json
{
  "keys": [
    {
      "kty": "OKP",
      "crv": "Ed25519",
      "kid": "key-2026-01",
      "x": "Base64urlEncodedPublicKey"
    }
  ]
}
```

**For sites** (you want to verify incoming bots):

Use the `web-bot-auth` npm package from Cloudflare or integrate RFC 9421 verification into your middleware. With Cloudflare WAF, Verified Bots are already built into the platform.

## How do we check Web Bot Auth?

The check is **informational** (`informational: true`) — absence does not penalize the score, because the directory is needed by bots, not sites. Most sites do not need to publish a directory.

The scanner performs `GET /.well-known/http-message-signatures-directory` and verifies:

1. **HTTP 200** — the file is accessible
2. **Valid JSON** — parses without errors
3. **JWKS format** — a `keys` field is present as an array

Status **pass** — a valid JWKS with keys is published. Status **neutral** — file not found (HTTP 404) or inaccessible: this is normal for most sites. Status **neutral** — JSON found but invalid or missing the `keys` field.

[← All glossary terms](/en/glossary)