agentsready
Bot Access Control Easy

Web Bot Auth: how a bot cryptographically proves who it is

What Web Bot Auth and HTTP Message Signatures (RFC 9421) are, why agents sign requests, an example, mistakes, and how to verify.

Updated:

What it is

Web Bot Auth is how a bot cryptographically proves its identity. The request is signed per RFC 9421 (HTTP Message Signatures), and the site publishes its public keys in a directory at a well-known path (/.well-known/http-message-signatures-directory, JWKS format). A verifier checks the signature against the published key — and sees the bot is genuine, not a User-Agent spoof.

Why it matters for AI agents

A User-Agent is trivial to fake: malware pretends to be “GPTBot.” A signature can’t be forged without the private key. This lets sites admit verified agents and block impostors — opening content to “good” bots without the risk. For an agent operator it’s a pass into places where anonymous bots get cut off.

Minimal working example

The bot signs the request (Ed25519) and sends headers:

GET /data HTTP/1.1
Host: example.com
Signature-Input: sig1=("@method" "@authority" "@path");keyid="agent-key-1";alg="ed25519"
Signature: sig1=:MEUCIQ...base64...:

The site owner publishes its public keys:

GET /.well-known/http-message-signatures-directory HTTP/1.1
{ "keys": [ { "kty": "OKP", "crv": "Ed25519", "x": "...", "kid": "scanner-key-1" } ] }

Right vs wrong

RightWrong
Signature per RFC 9421, Ed25519 keyA home-grown signing scheme
A JWKS directory at the well-known pathKeys published nowhere — the signature can’t be checked
keyid in the signature points to a key in the JWKSkeyid doesn’t match the published key
Key components covered (@method, @path…)Only an empty set signed — meaningless

Common mistakes

  • JWKS unreachable (404) — the verifier can’t validate the signature.
  • keyid in Signature-Input doesn’t match a key in the directory.
  • Keys out of sync after rotation — the old key was removed, signatures break.
  • Too narrow a set of components signed — the signature doesn’t protect the request.

How to verify

A scan checks for a valid JWKS directory. Manually:

curl -s https://example.com/.well-known/http-message-signatures-directory | jq .

It should return a valid JWKS with a keys array (Ed25519, with kid).

Sources