Markdown for agents: content negotiation for LLMs

What it is

“Markdown for agents” means serving a clean Markdown version of a page when the client asks for it with the Accept: text/markdown header. Humans get HTML, agents get Markdown. It’s done via content negotiation (one URL, different format) or via a stable parallel .md address for each page.

Why it matters for AI agents

HTML is noise to an LLM: navigation, scripts, markup, banners. The model spends tokens and context on junk and extracts the substance less reliably. Markdown is dense and clean: fewer tokens, sharper comprehension, a better chance of an accurate citation in an AI answer.

Minimal working example

The server inspects the Accept header and serves the right format:

GET /guides/llms-txt HTTP/1.1
Accept: text/markdown

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

# How to write llms.txt

> In brief: ...

A simpler alternative: publish a static /guides/llms-txt.md next to the HTML and advertise it (<link rel="alternate" type="text/markdown" ...>).

Right vs wrong

Right	Wrong
Accept: text/markdown → Content-Type: text/markdown	Always HTML, header ignored
A Vary: Accept header (for correct caching)	No Vary — a proxy serves the wrong format
Markdown mirrors the main content	A trimmed/stale Markdown version

Common mistakes

• Ignoring the Accept header — the server always returns HTML.
• No Vary: Accept — CDNs/proxies cache one format for everyone.
• Markdown ≠ HTML in content — the versions drift apart.
• Serving text/plain instead of text/markdown.

How to verify

A scan checks content negotiation. Manually — compare two requests:

curl -s -H "Accept: text/markdown" https://example.com/ | head
curl -sI -H "Accept: text/markdown" https://example.com/ | grep -i 'content-type\|vary'

The second response should show Content-Type: text/markdown and Vary: Accept.