Skip to content

Standard

Web Bot Auth

IETF draft standard built on HTTP Message Signatures (RFC 9421) that lets AI agents and bots cryptographically sign requests so sites can verify the operator.

Current stories

buildOne report1 publisher

Cloudflare starts measuring Verified bots against robots.txt and 429 responses over time

Cloudflare has introduced a system that scores Verified bots on robots.txt compliance and on whether they slow down after an HTTP 429. It reports the result as a trust band, and Cloudflare says it will share scores on Radar later.

Reality

Evidence35
Adoption10
Hype gap+10
Incentives60
Confidence65
buildConfirmed6 publishers

Signed requests would meet two of the four conditions Amazon sets for agents

Amazon cut Meta's Muse off for arriving unannounced, browsing without identifying itself, and appearing to keep customer credentials. Signed-request verification, already running at Cloudflare and CloudFront, answers the middle objection.

Perspective Coverage

7 publishers
Builder
Builder 25%
Operator
Operator 36%
Investor
Investor 39%

Reality

Evidence68
Adoption40
Hype gap+15
Incentives78
Confidence64
buildOne report1 publisher

Ahrefs and three rival bot-analytics tools run a Cloudflare Worker on every file a page loads

Ahrefs, Known Agents, Profound and Scrunch run a Worker on every request, so a 20-file page spends 21 of Cloudflare's 100,000 free daily invocations. A site with 10,000 page views a day would need more than twice that.

Publishers:dev.to

Reality

Evidence64
Adoption
Not scored
Hype gap+8
Incentives
Not scored
Confidence58
buildOne report1 publisher

AI-agent requests on Cloudflare's network grew more than 1,700% in a year

Cloudflare says AI-agent requests on its network grew more than 1,700% in a year as non-human traffic passed half the total. It now wants sites to identify and price agents as customers, after advising last year that new domains block AI training crawlers.

Reality

Evidence35
Adoption50
Hype gap+25
Incentives85
Confidence40
buildOne report1 publisher

DataDome's threat research VP puts AI login traffic at 1.08 percent of classified requests

The 51.7 percent in DataDome's 2026 bot report is login's share of five sensitive endpoint types, and Jerome Segura told Lets Data Science the same 313 million requests work out to about 1.08 percent of the 29.02 billion requests the company could classify by path.

Reality

Evidence58
Adoption42
Hype gap+18
Incentives80
Confidence55
buildOne report1 publisher

Google's ARD spec stops at the endpoint, before the agent proves which workload is calling

The spec's own boundary list says a verified catalog entry establishes which domain claims a capability and what trust material it published. It says nothing about whether the calling process is the agent it names.

Publishers:dev.to

Reality

Evidence45
Adoption18
Hype gap−10
Incentives30
Confidence45
buildOne report1 publisher

Paying for a crawl means retrying every priced URL with a transaction hash

HTTP 402 sat in the spec unused since 1991. The x402 flow makes it callable. The work lands inside your crawler: two origin requests per priced page, a funded wallet on the server's chain, and settlement checked before any retry.

Publishers:dev.to

Reality

Evidence44
Adoption18
Hype gap+42
Incentives58
Confidence46
buildOne report1 publisher

Cloudflare attaches a status and a rejection reason to every BotBase submission

The bot submission form moves into Application Security and grows a history tab, which means the entry describing your crawler becomes a record you are expected to keep matching your traffic rather than a form you filed once.

Reality

Evidence47
Adoption20
Hype gap+12
Incentives72
Confidence56