Skip to content

Benchmark

AutomationBench v1.0.6

Zapier's AutomationBench, a benchmark that scores AI agents on multistep business workflows across apps and tools, used to compare frontier models.

Known aliases

  • AutomationBench
  • AutomationBench 1.0.6
  • AutomationBench-AA
  • AutomationBench v1.0.6
  • Zapier AutomationBench

Current stories

buildConfirmed6 publishers

DeepSeek's V4.1-Flash report prices a live agent's memory at 890 bytes per token

The quarter-size global KV cache and eighth-size persistent storage are serving-cost claims rather than benchmark scores, and banking the second one requires a prefix-cache tier your stack has to already run.

Perspective Coverage

6 publishers
Builder
Builder 53%
Operator
Operator 33%
Investor
Investor 14%

Reality

Evidence66
Adoption42
Hype gap+14
Incentives64
Confidence63
buildOne report1 publisher

H Company's Holo4 merges its desktop and API specialists into one open-weight model

H Company's open-weight Holo4 27B handles GUIs, code, MCP servers and REST APIs in one model, scoring 61.7% on OSWorld 2.0 at an estimated $1.22 per task. The choice between specialists happens during training, so agent teams could drop the router from their stack if the scores hold on their own workloads.

Publishers:dev.to

Reality

Evidence35
Adoption
Not scored
Hype gap+25
Incentives
Not scored
Confidence40
productConfirmed5 publishers

OpenAI's 10-cent GPT-6.1 Sol price covers only cached input

OpenAI lists GPT-6.1 Sol at $2 per million input tokens and $10 per million output; the 10-cent figure in early coverage is its cached-input rate. Teams moving work off Astra should budget on the list rates and OpenAI's per-task costs.

Perspective Coverage

5 publishers
Builder
Builder 44%
Operator
Operator 34%
Investor
Investor 22%

Reality

Evidence50
Adoption
Not scored
Hype gap+35
Incentives70
Confidence60
leadershipOne report1 publisher

OpenAI's GPT-6.1 Sol price cut rewards teams whose agents resend the same long context

OpenAI's GPT-6.1 Sol halves cached-input pricing to $0.10 per million tokens and leaves standard rates at $2 and $10. Agents that resend long context collect the saving, while other buyers weigh gains shown mostly in OpenAI's own tests.

Publishers:implicator.ai

Reality

Evidence55
Adoption30
Hype gap+20
Incentives65
Confidence55
productConfirmed4 publishers

Meta hires MongoDB's CEO to sell its weeks-old Muse agent to businesses

Meta hired CJ Desai, MongoDB's CEO of about 11 months, to run Meta Enterprise Platform, a new unit selling its Muse agent and AI stack to businesses. So far a buyer can weigh Desai's enterprise record and a product list, with no prices or contract terms.

Perspective Coverage

4 publishers
Builder
Builder 25%
Operator
Operator 36%
Investor
Investor 39%

Reality

Evidence62
Adoption
Not scored
Hype gap+35
Incentives65
Confidence60
investOne report1 publisher

Claude Opus 5.5 charges 5.3 times GPT-6 Sol's price per task for a nine-point benchmark lead

US companies are buying the cheapest AI model that can finish a job, the FT reports, with Ramp data pointing to a 41% fall in effective token prices. Investors now have to value AI vendors on cost per completed task, a figure that shifts with the workload.

Reality

Evidence42
Adoption50
Hype gap+20
Incentives
Not scored
Confidence40
productConfirmed5 publishers

Price cuts minutes apart send agent routing back to the spreadsheet

Anthropic took 20% off Opus 5.5 and OpenAI halved its two new GPT-6 tiers the same day. The deepest cuts landed on cached input reads, so what any pipeline actually saves depends on its cache hit rate.

Perspective Coverage

5 publishers
Builder
Builder 38%
Operator
Operator 37%
Investor
Investor 25%

Reality

Evidence55
Adoption20
Hype gap+25
Incentives70
Confidence60
productConfirmed8 publishers

OpenAI halves the API price of Sol and Luna against GPT-5.6's promotional rates

OpenAI says better caching and inference let it cut API prices for Sol and Luna by half, and the cost advantage it claims for the cheap tier over the old top tier comes in at one tenth on the benchmark it published and one hundredth in its summary.

Perspective Coverage

8 publishers
Builder
Builder 36%
Operator
Operator 42%
Investor
Investor 22%

Reality

Evidence40
Adoption
Not scored
Hype gap+35
Incentives70
Confidence60
investConfirmed16 publishers

Sol's 27-cent benchmark task undercuts Opus 5 by more than eleven times

Anthropic and OpenAI shipped cheaper model tiers minutes apart on Tuesday. OpenAI halved Sol's posted token prices, and the only per-task cost comparison between the two labs so far comes from OpenAI itself.

Perspective Coverage

16 publishers
Builder
Builder 29%
Operator
Operator 35%
Investor
Investor 36%

Reality

Evidence62
Adoption25
Hype gap+25
Incentives70
Confidence60
securityOne report1 publisher

OpenAI's 50 percent API price cut doubles the token volume a flat budget buys

OpenAI's own benchmarks put GPT-6 Sol and Luna at half the list price and a fraction of a rival's cost per finished task. For defenders, the thing getting cheaper is autonomous tool calls into business systems.

Reality

Evidence32
Adoption38
Hype gap+34
Incentives80
Confidence52
investOne report1 publisher

Anthropic cuts Opus 5.5 prices 20% on tokens, 60% on cache reads, citing fewer tokens burned for 40% total savings

Every token price on Claude Opus 5.5 fell 20% except cache reads, which dropped 60% to 20 cents a million. For the advertised 40% saving to come from price alone, half a buyer's Opus 5 bill has to be cache reads.

Reality

Evidence38
Adoption24
Hype gap+32
Incentives78
Confidence44
productOne report1 publisher

OpenAI cuts prices on new GPT-6 Sol and Luna models

Sol now bills $2 and $10 per million tokens and Luna $0.10 and $0.50, while OpenAI quotes its own benchmark results per task, where the cheap model lands 2.2 points behind Sol on the software engineering test.

Publishers:thenextweb.com

Reality

Evidence42
Adoption55
Hype gap+25
Incentives78
Confidence52
scienceOne report1 publisher

Xiaomi streamed the cost and reward of a trillion-parameter RL run to a public dashboard

Two agentic post-training jobs, a 1.02-trillion-parameter model and a 309-billion cousin, ran with step time, reward and infrastructure error rates on the open web. Both stopped near step 30, and both models are still unreleased.

Publishers:rajeshparikh.substack.com

Reality

Evidence34
Adoption14
Hype gap+31
Incentives62
Confidence41
productOne report1 publisher

DeepSeek prices cached agent input at $0.003 a million tokens off-peak

The rate undercuts the cached-input prices OpenAI and Anthropic publish by more than 130 times, and DeepSeek's own release says the sparse-attention design behind it has untested limits at cache boundaries.

Publishers:dataconomy.com

Reality

Evidence46
Adoption32
Hype gap+18
Incentives72
Confidence55
buildOne report1 publisher

Nex-AGI's runnable N2.5 tiers ask for two H100s or sixteen H200s

Publishing weights and publishing something a team can actually deploy are different acts, and N2.5 lands on both sides of that line depending on which tier you pick and whether its weights exist yet.

Publishers:runtimewire.com

Reality

Evidence45
Adoption15
Hype gap+30
Incentives68
Confidence50
leadershipOne report1 publisher

GLM-5.3-Flash benchmarks its tenth-of-the-price claim against its own predecessor

The MIT-licensed 320B model card claims it beats GLM-5.2 at a tenth of the price and approaches Claude Opus 4.8 on coding, but it names no dollar rate, and the comparisons are largely the vendor's own.

Publishers:huggingface.co

Reality

Evidence34
Adoption18
Hype gap+46
Incentives82
Confidence61
productOne report1 publisher

Ox Alpha was GLM-5.3-Flash, and the number that decides displacement is 18 billion

Z.ai says the model runs at a tenth the cost of its last one. The comparison an operator needs is against the API invoice they already pay, and the release does not make it.

Publishers:siliconangle.com

Reality

Evidence32
Adoption24
Hype gap+38
Incentives72
Confidence44

Earlier coverage

  1. Gemini 3.7 Flash Is Cheap Until January 1, When The Agent Bill Doubles

    Invest · August 16, 2026 · Confirmed2 publishers