Skip to content

Benchmark

OSWorld

OSWorld is a benchmark that evaluates AI agents on real computer tasks across executable desktop and OS environments, measuring end-to-end task completion.

Known aliases

  • OSWorld 2.0
  • OSWorld-2.0
  • OSWorld 2.0 offline
  • OSWorld 2.1
  • OSWorld-2.1

Current stories

buildConfirmed22 publishers

Prompt length sets what Anthropic's Haiku 5.5 price cut is worth to each workload

Anthropic launched Claude Haiku 5.5 at an average price about 75% below Haiku 4.5. The saving varies widely with prompt length, so teams moving classification, support or query traffic need to price their own requests before they switch models.

Perspective Coverage

22 publishers
Builder
Builder 47%
Operator
Operator 36%
Investor
Investor 17%

Reality

Evidence66
Adoption30
Hype gap+25
Incentives70
Confidence65
buildOne report1 publisher

H Company's Holo4 merges its desktop and API specialists into one open-weight model

H Company's open-weight Holo4 27B handles GUIs, code, MCP servers and REST APIs in one model, scoring 61.7% on OSWorld 2.0 at an estimated $1.22 per task. The choice between specialists happens during training, so agent teams could drop the router from their stack if the scores hold on their own workloads.

Publishers:dev.to

Reality

Evidence35
Adoption
Not scored
Hype gap+25
Incentives
Not scored
Confidence40
productConfirmed5 publishers

OpenAI's 10-cent GPT-6.1 Sol price covers only cached input

OpenAI lists GPT-6.1 Sol at $2 per million input tokens and $10 per million output; the 10-cent figure in early coverage is its cached-input rate. Teams moving work off Astra should budget on the list rates and OpenAI's per-task costs.

Perspective Coverage

5 publishers
Builder
Builder 44%
Operator
Operator 34%
Investor
Investor 22%

Reality

Evidence50
Adoption
Not scored
Hype gap+35
Incentives70
Confidence60
leadershipOne report1 publisher

OpenAI's GPT-6.1 Sol price cut rewards teams whose agents resend the same long context

OpenAI's GPT-6.1 Sol halves cached-input pricing to $0.10 per million tokens and leaves standard rates at $2 and $10. Agents that resend long context collect the saving, while other buyers weigh gains shown mostly in OpenAI's own tests.

Publishers:implicator.ai

Reality

Evidence55
Adoption30
Hype gap+20
Incentives65
Confidence55
buildConfirmed8 publishers

Gemini 3.8 Flash's introductory price doubles on December 31, 2026

Google's third Flash release in six weeks keeps the $0.75/$3.75 rate card. But the model also spends more tokens per task. Both numbers in your cost model are moving before the price even changes.

Perspective Coverage

8 publishers
Builder
Builder 53%
Operator
Operator 29%
Investor
Investor 18%

Reality

Evidence58
Adoption35
Hype gap+22
Incentives72
Confidence62
buildOne report1 publisher

Astra bills at long-context rates once a request passes 30 percent of its input window

OpenAI's spec lets gpt-6-astra take 922,000 input tokens, but requests above 272,000 move to higher long-context rates. Provisioning a model that drives a desktop, a shell and MCP servers starts with that threshold.

Publishers:dev.to

Reality

Evidence40
Adoption
Not scored
Hype gap+25
Incentives55
Confidence40
productConfirmed8 publishers

OpenAI halves the API price of Sol and Luna against GPT-5.6's promotional rates

OpenAI says better caching and inference let it cut API prices for Sol and Luna by half, and the cost advantage it claims for the cheap tier over the old top tier comes in at one tenth on the benchmark it published and one hundredth in its summary.

Perspective Coverage

8 publishers
Builder
Builder 36%
Operator
Operator 42%
Investor
Investor 22%

Reality

Evidence40
Adoption
Not scored
Hype gap+35
Incentives70
Confidence60
securityOne report1 publisher

OpenAI's 50 percent API price cut doubles the token volume a flat budget buys

OpenAI's own benchmarks put GPT-6 Sol and Luna at half the list price and a fraction of a rival's cost per finished task. For defenders, the thing getting cheaper is autonomous tool calls into business systems.

Reality

Evidence32
Adoption38
Hype gap+34
Incentives80
Confidence52
productOne report1 publisher

OpenAI cuts prices on new GPT-6 Sol and Luna models

Sol now bills $2 and $10 per million tokens and Luna $0.10 and $0.50, while OpenAI quotes its own benchmark results per task, where the cheap model lands 2.2 points behind Sol on the software engineering test.

Publishers:thenextweb.com

Reality

Evidence42
Adoption55
Hype gap+25
Incentives78
Confidence52
scienceOne report1 publisher

A Berkeley scanning agent scores 100% on five AI agent benchmarks without solving a task

Berkeley's RDI center attacked the step where each benchmark computes its score, and without solving a task its own scorecard reports 100% on five of the eight, about 98% on GAIA and 73% on OSWorld.

Publishers:rdi.berkeley.edu

Reality

Evidence45
Adoption35
Hype gap+20
Incentives55
Confidence50
buildOne report1 publisher

Anthropic prices its newer Sonnet a third below Sonnet 4.5

Sonnet 4.5 still leads GPT-5 on the coding leaderboards, and GPT-5 lists about 46 percent below it on a 5:1 token mix. Anthropic's current Sonnet undercuts both of Sonnet 4.5's list prices, and that complicates a routing plan built on the older pair.

Publishers:dev.to

Reality

Evidence40
Adoption20
Hype gap+15
Incentives55
Confidence45
buildOne report1 publisher

Mano-CUA's ten-point win over Claude shrinks to 0.8 points against Gemini

Mininglamp published NavEval scores for its own model on its own benchmark. Across the three entries, the spread tracks how each stack reads a page. It is not a case of specialists beating frontier models.

Publishers:dev.to

Reality

Evidence24
Adoption12
Hype gap+46
Incentives86
Confidence58
buildOne report1 publisher

Harness-Bench makes the scaffold a measured variable, and model-only scores lose meaning

A new diagnostic benchmark treats the execution layer as something to vary rather than a fixed backdrop. Its conclusion is that agent capability belongs to a model-harness pair.

Publishers:arxiv.org

Reality

Evidence42
Adoption10
Hype gap+22
Incentives55
Confidence40