Topic
Task suites that measure whether agents complete realistic business work, and what their scores omit.
Anthropic opened dynamic workflows in Claude Managed Agents on 9 October, letting one run spawn up to 1,000 agents, 64 at a time. The only performance evidence offered is Anthropic's own planted-bug test.
Reality
SpaceXAI lists Grok 4.7 at $2 and $6 per million tokens against $10 and $50 for GPT-6 Astra and Claude Fable 5.1, and the only per-task figures in the record put a Cursor run at $4.69 against about $9.
Perspective Coverage
8 publishersReality
Microsoft researchers say CASD writes better agent prompts from saved logs for about $1.60 a pass, over 22 times cheaper than iterative tuning. Outside one telecom test its accuracy edge over GEPA is thin, so price and setup carry the case.
Reality
Paper Instruments says agents on its Paper Office libraries and guidance passed 92.5% of trials across 61 Office tasks, against 80.7% on upstream libraries. The test runs the code and the instructions together, so teams have to check on their own files which one earned the gain.
Reality
NVIDIA's SWE-Serve scores the same 627 patches twice on 19 SGLang tasks, once with the live-serving tests and once without. The pass rate falls from 69.4% to 45.9%, and 242 of the 276 live tests came from SGLang itself.
Reality
AWS has open-sourced a preconfigured general-purpose agent on top of its Strands SDK and says it runs 45% cheaper than Claude Code and Codex. The model call defaults to Amazon Bedrock, and Marc Brooker says one line changes that.
Reality
An arXiv paper names the failure behavioral state decay and runs a side-car memory agent next to an unmodified action agent, reporting gains of 8.3 and 6.8 points of pass@1 on two long-horizon benchmarks.
Reality
The 769 task records behind Atria Dawn Preview are process data from the people who built it, and the one-third feasibility figure is their own retrospective judgement about work they had already completed.
Reality
IBM Research's ALTK-Evolve distils an agent's own trajectories into scored guidelines and injects the top five at inference time, and a companion post puts a number on the reliability an average success rate hides.
Reality
Vals AI's 141-hour Minecraft run ended with OpenAI's newest model farming potatoes for hours after losing its end-game loot and its spawn point in one explosion, and the recovery policy it came away with was a note to self.
Reality
MemTensor's case for checking agent memory at write time rests on a Microsoft experiment in which read-only environment probing reached 73% against 70% for memory alone, a gap of about one question in 40.
Reality
LangChain's new harness profiles set prompts, tool implementations and tool names per model family. The company measures a 10 to 20 point gain on a tau2-bench subset it curated from tasks frontier models have not saturated.
Reality
Every task in the benchmark is checked against its required terminal state, so any tool order can pass and any extra side effect fails. Claude Opus 5, the strongest model tested, cleared 66.50% at pass@1.
Reality
Anthropic's December 2024 guidance and three agent benchmarks converge on an awkward result, because the workloads whose steps cannot be enumerated in advance are also the ones where measured agent completion is lowest.
Reality
The band it joined is one where the best closed models still finish under 10% of tasks end to end at roughly $50 and 20 minutes each, so what post-training buys a law firm is price and hosting rather than capability.
Reality
Harvey and Baseten report rubric pass rates rising from 23.3% to 62.4% across seven models once the data room sits in a Python REPL and sub-agents do the reading, while Claude Code with Opus-5 manages 24.6%.
Reality
An empirical study of publicly released PostTrainBench trajectories finds the training strategy is fixed at step one and the whole remaining budget goes on local tweaks, with scaffolding and human hints improving execution and leaving that pattern intact.
Reality
The Accio team's verifier ignores what the agent says it did and inspects the containers it leaves behind. That design is the news; the two-pass open-weight lead Alibaba reports from it sits inside its own harness spread.
Reality
The company's own benchmark keeps every flagship model it tested below 60 percent completion across 2140 CRM task instances, and the fix its researchers report is domain knowledge somebody inside your org has to write down.
Reality