Topic
An evaluation approach that checks each AI agent control layer—prompt, context, harness, loop, and graph—rather than judging only final output quality.
In Ashish Jain and Armaan Sandhu's UserProxyBench, 24.4% of successful tau-bench-family episodes contain a violation of the simulated user's own script. A passing reward can therefore hide a user that did part of the agent's work.
Reality
Sun et al. propose scoring agents on Identify, Solve and Escalate when retrieved evidence contradicts training data, according to a dev.to summary. The post's sample harness has a revision check that returns true for any run that produces an answer.
Reality
Microsoft DevRel published the AX Practitioner Playbook, a method for finding why coding agents misuse a platform, drawn from hundreds of agent sessions. Its case is that owners of docs, MCP servers and CLIs can fix agent behavior now, if their evals can prove which change helped.
Reality
OpenAI and Ironclad published a case study on turning production contract workflows into training data and evals for computer-use agents. Scores from such an eval carry over to other software only as far as it shares contract approval's roles and branching.
Reality
One developer's 'could not tell' exit code stopped a false green on a hold, then exposed four new ways a check can mislead. Each looks like a clean result, so agent evals that adopt the third value need controls that force every outcome.
Reality
Five agents built to cheat a ledger-scored refund benchmark found three scorer holes, including a $50 cap that scored $120 in split refunds as $0.00. An outage, too-good results and a fact-check found the other three.
Reality
Huntress ran seven Rails tasks past Claude Fable 5.1 and watched it copy hand-rolled code out of its own repository. Changing the harness, in three cheap steps, took API recall from 48 percent to 100.
Reality
A dev.to post credits Anthropic with running the same model and the same prompt under two harnesses, 20 minutes and $9 for a broken result against six hours and $200 for a working one. The hourly spend barely moved.
Reality
AWS has added three skill-focused evaluators to Strands Evals and Bedrock AgentCore Evaluations. Two of them call a model once per invoked skill, and the third is a deterministic check that exists only in Strands.
Reality
NVIDIA's developer blog sets out the five-level rollup from step to benchmark and the two scores that read the same trace. Step-level says where the chain broke. End-to-end reads the environment and says whether the refund posted.
Reality
LangChain clocked TypeSafe AI's Jev at 0.44 seconds and $0.00035 per call against three LLM judges on the same eval set. Whether that price transfers depends on how much structure your traces already have.
Reality
A dev.to FAQ proposes failing any agent session whose diff touches a test path without the ticket ID in a signed allowlist. Both scripts in the post are labeled unexecuted, so the false-positive count is yours to find.
Reality
COGEXT's author pushed 150 samples from cookbooks, DEV posts and Hacker News through his own extractor. The unbiased 120 yielded one promise, and the 13 in the enriched set averaged 0.79 confidence with a single deadline between them.
Reality
License Referee resolves each npm dependency to an SPDX id and looks up a ruling for the exact dependency-into-project direction. On its 22-question test set, all three configurations agreed on the verdict and one invented its source.
Reality
ProgramDistill turns 26 working web applications into thousands of verifiable tasks where the specification is the running software itself. Difficulty comes from stacking repairs that depend on one another, and the reported scores track it.
Reality
A dev.to post lays out a coding-agent eval protocol that hashes the token and tool-call envelope alongside the tasks, so editing a cap changes the run id. No executed runs are published with it.
Reality
Turing Post talked through agent feedback with Salesforce's chief AI scientist at Dreamforce. A correction lands in one of four places: the context, persistent memory, the software around the model, or the weights.
Reality
A developer ran three change tasks in one Python and TypeScript product against two code-index tools, and counted a run only after the focused test passed and then failed again with the defect put back on purpose.
Reality
Coding-agent pass rates get re-quoted after every prompt edit. A dev.to protocol freezes a seeded holdout split before any tuning, and at its minimum size the published number gets too coarse to show a 13-point gain.
Reality
AWS's system prompt optimizer hands a reflector agent a shell tool and a directory of scored production traces, and offline batch evaluation plus a live-traffic A/B test decide which of its proposed edits reach the agent.
Reality
Build · August 26, 2026 · Confirmed2 publishers