Topic
Use of language models as nondeterministic reviewers within otherwise mechanical validation pipelines.
In Ashish Jain and Armaan Sandhu's UserProxyBench, 24.4% of successful tau-bench-family episodes contain a violation of the simulated user's own script. A passing reward can therefore hide a user that did part of the agent's work.
Reality
Playtomic ran eight newly released models through its Slack agent Dory on 39 real questions, graded by two judges from different labs. The company says public rankings measure generic skills, so other teams would have to rerun the method on their own questions.
Reality
Seven chain-of-thought monitors in a LessWrong study flagged 66 points fewer flawed physics solutions when their 'known correct' answer was wrong. Any pipeline that hands its monitor a reference answer depends on that reference being right.
Reality
GitHub released ReviewBench, an open benchmark that scores AI code reviewers on 219 pull requests from 187 public repositories in 19 languages. Buyers can rerun any reviewer on the public dataset and published judge, though every score depends on one LLM's view of what counts as relevant.
Reality
Red Hat's 200M-parameter classifier scored 89.01% on prompt injection to a 35B LLM judge's 89.31%, answering in a median 54.1 ms against 312.5 ms. For injection screening, a large judge in the request path now has to justify nearly six times the latency for 0.3 points.
Reality
MIT and Sakana AI's SIFT ran a coding agent's full self-improvement search on roughly $34 of API calls, about a tenth of the Darwin Godel Machine's resources. The saving comes from a language-model judge screening patches, so it holds only while that judge picks correctly.
Reality
Apollon Labs' Greek benchmark found gpt-oss-20b making up 48 words per 1,000, against zero for six of the 15 models tested. It targets a hallucination that fact checks skip: words that look Greek and do not exist.
Reality
Retirement-answer-check's injection regex caught all 16 first-round attacks and none of the 20 written by a second red team that had read it. Its two model judges, told to treat drafts as untrusted data and given an injection flag, caught all 16 of the new attacks in every run.
Reality
CDV's adaptive stop rule truly reached its quality bar in 71% of runs under judge noise of 0.10, against 93% for a fixed six-step budget. Requiring two passing scores in a row lifts that to 97% for 1.4 extra steps, so a one-line change beats both the Bayesian policy and the budget.
Reality
A PyTorch case study credits Shopify's GraphQL agent with beating frontier-model quality at 96 percent lower serving cost. Most of what it documents is how the reward signal behind that gets defined and checked.
Reality
A Magic: The Gathering rules agent works out which source is entitled to settle a question before it answers. Its evaluation pits 496 structured documents against a single BM25 pass over the same corpus, graded blind.
Reality
With documentation and web search removed, GPT-5.6 Luna passed 18% of 336 Dev Proxy tasks and 15% of 413 SPFx tasks, and the passes appear throughout both product histories instead of stopping at one release.
Reality
A dev.to developer prices each API request after the work by asking a second model how much the answer represents, and the first version threw away the calibration he was paying for by rounding its probabilities into three tiers.
Reality
CommentBench splits human comments on AI-safety posts and drafts into target points, filters out the ones a model could not reach without extra context, and has Opus 5 judge which of the rest a model hit.
Reality
The four RAGAS-lineage metrics were built to separate a retriever's failures from a generator's. Read in pairs, they also expose the case where the model skipped the context and got the answer right anyway.
Reality
A LessWrong write-up planted invalidating flaws in ML experiment logs and asked models to write the conference abstract. A second model scored the disclosure on three levels, and the published example is one before-and-after pair.
Reality
Each case in claude plugin eval runs three times with the plugin loaded and three times without it. That doubling is what produces the delta column, and every one of those calls is billed to your own credentials.
Reality
Cloudflare says its graph model flagged eight malicious scripts running on live storefronts. A retrospective check put seven of the eight outside VirusTotal entirely. URLScan flagged none of the eight as malicious.
Reality
A dev.to post publishes an unexecuted sketch that splits deterministic structural checks from a model-graded rubric and pins both graders to a changelog file, so a case citing a missing version stops the run before anyone reads a pass rate.
Reality
Its engineering post argues that automated tests for multi-turn tool use belong in place before an agent scales. The evidence behind the advice is three deployments, one of them Anthropic's own.
Reality