Skip to content

Topic

AI Agent Reliability

Failure modes of tool-using LLM agents in production workflows, including false completion reports on side-effecting operations.

Current stories

investOne report1 publisher

Nvidia's Mid-Harness buys an 18-point agent reliability gain with eight candidate commands per step

Nvidia researchers lifted a terminal agent's first-try success from 50.00% to 68.03% by letting a judge model pick among eight candidate commands. For teams budgeting agents, the spend moves from model size to calls per step.

Reality

Evidence30
Adoption
Not scored
Hype gap+25
Incentives
Not scored
Confidence35
buildOne report1 publisher

n8n 2.43.0 blocks publishing of workflows with unconnected required inputs

n8n 2.43.0, shipped October 6, refuses to publish any workflow whose required node inputs are unconnected, according to a dev.to review of the changelog. The check covers wiring only, so wrong credentials and bad prompts still need their own tests before an agent goes live.

Publishers:dev.to

Reality

Evidence55
Adoption
Not scored
Hype gap+5
Incentives
Not scored
Confidence55
leadershipOne report1 publisher

Business Insider testers lost 8 pounds and a Notion schedule when an AI agent acted for them

Instinct, an AI agent Business Insider staff tested, paid 8 pounds for the wrong Oxford congestion charge and wiped an editor's Notion schedule. Its research held up, so the thing to settle before delegating is what the agent may do without a person checking first.

Reality

Evidence45
Adoption
Not scored
Hype gap+10
Incentives
Not scored
Confidence40
buildOne report1 publisher

Agent retries that feed errors back into the prompt make every attempt cost more than the last

Retry math in a dev.to post on agent circuit breakers has a failing agent's calls growing from 2,000 tokens to 20,000 by attempt 50. Summed, that spend rises with the square of the attempts, so the cap belongs in the orchestrator.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+40
Incentives
Not scored
Confidence35
buildOne report1 publisher

An explicit uncertainty gate removed 60% of one team's production agent incidents

The gate is a few lines of Python, and the engineering sits in whatever populates response.confidence and in whether your incident mix resembles the one where that 60 percent was measured.

Publishers:dev.to

Reality

Evidence25
Adoption
Not scored
Hype gap+35
Incentives
Not scored
Confidence35
buildOne report1 publisher

CauterRule traced its 0.08 extraction score to a token-F1 gate inside the metric

CauterRule's v0.3.1 field test scored extracted rules against ground truth for the first time and read 0.08 on the golden corpus. The trigger half of those rules was matching at 0.6 or better, while the directive comparator counted tokens.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap+20
Incentives60
Confidence55
scienceOne report1 publisher

Handing a stuck drive-thru order to a human lifted accuracy from 81% to 95%

McDonald's ran an automated voice at about 100 US drive-thrus for three years and shut it down in 2024. A 2025 audit of three chains using voice AI measured what happens when the machine stops understanding.

Publishers:cio.com

Reality

Evidence44
Adoption38
Hype gap+30
Incentives58
Confidence46
buildOne report1 publisher

Hashing the arguments lets a reworded retry fire the destructive call twice

Across 34 runs on three agent frameworks, a recorder proxy shows that what the dedup key names decides whether a retried publish executes once or twice, and that surviving a SIGKILL is a separate question about where state is kept.

Publishers:dev.to

Reality

Evidence46
Adoption
Not scored
Hype gap+18
Incentives36
Confidence52
productOne report1 publisher

Beacon folds Haize Labs into an in-house research group for its 45 software companies

Haize Labs sold red teaming and evaluation tooling to frontier labs and large enterprises. Its engineers are now an internal research group inside a Toronto holding company that buys small-business software.

Publishers:thenextweb.com

Reality

Evidence45
Adoption30
Hype gap+12
Incentives70
Confidence52
buildOne report1 publisher

Swapping every leaf value for a marker string clears the schema on 42% of 265 MCP tools

A fault-injection scan of 31 popular MCP servers found only 3% of their 265 tools declare an output contract that would reject a well-typed but wrong response. The failing 97% split into two different problems.

Publishers:dev.to

Reality

Evidence62
Adoption18
Hype gap+18
Incentives72
Confidence55
buildOne report1 publisher

A cached empty list made a read-back commit rule ask for 70 paragraphs

The engineering lead at app platform GoodBarber traced 70 duplicate paragraphs to a read path serving a 742-byte empty list out of a 60-second cache. His debugging order now puts the prompt last.

Publishers:dev.to

Reality

Evidence58
Adoption20
Hype gap−12
Incentives35
Confidence47
productOne report1 publisher

Claude spent most of its 1,000-page PyPI attack transcript stuck on hCaptcha

Anthropic published the chain-of-thought from an agent trying to upload a malicious file to PyPI. The data scientist Colin Fraser counted image puzzles filling roughly 95 percent of more than 1,000 pages.

Publishers:gizmodo.com

Reality

Evidence38
Adoption
Not scored
Hype gap+30
Incentives55
Confidence42
productOne report1 publisher

A CEO's $25-a-day agent filled the one gap in its data with a confident guess

Writing in Fast Company, a non-technical CEO prices a day of his homemade agent at no more than $25 in tokens against work he sizes at half a chief of staff's load, then describes the customer call he nearly walked into with the wrong numbers.

Publishers:fastcompany.com

Reality

Evidence30
Adoption18
Hype gap+38
Incentives55
Confidence45
buildOne report1 publisher

A support-ticket agent handed back the production connection string it was given

One engineer's account of an MCP rejection that cascaded into a secrets leak and a loop that would not stop carries no incident numbers, but the config blocks and handler code it prints show where the failure boundary has to sit.

Publishers:dev.to

Reality

Evidence34
Adoption12
Hype gap+24
Incentives38
Confidence46
buildOne report1 publisher

An unsupervised agent loop billed $38 before anything in the system said stop

A dev.to engineer argues the shipping bottleneck has moved from prompt wording to the environment around the loop, and his own postmortems carry that case a good deal better than the 40% failure figure he opens with.

Publishers:dev.to

Reality

Evidence40
Adoption28
Hype gap+38
Incentives40
Confidence33
investOne report1 publisher

Microsoft's ThinkingBox puts a number on agent reliability: 65% once, 25% every time

The best of 12 models passed 65.36% of 507 business tasks on the first attempt and only 25.25% across all 20 trials. Agent procurement should be priced on the second number.

Reality

Evidence40
Adoption16
Hype gap+12
Incentives62
Confidence38
buildOne report1 publisher

Long Horizon: Google open-sources the agent bugs that never threw an error

Five patterns, each traced to a bug the team hit running agents on their own work for weeks. None of the bugs raised an error, and the API bill looked normal throughout.

Publishers:dev.to

Reality

Evidence32
Adoption18
Hype gap+14
Incentives62
Confidence38
productOne report1 publisher

A retry that sends the email twice is a correctness bug, not a slow job

A systems newsletter makes the case for durable execution in agent workflows. The consequence for operators is narrower and harder: side effects have to be safe to replay.

Publishers:newsletter.systemdesign.one

Reality

Evidence26
Adoption
Not scored
Hype gap+22
Incentives74
Confidence34
scienceOne report1 publisher

Two governments got AI slop in writing, and the review chain caught none of it

South Africa pulled a draft national AI policy over invented citations, and Deloitte refunded part of a A$440,000 report for the same defect. The failure was institutional, not personal.

Publishers:phys.org

Reality

Evidence34
Adoption62
Hype gap+18
Incentives46
Confidence41

Earlier coverage