Five retrieved guidelines lift AppWorld's hard-task success by 14.2 points
IBM Research's ALTK-Evolve distils an agent's own trajectories into scored guidelines and injects the top five at inference time, and a companion post puts a number on the reliability an average success rate hides.
The Engineer · Build desk
What happened
- IBM Research's ALTK-Evolve distils an agent's raw trajectories into reusable guidelines, filters them for quality, and injects only the relevant ones back into context at the moment of action.
- On AppWorld, a ReAct agent given the task instruction plus the top five retrieved guidelines gained 14.2 points on hard multi-step tasks, which the post reports as a 74 percent relative increase.
- A companion post reports the baseline it is measured against: a GPT-4.1 ReAct agent succeeded on 77.4 percent of runs but on all five repetitions for only 53.0 percent of tasks.
- Guidelines derived from that diagnosis cut the 24.4-point gap to 12.0 points, with same-task Pass^5 up 16.0 points and no cost in average accuracy.
Why it matters
- Who has to decide decision A team chasing a flaky agent now has to choose where the budget goes, because the Hugging Face post argues that pinning a seed or forcing greedy decoding leaves the token distribution untouched.
- What's now possible capability A single trace you already logged is enough to locate the steps that flip, so flakiness can be diagnosed on production traffic without labeled outcomes or a rerun harness.
- What it limits constraint The top-five retrieval cap fixes how much learned guidance reaches the model per call, and guidelines mined on one app surface only help where the same failures recur.
- What it sets up precedent With Pass^k defined and measured, a demo that passed once stops counting as evidence, and a vendor quoting an average success rate can be asked for the all-runs number.
Next story · Build · Oct 1
CloudWatch Omni opens incident investigation to staff without AWS console accessConfirmed2 publishers
Two numbers in the dev.to writeup describe the same hard-task result: a 14.2-point delta on AppWorld [3] and a 74 percent relative increase in success on hard tasks [4]. Divide one by the other to recover the baseline. 14.2 / 0.74 is about 19.2, so the unaided agent solved roughly 19 percent of hard tasks and the guideline-assisted one lands near 33 percent [22]. About two thirds of the hard partition still fails [23].
The context claim is narrower than the phrasing suggests. The evaluated agent got the task instruction plus the top five retrieved guidelines generated on a prior run [5]. Five rules is a fixed per-call cost, and it does not grow as the agent accumulates experience, because a background consolidate-and-score job merges duplicates, prunes weak rules and boosts proven strategies instead of appending to the prompt [6]. Capture sits in an observability layer, Langfuse or another OpenTelemetry-based tool, with pluggable extractors persisting candidate entities [7].
For the 14.2 to mean anything on another workload, the environment has to look like AppWorld's: tasks calling about 9.5 APIs across 1.8 apps, with hard cases marked by more complex control flow [8]. The guidelines were mined from train and dev runs and tested on an unseen test-normal partition of the same benchmark [5]. That tests generalization across tasks on a single fixed API surface. The headline metric is Scenario Goal Completion, which requires success across scenario variants, and the post says SGC gains exceeded raw pass-rate improvements [9].
The Consistency Analyzer is the cheaper half of the system to try. It resamples each decision point in one recorded trace with a single call requesting k completions, k=5 by default, needs no ground truth, and never re-runs the task end to end [13]. Measuring Pass^5 directly means five complete runs. If decision points track API calls, a 9.5-step AppWorld task costs about ten resampling calls against about 48 agent steps across five runs [19].
Whether that diagnosis applies to your stack rests on one claim in the Hugging Face post: greedy decoding and a fixed seed govern how a distribution gets turned into a token, and neither one changes the distribution, so near-ties still reorder under GPU floating-point non-associativity and request batching [16]. Pass^k is the pessimistic mirror of the familiar Pass@k, requiring every attempt to succeed, and it sits at or below Mean@k [15]. Standard agent evaluation reports Mean@k, often with k=3, sometimes k=1 [20].
The dev.to post does not say which model backed the ReAct agent behind the 14.2-point delta [18]; the consistency post names GPT-4.1 for its baseline [11]. It also cites an unnamed recent MIT study for the claim that 95 percent of pilots fail because agents do not adapt and learn on the job [21].
Trying the lite version costs two commands in Claude Code, after which the plugin extracts entities from trajectories, stores them as files on the filesystem, and retrieves them through Claude Code hooks [10].
What to watch
- The arXiv technical report should show per-partition baselines and name the model behind the 14.2-point hard-task delta.
- Whether agent leaderboards start publishing Pass^k next to Mean@k, and at what k.
- Whether consistency guidelines hold outside AppWorld, on workloads where each task hits a different API surface.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence45
- Adoption20
- Hype gap+15
- Incentives85
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
ALTK-Evolve turns raw agent trajectories into reusable guidelines, converting interaction traces into candidate guidelines, filtering for quality, and injecting only relevant guidance at the moment of action.
- [2]
Full methodology and evaluations are in a technical report on arXiv.
- [3]
In benchmarks the approach boosted reliability, especially on hard multi-step tasks, with a delta of 14.2% on AppWorld, without bloating context.
- [4]
The Hard tasks saw a 74% relative increase in success, where guidelines are useful to navigate the intricate control flows.
- [5]
A ReAct agent received the task instruction plus the top 5 retrieved guidelines generated on a prior run (train/dev) and was tested on an unseen partition (test-normal).
- [6]
A background consolidate-and-score job merges duplicates, prunes weak rules and boosts proven strategies, evolving a library of guidelines, policies and SOPs.
- [7]
Full agent trajectories are captured in an Interaction Layer such as Langfuse or another OpenTelemetry-based observability tool, and pluggable extractors mine traces for structural patterns and persist them as candidate entities.
- [8]
AppWorld agents complete realistic multi-step tasks via APIs, averaging 9.5 APIs across 1.8 apps, with hard cases requiring more complex control flow.
- [9]
The reported metric is Scenario Goal Completion (SGC), a strict consistency metric requiring success across variants, and SGC gains exceeded raw pass-rate improvements.
- [10]
Evolve-Lite installs into Claude Code with two commands (claude plugin marketplace add AgentToolkit/altk-evolve, then claude plugin install evolve-lite@evolve-marketplace); the plugin extracts entities from trajectories, stores them as files on the filesystem, and uses Claude Code hooks for automatic retrieval.
- [11]
On AppWorld test_normal, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions but succeeded in all five runs for only 53.0% of tasks, a 24.4-point consistency gap.
- [12]
On hard tasks the consistency gap reaches 30 points.
- [13]
The Consistency Analyzer resamples an agent's own recorded trajectory to find flip-prone decision points, needs one trace and no ground truth, and resamples each decision point with a single call requesting k completions (k=5 by default) rather than re-running the task end to end.
- [14]
Turning that diagnosis into guidelines halves the gap from 24.4pp to 12.0pp (same-task Pass^5 +16.0pp, similar-task +13.0pp) without costing anything in average accuracy.
- [15]
Pass^k is the fraction of tasks where the agent succeeds on all k runs, the pessimistic mirror of Pass@k, and Pass^k is always at or below Mean@k, which is at or below Pass@k.
- [16]
Greedy decoding and a fixed seed both govern how a distribution gets turned into a token and say nothing about the distribution itself; flat distributions with near-tied tokens may reorder under GPU floating-point non-associativity, request batching and other platform-side effects.
- [17]
Consistency guidelines are a new guideline type in altk-evolve built on top of the Consistency Analyzer diagnostic.
- [18]
The dev.to post does not state which model backed the ReAct agent that produced the 14.2-point AppWorld result.
- [19]
For a 9.5-step AppWorld task, one resampling call per decision point is about ten calls, against about 48 agent steps for the five complete runs a Pass^5 measurement needs.
- [20]
Standard agent evaluation reports Mean@k, running a benchmark k times and averaging the pass rate, often with k=3 and sometimes just 1.
- [21]
The dev.to post states that a recent MIT study found 95% of pilots fail because agents do not adapt and learn on the job.
- [22]
If the 14.2-point delta and the 74% relative increase share a baseline, the unaided hard-task success rate was about 19.2% and the guideline-assisted rate about 33.4%.
- [23]
About two thirds of AppWorld hard tasks still fail after guidelines are injected.
Sources
2 independent publishers whose own reporting we read for this story.
- dev.toALTK-Evolve: On-the-Job Learning for AI Agents
1 article · September 17, 2026
- huggingface.coYour Agent Aced the Task. Will It Do It Again?
1 article · September 15, 2026
Topics and entities
Topics
- Inference determinism
- Agent Benchmarks
- Agent Reliability
- LLM Runtime Observability
- Agent Memory Systems