Vercel's AI Gateway bills two model runs for every low-confidence decision it escalates
Vercel's AI Gateway now reruns successful but low-confidence decisions on a second model and bills both runs. Each escalation adds a billed call and its latency, so the threshold belongs on decisions where labelled tests show the second run fixes costly errors.
The Engineer · Build desk
What happened
- The fallback is a single conditional object that must sit first in providerOptions.gateway.models, with plain model names allowed after it.
- Choice and Score questions trigger on confidenceBelow, while Boolean questions use probabilityBetween, an inclusive range on P(true).
- If the question field is left out, the condition checks every question of the matching type, and any single match escalates.
- Plain model names already in the list keep catching outright execution errors, so existing fallback setups are unaffected, according to Vercel.
Why it matters
- What it costs cost Spend on a decision endpoint becomes the primary's price plus the escalation rate times the fallback's price, so the threshold is a budget setting as much as a quality one.
- What it limits constraint A request gets one confidence-escalation model, so a call mixing an intent choice and a priority score cannot send each uncertain answer to a different fallback.
- Who's exposed exposure Any critical path built on this depends on beta behaviour, and the dev.to guide advises checking the exact request shape and observed behaviour in your own environment first.
Next story · Build · 23h ago
Vercel sells Pro teams $80 of GLM 5.3 and Kimi K3 credit for $30 a monthConfirmed2 publishers
The primary model answers first. The gateway checks that completed answer against the condition and calls the fallback only on a match [1][2]. An escalated request therefore waits for two model runs in sequence, and its latency is roughly the primary's duration plus the fallback's [22]. Requests that never trip the condition pay for one run [2].
The value being compared needs care. According to the dev.to guide, confidence describes how concentrated the answer's probability distribution is, and it is not the probability that a Boolean answer is true [8]. A tightly concentrated intent choice can still name the wrong team [8]. Only a run against labels shows how often that happens [17]. Vercel's examples use 0.6 for a Choice threshold and [0.4, 0.6] for a Boolean band [9]. The guide says to treat both as syntax examples, not recommended production values [9].
Scope is where a multi-question call goes wrong, and the guide flags it directly [11]. With the question field left out, each extra Choice or Score question is one more chance to trip the condition, so the call escalates at least as often as its most uncertain question [23]. Conditions can also be combined, so escalation can depend on more than one signal [13]. The changelog's sample config scopes its condition to a question named `intent`, with a 0.6 threshold and `openai/gpt-6-astra` as the fallback [12].
The guide proposes a sequence before rollout:
1. Build a small evaluation set from real request shapes, with secrets and personal data removed. Store each input with its expected choice or score and the production consequence of a wrong answer [16]. 2. Run the primary model alone. Record its output, confidence, correctness against the label and request duration [17]. 3. Tune the threshold on one set, then check it on a held-out slice so the tuning has not memorised the first examples [16].
Because the baseline records confidence for every example, it also shows what share of traffic each candidate threshold would send to the second model [24]. Its pass condition fits in one sentence: "A confidence threshold is useful only if confidence separates cases your fallback can improve from cases it cannot." [18]
I think this is the right design for a team routing support tickets across a few labelled intents. Uncertainty escalation and error fallback live in one list as separate triggers, and an unsure answer and a failed call are different failures [4]. For a first target, the guide suggests a support-intent choice, where a billing issue sent to the wrong team has a clear cost [15]. Free-form assistant replies are a worse fit, because without defined labels there is no consistent way to tell whether the second run helped [15]. Adoption can go one question at a time, since requests without the conditional object keep their existing behaviour [6]. The whole feature hangs off an SDK function that the guide's code imports as `experimental_decide` [19].
What to watch
- Vercel moving decision fallbacks out of beta, and whether the request shape in providerOptions.gateway.models changes when it does.
- Published pricing or latency figures for an escalated decision, so teams could estimate the cost of a threshold before running a baseline.
- Whether experimental_decide drops its experimental status in the AI SDK, since the gateway's confidence fallback is built around that call.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap0
- Incentives40
- Confidence65
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
AI Gateway can now escalate a decision request to a fallback model when the primary model's answer trips a confidence condition the user sets.
- [2]
Vercel AI Gateway's confidence-based decision fallback works with experimental_decide and runs the configured fallback model when a successful decision matches a confidence condition.
- [3]
Because a triggered fallback runs a second decision, it bills both stages.
- [4]
The setting lives in providerOptions.gateway.models as a conditional model object; it must be the first entry, the array can contain only one conditional object, and plain model-name entries can follow it for execution-error fallback. The conditional model reruns a successful but uncertain decision; a string entry handles a model that cannot execute successfully.
- [5]
The plain model names already listed in models keep catching outright errors, so existing fallbacks are unaffected.
- [6]
Requests without the conditional object keep their existing behavior.
- [7]
For Choice and Score questions the condition can use confidenceBelow; Boolean questions use probabilityBetween, an inclusive range for P(true).
- [8]
The confidence value describes how concentrated the answer's probability distribution is; it is not the probability that a Boolean answer is true.
- [9]
Vercel's example uses 0.6 for a Choice threshold and [0.4, 0.6] for a Boolean band; the guide says to treat both as syntax examples, not recommended production values.
- [10]
Leaving out question makes the condition check every question of the matching type, so { "confidenceBelow": 0.6 } escalates when any Choice or Score answer falls below 0.6.
- [11]
If the question field is omitted, any one matching question triggers the fallback, and that broader scope is easy to miss when one decide call contains several questions.
- [12]
The changelog's example config is { "gateway": { "models": [ { "model": "openai/gpt-6-astra", "when": { "question": "intent", "confidenceBelow": 0.6 } } ] } }.
- [13]
Conditions can be combined, so an escalation can depend on more than one signal.
- [14]
The feature is explicitly beta, and the guide advises validating the exact request shape and observed behavior in your own environment before relying on it in a critical path.
- [15]
The guide recommends picking one decision where an incorrect low-confidence answer has a clear consequence; a support-intent choice that routes a billing issue to the wrong team may qualify, while a free-form assistant response with no defined labels is harder to evaluate because you cannot consistently determine whether the second stage improved it.
- [16]
The guide advises building a small representative evaluation set from real request shapes, removing secrets and personal data, storing the input, expected choice or score and production consequence of a wrong answer, keeping it separate from tuning examples, and using a held-out slice to check that tuning did not memorize the first set.
- [17]
The guide advises running the primary model alone first and recording its output, confidence, correctness against the label, and request duration as a baseline.
- [18]
"A confidence threshold is useful only if confidence separates cases your fallback can improve from cases it cannot."
- [19]
The guide's code example imports experimental_decide as decide from the 'ai' package and uses gateway.decisionModel from '@ai-sdk/gateway'.
- [20]
Expected model cost per decision request equals the primary decision's cost plus the escalation rate multiplied by the fallback decision's cost.
- [21]
Because the models array accepts only one conditional object and each conditional object names one model, a request has a single confidence-escalation model.
- [22]
Because the condition is evaluated on the primary model's completed answer, the fallback cannot start until the primary returns, so an escalated request's latency is roughly primary duration plus fallback duration.
- [23]
With the question field omitted, a multi-question decide call escalates at least as often as its most uncertain matching question, since any single match triggers the fallback.
- [24]
Because the baseline records confidence for every example, it shows what share of traffic a candidate threshold would escalate.
Sources
2 independent publishers whose own reporting we read for this story.
- blog.vercel.comAI Gateway adds confidence-based decision fallbacks
1 article · October 6, 2026
- dev.toVercel AI Gateway confidence fallbacks: measure before rollout
1 article · October 7, 2026
Topics and entities
Topics
- LLM model fallback routing
- AI gateways and proxy layers