Vercel has added confidence-based decision fallbacks to AI Gateway, allowing developers to send a model’s result to a second model when the first answer meets a predefined uncertainty condition. The feature is currently available in beta.
Unlike a conventional fallback, which is triggered when a model fails to execute, the new system can escalate a request after the primary model has successfully returned an answer. Developers can define rules based on the confidence or probability associated with a decision.
For Choice and Score decisions, developers can use a confidenceBelow condition. For Boolean decisions, the system supports probabilityBetween, which checks the probability assigned to the true outcome against a specified range.
The rules can target a specific question or apply to all applicable questions in a decision request. Multiple conditions can also be combined with any, all or atLeast logic, allowing developers to escalate only when one or more uncertainty signals meet their configured thresholds.
When a condition is triggered, AI Gateway reruns the original state and questions against the fallback model. The fallback result replaces the primary result rather than being combined with it. The system does not recursively evaluate the fallback, so a conditional decision request has a maximum of two successful decision stages.
The fallback model does not have to be another native decision model. Vercel supports using a conventional language model as the second stage, with structured output used to produce the requested decision.
There is also a distinction between uncertainty and execution failure. If the primary model fails before producing a usable decision, AI Gateway can use the configured conditional model as an ordinary execution fallback. If the conditional fallback itself fails, the primary result is not returned as a successful response.
Triggered fallbacks involve a second model execution, meaning developers incur the cost and latency of both stages. Vercel recommends measuring escalation rates, accuracy and the additional cost and latency rather than assuming a particular confidence threshold will work across applications.
The feature also accounts for cases where confidence information is unavailable. For Choice and Score decisions, Vercel treats missing or non-finite confidence conservatively, allowing the confidenceBelow condition to trigger and recording confidence_unavailable as the reason.
Developers can configure the behavior directly in AI Gateway requests or place the policy in a decision virtual model so callers using that model inherit the same conditional fallback configuration.
Vercel has also moved its terminology from “evaluation” to “decision” for these APIs. New integrations use experimental_decide and gateway.decisionModel(), while the older experimental_evaluate and gateway.evaluationModel() interfaces remain available as deprecated aliases. The HTTP endpoint remains /v1/evaluate.
Vercel’s documentation uses release-change classification as an example, where a decision model handles routine cases and an alternative model reviews cases whose confidence falls below a configured threshold. The company cautions that confidence-based routing cannot prevent a model from being confidently wrong.
Vercel’s announcement and its AI Gateway decision fallback documentation provide the configuration and validation details for the beta feature.
Discover more from Aree Blog
Subscribe now to keep reading and get access to the full archive.
