# Escalate when the answer falls short Try a lower-cost tier, check the response, and escalate only when it does not meet your acceptance rule — with the arithmetic that decides whether it pays. Source: https://tokentriage.com/docs/build/routing-cascades --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; import CascadeLadderDiagram from '../../../components/CascadeLadderDiagram.astro'; Most routing decisions are made before a model responds. A cascade does something different: it calls a lower-cost tier, checks the response, and escalates to a stronger tier only when the response does not meet your acceptance rule. Use one when request data alone cannot predict whether the lower-cost tier will be good enough. An escalated request pays for multiple model calls and may also pay for a judge, so a cascade can either reduce or increase total cost. Use it for response-based escalation, not as an availability failover. ## Before you start You need: - At least two configured tiers: a lower-cost first attempt and a stronger fallback. [Turn routing on](/docs/build/routing) defines tiers and validates a first policy. - `mode: route` for the cascade to make upstream calls. In `ledger` and `shadow`, TokenTriage records the planned route but does not execute the cascade. - A working judge credential if a stage uses `judge: true`. The inline judge reads the environment variable named by `classifier.oracle.api_key_env`, which defaults to `ANTHROPIC_API_KEY`. - A latency budget for every attempted stage. The judge is synchronous, and each conditional stage defaults to a 60-second timeout unless you set `timeout_ms`. Cascades do not run for streaming requests. TokenTriage forwards a streaming request to the first stage as an ordinary call; the decision row has no `cascade` block and no error announcing the bypass. ## Configure a minimal cascade This is the smallest useful shape. Add it to a non-production configuration first; the rollout steps below cover how to introduce it to production traffic. ```yaml mode: route routing: default_tier: frontier policy: version: 1 root: cascade: stages: - id: try-cheap to: { tier: cheap } timeout_ms: 15000 judge: true accept_when: "outcome.status = 200 and outcome.refusal_markers = 0 and outcome.judge.available and outcome.judge.ok" final: { tier: frontier } ``` The `try-cheap` stage makes a real call to the `cheap` tier. Its acceptance rule requires three things: 1. The provider returned HTTP `200`. 2. The response did not match a known refusal phrase. 3. The judge returned a verdict and approved the response. If all three checks pass, TokenTriage returns that response. Otherwise it calls `frontier`. `final` is the fallback target: it has no acceptance check, so its response is the last attempt. Every entry in `stages` must have an `accept_when`; only `final` may omit one. The example limits the conditional stage to 15 seconds. Choose `timeout_ms` from your own latency budget rather than copying that value unchanged. ## Validate and inspect one request Validate the configuration before starting the proxy: ```console $ tokentriage validate --config tokentriage.yaml ``` Then enable the cascade in a non-production environment or a low-weight production split, send one identifiable request, and inspect its decision: ```console $ tokentriage policy explain --req-id --log ~/.tokentriage/decisions.jsonl ``` For the one-stage cascade above, an accepted cheap response produces one provider attempt; an escalation produces two. `policy explain` prints `cascade: attempts=N final=` and one line per attempted stage. The inputs read by `accept_when` and `judge_cost` remain in the decision row's `cascade` block, available in the JSON log or dashboard drawer. [Routing explanation fields](/docs/build/routing-explanation-fields#cascade-attempts) defines every printed field. ## Every call that runs is billed The first attempt is a real upstream call your provider bills. If the stage asks for a judge, that is another billed call. When `accept_when` holds, the response goes to the client and the cascade stops. When it does not, the response is discarded but still billed, and the next stage runs. For the one-stage cascade above, let `a` be the share of requests accepted at `cheap`. The expected cost per request is: ```text expected cost = cheap call + judge call + (1 - a) × frontier call break-even a = (cheap call + judge call) / frontier call ``` With per-stage figures from a real decision row — a cheap attempt at `$0.000381`, a judge call at `$0.000226`, and a frontier attempt at `$0.011902`: ```text (0.000381 + 0.000226) / 0.011902 = 0.051 → about 5.1% ``` Above a 5.1% accept rate, this cascade costs less than sending every request directly to `frontier`. Below it, the cascade costs more than that baseline. At a 70% accept rate, the expected cost per request is `0.000381 + 0.000226 + 0.30 × 0.011902 = $0.004178`, about 65% below the `$0.011902` baseline. ### Measure before you widen it Shadow duplication can collect counterfactual quality evidence before the cascade serves traffic, but it cannot measure the cascade's acceptance rate because the cascade does not execute in `shadow` mode. Roll out in two phases: 1. Use shadow duplication or an offline calibration to confirm that the lower-cost tier is a credible candidate. 2. Put the cascade behind a low-weight [split arm](/docs/build/routing-splits) in `route` mode. Measure its acceptance rate, latency, and total spend before increasing the weight. Compute break-even from your own tier and judge prices. If the observed acceptance rate remains below it, disable or retune the cascade rather than widening the rollout. ### Total cascade spend A decision row's top-level `cost` prices the **accepted** attempt alone. The `cascade` block records every attempted stage, accepted and discarded, plus the judge. Summing only `cost.usd` therefore understates cascade spend: ```text total cascade spend = Σ per-stage costs + judge cost = row cost + Σ discarded-stage costs + judge cost ``` ## Define what counts as acceptable `accept_when` runs in **outcome scope**, which exists only once an attempt has returned. These conditions can use response status, size, latency, token usage, finish reason, refusal markers, and the optional judge verdict.
| Input | Type | When unavailable | |---|---|---| | `outcome.status` | number | always present | | `outcome.response_bytes` | number | always present | | `outcome.latency_ms` | number | always present | | `outcome.output_tokens` | number | usage could not be measured | | `outcome.finish_reason` | string | the provider sent none | | `outcome.refusal_markers` | number | always present | | `outcome.judge.ok` | bool | the stage did not ask for a judge | | `outcome.judge.available` | bool | the stage did not ask for a judge | `outcome.refusal_markers` counts how many distinct decline patterns appear in the first 8 KiB of the response, matched case-insensitively over a fixed list of phrases like "I cannot" and "unable to assist". The dependable use is the clean gate `= 0`; a threshold above zero reads as "at least N different phrasings matched", which one strongly worded refusal can satisfy on its own.
When using a judge, require both `outcome.judge.available and outcome.judge.ok`. If a requested judge fails, times out, or has no credential, both values are `false`, so this expression rejects the stage and escalates. Do not use `not outcome.judge.ok`: it becomes `true` when no verdict was obtained and can accept an ungraded response. ## Configure the inline judge `judge: true` on a stage runs a grading call between the attempt and the check, so `accept_when` can use a per-response accept-or-reject verdict instead of relying only on response metadata and refusal markers. The judge is synchronous and on the serving path, so the client waits for it. Its wall time is included in the row's `total_ms` and also recorded as `judge_ms`; its spend is recorded as `judge_cost`. The inline judge reads the environment variable configured by `classifier.oracle.api_key_env`; that setting defaults to `ANTHROPIC_API_KEY`. It calls the built-in judge model over the Anthropic Messages API. `classifier.oracle.model` configures the shadow-duplication judge, not this inline judge. Because this is an outbound provider call on the request path, confirm that the data path and latency fit your deployment before enabling it. Treat `outcome.judge.ok` as a decision for this response, not as a calibrated probability or evidence that a tier is broadly equivalent to another. That broader claim requires held-out evaluation. ## Operational limits | Limit | Value | What happens past it | |---|---|---| | stages per cascade | 4, plus `final` | validation refuses the policy | | nesting | a cascade may not sit inside a cascade stage | validation refuses the policy | | per-stage timeout | 60 seconds unless `timeout_ms` is set | four stages plus `final` can wait five minutes | | streaming | not supported | the request is served by stage 1 as a single ordinary call | | mode | `route` only | `ledger` and `shadow` record the plan without executing it | Set an explicit `timeout_ms` on every conditional stage. With four stages plus `final`, the default limits can allow five sequential model attempts to occupy the request path; judge calls add more time. Streaming workloads bypass the cascade as described at the start of this page.
The client receives exactly one response, the accepted attempt, with its upstream headers copied through. Discarded attempts never reach the client writer. Governance checks each candidate model before calling it against the caller's own scope, so a key allowed to use the cheap model but not the frontier one has its escalation skipped rather than served. If every viable stage is out of scope, the request gets a `403` naming a scope violation rather than a generic failure.
## What a cascade is not **It is not an availability failover.** A cascade decides from the returned outcome. An upstream that returns `502` produces `outcome.status = 502`, which fails the example's `accept_when` and continues to `final`, but the cascade does not rank targets by health. For deliberate ordered fallback on health, use [a select with `first_eligible`](/docs/build/routing-pools#try-tiers-in-a-deliberate-order). **It is not evidence of general model quality.** An inline judge decides whether to accept one response. To decide whether the lower-cost tier is good enough across a workload, use [calibration](/docs/calibration/prove-a-cheaper-tier) with held-out data.