Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Escalate when the answer falls short

Try a lower-cost tier, check the response, and escalate only when it does not meet your acceptance rule — with the arithmetic that decides whether it pays.

How-to guideLast updated

On this page

Most routing decisions are made before a model responds. A cascade does something different: it calls a lower-cost tier, checks the response, and escalates to a stronger tier only when the response does not meet your acceptance rule. Use one when request data alone cannot predict whether the lower-cost tier will be good enough.

An escalated request pays for multiple model calls and may also pay for a judge, so a cascade can either reduce or increase total cost. Use it for response-based escalation, not as an availability failover.

Before you start

You need:

  • At least two configured tiers: a lower-cost first attempt and a stronger fallback. Turn routing on defines tiers and validates a first policy.
  • mode: route for the cascade to make upstream calls. In ledger and shadow, TokenTriage records the planned route but does not execute the cascade.
  • A working judge credential if a stage uses judge: true. The inline judge reads the environment variable named by classifier.oracle.api_key_env, which defaults to ANTHROPIC_API_KEY.
  • A latency budget for every attempted stage. The judge is synchronous, and each conditional stage defaults to a 60-second timeout unless you set timeout_ms.

Configure a minimal cascade

This is the smallest useful shape. Add it to a non-production configuration first; the rollout steps below cover how to introduce it to production traffic.

mode: route

routing:
  default_tier: frontier
  policy:
    version: 1
    root:
      cascade:
        stages:
          - id: try-cheap
            to: { tier: cheap }
            timeout_ms: 15000
            judge: true
            accept_when:
              "outcome.status = 200 and outcome.refusal_markers = 0
              and outcome.judge.available and outcome.judge.ok"
        final: { tier: frontier }

The try-cheap stage makes a real call to the cheap tier. Its acceptance rule requires three things:

  1. The provider returned HTTP 200.
  2. The response did not match a known refusal phrase.
  3. The judge returned a verdict and approved the response.

If all three checks pass, TokenTriage returns that response. Otherwise it calls frontier. final is the fallback target: it has no acceptance check, so its response is the last attempt. Every entry in stages must have an accept_when; only final may omit one.

The example limits the conditional stage to 15 seconds. Choose timeout_ms from your own latency budget rather than copying that value unchanged.

Validate and inspect one request

Validate the configuration before starting the proxy:

$ tokentriage validate --config tokentriage.yaml

Then enable the cascade in a non-production environment or a low-weight production split, send one identifiable request, and inspect its decision:

$ tokentriage policy explain --req-id <req-id> --log ~/.tokentriage/decisions.jsonl

For the one-stage cascade above, an accepted cheap response produces one provider attempt; an escalation produces two. policy explain prints cascade: attempts=N final=<tier> and one line per attempted stage. The inputs read by accept_when and judge_cost remain in the decision row’s cascade block, available in the JSON log or dashboard drawer. Routing explanation fields defines every printed field.

Every call that runs is billed

A one-stage cascade with a judge and final fallbackThe cascade calls the cheap tier, runs the optional inline judge, and checks accept_when. If the condition holds, that first response goes to the client. Otherwise TokenTriage discards it and calls the final frontier fallback, whose response goes to the client without another acceptance check. Every model attempt and judge call that runs is billed.stage: try-cheapmodel call, billedjudge?another billed callaccept_whenchecks the outcomeThe client's responseone returned attemptacceptescalatefinal: frontierno acceptance checkreturn fallbackEvery call that runs is billed. Top-level cost covers the returned attempt; discarded attempts are recorded per stage.

The first attempt is a real upstream call your provider bills. If the stage asks for a judge, that is another billed call. When accept_when holds, the response goes to the client and the cascade stops. When it does not, the response is discarded but still billed, and the next stage runs.

For the one-stage cascade above, let a be the share of requests accepted at cheap. The expected cost per request is:

expected cost  =  cheap call + judge call + (1 - a) × frontier call
break-even a   =  (cheap call + judge call) / frontier call

With per-stage figures from a real decision row — a cheap attempt at $0.000381, a judge call at $0.000226, and a frontier attempt at $0.011902:

(0.000381 + 0.000226) / 0.011902  =  0.051   →  about 5.1%

Above a 5.1% accept rate, this cascade costs less than sending every request directly to frontier. Below it, the cascade costs more than that baseline. At a 70% accept rate, the expected cost per request is 0.000381 + 0.000226 + 0.30 × 0.011902 = $0.004178, about 65% below the $0.011902 baseline.

Measure before you widen it

Shadow duplication can collect counterfactual quality evidence before the cascade serves traffic, but it cannot measure the cascade’s acceptance rate because the cascade does not execute in shadow mode. Roll out in two phases:

  1. Use shadow duplication or an offline calibration to confirm that the lower-cost tier is a credible candidate.
  2. Put the cascade behind a low-weight split arm in route mode. Measure its acceptance rate, latency, and total spend before increasing the weight.

Total cascade spend

A decision row’s top-level cost prices the accepted attempt alone. The cascade block records every attempted stage, accepted and discarded, plus the judge. Summing only cost.usd therefore understates cascade spend:

total cascade spend  =  Σ per-stage costs + judge cost
                     =  row cost + Σ discarded-stage costs + judge cost

Define what counts as acceptable

accept_when runs in outcome scope, which exists only once an attempt has returned. These conditions can use response status, size, latency, token usage, finish reason, refusal markers, and the optional judge verdict.

Outcome fields available to accept_when
Input Type When unavailable
outcome.status number always present
outcome.response_bytes number always present
outcome.latency_ms number always present
outcome.output_tokens number usage could not be measured
outcome.finish_reason string the provider sent none
outcome.refusal_markers number always present
outcome.judge.ok bool the stage did not ask for a judge
outcome.judge.available bool the stage did not ask for a judge

outcome.refusal_markers counts how many distinct decline patterns appear in the first 8 KiB of the response, matched case-insensitively over a fixed list of phrases like “I cannot” and “unable to assist”. The dependable use is the clean gate = 0; a threshold above zero reads as “at least N different phrasings matched”, which one strongly worded refusal can satisfy on its own.

Configure the inline judge

judge: true on a stage runs a grading call between the attempt and the check, so accept_when can use a per-response accept-or-reject verdict instead of relying only on response metadata and refusal markers.

The judge is synchronous and on the serving path, so the client waits for it. Its wall time is included in the row’s total_ms and also recorded as judge_ms; its spend is recorded as judge_cost.

The inline judge reads the environment variable configured by classifier.oracle.api_key_env; that setting defaults to ANTHROPIC_API_KEY. It calls the built-in judge model over the Anthropic Messages API. classifier.oracle.model configures the shadow-duplication judge, not this inline judge. Because this is an outbound provider call on the request path, confirm that the data path and latency fit your deployment before enabling it.

Treat outcome.judge.ok as a decision for this response, not as a calibrated probability or evidence that a tier is broadly equivalent to another. That broader claim requires held-out evaluation.

Operational limits

Limit Value What happens past it
stages per cascade 4, plus final validation refuses the policy
nesting a cascade may not sit inside a cascade stage validation refuses the policy
per-stage timeout 60 seconds unless timeout_ms is set four stages plus final can wait five minutes
streaming not supported the request is served by stage 1 as a single ordinary call
mode route only ledger and shadow record the plan without executing it

Set an explicit timeout_ms on every conditional stage. With four stages plus final, the default limits can allow five sequential model attempts to occupy the request path; judge calls add more time. Streaming workloads bypass the cascade as described at the start of this page.

What the client sees, and what governance does per stage

The client receives exactly one response, the accepted attempt, with its upstream headers copied through. Discarded attempts never reach the client writer.

Governance checks each candidate model before calling it against the caller’s own scope, so a key allowed to use the cheap model but not the frontier one has its escalation skipped rather than served. If every viable stage is out of scope, the request gets a 403 naming a scope violation rather than a generic failure.

What a cascade is not

It is not an availability failover. A cascade decides from the returned outcome. An upstream that returns 502 produces outcome.status = 502, which fails the example’s accept_when and continues to final, but the cascade does not rank targets by health. For deliberate ordered fallback on health, use a select with first_eligible.

It is not evidence of general model quality. An inline judge decides whether to accept one response. To decide whether the lower-cost tier is good enough across a workload, use calibration with held-out data.