# Cut cost with a model fallback

Try a cheap tier first and escalate only when the answer fails an inline quality check, so the bill drops without quality cliffs.

Source: https://tokentriage.com/docs/recipes/cut-cost-with-fallback

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';

Serve every request from a cheap tier first, grade the cheap response inline, and
escalate to a stronger tier **only** for the rows the cheap tier can't handle. The
mechanism is a `cascade` node in the [routing policy](/docs/build/policy-language):
`stages` try tiers in order, an `accept_when` predicate grades each attempt, and a
required `final` tier is the escalation floor. You promote it to `route` mode only after
shadow evidence proves the cheaper path holds quality.

## Prerequisites

- A running TokenTriage with at least two tiers ordered cheapest → most capable (e.g.
  `cheap` and `frontier`) and a decision log configured (`ledger.decision_log.path`).
- The oracle judge configured (`classifier.oracle`) — the inline cascade judge reuses it.
  Without a wired judge, a `judge: true` stage yields `outcome.judge.available = false`
  and the `accept_when` below escalates rather than inventing a verdict.
- `shadow.duplicate` available for the promotion step (below).

## 1. Write the cheap-first cascade policy

`routing.policy` is the v2 routing form and is **mutually exclusive** with the legacy
`routing.rules` list — a config sets exactly one. Make the root a `cascade`: try `cheap`,
run the inline judge, and escalate to `frontier` only when the attempt isn't a clean,
non-refusing, judge-approved 200.

```yaml
routing:
  default_tier: frontier          # fail-safe target on classifier error (never a downgrade)
  policy:
    version: 1
    root:
      cascade:
        stages:
          - id: try-cheap
            to: { tier: cheap }
            judge: true           # run the inline shadow judge before accept_when
            accept_when: 'outcome.status = 200 and outcome.refusal_markers = 0 and outcome.judge.available and outcome.judge.ok'
        final: { tier: frontier } # always accepted — the escalation floor
```

Only outcome-scope inputs are legal inside `accept_when`: `outcome.status`,
`outcome.refusal_markers`, `outcome.judge.ok`, `outcome.judge.available`,
`outcome.finish_reason`, `outcome.latency_ms`, and the token/byte facts. An **absent**
`outcome.judge.available` reads false, so the `... and outcome.judge.available and
outcome.judge.ok` form escalates when the judge is down — it never accepts on a fabricated
verdict.

Validate it strictly before running:

```console
$ tokentriage validate --config configs/tokentriage.yaml
```

<Callout type="warning">
Escalation pays for both attempts. When a row escalates, TokenTriage really does call the
cheap tier (discarded), run the judge, **and** call `frontier` — so an escalated row costs
`cheap + judge + frontier`, more than routing straight to `frontier`. The row's top-level
`cost` prices the *accepted* attempt alone; each discarded attempt's cost is disclosed
per-stage, and `judge_cost` (the summed cost of the accept-check judge calls) is a single
block-level field — both in the decision's `cascade` block. The bill drops only when the
cheap tier's accept rate is high enough that the saved rows outweigh the escalated ones —
which is exactly what the next step measures before you enforce anything.
</Callout>

## 2. Climb the ladder: run it in shadow first

Don't flip straight to `route`. Set the top-level `mode: shadow` and turn on shadow
duplication so TokenTriage grades the cheaper tier against the stronger one on the *same*
query — the only data that can prove the cheaper path holds quality:

```yaml
mode: shadow                      # observe + grade the counterfactual; client bytes unchanged
shadow:
  duplicate:
    enabled: true
    sample_rate: 0.01
    max_usd_per_day: 5.00         # hard cap; the sampler stops when exhausted
    judge: oracle
```

In `shadow` mode the client still sees identical bytes — you're accumulating graded rows,
not enforcing. Let traffic run, then read the shadow-savings total:

```console
$ tokentriage report --log ~/.tokentriage/decisions.jsonl
=== TokenTriage shadow-savings report ===

routed rows:        560 of 560 (rows with a routing decision + would_usd)
not-routing spend:  $3.731000  (served on the top tier)
routed spend:       $1.305850  (under the routing decision)
savings:            $2.425150  (65.0% vs not routing; 455 of 560 rows routed cheaper)
```

*(Illustrative output — your totals come from your own log.)*

## 3. Read the honesty verdict before you promote

`report` shows the *savings*; `eval --quality shadow` answers the question that licenses
enforcement — **does the cheaper routing actually beat a blind model mix at its own cost?**
Only the shadow (counterfactual) source can return a `BEATS` verdict:

```console
$ tokentriage eval --quality shadow --log ~/.tokentriage/decisions.jsonl
=== TokenTriage honesty eval ===
verdict: routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication)
quality source: shadow (labeled iff judged-shadow grade present (chosen_ok != nil); counterfactual per-cost verdict from both tiers' grades)
labeled rows:   8 of 8
… (per-tier TIER POINTS table elided)

FRONTIER  (per-cost verdict basis: router quality vs zero-router at the router's own cost)
  router quality:        1.0000  @ cost $5.500000
  zero-router @ router:  0.7500
  delta:                 +0.2500
```

If instead you see `insufficient labels to judge routing` or a served-model source sitting
`ON the observed-model frontier` (Δ 0.0000), you do **not** have a promotable win yet —
oracle/outcome labels score only the tier that served each row and can never show routing
skill. Keep it in shadow and let judged-shadow grades accumulate.

## 4. Promote to route mode

Once the verdict is a `BEATS` and the promote gate is satisfied, enforce it:

```yaml
mode: route                       # enforce the cascade — cheap first, escalate on failure
```

You should see the cheap tier serving the accepted rows and only the failing rows carrying
a `frontier` attempt. Inspect a decision's cascade with `policy explain`, which prints
`cascade: attempts=N final=<tier>` and one line per stage (`id tier model accepted cost=$…`):

```console
$ tokentriage policy explain --req-id <req-id> --log ~/.tokentriage/decisions.jsonl
```

(The `accept_when` reads and the block-level `judge_cost` are recorded in the decision-log
`cascade` block itself — the JSON row / dashboard drawer — rather than printed by this command.)

<Callout type="warning">
**Only promote on evidence — a BEAT plus fresh labels under the *current* policy.** The
promote gate reads **only** the shadow source, so served-model labels can never satisfy it.
It requires the shadow verdict to be `BEAT` *and* enough labeled rows whose newest graded
row is within the freshness window (a fixed 30-day / `720h` default). Critically,
the gate is keyed to a **policy epoch**: when you change the routing config its fingerprint
changes, and graded rows from before the change stop counting — a promote of the *new*
cascade needs fresh graded rows produced under it. That's why every edit sends you back to
shadow to re-earn the win, and why a stale or policy-drifted eval can never launder into a
satisfied gate.
</Callout>

## Related: availability fallback vs. quality escalation

The cascade above is a **quality** fallback — it escalates on a graded outcome. If instead
you want an **availability** fallback — try tiers in a deliberate order and serve the first
one that isn't cooling down — use a `select` with the `first_eligible` selector over an
ordered candidate list. It picks the first survivor of the `where` filter and never draws
randomness:

```yaml
routing:
  policy:
    version: 1
    root:
      select:
        from: { tiers: [mid, frontier, cheap] }   # deliberate order, not cost rank
        where: 'candidate.healthy'
        by: { selector: first_eligible }
        on_empty: { refuse: { status: 503, reason: "all_upstreams_cooling" } }
```

This cuts cost by preferring a cheaper-but-healthy tier and only failing over on a
cooldown — it does **not** grade the response, so pair it with the cascade above when you
also need a quality safety net.

<NextSteps items={[
  {title: "Escalate when the answer falls short", href: "/docs/build/routing-cascades", description: "Cascades in depth, including the break-even accept rate that decides whether one pays."},
  {title: "The policy language", href: "/docs/build/policy-language", description: "The five node kinds, the cascade grammar, and the outcome-scope inputs accept_when may read."},
  {title: "The honesty model", href: "/docs/concepts/honesty-model", description: "Why only judged-shadow labels can show routing skill — and why the promote gate ignores served-model labels."},
  {title: "Cut your bill further", href: "/docs/build/cost-optimization", description: "Combine model fallback with caching, output caps, and provider arbitrage."},
  {title: "Prove a cheaper tier", href: "/docs/calibration/prove-a-cheaper-tier", description: "Run an eval experiment to decide, on held-out data, whether the cheaper tier answers as well — before you enforce the fallback."}
]} />