Cut cost with a model fallback — TokenTriage Docs

Serve every request from a cheap tier first, grade the cheap response inline, and escalate to a stronger tier only for the rows the cheap tier can’t handle. The mechanism is a cascade node in the routing policy: stages try tiers in order, an accept_when predicate grades each attempt, and a required final tier is the escalation floor. You promote it to route mode only after shadow evidence proves the cheaper path holds quality.

Prerequisites

A running TokenTriage with at least two tiers ordered cheapest → most capable (e.g. cheap and frontier) and a decision log configured (ledger.decision_log.path).
The oracle judge configured (classifier.oracle) — the inline cascade judge reuses it. Without a wired judge, a judge: true stage yields outcome.judge.available = false and the accept_when below escalates rather than inventing a verdict.
shadow.duplicate available for the promotion step (below).

1. Write the cheap-first cascade policy

routing.policy is the v2 routing form and is mutually exclusive with the legacy routing.rules list — a config sets exactly one. Make the root a cascade: try cheap, run the inline judge, and escalate to frontier only when the attempt isn’t a clean, non-refusing, judge-approved 200.

routing:
  default_tier: frontier          # fail-safe target on classifier error (never a downgrade)
  policy:
    version: 1
    root:
      cascade:
        stages:
          - id: try-cheap
            to: { tier: cheap }
            judge: true           # run the inline shadow judge before accept_when
            accept_when: 'outcome.status = 200 and outcome.refusal_markers = 0 and outcome.judge.available and outcome.judge.ok'
        final: { tier: frontier } # always accepted — the escalation floor

Only outcome-scope inputs are legal inside accept_when: outcome.status, outcome.refusal_markers, outcome.judge.ok, outcome.judge.available, outcome.finish_reason, outcome.latency_ms, and the token/byte facts. An absent outcome.judge.available reads false, so the ... and outcome.judge.available and outcome.judge.ok form escalates when the judge is down — it never accepts on a fabricated verdict.

Validate it strictly before running:

$ tokentriage validate --config configs/tokentriage.yaml

2. Climb the ladder: run it in shadow first

Don’t flip straight to route. Set the top-level mode: shadow and turn on shadow duplication so TokenTriage grades the cheaper tier against the stronger one on the same query — the only data that can prove the cheaper path holds quality:

mode: shadow                      # observe + grade the counterfactual; client bytes unchanged
shadow:
  duplicate:
    enabled: true
    sample_rate: 0.01
    max_usd_per_day: 5.00         # hard cap; the sampler stops when exhausted
    judge: oracle

In shadow mode the client still sees identical bytes — you’re accumulating graded rows, not enforcing. Let traffic run, then read the shadow-savings total:

$ tokentriage report --log ~/.tokentriage/decisions.jsonl
=== TokenTriage shadow-savings report ===

routed rows:        560 of 560 (rows with a routing decision + would_usd)
not-routing spend:  $3.731000  (served on the top tier)
routed spend:       $1.305850  (under the routing decision)
savings:            $2.425150  (65.0% vs not routing; 455 of 560 rows routed cheaper)

(Illustrative output — your totals come from your own log.)

3. Read the honesty verdict before you promote

report shows the savings; eval --quality shadow answers the question that licenses enforcement — does the cheaper routing actually beat a blind model mix at its own cost? Only the shadow (counterfactual) source can return a BEATS verdict:

$ tokentriage eval --quality shadow --log ~/.tokentriage/decisions.jsonl
=== TokenTriage honesty eval ===
verdict: routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication)
quality source: shadow (labeled iff judged-shadow grade present (chosen_ok != nil); counterfactual per-cost verdict from both tiers' grades)
labeled rows:   8 of 8
… (per-tier TIER POINTS table elided)

FRONTIER  (per-cost verdict basis: router quality vs zero-router at the router's own cost)
  router quality:        1.0000  @ cost $5.500000
  zero-router @ router:  0.7500
  delta:                 +0.2500

If instead you see insufficient labels to judge routing or a served-model source sitting ON the observed-model frontier (Δ 0.0000), you do not have a promotable win yet — oracle/outcome labels score only the tier that served each row and can never show routing skill. Keep it in shadow and let judged-shadow grades accumulate.

4. Promote to route mode

Once the verdict is a BEATS and the promote gate is satisfied, enforce it:

mode: route                       # enforce the cascade — cheap first, escalate on failure

You should see the cheap tier serving the accepted rows and only the failing rows carrying a frontier attempt. Inspect a decision’s cascade with policy explain, which prints cascade: attempts=N final=<tier> and one line per stage (id tier model accepted cost=$…):

$ tokentriage policy explain --req-id <req-id> --log ~/.tokentriage/decisions.jsonl

(The accept_when reads and the block-level judge_cost are recorded in the decision-log cascade block itself — the JSON row / dashboard drawer — rather than printed by this command.)

The cascade above is a quality fallback — it escalates on a graded outcome. If instead you want an availability fallback — try tiers in a deliberate order and serve the first one that isn’t cooling down — use a select with the first_eligible selector over an ordered candidate list. It picks the first survivor of the where filter and never draws randomness:

routing:
  policy:
    version: 1
    root:
      select:
        from: { tiers: [mid, frontier, cheap] }   # deliberate order, not cost rank
        where: 'candidate.healthy'
        by: { selector: first_eligible }
        on_empty: { refuse: { status: 503, reason: "all_upstreams_cooling" } }

This cuts cost by preferring a cheaper-but-healthy tier and only failing over on a cooldown — it does not grade the response, so pair it with the cascade above when you also need a quality safety net.

Prerequisites

1. Write the cheap-first cascade policy

2. Climb the ladder: run it in shadow first

3. Read the honesty verdict before you promote

4. Promote to route mode

Related: availability fallback vs. quality escalation