GitHubBook a demoStart routing
Browse docs

Your first routing rollout

A hands-on walk up the trust ladder — ledger, shadow, eval, route — with the real commands and the exact output you see at each rung, ending in a promotion gated on evidence rather than hope.

Last updated

On this page

The Quickstart got you a costed request in ledger mode. This tutorial takes the next step you actually came for: cutting spend by routing, without flipping a switch and hoping. You climb one rung at a time — ledger → shadow → eval → route — and the evidence promotes you. Every command below is real, and so is every line of output.

The trust ladder: ledger, shadow, routeThree ascending rungs. Ledger observes and costs every request while changing nothing; shadow computes the would-route decision as a counterfactual without serving it; route enforces the decision. You climb one rung at a time, promoting only on evidence.ledgerobservecost every request, mutate nothingshadowprovecounterfactual — nothing servedrouteactenforce the cheaper decisionYou climb one rung at a time —promote only on evidence.

What you need to know to follow along: you’ll edit one tokentriage.yaml (copy the example config), restart the proxy between rungs, and send it some traffic. If you’d rather watch the whole journey on synthetic data first, run tokentriage demo --rich — the output in this tutorial is captured from exactly that.

Rung 1 — ledger: see the bill

You’re here after the Quickstart. The proxy costs every request and changes nothing. Roll the ledger up by model:

$ tokentriage top --by model
MODEL                        REQS  INPUT     OUTPUT  CACHE-R  CACHE-W  REASON  CACHE-SAV  COST        P50        P90        P99
acme/private-model-1         2     7008      3423    194      0        0       2.7%       -           -          -          -
anthropic/claude-haiku-4-5   81    204790    91558   19297    0        0       8.6%       $0.664510   $0.008663  $0.011766  $0.014075
anthropic/claude-opus-4-8    158   13328588  138619  140315   0        21312   1.0%       $70.123659  $0.043616  $0.100000  $10.000000
anthropic/claude-sonnet-4-5  146   297484    175631  29822    0        0       9.1%       $2.654674   $0.018203  $0.033462  $0.038363
openai/gpt-4o                145   313768    164971  32203    0        0       9.3%       $2.177474   $0.015441  $0.025441  $0.030319
openai/gpt-4o-mini           95    244418    89687   23123    0        0       8.6%       $0.092209   $0.000922  $0.001437  $0.001874
openai/o3                    127   304006    124211  97004    0        25998   24.2%      $1.618597   $0.012851  $0.020269  $0.024183
TOTAL                        783   14727438  798250  349382   0        47310   2.3%       $77.339009
COST excludes 2 unpriced rows of 783 (no price resolved; the $0 is a missing price, not $0 of spend)

Read the honesty in that output. P50/P90/P99 are the per-request cost percentiles for the group — not latency. And look at the top row: acme/private-model-1 shows - for cost, and the footer says COST excludes 2 unpriced rows. That model has no resolved price, so its cost is null and counted, never silently folded in as $0. (--by also takes agent, tool, step-class, and trace; window with --since 24h.)

Rung 2 — shadow: see what routing would save

To route, the proxy needs tiers (cheapest → most capable) and rules. Copy the example config, then set the master switch to shadow:

mode: shadow          # compute the would-route decision + its counterfactual cost; enforce nothing

Restart the proxy and send traffic again. In shadow mode every request still returns the same response, but the ledger now also records the tier routing would have chosen and what that path would have cost. report totals those counterfactual deltas:

$ tokentriage report
=== TokenTriage shadow-savings report ===
verdict: insufficient labels to judge routing — enable oracle sampling (classifier.oracle) or shadow duplication (shadow.duplicate)

routed rows:        292 of 784 (rows with a routing decision + would_usd)
not-routing spend:  $7.215266  (served on the top tier)
routed spend:       $6.700382  (under the routing decision)
savings:            $0.514884  (7.1% vs not routing; 57 of 292 rows routed cheaper)

BY RULE
RULE                REQS  EXCL  NO-ROUTE   ROUTED     SAVINGS    SAVINGS%
arbitrage-pair      20    0     $0.400420  $0.400413  $0.000006  0.0%
bulk-summarize      120   0     $2.853067  $2.853067  $0.000000  0.0%
default             2     0     $0.159642  $0.049888  $0.109754  68.8%
routed-down         40    0     $0.573925  $0.168801  $0.405124  70.6%
thin-evidence       50    0     $1.500398  $1.500398  $0.000000  0.0%
TOTAL               292   30    $7.215266  $6.700382  $0.514884  7.1%

Each row’s saving is would_usd[top-tier baseline] − would_usd[chosen tier]. A rule that only ever lands on the top tier — bulk-summarize here — shows exactly $0.000000: no downgrade, no saving, and the report says so rather than inventing one. The routed-down rule is where the money is (70.6%). Note also the EXCL column: 30 rows are honestly excluded from savings (a would-route tier that carries no price would deflate the baseline), never counted as free wins.

Rung 3 — eval: the honest verdict

eval answers the only question that licenses enforcing: does the router beat the zero-router baseline — the best you could do by blindly mixing the individual models — at the router’s own cost? With only the labels a plain ledger has, it refuses:

$ tokentriage eval
=== TokenTriage honesty eval ===
verdict: insufficient labels to judge routing — enable oracle sampling (classifier.oracle) or shadow duplication (shadow.duplicate)
note: served-model labels cannot demonstrate routing skill — see verdict
quality source: oracle (labeled iff outcome.oracle present; quality = oracle.score/100)
labeled rows:   5 of 784

That is CANNOT_MEASURE — the refusal, because only 5 of 784 rows carry a quality label: far too few to judge anything. And there’s a deeper limit even once you have enough labels: served-model labels only ever score the tier that actually served a row, so at best they place routing on the frontier (ON_FRONTIER) — they can never show a beat. To earn a BEAT-capable verdict you enable shadow duplication (shadow.duplicate — a sampled, cost-capped second call graded against the first). With those judged counterfactual labels present, eval can finally decide:

$ tokentriage eval --quality shadow
=== TokenTriage honesty eval ===
verdict: routing does NOT beat zero-router (Δ -0.0145) — keep shadow mode
quality source: shadow (labeled iff judged-shadow grade present (chosen_ok != nil); counterfactual per-cost verdict from both tiers' grades)
labeled rows:   69 of 784

FRONTIER  (per-cost verdict basis: router quality vs zero-router at the router's own cost)
  router quality:        0.9855  @ cost $0.026147
  zero-router @ router:  1.0000
  delta:                 -0.0145

This run says NOT_BEAT: the counterfactual ran with enough labels, but the delta didn’t clear the bar — so the honest instruction is keep shadow mode, not promote. That is a good outcome, not a failure: the tool just saved you from enforcing a downgrade that would have cost quality. A BEAT verdict reads routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication). See the honesty model for all four states and the exact lines each emits.

Rung 4 — route: enforce, once the evidence says so

When eval returns BEAT with enough fresh labels and a policy fingerprint that matches the running config, you’ve earned the top rung. Set the switch and restart:

mode: route           # now the chosen tier is enforced on the wire
How a request becomes a routing decisionA request's signals (complexity, model, agent, tools, budget) feed the routing policy, which first-match selects the cheapest tier that fits from tiers ordered cheapest to most capable. The mode gate then either enforces the decision in route mode or records it as a counterfactual in ledger and shadow mode. Every decision, served or counterfactual, appends one row with would_usd to the decision ledger.Requestcomplexity · modelagent · tools · budgetRouting policycheaptier 0frontiertier Nfirst-match · cheapest that fitsmode gateledger · shadowrouteroute → enforcerewrite model · upstreamledger / shadowobserve · would_usdDecision ledgerone row + would_usd — served or counterfactualoff-path — every decision recorded

Now the routing decision that was only recorded in shadow is served: simple traffic lands on the cheap tier, tool loops pin to frontier, and the decision headers (x-tt-decision-*) appear on the response. The ledger keeps costing everything exactly as before, so top and report still tell you the truth about what enforcing actually did.

localhost:9090/ui/routingDemo data
A real capture of the dashboard in demo mode: the trust ladder shows the shadow and oracle sources as CANNOT_MEASURE and the outcome source as ON_FRONTIER, and the Promote-to-route panel is gated — 'shadow evidence is not green' — behind an explicit acknowledgment that promoting anyway badges the decision 'routing without shadow evidence'. Watermarked DEMO DATA.
The promote gate on the dashboard — the same verdicts eval printed, and the gate refusing to open until shadow evidence is green (real dashboard, demo mode).

You’ve climbed the whole ladder: observed the bill, measured the would-route saving, judged whether routing is genuinely as good, and enforced only what the evidence backed. That refusal to promote on hope — the NOT_BEAT you saw at rung 3 — is the entire point of the tool.