Routing & the trust ladder

Routing is opt-in and earned. You don’t flip it on and hope — you climb the trust ladder one rung at a time, and the tooling refuses to claim a win it can’t prove.

Start routing (then slow down)

You can front a cheap tier and a frontier tier with one command — this boots into shadow mode with a built-in complexity rule (deliberately not route: the trust ladder is enforced by the defaults, so nothing is re-routed until you promote it):

$ tokentriage run --simple ollama/llama3.1:8b --complex anthropic/claude-sonnet-4-6

A real config names upstreams, orders tiers cheapest → most capable, and matches with first-match-wins routing.rules. Validate it strictly before you run:

$ tokentriage validate --config configs/tokentriage.example.yaml

Key routing settings: default_tier (unmatched requests, and the fail-safe target on classifier error — never a downgrade on error), on_internal_error: passthrough (the default), overrides.tier_header: x-tt-tier (a client may force a tier; always honored in route mode, logged in all modes), cache_affinity, cooldown, and allow_cross_dialect: false (opt-in translation, off by default).

The routing strategies

Simple first-match routing.rules is the entry point. For richer routing, the typed policy language composes a set of strategy recipes — ten driven end-to-end through the real proxy, with ordered fallback covered by selector and evaluator tests, plus expected-cost routing as a follow-on:

• Complexity thresholds — route by a signal.complexity score vs. a calibratable cut.
• Semantic topic pins — route by content similarity (signal.semantic.<topic>).
• Cost-aware selection — cheapest_meeting an SLA, or a weighted_objective.
• Load-balanced pools — weighted / least-latency distribution over a pool.
• A/B / canary splits — seeded, sticky traffic splits.
• Time-of-day — route by ctx.time.* (off-peak → cheap).
• Budget degrade → refuse — ctx.budget.* conditions with a typed refusal.
• Ordered fallback — first_eligible over an ordered candidate list.
• Judge-graded cascades — try a cheap tier, grade the response, escalate on failure.
• Learned / bandit routing — epsilon_greedy over rewarded tiers.
• Expected-cost routing (est_cost) — price a request at a reference tier (a labeled estimate).

Any of these compose in one policy, and adding a strategy is a primitive registration — no grammar change. The full grammar and worked examples are in The routing policy language.

Calibrate from your own logs

Routing thresholds aren’t magic numbers. calibrate fits them from your decision log using RouteLLM’s quantile method and writes a versioned candidate artifact — nothing changes routing until you explicitly --activate it:

$ tokentriage calibrate --config configs/tokentriage.example.yaml --target-frontier-pct 0.4
param simple_cut  complexity below: 0.35000 -> 0.73700

localhost:9090/ui/calibrationDemo data

Shadow — prove the savings first

In shadow mode the proxy computes the would-route decision and reprices it without touching any response. report totals the counterfactual savings, computed as would_usd[top tier] − would_usd[chosen tier]:

$ tokentriage report --log ~/.tokentriage/decisions.jsonl
=== TokenTriage shadow-savings report ===
routed rows:        560 of 560
not-routing spend:  $3.731000  (served on the top tier)
routed spend:       $1.305850  (under the routing decision)
savings:            $2.425150  (455 of 560 rows routed cheaper)

The honesty verdict

This is the part most routers won’t show you. eval asks the only question that matters: does the router’s realized quality exceed the zero-router baseline at the router’s own cost? Pick the quality source with --quality oracle | outcome | shadow.

The verdict is honest about what your labels can prove. Served-model labels can only place routing on the frontier; only judged shadow duplication can show a beat; and with too few labels, eval refuses:

$ tokentriage eval --quality shadow --log ~/.tokentriage/decisions.jsonl
verdict: routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication)

$ tokentriage eval --log ~/.tokentriage/decisions.jsonl
verdict: routing sits ON the observed-model frontier (Δ 0.0000) — served-model labels
score only the tier that served each row, so routing cannot be shown to beat a blind
model mix. Enable shadow duplication (shadow.duplicate) for a counterfactual verdict.

The dashboard renders the identical verdict, and promoting from shadow to route is gated on green shadow evidence — a move to route without it wears a loud route without shadow evidence badge.

localhost:9090/ui/routingDemo data

Honest disclosures

A {model, upstream} policy target does not enforce in route mode today (it needs a tier; a tier-less model target safely drops to passthrough). est_cost and tokentriage policy preview are honest estimates, with their approximation labels always emitted. Both are covered in the policy-language spec.