Turn routing on

Use this guide when TokenTriage is already proxying requests and you want to add a two-tier routing policy. You will validate the policy, preview its choices on live traffic, check the cost and answer quality, and then turn it on.

If this is your first routing setup, follow Your first routing rollout. It includes a complete configuration and sample requests. This page assumes the Quickstart is complete and requests are reaching the decision log. Nothing here changes a response until the final section.

Define the tiers

A tier is one routing target: a name, an upstream, and the model that upstream serves. This example uses an existing upstream named anthropic; replace that name if your config uses a different one. Start in shadow mode so TokenTriage records the tier it would choose while your application still receives the response it requested.

mode: shadow

tiers: # ordered cheapest → most capable
  - name: cheap
    upstream: anthropic
    model: claude-haiku-4-5
  - name: frontier
    upstream: anthropic
    model: claude-opus-4-8

TokenTriage uses list order as the cost rank. Put the least expensive tier first and the most capable tier last. The router also uses this order when it needs to move a request to a more capable eligible tier.

Write a first policy

The policy below sends prompts with a low complexity score to cheap and everything else to frontier. The built-in classifier assigns signal.complexity a value from 0 to 1 on every request.

routing:
  default_tier: frontier
  policy:
    version: 1
    params:
      simple_cut:
        value: 0.35
        calibrate: { signal: complexity, direction: below }
    root:
      rules:
        - id: simple-to-cheap
          when: "signal.complexity < $simple_cut"
          to: { tier: cheap }
      default: { tier: frontier }

Three parts of the policy matter for the rollout:

• default: is required on every rules node. It sends requests that do not match a rule to frontier.
• $simple_cut is a named parameter. Naming it lets calibrate suggest a better value later.
• routing.default_tier is the fail-safe when the router cannot choose an eligible tier. It is required outside ledger mode and is separate from the default: branch above.

Use either routing.policy or the older routing.rules list. Setting both is a configuration error.

Validate before you run. The command reads ./tokentriage.yaml when you omit --config.

$ tokentriage validate --config tokentriage.yaml

Successful validation ends with OK: configuration is valid. It catches unknown fields, missing defaults, invalid policy dependencies, and references to tiers or upstreams that do not exist.

The one-command demo, and why it stops short of enforcing

tokentriage run --simple ollama/llama3.1:8b --complex anthropic/claude-sonnet-4-6 starts a two-tier preview without a config file. It creates simple and complex tiers, uses a complexity threshold of 0.35, and starts in shadow mode. The command accepts anthropic, openai, ollama, and groq upstream shorthands. Use it to explore routing; use a config file for a production rollout.

Preview one routing decision

Restart TokenTriage with the validated config:

$ tokentriage run --config tokentriage.yaml

Send a few representative requests. Include both short prompts and prompts that require several steps so you can see both branches.

Copy the x-tt-request-id response header from one request, then explain its routing decision:

$ tokentriage policy explain --req-id <request-id> \
    --log ~/.tokentriage/decisions.jsonl

Check three fields before continuing:

• decision names the expected rule and tier.
• path ends at rules[simple-to-cheap] for a matching request.
• trace shows the complexity value and the 0.35 threshold the rule compared it with.

In shadow mode, these are previewed choices. The client still receives the original model’s response. Routing explanation fields covers the complete output.

Calibrate from your own logs

The 0.35 threshold is a starting point, not a universal definition of a simple prompt. After you have representative traffic, calibrate can suggest a threshold for the share of requests you want to keep on frontier.

$ tokentriage calibrate --config tokentriage.yaml \
    --log ~/.tokentriage/decisions.jsonl --target-frontier-pct 0.30
fitted complexity threshold: 0.66500
config-diff (effect when active):
  rule simple-to-cheap          complexity below: 0.35000 -> 0.66500
wrote candidate artifact: cal_20260809T183520Z_d5db85f2 (status=candidate)
activate it with: tokentriage calibrate --activate cal_20260809T183520Z_d5db85f2

Here, --target-frontier-pct 0.30 asks for a threshold that keeps about 30% of the observed traffic on frontier. The command saves its suggestion as a candidate; it does not change the active policy or rewrite your YAML. Review the printed diff, configure classifier.calibration_file, and then activate the candidate:

$ tokentriage calibrate --activate cal_20260809T183520Z_d5db85f2 --config tokentriage.yaml

You can skip calibration for an initial preview and return after the first shadow report. Calibrate from logs covers artifact storage, activation, and rollback in detail.

Total the saving in shadow mode

After you have sent a representative batch of traffic in shadow mode, total the previewed decisions:

$ tokentriage report --log ~/.tokentriage/decisions.jsonl

Read these lines first:

• routed rows is the number of requests with a routing decision and enough pricing data for the comparison.
• not-routing spend is what those requests would cost on the most capable tier.
• routed spend is what the selected tiers would cost.
• savings is the difference for those routed rows, not for every request in the log.

Then find simple-to-cheap under BY RULE. That row isolates the requests your policy moved to the cheaper tier. Rows without enough pricing data appear under EXCL and do not count as zero. Before you configure quality labels, the report may also say that routing quality cannot be measured. That is expected; the cost totals are still useful.

The honesty verdict

report answers a cost question. It cannot tell you whether the cheaper model answered well enough, and a lower bill is not a saving when answer quality drops.

To compare quality, TokenTriage needs to grade both models on the same requests. Shadow duplication sends sampled requests to the alternative tier outside the client path and grades both answers. The copied answer is never returned to your application. Follow Check both models on the same requests to enable duplication with a daily spend cap, then run:

$ tokentriage eval --quality shadow --log ~/.tokentriage/decisions.jsonl

eval compares the policy with the zero router, a blind mix of the same models at the same average cost. Read the first verdict line as the next action:

Verdict	Next action
insufficient labels to judge routing	collect more judged request pairs, then run eval again
routing sits ON the observed-model frontier	enable shadow.duplicate; one-sided labels cannot show which model was better for the same request
routing does NOT beat zero-router	stay in shadow and adjust the policy, models, or threshold
routing BEATS zero-router by …	the quality evidence supports promotion

The dashboard renders the same four states, and the honesty model explains the full method and the other supported quality sources.

localhost:9090/ui/routingDemo data

Turn routing on safely

Promote only after eval reports routing BEATS zero-router for the current policy. The dashboard gate checks that the labels are recent and belong to the same policy fingerprint. Do not edit the policy or tier definitions between evaluation and promotion; an edit needs a new shadow run.

Change one line:

mode: route

Validate, restart, and send a request that matched simple-to-cheap in shadow mode:

$ tokentriage validate --config tokentriage.yaml
$ tokentriage run --config tokentriage.yaml

The selected tier now serves the request, and x-tt-decision appears on the response. If you enable route without matching shadow evidence, TokenTriage allows it but adds a route_without_shadow_evidence badge to the dashboard and the audit row.

What happens when a tier or routing decision fails

• Cooldown routing is disabled by default. If you enable it and the selected upstream is cooling, TokenTriage moves toward the more capable end of the tier list and chooses the first eligible tier. It does not move down to a cheaper tier.
• Internal routing errors pass the original request through by default. Set routing.on_internal_error: default_tier if you want an internal error to use the configured fail-safe tier instead.

What routing does not do lists the rest, including the shapes that record a decision without applying it.