# Turn routing on

Define two tiers, preview a routing policy on live traffic, check cost and answer quality, then turn it on.

Source: https://tokentriage.com/docs/build/routing

---
import Callout from "../../../components/Callout.astro";
import Details from "../../../components/Details.astro";
import NextSteps from "../../../components/NextSteps.astro";
import Screenshot from "../../../components/Screenshot.astro";
import TrustLadderDiagram from "../../../components/TrustLadderDiagram.astro";

Use this guide when TokenTriage is already proxying requests and you want to add a two-tier
routing policy. You will validate the policy, preview its choices on live traffic, check the
cost and answer quality, and then turn it on.

If this is your first routing setup, follow
[Your first routing rollout](/docs/get-started/first-routing-rollout). It includes a complete
configuration and sample requests. This page assumes the
[Quickstart](/docs/get-started/quickstart) is complete and requests are reaching the decision
log. Nothing here changes a response until the final section.

<TrustLadderDiagram />

## Define the tiers

A **tier** is one routing target: a name, an upstream, and the model that upstream serves.
This example uses an existing upstream named `anthropic`; replace that name if your config uses
a different one. Start in `shadow` mode so TokenTriage records the tier it would choose while
your application still receives the response it requested.

```yaml
mode: shadow

tiers: # ordered cheapest → most capable
  - name: cheap
    upstream: anthropic
    model: claude-haiku-4-5
  - name: frontier
    upstream: anthropic
    model: claude-opus-4-8
```

TokenTriage uses list order as the cost rank. Put the least expensive tier first and the most
capable tier last. The router also uses this order when it needs to move a request to a more
capable eligible tier.

## Write a first policy

The policy below sends prompts with a low complexity score to `cheap` and everything else to
`frontier`. The built-in classifier assigns `signal.complexity` a value from `0` to `1` on every
request.

```yaml
routing:
  default_tier: frontier
  policy:
    version: 1
    params:
      simple_cut:
        value: 0.35
        calibrate: { signal: complexity, direction: below }
    root:
      rules:
        - id: simple-to-cheap
          when: "signal.complexity < $simple_cut"
          to: { tier: cheap }
      default: { tier: frontier }
```

Three parts of the policy matter for the rollout:

- `default:` is required on every `rules` node. It sends requests that do not match a rule to
  `frontier`.
- `$simple_cut` is a named parameter. Naming it lets `calibrate` suggest a better value later.
- `routing.default_tier` is the fail-safe when the router cannot choose an eligible tier. It is
  required outside `ledger` mode and is separate from the `default:` branch above.

Use either `routing.policy` or the older `routing.rules` list. Setting both is a configuration
error.

Validate before you run. The command reads `./tokentriage.yaml` when you omit `--config`.

```console
$ tokentriage validate --config tokentriage.yaml
```

Successful validation ends with `OK: configuration is valid`. It catches unknown fields,
missing defaults, invalid policy dependencies, and references to tiers or upstreams that do not
exist.

<Details summary="The one-command demo, and why it stops short of enforcing">

`tokentriage run --simple ollama/llama3.1:8b --complex anthropic/claude-sonnet-4-6` starts a
two-tier preview without a config file. It creates `simple` and `complex` tiers, uses a
complexity threshold of `0.35`, and starts in `shadow` mode. The command accepts `anthropic`,
`openai`, `ollama`, and `groq` upstream shorthands. Use it to explore routing; use a config file
for a production rollout.

</Details>

## Preview one routing decision

Restart TokenTriage with the validated config:

```console
$ tokentriage run --config tokentriage.yaml
```

Send a few representative requests. Include both short prompts and prompts that require several
steps so you can see both branches.

Copy the `x-tt-request-id` response header from one request, then explain its routing decision:

```console
$ tokentriage policy explain --req-id <request-id> \
    --log ~/.tokentriage/decisions.jsonl
```

Check three fields before continuing:

- `decision` names the expected rule and tier.
- `path` ends at `rules[simple-to-cheap]` for a matching request.
- `trace` shows the complexity value and the `0.35` threshold the rule compared it with.

In `shadow` mode, these are previewed choices. The client still receives the original model's
response. [Routing explanation fields](/docs/build/routing-explanation-fields) covers the
complete output.

<Callout type="warning">

Signal names are not spell-checked. A typo such as `signal.complexty` still loads, but the
signal has no value and the rule never matches. Before promotion, confirm in `explain` that the
expected rule and complexity value appear.

</Callout>

## Calibrate from your own logs

The `0.35` threshold is a starting point, not a universal definition of a simple prompt. After
you have representative traffic, `calibrate` can suggest a threshold for the share of requests
you want to keep on `frontier`.

```console
$ tokentriage calibrate --config tokentriage.yaml \
    --log ~/.tokentriage/decisions.jsonl --target-frontier-pct 0.30
fitted complexity threshold: 0.66500
config-diff (effect when active):
  rule simple-to-cheap          complexity below: 0.35000 -> 0.66500
wrote candidate artifact: cal_20260809T183520Z_d5db85f2 (status=candidate)
activate it with: tokentriage calibrate --activate cal_20260809T183520Z_d5db85f2
```

Here, `--target-frontier-pct 0.30` asks for a threshold that keeps about 30% of the observed
traffic on `frontier`. The command saves its suggestion as a candidate; it does not change the
active policy or rewrite your YAML. Review the printed diff, configure
`classifier.calibration_file`, and then activate the candidate:

```console
$ tokentriage calibrate --activate cal_20260809T183520Z_d5db85f2 --config tokentriage.yaml
```

You can skip calibration for an initial preview and return after the first shadow report.
[Calibrate from logs](/docs/calibration/calibrate-from-logs) covers artifact storage, activation,
and rollback in detail.

## Total the saving in shadow mode

After you have sent a representative batch of traffic in `shadow` mode, total the previewed
decisions:

```console
$ tokentriage report --log ~/.tokentriage/decisions.jsonl
```

Read these lines first:

- **`routed rows`** is the number of requests with a routing decision and enough pricing data
  for the comparison.
- **`not-routing spend`** is what those requests would cost on the most capable tier.
- **`routed spend`** is what the selected tiers would cost.
- **`savings`** is the difference for those routed rows, not for every request in the log.

Then find `simple-to-cheap` under `BY RULE`. That row isolates the requests your policy moved to
the cheaper tier. Rows without enough pricing data appear under `EXCL` and do not count as zero.
Before you configure quality labels, the report may also say that routing quality cannot be
measured. That is expected; the cost totals are still useful.

## The honesty verdict

`report` answers a cost question. It cannot tell you whether the cheaper model answered well
enough, and a lower bill is not a saving when answer quality drops.

To compare quality, TokenTriage needs to grade both models on the same requests. **Shadow
duplication** sends sampled requests to the alternative tier outside the client path and grades
both answers. The copied answer is never returned to your application. Follow
[Check both models on the same requests](/docs/get-started/first-routing-rollout#5-check-both-models-on-the-same-requests)
to enable duplication with a daily spend cap, then run:

```console
$ tokentriage eval --quality shadow --log ~/.tokentriage/decisions.jsonl
```

`eval` compares the policy with the **zero router**, a blind mix of the same models at the same
average cost. Read the first verdict line as the next action:

| Verdict                                       | Next action                                                                                         |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `insufficient labels to judge routing`        | collect more judged request pairs, then run `eval` again                                            |
| `routing sits ON the observed-model frontier` | enable `shadow.duplicate`; one-sided labels cannot show which model was better for the same request |
| `routing does NOT beat zero-router`           | stay in `shadow` and adjust the policy, models, or threshold                                        |
| `routing BEATS zero-router by …`              | the quality evidence supports promotion                                                             |

The dashboard renders the same four states, and
[the honesty model](/docs/concepts/honesty-model#the-honesty-verdict-for-routing) explains the
full method and the other supported quality sources.

<Screenshot
  stem="routing-promote"
  url="localhost:9090/ui/routing"
  alt="The routing dashboard showing the trust ladder, a gated promotion, and the evidence behind the gate."
  caption="The routing dashboard keeps promotion blocked until the shadow evidence supports it (real dashboard, demo mode)."
/>

## Turn routing on safely

Promote only after `eval` reports `routing BEATS zero-router` for the current policy. The
dashboard gate checks that the labels are recent and belong to the same policy fingerprint. Do
not edit the policy or tier definitions between evaluation and promotion; an edit needs a new
shadow run.

Change one line:

```yaml
mode: route
```

Validate, restart, and send a request that matched `simple-to-cheap` in shadow mode:

```console
$ tokentriage validate --config tokentriage.yaml
$ tokentriage run --config tokentriage.yaml
```

The selected tier now serves the request, and `x-tt-decision` appears on the response. If you
enable `route` without matching shadow evidence, TokenTriage allows it but adds a
`route_without_shadow_evidence` badge to the dashboard and the audit row.

<Details summary="What happens when a tier or routing decision fails">

- **Cooldown routing is disabled by default.** If you enable it and the selected upstream is
  cooling, TokenTriage moves toward the more capable end of the tier list and chooses the first
  eligible tier. It does not move down to a cheaper tier.
- **Internal routing errors pass the original request through by default.** Set
  `routing.on_internal_error: default_tier` if you want an internal error to use the configured
  fail-safe tier instead.

</Details>

[What routing does not do](/docs/build/routing-limits) lists the rest, including the shapes
that record a decision without applying it.

<NextSteps
  items={[
    {
      title: "Which routing strategy",
      href: "/docs/build/routing-strategies",
      description:
        "Choose an approach when one complexity threshold is not enough.",
    },
    {
      title: "Preview a routing policy change",
      href: "/docs/build/routing-preview",
      description:
        "Re-evaluate past decisions under a draft and interpret the route and cost differences.",
    },
    {
      title: "The honesty model",
      href: "/docs/concepts/honesty-model",
      description:
        "Understand why both models must be judged on the same requests.",
    },
    {
      title: "Calibration & evals",
      href: "/docs/calibration/overview",
      description: "Tune the threshold and add a measured quality requirement.",
    },
  ]}
/>