# Route by observed outcomes
Use graded outcomes to learn which model tier performs best. Configure epsilon-greedy routing, budget shadow grading, and verify each decision.
Source: https://tokentriage.com/docs/build/routing-bandit
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
An epsilon-greedy bandit learns from pass/fail grades. It sends most requests to the tier with
the best pass rate so far and reserves a small share for testing other eligible tiers.
Unlike a fixed routing rule, its preference changes as new grades arrive. The selector does not
grade answers itself: it depends on a working grading pipeline and enough traffic to build a
useful history.
## When to use it
Bandit routing is a good fit when all of these are true:
- You have a repeated workload and enough traffic to collect grades over time.
- You can judge each response as pass or fail with a criterion you trust.
- You have at least two tiers and can safely send some production traffic to alternatives.
- You can budget for the extra model and judge calls that produce the grades.
Use a [simpler routing strategy](/docs/build/routing-strategies) when traffic is low, grading is
unreliable, or every request must use your strongest tier. In bandit terminology, each selectable
tier is an *arm*. This page uses **tier** except where an output field uses the statistical term.
## Before you start
You need:
- at least two eligible [routing tiers](/docs/build/routing-pools);
- `mode: route`, introduced through a [first routing rollout](/docs/get-started/first-routing-rollout);
- a [configured judge](/docs/judges/configure-a-judge); and
- a daily budget for shadow grading.
The rewards come from **shadow duplication**. On a sampled request, TokenTriage sends one extra
request to a tier that did not serve the original, then grades both responses. The application
still receives the response selected by the routing policy; the duplicate supplies evidence.
```yaml
mode: route
shadow:
duplicate:
enabled: true
sample_rate: 0.01
max_usd_per_day: 5.00
judge: oracle
```
All four fields under `duplicate` matter. The sampler runs only when it is enabled, the sample
rate and daily cap are above zero, and the judge is available. Each sampled request adds one
upstream model call and two grading calls, one for each response. `max_usd_per_day` caps this
duplicate-and-grade work; it does not cap the application's original requests.
To estimate the volume before the budget cap applies:
```text
sampled requests per day = eligible requests per day × sample_rate
```
For example, 20,000 eligible requests per day at `sample_rate: 0.01` produces about 200 sampled
requests per day. How quickly a particular tier reaches `min_samples` also depends on which tiers
are sampled and how many judge runs produce usable grades.
Until at least one eligible tier has `min_samples` usable grades, the selector explores on
**every** request, regardless of `epsilon`. The decision row records
`explore_under_min_samples` during this warm-up. With a low shadow sample rate or low traffic,
warm-up can take much longer than the sample-rate percentage alone suggests.
## Configure the selector
```yaml
routing:
default_tier: frontier
policy:
version: 1
seed: "bandit-1"
root:
select:
from: { tiers: [cheap, mid, frontier] }
where: "candidate.healthy"
by:
selector: epsilon_greedy
params: { epsilon: 0.05, min_samples: 50, reward: shadow_grade }
on_empty: { tier: frontier }
```
| Parameter | Default | What it controls |
|---|---|---|
| `epsilon` | `0.1` | Share of requests assigned to exploration after at least one tier is learned. Before then, every request explores. |
| `min_samples` | `1` | Usable grades a tier needs before it can be selected during exploitation. |
| `reward` | none | Optional. The only accepted value is `shadow_grade`; it is validated but does not currently change the reward source. |
| `min_quality_lcb` | none | Optional calibrated lower-bound threshold. Tiers below it are excluded from selection. |
| `quality_metric` | none | Metric used by `min_quality_lcb`. Setting it without a quality floor is rejected. |
The example raises `min_samples` to `50`. The default of `1` allows a tier to become the leader
after a single successful grade, which is rarely enough evidence for a production decision.
Treat `50` as a starting example rather than a universal guarantee; tune it to your traffic,
judge consistency, and tolerance for a longer warm-up.
## How it chooses
With `epsilon: 0.05`, a learned selector uses the current leader on the exploitation branch for
95% of requests. The other 5% use a uniform draw over all eligible tiers. Because that draw can
also land on the leader, the leader's total selection probability is slightly higher than 95%.
Each request follows three steps:
1. **Read the rewards.** Each tier carries a `reward_mean` in `[0,1]` and a `reward_count`. A
tier is *learned* when it has a mean and its count reaches `min_samples`.
2. **Choose exploration or exploitation.** If no tier is learned, the request explores and is
tagged `explore_under_min_samples`. Otherwise a seeded draw against `epsilon` chooses the
branch; an epsilon exploration is tagged `explore_epsilon`.
3. **Select a tier.** Exploitation takes the learned tier with the highest pass rate, breaking
ties toward the lower configured tier rank. Exploration draws uniformly from the eligible
tiers.
A tier is marked `cold` when it has no mean or has fewer than `min_samples` observations. Cold
tiers remain reachable through exploration. A tier excluded by `min_quality_lcb` is not eligible
for either branch.
## What the reward measures
The reward answers a narrower question than "How good was this answer?"
- Only a successfully completed grade counts. A raw model response contributes nothing by
itself.
- Each observation contributes `1` or `0`, so `reward_mean` is a **pass rate**, not a rating or
quality score.
- Rewards are attributed **per tier**, not per model or upstream. Models behind the same tier
contribute to the same tier history.
This distinction matters. A pass-rate reward answers "Did this response meet the judge's pass
criterion?" It does not answer "Was this the best possible response?" A tier that reliably
produces acceptable answers can outrank one that produces better answers less consistently.
Reward state is updated outside the request path and read as a snapshot. Every decision records
the version it used. `ctx_ver` contains a `banditstate/r` component that identifies that reward
snapshot, so you can investigate a surprising decision using what the router knew at the time
rather than today's state.
The seeded exploration draws are recorded as well, allowing the decision to be recomputed from
the row.
## What it costs
The selector itself does not add another model call to the request path. The additional spend
comes from the shadow sampler: one upstream model call and two judge calls for every sampled
request, bounded by `max_usd_per_day`.
Exploration has a separate product cost. It deliberately sends some production requests to a
tier that does not currently have the best observed pass rate. After warm-up, `epsilon: 0.1`
uses the exploration branch on about one request in ten.
## How to tell it is working
The score table shows the pass rate, evidence count, and reason for the selected tier:
```console
selector=epsilon_greedy
score cheap 0.8571 {count=140, reward_mean=0.8571}
score mid 0.9167 {cold=1, count=12, reward_mean=0.9167}
score frontier 0.9400 {count=300, exploit=1, reward_mean=0.94}
```
| Row | Interpretation |
|---|---|
| `cheap` | Learned, but its `0.8571` pass rate is below the leader's. |
| `mid` | Still cold: `12` observations is below `min_samples: 50`, so it cannot win exploitation yet. |
| `frontier` | Highest pass rate among learned tiers. `exploit=1` shows that the exploitation branch selected it. |
An exploration that selects `cheap` instead records `explore=1` and `explore_epsilon=1` on the
`cheap` row, with no `exploit` term.
Look for these signals over time:
- `count` increases as shadow grading succeeds;
- `cold` disappears from tiers as they reach `min_samples`; and
- after warm-up, the exploration branch settles near `epsilon` and exploitation near
`1 - epsilon`.
If counts remain at zero, the selector has no new evidence. Check the shadow sampler and judge
before changing the routing policy.
Let `N` be the number of eligible tiers. After at least one tier is learned, the current leader's
selection probability is `(1 - epsilon) + (epsilon / N)`, because exploration can also select
it. Every other eligible tier has probability `epsilon / N`. Before any tier is learned, the
selector draws uniformly and each eligible tier has probability `1 / N`.
The decision row records the eligible count and the seeded draws because those facts cannot
always be reconstructed later. They are useful when evaluating a different policy against logged
decisions, often called off-policy analysis.
## Troubleshooting
| Symptom | Likely cause | What to check |
|---|---|---|
| Counts remain at zero | Duplication is off, the daily cap is exhausted, or the judge is failing. | Check the sampler counters, the `shadow duplication enabled` boot log, and judge errors. |
| The router exploits too early | `min_samples` is too low for the workload. | Raise it and allow enough time for every tier to collect usable grades. |
| A high-scoring tier is still `cold` | Its count has not reached `min_samples`. | Compare its `count` term with the configured threshold. |
| A tier never appears in exploration | A quality floor or the `where` filter excluded it. | Inspect the exclusion terms and the candidate's health and quality inputs. |
| The pass-rate leader is not the tier you consider best | The judge's pass criterion rewards acceptability, not relative excellence. | Revisit the grading criterion and validate it against the outcome you want to optimize. |
The de-duplication set of seen request IDs grows without bound over a long-running process, and
each catch-up pass re-reads the decision log from the beginning. If you expect a bandit to run for
months or retain a large decision log, measure memory use and catch-up time against your expected
traffic instead of assuming the defaults will remain inexpensive.
## What this page does not cover
A bandit can optimize only the pass/fail grades it receives. Building a quality measure you can
defend, with stated confidence and coverage, belongs to
[calibration and evals](/docs/calibration/overview). You can then use `min_quality_lcb` to exclude
tiers whose calibrated lower bound does not meet your requirement; the floor controls eligibility,
while the bandit chooses among the remaining tiers.