Route by observed outcomes

An epsilon-greedy bandit learns from pass/fail grades. It sends most requests to the tier with the best pass rate so far and reserves a small share for testing other eligible tiers.

Unlike a fixed routing rule, its preference changes as new grades arrive. The selector does not grade answers itself: it depends on a working grading pipeline and enough traffic to build a useful history.

When to use it

Bandit routing is a good fit when all of these are true:

• You have a repeated workload and enough traffic to collect grades over time.
• You can judge each response as pass or fail with a criterion you trust.
• You have at least two tiers and can safely send some production traffic to alternatives.
• You can budget for the extra model and judge calls that produce the grades.

Use a simpler routing strategy when traffic is low, grading is unreliable, or every request must use your strongest tier. In bandit terminology, each selectable tier is an arm. This page uses tier except where an output field uses the statistical term.

Before you start

You need:

• at least two eligible routing tiers;
• mode: route, introduced through a first routing rollout;
• a configured judge; and
• a daily budget for shadow grading.

The rewards come from shadow duplication. On a sampled request, TokenTriage sends one extra request to a tier that did not serve the original, then grades both responses. The application still receives the response selected by the routing policy; the duplicate supplies evidence.

mode: route
shadow:
  duplicate:
    enabled: true
    sample_rate: 0.01
    max_usd_per_day: 5.00
    judge: oracle

All four fields under duplicate matter. The sampler runs only when it is enabled, the sample rate and daily cap are above zero, and the judge is available. Each sampled request adds one upstream model call and two grading calls, one for each response. max_usd_per_day caps this duplicate-and-grade work; it does not cap the application’s original requests.

To estimate the volume before the budget cap applies:

sampled requests per day = eligible requests per day × sample_rate

For example, 20,000 eligible requests per day at sample_rate: 0.01 produces about 200 sampled requests per day. How quickly a particular tier reaches min_samples also depends on which tiers are sampled and how many judge runs produce usable grades.

Configure the selector

routing:
  default_tier: frontier
  policy:
    version: 1
    seed: "bandit-1"
    root:
      select:
        from: { tiers: [cheap, mid, frontier] }
        where: "candidate.healthy"
        by:
          selector: epsilon_greedy
          params: { epsilon: 0.05, min_samples: 50, reward: shadow_grade }
        on_empty: { tier: frontier }

Parameter	Default	What it controls
epsilon	0.1	Share of requests assigned to exploration after at least one tier is learned. Before then, every request explores.
min_samples	1	Usable grades a tier needs before it can be selected during exploitation.
reward	none	Optional. The only accepted value is shadow_grade; it is validated but does not currently change the reward source.
min_quality_lcb	none	Optional calibrated lower-bound threshold. Tiers below it are excluded from selection.
quality_metric	none	Metric used by min_quality_lcb. Setting it without a quality floor is rejected.

The example raises min_samples to 50. The default of 1 allows a tier to become the leader after a single successful grade, which is rarely enough evidence for a production decision. Treat 50 as a starting example rather than a universal guarantee; tune it to your traffic, judge consistency, and tolerance for a longer warm-up.

How it chooses

With epsilon: 0.05, a learned selector uses the current leader on the exploitation branch for 95% of requests. The other 5% use a uniform draw over all eligible tiers. Because that draw can also land on the leader, the leader’s total selection probability is slightly higher than 95%.

Each request follows three steps:

1. Read the rewards. Each tier carries a reward_mean in [0,1] and a reward_count. A tier is learned when it has a mean and its count reaches min_samples.
2. Choose exploration or exploitation. If no tier is learned, the request explores and is tagged explore_under_min_samples. Otherwise a seeded draw against epsilon chooses the branch; an epsilon exploration is tagged explore_epsilon.
3. Select a tier. Exploitation takes the learned tier with the highest pass rate, breaking ties toward the lower configured tier rank. Exploration draws uniformly from the eligible tiers.

A tier is marked cold when it has no mean or has fewer than min_samples observations. Cold tiers remain reachable through exploration. A tier excluded by min_quality_lcb is not eligible for either branch.

What the reward measures

The reward answers a narrower question than “How good was this answer?”

• Only a successfully completed grade counts. A raw model response contributes nothing by itself.
• Each observation contributes 1 or 0, so reward_mean is a pass rate, not a rating or quality score.
• Rewards are attributed per tier, not per model or upstream. Models behind the same tier contribute to the same tier history.

This distinction matters. A pass-rate reward answers “Did this response meet the judge’s pass criterion?” It does not answer “Was this the best possible response?” A tier that reliably produces acceptable answers can outrank one that produces better answers less consistently.

Advanced: state snapshots and auditability

Reward state is updated outside the request path and read as a snapshot. Every decision records the version it used. ctx_ver contains a banditstate/r<N> component that identifies that reward snapshot, so you can investigate a surprising decision using what the router knew at the time rather than today’s state.

The seeded exploration draws are recorded as well, allowing the decision to be recomputed from the row.

What it costs

The selector itself does not add another model call to the request path. The additional spend comes from the shadow sampler: one upstream model call and two judge calls for every sampled request, bounded by max_usd_per_day.

Exploration has a separate product cost. It deliberately sends some production requests to a tier that does not currently have the best observed pass rate. After warm-up, epsilon: 0.1 uses the exploration branch on about one request in ten.

How to tell it is working

The score table shows the pass rate, evidence count, and reason for the selected tier:

      selector=epsilon_greedy
      score  cheap      0.8571  {count=140, reward_mean=0.8571}
      score  mid        0.9167  {cold=1, count=12, reward_mean=0.9167}
      score  frontier   0.9400  {count=300, exploit=1, reward_mean=0.94}

Row	Interpretation
cheap	Learned, but its 0.8571 pass rate is below the leader’s.
mid	Still cold: 12 observations is below min_samples: 50, so it cannot win exploitation yet.
frontier	Highest pass rate among learned tiers. exploit=1 shows that the exploitation branch selected it.

An exploration that selects cheap instead records explore=1 and explore_epsilon=1 on the cheap row, with no exploit term.

Look for these signals over time:

• count increases as shadow grading succeeds;
• cold disappears from tiers as they reach min_samples; and
• after warm-up, the exploration branch settles near epsilon and exploitation near 1 - epsilon.

If counts remain at zero, the selector has no new evidence. Check the shadow sampler and judge before changing the routing policy.

Advanced: reconstruct the selection probability

Let N be the number of eligible tiers. After at least one tier is learned, the current leader’s selection probability is (1 - epsilon) + (epsilon / N), because exploration can also select it. Every other eligible tier has probability epsilon / N. Before any tier is learned, the selector draws uniformly and each eligible tier has probability 1 / N.

The decision row records the eligible count and the seeded draws because those facts cannot always be reconstructed later. They are useful when evaluating a different policy against logged decisions, often called off-policy analysis.

Troubleshooting

Symptom	Likely cause	What to check
Counts remain at zero	Duplication is off, the daily cap is exhausted, or the judge is failing.	Check the sampler counters, the shadow duplication enabled boot log, and judge errors.
The router exploits too early	min_samples is too low for the workload.	Raise it and allow enough time for every tier to collect usable grades.
A high-scoring tier is still cold	Its count has not reached min_samples.	Compare its count term with the configured threshold.
A tier never appears in exploration	A quality floor or the where filter excluded it.	Inspect the exclusion terms and the candidate’s health and quality inputs.
The pass-rate leader is not the tier you consider best	The judge’s pass criterion rewards acceptability, not relative excellence.	Revisit the grading criterion and validate it against the outcome you want to optimize.

Operational limits in this version

The de-duplication set of seen request IDs grows without bound over a long-running process, and each catch-up pass re-reads the decision log from the beginning. If you expect a bandit to run for months or retain a large decision log, measure memory use and catch-up time against your expected traffic instead of assuming the defaults will remain inexpensive.

What this page does not cover

A bandit can optimize only the pass/fail grades it receives. Building a quality measure you can defend, with stated confidence and coverage, belongs to calibration and evals. You can then use min_quality_lcb to exclude tiers whose calibrated lower bound does not meet your requirement; the floor controls eligibility, while the bandit chooses among the remaining tiers.