# Split traffic for an experiment

Send a slice of traffic down a different path, keep each conversation on one side, and recompute any draw from the row.

Source: https://tokentriage.com/docs/build/routing-splits

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import TrafficSplitDiagram from '../../../components/TrafficSplitDiagram.astro';

Every other strategy tries to make the best decision for each request. A split does the
opposite: it decides at random so that two paths get comparable traffic, which is
the only way to find out whether the change you are considering is an improvement.

This page builds a canary and an A/B, explains how the draw works and why the same
conversation keeps landing on the same side, and covers what you have to do yourself to
measure the result.

You need a policy that loads and two things worth comparing;
[Turn routing on](/docs/build/routing) covers the setup.

## Ship a change to a slice first

A canary sends most traffic down the path you trust and a small share down the new one.

```yaml
routing:
  default_tier: frontier
  policy:
    version: 1
    seed: "canary-2026-08"
    root:
      split:
        sticky: trace
        arms:
          - { id: control, weight: 95, to: { tier: frontier } }
          - { id: canary, weight: 5, to: { tier: mid } }
```

Weights are relative positive integers. `95` and `5` gives nineteen to one; so does `19` and
`1`. They do not have to sum to a hundred, and a weight of zero is a load error rather than a
parked arm. Remove the arm instead.

An arm's `to:` is a whole node, not a leaf, so an arm can hold an entire sub-policy. That is
how you A/B two *strategies* rather than two tiers: put a cascade on one side and a plain tier
on the other.

<Callout type="warning">

A document containing a split requires `seed:`, and validation refuses one
without it. Pick a string and keep it: changing the seed reshuffles which
conversations are in the canary, which throws away whatever comparison you had
accumulated.

</Callout>

## How an arm is picked

The word "split" suggests a coin flip. It is not one, and the difference is what makes the
result usable.

<TrafficSplitDiagram />

Read it left to right. Three inputs go in: the seed you set, the sticky key for this request,
and the node's own identity within the tree. They are hashed into a single number, which is
reduced onto the cumulative weight bands. There is no clock, no counter, and no randomness
source, so the same three inputs always produce the same arm.

The node's identity is in the hash for a reason: two splits with identical arms and weights
draw independently rather than agreeing with each other, so nesting one experiment inside
another does not correlate them.

## Stickiness, and what happens without a trace id

`sticky:` decides which key goes into the hash. It takes `trace`, the default, or `request`.

With `sticky: trace`, every request carrying the same trace id lands on the same arm. That is
what you want for anything conversational: a canary that flips a user between models
mid-conversation is measuring the flipping, not the model.

With `sticky: request`, every request is drawn independently. Use it for stateless,
single-shot workloads where a conversation is not a thing.

**A request with no trace id** cannot be sticky to one, so the draw falls back to the request
id — and the row records the scope it actually used rather than the one you asked for. If your
rows say `scope=request` under `sticky: trace`, your clients are not sending
[trace headers](/docs/reference/headers).

## Read the result

The draw is on the row, with enough detail to recompute:

```console
$ tokentriage policy explain --req-id tt_split --log ~/.tokentriage/decisions.jsonl
req_id:     tt_split
policy_rev: abf1f12
decision:   rule=default tier=mid upstream=openai
path:       root → split → arm:canary
trace:
  • split                        split (arm:canary)
  roll split#1 scope=trace key=51bcc7b value=0.462589 → arm:canary
```

`path` names the arm. The roll line names the node, the scope, a short hash of the sticky key,
the drawn value and the arm it produced. Anyone holding the row's trace id and the seed can recompute that
draw, which is what makes a disputed assignment checkable rather than a matter of trust.

<Details summary="Making the arms show up in the savings report">

`tokentriage report` groups by rule id, and a split arm does not set one — rows land under
whatever rule matched above the split, or under `default`. To get per-arm totals, make each
arm's target a `rules` node with one unconditional entry whose id names the arm:

```yaml
arms:
  - id: canary
    weight: 5
    to:
      rules:
        - { id: arm-canary, to: { tier: mid } }
      default: { tier: mid }
```

The arm id now appears in the `BY RULE` table beside its spend and saving. It costs one extra
node per arm and makes the experiment legible in the reporting you already run.

</Details>

## What can go wrong

**Editing the policy while the experiment runs.** Changing the policy changes its fingerprint,
which is what the promote gate keys on, so accumulated shadow evidence stops counting toward
promoting the new version. Finish the comparison, then edit.

**Expecting the shares to be exact.** The draw is uniform over conversations, not over
requests. A single very chatty trace lands wholly on one arm, so a 95/5 split can look like
90/10 by request count on low volume. Judge it by trace count, and give it enough traces that
the ratio settles.

**Treating a split as the measurement.** The split creates comparable traffic. It does not tell
you which side won: for dollars that is `tokentriage report`, and for whether the cheaper side
held up it is `tokentriage eval` with judged shadow duplication. A split with no measurement
attached is only a slower rollout.

## What this page does not cover

Splitting traffic is not the same as proving one side is better. Making a defensible claim
about a cheaper tier is [prove a cheaper tier](/docs/calibration/prove-a-cheaper-tier), which
runs the comparison on held-out data rather than on live traffic. The promotion gate that
reads the result is on [Turn routing on](/docs/build/routing#the-honesty-verdict).

<NextSteps items={[
  { title: 'Turn routing on', href: '/docs/build/routing', description: 'The verdict that says whether the side you were testing actually held up.' },
  { title: 'Prove a cheaper tier', href: '/docs/calibration/prove-a-cheaper-tier', description: 'The offline version: run the comparison on held-out data before touching live traffic.' },
  { title: 'The policy language', href: '/docs/build/policy-language', description: 'The split grammar in full, including the arm cap and what an arm target may contain.' },
  { title: 'Routing explanation fields', href: '/docs/build/routing-explanation-fields', description: 'Read the roll record and the inputs needed to recompute a logged draw.' }
]} />