Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Split traffic for an experiment

Send a slice of traffic down a different path, keep each conversation on one side, and recompute any draw from the row.

How-to guideLast updated

On this page

Every other strategy tries to make the best decision for each request. A split does the opposite: it decides at random so that two paths get comparable traffic, which is the only way to find out whether the change you are considering is an improvement.

This page builds a canary and an A/B, explains how the draw works and why the same conversation keeps landing on the same side, and covers what you have to do yourself to measure the result.

You need a policy that loads and two things worth comparing; Turn routing on covers the setup.

Ship a change to a slice first

A canary sends most traffic down the path you trust and a small share down the new one.

routing:
  default_tier: frontier
  policy:
    version: 1
    seed: "canary-2026-08"
    root:
      split:
        sticky: trace
        arms:
          - { id: control, weight: 95, to: { tier: frontier } }
          - { id: canary, weight: 5, to: { tier: mid } }

Weights are relative positive integers. 95 and 5 gives nineteen to one; so does 19 and 1. They do not have to sum to a hundred, and a weight of zero is a load error rather than a parked arm. Remove the arm instead.

An arm’s to: is a whole node, not a leaf, so an arm can hold an entire sub-policy. That is how you A/B two strategies rather than two tiers: put a cascade on one side and a plain tier on the other.

How an arm is picked

The word “split” suggests a coin flip. It is not one, and the difference is what makes the result usable.

How a split picks an arm, and why it keeps picking the same oneThree inputs go into the draw: the policy's seed, the sticky key — the trace id by default, falling back to the request id when there is no trace — and the node's own identity within the tree. They are hashed together into a single number, which is reduced modulo the total arm weight and lands in one of the cumulative weight bands. In this example the control arm has weight ninety-five and the canary arm has weight five, so the bands are zero to ninety-four and ninety-five to ninety-nine. Because the draw is a pure function of those three inputs, the same conversation lands on the same arm on every request and across restarts, and the recorded roll on the decision row can be recomputed by anyone holding it. Weights are positive integers and do not have to sum to one hundred.the policy'sseedthe sticky keytrace id(request id when there is none)this node's identitytwo splits never agreeone hashno clock, no randomnesssource, no counterreduced onto the weight bandscontrolweight: 95 → band 0–94canaryweight: 5 → band 95–99Recorded, and recomputableThe decision row carries the node, the scope, a short hash of the key, the drawn value and the arm it chose.Same seed, same conversation, same arm — on the next request and after a restart.

Read it left to right. Three inputs go in: the seed you set, the sticky key for this request, and the node’s own identity within the tree. They are hashed into a single number, which is reduced onto the cumulative weight bands. There is no clock, no counter, and no randomness source, so the same three inputs always produce the same arm.

The node’s identity is in the hash for a reason: two splits with identical arms and weights draw independently rather than agreeing with each other, so nesting one experiment inside another does not correlate them.

Stickiness, and what happens without a trace id

sticky: decides which key goes into the hash. It takes trace, the default, or request.

With sticky: trace, every request carrying the same trace id lands on the same arm. That is what you want for anything conversational: a canary that flips a user between models mid-conversation is measuring the flipping, not the model.

With sticky: request, every request is drawn independently. Use it for stateless, single-shot workloads where a conversation is not a thing.

A request with no trace id cannot be sticky to one, so the draw falls back to the request id — and the row records the scope it actually used rather than the one you asked for. If your rows say scope=request under sticky: trace, your clients are not sending trace headers.

Read the result

The draw is on the row, with enough detail to recompute:

$ tokentriage policy explain --req-id tt_split --log ~/.tokentriage/decisions.jsonl
req_id:     tt_split
policy_rev: abf1f12
decision:   rule=default tier=mid upstream=openai
path:       root → split → arm:canary
trace:
  • split                        split (arm:canary)
  roll split#1 scope=trace key=51bcc7b value=0.462589 → arm:canary

path names the arm. The roll line names the node, the scope, a short hash of the sticky key, the drawn value and the arm it produced. Anyone holding the row’s trace id and the seed can recompute that draw, which is what makes a disputed assignment checkable rather than a matter of trust.

Making the arms show up in the savings report

tokentriage report groups by rule id, and a split arm does not set one — rows land under whatever rule matched above the split, or under default. To get per-arm totals, make each arm’s target a rules node with one unconditional entry whose id names the arm:

arms:
  - id: canary
    weight: 5
    to:
      rules:
        - { id: arm-canary, to: { tier: mid } }
      default: { tier: mid }

The arm id now appears in the BY RULE table beside its spend and saving. It costs one extra node per arm and makes the experiment legible in the reporting you already run.

What can go wrong

Editing the policy while the experiment runs. Changing the policy changes its fingerprint, which is what the promote gate keys on, so accumulated shadow evidence stops counting toward promoting the new version. Finish the comparison, then edit.

Expecting the shares to be exact. The draw is uniform over conversations, not over requests. A single very chatty trace lands wholly on one arm, so a 95/5 split can look like 90/10 by request count on low volume. Judge it by trace count, and give it enough traces that the ratio settles.

Treating a split as the measurement. The split creates comparable traffic. It does not tell you which side won: for dollars that is tokentriage report, and for whether the cheaper side held up it is tokentriage eval with judged shadow duplication. A split with no measurement attached is only a slower rollout.

What this page does not cover

Splitting traffic is not the same as proving one side is better. Making a defensible claim about a cheaper tier is prove a cheaper tier, which runs the comparison on held-out data rather than on live traffic. The promotion gate that reads the result is on Turn routing on.