Spread traffic and fail over

When two or more routes can serve the same request, you need to decide how traffic should move between them. You can distribute requests by weight, prefer the route with the best recent latency, or try routes in a fixed order.

An upstream is a configured connection to a model provider. A tier is a named routing option that pairs an upstream with a model. A pool groups tiers that are interchangeable for the traffic you send to it.

These strategies choose a tier before a request is sent. They do not retry a request after its upstream fails. Cooldown only steers later requests away from an upstream that has been failing.

You need at least two tiers that can serve the same traffic. Turn routing on covers that setup.

Choose a strategy

Start with the behavior you want:

Your goal	Use	Important behavior
Distribute traffic in set proportions	weighted	weights apply across traces; requests in one trace stay on the same member
Prefer the fastest recently observed upstream	least_latency	reads stored measurements; it does not probe providers before routing
Try a primary route, then a backup	first_eligible	chooses the first tier already considered healthy; it does not retry after an error

All three can filter on candidate.healthy. That filter becomes useful only after you enable cooldown, as the complete example below does.

Declare a pool

This configuration creates a two-member pool, distributes traffic between its members, and routes later requests to frontier if cooldown removes every pool member:

routing:
  default_tier: frontier
  cooldown:
    enabled: true
    seconds: 30
    failure_threshold: 0.5
    min_requests: 5
  policy:
    version: 1
    seed: "lb-1"
    pools:
      cheap_pool:
        members:
          - { tier: cheap }
          - { tier: cheap-b, weight: 2 }
    root:
      select:
        from: { pool: cheap_pool }
        where: "candidate.healthy"
        by: { selector: weighted }
        on_empty: { tier: frontier }

A pool member has a tier name and an optional weight. Weights are relative: with weights 1 and 2, the first member receives about one third of traces and the second receives about two thirds. The weights do not have to add up to a particular total.

Writing to: { pool: cheap_pool } is shorthand for a weighted selection with no health filter and no on_empty branch. Use the longer form when you want unhealthy members excluded or need an explicit destination when no member qualifies.

Spread load in proportion

Use weighted when every eligible pool member can serve the request and you want to control its share of traffic:

by: { selector: weighted }

Set routing.policy.seed so the same trace makes the same choice across restarts and replicas. Weights are distributed across many traces, not across requests within one trace. Keeping a trace on one member also preserves prompt-cache affinity.

Check the recorded roll in the decision details, then compare the distribution of decisions over time:

  roll select[cheap_pool]#3 scope=trace key=9b1e77c value=0.418003 → tier:cheap-b

Do not use weighted distribution when the members are not interchangeable. A weight controls how often a member is chosen; it does not express that one member is safer, cheaper, or more capable.

How weighted selection works

The selector sums the eligible members’ weights, produces one seeded draw, and maps that draw onto their cumulative shares. Every candidate that passes the where filter receives a probability in the decision’s score table. The selection adds one hash operation to the decision.

An absent or non-positive weight is loaded as 1; it does not remove the member. The selector itself has no parameters because its distribution comes from the pool.

Why a weight filter usually excludes everything

candidate.pool_weight exists only when the candidate set came from a pool. In a select over { tiers: [...] } it is absent for every candidate, so a filter such as where: 'candidate.pool_weight >= 1' rejects all of them and follows on_empty. Filter on candidate.healthy or a price, and let the pool carry the weights.

Prefer whichever upstream is answering fastest

Use least_latency when recent response time matters more than a fixed traffic share. In the pool configuration above, replace the by line with:

by: { selector: least_latency, params: { stat: p50 } }

The selector compares stored latency observations for each upstream. It does not send a probe or wait for a new measurement before choosing. Use p50 for typical latency or p95 when tail latency matters more.

An upstream with no observations is marked cold_start. Any candidate with measurements is preferred over a cold candidate. When every candidate is cold, TokenTriage chooses the first one in the candidate list. After a restart, this can keep traffic on the first candidate while the others remain unmeasured.

Warm every candidate with representative traffic before relying on this selector. One practical path is to start with weighted, wait until every upstream has observations, and then switch to least_latency. In the decision details, check the millisecond score for each candidate and look for cold_start on any upstream that has not collected observations.

Use another selector when you want the cheapest candidate. least_latency does not consider price.

Reference details for least-latency selection

Param	Default	What it does
stat	p50	chooses the percentile to compare; accepts p50 or p95
window	none	accepts a duration but does not apply it in this version

Both percentiles are computed per upstream by nearest rank over the last 256 observations. The measurements are recorded after the client has been served, so measurement does not add work to the request path. Selection reads one stored value per candidate, and the decision’s ctx_ver records which latency snapshot it used.

A malformed window is a configuration error. A valid value has no effect because the sample window is currently fixed at 256 observations.

Try tiers in a deliberate order

Use first_eligible when you have a preferred route and one or more backups. This configuration tries mid, then frontier, then cheap, skipping any tier currently in cooldown:

routing:
  default_tier: frontier
  cooldown:
    enabled: true
    seconds: 30
    failure_threshold: 0.5
    min_requests: 5
  policy:
    version: 1
    root:
      select:
        from: { tiers: [mid, frontier, cheap] }
        where: "candidate.healthy"
        by: { selector: first_eligible }
        on_empty: { refuse: { status: 503, reason: "all_upstreams_cooling" } }

The order is exactly the order in from.tiers; it is not derived from price or capability. If every tier is filtered out, TokenTriage follows on_empty. If you omit on_empty, the request is relayed without applying a route.

Check the decision details for the selected tier and the filtered entries naming each tier that was skipped:

      selector=first_eligible
      score  frontier   1.0000  {position=0}
      filtered mid        (candidate.healthy=false)
      filtered cheap      (candidate.healthy=false)

Do not use this selector when you want cost order. It can prefer an expensive tier whenever that tier appears first.

Reference details for ordered selection

first_eligible chooses the first candidate that survives the filter. It accepts no parameters; supplying one is a configuration error. It reads no candidate facts beyond the filter and uses no randomness. The decision still records a position score for each eligible candidate so you can see the comparison that produced the choice.

Take a failing upstream out of rotation

Cooldown is what makes candidate.healthy reflect recent failures. It is disabled until you enable it:

routing:
  cooldown:
    enabled: true
    seconds: 30
    failure_threshold: 0.5
    min_requests: 5

Within each wall-clock minute, TokenTriage counts responses for each upstream. Once the upstream has seen at least min_requests, it enters cooldown when its failure rate is strictly greater than failure_threshold. A failure rate of exactly 0.5 does not trip a threshold of 0.5.

Not every unsuccessful request contributes to cooldown:

Result	Counts as a cooldown failure?
401, 404, 408, 429, or any 5xx	yes
any other 4xx	no; it counts in the denominator as a success
connection error	no; it is excluded from both successes and failures

This means a connection error does not cause cooldown by itself. Other 4xx responses are treated as request problems rather than evidence that the provider is unhealthy.

Cooldown expires after seconds. There is no half-open probe or trial request. The minute’s counters are not cleared when cooldown expires, so another failure in the same minute can put the upstream back into cooldown.

If every configured tier points to one upstream, TokenTriage exempts that upstream from cooldown because removing it would leave no route.

What this layer does not give you

Keep these boundaries in mind before relying on a pool during an incident:

• There is no retry. A request whose upstream fails returns that failure to your application. Cooldown can steer later requests away; it cannot rescue the failed request.
• There are no health probes. Health is inferred from your traffic. An idle upstream has no failure samples, so candidate.healthy remains true until observed traffic says otherwise.
• A cascade is not a failover. It escalates after grading an answer, not after receiving an upstream error. Cascades covers that distinction.
• Configuration changes have different lifecycles. Cooldown changes take effect on reload; cache-affinity changes require a restart.

If retrying a failed request across providers is the reason you want a gateway, How TokenTriage compares explains the current product boundary.