# Spread traffic and fail over Distribute traffic across interchangeable upstreams, prefer the fastest recently observed route, or try healthy tiers in a fixed order. Source: https://tokentriage.com/docs/build/routing-pools --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; When two or more routes can serve the same request, you need to decide how traffic should move between them. You can distribute requests by weight, prefer the route with the best recent latency, or try routes in a fixed order. An **upstream** is a configured connection to a model provider. A **tier** is a named routing option that pairs an upstream with a model. A **pool** groups tiers that are interchangeable for the traffic you send to it. These strategies choose a tier before a request is sent. They do not retry a request after its upstream fails. Cooldown only steers later requests away from an upstream that has been failing. You need at least two tiers that can serve the same traffic. [Turn routing on](/docs/build/routing) covers that setup. ## Choose a strategy Start with the behavior you want: | Your goal | Use | Important behavior | |---|---|---| | Distribute traffic in set proportions | `weighted` | weights apply across traces; requests in one trace stay on the same member | | Prefer the fastest recently observed upstream | `least_latency` | reads stored measurements; it does not probe providers before routing | | Try a primary route, then a backup | `first_eligible` | chooses the first tier already considered healthy; it does not retry after an error | All three can filter on `candidate.healthy`. That filter becomes useful only after you enable cooldown, as the complete example below does. ## Declare a pool This configuration creates a two-member pool, distributes traffic between its members, and routes later requests to `frontier` if cooldown removes every pool member: ```yaml routing: default_tier: frontier cooldown: enabled: true seconds: 30 failure_threshold: 0.5 min_requests: 5 policy: version: 1 seed: "lb-1" pools: cheap_pool: members: - { tier: cheap } - { tier: cheap-b, weight: 2 } root: select: from: { pool: cheap_pool } where: "candidate.healthy" by: { selector: weighted } on_empty: { tier: frontier } ``` A pool member has a tier name and an optional weight. Weights are relative: with weights `1` and `2`, the first member receives about one third of traces and the second receives about two thirds. The weights do not have to add up to a particular total. `weight: 0` does not disable a member. An omitted weight and an explicit zero both become `1`, so the member still receives traffic. In the example above, changing the second member's weight to zero would give each member half. Remove a member from the pool when you want to stop routing to it. Writing `to: { pool: cheap_pool }` is shorthand for a `weighted` selection with no health filter and no `on_empty` branch. Use the longer form when you want unhealthy members excluded or need an explicit destination when no member qualifies. ## Spread load in proportion Use `weighted` when every eligible pool member can serve the request and you want to control its share of traffic: ```yaml by: { selector: weighted } ``` Set `routing.policy.seed` so the same trace makes the same choice across restarts and replicas. Weights are distributed across many traces, not across requests within one trace. Keeping a trace on one member also preserves prompt-cache affinity. Check the recorded roll in [the decision details](/docs/build/routing-explain), then compare the distribution of decisions over time: ```console roll select[cheap_pool]#3 scope=trace key=9b1e77c value=0.418003 → tier:cheap-b ``` Do not use weighted distribution when the members are not interchangeable. A weight controls how often a member is chosen; it does not express that one member is safer, cheaper, or more capable.
The selector sums the eligible members' weights, produces one seeded draw, and maps that draw onto their cumulative shares. Every candidate that passes the `where` filter receives a probability in the decision's score table. The selection adds one hash operation to the decision. An absent or non-positive weight is loaded as `1`; it does not remove the member. The selector itself has no parameters because its distribution comes from the pool.
`candidate.pool_weight` exists only when the candidate set came from a pool. In a `select` over `{ tiers: [...] }` it is absent for every candidate, so a filter such as `where: 'candidate.pool_weight >= 1'` rejects all of them and follows `on_empty`. Filter on `candidate.healthy` or a price, and let the pool carry the weights.
## Prefer whichever upstream is answering fastest Use `least_latency` when recent response time matters more than a fixed traffic share. In the pool configuration above, replace the `by` line with: ```yaml by: { selector: least_latency, params: { stat: p50 } } ``` The selector compares stored latency observations for each upstream. It does not send a probe or wait for a new measurement before choosing. Use `p50` for typical latency or `p95` when tail latency matters more. An upstream with no observations is marked `cold_start`. Any candidate with measurements is preferred over a cold candidate. When every candidate is cold, TokenTriage chooses the first one in the candidate list. After a restart, this can keep traffic on the first candidate while the others remain unmeasured. Warm every candidate with representative traffic before relying on this selector. One practical path is to start with `weighted`, wait until every upstream has observations, and then switch to `least_latency`. In [the decision details](/docs/build/routing-explain), check the millisecond score for each candidate and look for `cold_start` on any upstream that has not collected observations. Use another selector when you want the cheapest candidate. `least_latency` does not consider price.
| Param | Default | What it does | |---|---|---| | `stat` | `p50` | chooses the percentile to compare; accepts `p50` or `p95` | | `window` | none | accepts a duration but does not apply it in this version | Both percentiles are computed per upstream by nearest rank over the last 256 observations. The measurements are recorded after the client has been served, so measurement does not add work to the request path. Selection reads one stored value per candidate, and the decision's `ctx_ver` records which latency snapshot it used. A malformed `window` is a configuration error. A valid value has no effect because the sample window is currently fixed at 256 observations.
## Try tiers in a deliberate order Use `first_eligible` when you have a preferred route and one or more backups. This configuration tries `mid`, then `frontier`, then `cheap`, skipping any tier currently in cooldown: ```yaml routing: default_tier: frontier cooldown: enabled: true seconds: 30 failure_threshold: 0.5 min_requests: 5 policy: version: 1 root: select: from: { tiers: [mid, frontier, cheap] } where: "candidate.healthy" by: { selector: first_eligible } on_empty: { refuse: { status: 503, reason: "all_upstreams_cooling" } } ``` The order is exactly the order in `from.tiers`; it is not derived from price or capability. If every tier is filtered out, TokenTriage follows `on_empty`. If you omit `on_empty`, the request is relayed without applying a route. Check the decision details for the selected tier and the `filtered` entries naming each tier that was skipped: ```console selector=first_eligible score frontier 1.0000 {position=0} filtered mid (candidate.healthy=false) filtered cheap (candidate.healthy=false) ``` Do not use this selector when you want cost order. It can prefer an expensive tier whenever that tier appears first.
`first_eligible` chooses the first candidate that survives the filter. It accepts no parameters; supplying one is a configuration error. It reads no candidate facts beyond the filter and uses no randomness. The decision still records a position score for each eligible candidate so you can see the comparison that produced the choice.
## Take a failing upstream out of rotation Cooldown is what makes `candidate.healthy` reflect recent failures. It is disabled until you enable it: ```yaml routing: cooldown: enabled: true seconds: 30 failure_threshold: 0.5 min_requests: 5 ``` Within each wall-clock minute, TokenTriage counts responses for each upstream. Once the upstream has seen at least `min_requests`, it enters cooldown when its failure rate is strictly greater than `failure_threshold`. A failure rate of exactly `0.5` does not trip a threshold of `0.5`. Not every unsuccessful request contributes to cooldown: | Result | Counts as a cooldown failure? | |---|---| | `401`, `404`, `408`, `429`, or any `5xx` | yes | | any other `4xx` | no; it counts in the denominator as a success | | connection error | no; it is excluded from both successes and failures | This means a connection error does not cause cooldown by itself. Other `4xx` responses are treated as request problems rather than evidence that the provider is unhealthy. Cooldown expires after `seconds`. There is no half-open probe or trial request. The minute's counters are not cleared when cooldown expires, so another failure in the same minute can put the upstream back into cooldown. If every configured tier points to one upstream, TokenTriage exempts that upstream from cooldown because removing it would leave no route. Zero is not expressible for these cooldown settings. `failure_threshold: 0` becomes `0.5`, `seconds: 0` becomes `30`, and `min_requests: 0` becomes `5`. Use a small positive value instead. ## What this layer does not give you Keep these boundaries in mind before relying on a pool during an incident: - **There is no retry.** A request whose upstream fails returns that failure to your application. Cooldown can steer later requests away; it cannot rescue the failed request. - **There are no health probes.** Health is inferred from your traffic. An idle upstream has no failure samples, so `candidate.healthy` remains `true` until observed traffic says otherwise. - **A cascade is not a failover.** It escalates after grading an answer, not after receiving an upstream error. [Cascades](/docs/build/routing-cascades) covers that distinction. - **Configuration changes have different lifecycles.** Cooldown changes take effect on reload; cache-affinity changes require a restart. If retrying a failed request across providers is the reason you want a gateway, [How TokenTriage compares](/docs/introduction/how-tokentriage-compares) explains the current product boundary.