Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Route to the cheapest eligible tier

Route each request to the lowest-cost tier that meets your health, latency and quality requirements.

How-to guideLast updated

On this page

Use a cost-aware select policy when several configured tiers can serve a request and you want the least expensive one that still meets hard requirements. A tier is a named model and provider target.

TokenTriage handles this in two steps: remove tiers that miss requirements such as health or latency, then estimate the request cost for the rest. The comparison is local and adds no model call or provider spend. In route mode, only the chosen tier receives the routed request.

This guide assumes routing is turned on and your configuration already has at least two tiers. The captured example uses three existing tiers named cheap, mid and frontier, ordered from cheapest to most capable.

Add the cost-aware policy

Replace the routing: block in your existing configuration with this policy. Adjust the latency target to match your own service-level requirement.

routing:
  default_tier: frontier
  policy:
    version: 1
    root:
      select:
        from: { tiers: all }
        where: "candidate.latency.p95_ms <= 2000 and candidate.healthy"
        by: { selector: cheapest_meeting }
        on_empty: { tier: frontier }

Four lines control the decision:

  • from supplies every configured tier as a candidate.
  • where requires a healthy tier whose 95th-percentile latency is no more than two seconds.
  • by tells TokenTriage to choose the lowest estimated request cost among the tiers left.
  • on_empty sends the request to frontier if no tier passes the requirements.

Save the configuration and validate it before restarting TokenTriage:

$ tokentriage validate --config tokentriage.yaml

Successful validation ends with OK: configuration is valid. It catches unknown fields, invalid expressions and references to tiers that do not exist.

Understand filter, then score

Separating requirements from ranking prevents a cheap but unsuitable tier from winning the comparison.

A select node filters candidates, then scores the survivorsA select node starts from a candidate set, drawn from a named tier list, from every tier, or from a pool. The where clause is a hard filter applied to each candidate in turn; a candidate it rejects never reaches the selector, and is recorded in the filtered list alongside the exact input that rejected it. Every candidate that survives goes to the selector, which scores all of them and picks one; every survivor gets a row in the score table, including the ones the selector declined, so a choice always carries the comparison behind it. When no candidate survives, the node takes its on_empty branch instead, and a policy with no on_empty branch falls to the document-level on_no_candidate behaviour.Candidatestiers: [a, b]tiers: allpool: <name>wherea hard filter,one candidateat a timeby: selectora soft scoreover everysurvivorOne candidatebecomes the plansurvivorsfiltered[]each rejected candidate,and the input that rejected itscores[]every survivor scored,chosen and unchosen alikeboth land in the decision's explanationnobody survives → on_empty, else on_no_candidate

Read the diagram from left to right. where decides which candidates are eligible. by ranks only the candidates that remain. Rejected candidates appear in filtered[], together with the value that rejected them. Every surviving candidate appears in scores[], whether or not it won. If no candidate passes the filter, the node takes its on_empty branch.

See how request cost is estimated

cheapest_meeting estimates what the current request would cost at each surviving tier and takes the lowest result:

cost = prompt_tokens × input_per_tok  +  est_output × output_per_tok

Prompt tokens come from TokenTriage’s request estimate. If no positive estimate is available, it uses 1000. By default, it assumes the response will contain the same number of tokens as the prompt. A 600-token prompt therefore uses a 600-token response estimate unless you set default_output_tokens.

Ties go to the tier with the lower rank, meaning the tier that appears earlier in your tiers: list.

Parameter Default What it does
default_output_tokens prompt-token estimate uses this fixed response-length estimate instead
min_quality_lcb none requires a calibrated quality floor; see below
quality_metric none names the calibrated metric when a floor is set

What it costs. Selection adds no model calls or provider spend. TokenTriage calculates the scores locally from its request estimate and the prices loaded with your configuration.

If a tier has no usable price. TokenTriage excludes it and marks it unpriced in the score table. It does not treat a missing price as zero. A tier you explicitly declared free carries a real $0 price and can win. If no surviving tier has a usable price, the node takes on_empty.

Verify one decision

Restart TokenTriage with the validated configuration and send a representative request. Copy the x-tt-request-id response header, then explain that request’s routing decision. The request ID in this capture was tt_select:

$ tokentriage policy explain --req-id tt_select \
    --log ~/.tokentriage/decisions.jsonl
req_id:     tt_select
policy_rev: 8067f04
decision:   rule=default tier=mid upstream=openai
path:       root → select[tiers]
trace:
  • select[tiers]                selected
      selector=cheapest_meeting
      score  mid        0.0032  {input_cost=0.0009, output_cost=0.00225}
      score  frontier   0.0189  {input_cost=0.0054, output_cost=0.0135}
      filtered cheap      (candidate.latency.p95_ms=3400)

The important lines show why mid won:

  • cheap was removed because its 3,400 ms p95 latency exceeded the 2,000 ms limit.
  • mid and frontier passed the filter, so both received an estimated cost.
  • mid won because its $0.0032 score was lower than frontier’s $0.0189.

Read a routing decision covers the remaining fields and shows where to find the same explanation in the dashboard.

Add a measured quality floor

Price and latency do not tell you whether a tier produces answers that are good enough for your workload. After you have calibrated a quality metric, replace the by block with this one:

by:
  selector: cheapest_meeting
  params:
    min_quality_lcb: 0.98
    quality_metric: output_nonempty

Lower confidence bound (LCB) is a conservative quality estimate that accounts for measurement uncertainty. A floor of 0.98 admits only tiers whose lower bound is at least 0.98 for output_nonempty.

The floor controls eligibility, not ranking. TokenTriage removes a tier below the floor before comparing costs, so the floor does not reorder the tiers that pass. A tier with no usable quality measurement is also removed. The explanation distinguishes the two cases:

  • quality_below_min_lcb: the tier was measured and fell below the floor
  • quality_absent:<reason>: TokenTriage could not read a usable measurement

Quality-aware routing explains the distinction. Gate routing on quality walks through calibration, configuration and verification from start to finish.

When nobody qualifies

An empty candidate set is an expected outcome. Set on_empty on the node so the fallback is visible next to the requirements that can trigger it. TokenTriage handles an empty set in this order:

  1. The node’s on_empty branch, when present.
  2. The document-level on_no_candidate setting. default_tier sends the request to routing.default_tier; refuse returns a typed 503 with the reason no_candidate_available.
  3. The default fail_safe behavior. TokenTriage sends the request without applying a tier and records that the filter emptied without a configured branch.
Why a selector choice can become a fail-safe

A valid selector result includes a score table with the candidate it chose. If a selector returns a choice without that supporting row, TokenTriage treats the result as a fault, sends the request without applying a tier and records the reason. An empty candidate set is different: the decision includes the filter and score results, then follows on_empty.

What this page does not cover

Use weighted_objective when cost, latency or quality should trade off against one another instead of acting as hard requirements. Its weights, normalization and missing-input behavior are in the selector reference.

Spreading traffic across interchangeable upstreams and falling back when one is unhealthy is pools and fallback. Learning from graded responses is bandit routing. Producing the quality measurements used by min_quality_lcb starts with calibration from logs.