Choose a routing strategy

If you are unsure where to start, use a rule and test it in shadow mode. A rule makes one model call and is easy to explain when something about the request or caller tells you where it should go. Shadow mode records the decision without changing live traffic.

Use selection when you need to balance several measurements, a split when you are running an experiment, and a cascade when you must inspect an answer before deciding whether to try a stronger model.

A tier is a model option you have configured, such as a lower-cost default and a stronger fallback. A routing policy is the set of decisions that chooses among those tiers. How a routing decision is made shows how TokenTriage walks that policy.

Four ways to make the decision

TokenTriage policies use four node kinds. Each one answers the routing question from different information.

Node kind	Use it when	What it decides from
rules	one condition can express the choice	the prompt, caller, time, budget or other request context
select	several eligible models need to be compared	price, latency, measured quality or another score
split	you want a stable traffic allocation	a seeded assignment that keeps related requests together
cascade	you need to inspect an answer before accepting it	the answer from one model and the result of an acceptance check

Start from the job

Find the row that sounds like your problem. Start with the plain-language strategy in the middle column; the configuration names appear in backticks.

Your goal	Start with	Build it
route from the prompt’s complexity, topic or estimated cost	a prompt rule: rules using signal.*	Route on the prompt
route by team, agent, feature, time or remaining budget	a context rule: rules using trace.*, dim.* or ctx.*	Route on context
choose the cheapest qualified model, or balance cost, latency and quality	scored selection: select using cheapest_meeting or weighted_objective	Pick the cheapest candidate that qualifies
spread traffic, prefer the fastest provider or follow an ordered fallback	pool selection: select using weighted, least_latency or first_eligible	Spread traffic and fail over
send a stable share of traffic to a new policy	a traffic split: split	Split traffic for an experiment
try a lower-cost model, check its answer and escalate when needed	an answer cascade: cascade	Escalate when the answer falls short
learn which model to prefer from graded outcomes	learned selection: select using epsilon_greedy	Let the router learn from outcomes

You can combine these strategies. For example, a policy can route by team, send 5% of one team’s traffic to a new branch, and use a cascade inside that branch. The policy language shows how those nodes fit together.

Compare cost and operational requirements

Compare model attempts first. Rules, selection and splits send each request to one target model. A cascade can make several attempts and may also pay for a judge.

Node kind	Target-model attempts	Other work	Data it needs
rules	one	evaluate a condition; prompt rules may also classify the request	request and context fields named by the condition
select	one	filter and score the eligible tiers	candidate tiers and the price, latency, quality or reward data used by the selector
split	one	assign the request to a stable traffic group	the split seed and the identity used to keep related requests together
cascade	one or more	check each answer; a model-based judge adds another call	the model answer and an acceptance check

For an escalated request, a two-tier cascade pays for the first model attempt, the judge when the judge is another model, and the stronger model attempt. It saves money only when enough requests stop before that stronger attempt. Escalate when the answer falls short calculates the break-even acceptance rate.

How to tell it is working

A routing decision and a useful outcome are different things. Each request records an explanation in the decision ledger. Inspect that explanation first, then use reports and evaluations to decide whether the strategy improved cost, latency or answer quality.

Node kind	Read this	It is working when
rules	path and read in tokentriage policy explain	the expected rule appears and the recorded input has the value you expected
select	score and filtered lines	the intended candidate ranks first and every excluded candidate has a stated reason
split	the roll line and tokentriage report grouped by rule	traffic approaches the configured percentages and one conversation stays in the same group
cascade	the cascade: block	the acceptance and escalation rates match the break-even assumption you used

Across all four, tokentriage report shows the observed cost change. tokentriage eval shows whether answer quality held up when you have configured the outcomes or evaluators it needs. If a strategy routes requests as expected but does not change cost, revisit the assumption behind it and check that the relevant requests have price data.

Use these after simpler strategies

Two strategies need more evidence or more calls than a rule.

Learned routing needs graded outcomes: a recorded score that says whether an answer was good enough. While it gathers evidence, it intentionally tries alternatives on some requests. Expect early routing to be less predictable than a fixed rule.

Cascades can increase your bill because one request may produce several model attempts. Use a cascade when the request alone does not predict answer quality well enough. Prefer a rule when information available before the first model call can make the choice.

How the routing strategies are covered by the product's test suite

Most strategies are exercised by sending requests through the proxy and reading the resulting explanation from the dashboard API. This covers complexity, semantic topic, cost-aware selection, expected-cost routing, cascades, pool load balancing, traffic splits, learned routing, budget handling and time-of-day routing.

Attribute routing on trace.* and dim.*, plus ordered fallback with first_eligible, are tested against the evaluator and policy compiler rather than in dedicated proxy-to-dashboard scenarios. least_latency shares the end-to-end pool path exercised by weighted, but it does not have a dedicated proxy-to-dashboard scenario of its own.

Test ordered fallback and least_latency in shadow mode on your traffic before enabling them.

What this page does not cover

This page helps you choose a strategy; it does not estimate savings for your workload. tokentriage report measures the cost change in your recorded traffic, while tokentriage eval checks answer quality when its required evidence is available. Turn routing on explains how to collect those results safely. Use calibration and evals to decide whether a lower-cost tier is good enough for the requests it receives.