Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Routing

Route requests across configured models, preview decisions before enforcing them, and record why every choice was made.

ExplanationLast updated

On this page

Routing chooses which configured model handles each request. For example, you might send routine summaries to a lower-cost model while reserving a stronger model for complex coding requests. The goal is to lower cost without lowering your quality bar.

You define the routing policy. TokenTriage evaluates it before serving each request and records what the policy evaluated and matched. Two terms appear throughout this section:

  • An upstream is a configured connection to a model provider.
  • A tier is a named routing option that pairs an upstream with a model.

Routing chooses between tiers. This page explains how to preview those choices safely, the four ways a policy can choose, and where to go next. If you prefer to learn by doing, follow Your first routing rollout.

Nothing is routed until you say so

A new installation does not reroute requests. mode: sits at the top level of your config file, not inside the routing: block, and defaults to ledger.

mode: ledger # ledger | shadow | route

Your policy is evaluated in all three modes. The mode controls how much of the result is recorded and whether it changes the model that serves your application.

Mode Changes the model serving your app? What the decision log contains
ledger No The matched rule, the inputs it read, and the decision trace. The tier is left blank.
shadow No The proposed tier and available comparison costs, in addition to the trace.
route Yes The tier that served the request and the complete trace.

These modes form the trust ladder. Start in ledger to confirm that the policy reads the inputs you expect. Move to shadow to review the exact tier and projected cost without changing the model that returns the response. Use route after you have reviewed the cost and any quality evidence your rollout requires.

How a routing decision is made explains evaluation order, passthroughs, refusals, and fail-safes.

Choose a routing approach

The policy language offers four node types. Choose one based on the information available when TokenTriage needs to make the decision.

Choosing a routing approachChoose rules when request data is enough to make the decision. Choose select to filter eligible tiers and score the remaining options. Choose split to assign traffic for an experiment or canary. Choose cascade to try a lower-cost tier, grade its answer, and escalate when needed.What do you knowwhen you decide?one question, four answersUse request data → rulesMatch on complexity, topic, caller, time, or budgetFirst matching rule wins · one provider callChoose the best eligible tier → selectFilter by requirements, then score price, latency, health, or qualityOne provider callRun an experiment or canary → splitAssign requests reproducibly across weighted tiersOne provider callJudge an answer before deciding → cascadeTry a lower-cost tier, grade the answer, and escalate if neededMay make another provider call when it escalates
  • Rules (rules) choose from information already on the request, such as complexity, topic, caller, time, or budget. The first matching rule wins.
  • Selection (select) filters out tiers that do not meet your requirements, then scores the remaining tiers by price, latency, health, or measured quality.
  • Traffic splitting (split) assigns requests reproducibly across weighted tiers for an A/B test or canary rollout.
  • Cascades (cascade) try a lower-cost tier, grade its answer, and escalate when the answer falls short. An escalated request makes more than one provider call.

If you are unsure, start with rules: they are the simplest option to predict and debug, and they do not add provider calls. Which routing strategy compares the options by goal and trade-off.

See why each decision was made

TokenTriage writes one entry per proxied request to ~/.tokentriage/decisions.jsonl. The decision trace shows the rules the policy visited, the inputs it used, and any scores or random assignments used to make the decision. The trace is recorded for every request rather than sampled.

In this example, the request matched simple-to-cheap. Its complexity score is 0.21, below the configured cutoff of 0.35, and the request does not use tools.

$ tokentriage policy explain --req-id tt_route
req_id:     tt_route
policy_rev: f9903ca
decision:   rule=simple-to-cheap tier=cheap upstream=oai_cheap model_rewrite=gpt-4o→gpt-4o-mini
path:       root → rules[simple-to-cheap]
trace:
  • rules[simple-to-cheap]       matched
      read signal.complexity = 0.21
      read $simple_cut = 0.35
      read request.has_tools = false

The CLI, dashboard, and API all read this trace from the same decision-log entry. Routing overhead is measured and capped in CI; see the hot-path benchmarks for current results. Explain a routing decision walks through the diagnostic workflow. Routing explanation fields defines each field.

What routing does not decide

Routing chooses the tier that should serve a request. Other systems answer separate questions:

  • Governance decides whether a request is allowed, including spending limits, rate limits, and per-key permissions.
  • Guardrails inspect content for personal data, secrets, and other configured risks.
  • Calibration and evals measure quality and help you set a quality requirement for routing.
Where routing sits in the request path

A request passes through governance admission, guardrails, routing, a second governance check for the selected model, and then the response cache. Routing comes before the cache because the selected model is part of the cache key. How TokenTriage works explains the complete request path.