GitHubBook a demoStart routing
Browse docs

Route by prompt complexity

Send simple prompts to a cheap tier and hard ones to a strong tier with a single policy condition on signal.complexity — rolled out shadow-first so you prove the savings before you enforce them.

Last updated

On this page

Route cheap-looking prompts to a cheap tier and hard ones to a strong tier by branching a routing policy on the built-in signal.complexity score — and roll it out shadow-first so a report proves the savings before a single served response changes.

Prerequisites

  • TokenTriage running with at least two tiers — a cheap one and a strong one — defined in tiers: (see Providers).
  • The built-in classifier ladder enabled (the default). It emits signal.complexity, a [0,1] score, per request.
  • Familiarity with the policy language — this recipe assumes you know what a policy node is.

Steps

1. Confirm your tiers, cheapest → most capable

tiers: is an ordered list; the names are free-form and are what a policy target references. A minimal two-tier setup:

tiers:
  - name: cheap
    upstream: anthropic
    model: claude-haiku-4-5
  - name: frontier
    upstream: anthropic
    model: claude-opus-4-8

2. Add a routing.policy that branches on complexity

The policy tree has five node kinds: rules (first-match-wins), select (hard-filter → soft-score over candidates), split (seeded A/B/canary), cascade (try-grade-escalate), and bare leaf targets ({ tier }, { pool }, { refuse }, { passthrough }, { model, upstream }). Complexity routing is the simplest case — a single rules node whose one rule sends anything below a threshold to cheap and lets everything else fall through to the default:

routing:
  policy:
    version: 1
    params:
      simple_cut:
        value: 0.35
        calibrate: { signal: complexity, direction: below }
    root:
      rules:
        - id: simple-to-cheap
          when: 'signal.complexity < $simple_cut'
          to: { tier: cheap }
      default: { tier: frontier }

What each piece is doing, all verified against the frozen grammar:

  • signal.complexity is a genuine signal.<key> input emitted by the ladder; $simple_cut is a $param reference; < is one of the six comparison operators (<, <=, >, >=, =, !=).
  • default: is required on every rules node — it is the target when no rule matches.
  • The calibrate block makes $simple_cut a fittable param (see step 5); it is inert until you run the fitter.

3. Roll out in shadow, not route

Set the top-level trust-ladder mode to shadow. In shadow, TokenTriage records the would-route decision (and can duplicate to the counterfactual tier and grade it) without touching any served response — the served model is still whatever the client asked for.

mode: shadow          # ledger | shadow | route  — the trust ladder

Send some real traffic, then read a decision back with the explain command:

$ tokentriage policy explain --req-id 01J8Z... --log ./decisions.log

policy explain prints human-readable text (not JSON) — the path taken and the exact inputs each node read, in first-read order. A simple prompt (illustrative):

req_id:     01J8Z...
policy_rev: r7
decision:   rule=simple-to-cheap tier=cheap upstream=openai-cheap
path:       root → rules[simple-to-cheap]
trace:
  • rules[simple-to-cheap]     matched
      read signal.complexity = 0.21
      read $simple_cut = 0.35

A hard prompt instead falls through to default (illustrative):

req_id:     01J9A...
policy_rev: r7
decision:   rule=default tier=frontier upstream=openai-frontier
path:       root → rules[default]
trace:
  • rules[simple-to-cheap]     skipped
      read signal.complexity = 0.71
      read $simple_cut = 0.35
  • rules[default]             matched

4. Prove the savings before enforcing

Because you are in shadow, the decision log holds real would-route decisions you can score offline. Re-decide the log under the same (or a draft) policy to see how many requests would move and the cost delta:

$ tokentriage policy preview --config ./config.yaml --log ./decisions.log

policy preview re-decides each recorded row through the real evaluator and reports changed / cost-delta / by-rule. It always emits two approximation labels (a decision is re-simulated, not re-served) and discloses per-row any draft-referenced input a row did not record — so you never get a confident wrong delta.

5. (Optional) Calibrate the cut to a target frontier share

Instead of guessing 0.35, fit $simple_cut from the log to hit a target share of frontier traffic. A fit writes a dormant candidate artifact (under ~/.tokentriage/calibrations); it does not edit your YAML, and it stays inert until you activate it:

$ tokentriage calibrate -c ./config.yaml --target-frontier-pct 0.30

The fit reports the proposed change, e.g. simple_cut: 0.35000 → 0.28400. Nothing affects routing until you promote the candidate:

$ tokentriage calibrate --activate <candidate-id>

6. Promote to route

Once the shadow report shows the split you want, flip the mode to enforce it. Now the matched rule actually rewrites the model / picks the tier:

mode: route

Why this is safe

  • Shadow-first is the point. The ledger → shadow → route ladder means you observe and price every would-route decision, then prove the savings on real traffic, and only then enforce — no served response changes until the last step.
  • A missing signal fails safe. This is absent-input comparison semantics: if signal.complexity is ever absent, the ordered comparison signal.complexity < $simple_cut evaluates to false — the rule does not fire and the request falls through to the default (frontier), your strong tier. Complexity routing therefore never downgrades a request on a missing signal.
  • Every decision is explainable. The read block records exactly which inputs the condition saw, in first-read order, on the decision-log row — so a “why did this go cheap?” question is answered from the record, not reconstructed.