Choosing a prompt routing signal

Some requests give you useful clues before a model sees them. A short factual question, a legal drafting request, and a large code review may each deserve a different model.

TokenTriage exposes those clues as signals: values computed from the request before it is routed. This page helps you choose one of three prompt signals, add it to a rules node, and check the resulting decision without adding another model call.

You need at least two tiers and a working policy. If you do not have those yet, start with Turn routing on. The examples use tiers named cheap, mid, and frontier; replace them with the names from your config.

Choose a signal

Start with the difference you can observe reliably in your own traffic.

All three signals are computed locally, so none adds a second upstream call. Complexity is available on every request. Topic matching and estimated cost run only when you configure their scorers. If a signal is unavailable, its comparison is false and the request follows the default branch.

The thresholds below are examples, not recommended production values. Start in shadow, observe your own requests, and adjust the threshold before you let the rule change traffic.

signal.complexity — estimated difficulty

Use complexity when a broad simple-versus-difficult split describes your workload. A lower score means the prompt looks more like something your lower-cost tier can handle.

routing:
  default_tier: frontier
  policy:
    version: 1
    params:
      simple_cut:
        value: 0.35
        calibrate: { signal: complexity, direction: below }
    root:
      rules:
        - id: simple-to-cheap
          when: "signal.complexity < $simple_cut and not request.has_tools"
          to: { tier: cheap }
      default: { tier: frontier }

This example sends prompts below 0.35 to cheap. Requests that declare tools stay on frontier, because an error can compound across a tool loop. Remove that condition only if you have evaluated tool-using requests separately.

The score comes from fixed patterns in the prompt, including length, multi-step structure, question patterns, and code or reasoning vocabulary. It is a local heuristic, not a model call and not a prediction of answer quality.

Main limitation. A short question about a subtle API contract can receive a low score. Use another signal when difficult prompts in your workload are often terse.

signal.semantic.<topic> — recognizable topic wording

Use topic matching when each route has a stable vocabulary: legal drafting, billing support, or another subject you can describe with representative phrases. Despite the scorer’s semantic name, it matches words rather than general meaning.

routing:
  default_tier: mid
  policy:
    version: 1
    scorers:
      semantic:
        budget_ms: 200
        topics:
          legal:
            exemplars: ["draft a contract clause", "review this NDA"]
          support:
            exemplars: ["track my shipment", "billing question"]
    params:
      topic_conf: { value: 0.8 }
    root:
      rules:
        - id: legal-to-frontier
          when: "signal.semantic.legal >= $topic_conf"
          to: { tier: frontier }
        - id: support-to-cheap
          when: "signal.semantic.support >= $topic_conf"
          to: { tier: cheap }
      default: { tier: mid }

Each topic receives a score from 0 to 1. A higher score means the prompt shares more distinctive words with at least one of that topic’s example phrases. The condition owns the threshold; there is no separate threshold setting on the scorer.

The scorer’s default time budget is 1 millisecond. The example raises it to 200, but that is a starting value rather than a production recommendation. Choose a budget that fits your latency target, then confirm in shadow that the topic signal is present under load. When the scorer exceeds its budget, its signals are omitted and the request follows default.

Main limitation. “Terminate the agreement” and “cancel the contract” share no content words, so they can score near zero against each other. Use another signal when paraphrases or multilingual prompts are common.

signal.est_cost — estimated request cost

Use estimated cost when prompt size is a useful predictor of spend. The scorer answers one specific question: what would this request cost if the configured reference tier served it?

routing:
  default_tier: frontier
  policy:
    version: 1
    scorers:
      est_cost:
        reference_tier: frontier
    params:
      frontier_cost_limit: { value: 0.02 }
    root:
      rules:
        - id: pricey-to-cheap
          when: "signal.est_cost >= $frontier_cost_limit"
          to: { tier: cheap }
      default: { tier: frontier }

This example sends a request to cheap when its estimated cost on frontier is at least $0.02. Set that threshold from your own price and quality targets, and verify the lower-cost tier on representative requests before enforcing the rule.

The scorer estimates input tokens from prompt bytes, adds an assumed 256 output tokens, and applies the reference tier’s catalog rates. It is computed locally and does not make a network or model call.

Main limitation. The output estimate is constant because the response does not exist yet. If one request returns a sentence and another returns a file, the signal mostly distinguishes their prompt sizes rather than their final bills.

If the reference tier has no catalog price, the signal is omitted and the request follows default. A reference_tier that does not name a configured tier prevents the policy from loading.

signal.est_cost = (prompt_bytes / chars_per_token) × input_rate
                + est_output_tokens × output_rate

Verify one request before routing

A matching rule and a useful rule are different claims. Keep the top-level mode set to shadow, then validate the edited config:

$ tokentriage validate --config ./tokentriage.yaml

After you send one representative request through your normal client, explain its recorded decision using the request ID:

$ tokentriage policy explain --req-id <request-id> --log ~/.tokentriage/decisions.jsonl

Look for both the rule ID in path and the expected value in read:

If the signal is absent, the comparison is false. Check the scorer configuration, its time budget, or the reference tier’s price before changing the default branch. Routing explanation fields covers the rest of the output.

What these signals cannot tell you

These signals describe a request before an answer exists. None proves that a lower-cost tier will produce an acceptable answer. Measure that separately before moving from shadow to route.

Use Route on context when the caller, time, or remaining budget should decide. Use Cheapest that qualifies to compare tiers against runtime criteria. Use Cascades when the decision depends on inspecting an answer.