How TokenTriage works

TokenTriage is a single Go binary you place in front of your LLM traffic. It costs every request, and — only once the evidence says it is safe — routes each request to the cheapest tier that can still do the job. This page explains how the pieces fit together, so the how-to guides make sense when you get there.

The design goal is trust. TokenTriage never asks you to route blind, and it never shows a number it cannot defend. Everything below follows from those two commitments.

The request flow

Point your agent’s base URL at TokenTriage instead of at the provider. From the client’s perspective, nothing changes: TokenTriage speaks both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) dialects, and the bytes it forwards back are identical to going direct — streaming included.

A request travels through four stops:

1. Client → data plane. Your agent sends a normal chat/messages request to the TokenTriage data plane (loopback 127.0.0.1:8787 by default).
2. Data plane → upstream provider. TokenTriage forwards the request to the real provider (OpenAI, Anthropic, Bedrock, Gemini, or any OpenAI-compatible upstream). It is a byte-faithful proxy: the proxied bytes are equal to going direct.
3. Upstream → data plane → client. The provider’s response streams back through TokenTriage to your agent, verbatim.
4. Data plane → decision ledger. As the response passes through, TokenTriage costs the request and writes one row to the decision ledger (~/.tokentriage/decisions.jsonl) — model, token types, dollars, the pricing snapshot, and the routing decision.

Prompt text is not stored by default — a row carries a prompt_sha256 and counted features, not the words. That is what lets you run TokenTriage in front of real traffic without turning your ledger into a second copy of your prompts.

The trust ladder: observe, then prove, then act

TokenTriage earns enforcement one rung at a time. Each rung is a mode: setting, and you climb only when the evidence from the rung below says the next one is safe.

• ledger — observe. Cost every request; mutate nothing. Zero client impact. This is where you learn where your bill went.
• shadow — prove. Also compute the would-route decision and reprice it as a counterfactual, so a report can show the savings you would get — still without touching a single response. Zero client impact.
• route — act. Enforce decisions: rewrite the model, pick the cheaper upstream, honor cache pins. Now, and only now, requests are actually routed.

The whole point is ledger → shadow → route as a deliberate progression. Observe first, prove the savings are real and safe, then act. You are never asked to flip on routing and hope.

Data plane vs admin plane

TokenTriage binds two listeners, and keeping them separate is a security and clarity decision.

Plane	Default address	Serves	Who talks to it
Data plane	127.0.0.1:8787	The proxy itself — your agents’ chat/messages traffic	Your agents
Admin plane	127.0.0.1:9090	/health, /metrics, /tt/api/status, and the embedded dashboard at /ui/	You (operators, monitoring)

Your LLM traffic and your operational surface never share a door. The data plane carries requests; the admin plane exposes health, Prometheus metrics, provider reachability (/tt/api/status), and the trust dashboard. An admin-plane bind failure is deliberately non-fatal — the data plane keeps serving even if the admin listener can’t come up, because proxying your traffic matters more than serving a chart.

The dashboard on the admin plane renders the same numbers as the CLI with the same honesty posture: every dollar carries its pricing-snapshot provenance, estimates wear an EST flag, and unpriced spend is an explicit excluded slice — never rendered as a fake $0.00.

The honesty flywheel

Put the pieces together and you get a loop that keeps turning:

1. Route / observe. Every request flows through the data plane and lands in the ledger.
2. Cost / attribute. Each row is priced against the embedded snapshot and attributed — per model, per token type (input / output / cache / reasoning), per trace, per agent, per MCP tool.
3. Prove savings. In shadow mode, the ledger accumulates would-route counterfactuals, and the honesty verdict asks the only question that matters: does routing actually beat a blind model mix at the router’s own cost?
4. (Maybe) promote a route. If — and only if — the evidence holds up, you promote a route from shadow to enforced. If it doesn’t, TokenTriage tells you so, plainly.

Because attribution runs from the very first request (in ledger mode, before any routing exists), the flywheel produces evidence before it asks you to act on it. That ordering is the honest part: measurement leads, enforcement follows.

Where to go next

• To turn observation into cheaper traffic, see routing — the strategies and the policy language behind the route rung.
• To read where the money actually went, see cost analytics — attribution by model, token type, trace, agent, and tool.
• To understand what a verdict can and cannot prove, see the honesty model.
• To try the whole flow in five minutes, start with the quickstart.