Core concepts

Your provider invoice tells you the total amount due. It doesn’t tell you which requests caused the cost, which model handled them, or what changed from last month. TokenTriage fills in that missing detail by recording each request as it passes through the proxy.

When you’re ready to reduce that cost, TokenTriage can also choose among models and providers you configure. It starts by observing traffic without changing it. You decide when it may compare other routes and when it may apply them.

Two ideas explain that progression: the decision ledger records what happened, and the trust ladder controls what TokenTriage is allowed to change. If LLM gateways are new to you, start with LLM gateways, proxies, and routers. For a guided tour of one request, read How TokenTriage works.

The decision ledger

Each proxied request appends one decision row to ~/.tokentriage/decisions.jsonl. JSONL is a plain-text format with one JSON object per line. You can inspect it with ordinary text tools, and TokenTriage doesn’t require a database.

The same file also records evaluations, guardrail actions, feedback, and oracle results. Those events use their own row types, so not every line represents a request. Prompt text isn’t stored by default; a decision row stores a hash and measurements such as prompt size instead.

Here is a shortened decision row:

{
  "req_id": "tt_d5k7ntwm6utqbk4eigg3am3udm",
  "request": {
    "model": "gpt-5"
  },
  "decision": {
    "mode": "ledger",
    "upstream": "local"
  },
  "usage": {
    "input_tokens": 88,
    "output_tokens": 40,
    "source": "provider"
  },
  "cost": {
    "usd": 0.000514,
    "estimated": false
  }
}

From this row you can see which model was requested, how TokenTriage handled the request, how many tokens the provider reported, and what the request cost. usage.source: provider means the token counts came from the provider. estimated: false means the dollar figure is based on those reported counts rather than a local estimate.

An upstream is a provider endpoint you configure. In this example, the request went to the upstream named local. The complete row contains more detail for tracing, caching, pricing, and routing; Ledger fields documents every field when you need it.

The trust ladder

The next question is what TokenTriage may change about your traffic. The top-level mode setting answers it:

mode: ledger   # ledger | shadow | route

ledger is the default. A new installation records and costs requests but doesn’t choose a different model or provider. Each mode keeps the behavior of the mode before it and adds one capability.

mode	What TokenTriage does	Can the selected model or provider change?
ledger	Records and costs the request, then sends it to the model your application requested	No
shadow	Also calculates which configured route it would choose and records the comparison	No
route	Applies the routing decision and sends the request to the selected target	Yes

A routing tier is a named combination of one upstream and one model. In shadow mode, TokenTriage compares tiers without changing the response your application receives. In route mode, it may select a different tier, but your application still receives the same API format it sent.

eval is a command, not a mode. Run it between shadow and route to check whether the less expensive path still produces acceptable results. Your first routing rollout walks through the complete sequence.

One proxy, two ports

TokenTriage keeps application traffic separate from its administrative tools. The docs call these the data plane and the admin plane:

• The data plane listens on 127.0.0.1:8787 by default. Your application sends its OpenAI- or Anthropic-formatted requests here.
• The admin plane listens on 127.0.0.1:9090 by default. It serves health checks, metrics, API status, and the optional dashboard. It isn’t part of the path a model request waits on.

Both ports listen only on the same machine by default. Architecture shows the full topology and explains what to expose when you deploy TokenTriage elsewhere.

When a cost can’t be calculated

A successful request doesn’t always produce a dollar figure. If the provider omits token counts, TokenTriage estimates them when it can and labels the cost EST. If the model has no entry in the price catalog, the cost is UNPRICED. Neither label means the request cost $0.00.

The decision row records which case occurred, so reports don’t mix measured, estimated, and missing costs. Provenance and verdicts explains how those labels work across the rest of TokenTriage.