# Why TokenTriage

The design choices behind TokenTriage, their operational trade-offs, and when another gateway is a better fit.

Source: https://tokentriage.com/docs/introduction/why-tokentriage

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';

Most LLM gateways can sit between your application and its model providers, record usage, and
route traffic. The meaningful differences are operational: who runs the gateway, what changes by
default, how it calculates cost, and what happens when something fails.

TokenTriage is a strong fit if you want a self-hosted gateway that starts by recording
per-request cost and leaves routing off until you enable it. It is a weaker fit if you want a
hosted control plane, the broadest provider catalog, or advanced failover.

This page explains those trade-offs. If **LLM gateway** is a new term, start with
[What is TokenTriage?](/docs/introduction/what-is-tokentriage). For a feature-by-feature comparison
with LiteLLM, Portkey, Helicone, OpenRouter, and Bifrost, see
[How TokenTriage compares](/docs/introduction/how-tokentriage-compares).

## Five design choices and their trade-offs

### Self-hosted, with no required external database

TokenTriage is a single static Go binary, about **10.7 MB**. Its price table is compiled in, so it
can calculate request costs without a network call. By default, it writes one line of JSON per
request to `~/.tokentriage/decisions.jsonl`.

Some optional features add more state. The dashboard builds a local SQLite index that you can
delete and rebuild. Virtual keys add a local SQLite database for key hashes and tenant settings.
Redis or Postgres is needed only when you opt into shared state across replicas or Redis-backed
caching. The trade-off is straightforward: you avoid a required database service, but you run and
monitor the gateway yourself.

### Rollback is one configuration change

In the default `ledger` mode, TokenTriage returns the provider's response body byte-for-byte,
including streamed responses. It adds an `x-tt-request-id` response header so you can find the
corresponding ledger row. If you configure `api_key_env`, it also replaces the credential sent by
your client with the provider key from that environment variable.

If an internal processing step fails, TokenTriage records the error and falls through to
passthrough instead of replacing the provider response with an accounting error. This protection
does not cover an unavailable process or network path: TokenTriage is still on the request path
and needs the same availability planning as any other proxy.

To remove it, point your application's base URL at the provider again. There is no provider-side
schema to undo, data migration to run, or TokenTriage client library to remove. The
[deployment guide](/docs/operate/deployment) covers availability and multi-replica options.

### Observe before you automate

A new installation starts in `mode: ledger`. It calculates and records the cost of each request,
but it does not route requests, return cached responses, enforce budgets, inspect content, or
export telemetry. [Features at a glance](/docs/introduction/features) shows every optional system
and whether it is enabled by default.

If you want to evaluate routing, the next mode is `shadow`. The router records the model it would
have selected and the savings that decision would have produced, while your application still
receives the response from the model it requested. This lets you review proposed changes before
routing affects live traffic. The full `ledger → shadow → route` progression is explained in
[Core concepts](/docs/concepts/core-concepts#the-trust-ladder).

<Callout type="note">
Governance is off in a new installation. After you enable it, a new limit defaults to enforcing.
Set the limit to `mode: shadow` first if you want to review what it would block.
</Callout>

### Every cost names its inputs

Every calculated cost includes the token counts, the catalog entry used for pricing, and the
pricing snapshot. Here is the relevant part of one ledger row:

```json
{
  "usage": {
    "input_tokens": 4213,
    "output_tokens": 618,
    "source": "provider"
  },
  "cost": {
    "usd": 0.007303,
    "pricing_key": "anthropic/claude-haiku-4-5",
    "pricing_snapshot": "litellm@2026-07-14#179affb"
  }
}
```

At this snapshot, `anthropic/claude-haiku-4-5` costs $1.00 per million input tokens and $5.00 per
million output tokens. The calculation is:

```text
4,213 input  × $1.00 / 1M = $0.004213
  618 output × $5.00 / 1M = $0.003090
                            ----------
                            $0.007303
```

The same figure appears in terminal summaries, the dashboard, and SDK receipts.
`source: "provider"` means the token counts came from the provider's response rather than an
estimate.

If the catalog cannot price a model, the row is marked `unpriced` and excluded from dollar totals.
The summary also reports how many rows it excluded. A genuinely free model receives a real `$0`,
so an unknown price is not mistaken for a free request. The full rules are in
[the honesty model](/docs/concepts/honesty-model).

### Your provider accounts and traffic remain yours

Configuration stores the name of an environment variable, such as `api_key_env: OPENAI_API_KEY`,
instead of storing the provider credential itself. If you omit `api_key_env`, TokenTriage forwards
the inbound `Authorization` header. You keep your provider accounts and continue paying providers
directly.

TokenTriage does not send usage telemetry to a TokenTriage-operated service. Pricing works offline
because the table is compiled into the binary. Guardrail options that call an external service
require an explicit consent setting before they start. The details, including the limits of an
air-gapped deployment, are in
[Air gap and limits](/docs/guardrails/air-gap-and-limits).

## What you take on by self-hosting

Self-hosting means you are responsible for running, monitoring, and updating a process on your
application's request path. It also adds one network hop.

In the published benchmark, TokenTriage added **0.148 ms** of gateway overhead, with a 95%
confidence interval of 0.146 to 0.150 ms. The test used a zero-delay mock provider, so it isolates
gateway work rather than measuring end-to-end application speed. The full setup, limitations, and
competitor results are in
[Performance evidence](/docs/introduction/how-tokentriage-compares#performance-evidence).
The historical comparison harness remains in the repository. TokenTriage's separate
micro-benchmarks are listed in [Benchmarks](/docs/reference/benchmarks).

## When another gateway is a better fit

TokenTriage makes specific operational trade-offs. Another gateway is likely a better fit when:

- **You do not want to operate another service.** A hosted gateway gives you a managed control
  plane and no proxy process to maintain.
- **Broad provider and endpoint coverage is your first requirement.** TokenTriage's catalog is
  smaller than LiteLLM's. Check [Providers and dialects](/docs/build/providers) for the endpoint
  you need before adopting it.
- **You need advanced failover or load balancing.** TokenTriage supports weighted pools,
  least-latency selection, and passive cooldown. It does not provide automatic retry to another
  upstream, active health probes, hedged requests, or ordered per-model fallback chains in
  configuration.
- **You need a large managed ecosystem.** Hosted gateways have more integrations, plugins,
  community history, and existing operational guidance. TokenTriage is a smaller project.

The full list, including streaming guardrails, per-request overrides and agent attribution, is the
gap table in
[Where TokenTriage is behind today](/docs/introduction/how-tokentriage-compares#where-tokentriage-is-behind-today).

<Callout type="warning">
TokenTriage is beta software. The wire behaviour and the ledger schema are stable and versioned,
but treat it as beta when you plan a rollout.
</Callout>

## What adopting it looks like

Start with two steps, then decide whether you want TokenTriage to change traffic.

1. **Change one base URL.** Your SDK, your calls and your keys stay put. See the
   [Quickstart](/docs/get-started/quickstart).
2. **Read the ledger.** Every request now has a costed row and a request id you can join on.
   Roll it up from the terminal:

   ```console
   $ tokentriage top --by model
   ```

3. **Optionally enable one control in shadow mode.** Review the proposed routing decisions or
   budget blocks before enforcing them. You can also stop after step two and use `ledger` mode only.

<NextSteps items={[
  {
    title: "Quickstart",
    href: "/docs/get-started/quickstart",
    description: "Install, point one base URL at the proxy, and read the dollar cost of your first request."
  },
  {
    title: "How TokenTriage compares",
    href: "/docs/introduction/how-tokentriage-compares",
    description: "Compare TokenTriage with LiteLLM, Portkey, Helicone, OpenRouter, and Bifrost, including where each is stronger."
  },
  {
    title: "Features at a glance",
    href: "/docs/introduction/features",
    description: "Every subsystem, what it does, and whether it's on by default."
  },
  {
    title: "Benchmarks",
    href: "/docs/reference/benchmarks",
    description: "See the measured gateway overhead, binary size, test conditions, and limitations."
  }
]} />