Why TokenTriage

Most LLM gateways can sit between your application and its model providers, record usage, and route traffic. The meaningful differences are operational: who runs the gateway, what changes by default, how it calculates cost, and what happens when something fails.

TokenTriage is a strong fit if you want a self-hosted gateway that starts by recording per-request cost and leaves routing off until you enable it. It is a weaker fit if you want a hosted control plane, the broadest provider catalog, or advanced failover.

This page explains those trade-offs. If LLM gateway is a new term, start with What is TokenTriage?. For a feature-by-feature comparison with LiteLLM, Portkey, Helicone, OpenRouter, and Bifrost, see How TokenTriage compares.

Five design choices and their trade-offs

Self-hosted, with no required external database

TokenTriage is a single static Go binary, about 10.7 MB. Its price table is compiled in, so it can calculate request costs without a network call. By default, it writes one line of JSON per request to ~/.tokentriage/decisions.jsonl.

Some optional features add more state. The dashboard builds a local SQLite index that you can delete and rebuild. Virtual keys add a local SQLite database for key hashes and tenant settings. Redis or Postgres is needed only when you opt into shared state across replicas or Redis-backed caching. The trade-off is straightforward: you avoid a required database service, but you run and monitor the gateway yourself.

Rollback is one configuration change

In the default ledger mode, TokenTriage returns the provider’s response body byte-for-byte, including streamed responses. It adds an x-tt-request-id response header so you can find the corresponding ledger row. If you configure api_key_env, it also replaces the credential sent by your client with the provider key from that environment variable.

If an internal processing step fails, TokenTriage records the error and falls through to passthrough instead of replacing the provider response with an accounting error. This protection does not cover an unavailable process or network path: TokenTriage is still on the request path and needs the same availability planning as any other proxy.

To remove it, point your application’s base URL at the provider again. There is no provider-side schema to undo, data migration to run, or TokenTriage client library to remove. The deployment guide covers availability and multi-replica options.

Observe before you automate

A new installation starts in mode: ledger. It calculates and records the cost of each request, but it does not route requests, return cached responses, enforce budgets, inspect content, or export telemetry. Features at a glance shows every optional system and whether it is enabled by default.

If you want to evaluate routing, the next mode is shadow. The router records the model it would have selected and the savings that decision would have produced, while your application still receives the response from the model it requested. This lets you review proposed changes before routing affects live traffic. The full ledger → shadow → route progression is explained in Core concepts.

Every cost names its inputs

Every calculated cost includes the token counts, the catalog entry used for pricing, and the pricing snapshot. Here is the relevant part of one ledger row:

{
  "usage": {
    "input_tokens": 4213,
    "output_tokens": 618,
    "source": "provider"
  },
  "cost": {
    "usd": 0.007303,
    "pricing_key": "anthropic/claude-haiku-4-5",
    "pricing_snapshot": "litellm@2026-07-14#179affb"
  }
}

At this snapshot, anthropic/claude-haiku-4-5 costs $1.00 per million input tokens and $5.00 per million output tokens. The calculation is:

4,213 input  × $1.00 / 1M = $0.004213
  618 output × $5.00 / 1M = $0.003090
                            ----------
                            $0.007303

The same figure appears in terminal summaries, the dashboard, and SDK receipts. source: "provider" means the token counts came from the provider’s response rather than an estimate.

If the catalog cannot price a model, the row is marked unpriced and excluded from dollar totals. The summary also reports how many rows it excluded. A genuinely free model receives a real $0, so an unknown price is not mistaken for a free request. The full rules are in the honesty model.

Your provider accounts and traffic remain yours

Configuration stores the name of an environment variable, such as api_key_env: OPENAI_API_KEY, instead of storing the provider credential itself. If you omit api_key_env, TokenTriage forwards the inbound Authorization header. You keep your provider accounts and continue paying providers directly.

TokenTriage does not send usage telemetry to a TokenTriage-operated service. Pricing works offline because the table is compiled into the binary. Guardrail options that call an external service require an explicit consent setting before they start. The details, including the limits of an air-gapped deployment, are in Air gap and limits.

What you take on by self-hosting

Self-hosting means you are responsible for running, monitoring, and updating a process on your application’s request path. It also adds one network hop.

In the published benchmark, TokenTriage added 0.148 ms of gateway overhead, with a 95% confidence interval of 0.146 to 0.150 ms. The test used a zero-delay mock provider, so it isolates gateway work rather than measuring end-to-end application speed. The full setup, limitations, and competitor results are in Performance evidence. The historical comparison harness remains in the repository. TokenTriage’s separate micro-benchmarks are listed in Benchmarks.

When another gateway is a better fit

TokenTriage makes specific operational trade-offs. Another gateway is likely a better fit when:

• You do not want to operate another service. A hosted gateway gives you a managed control plane and no proxy process to maintain.
• Broad provider and endpoint coverage is your first requirement. TokenTriage’s catalog is smaller than LiteLLM’s. Check Providers and dialects for the endpoint you need before adopting it.
• You need advanced failover or load balancing. TokenTriage supports weighted pools, least-latency selection, and passive cooldown. It does not provide automatic retry to another upstream, active health probes, hedged requests, or ordered per-model fallback chains in configuration.
• You need a large managed ecosystem. Hosted gateways have more integrations, plugins, community history, and existing operational guidance. TokenTriage is a smaller project.

The full list, including streaming guardrails, per-request overrides and agent attribution, is the gap table in Where TokenTriage is behind today.

What adopting it looks like

Start with two steps, then decide whether you want TokenTriage to change traffic.

1. Change one base URL. Your SDK, your calls and your keys stay put. See the Quickstart.

2. Read the ledger. Every request now has a costed row and a request id you can join on. Roll it up from the terminal:

   $ tokentriage top --by model

3. Optionally enable one control in shadow mode. Review the proposed routing decisions or budget blocks before enforcing them. You can also stop after step two and use ledger mode only.