Architecture and components

Before you put anything in front of production traffic, you want to know what it adds to your system and what you’re responsible for operating. In the default setup, you run one TokenTriage process. It listens on two network ports and appends request records to one file called the decision ledger.

You don’t need to run a database, message broker, or helper on every application host. TokenTriage is one static Go binary of about 10.7 MB, and its container image is about 11 MB. The price table is compiled into the binary, so a host without outbound internet access can still calculate dollar figures. Some optional features add state; those exceptions are called out below.

Two things stay outside, and they’re the ones you already run: your provider accounts, and your application.

You can read this page on its own. If you prefer to start with a running process, use the Quickstart. How it works follows one request end to end, while Core concepts collects the product vocabulary in one place.

The map

Start with the picture. The rest of the page walks through it.

Read it left to right: your application sends a request through the TokenTriage process to a provider, and the provider’s response returns along the same solid path.

Dashed lines mark work outside that live request path: recording the ledger row, building dashboard views, reading data with the CLI, and using the optional governance store. The main takeaway is that provider traffic crosses one TokenTriage process. It doesn’t wait for the operator tools or the ledger index.

Two ports, and what each one is for

A listener is a TCP port where a process accepts connections. TokenTriage opens two and keeps them separate. Both use 127.0.0.1 by default, which means only applications on the same machine can connect until you change the address.

Port 8787 handles application requests. TokenTriage calls this the data plane. It accepts OpenAI requests at /v1/chat/completions and Anthropic requests at /v1/messages, forwards them to the provider endpoint in your config, and returns the provider’s response unchanged, including streamed responses. Elsewhere in the docs, these request formats are called dialects, and a configured provider endpoint is an upstream.

Port 9090 serves operator-facing endpoints. TokenTriage calls this the admin plane; your application doesn’t call it. GET /health reports that the process is running, /metrics exposes metrics in Prometheus format, and /tt/api/status reports whether each provider can be reached. The same port serves the management API and, when enabled, the dashboard at /ui/.

Keeping the listeners separate has three benefits:

• You can expose the data plane to your applications while keeping the admin plane on the same machine, or apply different firewall rules to each port.
• An operator pulling a heavy report doesn’t compete with proxy traffic.
• If port 9090 is already in use, the data plane still starts and serves traffic. The startup message identifies the failed admin port and omits the dashboard address.

tokentriage validate prints both addresses before you start anything. Other details are trimmed from this example:

$ tokentriage validate --config configs/tokentriage.example.yaml
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 3
tiers:     4 (cheapest -> most capable)
rules:     3
OK: configuration is valid

listen is the data-plane address, while admin is the operator-facing address. A tier is a named model target ordered from cheapest to most capable. It is defined with the rest of the vocabulary in Core concepts.

The file you need to preserve

For every provider response, TokenTriage appends one JSON record to the decision ledger at ~/.tokentriage/decisions.jsonl. The .jsonl suffix means JSON Lines: a text file with one JSON object per line.

The request doesn’t wait for the ledger write. A slow or full disk won’t delay the provider response, although the ledger row can be missing if the write fails.

The dashboard builds a SQLite index from the ledger. That index can be rebuilt; the ledger is the file you preserve. Governance is the exception: enabling it adds a separate store for keys and limits, described in the following section.

What a ledger row holds, including the hash stored in place of prompt text, is introduced under the decision ledger and listed field by field in the ledger field reference.

What runs by default and what is optional

The components are easier to reason about when they’re grouped by role. This table shows what is available in the default setup and what changes only when you opt in.

Group	Part	Default	What it does
request path	data plane	on	forwards requests and calculates their cost; routing and guardrails stay off until you enable them
operator path	admin plane	on unless its port is unavailable	serves health, metrics, status, and management endpoints without sharing the request port
recorded state	decision ledger	on	appends one JSON record per proxied request
configuration	flags or tokentriage.yaml	file optional	names providers and controls optional routing, caching, guardrails, and governance
optional interface	dashboard	on in zero-config quickstarts; otherwise off	shows the same ledger data that the CLI reads; --no-dashboard turns it off
optional state	governance store	off	holds virtual keys and per-tenant limits in embedded SQLite
developer tools	CLI and SDKs	CLI included; SDKs optional	inspect data or add attribution headers and typed cost receipts to your application

Guardrails can inspect request and response text

Guardrails run on the data plane only after you enable them. Built-in detectors stay inside the TokenTriage process. Sending text to another model for scoring or to a classifier you run yourself requires the explicit setting external: allow. The same rule applies when that classifier runs on the same machine. See Air-gap and known limits.

Governance adds stored state

A virtual key is a token you give to a team or tenant so its spending and limits can be tracked separately from your provider key. On one TokenTriage process, those keys and limits live in embedded SQLite. Multiple replicas require shared Redis or Postgres state so every replica counts against the same limits. See Governance and Scale out.

The SDKs are conveniences rather than dependencies. A base-URL change is enough to use TokenTriage without adding a library. See SDK overview.

What stays outside the binary

Your providers. TokenTriage forwards to accounts you already hold at OpenAI, Anthropic, Bedrock, Gemini, or any OpenAI-compatible endpoint. Your config names the environment variable that holds each credential; it doesn’t contain the credential itself. If you don’t name one, TokenTriage forwards the request’s Authorization header unchanged. There’s no billing or settlement code in the binary, so you continue buying tokens directly from your providers.

Your application. The integration is a base-URL change. Your prompts, your framework, and your deployment stay where they are.

Latency and failure behavior

TokenTriage adds under 0.15 ms of overhead in the published benchmark run, whose pinned versions, equal-core budget, and confidence interval are in Benchmarks.

Where it can surprise you

• The dashboard default flips with how you start the process. A config-file run leaves it off; the zero-config quickstarts turn it on.
• Governance as a whole is off, but a limit you write defaults to mode: on, meaning enforcing. Introduce a new limit in mode: shadow first.
• Both ports accept connections only from the same machine by default. Opening either one to another host is a deployment decision you make yourself. This is beta software.

This page doesn’t cover how a routing decision gets made, which is How it works, or how to run several replicas, which is Scale out.