GitHubBook a demoStart routing
Browse docs

Architecture & components

The whole of TokenTriage on one page — one Go binary running two planes around a decision ledger, the tools you drive it with (SDKs, CLI, dashboard), and what stays external. A very high-level map.

Last updated

On this page

TokenTriage is deliberately small: one Go binary you place in front of your LLM traffic, with no sidecar runtime and no external database to stand up. This page is the map — the handful of parts, and how they fit together — so the how-to guides land in a frame you already have.

TokenTriage components at a glanceOne TokenTriage Go binary runs a data plane on port 8787 and an admin plane on port 9090 around an off-path decision-ledger JSONL file. Your agents on the left and upstream providers on the right exchange verbatim traffic through the data plane. The Python and TypeScript SDKs and the CLI are the tools you drive it with, a config file feeds it, and an opt-in SQLite governance store backs it. The admin plane serves the dashboard, metrics and management API to operators.Config — tokentriage.yamlvalidated · config-as-contractTokenTriage — one Go binaryData plane127.0.0.1:8787byte-faithful proxy + routerDecision ledger~/.tokentriage/decisions.jsonlone row per request · off-pathAdmin plane127.0.0.1:9090dashboard /ui/ · /metrics · /healthmanagement APIcost + decisionread modelsYour agents / appsbase-URL swapopenai / anthropic dialectUpstream providersopenai · anthropicbedrock · gemini · …verbatimverbatimSDKspython · typescriptCLItop · report · eval · keysinstrumentreadGovernance storeSQLite (opt-in)You (operator)read-only surfacesdashboard · APIOne binary, two planes, one ledger — the proxy hot path never depends on any off-path surface

The one binary, two planes

Everything ships as a single static binary (no CGO; the pricing table is compiled in via go:embed). At runtime it binds two listeners that never share a door:

  • The data plane (127.0.0.1:8787) is the hot path — a byte-faithful proxy and router that your agents talk to. It speaks the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) dialects and forwards to your configured upstreams.
  • The admin plane (127.0.0.1:9090) is everything off the hot path — /health, /metrics, /tt/api/status, the embedded dashboard at /ui/, and the management API.

Keeping them separate is a security and reliability decision: your LLM traffic and your operational surface never mix, and an admin-plane bind failure is non-fatal — the data plane keeps serving even if the admin listener can’t come up.

The decision ledger is the source of truth

As each response passes through, the data plane writes one JSONL row to the decision ledger (~/.tokentriage/decisions.jsonl) — off the request path, so it can never backpressure your traffic. Prompt text is not stored by default: a row carries a prompt_sha256 and counted features, not the words.

The parts

Component Where it runs What it is
Data plane in the binary, :8787 The byte-faithful proxy + router your agents call. Costs every request; routes when you opt in.
Admin plane in the binary, :9090 Off-path surfaces: health, Prometheus metrics, provider status, the dashboard, and the management API.
Decision ledger decisions.jsonl on disk The append-only system of record — one costed row per request.
Config tokentriage.yaml Declares upstreams, tiers, routing, caching, and governance. Config-as-contract: administrative writes are validated and diffable, never a silent mutation.
Governance store embedded SQLite (opt-in) Backs virtual keys and per-tenant limits. Off by default; a shared_state Redis/Postgres tier is opt-in for multi-replica coordination.
SDKs in your app (py / ts) Optional zero-dependency instrumentation — emit the x-tt-* headers, read a typed receipt, pull the honest dollar.
CLI your terminal run, wrap, top, report, eval, validate, calibrate, keys, demo — drive and read the proxy.
Dashboard admin plane /ui/ The trust dashboard — the same numbers as the CLI, with the same honesty posture.

What stays external

Two things are deliberately not part of the binary, and never move into it:

  • Your upstream providers. TokenTriage forwards to OpenAI, Anthropic, Bedrock, Gemini, or any OpenAI-compatible endpoint — with your keys, resolved per-upstream from env-var names. It brokers no third-party marketplace and never holds your provider secret.
  • Your agents. Nothing about your application changes beyond the base-URL swap; the SDKs are an optional convenience, never a requirement.