GitHubBook a demoStart routing
Browse docs

Core concepts

The trust ladder, the decision ledger, and the request flow — the small set of ideas that make everything else in TokenTriage predictable.

Last updated

On this page

TokenTriage is a byte-faithful LLM proxy that costs every request and, when you opt in, routes each one to the cheapest tier that can still do the job. Four ideas explain how it behaves: the planes it runs on, the request flow, the trust ladder, and the decision ledger. The honesty model then explains why the numbers can be trusted.

Two planes

TokenTriage exposes two listeners, and they never mix:

  • The data plane (default 127.0.0.1:8787) carries your traffic. It speaks the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) dialects.
  • The admin plane (default 127.0.0.1:9090) serves derived state only — /health, /metrics, /tt/api/status, and, when enabled, the dashboard at /ui/. Nothing on the admin plane sits on the request hot path.

GET /health returns 200 with a bare OK body (process-up only); provider reachability is reported separately at /tt/api/status, never fabricated.

The request flow

Every request takes one hop in and one hop out. The costing, deciding, and logging happen beside the request, not in front of it — so the ledger can never become a new way for your agents to break. Governance admission and the response cache are both opt-in and off by default.

TokenTriage request flowYour app calls the data plane on port 8787, which runs the optional governance, route and cache stages, then forwards to an upstream provider and returns a byte-faithful response. A decision ledger runs off-path and feeds the admin plane on port 9090.Your appbase-URL swapData plane :8787Governanceoff by defaultRouteledger→shadow→routeCacheoff by defaultUpstream80+ providersoff-pathDecision ledgernever blocksAdmin plane :9090derived viewsbyte-faithful response

The trust ladder

TokenTriage earns enforcement one rung at a time. The mode: setting is the whole safety story — you never route blind, and eval will tell you honestly when the next rung is not yet safe.

The trust ladder: ledger, shadow, routeThree ascending rungs. Ledger observes and costs every request while changing nothing; shadow computes the would-route decision as a counterfactual without serving it; route enforces the decision. You climb one rung at a time, promoting only on evidence.ledgerobservecost every request, mutate nothingshadowprovecounterfactual — nothing servedrouteactenforce the cheaper decisionYou climb one rung at a time —promote only on evidence.
Rung mode: What it does Client impact
Ledger ledger Observe and cost every request. Zero request mutation. None
Shadow shadow Also compute the would-route decision and its counterfactual cost, so report shows the savings you’d get — without touching a single response. None
Route route Enforce decisions: rewrite the model, pick the cheaper upstream, honor cache-affinity pins. Requests are routed

The same shadow-first ladder is reused by the other opt-in subsystems: response-cache rules default to mode: shadow before mode: on, and governance limits climb off → shadow → on. Walking the routing rung in practice is covered in the Routing guide.

The decision ledger

One ledger, one truthEvery request appends exactly one row to the append-only decision ledger (decisions.jsonl). Every derived view — cost analytics, routing evidence, governance — reads those same rows, so there is a single source of truth.Every requestone row appendedDecision ledgerdecisions.jsonlappend-onlyCost analyticsRouting evidenceGovernanceone ledger, one truth — every view reads the same rows

Every proxied request appends one JSONL row to the decision log (~/.tokentriage/decisions.jsonl). Prompt text is never stored by default — only a prompt_sha256 and counted features. A trimmed row:

{
  "req_id": "tt_d5k7ntwm6utqbk4eigg3am3udm",
  "trace": {
    "trace_id": "smoke-1",
    "agent": "smoke-agent",
    "step_class": "user_turn",
    "source": "x-tt"
  },
  "http": {
    "path": "/v1/chat/completions",
    "dialect": "openai",
    "stream": false,
    "status": 200
  },
  "request": {
    "model": "gpt-5",
    "prompt_bytes": 5,
    "messages": 1
  },
  "decision": {
    "mode": "ledger",
    "tier": "",
    "upstream": "local"
  },
  "usage": {
    "input_tokens": 88,
    "output_tokens": 40,
    "cache_read_tokens": 32,
    "total_tokens": 160,
    "source": "provider"
  },
  "cost": {
    "usd": 0.000514,
    "pricing_key": "openai/gpt-5",
    "pricing_snapshot": "litellm@2026-07-14#179affb",
    "estimated": false,
    "would_usd": null
  }
}

The dollar is hand-checkable against the embedded price table: multiply the token counts by the stamped snapshot and you get the same number every surface shows. would_usd records the counterfactual, so shadow savings are auditable.

This same ledger is what the how-to guides read from and act on: Cost analytics turns these rows into savings you can report, and Governance enforces limits against them. This is the mechanism; for the accessible narrative version — the flow diagram and the invariants in plain language — see How it works. For the precise rules that make the ledger trustworthy, continue to The honesty model.