Glossary — TokenTriage Docs

This page defines every word these docs use that isn’t ordinary English. Read the first section straight through, then look terms up as you hit them.

Start with these ten

Ten terms carry the rest of the documentation. Read them before anything past What is TokenTriage?.

Token — the unit a model reads and charges in. One token is a few characters, so a short paragraph is a few hundred of them. Where the money goes
Upstream — the account and endpoint a request is actually sent to, such as your OpenAI account or your AWS Bedrock region. Providers
Tier — a nickname you give one model at one provider, so rules can say cheap and strong instead of model strings. Routing
Dialect — the shape of request and response body a provider expects. Every shipped dialect is recognised on its own paths; two of them, OpenAI’s and Anthropic’s, are also what cross-dialect translation can read from your code. Core concepts
Data plane and admin plane — two separate ports. 127.0.0.1:8787 takes your traffic; 127.0.0.1:9090 serves health, metrics and the dashboard. Architecture
The decision ledger — a text file with one line per request that went through, holding what was asked, what it cost and what was decided. Default path ~/.tokentriage/decisions.jsonl. Core concepts
Trust ladder — the three settings TokenTriage runs in, ordered by how much it may change: ledger, then shadow, then route. Core concepts
Shadow mode — the middle setting. TokenTriage works out which model it would have picked and what that would have cost, writes both down, and sends your request untouched. First routing rollout
Provenance — the fields stamped beside every dollar figure saying where it came from: which price snapshot, whether it was calculated or reported, and by whom. Cost provenance
UNPRICED — the mark on a request whose dollar figure is not known, either because the model has no price on file or because an async provider has not reported the money yet. The cost is left blank and the request counts as excluded, unlike a model priced at zero because it’s free. Where the money goes

The rest groups every other term by where you’ll meet it, without repeating those ten.

Running the proxy

Proxy — a server your application sends requests to instead of the provider. It passes them on and returns the reply.
Gateway — a proxy that also does something with the traffic on the way through: costing, logging, routing, capping. LLM gateways explained
Base URL — the address your SDK sends requests to. Point it at TokenTriage and the rest of your code is unchanged.
Ledger mode — the default. Every request is watched and costed; nothing about it changes.
Route mode — the top rung. Decisions take effect: the model name can be rewritten and a cheaper upstream chosen.
Streaming — the provider sends its answer in pieces as it’s written, instead of one complete response.
Context window — the most tokens a model accepts in one request, counting your prompt and its reply together.
Wire dialect — a provider protocol the binary speaks. 11 wire-dialect drivers are compiled in: openai, anthropic, bedrock, gemini, anthropic_vertex, elevenlabs, deepgram, assemblyai, replicate, soniox and stability. The generator counts a driver where an internal/dialect package name equals the dialect id a descriptor names. Wire dialects
Provider descriptor — the data defining a provider: authentication, price prefix, endpoints. TokenTriage ships 95 built-in provider descriptors, and adding an OpenAI-compatible one is a config change with no new Go code. Where a test drives a request through one, the generated matrix names that test; where none does, it prints the cell as ungated rather than hiding it, and ten descriptors carry no gated cell at all. The verification is against mocks, not against the providers’ live endpoints.
Cross-dialect translation — rendering a request written in one client format into the shape a different upstream dialect expects. Two client formats, OpenAI’s and Anthropic’s, reach all four translation upstreams (openai, anthropic, gemini, bedrock), streaming and not, and each of the 12 translated cells is covered by an end-to-end test you can run against a mock shaped like that upstream. It’s off until you turn it on, and any parameter that couldn’t cross is listed on the row. The dialect matrix
x-tt-* headers — the headers you set to say which trace, step, agent and tier a request belongs to. Identity is read from these rather than guessed. Headers
Trace and step — a trace is one unit of work, such as an agent run; a step is one model call inside it. Headers name both.
Step-class — what kind of step a call is: user_turn, tool_loop, or provider_state. Routing rules can match on it.
Fail open — when something inside TokenTriage goes wrong, your request is forwarded to the provider unchanged rather than failed.

Cost and pricing

Input tokens and output tokens — what you sent and what came back. They bill at different rates, output usually dearer, so both are counted separately on every row.
Prompt cache — the provider’s own reuse of a prompt prefix it has already processed. Writing costs a premium; later reads are discounted.
Response cache — TokenTriage’s own store of answers it has seen before. A hit is served locally, so there’s no provider call and no new tokens. Cache accounting
Price catalog — the price table compiled into the binary: 2,889 priced model keys, each keyed <provider>/<model_key>. That is a statistic about a vendored price table, not a claim about what TokenTriage can reach. Costing a request needs no network call.
Pricing snapshot — the stamp naming which build of that table produced a figure. This build carries litellm@2026-07-14#179affb.
Settlement — the second ledger row an async provider’s job gets, written when the provider reports what the job cost. The submit row is written at once and carries no dollar; a reader takes the settlement’s figure in place of it rather than adding the two. Reading a settled cost
EST — the mark on a figure TokenTriage worked out from token counts and its price table, rather than one the provider reported.
Metacost — what the measuring cost you. The semantic response cache makes billable embedding calls to decide whether two prompts match; metacost is that spend, reported apart from the traffic it measured.
Reconciliation residual — the part of a provider invoice your ledger rows don’t account for, shown as “unexplained” rather than folded into the matched total.
FOCUS — the FinOps industry standard for cost-export columns, so an export lands in the same tools as your cloud bills.
Chargeback and showback — two ways to split spend by team, tenant or feature. Chargeback moves money in your accounting; showback only reports who spent it.

Routing and evidence

Counterfactual — a figure for a call that didn’t happen. If TokenTriage says a cheaper tier would have cost $0.0004, that’s a counterfactual: the same request repriced against a model it never saw.
Savings verdicts — how much weight a savings figure carries. PROVEN was measured after a change went live, ESTIMATED was repriced from a model, UNPROVABLE means the evidence isn’t there. A recommendation you haven’t acted on can’t reach PROVEN.
Honesty verdict — the answer eval gives to “is routing earning its keep?”, in one of four words printed as the API returns them: BEAT, NOT_BEAT, ON_FRONTIER, CANNOT_MEASURE. Grading only served answers can place routing on the frontier; a beat takes shadow duplication.
Zero-router baseline — the best score reachable with no per-request decision, by splitting traffic between your models in fixed proportions set in advance. It’s the bar eval measures against.
Shadow duplication — sending the same query to both tiers and grading both answers. It’s the only way to see what the other tier would have said, and the only evidence that a router beats the baseline.
Calibration — fitting your routing thresholds to your own decision log rather than a guessed number. It writes a dated candidate file that changes nothing until you activate it. Calibration and evals
Cache affinity — a short-lived pin keeping later turns of one trace on the tier that warmed the provider’s prompt cache, so a warm prefix isn’t discarded.

LLM judges

LLM judge — a model asked to grade another model’s output, so quality gets a score without a person reading every row. Each graded request costs one extra model call. LLM judges
Judge subject — what a validation result is about: one judge, on one metric, in one segment, at one version of itself. Six coordinates in all, three that file the result and three that pin it to a version. How a judge reaches a measured basis
Judge method — the shape of question a judge is asked. Five are registered: pointwise-bool (pass or fail on one answer), pointwise-rubric (a score against written criteria), pairwise (which of two answers is better), reference (how close an answer is to a known-good one, also called reference-based), and panel (several judges vote). Two of the five compose from a config file; the other three are turned away at validate and again at boot. Configure a judge
Pointwise and pairwise — the two shapes of judging question. Pointwise grades one answer on its own; pairwise shows a judge two answers and asks which is better, which is why only pairwise has a presentation order to swap.
Rubric — the written criteria a rubric judge scores an answer against, named by an id in config rather than pasted into it.
Panel — one judge made of several seats drawn from more than one model family, all voting on the same item. Configuration enforces a floor of three seats and sets no ceiling, and an odd panel always decides. Run a cross-family panel
Same-family refusal — the rule that turns away a judging setup whose grader belongs to the same model family as a tier it would grade, so no model marks its own family’s homework. Position bias, self-preference and panels
Both-orderings trial — running a pairwise comparison twice with the two answers presented in each order, so a judge that prefers whichever it reads first is caught rather than believed.
Swap flip — the share of both-orderings trials whose two runs named different winners. It has a ceiling of its own, 0.10, and a flip rate of 0.0 is a measurement rather than a missing value.
Judge-uncertain — the name for a pair whose two orderings disagreed. It is counted and left out of the arm’s estimate rather than averaged in. On the wire it arrives as a flipped count, and as an abstention reason it is spelled swap_inconsistent. Judge vocabularies
Gold set — the human-labelled rows a judge is measured against, assembled from your own ledger. Rows carrying demo, synthetic, unattributed or superseded provenance are excluded and counted before anything is measured. Collect the labels a judge is measured against
Gold-set revision — a short digest naming which version of a gold set a validation run was measured against, so two runs can be told apart. It is empty wherever the record it came from has none.
Label event — one (annotator, request) pair that survived deduplication and carries a judge verdict to pair against. It is the unit the sample-size floor counts, which is why one gold set can hold 105 label events over 80 requests.
Overlap sample — the requests two annotators both labelled. Agreement between the people is measured on these rows and nowhere else, so a queue that hands every row to one person produces no overlap and no agreement figure.
Inter-annotator agreement — how far two people labelling the same rows agree, measured over the overlap sample. It gates the gold set: where the people agree too little, the judge is not measured against them at all.
Cohen’s kappa — agreement between two labellers, corrected for the agreement chance alone would produce. A kappa of 0.0 means chance level, which is a finding rather than a missing number, and it travels as a point estimate beside the count it was computed over.
Promotion — the record that a judge subject cleared every floor, filed under its judge id, metric and segment, and pinned to its model, prompt digest and evaluator version. It is what licenses a measured quality basis. Validate a judge and promote it
Demotion — a promotion withdrawn. Re-running the validation check against a subject whose promotion still stands is what withdraws one, and the subject’s timeline keeps a demoted entry saying so. Judge drift, with no monitor between checks
Method version — the digest identifying which judge method and which prompt version a reading was taken under. A prompt-regression reading means nothing without it.
Abstention — a verdict whose content is that there is no verdict, carrying one of four named reasons. It lands in a counted bucket of its own rather than in a pass rate, and it cannot also carry a score. When a judge can’t tell
Selective risk — a certification about the answers a judge is confident enough to give: how often it is wrong inside that accept region, bounded on held-out data. Where the process that answered holds no fit, the reading comes back null rather than zero.
Clopper–Pearson lower bound — the exact one-sided bound the licensing arithmetic prints as CP-LCB, computed by clopper-pearson-v1. It is the number a floor is compared against, not the estimate printed beside it.
Benchmark agreement — a judge’s agreement with a public meta-evaluation corpus. It is a pre-screen and a prompt-regression check, and it licenses no promotion, no ceiling and no measured basis, because leaderboard validity is not your-traffic validity.
Annotation queue — the surface that offers unlabelled rows to a person to grade, ranked by a strategy you choose. Labels come back through the outcomes endpoint rather than through a second door of its own. Collect the labels a judge is measured against
Lease — the hold a claim puts on the rows it offered, so two annotators are not handed the same request. Submitting a label does not release it; only expiry does.
Quality basis — what kind of claim a quality number is: measured, reported, or insufficient. It is a different enum from the guardrail basis below, and the two spell reported with different meanings. Why your number is reported, not measured

Governance and keys

Virtual key — a bearer token you mint for a team or application. It starts with ttv_, attaches to one place in your org chart, is stored as a SHA-256 hash, and is shown in full once, at mint. Governance
Tenancy node — a position in the org / team / user / key hierarchy. Limits and keys attach to a node; spend rolls up the tree.
The six limit types — budget_usd, quota_tokens, quota_requests, rpm, tpm, and concurrency. Everything you can cap is one of these.
Limit mode — whether one limit is off, shadow (records what it would have stopped) or on (stops it). Governance stays off until you enable it, but a limit you write defaults to on, so start new ones in shadow.
Row == wire — the response your caller gets and the ledger line are built from the same value before any byte goes out, so they can’t disagree about why a request was turned away.
Fail-open with disclosure — when the counter behind a limit can’t be reached, the limit is suspended. Traffic keeps flowing and the row records suspended:[<limit_id>], so you get a working request plus a note that the cap went unchecked.

Guardrails

Binding — one configured guardrail: its category, its detectors, where in the request they run, and what happens on a finding. Its id is what the row’s engaged[], fired[] and suspended[] carry. Guardrails overview
Detector — the code that reads text and reports a finding, such as a credit-card matcher. 21 are registered in this build.
Category — what a binding looks for: pii, secrets, injection, and so on. The vocabulary is a fixed set of nine, seven with a detector here. It says what the ledger can record, not what the binary can find.
Phase — when a check runs: input before the provider sees it, output on a complete reply, or stream on a reply arriving in pieces. What gets inspected
Locality — where a detector runs. embedded is in-process with no network access, judge is an LLM call, sidecar is an HTTP classifier you host. The last two need an explicit consent string in config before the proxy starts. Air gap and limits
Basis (guardrail) — where a confidence number came from, and so how much to trust it. exact: a deterministic check passed. fixture: this detector version’s measured precision. calibrated: a calibration file licenses it, and it alone may carry a range. reported: the detector’s raw score. insufficient: too little evidence for any of those. Five values, and a different enum from the quality basis above, which has three and spells reported to mean something else. Guardrail confidence
Effect — what happened to the bytes: mutated, blocked, annotated, or warned. That’s a separate fact from what the binding asked for, since redacting something absent changes nothing.
Suspension — a check that couldn’t run, recorded on the row with a reason from a fixed list such as detector_timeout. It tells a clean row from an unguarded one.

Observability and analytics

Business dimensions — the labels you attach to spend so it can be sliced later: dim.user, dim.tenant, dim.feature, dim.tag.<key>. They come from your x-tt-* headers.
Rollup tuple — the eight columns cost analytics pre-totals by, once an hour, so a dashboard query doesn’t rescan the ledger: tenant, team, key, feature, model_served, tier, provider, endpoint. Cost analytics
BubbleUp — the ranking that answers “which slice moved?”. When spend jumps, it sorts your dimensions by how much each contributed.
robust-seasonal-v1 — the method behind cost anomaly detection, printed beside every finding so you can tell what produced it. Anomaly detection
Coverage merge — how a multi-replica deployment answers without a shared log. Each replica answers from its own ledger, the results are combined, and the total says how many replicas it covers. Scale out