The honesty model

Documentation is where credibility is won or lost, and so is a cost dashboard. TokenTriage is built so that a number it cannot defend is unrepresentable — not rounded, not zeroed, not guessed. This page states the rules precisely; How it works tells the same story in plain language.

The invariants

These aren’t taglines — they’re constraints the system holds itself to, re-proven each release:

• Off the request hot path. No dashboard or analytics work sits on, blocks, or backpressures the proxy pipeline. A dead dashboard is invisible to your traffic.
• Fail-open, byte-faithful passthrough. Any internal error fails open to a verbatim passthrough; the proxied bytes are bytes.Equal to going direct, streaming included. Nothing here adds a new way for your agents to break.
• Provenance on every dollar. Any number that is produced or transformed keeps its pricing_snapshot, estimated flag, and source lineage.
• No heuristic identity stitching. Attribution comes exclusively from explicit identity on the wire (x-tt-* headers). A wrong ledger is worse than a narrow one, so nothing is guessed from timing or prompt prefixes.
• No stored secrets. Secret values are never persisted or displayed — only references (env-var names, secret-manager refs) and presence checks. Virtual keys are stored as SHA-256 hashes.
• Config-as-contract. Administrative writes are emitted, validated configuration — diffable, auditable, never a silent mutation of live behavior.
• The honest-verdict rule. Served-model labels can only place routing on the frontier; only counterfactual shadow duplication can show a beat; insufficient labels produce a refusal, never a number.
• JSONL is the system of record. Anything derived (the SQLite index, the dashboard) must be rebuildable from the decision log — and disposable.

Provenance & UNPRICED

Every dollar renders through a typed Money value (usd, plus estimated and unpriced flags); the ledger cost record it came from is stamped with the pricing snapshot it was computed against, and an estimated figure wears an EST flag. The load-bearing rule is what happens when a model has no price:

The mirror image matters too: a model you have declared free — a local endpoint, say — gets an explicit $0 with unpriced: false and a provenance stamp. Genuine zero and unknown are kept apart on purpose, because conflating them is exactly the failure that makes cost dashboards untrustworthy.

“We spent nothing on this model” and “we don’t know what we spent on this model” are opposite facts, and only one of them is good news.

The same discipline extends to reconciliation: when a provider invoice is matched against the ledger, the leftover delta is shown as “unexplained,” never zeroed to force the two sides to agree (see Cost optimization).

The savings verdicts

A savings figure is never a bare dollar amount — it carries a verdict and a basis. The vocabulary is closed at three, and they are different kinds of claim, not degrees of confidence. Each renders through the same typed Assessed figure, and the enum values are wire-exact.

Verdict	What it means	What licenses it
PROVEN	A measured figure — a realized per-row counterfactual on rows that actually happened after a change, or a mechanism whose avoided spend is itself replay-measured	a proven-capable method: a realized-counterfactual-v1 post-apply measurement or a cache-licensed-replay-v1 avoided-spend measurement (ProvenCapableMethods())
ESTIMATED	A model — recorded usage repriced under a different mechanism; Basis.Method names the model and Approximations lists every simulation label	the query + projection methods
UNPROVABLE	The refusal — evidence for this slice is insufficient. The figure renders no achievable dollar (null Money) and Basis.Reason says what was missing and how to get it	any method, when the licensing bound isn’t met

Quality (“did the cheaper alternative answer as well?”) and impact (“how many dollars?”) are two separate verdicts on a recommendation, and they never blend: a projection can carry shadow_proven quality and still be an ESTIMATED impact, because extrapolating measured deltas over future volume is a model. See Cost analytics for how each renders.

Two verdict vocabularies — don’t conflate them

TokenTriage surfaces two closed verdict vocabularies, and they read alike but are unrelated enums answering different questions. The savings verdict above judges a dollar figure; the routing verdict below judges whether routing beat the baseline. Neither shares a value with the other — there is no PROVEN routing verdict and no BEAT savings verdict.

	Savings / impact verdict	Routing verdict
Question it answers	Is this claimed dollar figure measured, modeled, or unprovable?	Does the router’s realized quality beat the zero-router baseline at its own cost?
Closed vocabulary	PROVEN · ESTIMATED · UNPROVABLE	BEAT · NOT_BEAT · ON_FRONTIER · CANNOT_MEASURE
Produced by	the assessment pipeline behind savings, recommendations, and reconciliation (ImpactVerdict)	eval over shadow/oracle labels (routing Verdict)
Where you see it	a dollar figure — a recommendation’s impact, a realized saving, a reconciliation delta	the routing evidence panel and the eval CLI line
Its strongest state	PROVEN — a realized post-apply or replay measurement	BEAT — a judged shadow counterfactual cleared the threshold
Its refusal	UNPROVABLE — renders no dollar, names what was missing	CANNOT_MEASURE — not enough labels to judge at all

The offline eval program that compares a cheaper tier against a baseline reports its result in the same four-state routing vocabulary — BEAT / NOT_BEAT / ON_FRONTIER / CANNOT_MEASURE, not a third enum — for the same reason: a non-counterfactual instrument can only place an arm on the frontier, and only a live counterfactual can license a BEAT. It is one honesty posture applied on the request path and off it. The measured-quality substrate is where that posture is built.

The honesty verdict for routing

Routing has its own honest question, answered by eval: does the router’s realized quality exceed the zero-router baseline at the router’s own cost? The answer is one of four frozen, wire-exact states — the enum the dashboard renders verbatim from the API, never re-worded client-side, paired with the prose verdict line the CLI emits:

Wire verdict	When it’s produced	The line the CLI emits
BEAT	Judged shadow duplication ran, labels were sufficient, and the per-cost frontier delta cleared the epsilon	verdict: routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication)
NOT_BEAT	The judged counterfactual ran with enough labels, but the delta did not clear the threshold	verdict: routing does NOT beat zero-router (Δ +0.0000) — keep shadow mode
ON_FRONTIER	Only served-model labels (oracle/outcome) exist; the router point can only sit ON or BELOW the frontier at its own cost	verdict: routing sits ON the observed-model frontier (Δ +0.0000) — served-model labels … cannot be shown to beat a blind model mix. Enable shadow duplication (shadow.duplicate) …
CANNOT_MEASURE	Not enough labels to judge routing at all — the refusal	verdict: insufficient labels to judge routing — enable oracle sampling (classifier.oracle) or shadow duplication (shadow.duplicate)

The distinctions between these four are the whole point, so it is worth seeing each verdict in the CLI’s own words.

Served-model labels only → ON_FRONTIER. With only served-model labels (oracle or outcome), each row is scored for the tier that actually served it, so the router point is a convex combination of per-model means — it can at best sit on the frontier:

verdict: routing sits ON the observed-model frontier (Δ +0.0000) — served-model labels
(quality source 'oracle') score only the tier that served each row, so routing cannot be
shown to beat a blind model mix. Enable shadow duplication (shadow.duplicate) for a
counterfactual routing-skill verdict; use calibrate to sweep thresholds.

When the same served-only labels place the router point under the frontier, the delta is signed and the served-label word flips to BELOW — but the wire verdict is still ON_FRONTIER (the state covers the on-and-below case; served labels can never demonstrate skill either way):

verdict: routing sits BELOW the observed-model frontier (Δ -0.0300) — served-model labels
(quality source 'oracle') score only the tier that served each row, so routing cannot be
shown to beat a blind model mix. Enable shadow duplication (shadow.duplicate) for a
counterfactual routing-skill verdict; use calibrate to sweep thresholds.

Judged shadow duplication clears the threshold → BEAT. Only judged shadow duplication (both tiers graded on the same query) can license a win:

verdict: routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication)

Judged counterfactual, but the delta doesn’t clear the threshold → NOT_BEAT. This is the honest “the counterfactual ran and had enough labels, but routing did not beat a blind model mix” state — distinct from the refusal, and the signal to keep shadow mode rather than promote:

verdict: routing does NOT beat zero-router (Δ +0.0000) — keep shadow mode

Not enough labels → CANNOT_MEASURE. When there aren’t enough labels to judge at all, eval refuses rather than guesses, and names the config that would fix it:

verdict: insufficient labels to judge routing — enable oracle sampling (classifier.oracle)
or shadow duplication (shadow.duplicate)

The dashboard renders these four states as chips through a single <VerdictChip> component; a promote to route without shadow evidence wears a loud route without shadow evidence badge, and the promote gate is satisfied only when the shadow source is BEAT with enough fresh labels and a matching policy fingerprint. That refusal — never a fabricated number — is the whole value of the tool.