The honesty model — TokenTriage Docs

A cost total is useful only if you know what it leaves out. A routing result is useful only if it was compared with a meaningful baseline.

TokenTriage keeps those limits attached to the number. Missing prices remain unpriced instead of becoming $0. Savings say whether they were measured or estimated. Routing gets a win only when both choices were scored on the same requests.

Checking a cost or savings figure? Start with Provenance & UNPRICED.
Evaluating routing? Skip to why routing needs a baseline.

How TokenTriage works follows a request end to end. If ledger, shadow, and route are new to you, read the trust ladder first.

The invariants

Two rules explain the rest of this page:

Unknown stays visible. A missing price does not become zero, and missing evidence does not become a claim.
The evidence limits the conclusion. A forecast remains an estimate. A routing win requires a comparison that can show the router made a better choice.

Those rules also shape how TokenTriage behaves outside cost and routing.

Where these rules show up elsewhere

Costing, analytics, and the dashboard run off the request path, so bookkeeping does not delay your traffic.
On an internal error, the proxy falls back to verbatim passthrough, including streaming.
Request attribution comes from the x-tt-* headers your client sends. TokenTriage does not infer ownership from timing or prompt text.
The JSONL decision ledger is the record. The SQLite index and dashboard views can be rebuilt from it.
A guard that could not run records why in suspended[]; it does not look like a clean row.

Security and administrative guarantees live in security and governance.

Provenance & UNPRICED

A cost figure needs enough context to tell you whether it is complete. TokenTriage keeps three facts beside each amount:

which price-list version was used;
whether token counts were measured or estimated; and
whether TokenTriage found a price for the model.

An estimated figure carries an EST flag. A model with no matching price follows a different rule.

That $41.20 is the known spend for priced rows, not a complete total. The 116 excluded rows are not treated as free.

A model you declared free, such as a local endpoint, is different. It gets an explicit $0, unpriced: false, and the price-list version that supplied the zero rate. This keeps “we spent nothing” separate from “we do not know what we spent.”

The same rule applies during reconciliation. If the provider invoice and ledger do not match, the remaining amount is shown as unexplained instead of being changed to zero. See provider-bill reconciliation.

The ledger fields behind a cost figure

A cost is an amount in USD plus the estimated and unpriced flags. Its ledger record also carries pricing_snapshot, which identifies the price list used for the calculation. Ledger fields defines each field.

The savings verdicts

A savings figure also tells you what kind of claim it makes. TokenTriage uses three labels:

Verdict	Plain meaning	Typical example
`PROVEN`	measured from events that have already happened	savings measured after a change, or avoided spend measured by replay
`ESTIMATED`	calculated from recorded usage and stated assumptions	a recommendation about future traffic
`UNPROVABLE`	the available evidence cannot support a dollar figure	a slice with missing or insufficient evidence

These labels describe the source of the figure, not how confident the UI feels about it. An UNPROVABLE result renders no dollar and names the missing evidence.

Only two method IDs can produce a PROVEN figure. The cost figure reference lists them and the evidence each one requires.

Two verdict vocabularies — don’t conflate them

Savings and routing use different verdicts because they answer different questions.

	Savings / impact verdict	Routing verdict
Question	Is this dollar figure measured, estimated, or unavailable?	Did routing beat a blind model mix at the same cost?
Values	`PROVEN` · `ESTIMATED` · `UNPROVABLE`	`BEAT` · `NOT_BEAT` · `ON_FRONTIER` · `CANNOT_MEASURE`
Where it appears	recommendations, measured savings, and reconciliation	the routing evidence panel and `eval` output
Missing evidence	`UNPROVABLE`; no dollar is shown	`CANNOT_MEASURE`; no routing judgment is made

The verdicts can differ without contradicting each other. A recommendation may have strong quality evidence and still show ESTIMATED savings because future traffic has not happened. When you quote a verdict, say whether it describes savings or routing.

Quality labels and evidence bases are a separate vocabulary. They are collected in Calibration & evals vocabularies.

Why “routing beat the baseline” is hard to show

Suppose you use a cheap model for 70% of requests and an expensive model for 30%, without looking at the request first. That blind mix has an average cost and quality. Change the mix and you trace a line between the two models.

TokenTriage calls the best line available from blind mixes the zero-router frontier. A router must land above that line to show it chose well for individual requests. Landing on the line is no better than choosing the same proportions at random.

The line in the diagram is what a blind mix can achieve. A result above it has better quality at the same cost. A point on or below it does not show routing skill.

Your normal logs record the model the router chose and how that answer scored. They do not record how another model would have answered the same request. With only those served-model labels, a good choice and a lucky mix look the same.

Judged shadow duplication supplies the missing comparison. It sends the same request to both models and grades both answers, while your application still receives the primary response. The second graded answer is the counterfactual: what would have happened under the other choice.

This page does not cover how a model grader becomes trusted measurement. That process is explained in How a judge reaches a measured basis and Why your number is reported, not measured.

The honesty verdict for routing

eval turns the available routing evidence into one of four verdicts. The CLI, API, and dashboard use the same names.

Verdict	What it means	What to do next
`BEAT`	both choices were graded on the same requests, enough labels were available, and routing cleared the improvement threshold	if the evidence is fresh and matches the current policy, promote it to `route`
`NOT_BEAT`	the paired comparison ran with enough labels, but routing did not clear the threshold	keep `shadow` mode and adjust the policy
`ON_FRONTIER`	only served-model labels were available, so the result cannot show routing skill	enable judged shadow duplication
`CANNOT_MEASURE`	too few rows were labeled, or the filter matched no rows	collect labels; use judged shadow duplication if you need a routing-skill verdict

Despite its name, ON_FRONTIER also covers a point below the frontier. Served-model labels can place the result on or below the line, but they cannot show that per-request choices caused either position.

The CLI output for each routing verdict

With served-model labels and a point on the frontier:

verdict: routing sits ON the observed-model frontier (Δ +0.0000) — served-model labels
(quality source 'oracle') score only the tier that served each row, so routing cannot be
shown to beat a blind model mix. Enable shadow duplication (shadow.duplicate) for a
counterfactual routing-skill verdict; use calibrate to sweep thresholds.

The same wire verdict prints BELOW when the point falls under the line:

verdict: routing sits BELOW the observed-model frontier (Δ -0.0300) — served-model labels
(quality source 'oracle') score only the tier that served each row, so routing cannot be
shown to beat a blind model mix. Enable shadow duplication (shadow.duplicate) for a
counterfactual routing-skill verdict; use calibrate to sweep thresholds.

With a judged shadow comparison that cleared the threshold:

verdict: routing BEATS zero-router by 0.2500 (counterfactual: judged shadow duplication)

With a judged shadow comparison that did not clear it:

verdict: routing does NOT beat zero-router (Δ +0.0000) — keep shadow mode

With too few labels:

verdict: insufficient labels to judge routing — enable oracle sampling (classifier.oracle)
or shadow duplication (shadow.duplicate)

A filter that matches zero rows also produces CANNOT_MEASURE; the evidence API reports matched_rows: 0 beside it. Promotion checks two more conditions after BEAT: the labels must be fresh enough, and the policy fingerprint must match the policy being promoted. The fingerprint identifies the policy version under which the evidence was collected.

What this page doesn’t cover

Ledger fields defines the stored records. The trust ladder explains ledger, shadow, and route. Cache accounting explains why provider prompt-cache discounts and TokenTriage response-cache savings remain separate.