GitHubBook a demoStart routing
Browse docs

Calibration & Evals: measured quality

The measured-quality substrate — how TokenTriage attaches honest, statistically-gated quality numbers to routed traffic, proves them on held-out data, and refuses rather than fabricates.

Last updated

On this page

A router that sends a request to a cheaper model has to answer one question honestly: did quality drop? Most routers cannot. They either never measure it, or they report a raw evaluator score as if it were a probability — a number nobody has checked against reality. The calibration & evals substrate is TokenTriage’s answer: it attaches quality numbers to the router’s own traffic, proves each one on a held-out slice of the same data, and — when a number cannot be defended — says so instead of guessing. It is the mechanism behind the promise: cut your LLM bill, after you prove the cheaper path is just as good.

This page is the front door to the section. It names the two nouns everything else is built from, states the honesty posture, draws the substrate end to end, and points you at the rest.

The two nouns

Almost everything here is either a quality estimate or a calibration, and keeping them apart is the whole discipline.

A quality estimate is a number about how good outputs are on a metric — schema_validity, groundedness, task_completion, and the rest. It never travels as a bare float; it carries an epistemic basis that says what kind of claim it is:

Basis What it means Can it be a probability?
measured A real evaluator ran, and the metric’s ground-truth class licenses measurement — an execution or rule check, or a judge that has been validated on this segment. Yes.
reported An unvalidated proxy: a raw LLM-judge score, a heuristic. Displayable and filterable, but nobody has measured its error. No — never a probability.
insufficient Not enough evidence to say anything yet. A first-class, routable state that carries a reason and need/have counts. No — it is a refusal, never a silent 0.0.

A calibration is the machine that turns an evaluator’s raw score into a probability whose error has been measured on a held-out slice of the same window. That measurement lives inside the artifact the fit produces, and it gates promotion: an artifact that failed its own measurement can be written to disk, but it can never be served. This is what earns a reported signal the right to become a measured probability — and how a measured estimate gets an interval it can defend.

The honesty posture

The honesty model states the rule that governs every number in TokenTriage: a figure the system cannot defend is unrepresentable — not rounded, not zeroed, not guessed. Calibration inherits it whole.

  • Refuse, don’t fabricate. A too-thin segment is emitted as an insufficient entry with a reason and need/have counts — never omitted, never a quiet 0.0. Absence of evidence is disclosed as absence, not printed as zero.
  • Measured, not asserted. The gate compares each segment’s measured holdout error against a threshold; a fit that cannot clear its own measurement is refused with the numbers attached.
  • One basis per claim. measured and reported are different kinds of claim, not degrees of confidence, and they are never blended into one aggregate.

This is the same posture as the router’s own honesty verdict, and the two even share a verdict vocabulary — see Eval programs & verdicts and the two verdict vocabularies note.

The substrate at a glance

Every quality number traces back to the decision ledger. Evaluators score the outputs recorded there; each score becomes an estimate carrying its basis; the calibration engine fits and gates those scores into a versioned qcal_ artifact; and the router — when you opt in — reads the active artifact to decide whether a cheaper tier has proven itself.

The measured-quality substrateA left-to-right flow that wraps onto a second row. The decision ledger feeds the evaluators, which emit a quality estimate carrying a basis of measured, reported, or insufficient. The estimate flows into calibration fit plus the B.5 reliability gate, which produces an active qcal_ artifact. The router consults that artifact through min_quality_lcb. A dashed branch from the artifact reaches the offline evals track, whose Pareto frontier and verdict form the second of the two tracks.decision ledgerevery request — costed & loggedevaluatorsrule · code · statisticaljudge · sidecarquality estimatebasis: measured, reported,or insufficientcalibration fit+ B.5 reliability gateqcal_ artifactstatus: activeroutermin_quality_lcbevalsPareto frontier · verdictoffline track

The diagram also shows the substrate’s two tracks, which share the same estimates and the same ledger but answer different questions:

  • Calibration is the online track: the probability engine that fits per-segment quality, measures its error, and publishes an artifact the router can consult on the hot path. It is what lets quality-aware routing gate a cheaper tier on a proven floor.
  • Evals is the offline track: a program that re-scores stored outputs (or, with a live data plane, re-runs them) to compare candidate routing choices on a cost/quality frontier and return a four-state verdict. It is how you prove a cheaper tier before you trust it in production.

The two meet at the estimate: an eval experiment reads the same measured/reported/insufficient bases the calibration engine produces, and a calibration is fit from the same ledger an eval program samples.

Where to go next

The section is organised so you can read the concepts first or jump straight to a task.