Calibration & Evals: measured quality

Routing a request to a cheaper model raises one question: did quality drop? This section is how you answer it with numbers rather than a hunch.

Calibration and evals let you:

• score your own traffic on metrics like schema_validity, groundedness, or task_completion;
• turn those scores into probabilities whose error has been measured on data the fit never saw;
• gate a cheaper tier on a quality floor, so routing only downgrades where the floor holds.

Start with Calibrate quality from your logs if you want to run it; read on for the model behind it.

Two nouns to know

Almost everything here is either a quality estimate or a calibration.

A quality estimate is a number about how good outputs are on one metric. It does not travel as a bare float: it carries a basis saying what kind of claim it is.

Basis	What it means	Can it be a probability?
measured	A real evaluator ran, and the metric’s ground-truth class licenses measurement — an execution or rule check, or a judge validated on this segment.	Yes.
reported	An unvalidated proxy: a raw LLM-judge score, a heuristic. Displayable and filterable, but nobody has measured its error.	No.
insufficient	Not enough evidence to say anything yet. A first-class, routable state carrying a reason and need/have counts.	No — it reports the gap rather than a silent 0.0.

A calibration turns an evaluator’s raw score into a probability whose error has been measured on a held-out slice of the same window. That measurement lives in the artifact the fit produces, and it gates promotion: an artifact that fails its own measurement can be written to disk and can then be served by nothing.

A too-thin segment therefore comes back as an insufficient entry with its need/have counts rather than a quiet 0.0, and measured and reported are never blended into one aggregate — they are different kinds of claim, not degrees of confidence. The router’s own verdict follows the same rule; see Eval programs & verdicts and the two verdict vocabularies note, which are the pair most often mixed up.

How the pieces fit

Every quality number traces back to the decision ledger. Evaluators score the outputs recorded there; each score becomes an estimate carrying its basis; the calibration engine fits and gates those scores into a versioned qcal_ artifact; and the router, when you opt in, reads the active artifact to decide whether a cheaper tier has earned the traffic.

The diagram also shows the two tracks, which share the same estimates and the same ledger but answer different questions:

• Calibration is the online track: the probability engine that fits per-segment quality, measures its error, and publishes an artifact the router can consult on the hot path. It is what lets quality-aware routing gate a cheaper tier on a proven floor.
• Evals is the offline track: a program that re-scores stored outputs (or, with a live data plane, re-runs them) to compare candidate routing choices on a cost/quality frontier and return a four-state verdict. It is how you prove a cheaper tier before you trust it in production.

The two meet at the estimate: an eval experiment reads the same measured/reported/insufficient bases the calibration engine produces, and a calibration is fit from the same ledger an eval program samples.

Where to go next

The section is organised so you can read the concepts first or jump straight to a task.

• Concepts explain how it works: this page, How calibration works (the fit engine), The calibration lifecycle (states, refit, and audit), Evaluators & metrics, Eval programs & verdicts, and Quality-aware routing.
• How-tos get a job done: calibrate quality from your logs, prove a cheaper tier, gate routing on a proven floor, validate an LLM judge, and keep calibrations fresh.
• Reference is the exact surface: the evals & calibration configuration, the quality & evals API, and the closed vocabularies.

One evaluator kind has a section of its own, because a model that grades your outputs has to be graded too. Configuring a judge, collecting the human labels it is measured against, running the check that promotes it, and reading why a promoted judge’s figure has not moved are all in LLM judges. The page above on validating one is the orientation; the procedure lives there.