How calibration works

A quality estimate starts life as a raw number an evaluator produced — a judge’s claimed probability, an NLI score, a rule’s 1/0. Calibration is the pure function that turns that raw signal into a probability whose error has been measured on a held-out slice of the same window, and then decides — segment by segment — whether the result may ever be served. This page walks the fit engine end to end.

Everything here runs tokentriage quality calibrate, which fits a candidate artifact and changes nothing else. (This is the probability calibration; do not confuse it with the routing-threshold sibling tokentriage calibrate — see The calibration lifecycle.)

From a raw score to a measured probability

The engine is a pure function of the joined decision log: same observations, same options, same bytes out, so a refit over the same window reproduces the artifact exactly. It does exactly one honest thing — it takes an evaluator’s raw scores paired with the realized outcomes they should have predicted, fits a map from score to probability, and then measures how wrong that map is on data it never saw during fitting. That measured error is the number the whole product hangs on: it is stored in the artifact, and it is what later licenses (or refuses) a claim.

Two facts shape the fit before any arithmetic happens:

• It excludes what it must not learn from. Demo rows and synthetic / dataset rows are dropped unconditionally and counted — there is no config to turn the exclusion off, because synthetic evidence licensing a real probability claim is exactly the failure the substrate exists to prevent. The counts appear in the artifact’s exclusion ledger.
• It fits per (segment × evaluator), not per segment. Two evaluators’ raw scores live on different scales — a judge’s 0.1–0.9 grid is not a human annotator’s 1/0 — so pooling them would calibrate a mixture of incomparable numbers. Each evaluator’s signal is fit separately inside the segment.

Segments — the Mondrian coordinates

A segment is the unit a probability is fit for. Its key is four coordinates:

{ metric, tier, model, endpoint }

The load-bearing coordinate is model, and it is the model served, not the model requested. Routing rewrites the model — that is the whole point of routing — so on a routed row the requested and served models are different by construction. Keying the fit on the request would file the cheap model’s observed pass-rate under the frontier model’s name: a segment that silently describes the wrong population. The fit reads the served model (decision.model_rewrite.to), which is the same coordinate the router later probes, so the router’s lookup joins this segment rather than missing it forever behind a fail-open absence.

The 50/25/25 split and the held-out measurement

Within each unit, the observations are ordered in request time and split into three contiguous bands:

• Fit (50%) — the data the probability map is trained on.
• Calibrate (25%) — the slice the conformal step stands on, which the map has not seen.
• Holdout (25%) — the untouched final quarter the error is measured on.

Ordering by time, never shuffling, is deliberate: a fitter that could re-order its own training set could quietly launder tomorrow’s data into today’s fit. By the time any data reaches the fitter the split has already happened and is not re-litigable.

What gets fit depends on the score’s own shape. A segment whose evaluator emits three or more distinct score values gets a fitted probability map, conformalized on the calibrate split; its holdout error is a coverage gap (coverage_gap) — the distance between the interval’s nominal coverage and the coverage it actually delivered. A segment whose score takes fewer than three distinct values (a rule’s pass/fail) has no map to fit at all; it takes a base-rate branch, where the fitted object is the segment’s pass rate as a constant forecast and the holdout error is an ECE (ece) — does the past rate transfer to the future? Either way the measured error is a single non-negative number, and either way it comes off the holdout.

The B.5 gate

Measuring an error is not enough; the engine has to decide whether the error is small enough to serve. That is the gate, and it compares the measured error against a threshold that is not a flat constant:

threshold = tolerance + allowance

• tolerance is the configured target — 0.02 for a coverage gap, 0.05 for an ECE by default. It is raise-only: an operator with a safety-critical workload can demand a tighter gate, but a tolerance looser than the default is refused at construction, because a knob that could quietly loosen the product’s one credibility asset is not a knob worth having.
• allowance is the sampling-noise critical value at this holdout’s size and structure — the error a perfectly calibrated segment would show anyway, purely because the holdout is finite. At the licensing floor the allowance can be several times the tolerance, so the effective gate runs well above the bare 0.05.

Because the allowance is the number on a tiny holdout, the gate refuses to run at all below MinHoldoutM = 30 rows — beneath that there is no measurement to correct, and a gate that passed a maximally-wrong two-row holdout would license anything.

The gate also stores what “passed” could not have told you on its own, and each is a disclosure rather than a gate:

• min_detectable_error — the smallest true error this gate catches at least half the time. A gate can control its false-refusal rate exactly and still be unable to see a 0.10 overconfidence at sixty rows; publishing this turns “passed” into “passed, and here is what passing could not rule out.”
• false_refusal_rate (β) — the rate at which a segment whose true error sits at the tolerance is refused anyway.
• null_mean — what a perfectly calibrated segment would measure here, on average. An error sitting within a hair of it is the signature of a measurement that measured the holdout size rather than the calibration.
• raw_error — the same measurement made on the uncalibrated signal, so the report can say what calibration bought: “the judge’s own probabilities were off by 0.35; the fitted map is off by 0.02.”

Insufficiency is emitted, not omitted

The common case is not a passing gate; it is a segment too thin to make a claim about at all. The substrate’s differentiator is saying so out loud. Such a segment is written into the artifact as an insufficient entry — never dropped, because a segment missing from the artifact is indistinguishable from a segment that had no traffic — carrying a reason, a need, a have, and a hint. The reasons are a closed set:

Reason	What it means
below_min_n	The calibration split (or the holdout) is under its evidence floor.
alpha_infeasible	There is enough data to fit, but not enough to attain the requested confidence at all at this size.
no_variation	Every observation carries the same outcome, so no probability map is identifiable — a constant fit is not a calibration.
gate_failed	The fit was measured and its holdout error did not clear the gate.

The evidence floor is demanding on purpose. The default min_n of 50 floors the calibration split — a quarter of the segment’s rows under the 50/25/25 split — so a segment needs roughly 200 fit-eligible observations before any claim about it is licensable. insufficient is meant to be the frequent answer; the coverage meter (quality coverage) exists precisely to show you how much of your traffic clears it and what the biggest gaps need.