# How calibration works How a raw evaluator score becomes a probability whose error is measured on held-out data, and gated before it can be served. Source: https://tokentriage.com/docs/calibration/how-calibration-works --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import CalibrationFitDiagram from '../../../components/CalibrationFitDiagram.astro'; A [quality estimate](/docs/calibration/overview#two-nouns-to-know) starts life as a raw number an evaluator produced — a judge's claimed probability, an NLI score, a rule's `1`/`0`. Calibration is the pure function that turns that raw signal into a probability **whose error has been measured on a held-out slice of the same window**, and then decides — segment by segment — whether the result may ever be served. This page walks the fit engine end to end. Everything here runs `tokentriage quality calibrate`, which fits a **candidate** artifact and changes nothing else. (This is the *probability* calibration; do not confuse it with the routing-threshold sibling `tokentriage calibrate` — see [The calibration lifecycle](/docs/calibration/lifecycle#two-calibrations-one-chassis).) ## From a raw score to a measured probability The engine is a pure function of the joined decision log: same observations, same options, same bytes out, so a refit over the same window reproduces the artifact byte for byte. It does exactly one honest thing — it takes an evaluator's raw scores paired with the realized outcomes they should have predicted, fits a map from score to probability, and then *measures how wrong that map is* on data it never saw during fitting. That measured error is the number the whole product hangs on: it is stored in the artifact, and it is what later licenses (or refuses) a claim. Two facts shape the fit before any arithmetic happens: - **It excludes what it must not learn from.** Demo rows and synthetic / dataset rows are dropped unconditionally and *counted* — there is no config to turn the exclusion off, because synthetic evidence licensing a real probability claim is the failure calibration exists to prevent. The counts appear in the artifact's exclusion ledger. - **It fits per `(segment × evaluator)`, not per segment.** Two evaluators' raw scores live on different scales, since a judge's `0.1`–`0.9` grid is not a human annotator's `1`/`0`, so pooling them would calibrate a mixture of incomparable numbers. Each evaluator's signal is fit separately inside the segment. For a judge that also has to reach a routing decision, the fit is the second of two steps and not the first. A judge's verdicts carry `reported` until the judge has been measured against human labels on the segment, and [Validate a judge and promote it](/docs/judges/validate-and-promote) is the check that records that answer. ## Segments — the Mondrian coordinates A **segment** is the unit a probability is fit for. Its key is four coordinates: ```text { metric, tier, model, endpoint } ``` The coordinate that carries the weight is **model**, and it is the model *served*, not the model *requested*. Routing rewrites the model, which is what routing does, so on a routed row the requested and served models are different by construction. Keying the fit on the request would file the cheap model's observed pass-rate under the frontier model's name: a segment that silently describes the wrong population. The fit reads the served model (`decision.model_rewrite.to`), which is the same coordinate the router later probes, so the router's lookup joins this segment rather than missing it forever behind a fail-open absence. The segment key is computed identically in two places — the fitter and the coverage meter — and a test pins them together, because a calibrator fitting a different population than the coverage meter reports on would be two surfaces telling two truths. See [Quality-aware routing](/docs/calibration/quality-aware-routing) for how the router probes this same key on the hot path. ## The 50/25/25 split and the held-out measurement Within each unit, the observations are ordered *in request time* and split into three contiguous bands: - **Fit (50%)** — the data the probability map is trained on. - **Calibrate (25%)** — the slice the conformal step stands on, which the map has not seen. - **Holdout (25%)** — the untouched final quarter the error is *measured* on. Ordering by time rather than shuffling is a decision, not an accident: a fitter that could re-order its own training set could quietly launder tomorrow's data into today's fit. By the time any data reaches the fitter the split has already happened and is not re-litigable. What gets fit depends on the score's own shape. A segment whose evaluator emits **three or more distinct score values** gets a fitted probability map, conformalized on the calibrate split; its holdout error is a **coverage gap** (`coverage_gap`) — the distance between the interval's nominal coverage and the coverage it actually delivered. A segment whose score takes **fewer than three distinct values** (a rule's pass/fail) has no map to fit at all; it takes a base-rate branch, where the fitted object is the segment's pass rate as a constant forecast and the holdout error is an **ECE** (`ece`) — does the past rate transfer to the future? Either way the measured error is a single non-negative number, and either way it comes off the holdout. ## The B.5 gate Measuring an error is not enough; the engine has to decide whether the error is *small enough* to serve. That is the gate, and it compares the measured error against a threshold that is not a flat constant: ```text threshold = tolerance + allowance ``` - **`tolerance`** is the configured target — `0.02` for a coverage gap, `0.05` for an ECE by default. It is **raise-only**: an operator with a safety-critical workload can demand a *tighter* gate, but a tolerance *looser* than the default is refused at construction, because a knob that could quietly loosen the product's one credibility asset is not a knob worth having. - **`allowance`** is the sampling-noise critical value at *this* holdout's size and structure — the error a perfectly calibrated segment would show anyway, purely because the holdout is finite. At the licensing floor the allowance can be several times the tolerance, so the effective gate runs well above the bare `0.05`. Because the allowance *is* the number on a tiny holdout, the gate refuses to run at all below `MinHoldoutM = 30` rows — beneath that there is no measurement to correct, and a gate that passed a maximally-wrong two-row holdout would license anything. The gate also stores what "passed" could not have told you on its own, and each is a disclosure rather than a gate: - **`min_detectable_error`** — the smallest *true* error this gate catches at least half the time. A gate can control its false-refusal rate exactly and still be unable to see a `0.10` overconfidence at sixty rows; publishing this turns "passed" into "passed, and here is what passing could not rule out." - **`false_refusal_rate` (β)** — the rate at which a segment whose true error sits at the tolerance is refused anyway. - **`null_mean`** — what a perfectly calibrated segment would measure *here*, on average. An error sitting within a hair of it is the signature of a measurement that measured the holdout size rather than the calibration. - **`raw_error`** — the *same* measurement made on the uncalibrated signal, so the report can say what calibration bought: "the judge's own probabilities were off by 0.35; the fitted map is off by 0.02." An artifact that fails its own B.5 measurement can be written to disk as a candidate, but it can **never** be served. The gate is checked before activation, and the refusal carries every measured number — not a bare "gate failed." ## Insufficiency is emitted, not omitted The common case is not a passing gate; it is a segment too thin to make a claim about at all. The differentiator is saying so out loud. Such a segment is written into the artifact as an `insufficient` entry and not dropped, because a segment missing from the artifact is indistinguishable from a segment that had no traffic — carrying a reason, a `need`, a `have`, and a hint. The reasons are a closed set: | Reason | What it means | |---|---| | `below_min_n` | The calibration split (or the holdout) is under its evidence floor. | | `alpha_infeasible` | There is enough data to fit, but not enough to attain the requested confidence at all at this size. | | `no_variation` | Every observation carries the same outcome, so no probability map is identifiable — a constant fit is not a calibration. | | `gate_failed` | The fit was measured and its holdout error did not clear the gate. | The evidence floor is demanding by intent. The default `min_n` of `50` floors the **calibration split**, a quarter of the segment's rows under the 50/25/25 split, so a segment needs roughly **200 fit-eligible observations** before any claim about it is licensable. `insufficient` is meant to be the frequent answer; the coverage meter (`quality coverage`) exists precisely to show you how much of your traffic clears it and what the biggest gaps need.