# Calibrate quality from your logs Fit a quality calibration from your decision log, read its reliability report, and activate it for the router. Source: https://tokentriage.com/docs/calibration/calibrate-from-logs --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import Screenshot from '../../../components/Screenshot.astro'; Turn the eval outcomes already sitting in your decision log into a **calibration** — a probability whose error has been *measured* on a held-out slice of the same window — then promote it so the router may read it. This is the quality half of *"cut your LLM bill — after you prove the cheaper path is just as good."* The whole flow is `tokentriage quality`: fit a candidate, read what the fit measured, activate. `tokentriage quality calibrate` (this page — **probability** calibration, stored under `/calibrations/quality/` as `qcal_…` artifacts) is a different command from `tokentriage calibrate` (the **routing-threshold** calibration documented in [Routing](/docs/build/routing#calibrate-from-your-own-logs)). Same lifecycle chassis, different artifact kind — don't conflate them. ## Prerequisites - A joined decision log with realized **eval outcomes** on it — the rows the router produced, each joined to at least one evaluator's score (see [Evaluators & metrics](/docs/calibration/evaluators) for what emits them). - A config file: the fit reads its policy epoch (`evidence.Fingerprint`), its state dir, and the `evals.calibration.quality` knobs from `--config`. - Nothing routes yet — a fit only writes an inert candidate. ## 1. Check coverage before you fit Before fitting, ask what share of your traffic can even *license* a probability claim. The B.9 coverage meter answers exactly that — `licensed rows / applicable rows` per metric — and names the biggest uncovered segments with their need/have gap: ```console $ tokentriage quality coverage --config tokentriage.yaml ``` ```text evidence coverage (B.9) over 2026-06-30T00:00:00Z .. 2026-07-30T00:00:00Z overall 62.4% (1873 licensed / 3002 applicable row(s); 240 demo/synthetic row(s) excluded from BOTH terms) predicate b9_full — which of §2.5's five licensing conjuncts the numerator actually tested schema_validity 88.1% (1401 / 1590; demo excluded 120) groundedness n/a (no applicable rows — an ABSENCE, not 0%) the 1 biggest UNCOVERED segment(s) — a worklist, not a census: groundedness x cheap x gpt-5-mini x chat rows=430 below_min_n: need 200, have 46 (short by 154) eval spend UNPRICED ``` *(Illustrative — your figures depend on your own traffic.)* A `n/a` ratio means the denominator was zero: **"this metric applies to none of your traffic"** is a different fact from "0% is calibrated", and the meter never collapses the two into a `0.0`. ## 2. Fit a candidate Fit every `(segment × evaluator)` unit over a window. The fit uses a time-ordered **50/25/25** split — fit, conformalize, then *measure error on the untouched holdout* — and writes the result as a **candidate**, which is inert: ```console $ tokentriage quality calibrate --config tokentriage.yaml --since 30d --confidence 0.90 ``` ```text fitted 2841 observation(s) from ~/.tokentriage/decisions.jsonl window 2026-06-30T00:11:04Z .. 2026-07-30T00:02:59Z policy epoch 4f2a9c1 alpha 0.1 (nominal confidence 0.9) min_n 50 ledger 3061 record(s), 2903 eval outcome(s), 62 skipped for an ABSENT pass verdict, 0 outside the window reliability report: gate PASS — 3 of 4 measured segment(s) passed … wrote candidate artifact: qcal_20260730T0002_ab12cd34 (status=candidate) activate it with: tokentriage quality activate qcal_20260730T0002_ab12cd34 ``` *(Illustrative output — your window, counts, and artifact id come from your own log.)* The `--confidence` is the nominal coverage of the fitted intervals (it *is* the claim, so it has no silent default — set it, or the config's `evals.calibration.quality.confidence` supplies it). `--fingerprint` defaults to the policy epoch derived from `--config`; the fit refuses without one, because a calibration with no epoch could never be found stale. The per-segment floor `min_n` (default 50) is on the **calibration split**, so a segment needs roughly **200 fit-eligible rows** before any claim is licensable. Thinner segments are emitted as `insufficient` with their `need`/`have` gap — never fitted anyway, never silently dropped. `--min-n` is **raise-only**: it can demand more evidence, never less (a flag that could lower the floor is refused). ## 3. Read the reliability report The candidate carries its own measurement. Print it verbatim — this is a read, so it writes no audit row: ```console $ tokentriage quality reliability qcal_20260730T0002_ab12cd34 --config tokentriage.yaml ``` Each measured segment prints `PASS`/`FAIL` against a threshold that is `tolerance + allowance` — the tolerance you configured plus the sampling-noise critical value at *this* holdout size — alongside `min_detectable_error` (the smallest true error the gate could catch) and the uncalibrated error (what calibration bought). A segment whose measured error exceeds its threshold is a `FAIL`. An artifact that fails its own holdout measurement **can be written but can never be served**. The candidate is kept so the measurement that refused it stays readable — the refusal is the product, not an error to hide. ## 4. Activate Promote the candidate to `active`. This retires the previous active artifact atomically, and is audited: ```console $ tokentriage quality activate qcal_20260730T0002_ab12cd34 --config tokentriage.yaml ``` Activation is **per segment** (SP-43): an artifact activates if at least one measured segment passed its own gate, and any failing segments serve *nothing* while the healthy ones serve. A wholly gate-failed artifact is **refused**, with its reliability report attached. You can watch the lifecycle board and the gated-activation surface in the studio: **Activation does not move live routing until the next config reload.** The CLI repoints the manifest and the studio confirms it, but the running data plane keeps its current snapshot until you reload — both surfaces disclose this. Every mutation here — the fit, the activation, **and their refusals** — writes exactly one audit row, so a refused activation is as visible as a successful one.