Calibrate quality from your logs

Turn the eval outcomes already sitting in your decision log into a calibration — a probability whose error has been measured on a held-out slice of the same window — then promote it so the router may read it. This is the quality half of “cut your LLM bill — after you prove the cheaper path is just as good.” The whole flow is tokentriage quality: fit a candidate, read what the fit measured, activate.

Prerequisites

• A joined decision log with realized eval outcomes on it — the rows the router produced, each joined to at least one evaluator’s score (see Evaluators & metrics for what emits them).
• A config file: the fit reads its policy epoch (evidence.Fingerprint), its state dir, and the evals.calibration.quality knobs from --config.
• Nothing routes yet — a fit only writes an inert candidate.

1. Check coverage before you fit

Before fitting, ask what share of your traffic can even license a probability claim. The B.9 coverage meter answers exactly that — licensed rows / applicable rows per metric — and names the biggest uncovered segments with their need/have gap:

$ tokentriage quality coverage --config tokentriage.yaml

evidence coverage (B.9) over 2026-06-30T00:00:00Z .. 2026-07-30T00:00:00Z
  overall     62.4%  (1873 licensed / 3002 applicable row(s); 240 demo/synthetic row(s) excluded from BOTH terms)
  predicate   b9_full — which of §2.5's five licensing conjuncts the numerator actually tested

  schema_validity          88.1%  (1401 / 1590; demo excluded 120)
  groundedness             n/a (no applicable rows — an ABSENCE, not 0%)
      the 1 biggest UNCOVERED segment(s) — a worklist, not a census:
        groundedness x cheap x gpt-5-mini x chat
          rows=430  below_min_n: need 200, have 46 (short by 154)  eval spend UNPRICED

(Illustrative — your figures depend on your own traffic.) A n/a ratio means the denominator was zero: “this metric applies to none of your traffic” is a different fact from “0% is calibrated”, and the meter never collapses the two into a 0.0.

2. Fit a candidate

Fit every (segment × evaluator) unit over a window. The fit uses a time-ordered 50/25/25 split — fit, conformalize, then measure error on the untouched holdout — and writes the result as a candidate, which is inert:

$ tokentriage quality calibrate --config tokentriage.yaml --since 30d --confidence 0.90

fitted 2841 observation(s) from ~/.tokentriage/decisions.jsonl
  window      2026-06-30T00:11:04Z .. 2026-07-30T00:02:59Z
  policy epoch 4f2a9c1
  alpha 0.1 (nominal confidence 0.9)  min_n 50
  ledger      3061 record(s), 2903 eval outcome(s), 62 skipped for an ABSENT pass verdict, 0 outside the window

reliability report: gate PASS — 3 of 4 measured segment(s) passed
  …
wrote candidate artifact: qcal_20260730T0002_ab12cd34 (status=candidate)
activate it with: tokentriage quality activate qcal_20260730T0002_ab12cd34

(Illustrative output — your window, counts, and artifact id come from your own log.)

The --confidence is the nominal coverage of the fitted intervals (it is the claim, so it has no silent default — set it, or the config’s evals.calibration.quality.confidence supplies it). --fingerprint defaults to the policy epoch derived from --config; the fit refuses without one, because a calibration with no epoch could never be found stale.

3. Read the reliability report

The candidate carries its own measurement. Print it verbatim — this is a read, so it writes no audit row:

$ tokentriage quality reliability qcal_20260730T0002_ab12cd34 --config tokentriage.yaml

Each measured segment prints PASS/FAIL against a threshold that is tolerance + allowance — the tolerance you configured plus the sampling-noise critical value at this holdout size — alongside min_detectable_error (the smallest true error the gate could catch) and the uncalibrated error (what calibration bought). A segment whose measured error exceeds its threshold is a FAIL.

4. Activate

Promote the candidate to active. This retires the previous active artifact atomically, and is audited:

$ tokentriage quality activate qcal_20260730T0002_ab12cd34 --config tokentriage.yaml

Activation is per segment (SP-43): an artifact activates if at least one measured segment passed its own gate, and any failing segments serve nothing while the healthy ones serve. A wholly gate-failed artifact is refused, with its reliability report attached. You can watch the lifecycle board and the gated-activation surface in the studio:

localhost:9090/ui/calibrationDemo data