# Calibration & evals vocabularies Every closed enum in the calibration and evals surface, one table each, cross-linked to the page that explains it. Source: https://tokentriage.com/docs/reference/quality-vocabularies --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; Every classification the calibration & evals surface emits is a **closed vocabulary** — a member is named, or the value is refused. This page enumerates each enum with its source of truth and links the page that explains it. A refusal is a first-class member here — not an error, and not a silent `0.0`. ## Basis What kind of claim a [`QualityEstimate`](/docs/reference/quality-api#load-bearing-schemas) is making. The three never blend in an aggregate. | Value | Meaning | |---|---| | `measured` | A real evaluator ran and the metric's ground-truth class licenses measurement (or a judge validated on this segment). A probability. | | `reported` | An unvalidated proxy (raw judge score, heuristic). Displayable and filterable, **never** a probability — it can never carry a calibrated interval. | | `insufficient` | Not enough evidence. A first-class, routable state carrying a `reason` + `need`/`have` counts. Never a `0.0`, never a silent omission. | See [Evaluators & metrics](/docs/calibration/evaluators) for the basis ceiling that caps what each evaluator may claim. ## Lifecycle states The status of a `qcal_` calibration artifact (`internal/calib`). One-way, from left to right. | State | Role | Legal transitions | |---|---|---| | `candidate` | Fresh fit, inert (safe to run on a schedule). | → `shadow`, `active`, `retired` | | `shadow` | Fitted and evaluated but **never served** — the slot the scheduler populates. | → `active`, `retired` | | `active` | The **only** status that can license a claim. Activation retires the incumbent atomically. | → `retired` | | `retired` | Terminal. A successor is a new content-addressed id, never rewritten bytes. | *(none)* | Illegal moves (`active → shadow`, `active → active`, `retired → *`) are refused, naming the from-state, the to-state, and the legal moves. See [The calibration lifecycle](/docs/calibration/lifecycle). ## Licensing refusal tokens Returned by `calib.Licensed` (`NotLicensedError.Reason`) — the five conjuncts of the §2.5 emission rule, each refused by name. On the routing seam these surface as the `quality_absent:` score-table term. | Token | The conjunct that failed | |---|---| | `no_active_calibration` | There is no active quality calibration at all (artifact-level). | | `artifact_not_active` | An artifact was consulted but its status is not `active` — it may exist as a candidate and may never license a claim. | | `segment_uncalibrated` | The artifact has no fitted entry for this segment. | | `gate_failed` | The entry exists but failed its own B.5 holdout gate — written, but never served. | | `calibration_stale` | The policy epoch has changed, **or** the fit is older than the 30-day freshness horizon. | A too-thin entry additionally forwards its own **insufficiency reason** (below) as the token. See [Quality-aware routing](/docs/calibration/quality-aware-routing). ## Insufficiency reasons (fit time) Emitted by the fitter (`internal/calib`) when a segment is written as an `Insufficient` entry rather than fitted. Each carries `need`/`have`/`hint`. | Reason | Meaning | |---|---| | `below_min_n` | The segment's calibration split is under the `min_n` floor (default `50`). | | `alpha_infeasible` | `n` clears the floor but is below `⌈(1−α)/α⌉` — the conformal quantile index does not exist at this confidence. | | `no_variation` | Every observation carries the same outcome, so no error can be measured. | | `gate_failed` | The artifact exists but its B.5 holdout measurement did not pass. | See [How calibration works](/docs/calibration/how-calibration-works#insufficiency-is-emitted-not-omitted). **The API-facing insufficiency vocabulary is broader.** `QualityEstimate.insufficient.reason` on the wire (§2.1, `internal/quality`) draws from `below_min_n`, `segment_uncalibrated`, `calibration_stale`, `drift_invalidated`, `no_evaluator`, and `basis_ceiling_reported` — the fit-time set above plus the read-time states (a stale or drift-invalidated segment flips to an insufficient refusal at read time, never rewriting the artifact). A consumer reading the wire switches on this larger set. ## Eval verdict The four-state verdict of an eval comparison (`internal/evalprog`, `VerdictOf`), carried by `EvalComparison.verdict`. | Verdict | Meaning | |---|---| | `BEAT` | The candidate beats the baseline by more than the noise floor — reachable **only** from a counterfactual instrument (`shadow_dual_run`; Inv-7). | | `NOT_BEAT` | Measured, and the candidate does not beat the baseline (delta at or below the noise floor). | | `ON_FRONTIER` | On-and-below the frontier. A non-counterfactual instrument (`judged_dataset_run`) is forced here at best — real evidence about what happened, structurally unable to show a different choice would be better. | | `CANNOT_MEASURE` | Always carries a reason: an arm is missing, the arms span different windows, mixed bases, or insufficient evidence. | **This is the same four-state enum as the routing verdict.** When you see it, link [the two verdict vocabularies](/docs/concepts/honesty-model#two-verdict-vocabularies--dont-conflate-them): the eval/routing verdict (`BEAT`/`NOT_BEAT`/`ON_FRONTIER`/`CANNOT_MEASURE`) is a **different** enum from the **savings** verdict (`PROVEN`/`ESTIMATED`/`UNPROVABLE`). Do not conflate them. ## Comparison refusal codes `EvalComparison.refusal.code` — why a delta could not be computed (the never-subtract rules). | Code | Meaning | |---|---| | `comparison_window_blend` | The two arms were measured over different windows — two facts, not a delta. | | `comparison_segment_blend` | The arms span different segments. | | `comparison_basis_blend` | The arms carry different bases; bases never blend (QH2). | | `comparison_insufficient` | One or both arms fall below the evidence floor. | | `comparison_arm_absent` | A named arm is not carried by this run. | | `comparison_no_quality` | Neither arm states a quality point to compare. | See [Eval programs & verdicts](/docs/calibration/eval-programs). ## Pairwise swap disclosure A pairwise comparison runs both presentation orderings on every item. A pair whose two orderings name different winners is **judge-uncertain**: it is counted, and excluded from the arm's estimate rather than averaged in. `EvalArmResult.pairwise` carries the block (`internal/api/contract/evals.go:290-303`). | Field | Meaning | |---|---| | `pairs` | Items judged in **both** orderings. Always emitted. | | `flipped` | Pairs whose two orderings disagreed — the count of judge-uncertain items. A flipped pair carries no winner at all (`internal/evalprog/experiments.go:1041-1046`). | | `flip_rate` | `flipped / pairs`, or **null** when `pairs` is `0`. Never `0.0` for "no pairs were run": a perfect flip rate and an unrun swap set are opposite facts. | | `swap_gate_passed` | The B.7 swap gate for this comparison: at least `evalprog.MinSwapPairs` (30) pairs **and** a flip rate at or below `evalprog.MaxPairwiseFlipRate` (0.10). `false` is the safe default, and zero pairs returns it. | The mark also travels as counted run refusals, so the flip rate survives into the frozen run record and therefore into the run history (`internal/evalprog/experiments.go:1164-1196`). | Run refusal code | Fired when | |---|---| | `run_pairwise_flipped` | At least one pair flipped. The detail string names those pairs `JUDGE-UNCERTAIN` verbatim, and says they are counted here and excluded from the arm's estimate. | | `run_pairwise_swap_set_short` | Fewer than 30 pairs were judged in both orderings, so the swap gate cannot be evaluated and therefore cannot pass. | | `run_pairwise_swap_gate` | 30 or more pairs, and the measured flip rate exceeds 0.10. The pairwise judge is position-biased on this comparison. | The judge promotion path applies the same two bars under different constant names, `SwapSetMinPairs` and `SwapFlipCeiling`. Those, the abstention reasons, the validation codes and every other closed set the judge surfaces print are in [Judge vocabularies](/docs/reference/judge-vocabularies). ## Routing score-table terms Minted by the selector (`internal/selector`) on each candidate's decision-row score table. `quality_absent` and `quality_below_min_lcb` are **different facts with different operator actions**. | Term | Meaning | |---|---| | `quality_absent:` | Could **not** measure this tier — the calibrated fact is absent. The `` suffix is a licensing token (above). Excluded from candidacy, never scored `0`. | | `quality_below_min_lcb` | Measured, and the calibrated lower bound is **below** the `min_quality_lcb` floor — admitted-but-not-good-enough. | | `quality_lcb` | The calibrated Clopper–Pearson lower bound a candidate was admitted on. | | `quality_estimand_mixed:` | The candidate set mixed guarantee classes and was refused (SP-47) — not comparable. The distinct estimand classes ride the term name; the term's **value** is their count. | | `excluded` | The shipped exclusion marker carried alongside the term above. | See [Quality-aware routing](/docs/calibration/quality-aware-routing). ## Refit triggers The five named events the refit scheduler fires (`internal/calib`), each as an edge (once per `(artifact, epoch)`). | Trigger | Fired when | |---|---| | `drift_finding` | A registered detector flags drift on a segment. | | `policy_epoch_change` | The policy fingerprint changed. | | `staleness` | The active fit passed the 30-day freshness horizon. | | `operator_request` | An operator asked for a refit. | | `scheduled_interval` | The configured `refit_interval` cadence elapsed. | An automatic refit lands in `shadow` (the `PromoteShadow` default) — it never promotes itself into the serving slot; an operator activates, and that is audited. See [Keep calibrations fresh: drift & refit](/docs/calibration/drift-and-refit).