Calibration & evals vocabularies

Every classification the calibration & evals surface emits is a closed vocabulary — a member is named, or the value is refused. This page enumerates each enum with its source of truth and links the page that explains it. A refusal is a first-class member here — not an error, and not a silent 0.0.

Basis

What kind of claim a QualityEstimate is making. The three never blend in an aggregate.

Value	Meaning
measured	A real evaluator ran and the metric’s ground-truth class licenses measurement (or a judge validated on this segment). A probability.
reported	An unvalidated proxy (raw judge score, heuristic). Displayable and filterable, never a probability — it can never carry a calibrated interval.
insufficient	Not enough evidence. A first-class, routable state carrying a reason + need/have counts. Never a 0.0, never a silent omission.

See Evaluators & metrics for the basis ceiling that caps what each evaluator may claim.

Lifecycle states

The status of a qcal_ calibration artifact (internal/calib). One-way, from left to right.

State	Role	Legal transitions
candidate	Fresh fit, inert (safe to run on a schedule).	→ shadow, active, retired
shadow	Fitted and evaluated but never served — the slot the scheduler populates.	→ active, retired
active	The only status that can license a claim. Activation retires the incumbent atomically.	→ retired
retired	Terminal. A successor is a new content-addressed id, never rewritten bytes.	(none)

Illegal moves (active → shadow, active → active, retired → *) are refused, naming the from-state, the to-state, and the legal moves. See The calibration lifecycle.

Licensing refusal tokens

Returned by calib.Licensed (NotLicensedError.Reason) — the five conjuncts of the §2.5 emission rule, each refused by name. On the routing seam these surface as the quality_absent:<token> score-table term.

Token	The conjunct that failed
no_active_calibration	There is no active quality calibration at all (artifact-level).
artifact_not_active	An artifact was consulted but its status is not active — it may exist as a candidate and may never license a claim.
segment_uncalibrated	The artifact has no fitted entry for this segment.
gate_failed	The entry exists but failed its own B.5 holdout gate — written, but never served.
calibration_stale	The policy epoch has changed, or the fit is older than the 30-day freshness horizon.

A too-thin entry additionally forwards its own insufficiency reason (below) as the token. See Quality-aware routing.

Insufficiency reasons (fit time)

Emitted by the fitter (internal/calib) when a segment is written as an Insufficient entry rather than fitted. Each carries need/have/hint.

Reason	Meaning
below_min_n	The segment’s calibration split is under the min_n floor (default 50).
alpha_infeasible	n clears the floor but is below ⌈(1−α)/α⌉ — the conformal quantile index does not exist at this confidence.
no_variation	Every observation carries the same outcome, so no error can be measured.
gate_failed	The artifact exists but its B.5 holdout measurement did not pass.

See How calibration works.

Eval verdict

The four-state verdict of an eval comparison (internal/evalprog, VerdictOf), carried by EvalComparison.verdict.

Verdict	Meaning
BEAT	The candidate beats the baseline by more than the noise floor — reachable only from a counterfactual instrument (shadow_dual_run; Inv-7).
NOT_BEAT	Measured, and the candidate does not beat the baseline (delta at or below the noise floor).
ON_FRONTIER	On-and-below the frontier. A non-counterfactual instrument (judged_dataset_run) is forced here at best — real evidence about what happened, structurally unable to show a different choice would be better.
CANNOT_MEASURE	Always carries a reason: an arm is missing, the arms span different windows, mixed bases, or insufficient evidence.

Comparison refusal codes

EvalComparison.refusal.code — why a delta could not be computed (the never-subtract rules).

Code	Meaning
comparison_window_blend	The two arms were measured over different windows — two facts, not a delta.
comparison_segment_blend	The arms span different segments.
comparison_basis_blend	The arms carry different bases; bases never blend (QH2).
comparison_insufficient	One or both arms fall below the evidence floor.
comparison_arm_absent	A named arm is not carried by this run.
comparison_no_quality	Neither arm states a quality point to compare.

See Eval programs & verdicts.

Pairwise swap disclosure

A pairwise comparison runs both presentation orderings on every item. A pair whose two orderings name different winners is judge-uncertain: it is counted, and excluded from the arm’s estimate rather than averaged in. EvalArmResult.pairwise carries the block (internal/api/contract/evals.go:290-303).

Field	Meaning
pairs	Items judged in both orderings. Always emitted.
flipped	Pairs whose two orderings disagreed — the count of judge-uncertain items. A flipped pair carries no winner at all (internal/evalprog/experiments.go:1041-1046).
flip_rate	flipped / pairs, or null when pairs is 0. Never 0.0 for “no pairs were run”: a perfect flip rate and an unrun swap set are opposite facts.
swap_gate_passed	The B.7 swap gate for this comparison: at least evalprog.MinSwapPairs (30) pairs and a flip rate at or below evalprog.MaxPairwiseFlipRate (0.10). false is the safe default, and zero pairs returns it.

The mark also travels as counted run refusals, so the flip rate survives into the frozen run record and therefore into the run history (internal/evalprog/experiments.go:1164-1196).

Run refusal code	Fired when
run_pairwise_flipped	At least one pair flipped. The detail string names those pairs JUDGE-UNCERTAIN verbatim, and says they are counted here and excluded from the arm’s estimate.
run_pairwise_swap_set_short	Fewer than 30 pairs were judged in both orderings, so the swap gate cannot be evaluated and therefore cannot pass.
run_pairwise_swap_gate	30 or more pairs, and the measured flip rate exceeds 0.10. The pairwise judge is position-biased on this comparison.

The judge promotion path applies the same two bars under different constant names, SwapSetMinPairs and SwapFlipCeiling. Those, the abstention reasons, the validation codes and every other closed set the judge surfaces print are in Judge vocabularies.

Routing score-table terms

Minted by the selector (internal/selector) on each candidate’s decision-row score table. quality_absent and quality_below_min_lcb are different facts with different operator actions.

Term	Meaning
quality_absent:<reason>	Could not measure this tier — the calibrated fact is absent. The <reason> suffix is a licensing token (above). Excluded from candidacy, never scored 0.
quality_below_min_lcb	Measured, and the calibrated lower bound is below the min_quality_lcb floor — admitted-but-not-good-enough.
quality_lcb	The calibrated Clopper–Pearson lower bound a candidate was admitted on.
quality_estimand_mixed:<classes>	The candidate set mixed guarantee classes and was refused (SP-47) — not comparable. The distinct estimand classes ride the term name; the term’s value is their count.
excluded	The shipped exclusion marker carried alongside the term above.

See Quality-aware routing.

Refit triggers

The five named events the refit scheduler fires (internal/calib), each as an edge (once per (artifact, epoch)).

Trigger	Fired when
drift_finding	A registered detector flags drift on a segment.
policy_epoch_change	The policy fingerprint changed.
staleness	The active fit passed the 30-day freshness horizon.
operator_request	An operator asked for a refit.
scheduled_interval	The configured refit_interval cadence elapsed.

An automatic refit lands in shadow (the PromoteShadow default) — it never promotes itself into the serving slot; an operator activates, and that is audited. See Keep calibrations fresh: drift & refit.