Calibration & evals vocabularies

Every classification the calibration & evals substrate emits is a closed vocabulary — a member is named, or the value is refused. This page enumerates each enum with its source of truth and links the page that explains it. A refusal is a first-class member here, never an error and never a silent 0.0.

Basis

The epistemic basis of a QualityEstimate. The three never blend in an aggregate.

Value	Meaning
measured	A real evaluator ran and the metric’s ground-truth class licenses measurement (or a judge validated on this segment). A probability.
reported	An unvalidated proxy (raw judge score, heuristic). Displayable and filterable, never a probability — it can never carry a calibrated interval.
insufficient	Not enough evidence. A first-class, routable state carrying a reason + need/have counts. Never a 0.0, never a silent omission.

See Evaluators & metrics for the basis ceiling that caps what each evaluator may claim.

Lifecycle states

The status of a qcal_ calibration artifact (internal/calib). One-way, from left to right.

State	Role	Legal transitions
candidate	Fresh fit, inert (safe to run on a schedule).	→ shadow, active, retired
shadow	Fitted and evaluated but never served — the slot the scheduler populates.	→ active, retired
active	The only status that can license a claim. Activation retires the incumbent atomically.	→ retired
retired	Terminal. A successor is a new content-addressed id, never rewritten bytes.	(none)

Illegal moves (active → shadow, active → active, retired → *) are refused, naming the from-state, the to-state, and the legal moves. See The calibration lifecycle.

Licensing refusal tokens

Returned by calib.Licensed (NotLicensedError.Reason) — the five conjuncts of the §2.5 emission rule, each refused by name. On the routing seam these surface as the quality_absent:<token> score-table term.

Token	The conjunct that failed
no_active_calibration	There is no active quality calibration at all (artifact-level).
artifact_not_active	An artifact was consulted but its status is not active — it may exist as a candidate and may never license a claim.
segment_uncalibrated	The artifact has no fitted entry for this segment.
gate_failed	The entry exists but failed its own B.5 holdout gate — written, but never served.
calibration_stale	The policy epoch has changed, or the fit is older than the 30-day freshness horizon.

A too-thin entry additionally forwards its own insufficiency reason (below) as the token. See Quality-aware routing.

Insufficiency reasons (fit time)

Emitted by the fitter (internal/calib) when a segment is written as an Insufficient entry rather than fitted. Each carries need/have/hint.

Reason	Meaning
below_min_n	The segment’s calibration split is under the min_n floor (default 50).
alpha_infeasible	n clears the floor but is below ⌈(1−α)/α⌉ — the conformal quantile index does not exist at this confidence.
no_variation	Every observation carries the same outcome, so no error can be measured.
gate_failed	The artifact exists but its B.5 holdout measurement did not pass.

See How calibration works.

Eval verdict

The four-state verdict of an eval comparison (internal/evalprog, VerdictOf), carried by EvalComparison.verdict.

Verdict	Meaning
BEAT	The candidate beats the baseline by more than the noise floor — reachable only from a counterfactual instrument (shadow_dual_run; Inv-7).
NOT_BEAT	Measured, and the candidate does not beat the baseline (delta at or below the noise floor).
ON_FRONTIER	On-and-below the frontier. A non-counterfactual instrument (judged_dataset_run) is forced here at best — real evidence about what happened, structurally unable to show a different choice would be better.
CANNOT_MEASURE	Always carries a reason: an arm is missing, the arms span different windows, mixed bases, or insufficient evidence.

Comparison refusal codes

EvalComparison.refusal.code — why a delta could not be computed (the never-subtract rules).

Code	Meaning
comparison_window_blend	The two arms were measured over different windows — two facts, not a delta.
comparison_segment_blend	The arms span different segments.
comparison_basis_blend	The arms carry different bases; bases never blend (QH2).
comparison_insufficient	One or both arms fall below the evidence floor.
comparison_arm_absent	A named arm is not carried by this run.
comparison_no_quality	Neither arm states a quality point to compare.

See Eval programs & verdicts.

Routing score-table terms

Minted by the selector (internal/selector) on each candidate’s decision-row score table. quality_absent and quality_below_min_lcb are different facts with different operator actions.

Term	Meaning
quality_absent:<reason>	Could not measure this tier — the calibrated fact is absent. The <reason> suffix is a licensing token (above). Excluded from candidacy, never scored 0.
quality_below_min_lcb	Measured, and the calibrated lower bound is below the min_quality_lcb floor — admitted-but-not-good-enough.
quality_lcb	The calibrated Clopper–Pearson lower bound a candidate was admitted on.
quality_estimand_mixed:<classes>	The candidate set mixed guarantee classes and was refused (SP-47) — not comparable. The distinct estimand classes ride the term name; the term’s value is their count.
excluded	The shipped exclusion marker carried alongside the term above.

See Quality-aware routing.

Refit triggers

The five named events the refit scheduler fires (internal/calib), each as an edge (once per (artifact, epoch)).

Trigger	Fired when
drift_finding	A registered detector flags drift on a segment.
policy_epoch_change	The policy fingerprint changed.
staleness	The active fit passed the 30-day freshness horizon.
operator_request	An operator asked for a refit.
scheduled_interval	The configured refit_interval cadence elapsed.

An automatic refit lands in shadow (the PromoteShadow default) — it never promotes itself into the serving slot; an operator activates, and that is audited. See Keep calibrations fresh: drift & refit.