Eval programs & verdicts

Calibration is the online half of the substrate — the probability engine the router consults on the hot path. Eval programs are the offline half: a way to ask, of a fixed set of past requests, would a different tier have answered as well, and at what cost? An eval program pins a dataset, re-scores it under one or more candidate arms, folds the results honestly, plots them on a cost/quality frontier, and returns a verdict — one of four frozen states that says, in particular, when it cannot answer. This page explains the pipeline and what licenses each verdict.

Datasets are ledger references

A dataset is not a copy of rows. It is a versioned artifact of ledger references — a filter, a window, and an explicit snapshot of req_ids — so the underlying prompts and responses stay in the decision ledger and are never duplicated into an eval store. The dataset’s version is a content hash over that snapshot, so refitting the same window is reproducible and a rotated-out row is detected rather than silently missing.

Five builders mint a dataset, each a member of a closed vocabulary:

Builder	What it captures
from_filter	Every row matching a filter over a window.
from_failures	The low-scoring outcomes — the rows worth re-examining.
from_annotation_queue	Rows a human queued for labelling.
import	Rows brought in from outside the ledger.
synthetic	Generated rows, for a regression suite.

Fit-eligibility is derived on read, never asserted: a dataset holding any synthetic or imported item is not fit-eligible, because those rows can never calibrate a probability. The census that travels with the dataset always distinguishes production traffic from a sanctioned synthetic population, so a run over generated rows is never byte-indistinguishable from one over real traffic.

Instruments license the verdict

An instrument is how an arm produces its evidence, and it is what decides the ceiling of the verdict that evidence can support. The three are a closed set:

Instrument	What it does	Strongest verdict it can license
judged_dataset_run	Re-scores the stored outputs of the referenced rows. Real, served-model evidence.	ON_FRONTIER
shadow_dual_run	The shadow duplicator re-runs items against a candidate tier live — the only counterfactual. Needs a running data plane and a budget cap.	BEAT
replay	Cost/decision simulation only. Contributes no quality at all.	— (cost half only)

The distinction is structural, not a matter of degree. A judged_dataset_run scores outputs the tier that was actually served produced, so however good those scores are they cannot show that a choice nobody made would have been better — that is the Inv-7 rule, and it caps a non-counterfactual arm at ON_FRONTIER no matter how large the delta. Only a shadow_dual_run, which grades both tiers on the same query, can license a BEAT.

The fold

Every arm’s estimate is built the same way: one atom per scored item, merged through the single aggregator (quality.Merge) — the pipeline never takes a plain average. Three rules arrive with that aggregator and cannot be bypassed:

• Bases never blend. Items are partitioned by the arm’s declared basis; a measured item and a reported item are never pooled. Off-basis items are excluded and counted, not dropped. The honest presentation of mixed evidence is two parallel arms, not one averaged number.
• A floor, named. An arm with fewer than MinArmObservations (8) merged observations becomes insufficient{below_min_n} carrying need/have — never a 0.0, never an omission — while keeping the eval-cost disclosure, because producing a refusal still spent money.
• The empty-but-successful run is unrepresentable. A run that scored nothing on every arm and recorded no refusal would claim to have measured a dataset and found nothing worth saying, which is never what happened; the engine refuses to write it.

The Pareto frontier

The comparison surface plots cost against quality. Only an arm that has both a quality point and a priced dollar appears on the frontier. An arm that is dominated, insufficient, or unpriced is excluded and named beneath the chart — never drawn at the origin, which would read as “free and worthless” when the truth is “not measurable.” Unpriced dollars are excluded from a total and counted beside it, exactly as UNPRICED is handled everywhere else.

The four-state verdict

A comparison of a baseline arm and a candidate arm resolves to one of four frozen, wire-exact states:

Verdict	When it is produced
BEAT	A counterfactual (shadow_dual_run) candidate beat the baseline by more than the noise floor.
NOT_BEAT	A counterfactual arm ran with enough evidence, but the delta did not clear the floor.
ON_FRONTIER	A non-counterfactual instrument — the delta is real but can only sit on or below the frontier (Inv-7).
CANNOT_MEASURE	The refusal. It always carries a reason: a missing arm, different windows, mixed bases, or insufficient evidence.

An eval program is therefore the honest path to a BEAT: judged_dataset_run shows you where a tier sits, and only a shadow_dual_run can show that a cheaper tier would have won — which is exactly what you need before letting the router prefer it.