Eval programs & verdicts

Calibration is the online half: the probability engine the router consults on the hot path. Eval programs are the offline half: a way to ask, of a fixed set of past requests, would a different tier have answered as well, and at what cost? An eval program pins a dataset, re-scores it under one or more candidate arms, folds the results, plots them on a cost/quality frontier, and returns a verdict, one of four frozen states that says, in particular, when it cannot answer. This page explains the pipeline and what licenses each verdict.

Datasets are ledger references

A dataset is not a copy of rows. It is a versioned artifact of ledger references: a filter, a window, and an explicit snapshot of req_ids, so the underlying prompts and responses stay in the decision ledger and are not duplicated into an eval store. The dataset’s version is a content hash over that snapshot, so refitting the same window is reproducible and a rotated-out row is detected rather than silently missing.

Five builders mint a dataset, each a member of a closed vocabulary:

Builder	What it captures
from_filter	Every row matching a filter over a window.
from_failures	The low-scoring outcomes — the rows worth re-examining.
from_annotation_queue	Rows a human queued for labelling.
import	Rows brought in from outside the ledger.
synthetic	Generated rows, for a regression suite.

Fit-eligibility is derived on read rather than asserted: a dataset holding any synthetic or imported item is not fit-eligible, because those rows cannot calibrate a probability. The census that travels with the dataset always distinguishes production traffic from a sanctioned synthetic population, so a run over generated rows is not byte-indistinguishable from one over real traffic.

Instruments license the verdict

An instrument is how an arm produces its evidence, and it is what decides the ceiling of the verdict that evidence can support. The three are a closed set:

Instrument	What it does	Strongest verdict it can license
judged_dataset_run	Re-scores the stored outputs of the referenced rows. Real, served-model evidence.	ON_FRONTIER
shadow_dual_run	The shadow duplicator re-runs items against a candidate tier live — the only counterfactual. Needs a running data plane and a budget cap.	BEAT
replay	Cost/decision simulation only. Contributes no quality at all.	— (cost half only)

The distinction is structural, not a matter of degree. A judged_dataset_run scores outputs the tier that was actually served produced, so however good those scores are they cannot show that a choice nobody made would have been better — that is the Inv-7 rule, and it caps a non-counterfactual arm at ON_FRONTIER no matter how large the delta. Only a shadow_dual_run, which grades both tiers on the same query, can license a BEAT.

The fold

Every arm’s estimate is built the same way: one atom per scored item, merged through the single aggregator (quality.Merge), and the pipeline takes no plain average. Three rules arrive with that aggregator and cannot be bypassed:

• Bases never blend. Items are partitioned by the arm’s declared basis; a measured item and a reported item are not pooled. Off-basis items are excluded and counted, not dropped. Mixed evidence is presented as two parallel arms, not one averaged number.

  Which partition a judge-scored arm lands in is decided elsewhere. A judge’s verdicts carry reported until that judge has been measured against human labels on the segment and promoted, so an arm graded by a judge and an arm graded by a rule sit in different partitions until the promotion lands. How a judge reaches a measured basis is the walk that moves it, and the labels it needs are collected through the annotation queue.

• A floor, named. An arm with fewer than MinArmObservations (8) merged observations becomes insufficient{below_min_n} carrying need/have, rather than a 0.0 or an omission, while keeping the eval-cost disclosure, because turning the arm away still spent money.

• There is no way to write an empty-but-successful run. A run that scored nothing on every arm and recorded nothing turned away would claim to have measured a dataset and found nothing to say, which is not what happened; the engine stops it.

The Pareto frontier

The comparison surface plots cost against quality. Only an arm that has both a quality point and a priced dollar appears on the frontier. An arm that is dominated, insufficient, or unpriced is excluded and named beneath the chart, never drawn at the origin, which would read as “free and worthless” when the truth is “not measurable.” Unpriced dollars are excluded from a total and counted beside it, exactly as UNPRICED is handled everywhere else.

The four-state verdict

A comparison of a baseline arm and a candidate arm resolves to one of four frozen, wire-exact states:

Verdict	When it is produced
BEAT	A counterfactual (shadow_dual_run) candidate beat the baseline by more than the noise floor.
NOT_BEAT	A counterfactual arm ran with enough evidence, but the delta did not clear the floor.
ON_FRONTIER	A non-counterfactual instrument — the delta is real but can only sit on or below the frontier (Inv-7).
CANNOT_MEASURE	The refusal. It always carries a reason: a missing arm, different windows, mixed bases, or insufficient evidence.

An eval program is therefore the path to a BEAT: judged_dataset_run shows you where a tier sits, and only a shadow_dual_run can show that a cheaper tier would have won, which is exactly what you need before letting the router prefer it.