# Eval programs & verdicts

The offline experiment half, from ledger-reference datasets and the instruments that license a verdict to the basis-partitioned fold and the four-state verdict.

Source: https://tokentriage.com/docs/calibration/eval-programs

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import EvalProgramDiagram from '../../../components/EvalProgramDiagram.astro';
import RoutingVerdictDiagram from '../../../components/RoutingVerdictDiagram.astro';

Calibration is the *online* half: the probability engine the router consults on
the hot path. **Eval programs** are the *offline* half: a way to ask, of a fixed set of past
requests, *would a different tier have answered as well, and at what cost?* An eval program pins
a dataset, re-scores it under one or more candidate arms, folds the results, plots them
on a cost/quality frontier, and returns a **verdict**, one of four frozen states that says, in
particular, when it *cannot* answer. This page explains the pipeline and what licenses each
verdict.

<EvalProgramDiagram />

## Datasets are ledger references

A dataset is **not a copy of rows**. It is a versioned artifact of *ledger references*: a
filter, a window, and an explicit snapshot of `req_id`s, so the underlying prompts and
responses stay in the [decision ledger](/docs/concepts/core-concepts#the-decision-ledger) and
are not duplicated into an eval store. The dataset's version is a content hash over that
snapshot, so refitting the same window is reproducible and a rotated-out row is *detected*
rather than silently missing.

Five builders mint a dataset, each a member of a closed vocabulary:

| Builder | What it captures |
|---|---|
| `from_filter` | Every row matching a filter over a window. |
| `from_failures` | The low-scoring outcomes — the rows worth re-examining. |
| `from_annotation_queue` | Rows a human queued for labelling. |
| `import` | Rows brought in from outside the ledger. |
| `synthetic` | Generated rows, for a regression suite. |

**Fit-eligibility is derived on read** rather than asserted: a dataset holding any synthetic or
imported item is *not* fit-eligible, because those rows cannot calibrate a probability. The
census that travels with the dataset always distinguishes production traffic from a sanctioned
synthetic population, so a run over generated rows is not byte-indistinguishable from one over
real traffic.

## Instruments license the verdict

An **instrument** is how an arm produces its evidence, and it is what decides the *ceiling* of
the verdict that evidence can support. The three are a closed set:

| Instrument | What it does | Strongest verdict it can license |
|---|---|---|
| `judged_dataset_run` | Re-scores the **stored outputs** of the referenced rows. Real, served-model evidence. | `ON_FRONTIER` |
| `shadow_dual_run` | The shadow duplicator re-runs items against a candidate tier **live** — the only counterfactual. Needs a running data plane and a budget cap. | `BEAT` |
| `replay` | Cost/decision **simulation only**. Contributes no quality at all. | — (cost half only) |

The distinction is structural, not a matter of degree. A `judged_dataset_run` scores outputs the
tier that was *actually served* produced, so however good those scores are they cannot show that
a choice nobody made would have been better — that is the **Inv-7** rule, and it caps a
non-counterfactual arm at `ON_FRONTIER` no matter how large the delta. Only a `shadow_dual_run`,
which grades *both* tiers on the same query, can license a `BEAT`.

<Callout type="warning">
Both run instruments need **captured bodies**, and capture defaults to hash-only. A
`judged_dataset_run` needs the captured *response*; a `shadow_dual_run` needs the captured
*request*. On a default deployment neither has bodies, so a run scores nothing and returns a
**counted refusal** naming the exact config leaf to enable, rather than an empty-but-successful run.
The two are opposite facts and the engine refuses to render them the same way.
</Callout>

## The fold

Every arm's estimate is built the same way: one atom per scored item, merged through the single
aggregator (`quality.Merge`), and the pipeline takes no plain average. Three rules arrive with
that aggregator and cannot be bypassed:

- **Bases never blend.** Items are partitioned by the arm's declared basis; a `measured` item and
  a `reported` item are not pooled. Off-basis items are *excluded and counted*, not dropped.
  Mixed evidence is presented as two parallel arms, not one averaged number.

  Which partition a judge-scored arm lands in is decided elsewhere. A judge's verdicts carry
  `reported` until that judge has been measured against human labels on the segment and promoted,
  so an arm graded by a judge and an arm graded by a rule sit in different partitions until the
  promotion lands. [How a judge reaches a measured basis](/docs/judges/agreement) is the walk that
  moves it, and the labels it needs are
  [collected through the annotation queue](/docs/judges/collect-labels).
- **A floor, named.** An arm with fewer than **`MinArmObservations` (8)** merged observations
  becomes `insufficient{below_min_n}` carrying `need`/`have`, rather than a `0.0` or an omission,
  while *keeping* the eval-cost disclosure, because turning the arm away still spent money.
- **There is no way to write an empty-but-successful run.** A run that scored nothing on every arm
  and recorded nothing turned away would claim to have measured a dataset and found nothing to say,
  which is not what happened; the engine stops it.

## The Pareto frontier

The comparison surface plots cost against quality. Only an arm that has **both a quality point
and a priced dollar** appears on the frontier. An arm that is dominated, `insufficient`, or
unpriced is **excluded and named beneath the chart**, never drawn at the origin, which would
read as "free and worthless" when the truth is "not measurable." Unpriced dollars are excluded
from a total and counted beside it, exactly as [UNPRICED](/docs/concepts/honesty-model#provenance--unpriced)
is handled everywhere else.

## The four-state verdict

A comparison of a baseline arm and a candidate arm resolves to one of four frozen, wire-exact
states:

<RoutingVerdictDiagram />

| Verdict | When it is produced |
|---|---|
| `BEAT` | A counterfactual (`shadow_dual_run`) candidate beat the baseline by more than the noise floor. |
| `NOT_BEAT` | A counterfactual arm ran with enough evidence, but the delta did not clear the floor. |
| `ON_FRONTIER` | A non-counterfactual instrument — the delta is real but can only sit on or below the frontier (Inv-7). |
| `CANNOT_MEASURE` | The refusal. It **always** carries a reason: a missing arm, different windows, mixed bases, or insufficient evidence. |

<Callout type="key">
**This is the same closed vocabulary as the routing verdict** — `BEAT` · `NOT_BEAT` ·
`ON_FRONTIER` · `CANNOT_MEASURE` — because it answers the same question (*did a candidate beat a
baseline at its own cost?*) about an offline experiment rather than live traffic. See the
[two verdict vocabularies](/docs/concepts/honesty-model#two-verdict-vocabularies--dont-conflate-them)
note. It is a **different** enum from the *savings* verdict (`PROVEN` · `ESTIMATED` ·
`UNPROVABLE`), which judges a dollar figure, not a quality comparison. When you cite a verdict,
name which vocabulary it belongs to, because there is no `BEAT` dollar and no `PROVEN` routing outcome.
</Callout>

An eval program is therefore the path to a `BEAT`: `judged_dataset_run` shows you where a
tier *sits*, and only a `shadow_dual_run` can show that a cheaper tier would have *won*, which
is exactly what you need before letting the router prefer it.

<NextSteps items={[
  { title: 'Prove a cheaper tier with an eval experiment', href: '/docs/calibration/prove-a-cheaper-tier', description: 'Run baseline vs candidate arms, read the Pareto frontier, and understand why a BEAT needs a shadow dual-run.' },
  { title: 'Quality-aware routing', href: '/docs/calibration/quality-aware-routing', description: 'How a proven calibration feeds the router — the online half these offline verdicts inform.' },
  { title: 'The honesty model', href: '/docs/concepts/honesty-model#two-verdict-vocabularies--dont-conflate-them', description: 'The two verdict vocabularies, and why the eval verdict must never be confused with the savings verdict.' }
]} />