# Prove a cheaper tier with an eval experiment

Run an offline eval over your ledger, place the candidates on a cost/quality frontier, and read the verdict before you route.

Source: https://tokentriage.com/docs/calibration/prove-a-cheaper-tier

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';

Before you route traffic to a cheaper tier, prove it answers as well. An **eval
experiment** re-scores the outputs your rows already produced, folds each arm's quality
and dollar, and returns a cost/quality frontier plus a four-state verdict with its
reason. The whole flow is `tokentriage evals`: pick a dataset, run the arms, compare.

## Prerequisites

- Response bodies captured for the rows you want to score (step 1).
- At least one **dataset** — a versioned set of ledger references (step 2).
- A cheap tier and a stronger tier already serving traffic, so both have rows to score.

## 1. Turn on capture

A `judged_dataset_run` re-scores **stored outputs**, so it needs the response bodies. A
default deployment captures hash-only, so the run finds no bodies and returns a *counted
refusal* — a documented default state, not a crash:

```yaml
dashboard:
  enabled: true
  capture:
    request: true
    response: true            # judged_dataset_run needs the stored body
    ttl: 720h
    path: ~/.tokentriage/content
```

<Callout type="note">
Without captured bodies you'll see the refusal spelled out, naming its floor and
arithmetic — never an empty-but-successful run:

```text
capture: NONE of the 412 member(s) carries a captured body. A `judged_dataset_run`
over this dataset is a COUNTED REFUSAL (captured_responses: need 412, have 0) rather
than an empty-but-successful run — turn on dashboard.capture to score it.
```

</Callout>

## 2. Choose a dataset

A dataset is a versioned set of **ledger references** — a filter, a window and an
explicit `req_id` snapshot — never a copy of your rows. List what exists, then inspect
one, re-resolving its snapshot against the live ledger:

```console
$ tokentriage evals datasets
$ tokentriage evals datasets ds_checkout --resolve --config tokentriage.yaml
```

`--resolve` reports members retention has rotated out (the snapshot *shrank*), and the
detail view prints **fit-eligibility**: a dataset holding synthetic or imported rows is
fit-ineligible and says so, because those rows are excluded from every calibration fit.
Create datasets over the management API (`POST /tt/api/v1/evals/datasets`) — the CLI
reads and runs them.

## 3. Run the arms

Score each tier as an arm. An arm is `key=value,…`: a name, a registered evaluator, a
registered metric, and optionally the tier it scopes. Write the run record out so you can
read the comparison later without re-executing:

```console
$ tokentriage evals run ds_checkout \
    --arm name=frontier,evaluator=code-output-nonempty,metric=output_nonempty,tier=frontier \
    --arm name=cheap,evaluator=code-output-nonempty,metric=output_nonempty,tier=cheap \
    --out run.json --config tokentriage.yaml
```

<Callout type="warning">
The CLI **refuses external evaluators by name** — a `judge-*` or `sidecar` arm needs
`external: allow` plus a key env-ref (the air-gap gate), which this command has no config
surface for. It also refuses a `shadow_dual_run` arm: that instrument re-issues requests
*live* under a budget cap and must be driven from
`POST /tt/api/v1/evals/experiments/{id}/run`. Arms below the `MinArmObservations` floor
report an `insufficient` estimate with need/have — never a fabricated `0.00`.
</Callout>

## 4. Compare

Project the same execution onto the comparison surface — the arms table, the Pareto
frontier, and the verdict:

```console
$ tokentriage evals compare ds_checkout --run run.json \
    --baseline frontier --candidate cheap --config tokentriage.yaml
```

```text
  ARM              TIER   BASIS        N     SUBJECT $   EVAL $      QUALITY
  frontier         frontier measured  384   $2.140000   $0.000000   0.9740
  cheap            cheap    measured  384   $0.310000   $0.000000   0.9610

  pareto (cheapest first — 2 point(s)):
    * cheap             cost $0.310000 quality 0.9610 (measured, n=384)
    * frontier          cost $2.140000 quality 0.9740 (measured, n=384)

  verdict     ON_FRONTIER
              both arms sit on the non-dominated frontier; a judged_dataset_run
              re-scores served outputs and cannot show a different choice would beat.
```

*(Illustrative — your figures depend on your own traffic.)* Only arms with **both** a
point and a priced dollar are plotted; insufficient or unpriceable arms are
excluded-and-named beneath the chart, never dropped at the origin. Add `--json` to print
exactly the bytes `GET /tt/api/v1/evals/experiments/{id}` serves.

## 5. Read the four-state verdict

The verdict is one of **`BEAT` / `NOT_BEAT` / `ON_FRONTIER` / `CANNOT_MEASURE`**, and
`CANNOT_MEASURE` always carries a reason (arms missing, different windows, mixed bases,
or insufficient evidence).

<Callout type="key">
**A `judged_dataset_run` can only reach `ON_FRONTIER`, never `BEAT`.** Re-scoring the
outputs the served tier actually produced is real evidence about what happened, but it
structurally cannot show that a *different* choice would have been better. Only a
**`shadow_dual_run`** — which re-runs items against the candidate tier live — is a
counterfactual, and it is the only instrument whose verdict may be `BEAT`. Drive it from
the API. This is the same four-state vocabulary as the routing verdict; see
[the two verdict vocabularies](/docs/concepts/honesty-model#two-verdict-vocabularies--dont-conflate-them)
(and note the *savings* verdict — PROVEN/ESTIMATED/UNPROVABLE — is a different enum).
</Callout>

<NextSteps items={[
  { title: 'Eval programs & verdicts', href: '/docs/calibration/eval-programs', description: 'Datasets, instruments, the fold, the Pareto frontier, and why each instrument licenses a different top verdict.' },
  { title: 'Gate routing on a proven floor', href: '/docs/calibration/gate-routing-on-quality', description: 'Once a tier is proven, make it eligible only while its calibrated lower bound clears a floor.' },
  { title: 'The honesty verdict for routing', href: '/docs/build/routing#the-honesty-verdict', description: 'How the same four-state vocabulary decides whether the router beats a zero-router baseline.' },
]} />