Prove a cheaper tier with an eval experiment

Before you route traffic to a cheaper tier, prove it answers as well. An eval experiment re-scores the outputs your rows already produced, folds each arm’s quality and dollar, and returns a cost/quality frontier plus a four-state verdict with its reason. The whole flow is tokentriage evals: pick a dataset, run the arms, compare.

Prerequisites

• Response bodies captured for the rows you want to score (step 1).
• At least one dataset — a versioned set of ledger references (step 2).
• A cheap tier and a stronger tier already serving traffic, so both have rows to score.

1. Turn on capture

A judged_dataset_run re-scores stored outputs, so it needs the response bodies. A default deployment captures hash-only, so the run finds no bodies and returns a counted refusal — a documented default state, not a crash:

dashboard:
  enabled: true
  capture:
    request: true
    response: true            # judged_dataset_run needs the stored body
    ttl: 720h
    path: ~/.tokentriage/content

2. Choose a dataset

A dataset is a versioned set of ledger references — a filter, a window and an explicit req_id snapshot — never a copy of your rows. List what exists, then inspect one, re-resolving its snapshot against the live ledger:

$ tokentriage evals datasets
$ tokentriage evals datasets ds_checkout --resolve --config tokentriage.yaml

--resolve reports members retention has rotated out (the snapshot shrank), and the detail view prints fit-eligibility: a dataset holding synthetic or imported rows is fit-ineligible and says so, because those rows are excluded from every calibration fit. Create datasets over the management API (POST /tt/api/v1/evals/datasets) — the CLI reads and runs them.

3. Run the arms

Score each tier as an arm. An arm is key=value,…: a name, a registered evaluator, a registered metric, and optionally the tier it scopes. Write the run record out so you can read the comparison later without re-executing:

$ tokentriage evals run ds_checkout \
    --arm name=frontier,evaluator=code-output-nonempty,metric=output_nonempty,tier=frontier \
    --arm name=cheap,evaluator=code-output-nonempty,metric=output_nonempty,tier=cheap \
    --out run.json --config tokentriage.yaml

4. Compare

Project the same execution onto the comparison surface — the arms table, the Pareto frontier, and the verdict:

$ tokentriage evals compare ds_checkout --run run.json \
    --baseline frontier --candidate cheap --config tokentriage.yaml

  ARM              TIER   BASIS        N     SUBJECT $   EVAL $      QUALITY
  frontier         frontier measured  384   $2.140000   $0.000000   0.9740
  cheap            cheap    measured  384   $0.310000   $0.000000   0.9610

  pareto (cheapest first — 2 point(s)):
    * cheap             cost $0.310000 quality 0.9610 (measured, n=384)
    * frontier          cost $2.140000 quality 0.9740 (measured, n=384)

  verdict     ON_FRONTIER
              both arms sit on the non-dominated frontier; a judged_dataset_run
              re-scores served outputs and cannot show a different choice would beat.

(Illustrative — your figures depend on your own traffic.) Only arms with both a point and a priced dollar are plotted; insufficient or unpriceable arms are excluded-and-named beneath the chart, never dropped at the origin. Add --json to print exactly the bytes GET /tt/api/v1/evals/experiments/{id} serves.

5. Read the four-state verdict

The verdict is one of BEAT / NOT_BEAT / ON_FRONTIER / CANNOT_MEASURE, and CANNOT_MEASURE always carries a reason (arms missing, different windows, mixed bases, or insufficient evidence).