# Quality & evals API
The management API for calibration and evals (contract 2.19) — lifecycle, datasets, experiments, and quality read surfaces, grouped by tag.
Source: https://tokentriage.com/docs/reference/quality-api
---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
The calibration & evals endpoints live in the management API (`docs/api/management.yaml`,
contract 2.19), all under the `/tt/api/v1` prefix served by the dashboard server. This page
enumerates them grouped by tag; the site renders the full typed spec — request/response schemas,
scopes, examples — at [/api](/api) (Scalar). Do not treat this page as the schema of record.
**This feature's API is `docs/api/management.yaml` (contract 2.19), not the retired
`governance.yaml`.** And **activation does not move live routing until the next config reload** —
the `POST …/activate` response `note` and the studio both disclose this.
## Calibration lifecycle — tag `evidence`
Scope `evidence:read` (read) / `evidence:write` (activate).
| Method & path | Role · scope | Purpose |
|---|---|---|
| `GET /tt/api/v1/calibrations` | viewer · `evidence:read` | The lifecycle board: the active/shadow manifest pointers and every versioned `qcal_` artifact. |
| `GET /tt/api/v1/calibrations/{id}` | viewer · `evidence:read` | One artifact, its threshold diff against the **running** config, and cross-check warnings. `?sweep=true` (opt-in — it reprices the joined ledger under the query budget) adds the repriced threshold sweep. |
| `POST /tt/api/v1/calibrations/{id}/activate` | operator · `evidence:write` | Point the calibration manifest at this artifact and audit it. A gate-failed artifact is refused (403) with its reliability report; a CAS conflict is 409. The response `note` discloses that **live routing does not change until the next config reload**. |
## Evals — tag `evals`
Scope `evals:read` (read) / `evals:write` (mutate). Datasets are ledger references, not copies;
experiments create/run are **API-only** (the studio experiment UI is read-only).
| Method & path | Role · scope | Purpose |
|---|---|---|
| `GET /tt/api/v1/evals/datasets` | viewer · `evals:read` | List dataset artifacts with their item census and fit-eligibility. Filter by `builder`, `fit_eligible`. |
| `POST /tt/api/v1/evals/datasets` | operator · `evals:write` | Build a dataset with one of the §2.10 builders (`from_filter`, `from_failures`, `from_annotation_queue`, `import`, `synthetic`). |
| `GET /tt/api/v1/evals/datasets/{id}` | viewer · `evals:read` | One dataset with its item snapshot — the explicit `req_id` list a run resolves, plus each member's exclusion markers. |
| `DELETE /tt/api/v1/evals/datasets/{id}` | operator · `evals:write` | Delete a dataset artifact. Ledger rows an `import`/`synthetic` build appended are **not** deleted — they stay, marked, forever. |
| `POST /tt/api/v1/evals/datasets/{id}/items` | operator · `evals:write` | Curate the snapshot — add/remove members, merge per-item annotations — re-versioning the artifact under the store lock. |
| `GET /tt/api/v1/evals/experiments` | viewer · `evals:read` | List experiments with their arms, the latest run's estimates, and the verdict that run licensed. |
| `POST /tt/api/v1/evals/experiments` | operator · `evals:write` | Create an experiment: a pinned dataset, one §2.10 instrument, its arms, and the `(baseline, candidate)` pair a verdict will be about. |
| `GET /tt/api/v1/evals/experiments/{id}` | viewer · `evals:read` | One experiment: its arms and their whole estimates, the cost/quality Pareto frontier, the verdict — and, when asked, the named comparison or its refusal. |
| `POST /tt/api/v1/evals/experiments/{id}/run` | operator · `evals:write` | Run the experiment over its pinned dataset: score arms, fold estimates, persist the run — **including, and especially, when it refuses**. |
| `GET /tt/api/v1/evals/runs` | viewer · `evals:read` | Run history *including refusals* — what was attempted, measured, and refused, newest first. |
Three further routes carry the `evals` tag and are documented on their own page. `GET /evals/judges`
and `GET /evals/judges/{subject}/history` read the B.7 promotion registry, and
`POST /evals/annotation-queue/claim` leases ledger rows for human labelling. See
[Judge API](/docs/reference/judge-api).
The opt-in instruments `shadow_dual_run` (live) and `replay` (simulation only) return
**`404 feature_absent`** by name when the deployment has not wired them. Only `shadow_dual_run`
is counterfactual: a `judged_dataset_run` re-scores stored outputs and can never reach the
`BEAT` verdict (Inv-7), `replay` carries no quality at all, and only a live `shadow_dual_run`
can license a `BEAT`. See
[Eval programs & verdicts](/docs/calibration/eval-programs).
## Quality read surfaces — tag `quality`
Scope `quality:read`. Read-only projections of the active artifact and the joined ledger.
| Method & path | Role · scope | Purpose |
|---|---|---|
| `GET /tt/api/v1/quality/coverage` | viewer · `quality:read` | The B.9 evidence-coverage meter: licensed rows over applicable rows, per metric and overall, with the biggest uncovered segments and their need/have gaps. |
| `GET /tt/api/v1/quality/drift` | viewer · `quality:read` | The §2.9 findings timeline: what each registered detector said about each calibration unit, with method, window, magnitude, and a named refusal wherever a detector could not look. |
| `GET /tt/api/v1/quality/reliability` | viewer · `quality:read` | The **active** artifact's B.5 holdout report, served verbatim: per-unit measured error beside the gate terms, the reliability-diagram bins, and the segments the fit refused. |
| `GET /tt/api/v1/quality/reliability/segment/{segment}` | viewer · `quality:read` | One §2.4 segment's B.5 report — every fit unit under that segment key. |
| `GET /tt/api/v1/quality/summary` | viewer · `quality:read` | Scoped quality estimates per segment **and per basis**, with typed refusals for thin segments and a calibrated interval only on the rows an active artifact licenses. |
## Outcome ingestion — tag `outcomes`
Scope `quality:write`, role floor operator. Append-only companion rows keyed on `req_id`.
| Method & path | Role · scope | Purpose |
|---|---|---|
| `POST /tt/api/v1/outcomes` | operator · `quality:write` | Submit a batch of eval outcomes as append-only ledger companion rows (registry-checked, basis-ceilinged, actor server-derived, audited). |
| `POST /tt/api/v1/outcomes/otel` | operator · `quality:write` | Ingest OTLP/HTTP JSON `gen_ai.evaluation.result` events as outcome rows — **`basis: reported`** (span money is ignored). |
## Metacost
Scope `analytics:read`, role floor viewer, tag `analytics`.
| Method & path | Role · scope | Purpose |
|---|---|---|
| `GET /tt/api/v1/metacost` | viewer · `analytics:read` | The **cost OF measurement** — what this platform spent to measure its own routing quality. |
## Load-bearing schemas
These are the shapes whose honesty rules matter most. Each is hand-pinned: its `MarshalJSON`
refuses every dishonest combination rather than emitting it, so the shapes admitted are narrower
than the keys suggest. The full definitions are at [/api](/api).
| Schema | What it guarantees |
|---|---|
| `QualityEstimate` | The **only** shape a quality figure crosses this API in. `point` is `null` **exactly** when `basis` is `insufficient` (a refusal may not whisper a number). `interval` is present **iff** `calibration` is (a `reported` estimate can never carry one). `n >= 1` on `measured`/`reported`. `eval_cost` is a typed `Money`. The three bases never blend in an aggregate. |
| `Money` | A typed dollar. `usd` is `null` when `unpriced` — **never `0`** — and an unpriced value is excluded from totals. `estimated`/`unpriced` carry the provenance; a raw float is unrepresentable. |
| `Provenance` | The window, path, index high-water mark, and the demo/estimated row counts a figure was computed over. |
| `EvalDataset` | A dataset artifact: `builder` (the five §2.10 builders), the item census (`items`, `synthetic_items`, `imported_items`, `captured_bodies`), and `fit_eligible` (synthetic/imported ⇒ not eligible). References, not copied rows. |
| `EvalExperiment` | An experiment's `arms`, the cost/quality `pareto[]` frontier, and the `comparison` carrying the four-state `verdict` + `verdict_reason`. |
| `EvalExperimentRun` | One run. `status` is `running` \| `complete` \| `refused` — a refusal is a persisted first-class run. `cost` is `Money`; `unpriced_excluded` counts dollars left out rather than summed as `$0`. |
| `QualityCoverageResponse` | The B.9 meter. Ratios are **`null`, not `0.0`**, when the denominator is 0. `licensing_predicate` names which of the five conjuncts the numerator tested (`b9_full` is real; anything else means every number is an **upper bound**); `denominator_basis` names what is being counted. |
| `QualityReliabilityResponse` | The active artifact's B.5 report verbatim: measured `error` beside `tolerance`, `allowance`, `threshold`, `min_detectable_error`, `false_refusal_rate`, `raw_error` (what calibration bought), `gate_passed`, and the reliability-diagram `bins`. |
| `QualitySummaryResponse` | Per-segment, per-basis estimates with parallel figures per basis and typed refusals on thin segments. |
| `EvalComparison` | A named comparison. `delta` is carried whole and is `null` (never absent, never `0`) when a never-subtract precondition failed; its `refusal.code` is a closed vocabulary. `verdict` ∈ `{BEAT, NOT_BEAT, ON_FRONTIER, CANNOT_MEASURE}`. |
See [Calibration & evals vocabularies](/docs/reference/quality-vocabularies) for the closed enums
(verdicts, bases, refusal tokens) these schemas carry.