Quality & evals API

The calibration & evals endpoints live in the management API (docs/api/management.yaml, contract 2.19), all under the /tt/api/v1 prefix served by the dashboard server. This page enumerates them grouped by tag; the site renders the full typed spec — request/response schemas, scopes, examples — at /api (Scalar). Do not treat this page as the schema of record.

Calibration lifecycle — tag evidence

Scope evidence:read (read) / evidence:write (activate).

Method & path	Role · scope	Purpose
GET /tt/api/v1/calibrations	viewer · evidence:read	The lifecycle board: the active/shadow manifest pointers and every versioned qcal_ artifact.
GET /tt/api/v1/calibrations/{id}	viewer · evidence:read	One artifact, its threshold diff against the running config, and cross-check warnings. ?sweep=true (opt-in — it reprices the joined ledger under the query budget) adds the repriced threshold sweep.
POST /tt/api/v1/calibrations/{id}/activate	operator · evidence:write	Point the calibration manifest at this artifact and audit it. A gate-failed artifact is refused (403) with its reliability report; a CAS conflict is 409. The response note discloses that live routing does not change until the next config reload.

Evals — tag evals

Scope evals:read (read) / evals:write (mutate). Datasets are ledger references, not copies; experiments create/run are API-only (the studio experiment UI is read-only).

Method & path	Role · scope	Purpose
GET /tt/api/v1/evals/datasets	viewer · evals:read	List dataset artifacts with their item census and fit-eligibility. Filter by builder, fit_eligible.
POST /tt/api/v1/evals/datasets	operator · evals:write	Build a dataset with one of the §2.10 builders (from_filter, from_failures, from_annotation_queue, import, synthetic).
GET /tt/api/v1/evals/datasets/{id}	viewer · evals:read	One dataset with its item snapshot — the explicit req_id list a run resolves, plus each member’s exclusion markers.
DELETE /tt/api/v1/evals/datasets/{id}	operator · evals:write	Delete a dataset artifact. Ledger rows an import/synthetic build appended are not deleted — they stay, marked, forever.
POST /tt/api/v1/evals/datasets/{id}/items	operator · evals:write	Curate the snapshot — add/remove members, merge per-item annotations — re-versioning the artifact under the store lock.
GET /tt/api/v1/evals/experiments	viewer · evals:read	List experiments with their arms, the latest run’s estimates, and the verdict that run licensed.
POST /tt/api/v1/evals/experiments	operator · evals:write	Create an experiment: a pinned dataset, one §2.10 instrument, its arms, and the (baseline, candidate) pair a verdict will be about.
GET /tt/api/v1/evals/experiments/{id}	viewer · evals:read	One experiment: its arms and their whole estimates, the cost/quality Pareto frontier, the verdict — and, when asked, the named comparison or its refusal.
POST /tt/api/v1/evals/experiments/{id}/run	operator · evals:write	Run the experiment over its pinned dataset: score arms, fold estimates, persist the run — including, and especially, when it refuses.
GET /tt/api/v1/evals/runs	viewer · evals:read	Run history including refusals — what was attempted, measured, and refused, newest first.

Quality read surfaces — tag quality

Scope quality:read. Read-only projections of the active artifact and the joined ledger.

Method & path	Role · scope	Purpose
GET /tt/api/v1/quality/coverage	viewer · quality:read	The B.9 evidence-coverage meter: licensed rows over applicable rows, per metric and overall, with the biggest uncovered segments and their need/have gaps.
GET /tt/api/v1/quality/drift	viewer · quality:read	The §2.9 findings timeline: what each registered detector said about each calibration unit, with method, window, magnitude, and a named refusal wherever a detector could not look.
GET /tt/api/v1/quality/reliability	viewer · quality:read	The active artifact’s B.5 holdout report, served verbatim: per-unit measured error beside the gate terms, the reliability-diagram bins, and the segments the fit refused.
GET /tt/api/v1/quality/reliability/segment/{segment}	viewer · quality:read	One §2.4 segment’s B.5 report — every fit unit under that segment key.
GET /tt/api/v1/quality/summary	viewer · quality:read	Scoped quality estimates per segment and per basis, with typed refusals for thin segments and a calibrated interval only on the rows an active artifact licenses.

Outcome ingestion — tag outcomes

Scope quality:write, role floor operator. Append-only companion rows keyed on req_id.

Method & path	Role · scope	Purpose
POST /tt/api/v1/outcomes	operator · quality:write	Submit a batch of eval outcomes as append-only ledger companion rows (registry-checked, basis-ceilinged, actor server-derived, audited).
POST /tt/api/v1/outcomes/otel	operator · quality:write	Ingest OTLP/HTTP JSON gen_ai.evaluation.result events as outcome rows — basis: reported (span money is ignored).

Metacost

Scope analytics:read, role floor viewer, tag analytics.

Method & path	Role · scope	Purpose
GET /tt/api/v1/metacost	viewer · analytics:read	The cost OF measurement — what this platform spent to measure its own routing quality.

Load-bearing schemas

These are the shapes whose honesty rules matter most. Each is hand-pinned: its MarshalJSON refuses every dishonest combination rather than emitting it, so the shapes admitted are narrower than the keys suggest. The full definitions are at /api.

Schema	What it guarantees
QualityEstimate	The only shape a quality figure crosses this API in. point is null exactly when basis is insufficient (a refusal may not whisper a number). interval is present iff calibration is (a reported estimate can never carry one). n >= 1 on measured/reported. eval_cost is a typed Money. The three bases never blend in an aggregate.
Money	A typed dollar. usd is null when unpriced — never 0 — and an unpriced value is excluded from totals. estimated/unpriced carry the provenance; a raw float is unrepresentable.
Provenance	The window, path, index high-water mark, and the demo/estimated row counts a figure was computed over.
EvalDataset	A dataset artifact: builder (the five §2.10 builders), the item census (items, synthetic_items, imported_items, captured_bodies), and fit_eligible (synthetic/imported ⇒ not eligible). References, not copied rows.
EvalExperiment	An experiment’s arms, the cost/quality pareto[] frontier, and the comparison carrying the four-state verdict + verdict_reason.
EvalExperimentRun	One run. status is running | complete | refused — a refusal is a persisted first-class run. cost is Money; unpriced_excluded counts dollars left out rather than summed as $0.
QualityCoverageResponse	The B.9 meter. Ratios are null, not 0.0, when the denominator is 0. licensing_predicate names which of the five conjuncts the numerator tested (b9_full is real; anything else means every number is an upper bound); denominator_basis names what is being counted.
QualityReliabilityResponse	The active artifact’s B.5 report verbatim: measured error beside tolerance, allowance, threshold, min_detectable_error, false_refusal_rate, raw_error (what calibration bought), gate_passed, and the reliability-diagram bins.
QualitySummaryResponse	Per-segment, per-basis estimates with parallel figures per basis and typed refusals on thin segments.
EvalComparison	A named comparison. delta is carried whole and is null (never absent, never 0) when a never-subtract precondition failed; its refusal.code is a closed vocabulary. verdict ∈ {BEAT, NOT_BEAT, ON_FRONTIER, CANNOT_MEASURE}.

See Calibration & evals vocabularies for the closed enums (verdicts, bases, refusal tokens) these schemas carry.