# Calibration & Evals: measured quality

Attach quality numbers to your routed traffic, check them against held-out data, and gate a cheaper tier on a floor you can defend.

Source: https://tokentriage.com/docs/calibration/overview

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import QualitySubstrateDiagram from '../../../components/QualitySubstrateDiagram.astro';

Routing a request to a cheaper model raises one question: *did quality drop?* This section is
how you answer it with numbers rather than a hunch.

Calibration and evals let you:

- score your own traffic on metrics like `schema_validity`, `groundedness`, or
  `task_completion`;
- turn those scores into probabilities whose error has been measured on data the fit never saw;
- gate a cheaper tier on a quality floor, so routing only downgrades where the floor holds.

Start with [Calibrate quality from your logs](/docs/calibration/calibrate-from-logs) if you want
to run it; read on for the model behind it.

## Two nouns to know

Almost everything here is either a **quality estimate** or a **calibration**.

A **quality estimate** is a number about how good outputs are on one metric. It does not travel
as a bare float: it carries a **basis** saying what kind of claim it is.

| Basis | What it means | Can it be a probability? |
|---|---|---|
| `measured` | A real evaluator ran, and the metric's ground-truth class licenses measurement — an execution or rule check, or a judge validated on this segment. | Yes. |
| `reported` | An unvalidated proxy: a raw LLM-judge score, a heuristic. Displayable and filterable, but nobody has measured its error. | No. |
| `insufficient` | Not enough evidence to say anything yet. A first-class, routable state carrying a reason and `need`/`have` counts. | No — it reports the gap rather than a silent `0.0`. |

A **calibration** turns an evaluator's *raw score* into a probability whose error has been
measured on a held-out slice of the same window. That measurement lives in the artifact the fit
produces, and it gates promotion: an artifact that fails its own measurement can be written to
disk and can then be served by nothing.

<Callout type="key">
An estimate says *how good*. A basis says *how much you may trust that*. A calibration is the
only thing allowed to move a signal from "we saw a score" to "we measured a probability, and
here is its error."
</Callout>

A too-thin segment therefore comes back as an `insufficient` entry with its `need`/`have`
counts rather than a quiet `0.0`, and `measured` and `reported` are never blended into one
aggregate — they are different kinds of claim, not degrees of confidence. The router's own
verdict follows the same rule; see
[Eval programs & verdicts](/docs/calibration/eval-programs) and the
[two verdict vocabularies](/docs/concepts/honesty-model#two-verdict-vocabularies--dont-conflate-them)
note, which are the pair most often mixed up.

## How the pieces fit

Every quality number traces back to the [decision ledger](/docs/concepts/core-concepts#the-decision-ledger).
Evaluators score the outputs recorded there; each score becomes an estimate carrying its basis;
the calibration engine fits and gates those scores into a versioned `qcal_` artifact; and the
router, when you opt in, reads the active artifact to decide whether a cheaper tier has earned
the traffic.

<QualitySubstrateDiagram />

The diagram also shows the **two tracks**, which share the same estimates and the same ledger
but answer different questions:

- **Calibration** is the *online* track: the probability engine that fits per-segment quality,
  measures its error, and publishes an artifact the router can consult on the hot path. It is
  what lets [quality-aware routing](/docs/calibration/quality-aware-routing) gate a cheaper
  tier on a proven floor.
- **Evals** is the *offline* track: a program that re-scores stored outputs (or, with a live
  data plane, re-runs them) to compare candidate routing choices on a cost/quality frontier
  and return a four-state verdict. It is how you [prove a cheaper tier](/docs/calibration/prove-a-cheaper-tier)
  before you trust it in production.

The two meet at the estimate: an eval experiment reads the same `measured`/`reported`/`insufficient`
bases the calibration engine produces, and a calibration is fit from the same ledger an eval
program samples.

## Where to go next

The section is organised so you can read the *concepts* first or jump straight to a *task*.

- **Concepts** explain how it works: this page, [How calibration works](/docs/calibration/how-calibration-works)
  (the fit engine), [The calibration lifecycle](/docs/calibration/lifecycle) (states, refit,
  and audit), [Evaluators & metrics](/docs/calibration/evaluators), [Eval programs & verdicts](/docs/calibration/eval-programs),
  and [Quality-aware routing](/docs/calibration/quality-aware-routing).
- **How-tos** get a job done: [calibrate quality from your logs](/docs/calibration/calibrate-from-logs),
  [prove a cheaper tier](/docs/calibration/prove-a-cheaper-tier), [gate routing on a proven
  floor](/docs/calibration/gate-routing-on-quality), [validate an LLM judge](/docs/calibration/validate-a-judge),
  and [keep calibrations fresh](/docs/calibration/drift-and-refit).
- **Reference** is the exact surface: the [evals & calibration configuration](/docs/reference/evals-config),
  the [quality & evals API](/docs/reference/quality-api), and the [closed vocabularies](/docs/reference/quality-vocabularies).

One evaluator kind has a section of its own, because a model that grades your outputs has to be
graded too. Configuring a judge, collecting the human labels it is measured against, running the
check that promotes it, and reading why a promoted judge's figure has not moved are all in
[LLM judges](/docs/judges/overview). The page above on validating one is the orientation; the
procedure lives there.

<NextSteps items={[
  { title: 'How calibration works', href: '/docs/calibration/how-calibration-works', description: 'The fit engine: from a raw score to a measured probability, split, measured, and gated.' },
  { title: 'Eval programs & verdicts', href: '/docs/calibration/eval-programs', description: 'The offline half — datasets, instruments, the Pareto frontier, and the four-state verdict.' },
  { title: 'Calibrate quality from your logs', href: '/docs/calibration/calibrate-from-logs', description: 'Fit and activate your first quality calibration from the decision log.' },
  { title: 'LLM judges', href: '/docs/judges/overview', description: 'The section for the one evaluator kind that has to be graded itself, from a first judged row to a routing decision.' }
]} />