# Evaluators & metrics
What produces a raw quality signal, and the ground-truth rules that cap how strong a claim each evaluator kind may make.
Source: https://tokentriage.com/docs/calibration/evaluators
---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
Before anything can be calibrated it has to be *scored*. **Evaluators** are the components that
turn a recorded output into a raw quality signal on a **metric**: `schema_validity`,
`groundedness`, `task_completion`, and the rest. What makes the result trustworthy is not the
score itself but the **basis** it is allowed to carry: whether it may claim to be a *measured*
probability or only a *reported* proxy is fixed by the metric's ground truth and the evaluator's
kind, before any operator sees a number. This page is the reference for that surface: the
registries, the metrics, the evaluator kinds, and the ceiling rules that bind them.
## The five registries
Everything here is declared, not discovered. `internal/evaluator` holds **five
self-registering registries**, each populated from its own descriptor file at process boot and
cross-checked by a single `ValidateRegistry` pass:
| Registry | What it holds |
|---|---|
| **Metrics** | The measurable properties (`schema_validity`, `groundedness`, …), each with a `GroundTruthClass`. |
| **Evaluators** | The scorers that produce outcomes on those metrics, each with a `Kind`, `External`, and `Metered` flag. |
| **Calibrators** | The methods that turn a raw score into a bounded probability (`clopper-pearson-v1`, `split-conformal-v1`, …). |
| **Outcome sources** | Where an outcome entered from (`evaluator`, `human`, `business`, `otel`, `feedback`), each with a basis ceiling. |
| **Drift detectors** | The registered detectors that decide whether a fitted segment has gone stale. |
Registration is partitioned. A duplicate id *panics* at boot, and an evaluator that names a
metric no registry knows about makes the whole registry invalid before the binary starts. The
practical consequence is honest: you cannot ship an evaluator that quietly scores a metric
nobody defined.
## Metrics and the basis ceiling
A metric's `GroundTruthClass` answers one question — *can a score on this metric ever be a real
measurement, or is it inherently a proxy?* — and that answer is the metric's **basis ceiling**:
the strongest basis any outcome on it may claim.
| Metric | Ground-truth class | Basis ceiling |
|---|---|---|
| `schema_validity` | `execution` | `measured` |
| `output_nonempty` | `execution` | `measured` |
| `cache_serve_correct` | `execution` | `measured` |
| `task_completion` | `human` | `measured` |
| `support_resolution` | `human` | `measured` |
| `refund_dispute_resolved` | `human` | `measured` |
| `answer_relevance` | `heuristic` | `reported` |
| `groundedness` | `judge` | `reported` |
The rule behind the column is small and closed:
- `execution`, `reference`, and `human` classes → the metric licenses **`measured`**. An
execution check observed a real fact about the output; a human annotation *is* the ground
truth; a resolved ticket is an observed downstream fact.
- `judge` and `heuristic` classes → the metric caps at **`reported`** until a judge is validated
on a specific segment (see below). A raw judge verdict or a token-overlap heuristic is a
proxy nobody has yet measured against reality.
A `measured` **ceiling** is a permission, not a promise. `task_completion` *can* be measured —
when a human actually labelled it — but a judge scoring the same metric is still capped at
`reported`, because the ceiling is the **minimum** of every constraint, not the metric's alone.
That minimum is the next section.
## Evaluator kinds
An evaluator's `Kind` says *where the verdict came from*; `External` says whether it makes an
outbound network call (the air-gap question); `Metered` says whether it spends money you must
cap. The built-in evaluators:
| Evaluator | Kind | External | Metered | Metric | Basis it may claim |
|---|---|---|---|---|---|
| `rule-json-schema` | rule | no | no | `schema_validity` | `measured` |
| `code-output-nonempty` | code | no | no | `output_nonempty` | `measured` |
| `statistical-overlap` | statistical | no | no | `answer_relevance` | `reported` |
| `judge-groundedness` | judge | **yes** | **yes** | `groundedness` | `reported` |
| `judge-task-completion` | judge | **yes** | **yes** | `task_completion` | `reported` |
| `nli-groundedness` | nli (sidecar) | **yes** | no | `groundedness` | `reported` |
Alongside these, four **ingestion adapters** carry outcomes that were formed outside the proxy
and arrive over `POST /tt/api/v1/outcomes`: `annotation-queue` (human), `ticket-resolution-webhook`
and `refund-dispute-webhook` (business), and `otel-gen-ai-eval` (relayed external scorer, always
`reported`). None of these dials out from the proxy, so all are `External: false`. (A fifth
evaluator, `cache-verified-equivalence`, scores `cache_serve_correct`, but it is a `judge`-kind
LLM call driven by the response cache's own off-path loop, not an ingestion adapter, so it is
`External: true` and `Metered: true`, and the shipped default wires no equivalence judge at all.)
### The effective ceiling is a minimum
The basis an outcome may claim is the **lowest** of the ceilings that apply to it, and `min()` is
the operator: adding a ceiling can only lower the answer, and the one lowest ceiling settles the
figure however many sit above it. Three ceilings apply to every quality outcome.
```text
effective basis = min( metric ground-truth class, evaluator kind, outcome source )
```
A judge-graded row carries a fourth, the judge disclosure on the row itself, so a figure read off a
judged observation is the minimum of **four**. Exactly three of those four yield to a standing
promotion, and [Why your number is reported, not measured](/docs/judges/basis-ceilings) does that
arithmetic with its values on it.
The `judge` arm is the part this page is the reference for, and it is the same at the write
boundary and at every read surface: a `judge` **kind** caps at `reported` until a promotion stands
for that observation's own subject. That is why a judge scoring the human-class `task_completion`
still lands at `reported`, while an `nli` sidecar scoring the judge-class `groundedness` is also
`reported`, this time capped by the **metric**. Neither ceiling alone gets both cases right; the
minimum does.
The other kinds are enumerated in two places in the tree, and the two enumerations disagree on
`statistical`, `nli` and `external`. So take the per-evaluator column in the table above, which is
correct for every evaluator this build registers, rather than deriving a kind's ceiling from a rule
stated here. The judge-tied ceilings are the ones a promotion moves, and they are the same wherever
they are computed.
## Local and free vs external and metered
The split in the `External`/`Metered` columns is the one an air-gapped operator relies on to be
exact.
- **Local and free** (`rule`, `code`, `statistical`): pure computation over already-captured
bytes. No network call, no budget, and no egress surface at all.
- **External and metered** (`judge`, `sidecar`): they leave the process, so they are gated
twice. The literal `external: allow` on the `judge:` or `sidecar:` config block **is** the
egress opt-in (SP-16); without it the entry is refused at config validation and is not
constructed. A metered judge additionally requires a finite `max_usd_per_day`; an uncapped
metered evaluator is refused as an uncapped spender.
Credentials are configured as **environment-variable names, never values**: `api_key_env`,
`endpoint_env`. TokenTriage stores and displays the *reference*, never the secret, exactly as
the [honesty model](/docs/concepts/honesty-model#the-invariants) requires. A judge is also
always configured on a **different model family** than the tiers it grades, so it cannot mark
its own homework.
## Validating a reported signal into measured
A `reported` judge signal is not stuck there forever. A judge can be *promoted* to `measured` on a
specific segment by the **B.7 validation protocol**, which is the whole walk rather than any one of
its checks. It assembles a gold set from human labels and admits it on the agreement between
annotators, computes Cohen's κ between the judge's verdicts and the admitted human ones, and
applies a position-swap ceiling that bounds the flip rate across both orderings and governs
pairwise use. A pair whose two orderings disagree is marked **judge-uncertain**, counted, and
excluded from the estimate rather than averaged in. Until the walk finishes without a floor
stopping it, the judge's scores stay `reported`: displayable and filterable, and not fed to the
router as a probability.
A heuristic-class signal is a different case, and no promotion reaches it. Whether a heuristic can
observe ground truth is not a question this protocol answers, so a `heuristic` metric stays
`reported` however good a judge turns out to be.
The promotion is per-segment and raise-only, gated further by `validation.require_iaa`. Walking it
end to end is a task in its own right, and it is
[Validate a judge and promote it](/docs/judges/validate-and-promote).
The basis on every quality number is not a label an author chose; it is the **floor** of what
the metric, the evaluator, and the source each permit. That is why a cheaper tier's pass rate
can be *shown* to a human as `reported` long before it is ever allowed to *gate a route* as
`measured`.