Evaluators & metrics

Before anything can be calibrated it has to be scored. Evaluators are the components that turn a recorded output into a raw quality signal on a metric: schema_validity, groundedness, task_completion, and the rest. What makes the result trustworthy is not the score itself but the basis it is allowed to carry: whether it may claim to be a measured probability or only a reported proxy is fixed by the metric’s ground truth and the evaluator’s kind, before any operator sees a number. This page is the reference for that surface: the registries, the metrics, the evaluator kinds, and the ceiling rules that bind them.

The five registries

Everything here is declared, not discovered. internal/evaluator holds five self-registering registries, each populated from its own descriptor file at process boot and cross-checked by a single ValidateRegistry pass:

Registry	What it holds
Metrics	The measurable properties (schema_validity, groundedness, …), each with a GroundTruthClass.
Evaluators	The scorers that produce outcomes on those metrics, each with a Kind, External, and Metered flag.
Calibrators	The methods that turn a raw score into a bounded probability (clopper-pearson-v1, split-conformal-v1, …).
Outcome sources	Where an outcome entered from (evaluator, human, business, otel, feedback), each with a basis ceiling.
Drift detectors	The registered detectors that decide whether a fitted segment has gone stale.

Registration is partitioned. A duplicate id panics at boot, and an evaluator that names a metric no registry knows about makes the whole registry invalid before the binary starts. The practical consequence is honest: you cannot ship an evaluator that quietly scores a metric nobody defined.

Metrics and the basis ceiling

A metric’s GroundTruthClass answers one question — can a score on this metric ever be a real measurement, or is it inherently a proxy? — and that answer is the metric’s basis ceiling: the strongest basis any outcome on it may claim.

Metric	Ground-truth class	Basis ceiling
schema_validity	execution	measured
output_nonempty	execution	measured
cache_serve_correct	execution	measured
task_completion	human	measured
support_resolution	human	measured
refund_dispute_resolved	human	measured
answer_relevance	heuristic	reported
groundedness	judge	reported

The rule behind the column is small and closed:

• execution, reference, and human classes → the metric licenses measured. An execution check observed a real fact about the output; a human annotation is the ground truth; a resolved ticket is an observed downstream fact.
• judge and heuristic classes → the metric caps at reported until a judge is validated on a specific segment (see below). A raw judge verdict or a token-overlap heuristic is a proxy nobody has yet measured against reality.

Evaluator kinds

An evaluator’s Kind says where the verdict came from; External says whether it makes an outbound network call (the air-gap question); Metered says whether it spends money you must cap. The built-in evaluators:

Evaluator	Kind	External	Metered	Metric	Basis it may claim
rule-json-schema	rule	no	no	schema_validity	measured
code-output-nonempty	code	no	no	output_nonempty	measured
statistical-overlap	statistical	no	no	answer_relevance	reported
judge-groundedness	judge	yes	yes	groundedness	reported
judge-task-completion	judge	yes	yes	task_completion	reported
nli-groundedness	nli (sidecar)	yes	no	groundedness	reported

Alongside these, four ingestion adapters carry outcomes that were formed outside the proxy and arrive over POST /tt/api/v1/outcomes: annotation-queue (human), ticket-resolution-webhook and refund-dispute-webhook (business), and otel-gen-ai-eval (relayed external scorer, always reported). None of these dials out from the proxy, so all are External: false. (A fifth evaluator, cache-verified-equivalence, scores cache_serve_correct, but it is a judge-kind LLM call driven by the response cache’s own off-path loop, not an ingestion adapter, so it is External: true and Metered: true, and the shipped default wires no equivalence judge at all.)

The effective ceiling is a minimum

The basis an outcome may claim is the lowest of the ceilings that apply to it, and min() is the operator: adding a ceiling can only lower the answer, and the one lowest ceiling settles the figure however many sit above it. Three ceilings apply to every quality outcome.

effective basis  =  min( metric ground-truth class,  evaluator kind,  outcome source )

A judge-graded row carries a fourth, the judge disclosure on the row itself, so a figure read off a judged observation is the minimum of four. Exactly three of those four yield to a standing promotion, and Why your number is reported, not measured does that arithmetic with its values on it.

The judge arm is the part this page is the reference for, and it is the same at the write boundary and at every read surface: a judge kind caps at reported until a promotion stands for that observation’s own subject. That is why a judge scoring the human-class task_completion still lands at reported, while an nli sidecar scoring the judge-class groundedness is also reported, this time capped by the metric. Neither ceiling alone gets both cases right; the minimum does.

The other kinds are enumerated in two places in the tree, and the two enumerations disagree on statistical, nli and external. So take the per-evaluator column in the table above, which is correct for every evaluator this build registers, rather than deriving a kind’s ceiling from a rule stated here. The judge-tied ceilings are the ones a promotion moves, and they are the same wherever they are computed.

Local and free vs external and metered

The split in the External/Metered columns is the one an air-gapped operator relies on to be exact.

• Local and free (rule, code, statistical): pure computation over already-captured bytes. No network call, no budget, and no egress surface at all.
• External and metered (judge, sidecar): they leave the process, so they are gated twice. The literal external: allow on the judge: or sidecar: config block is the egress opt-in (SP-16); without it the entry is refused at config validation and is not constructed. A metered judge additionally requires a finite max_usd_per_day; an uncapped metered evaluator is refused as an uncapped spender.

Validating a reported signal into measured

A reported judge signal is not stuck there forever. A judge can be promoted to measured on a specific segment by the B.7 validation protocol, which is the whole walk rather than any one of its checks. It assembles a gold set from human labels and admits it on the agreement between annotators, computes Cohen’s κ between the judge’s verdicts and the admitted human ones, and applies a position-swap ceiling that bounds the flip rate across both orderings and governs pairwise use. A pair whose two orderings disagree is marked judge-uncertain, counted, and excluded from the estimate rather than averaged in. Until the walk finishes without a floor stopping it, the judge’s scores stay reported: displayable and filterable, and not fed to the router as a probability.

A heuristic-class signal is a different case, and no promotion reaches it. Whether a heuristic can observe ground truth is not a question this protocol answers, so a heuristic metric stays reported however good a judge turns out to be.

The promotion is per-segment and raise-only, gated further by validation.require_iaa. Walking it end to end is a task in its own right, and it is Validate a judge and promote it.