Evaluators & metrics

Before anything can be calibrated it has to be scored. Evaluators are the components that turn a recorded output into a raw quality signal on a metric — schema_validity, groundedness, task_completion, and the rest. What makes the result trustworthy is not the score itself but the basis it is allowed to carry: whether it may claim to be a measured probability or only a reported proxy is fixed by the metric’s ground truth and the evaluator’s kind, before any operator sees a number. This page is the reference for that surface — the registries, the metrics, the evaluator kinds, and the ceiling rules that bind them.

The five registries

Everything here is declared, not discovered. internal/evaluator holds five self-registering registries, each populated from its own descriptor file at process boot and cross-checked by a single ValidateRegistry pass:

Registry	What it holds
Metrics	The measurable properties (schema_validity, groundedness, …), each with a GroundTruthClass.
Evaluators	The scorers that produce outcomes on those metrics, each with a Kind, External, and Metered flag.
Calibrators	The methods that turn a raw score into a bounded probability (clopper-pearson-v1, split-conformal-v1, …).
Outcome sources	Where an outcome entered from (evaluator, human, business, otel, feedback), each with a basis ceiling.
Drift detectors	The registered detectors that decide whether a fitted segment has gone stale.

Registration is partitioned — a duplicate id panics at boot, and an evaluator that names a metric no registry knows about makes the whole registry invalid before the binary starts. The practical consequence is honest: you cannot ship an evaluator that quietly scores a metric nobody defined.

Metrics and the basis ceiling

A metric’s GroundTruthClass answers one question — can a score on this metric ever be a real measurement, or is it inherently a proxy? — and that answer is the metric’s basis ceiling: the strongest basis any outcome on it may claim.

Metric	Ground-truth class	Basis ceiling
schema_validity	execution	measured
output_nonempty	execution	measured
cache_serve_correct	execution	measured
task_completion	human	measured
support_resolution	human	measured
refund_dispute_resolved	human	measured
answer_relevance	heuristic	reported
groundedness	judge	reported

The rule behind the column is small and closed:

• execution, reference, and human classes → the metric licenses measured. An execution check observed a real fact about the output; a human annotation is the ground truth; a resolved ticket is an observed downstream fact.
• judge and heuristic classes → the metric caps at reported until a judge is validated on a specific segment (see below). A raw judge verdict or a token-overlap heuristic is a proxy nobody has yet measured against reality.

Evaluator kinds

An evaluator’s Kind says where the verdict came from; External says whether it makes an outbound network call (the air-gap question); Metered says whether it spends money you must cap. The built-in evaluators:

Evaluator	Kind	External	Metered	Metric	Basis it may claim
rule-json-schema	rule	no	no	schema_validity	measured
code-output-nonempty	code	no	no	output_nonempty	measured
statistical-overlap	statistical	no	no	answer_relevance	reported
judge-groundedness	judge	yes	yes	groundedness	reported
judge-task-completion	judge	yes	yes	task_completion	reported
nli-groundedness	nli (sidecar)	yes	no	groundedness	reported

Alongside these, four ingestion adapters carry outcomes that were formed outside the proxy and arrive over POST /tt/api/v1/outcomes: annotation-queue (human), ticket-resolution-webhook and refund-dispute-webhook (business), and otel-gen-ai-eval (relayed external scorer, always reported). None of these dials out from the proxy, so all are External: false. (A fifth evaluator, cache-verified-equivalence, scores cache_serve_correct — but it is a judge-kind LLM call driven by the response cache’s own off-path loop, not an ingestion adapter, so it is External: true and Metered: true, and the shipped default wires no equivalence judge at all.)

The effective ceiling is a minimum

The basis a written outcome may actually claim is the lowest of three ceilings, computed at the write boundary — the one layer that can see all three:

effective basis  =  min( metric ground-truth class,  evaluator kind,  outcome source )

Only a judge kind caps at reported; rule, code, statistical, nli, human, and external kinds impose no ceiling of their own and defer to the metric. That is why a judge scoring the human-class task_completion still lands at reported — routed around by one word would be a hole, so the minimum closes it — while an nli sidecar scoring the judge-class groundedness is also reported, this time capped by the metric. Neither ceiling alone gets both cases right; the minimum does.

Local and free vs external and metered

The split in the External/Metered columns is the one an air-gapped operator relies on to be exact.

• Local and free (rule, code, statistical): pure computation over already-captured bytes. No network call, no budget — they run without any egress surface at all.
• External and metered (judge, sidecar): they leave the process, so they are gated twice. The literal external: allow on the judge: or sidecar: config block is the egress opt-in (SP-16) — without it the entry is refused at config validation and never constructed. A metered judge additionally requires a finite max_usd_per_day; an uncapped metered evaluator is refused as an uncapped spender.

Validating a reported signal into measured

A reported signal is not stuck there forever. A judge or heuristic can be promoted to measured on a specific segment by passing the B.7 validation gate — a pairwise swap test that runs both orderings on at least 30 pairs and demands a flip rate no higher than 0.10. A pair whose two orderings disagree is marked judge-uncertain, counted, and excluded from the estimate rather than averaged in. Until that gate passes, the judge’s scores stay reported and can be displayed and filtered but never fed to the router as a probability.

The promotion is per-segment and raise-only, gated further by validation.require_iaa. Walking it end to end — the config, the swap gate, and watching the cost of measurement — is a task in its own right.