Validate an LLM judge

A model that grades your outputs produces a reported score: displayable, filterable, and not yet a probability anybody has checked. Validating that judge is the work of measuring its verdicts against human ones on a specific segment, and recording the answer where the read path can find it.

This page is the short orientation. The procedure, the config surface, the statistics and the failure modes each have a page of their own in LLM judges, which is where the judge track now lives.

What a validation actually measures

The protocol the product calls B.7 is the whole walk, and its floors are named constants a refusal quotes back at you. It runs in this order, and each step can stop it.

1. Assemble a gold set from human labels already on ledger rows, and admit it or turn it away. Admission includes the agreement between annotators, κ_h, measured on the requests two of them both labelled rather than on the union of their work.
2. Compute Cohen’s κ between the judge’s verdicts and the admitted human ones, over the label events on that segment.
3. Apply the position-swap ceiling, which bounds the flip rate across both presentation orderings and governs pairwise use.
4. Record the outcome in the promotion registry, whether it promoted or declined.

“B.7” is the name of that protocol, not of any one of its steps. The annotator-agreement admission in step 1 and the swap ceiling in step 3 are two steps inside it, and both are worth naming by what they do rather than by the label, which appears in output only where the binary prints it.

What it takes, before you start

What	Why
A judge evaluator whose metric carries a pass floor	the gold-set assembler records a verdict only where one is set
Human labels on ledger rows	a second annotator with an overlapping sample is what makes agreement between them measurable; require_iaa is false by default, so plan the overlap before you need it
Captured request and response bodies	an annotator reads those bytes; a deployment storing hashes alone has nothing to offer
An egress path and a credential for the judge	a judge on this plane calls its provider on every sampled request
A daily budget on the evaluator	a metered evaluator without a finite cap is turned away at config

The judge and the tiers it grades are always different model families, and the config turns away a same-family pair by name.

Where each step lives now

The step	The page
Set the judge up and read every key it takes	Configure a judge
Get human labels onto rows, and plan the overlap	Collect the labels a judge is measured against
Run the check and read both answers	Validate a judge and promote it
Understand κ, the annotator agreement and the swap gate	How a judge reaches a measured basis
Find out why a promoted judge’s figure has not moved	Why your number is reported, not measured
Turn a promotion into a routing decision	Gate a rollout on a validated judge
Work out what a symptom means	Why didn’t my judge validate?

Two things this page used to say, corrected

Read the judge record, not the reliability report. tokentriage quality reliability reports on a calibration artifact and says nothing about a judge’s promotion state. The record with its named floor and its need and have counts comes from tokentriage evals validate, and the same record is served over GET /tt/api/v1/evals/judges.

Pick the judge id that can finish the walk. Of the two registered judge evaluators, only judge-task-completion can reach a promotion: judge-groundedness writes a scalar score and no pass verdict, so the gold-set assembler admits none of its rows and the label count on its subject stays at zero however many labels arrive. Why didn’t my judge validate? has the symptom and the fix.