# Validate an LLM judge

What a judge validation is, what it costs to get one, and which page in the LLM judges section runs each step.

Source: https://tokentriage.com/docs/calibration/validate-a-judge

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';

A model that grades your outputs produces a `reported` score: displayable, filterable, and not yet
a probability anybody has checked. Validating that judge is the work of measuring its verdicts
against human ones on a specific segment, and recording the answer where the read path can find it.

This page is the short orientation. The procedure, the config surface, the statistics and the
failure modes each have a page of their own in
[LLM judges](/docs/judges/overview), which is where the judge track now lives.

## What a validation actually measures

The protocol the product calls **B.7** is the whole walk, and its floors are named constants a
refusal quotes back at you. It runs in this order, and each step can stop it.

1. **Assemble a gold set** from human labels already on ledger rows, and admit it or turn it away.
   Admission includes the agreement between annotators, `κ_h`, measured on the requests two of
   them both labelled rather than on the union of their work.
2. **Compute Cohen's κ** between the judge's verdicts and the admitted human ones, over the label
   events on that segment.
3. **Apply the position-swap ceiling**, which bounds the flip rate across both presentation
   orderings and governs pairwise use.
4. **Record the outcome** in the promotion registry, whether it promoted or declined.

"B.7" is the name of that protocol, not of any one of its steps. The annotator-agreement admission
in step 1 and the swap ceiling in step 3 are two steps inside it, and both are worth naming by what
they do rather than by the label, which appears in output only where the binary prints it.

<Callout type="key">
There is no command that flips a judge to validated. A promotion is what the walk leaves behind
when no floor had anything to hold against the subject, and it is scoped to one judge id on one
metric on one segment, pinned to the judge model, the prompt sha and the evaluator version. Move
any of those three and the pin stops matching.
</Callout>

## What it takes, before you start

| What | Why |
|---|---|
| A judge evaluator whose metric carries a pass floor | the gold-set assembler records a verdict only where one is set |
| Human labels on ledger rows | a second annotator with an overlapping sample is what makes agreement between them measurable; `require_iaa` is `false` by default, so plan the overlap before you need it |
| Captured request and response bodies | an annotator reads those bytes; a deployment storing hashes alone has nothing to offer |
| An egress path and a credential for the judge | a judge on this plane calls its provider on every sampled request |
| A daily budget on the evaluator | a metered evaluator without a finite cap is turned away at config |

The judge and the tiers it grades are always different model families, and the config turns away a
same-family pair by name.

## Where each step lives now

| The step | The page |
|---|---|
| Set the judge up and read every key it takes | [Configure a judge](/docs/judges/configure-a-judge) |
| Get human labels onto rows, and plan the overlap | [Collect the labels a judge is measured against](/docs/judges/collect-labels) |
| Run the check and read both answers | [Validate a judge and promote it](/docs/judges/validate-and-promote) |
| Understand κ, the annotator agreement and the swap gate | [How a judge reaches a measured basis](/docs/judges/agreement) |
| Find out why a promoted judge's figure has not moved | [Why your number is reported, not measured](/docs/judges/basis-ceilings) |
| Turn a promotion into a routing decision | [Gate a rollout on a validated judge](/docs/judges/gate-a-rollout) |
| Work out what a symptom means | [Why didn't my judge validate?](/docs/judges/troubleshooting) |

## Two things this page used to say, corrected

**Read the judge record, not the reliability report.** `tokentriage quality reliability` reports on
a calibration artifact and says nothing about a judge's promotion state. The record with its named
floor and its need and have counts comes from `tokentriage evals validate`, and the same record is
served over `GET /tt/api/v1/evals/judges`.

**Pick the judge id that can finish the walk.** Of the two registered judge evaluators, only
`judge-task-completion` can reach a promotion: `judge-groundedness` writes a scalar score and no
pass verdict, so the gold-set assembler admits none of its rows and the label count on its subject
stays at zero however many labels arrive.
[Why didn't my judge validate?](/docs/judges/troubleshooting) has the symptom and the fix.

<NextSteps items={[
  { title: 'Validate a judge and promote it', href: '/docs/judges/validate-and-promote', description: 'The procedure: run the check, read the answer whether it promotes or declines, and find the floor a declined run names.' },
  { title: 'LLM judges', href: '/docs/judges/overview', description: 'The section this subject moved into: what a judge scores, and what that score licenses until somebody measures it.' },
  { title: 'How a judge reaches a measured basis', href: '/docs/judges/agreement', description: "Cohen's kappa, annotator agreement and the swap gate, walked in the order the binary walks them." }
]} />