Validate an LLM judge (air-gapped)

An LLM judge produces a reported score — displayable and filterable, but never a probability — until it has been validated against human labels on a specific segment. This page wires the judge behind the air-gap gate, supplies the gold set, and lets measurement earn the promotion. Nothing here makes a judge validated: you enable the ingestion, the scorer and the fitter, and promotion is earned by measurement.

Prerequisites

• A metric whose ground-truth class is judge-scored (e.g. groundedness, task_completion) — see Evaluators & metrics.
• An egress path you are willing to open for the judge, and a human-labelling source for the gold set.

1. Understand the basis ceiling

A metric’s ground-truth class sets a basis ceiling. Execution-, rule- and code-class signals may claim measured from row one; a judge (or heuristic) is capped at reported until validation promotes it on a specific (judge, prompt, metric, segment). Until then the segment stays insufficient(basis_ceiling_reported) — it is not a probability, and the product will not treat it as one.

2. Configure the judge behind the air-gap gate

A judge makes network calls, so it runs only under an explicit egress opt-in. The literal external: allow is the air-gap gate — without it the entry is refused at config validation:

evals:
  evaluators:
    - id: judge-groundedness
      judge:
        external: allow                 # the SP-16 air-gap gate — egress opt-in
        model: gemini-2.5-flash         # a DIFFERENT family than your gpt/claude tiers
        api_key_env: JUDGE_API_KEY      # an env-var NAME, never the key itself
      sample_rate: 0.05
      max_usd_per_day: 5.0              # metered ⇒ a daily budget is mandatory

3. Turn on the validation gate

validation.require_iaa gates promotion on inter-annotator agreement. Like the evidence floors, it is raise-only — you can demand the check, you cannot quietly waive it:

evals:
  validation:
    require_iaa: true                   # raise-only judge-validation gate

4. Feed the gold set

Validation is measured against human labels, delivered as eval outcomes through the annotation queue (POST /tt/api/v1/outcomes). B.7 admits the gold set first — a judge’s agreement is only meaningful against labels the annotators themselves agree on — then computes judge-vs-human agreement on the segment. Until enough labels accrue, the segment stays insufficient and licenses nothing.

5. Let measurement promote it — then read whether it did

There is no command that flips a judge to validated. Promotion is a fact earned by measurement: once judge-vs-human agreement clears the gate on a segment, the judge’s outcomes from that point forward carry measured; rows written before are not rewritten. Check whether it happened by reading the reliability report:

$ tokentriage quality reliability --config tokentriage.yaml

A promoted segment now measures on the measured basis; an unpromoted one still prints insufficient(basis_ceiling_reported) — the honest “not yet”.