How a judge reaches a measured basis

A model grading another model’s work hands you a number, and nothing in the number tells you whether a person would have said the same thing. Until somebody checks, that score is one machine’s opinion of another machine’s output: good enough to rank two prompts against each other, thin ground for a decision about money.

Checking it is measurable work. Label a sample of the same traffic by hand, count how often the judge and the people landed on the same answer, then take out the agreement two coin flips would have produced anyway. What is left is Cohen’s kappa (κ), and κ is the statistic this build promotes on.

This page walks that check in the order the binary walks it, names the floor and the constant each gate applies, and then says what a promotion changes for the figures you read afterwards. The last part is smaller than most readers expect, and it is the half that gets misremembered.

If you have not seen a graded row yet, How a judge call happens comes first. Getting human labels onto rows is Collect the labels a judge is measured against, and running the check is Validate a judge and promote it. This page is the model under both, and there is no procedure on it.

A promotion belongs to a subject, not to a judge

The sentence everybody reaches for first is “our judge is validated”, and this build cannot check that sentence, because it does not promote judges. It promotes subjects. A subject is a six-coordinate address, and every statistic below is scoped to one.

{
  "id": "1d7a361adad5",
  "judge_id": "judge-task-completion",
  "model": "claude-haiku-4-5",
  "prompt_sha": "1fc3923",
  "eval_ver": "judge-v1+cfg15acf67",
  "metric": "task_completion",
  "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat"
}

That is the subject block off GET /tt/api/v1/evals/judges, captured from a booted instance. The id is a truncated SHA-256 over the six coordinates under it, so it moves when any of them moves, and it is not a function of the rows. The six do two different jobs:

• Three of them file the promotion. judge_id, metric and segment are the key the promotion registry files a promotion under. Two subjects sharing that key overwrite each other’s promotion, which is why the shadow judging path carries a judge id of its own rather than reusing a registered evaluator’s.
• Three of them pin it. model, prompt_sha and eval_ver compose the version string a promotion is pinned to. It is stored readable rather than hashed, so an operator asking why a judge went back to reported reads the answer off the record instead of diffing two digests.

So κ here is judge-versus-human agreement on one metric, in one segment, for one version of one judge, and there is no deployment-wide, judge-wide or page-level κ. The product publishes none, and a sentence like “the judge scores 0.68” has dropped the part that makes it checkable.

The corpus and the binary both call the protocol B.7. B.7 is the whole validation protocol rather than any one of its tests: admitting a gold set on inter-annotator agreement is one step of it, and the position-swap gate is another. Where a log line says “the B.7 gate” with no step named, it means the protocol, and this page names the step instead.

One vocabulary note before the walk. Read quality basis everywhere below, because a second enum in this product also spells reported and means something else. A quality estimate’s basis is reported, measured or insufficient; a judge’s outcomes carry quality basis reported until a promotion stands for their subject. LLM judges has the three-row table.

What a null means on a judges row

Several fields on a judges row come back null, and each null is a different claim. Reading one of them as a zero is the most expensive mistake available on this surface, so the table comes before the walk rather than after it.

Value on the reading	What it records	What it is not
iaa: null	inter-annotator agreement was not measured	agreement of zero, and not any one named cause
iaa: 0.8397	κ_h over the rows two annotators both labelled	agreement over the union of the two label sets
kappa: null	κ was not computed, because a step ahead of it stopped the run	agreement of zero
kappa: 0.0	a measurement: judge and humans agreed at chance level	a missing value
swap_flip: 0.0	a measurement, and the best result available on that axis	not measured
spend: null	no shipped producer sums a subject’s judging dollars	$0, or a judge that cost nothing
anchor: null	no who-drifted diagnosis is joined to this reading	that nothing drifted
selective_risk: null	no B.2 fit exists in the process that answered	that risk control looked and found nothing

iaa: null carries a further trap. Four situations produce it, and the reading does not say which:

Reason, in the CLI’s text output	What happened
single_annotator	one annotator covered the admitted rows
no_annotators	the gold set is empty
overlap_below_floor	two or more annotators, and no pair shares IAAOverlapMinN rows
undefined_on_overlap	a pair shares enough rows, and κ_h has a zero denominator on them

The producer records the reason. It reaches one surface: the text output of tokentriage evals validate, which prints unmeasured (<reason>) on the gold-set line. It is carried by no wire field and no dashboard panel, so a null iaa read over the API alone does not say which of the four applied. Run the verb and read its text mode when you need the cause.

The walk, in the order the binary walks it

Validation is not a score with a threshold on it. It is an ordered walk of four gates, and each gate can do one thing to the answer: stop the subject, and say which gate stopped it. Nothing in the walk raises a number, waives a floor, or grades a subject back in after an earlier gate turned it away.

Read the left column downward as one validation run over one subject. Each row is a gate; the right column is what that gate turns a subject away with, and the floor each code names. The row above the four gates is a precondition rather than a gate, and the band at the bottom is the outcome.

That precondition is the subject key itself, because a promotion filed under an incomplete key is one nothing can look up. You meet it as a counter rather than as a stop.

The assembler screens every subject against the same completeness check before a validation run is planned. A row whose subject does not name its judge id, judge model, prompt sha, metric and segment is counted into incomplete_subject on the skipped: line of the assembly stanza, and reaches no gate at all. The library keeps a subject_incomplete code for the case, with the two words the other way round, and this verb does not reach it.

1. The gold set is admitted, and annotator agreement is measured inside it. Every human-labelled row on the subject’s segment is offered, and rows carrying demo, synthetic, unattributed or superseded provenance are excluded and counted before anything else runs. κ_h is then measured on the overlap sample, the rows both annotators labelled, against IAAOverlapMinN = 20 shared rows and GoldSetIAAFloor = 0.6. This gate can stop the walk with gold_set_empty, iaa_below_floor, iaa_undefined, or iaa_unmeasured where evals.validation.require_iaa is on, and it can do nothing else to the answer.
2. The label events are counted. N is the admitted (annotator, request) pairs whose surviving label carries a judge verdict to pair against, after a last-write-wins dedupe, against JudgeValidationMinN = 50. This gate can stop the walk with validation_below_n, and it can do nothing else to the answer.
3. κ is computed and compared to its floor. Judge-versus-human Cohen’s κ over those N label events, unweighted on a binary rubric, against JudgeKappaFloor = 0.5, which evals.validation.min_kappa may raise and cannot lower. This gate can stop the walk with kappa_undefined, or record a kappa_below_floor and walk on, and it can do nothing else to the answer.
4. The swap set is measured, and gates only for pairwise use. The flip rate over pairs run in both presentation orderings, against SwapSetMinPairs = 30 complete pairs and SwapFlipCeiling = 0.10; a pointwise judge whose operator ran a swap set gets the disclosure without the gate. This gate can stop the walk with swap_pair_incomplete, swap_below_min_pairs or swap_flip_above_ceiling, and it can do nothing else to the answer.

The verdict is what falls out, and it is not a fifth gate. A subject is promoted where the four gates recorded nothing against it and refused where any of them did, and the registry stores that fact at the subject’s current version.

Gates 1 and 2 stop where they stand, and so does an undefined κ. A κ below its floor behaves differently: it is recorded and the walk continues into gate 4, so a single run can name more than one floor. Either way the verb exits 0, because a stop is an answer rather than a failure. The answer persists too: a stopped run writes a jval_ artifact in the same way a promotion does, and the record carries each floor’s name with a need and a have. That pair is what says whether more labels would change the answer.

One constant in those four gates moves, and it moves in one direction. evals.validation.min_kappa raises JudgeKappaFloor and cannot lower it, and the two layers that enforce it use different words: a config file setting it below 0.5 is turned away at tokentriage validate, while the library clamps an out-of-range value back to the frozen floor instead. Both readings are correct, and crossing them is a common misquote. The other five constants take no configuration at all.

What the walk writes down

Here is the walk’s own arithmetic, from the run that promoted one subject. The gold set had grown to 105 admitted label events over 80 requests, with 25 of those requests labelled by both annotators. Three elisions are marked in the block, and each one is a stanza this page does not need; Validate a judge and promote it carries the same run whole.

$ tokentriage evals validate --config … --state-dir … --log …
[…the gold-set assembly and promotion-registry stanzas elided…]
validation runs (1)
  [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67
      metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat
      reason: operator — operator-forced validation of 105 admitted label(s) at revision 93f8396
      gold: admitted=105 annotators=2 licenses=true iaa=0.8397 revision=93f8396
      kappa: 0.6761 over n=105 (p_o=0.8381 p_e=0.5001)
      floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1
      verdict: PROMOTED   registry change: promoted
      selective-risk audit: outcome=fitted alpha=0.05 audits=0 reuses=0 holdout_read=false
        REFUSED: selective_no_feature
          selective_no_feature: the method publishes no scalar confidence feature, so there is no
          quantity to threshold; risk-controlled selective judging needs one and has none
          (alpha=0.05, delta=0.05). Structural abstention is unaffected.
        […8 lines of B.2 sealed-sample detail elided…]
      reading: state=promoted gold_revision=93f8396 auto_validate=false subjects_observed=1
[…the trailing spend note and the jval_ artifact count elided…]

Read the conditions before you quote a number out of that block. The machinery is the shipped binary’s: the assembler, Cohen’s κ, the inter-annotator agreement, all six floors, the promotion registry and the jval_ artifact store are the product’s own, and they ran against a ledger the proxy minted. Two inputs are a fixture. The judge verdicts came from a loopback listener answering a canned score, and the two annotators’ labels were designed by the harness that drove the run. So κ = 0.6761 is what a promoted run looks like, and is not what any judge scores against any human.

Everything left in the block is the run’s own bytes. revision and gold_revision are digests over the request ids the proxy minted, and they move on any run anywhere. Every floor constant on the floors applied: line, and verdict: PROMOTED, hold against any box and any registry state. Exit status was 0, and the same run printed two warnings on stderr, one of them the judge’s egress disclosure.

Two things in that fragment are worth stopping on. The rest of the run — the assembly counters, the registry state, the jval_ store — is read out on Validate a judge and promote it.

• admitted=105 counts label events, not requests. The gold set covered 80 requests: one annotator labelled all 80 and the other labelled the first 25, which is 105 (annotator, request) pairs. JudgeValidationMinN is measured against those pairs, and iaa=0.8397 on the 25 requests both annotators covered.
• There is a refusal inside a promoted run, and it is not a B.7 floor. REFUSED: selective_no_feature is B.2’s selective-risk audit reporting that this method publishes no scalar confidence feature to threshold. Reading it as a seventh floor failing is the trap this fragment is worth pasting for, and Judge methods says which methods publish one.

What promotion changes, and what it does not

There is no verb that flips a judge to validated. A promotion is what a measurement leaves behind when no step in the walk had anything to hold against the subject. The registry records that fact; nothing in the CLI or the API sets it directly, and nothing rewrites an old row when it lands.

Read it left to right along one subject’s life. Three windows, split by two instants: before any promotion stands, from the promotion instant onward, and after a version move. The basis line in each window is what a row written inside that window carries.

Promotion is forward-only. The quality basis of a row is re-derived at read time against the promotion registry, using the row’s own six coordinates and the row’s own timestamp, and measured is reached only where a promotion for that subject stood at that instant. A row’s own validated: true stamp is not trusted on its own. So every judged row written before a promotion keeps quality basis reported and keeps it permanently. On the run above, every judge row on that ledger predates the promotion instant, and each one carries validated: false.

Promotion is version-scoped. The registry files the promotion under the three key coordinates and pins it to the three version coordinates. Change the judge’s model, its prompt, or the evaluator version, and the pin no longer matches: the promotion is reset, the subject’s state reads version_changed, and its scores ride the reported ceiling until it is validated again. A prompt edit costs the whole gold set’s worth of labels, which is worth knowing before you make one.

The state a reading carries has three values, and one of them is read wrong often enough to be worth a row of its own:

state	What it says	What it does not say
promoted	a promotion stands, and the subject’s version still matches the one that produced it	that rows written before it became measured
never_validated	no promotion stands for this subject	that no run happened; a run that stopped at a floor also reads never_validated, and carries last_run where a never-run subject does not
version_changed	a promotion exists and was pinned to a different model, prompt or evaluator version	that the old reading was discarded; it is still served, which is what lets the row say what was lost

Three things a promotion does not change.

It does not make spend measurable. Judging costs money per row, and no shipped producer sums those dollars onto a subject.

It does not lift every cap on your figure. At a read surface the quality basis is the lowest of four caps, and three of them consult the promotion registry, while the outcome-source cap yields to nothing. Which one is holding your figure down is Why your number is reported, not measured.

It does not start anything watching the judge afterwards. No shipped binary can fire a drift demotion, because the gate’s observation entry point (Gate.ObserveAgreement) has no call site outside its tests. Re-running this check on a schedule of your own is the substitute, and Judge drift, with no monitor between checks is that story.

Why κ travels without an interval

κ is served beside its validated_n and with no confidence interval, and that is a position this build holds rather than a scheduling accident.

validated_n counts label events, which are (annotator, request) pairs, and not independent draws. Two annotators on one request are two label events, and that request’s single judge verdict votes in both of them. A count like that is clustered, so an interval computed from it as though it were N independent observations would overstate its own precision, and a bound that overstates its precision is worse than no bound at all. So the reading carries the point estimate, the N it was computed over, and the observed and chance agreement rates (p_o and p_e) that produced it, which is enough to recompute κ by hand.

A reader who wants a bound does have one, on a different quantity: the calibration track fits a lower bound on a rate, not on agreement, and the two are not interchangeable. See How calibration works.

What this page does not cover

The swap gate, on a judge that does not run pairwise. The flip-rate arithmetic is measured whenever swap pairs are supplied and gates only for pairwise use, so a pointwise-bool judge prints those two floors and reaches neither. What the gate is for, and what a flip means, is Position bias, self-preference and panels.

One of the two registered judge ids cannot be carried through this protocol at all. judge-groundedness publishes no pass floor, so the runner leaves every row it grades without a verdict, and the assembler records a subject’s verdict only where one is set. A judge-groundedness subject therefore accumulates none, and its run stops at the first gate with gold_set_empty. Use judge-task-completion, whose metric carries a floor.

Where the labels come from. Two annotators, an overlapping sample, and the door they arrive through are Collect the labels a judge is measured against. The overlap is the expensive part, and it is the one this walk measures agreement on.

Running the check, and reading a stopped run. The verb, its flags, and the floor a stopped run names are Validate a judge and promote it.