# How a judge reaches a measured basis Cohen's kappa, annotator agreement and the swap gate, walked in the order the binary walks them, and what promotion changes afterwards. Source: https://tokentriage.com/docs/judges/agreement --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import JudgeGateWalkDiagram from '../../../components/JudgeGateWalkDiagram.astro'; import JudgePromotionTimelineDiagram from '../../../components/JudgePromotionTimelineDiagram.astro'; A model grading another model's work hands you a number, and nothing in the number tells you whether a person would have said the same thing. Until somebody checks, that score is one machine's opinion of another machine's output: good enough to rank two prompts against each other, thin ground for a decision about money. Checking it is measurable work. Label a sample of the same traffic by hand, count how often the judge and the people landed on the same answer, then take out the agreement two coin flips would have produced anyway. What is left is **Cohen's kappa (κ)**, and κ is the statistic this build promotes on. This page walks that check in the order the binary walks it, names the floor and the constant each gate applies, and then says what a promotion changes for the figures you read afterwards. The last part is smaller than most readers expect, and it is the half that gets misremembered. If you have not seen a graded row yet, [How a judge call happens](/docs/judges/how-judging-works) comes first. Getting human labels onto rows is [Collect the labels a judge is measured against](/docs/judges/collect-labels), and running the check is [Validate a judge and promote it](/docs/judges/validate-and-promote). This page is the model under both, and there is no procedure on it. ## A promotion belongs to a subject, not to a judge The sentence everybody reaches for first is "our judge is validated", and this build cannot check that sentence, because it does not promote judges. It promotes **subjects**. A subject is a six-coordinate address, and every statistic below is scoped to one. ```json { "id": "1d7a361adad5", "judge_id": "judge-task-completion", "model": "claude-haiku-4-5", "prompt_sha": "1fc3923", "eval_ver": "judge-v1+cfg15acf67", "metric": "task_completion", "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat" } ``` That is the `subject` block off `GET /tt/api/v1/evals/judges`, captured from a booted instance. The `id` is a truncated SHA-256 over the six coordinates under it, so it moves when any of them moves, and it is not a function of the rows. The six do two different jobs: - **Three of them file the promotion.** `judge_id`, `metric` and `segment` are the key the promotion registry files a promotion under. Two subjects sharing that key overwrite each other's promotion, which is why the shadow judging path carries a judge id of its own rather than reusing a registered evaluator's. - **Three of them pin it.** `model`, `prompt_sha` and `eval_ver` compose the version string a promotion is pinned to. It is stored readable rather than hashed, so an operator asking why a judge went back to `reported` reads the answer off the record instead of diffing two digests. So κ here is judge-versus-human agreement on **one metric, in one segment, for one version of one judge**, and there is no deployment-wide, judge-wide or page-level κ. The product publishes none, and a sentence like "the judge scores 0.68" has dropped the part that makes it checkable. The corpus and the binary both call the protocol **B.7**. B.7 is the whole validation protocol rather than any one of its tests: admitting a gold set on inter-annotator agreement is one step of it, and the position-swap gate is another. Where a log line says "the B.7 gate" with no step named, it means the protocol, and this page names the step instead. One vocabulary note before the walk. Read **quality basis** everywhere below, because a second enum in this product also spells `reported` and means something else. A quality estimate's basis is `reported`, `measured` or `insufficient`; a judge's outcomes carry quality basis `reported` until a promotion stands for their subject. [LLM judges](/docs/judges/overview) has the three-row table. ## What a null means on a judges row Several fields on a judges row come back null, and each null is a different claim. Reading one of them as a zero is the most expensive mistake available on this surface, so the table comes before the walk rather than after it. | Value on the reading | What it records | What it is not | |---|---|---| | `iaa: null` | inter-annotator agreement was not measured | agreement of zero, and not any one named cause | | `iaa: 0.8397` | κ_h over the rows two annotators both labelled | agreement over the union of the two label sets | | `kappa: null` | κ was not computed, because a step ahead of it stopped the run | agreement of zero | | `kappa: 0.0` | a measurement: judge and humans agreed at chance level | a missing value | | `swap_flip: 0.0` | a measurement, and the best result available on that axis | not measured | | `spend: null` | no shipped producer sums a subject's judging dollars | `$0`, or a judge that cost nothing | | `anchor: null` | no who-drifted diagnosis is joined to this reading | that nothing drifted | | `selective_risk: null` | no B.2 fit exists in the process that answered | that risk control looked and found nothing | A κ of `0.0` is a measurement. It says the judge and the people agreed at the rate two independent coin flips would agree, which is a finding about the judge and a bad one. `kappa: null` is the other thing: no κ was computed at all, because a step before it stopped the run. Both fields are pointers that are always present on the wire, so the two cases cannot serialize identically. `iaa: null` carries a further trap. Four situations produce it, and the reading does not say which: | Reason, in the CLI's text output | What happened | |---|---| | `single_annotator` | one annotator covered the admitted rows | | `no_annotators` | the gold set is empty | | `overlap_below_floor` | two or more annotators, and no pair shares `IAAOverlapMinN` rows | | `undefined_on_overlap` | a pair shares enough rows, and κ_h has a zero denominator on them | The producer records the reason. It reaches one surface: the text output of `tokentriage evals validate`, which prints `unmeasured ()` on the gold-set line. It is carried by no wire field and no dashboard panel, so a null `iaa` read over the API alone does not say which of the four applied. Run the verb and read its text mode when you need the cause. The dashboard's Judges screen states one of those four causes for all four. It renders the literal string `IAA not measured (single-annotator gold set)` whenever `iaa` is null, which is the wrong cause three times in four. The running binary is the authority, and its text output names the cause it recorded. The discrepancy is filed. ## The walk, in the order the binary walks it Validation is not a score with a threshold on it. It is an ordered walk of four gates, and each gate can do one thing to the answer: stop the subject, and say which gate stopped it. Nothing in the walk raises a number, waives a floor, or grades a subject back in after an earlier gate turned it away. Read the left column downward as one validation run over one subject. Each row is a gate; the right column is what that gate turns a subject away with, and the floor each code names. The row above the four gates is a precondition rather than a gate, and the band at the bottom is the outcome. That precondition is the subject key itself, because a promotion filed under an incomplete key is one nothing can look up. You meet it as a counter rather than as a stop. The assembler screens every subject against the same completeness check before a validation run is planned. A row whose subject does not name its judge id, judge model, prompt sha, metric and segment is counted into `incomplete_subject` on the `skipped:` line of the assembly stanza, and reaches no gate at all. The library keeps a `subject_incomplete` code for the case, with the two words the other way round, and this verb does not reach it. 1. **The gold set is admitted, and annotator agreement is measured inside it.** Every human-labelled row on the subject's segment is offered, and rows carrying demo, synthetic, unattributed or superseded provenance are excluded and counted before anything else runs. κ_h is then measured on the **overlap sample**, the rows both annotators labelled, against `IAAOverlapMinN` = 20 shared rows and `GoldSetIAAFloor` = 0.6. This gate can stop the walk with `gold_set_empty`, `iaa_below_floor`, `iaa_undefined`, or `iaa_unmeasured` where `evals.validation.require_iaa` is on, and it can do nothing else to the answer. 2. **The label events are counted.** N is the admitted (annotator, request) pairs whose surviving label carries a judge verdict to pair against, after a last-write-wins dedupe, against `JudgeValidationMinN` = 50. This gate can stop the walk with `validation_below_n`, and it can do nothing else to the answer. 3. **κ is computed and compared to its floor.** Judge-versus-human Cohen's κ over those N label events, unweighted on a binary rubric, against `JudgeKappaFloor` = 0.5, which `evals.validation.min_kappa` may raise and cannot lower. This gate can stop the walk with `kappa_undefined`, or record a `kappa_below_floor` and walk on, and it can do nothing else to the answer. 4. **The swap set is measured, and gates only for pairwise use.** The flip rate over pairs run in both presentation orderings, against `SwapSetMinPairs` = 30 complete pairs and `SwapFlipCeiling` = 0.10; a pointwise judge whose operator ran a swap set gets the disclosure without the gate. This gate can stop the walk with `swap_pair_incomplete`, `swap_below_min_pairs` or `swap_flip_above_ceiling`, and it can do nothing else to the answer. The verdict is what falls out, and it is not a fifth gate. A subject is promoted where the four gates recorded nothing against it and refused where any of them did, and the registry stores that fact at the subject's current version. Gates 1 and 2 stop where they stand, and so does an undefined κ. A κ below its floor behaves differently: it is recorded and the walk continues into gate 4, so a single run can name more than one floor. Either way the verb exits 0, because a stop is an answer rather than a failure. The answer persists too: a stopped run writes a `jval_` artifact in the same way a promotion does, and the record carries each floor's name with a `need` and a `have`. That pair is what says whether more labels would change the answer. One constant in those four gates moves, and it moves in one direction. `evals.validation.min_kappa` raises `JudgeKappaFloor` and cannot lower it, and the two layers that enforce it use different words: a config file setting it below 0.5 is turned away at `tokentriage validate`, while the library clamps an out-of-range value back to the frozen floor instead. Both readings are correct, and crossing them is a common misquote. The other five constants take no configuration at all. ## What the walk writes down Here is the walk's own arithmetic, from the run that promoted one subject. The gold set had grown to 105 admitted label events over 80 requests, with 25 of those requests labelled by both annotators. Three elisions are marked in the block, and each one is a stanza this page does not need; [Validate a judge and promote it](/docs/judges/validate-and-promote) carries the same run whole. ```console $ tokentriage evals validate --config … --state-dir … --log … […the gold-set assembly and promotion-registry stanzas elided…] validation runs (1) [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67 metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat reason: operator — operator-forced validation of 105 admitted label(s) at revision 93f8396 gold: admitted=105 annotators=2 licenses=true iaa=0.8397 revision=93f8396 kappa: 0.6761 over n=105 (p_o=0.8381 p_e=0.5001) floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1 verdict: PROMOTED registry change: promoted selective-risk audit: outcome=fitted alpha=0.05 audits=0 reuses=0 holdout_read=false REFUSED: selective_no_feature selective_no_feature: the method publishes no scalar confidence feature, so there is no quantity to threshold; risk-controlled selective judging needs one and has none (alpha=0.05, delta=0.05). Structural abstention is unaffected. […8 lines of B.2 sealed-sample detail elided…] reading: state=promoted gold_revision=93f8396 auto_validate=false subjects_observed=1 […the trailing spend note and the jval_ artifact count elided…] ``` **Read the conditions before you quote a number out of that block.** The machinery is the shipped binary's: the assembler, Cohen's κ, the inter-annotator agreement, all six floors, the promotion registry and the `jval_` artifact store are the product's own, and they ran against a ledger the proxy minted. Two inputs are a fixture. The judge verdicts came from a loopback listener answering a canned score, and the two annotators' labels were designed by the harness that drove the run. **So κ = 0.6761 is what a promoted run looks like, and is not what any judge scores** against any human. Everything left in the block is the run's own bytes. `revision` and `gold_revision` are digests over the request ids the proxy minted, and they move on any run anywhere. Every floor constant on the `floors applied:` line, and `verdict: PROMOTED`, hold against any box and any registry state. Exit status was 0, and the same run printed two warnings on stderr, one of them the judge's egress disclosure. Two things in that fragment are worth stopping on. The rest of the run — the assembly counters, the registry state, the `jval_` store — is read out on [Validate a judge and promote it](/docs/judges/validate-and-promote). - **`admitted=105` counts label events, not requests.** The gold set covered 80 requests: one annotator labelled all 80 and the other labelled the first 25, which is 105 (annotator, request) pairs. `JudgeValidationMinN` is measured against those pairs, and `iaa=0.8397` on the 25 requests both annotators covered. - **There is a refusal inside a promoted run, and it is not a B.7 floor.** `REFUSED: selective_no_feature` is B.2's selective-risk audit reporting that this method publishes no scalar confidence feature to threshold. Reading it as a seventh floor failing is the trap this fragment is worth pasting for, and [Judge methods](/docs/reference/judge-methods) says which methods publish one. ## What promotion changes, and what it does not There is no verb that flips a judge to validated. A promotion is what a measurement leaves behind when no step in the walk had anything to hold against the subject. The registry records that fact; nothing in the CLI or the API sets it directly, and nothing rewrites an old row when it lands. Read it left to right along one subject's life. Three windows, split by two instants: before any promotion stands, from the promotion instant onward, and after a version move. The basis line in each window is what a row written inside that window carries. **Promotion is forward-only.** The quality basis of a row is re-derived at read time against the promotion registry, using the row's own six coordinates and the row's own timestamp, and `measured` is reached only where a promotion for that subject stood at that instant. A row's own `validated: true` stamp is not trusted on its own. So every judged row written before a promotion keeps quality basis `reported` and keeps it permanently. On the run above, every judge row on that ledger predates the promotion instant, and each one carries `validated: false`. **Promotion is version-scoped.** The registry files the promotion under the three key coordinates and pins it to the three version coordinates. Change the judge's model, its prompt, or the evaluator version, and the pin no longer matches: the promotion is reset, the subject's state reads `version_changed`, and its scores ride the `reported` ceiling until it is validated again. A prompt edit costs the whole gold set's worth of labels, which is worth knowing before you make one. The state a reading carries has three values, and one of them is read wrong often enough to be worth a row of its own: | `state` | What it says | What it does not say | |---|---|---| | `promoted` | a promotion stands, and the subject's version still matches the one that produced it | that rows written before it became `measured` | | `never_validated` | no promotion stands for this subject | that no run happened; a run that stopped at a floor also reads `never_validated`, and carries `last_run` where a never-run subject does not | | `version_changed` | a promotion exists and was pinned to a different model, prompt or evaluator version | that the old reading was discarded; it is still served, which is what lets the row say what was lost | Three things a promotion does not change. **It does not make `spend` measurable.** Judging costs money per row, and no shipped producer sums those dollars onto a subject. **It does not lift every cap on your figure.** At a read surface the quality basis is the lowest of four caps, and three of them consult the promotion registry, while the outcome-source cap yields to nothing. Which one is holding your figure down is [Why your number is reported, not measured](/docs/judges/basis-ceilings). **It does not start anything watching the judge afterwards.** No shipped binary can fire a drift demotion, because the gate's observation entry point (`Gate.ObserveAgreement`) has no call site outside its tests. Re-running this check on a schedule of your own is the substitute, and [Judge drift, with no monitor between checks](/docs/judges/drift) is that story. ## Why κ travels without an interval κ is served beside its `validated_n` and with no confidence interval, and that is a position this build holds rather than a scheduling accident. `validated_n` counts label events, which are (annotator, request) pairs, and not independent draws. Two annotators on one request are two label events, and that request's single judge verdict votes in both of them. A count like that is clustered, so an interval computed from it as though it were N independent observations would overstate its own precision, and a bound that overstates its precision is worse than no bound at all. So the reading carries the point estimate, the N it was computed over, and the observed and chance agreement rates (`p_o` and `p_e`) that produced it, which is enough to recompute κ by hand. A reader who wants a bound does have one, on a different quantity: the calibration track fits a lower bound on a *rate*, not on agreement, and the two are not interchangeable. See [How calibration works](/docs/calibration/how-calibration-works). ## What this page does not cover **The swap gate, on a judge that does not run pairwise.** The flip-rate arithmetic is measured whenever swap pairs are supplied and gates only for pairwise use, so a `pointwise-bool` judge prints those two floors and reaches neither. What the gate is for, and what a flip means, is [Position bias, self-preference and panels](/docs/judges/bias-controls). **One of the two registered judge ids cannot be carried through this protocol at all.** `judge-groundedness` publishes no pass floor, so the runner leaves every row it grades without a verdict, and the assembler records a subject's verdict only where one is set. A `judge-groundedness` subject therefore accumulates none, and its run stops at the first gate with `gold_set_empty`. Use `judge-task-completion`, whose metric carries a floor. **Where the labels come from.** Two annotators, an overlapping sample, and the door they arrive through are [Collect the labels a judge is measured against](/docs/judges/collect-labels). The overlap is the expensive part, and it is the one this walk measures agreement on. **Running the check, and reading a stopped run.** The verb, its flags, and the floor a stopped run names are [Validate a judge and promote it](/docs/judges/validate-and-promote).