# Validate a judge and promote it Run the validation check, read the answer whether it promotes or declines, and find the floor that a declined run names. Source: https://tokentriage.com/docs/judges/validate-and-promote --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; Your annotators have labelled a few hundred rows by hand, and the model that graded the same traffic is sitting on a screen with a score nobody has checked against them. One command settles whether those two agree closely enough for the score to be allowed near a decision about money. It answers in one of two ways. It promotes the subject, or it stops and names the floor it fell short of, with the number that floor wanted beside the number it had. Both outcomes are answers, both persist as records, and both exit 0. The second one happens more often and is worth more to you, because it says what to go and collect. This page runs the check and reads both answers off captured output. What each gate measures, and why Cohen's kappa is the statistic this build promotes on, is [the walk that promotes a judge](/docs/judges/agreement). Getting human labels onto rows in the first place is [Collect the labels a judge is measured against](/docs/judges/collect-labels). Start here once you have labels. ## Run the check One verb does the whole protocol. It assembles a gold set from the ledger, walks every judge subject that gold set supports, and writes a record for each. ```console $ tokentriage evals validate \ --config ./tokentriage.yaml \ --state-dir ./state/dash \ --log ./state/decisions.jsonl ``` `--config` supplies the other two: with a config file present, `--log` falls back to `ledger.decision_log.path` before the compiled-in default. Pass none of the three and the verb reads `~/.tokentriage/decisions.jsonl` and `~/.tokentriage/dash`. What the run needs, and what it does not: | What | Needed? | |---|---| | human labels on ledger rows, from two annotators wherever you want an agreement figure | yes | | a judge evaluator whose metric carries a pass floor, such as `judge-task-completion` | yes | | the decision log holding the judged rows, and the state directory holding the registry | yes | | a running server | no, this is a command-line verb | | `evals.validation.auto_validate.enabled` set to `true` | no, an operator run does not consult it | | outbound network access, or a judge credential | no, the run reads the ledger and makes no grading call | That last row is the run's own claim about itself. The note under its final stanza says no path this loop executes reaches a judge call, which is also why `spend` comes back as nil rather than as `$0`. Asking by hand is always allowed: the automatic loop is the server's, driven from boot, and this process is an operator run either way. Two flags belong to the verb, and that is the whole set. | Flag | What it does | |---|---| | `--json` | prints the runs as the wire shapes the judges resource serves, instead of the text report | | `--now` | stamps the records, the artifact and the recal-log entries with an RFC3339 instant you supply; absent reads as now, in UTC | A `--now` value that is not RFC3339 stops the run before it starts, and so does one at or before the newest artifact already in the store.
```console $ tokentriage evals validate --now 2026-08-24T00:00:00Z \ --state-dir D:/tt-jg-scratch/hwork-w24/run/state/dash \ --log D:/tt-jg-scratch/hwork-w24/run/state/decisions.jsonl Error: evals: refused: --now 2026-08-24T00:00:00Z is at or before the newest artifact in D:\tt-jg-scratch\hwork-w24\run\state\dash\quality (jval_20260824T1914_fe7ef520, created 2026-08-24T19:14:53.715Z). A run stamped there writes an artifact the next boot sorts UNDER that one, so this run's promotions and demotions would be silently reverted at restart while its recal-log entries stood. Use an instant after 2026-08-24T19:14:53.715Z, or a fresh state dir Error: evals: refused: --now 2026-08-24T00:00:00Z is at or before the newest artifact in D:\tt-jg-scratch\hwork-w24\run\state\dash\quality (jval_20260824T1914_fe7ef520, created 2026-08-24T19:14:53.715Z). A run stamped there writes an artifact the next boot sorts UNDER that one, so this run's promotions and demotions would be silently reverted at restart while its recal-log entries stood. Use an instant after 2026-08-24T19:14:53.715Z, or a fresh state dir ``` Exit status 1, and stdout is empty. The message prints twice on stderr because this binary prints every error once through its command framework and once from `main`, so a capture showing one copy has been edited. The path and the artifact id are that run's; the sentence is not.
`tokentriage evals judges` reads like the verb and is not one. The `evals` group defines no such command, so it prints the group's own help and exits 0, which means a scripted call to it passes CI while validating nothing. The verb is `tokentriage evals validate`; the group's other three commands are `compare`, `datasets` and `run`. ## Read a run that declines This is the answer you get first, most times. Forty admitted label events against a floor of fifty. ```console $ tokentriage evals validate --config … --state-dir … --log … gold-set assembly — D:/tt-jg-scratch/hwork2/run/state/decisions.jsonl sources: outcomes sets: 1 segments: 1 torn lines: 0 unorderable req_ts: 0 task_completion | task_completion|tier=frontier|model=gpt-5|endpoint=chat offered=40 admitted=40 (outcomes=40 feedback=0) requests=30 subjects=1 annotators=[annotator-a annotator-b] excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0 skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0 feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0 promotion registry — D:\tt-jg-scratch\hwork2\run\state\dash\quality validation_state: empty auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25 this CLI process does not drive the automatic loop; the SERVER does, from boot (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`. validation runs (1) [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67 metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat reason: operator — operator-forced validation of 40 admitted label(s) at revision 8b39113 gold: admitted=40 annotators=2 licenses=true iaa=unmeasured (overlap_below_floor) revision=8b39113 kappa: NOT COMPUTED — a gate before it refused (the refusals below name which); validated_n=40 floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1 verdict: REFUSED registry change: refused refusal validation_below_n: floor=JudgeValidationMinN need=50 have=40 admitted human-labeled rows validation_below_n: 40 admitted human-labeled rows; JudgeValidationMinN (50) requires 50. judge judge-task-completion (prompt 1fc3923) on task_completion|tier=frontier|model=gpt-5|endpoint=chat: 40 of the 40 admitted gold rows carry a judge verdict to pair against. The judge stays `reported` — no κ is computed and none is reported as 0 selective-risk audit: outcome=fitted alpha=0.05 audits=0 reuses=0 holdout_read=false REFUSED: selective_no_feature selective_no_feature: the method publishes no scalar confidence feature, so there is no quantity to threshold; risk-controlled selective judging needs one and has none (alpha=0.05, delta=0.05). Structural abstention is unaffected. […8 lines of B.2 sealed-sample detail elided…] reading: state=never_validated gold_revision=8b39113 auto_validate=false subjects_observed=1 (`spend` is NOT MEASURED and is reported as nil, not as $0: this loop cannot spend — no path it executes reaches a judge call, which JG4-06 restated from judgegate's import posture after an injected re-scorer seam made that argument unsound — and while a shadow row now records the judge id and metric its judging dollars belong to, no shipped producer SUMS those dollars onto a subject.) jval_ artifacts in the store: 1 newest: jval_20260824T1807_b950c1e1 ``` **Exit status 0**, and the same run wrote two warnings to stderr, one of them the judge's egress disclosure. That capture is from a source build on Windows driven by the documentation harness, so a release prints its own version where that build printed `dev`, and both paths, `revision`, `gold_revision`, the sealed-sample digest and the `jval_` id move on every run. Eight lines of sealed-sample accounting are elided where the block says so. The rest is the run's own bytes. The harness designed the human labels and answered the judge calls from a loopback stub, so `admitted=40` and `requests=30` are a fixture's counts; every floor, the code and the arithmetic around them are the binary's. Six lines carry the answer, and they read in this order. - **`verdict: REFUSED registry change: refused`** is the outcome, and `registry change` says the record went to the promotion registry rather than nowhere. - **`refusal validation_below_n: floor=JudgeValidationMinN need=50 have=40`** names the floor with what it wanted beside what it had. Ten more admitted label events change this answer. Nothing else on this run does. - **`gold: admitted=40 annotators=2 licenses=true iaa=unmeasured (overlap_below_floor)`**. `licenses=true` is a statement about the gold set being admissible as evidence, not about the judge, and it sits on the same line as a run that declined. `overlap_below_floor` is the reason agreement between the two annotators was not measured: they shared ten requests against a floor of twenty. That reason reaches this text output and no wire field, so the same run read back over the API says `iaa` is null and does not say why. - **`kappa: NOT COMPUTED — a gate before it refused`**, and the long form under it says the same thing again: no kappa is computed and none is reported as 0. A kappa of `0.0` would be a measurement, meaning agreement at the rate two coin flips would reach. - **`reading: state=never_validated`**. Not `demoted`, and not `failed`. A subject that has been through a run that declined reads the same state as one nobody has run, and is told apart by carrying a `last_run`. - **`jval_ artifacts in the store: 1`**, on a store that held none before this. A declined run persists the way a promotion does, which is what lets you diff two weeks apart. One line in that block is not one of the protocol's floors. `REFUSED: selective_no_feature`, under `selective-risk audit:`, belongs to a different appendix, and it reports that a `pointwise-bool` judge publishes no scalar confidence feature to threshold. It prints on promoted runs too. Reading it as a seventh floor failing is the mistake this block is worth pasting for. ## Read a run that promotes Same subject, same judge, same ledger, sixty-five more label events later. The gold set had grown to 105 admitted label events over 80 requests, with 25 of those requests labelled by both annotators. ```console $ tokentriage evals validate --config … --state-dir … --log … gold-set assembly — D:/tt-jg-scratch/hwork2/run/state/decisions.jsonl sources: outcomes sets: 1 segments: 1 torn lines: 0 unorderable req_ts: 0 task_completion | task_completion|tier=frontier|model=gpt-5|endpoint=chat offered=105 admitted=105 (outcomes=105 feedback=0) requests=80 subjects=1 annotators=[annotator-a annotator-b] excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0 skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0 feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0 promotion registry — D:\tt-jg-scratch\hwork2\run\state\dash\quality validation_state: empty auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25 this CLI process does not drive the automatic loop; the SERVER does, from boot (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`. validation runs (1) [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67 metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat reason: operator — operator-forced validation of 105 admitted label(s) at revision 93f8396 gold: admitted=105 annotators=2 licenses=true iaa=0.8397 revision=93f8396 kappa: 0.6761 over n=105 (p_o=0.8381 p_e=0.5001) floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1 verdict: PROMOTED registry change: promoted […13 lines of B.2 selective-risk audit elided…] reading: state=promoted gold_revision=93f8396 auto_validate=false subjects_observed=1 (`spend` is NOT MEASURED and is reported as nil, not as $0: this loop cannot spend — no path it executes reaches a judge call, which JG4-06 restated from judgegate's import posture after an injected re-scorer seam made that argument unsound — and while a shadow row now records the judge id and metric its judging dollars belong to, no shipped producer SUMS those dollars onto a subject.) jval_ artifacts in the store: 1 newest: jval_20260824T1808_40890fe5 ``` **Read the conditions before you quote a number from that block.** The machinery is the shipped binary's end to end: the assembler, Cohen's kappa, the inter-annotator agreement, all six floors, the promotion registry and the `jval_` artifact store are the product's own, and they ran against a ledger the proxy minted. Two inputs are a fixture. The judge verdicts came from a loopback listener answering a canned score, and the two annotators' labels were designed by the harness that drove the run. **So the kappa of 0.6761 is what a promoted run looks like, and is not what any judge scores** against any human. Quote the shape, not the value. Thirteen lines are elided and marked in the block. They are the same selective-risk paragraph the declined run printed, differing in a digest and two counters. Everything else is the run's own bytes, under the same conditions as the capture above. One of those conditions is not on the machine. Both runs started against a promotion registry holding nothing, which is why `validation_state` reads `empty` on each and the artifact count starts at zero. Run this over a state directory with history and the same two lines read `restored` and count up, with nothing else in either block moving. Five lines carry this answer. - **`verdict: PROMOTED registry change: promoted`**, and a promotion now stands for this subject's current version. - **`kappa: 0.6761 over n=105 (p_o=0.8381 p_e=0.5001)`**. Kappa travels with the N it was computed over and with the observed and chance agreement rates behind it, and with no confidence interval. - **`gold: admitted=105 annotators=2 licenses=true iaa=0.8397`**. That figure was measured on the 25 requests both annotators labelled, against a floor of 20 shared rows, and not on the union of the two label sets. - **`floors applied:`** prints all six on a pass. Two of them gate pairwise use, and a `pointwise-bool` judge runs no both-orderings trial, so this run applied `SwapSetMinPairs` and `SwapFlipCeiling` and reached neither. The line says what the protocol applies, not what your cohort exercised. - **`reading: state=promoted`**, beside a `spend` that is still nil. Promotion does not make judging spend measurable; judging costs money per row, and no shipped producer sums those dollars onto a subject. A declined run and a promoted run are the same kind of record. Both write a `jval_` artifact, both exit 0, both print the same six floors, and both are readable back months later. The one that declined additionally names what it needed and what it had, which is the part you can act on. ## One run can name more than one floor "It stopped at a gate" is true and singular, and the walk is not. Gates one and two stop where they stand, and so does a kappa that cannot be computed at all. A kappa below its floor behaves differently: the record keeps it and the walk carries on into the swap gate, so a single run can print two floor names under one subject. So read every `refusal` line on the record rather than the first. `floors applied:` is the ladder the protocol ran; the lines under it are the rungs that were not cleared, and there can be more than one. ## Two numbers that send a reader the wrong way Both are accurate about what they count and quiet about why, which is the combination that costs a week. ### A label count that will not move **`JudgeValidationMinN` need 50, have 0, on a ledger holding judged rows.** Read the evaluator id on the record before you go and collect more labels. `judge-groundedness` writes a scalar score and no pass verdict, and the gold-set assembler records only rows that carry one, so its subject is counted into the census, listed on the reading and given a run of its own while the count stays at zero however many labels arrive. Configure `judge-task-completion` on the metric `task_completion` for any judge that has to reach a promotion. `judge-groundedness` still produces a score you can read as `reported`. ### A count of rows that left before any gate `incomplete_subject` sits on the `skipped:` line of the assembly stanza, and both captures above read it at zero. A non-zero value there counts judged rows whose subject key did not name enough to file a promotion under, and the assembler drops them before the walk starts. No gate sees them, so no `refusal` line mentions them and no floor moves when you collect more labels. That line is the only place the run reports them. Enough means five coordinates: the judge id, the judge model, the prompt sha, the metric and the segment. A promotion is scoped by six, and `eval_ver` is the one this check does not read. The source records the omission without saying whether it was intended, and one shipped path depends on it: a shadow-plane grade projects its subject with an empty `eval_ver`, and that subject is admitted. The six coordinates a promotion is filed and pinned by are on [the walk that promotes a judge](/docs/judges/agreement). The protocol's vocabulary carries a `subject_incomplete` code for the same condition, applied as a precondition inside the validation routine. The one caller that reaches that routine is handed subjects the assembler has already filtered, so on this verb you meet the counter and not the code. ## Promotion is not a command No verb flips a judge to validated. A promotion is what the walk leaves behind when no gate had anything to hold against the subject, and the registry records that fact. Nothing on the command line and nothing in the API sets it. Two things follow for what you do next. **Re-read a later window, not the one you were looking at.** A row's quality basis is derived at read time against the promotion registry, using the row's own timestamp, so every judged row written before the promotion instant keeps its `reported` basis and keeps it permanently. Nothing rewrites them. Running the same report over the same window after a promotion and finding no movement is the most common reason an operator concludes the promotion failed. **Treat a prompt edit as a re-validation.** The promotion is pinned to the judge's model, its prompt sha and its evaluator version. Move any of the three and the pin stops matching, the subject reads `version_changed`, and its scores ride the `reported` ceiling until you run this check again over a fresh gold set. ## What this page does not cover **The run promoted and your figure did not move.** A quality basis is the lowest of several caps and only some of them consult the promotion registry, so a promotion can land and change nothing you are looking at. [Why your number is reported, not measured](/docs/judges/basis-ceilings) finds the cap that binds. **A symptom you cannot place.** Symptom, cause and fix for each answer this build prints, in one table, is [Why didn't my judge validate?](/docs/judges/troubleshooting). **What each gate measures, and why kappa.** The gold set, the overlap sample, the kappa floor and the swap gate are [the walk that promotes a judge](/docs/judges/agreement). This page runs that walk rather than explaining it. **Turning a promotion into a routing decision.** A quality floor reads a `measured` basis only, so a standing promotion is the precondition and not the mechanism. [Gate a rollout on a validated judge](/docs/judges/gate-a-rollout) is the mechanism.