Validate a judge and promote it

Your annotators have labelled a few hundred rows by hand, and the model that graded the same traffic is sitting on a screen with a score nobody has checked against them. One command settles whether those two agree closely enough for the score to be allowed near a decision about money.

It answers in one of two ways. It promotes the subject, or it stops and names the floor it fell short of, with the number that floor wanted beside the number it had. Both outcomes are answers, both persist as records, and both exit 0. The second one happens more often and is worth more to you, because it says what to go and collect.

This page runs the check and reads both answers off captured output. What each gate measures, and why Cohen’s kappa is the statistic this build promotes on, is the walk that promotes a judge. Getting human labels onto rows in the first place is Collect the labels a judge is measured against. Start here once you have labels.

Run the check

One verb does the whole protocol. It assembles a gold set from the ledger, walks every judge subject that gold set supports, and writes a record for each.

$ tokentriage evals validate \
    --config ./tokentriage.yaml \
    --state-dir ./state/dash \
    --log ./state/decisions.jsonl

--config supplies the other two: with a config file present, --log falls back to ledger.decision_log.path before the compiled-in default. Pass none of the three and the verb reads ~/.tokentriage/decisions.jsonl and ~/.tokentriage/dash.

What the run needs, and what it does not:

What	Needed?
human labels on ledger rows, from two annotators wherever you want an agreement figure	yes
a judge evaluator whose metric carries a pass floor, such as `judge-task-completion`	yes
the decision log holding the judged rows, and the state directory holding the registry	yes
a running server	no, this is a command-line verb
`evals.validation.auto_validate.enabled` set to `true`	no, an operator run does not consult it
outbound network access, or a judge credential	no, the run reads the ledger and makes no grading call

That last row is the run’s own claim about itself. The note under its final stanza says no path this loop executes reaches a judge call, which is also why spend comes back as nil rather than as $0. Asking by hand is always allowed: the automatic loop is the server’s, driven from boot, and this process is an operator run either way.

Two flags belong to the verb, and that is the whole set.

Flag	What it does
`--json`	prints the runs as the wire shapes the judges resource serves, instead of the text report
`--now`	stamps the records, the artifact and the recal-log entries with an RFC3339 instant you supply; absent reads as now, in UTC

A --now value that is not RFC3339 stops the run before it starts, and so does one at or before the newest artifact already in the store.

What a backdated --now prints, and why the run is stopped

$ tokentriage evals validate --now 2026-08-24T00:00:00Z \
    --state-dir D:/tt-jg-scratch/hwork-w24/run/state/dash \
    --log D:/tt-jg-scratch/hwork-w24/run/state/decisions.jsonl
Error: evals: refused: --now 2026-08-24T00:00:00Z is at or before the newest artifact in D:\tt-jg-scratch\hwork-w24\run\state\dash\quality (jval_20260824T1914_fe7ef520, created 2026-08-24T19:14:53.715Z). A run stamped there writes an artifact the next boot sorts UNDER that one, so this run's promotions and demotions would be silently reverted at restart while its recal-log entries stood. Use an instant after 2026-08-24T19:14:53.715Z, or a fresh state dir
Error: evals: refused: --now 2026-08-24T00:00:00Z is at or before the newest artifact in D:\tt-jg-scratch\hwork-w24\run\state\dash\quality (jval_20260824T1914_fe7ef520, created 2026-08-24T19:14:53.715Z). A run stamped there writes an artifact the next boot sorts UNDER that one, so this run's promotions and demotions would be silently reverted at restart while its recal-log entries stood. Use an instant after 2026-08-24T19:14:53.715Z, or a fresh state dir

Exit status 1, and stdout is empty. The message prints twice on stderr because this binary prints every error once through its command framework and once from main, so a capture showing one copy has been edited. The path and the artifact id are that run’s; the sentence is not.

Read a run that declines

This is the answer you get first, most times. Forty admitted label events against a floor of fifty.

$ tokentriage evals validate --config … --state-dir … --log …
gold-set assembly — D:/tt-jg-scratch/hwork2/run/state/decisions.jsonl
  sources: outcomes   sets: 1   segments: 1   torn lines: 0   unorderable req_ts: 0
  task_completion | task_completion|tier=frontier|model=gpt-5|endpoint=chat
      offered=40 admitted=40 (outcomes=40 feedback=0) requests=30 subjects=1 annotators=[annotator-a annotator-b]
      excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0
  skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0
  feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0

promotion registry — D:\tt-jg-scratch\hwork2\run\state\dash\quality
  validation_state: empty
  auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25
    this CLI process does not drive the automatic loop; the SERVER does, from boot
    (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`.

validation runs (1)
  [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67
      metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat
      reason: operator — operator-forced validation of 40 admitted label(s) at revision 8b39113
      gold: admitted=40 annotators=2 licenses=true iaa=unmeasured (overlap_below_floor) revision=8b39113
      kappa: NOT COMPUTED — a gate before it refused (the refusals below name which); validated_n=40
      floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1
      verdict: REFUSED   registry change: refused
      refusal validation_below_n: floor=JudgeValidationMinN need=50 have=40 admitted human-labeled rows
        validation_below_n: 40 admitted human-labeled rows; JudgeValidationMinN (50) requires
        50. judge judge-task-completion (prompt 1fc3923) on
        task_completion|tier=frontier|model=gpt-5|endpoint=chat: 40 of the 40 admitted gold rows
        carry a judge verdict to pair against. The judge stays `reported` — no κ is computed
        and none is reported as 0
      selective-risk audit: outcome=fitted alpha=0.05 audits=0 reuses=0 holdout_read=false
        REFUSED: selective_no_feature
          selective_no_feature: the method publishes no scalar confidence feature, so there is no
          quantity to threshold; risk-controlled selective judging needs one and has none
          (alpha=0.05, delta=0.05). Structural abstention is unaffected.
        […8 lines of B.2 sealed-sample detail elided…]
      reading: state=never_validated gold_revision=8b39113 auto_validate=false subjects_observed=1
  (`spend` is NOT MEASURED and is reported as nil, not as $0: this loop cannot
  spend — no path it executes reaches a judge call, which JG4-06 restated from judgegate's
  import posture after an injected re-scorer seam made that argument unsound — and while a
  shadow row now records the judge id and metric its judging dollars belong to, no shipped producer SUMS those dollars onto a subject.)

jval_ artifacts in the store: 1   newest: jval_20260824T1807_b950c1e1

Exit status 0, and the same run wrote two warnings to stderr, one of them the judge’s egress disclosure. That capture is from a source build on Windows driven by the documentation harness, so a release prints its own version where that build printed dev, and both paths, revision, gold_revision, the sealed-sample digest and the jval_ id move on every run.

Eight lines of sealed-sample accounting are elided where the block says so. The rest is the run’s own bytes. The harness designed the human labels and answered the judge calls from a loopback stub, so admitted=40 and requests=30 are a fixture’s counts; every floor, the code and the arithmetic around them are the binary’s.

Six lines carry the answer, and they read in this order.

verdict: REFUSED registry change: refused is the outcome, and registry change says the record went to the promotion registry rather than nowhere.
refusal validation_below_n: floor=JudgeValidationMinN need=50 have=40 names the floor with what it wanted beside what it had. Ten more admitted label events change this answer. Nothing else on this run does.
gold: admitted=40 annotators=2 licenses=true iaa=unmeasured (overlap_below_floor). licenses=true is a statement about the gold set being admissible as evidence, not about the judge, and it sits on the same line as a run that declined. overlap_below_floor is the reason agreement between the two annotators was not measured: they shared ten requests against a floor of twenty. That reason reaches this text output and no wire field, so the same run read back over the API says iaa is null and does not say why.
kappa: NOT COMPUTED — a gate before it refused, and the long form under it says the same thing again: no kappa is computed and none is reported as 0. A kappa of 0.0 would be a measurement, meaning agreement at the rate two coin flips would reach.
reading: state=never_validated. Not demoted, and not failed. A subject that has been through a run that declined reads the same state as one nobody has run, and is told apart by carrying a last_run.
jval_ artifacts in the store: 1, on a store that held none before this. A declined run persists the way a promotion does, which is what lets you diff two weeks apart.

One line in that block is not one of the protocol’s floors. REFUSED: selective_no_feature, under selective-risk audit:, belongs to a different appendix, and it reports that a pointwise-bool judge publishes no scalar confidence feature to threshold. It prints on promoted runs too. Reading it as a seventh floor failing is the mistake this block is worth pasting for.

Read a run that promotes

Same subject, same judge, same ledger, sixty-five more label events later. The gold set had grown to 105 admitted label events over 80 requests, with 25 of those requests labelled by both annotators.

$ tokentriage evals validate --config … --state-dir … --log …
gold-set assembly — D:/tt-jg-scratch/hwork2/run/state/decisions.jsonl
  sources: outcomes   sets: 1   segments: 1   torn lines: 0   unorderable req_ts: 0
  task_completion | task_completion|tier=frontier|model=gpt-5|endpoint=chat
      offered=105 admitted=105 (outcomes=105 feedback=0) requests=80 subjects=1 annotators=[annotator-a annotator-b]
      excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0
  skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0
  feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0

promotion registry — D:\tt-jg-scratch\hwork2\run\state\dash\quality
  validation_state: empty
  auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25
    this CLI process does not drive the automatic loop; the SERVER does, from boot
    (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`.

validation runs (1)
  [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67
      metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat
      reason: operator — operator-forced validation of 105 admitted label(s) at revision 93f8396
      gold: admitted=105 annotators=2 licenses=true iaa=0.8397 revision=93f8396
      kappa: 0.6761 over n=105 (p_o=0.8381 p_e=0.5001)
      floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1
      verdict: PROMOTED   registry change: promoted
      […13 lines of B.2 selective-risk audit elided…]
      reading: state=promoted gold_revision=93f8396 auto_validate=false subjects_observed=1
  (`spend` is NOT MEASURED and is reported as nil, not as $0: this loop cannot
  spend — no path it executes reaches a judge call, which JG4-06 restated from judgegate's
  import posture after an injected re-scorer seam made that argument unsound — and while a
  shadow row now records the judge id and metric its judging dollars belong to, no shipped producer SUMS those dollars onto a subject.)

jval_ artifacts in the store: 1   newest: jval_20260824T1808_40890fe5

Read the conditions before you quote a number from that block. The machinery is the shipped binary’s end to end: the assembler, Cohen’s kappa, the inter-annotator agreement, all six floors, the promotion registry and the jval_ artifact store are the product’s own, and they ran against a ledger the proxy minted. Two inputs are a fixture. The judge verdicts came from a loopback listener answering a canned score, and the two annotators’ labels were designed by the harness that drove the run. So the kappa of 0.6761 is what a promoted run looks like, and is not what any judge scores against any human. Quote the shape, not the value.

Thirteen lines are elided and marked in the block. They are the same selective-risk paragraph the declined run printed, differing in a digest and two counters. Everything else is the run’s own bytes, under the same conditions as the capture above.

One of those conditions is not on the machine. Both runs started against a promotion registry holding nothing, which is why validation_state reads empty on each and the artifact count starts at zero. Run this over a state directory with history and the same two lines read restored and count up, with nothing else in either block moving.

Five lines carry this answer.

verdict: PROMOTED registry change: promoted, and a promotion now stands for this subject’s current version.
kappa: 0.6761 over n=105 (p_o=0.8381 p_e=0.5001). Kappa travels with the N it was computed over and with the observed and chance agreement rates behind it, and with no confidence interval.
gold: admitted=105 annotators=2 licenses=true iaa=0.8397. That figure was measured on the 25 requests both annotators labelled, against a floor of 20 shared rows, and not on the union of the two label sets.
floors applied: prints all six on a pass. Two of them gate pairwise use, and a pointwise-bool judge runs no both-orderings trial, so this run applied SwapSetMinPairs and SwapFlipCeiling and reached neither. The line says what the protocol applies, not what your cohort exercised.
reading: state=promoted, beside a spend that is still nil. Promotion does not make judging spend measurable; judging costs money per row, and no shipped producer sums those dollars onto a subject.

One run can name more than one floor

“It stopped at a gate” is true and singular, and the walk is not. Gates one and two stop where they stand, and so does a kappa that cannot be computed at all. A kappa below its floor behaves differently: the record keeps it and the walk carries on into the swap gate, so a single run can print two floor names under one subject.

So read every refusal line on the record rather than the first. floors applied: is the ladder the protocol ran; the lines under it are the rungs that were not cleared, and there can be more than one.

Two numbers that send a reader the wrong way

Both are accurate about what they count and quiet about why, which is the combination that costs a week.

A label count that will not move

A count of rows that left before any gate

incomplete_subject sits on the skipped: line of the assembly stanza, and both captures above read it at zero. A non-zero value there counts judged rows whose subject key did not name enough to file a promotion under, and the assembler drops them before the walk starts. No gate sees them, so no refusal line mentions them and no floor moves when you collect more labels. That line is the only place the run reports them.

Enough means five coordinates: the judge id, the judge model, the prompt sha, the metric and the segment. A promotion is scoped by six, and eval_ver is the one this check does not read. The source records the omission without saying whether it was intended, and one shipped path depends on it: a shadow-plane grade projects its subject with an empty eval_ver, and that subject is admitted. The six coordinates a promotion is filed and pinned by are on the walk that promotes a judge.

The protocol’s vocabulary carries a subject_incomplete code for the same condition, applied as a precondition inside the validation routine. The one caller that reaches that routine is handed subjects the assembler has already filtered, so on this verb you meet the counter and not the code.

Promotion is not a command

No verb flips a judge to validated. A promotion is what the walk leaves behind when no gate had anything to hold against the subject, and the registry records that fact. Nothing on the command line and nothing in the API sets it.

Two things follow for what you do next.

Re-read a later window, not the one you were looking at. A row’s quality basis is derived at read time against the promotion registry, using the row’s own timestamp, so every judged row written before the promotion instant keeps its reported basis and keeps it permanently. Nothing rewrites them. Running the same report over the same window after a promotion and finding no movement is the most common reason an operator concludes the promotion failed.

Treat a prompt edit as a re-validation. The promotion is pinned to the judge’s model, its prompt sha and its evaluator version. Move any of the three and the pin stops matching, the subject reads version_changed, and its scores ride the reported ceiling until you run this check again over a fresh gold set.

What this page does not cover

The run promoted and your figure did not move. A quality basis is the lowest of several caps and only some of them consult the promotion registry, so a promotion can land and change nothing you are looking at. Why your number is reported, not measured finds the cap that binds.

A symptom you cannot place. Symptom, cause and fix for each answer this build prints, in one table, is Why didn’t my judge validate?.

What each gate measures, and why kappa. The gold set, the overlap sample, the kappa floor and the swap gate are the walk that promotes a judge. This page runs that walk rather than explaining it.

Turning a promotion into a routing decision. A quality floor reads a measured basis only, so a standing promotion is the precondition and not the mechanism. Gate a rollout on a validated judge is the mechanism.