Judge vocabularies

You are looking at a token in a log line, a JSON field or a command’s output, and you need to know which set it came from and what else is in that set. Every list on this page is closed: a member is named, or the value is turned away at the boundary that produced it.

Each table quotes the Go constant that defines the token, with the file and the line. Tokens are printed the way the product prints them, because a paraphrase will not match a grep.

Some members below have no producer that a configuration this build accepts can reach. Those rows say so in their own column, since a list that names a token without saying whether anything emits it sends you hunting for output that will not arrive.

Where each vocabulary is defined

Vocabulary	Members	Defined at
Judge methods	5	internal/judgecore/method.go:50-54
Winner	3	internal/judgecore/method.go:76-78
Panel ballot tokens	2	internal/judgecore/panel.go:179-180
Abstention reasons	4	internal/judgecore/method.go:99,103,107,112
Judge disclosure fields	9	internal/declog/eval.go:82-172
Gold sources	2	internal/judgeval/goldset_types.go:29-30
Exclusion reasons	6	internal/judgeval/goldset_types.go:53-58
Assembler skip counters	6	internal/judgeval/assemble.go:365-391
IAA reason codes	5	internal/evaluator/validate.go:510-514
Validation refusal codes	13	internal/evaluator/validate.go:331-345
Applied floors	6	internal/evaluator/validate.go:103-130
Promotion states	3	internal/evaluator/validate.go:1364,1366,1370
Promotion timeline kinds	5	internal/evaluator/validate.go:1423-1427
Registry restore states	3	internal/judgeval/registry.go:86,88,95
Validation run reasons	4	internal/judgeval/runs.go:47,50,52,56
Selective-risk refusal codes	5	internal/judgeval/selective.go:112-136
Selective per-verdict outcomes	4	internal/judgecore/selective_hook.go:95-107
Selective scan outcomes	4	internal/judgeval/selective.go:395-403
Selective audit outcomes	6	internal/judgeval/runs.go:417-438
Drift outcomes	4	internal/judgeval/driftwatch.go:318-325
Drift refusal codes	12	internal/judgeval/driftwatch.go:332-348
Anchor attribution	5	internal/judgeval/anchor.go:424-428
Anchor arm	4	internal/judgeval/anchor.go:452-455
Anchor refusal codes	13	internal/judgeval/anchor.go:476-488
Sidecar judge fault reasons	12	internal/judgecore/sidecar_transport.go:151-190

Method, winner and ballot tokens

Three small sets that travel together on a graded row. The method token is read off the scorer that ran, not off the configuration that asked for it.

Set	Members
MethodID	pointwise-bool · pointwise-rubric · pairwise · reference · panel
Winner	A · B · TIE
Panel ballot	pass · fail

Two of the five method ids compose from a configuration file, and one of those two carries a condition. Which two, what the condition is, and what each method decides, is Judge methods. The ballot vocabulary has two tokens, which is why a panel’s tally has at most two buckets, and that fact governs the next table.

Abstention reasons

An abstention is a verdict with no winner. Each reason names what to fix rather than reporting only that the judge declined.

Reason	What the judge is saying	What has to be true for a producer to reach it
swap_inconsistent	The two presentation orderings named different winners, so the verdict was partly a function of position and no winner is written	Minted by NewPairwiseTrial (method.go:278) and by the shadow sampler (internal/shadow/judge_config.go:428), and re-derived on the read side over stored shadow rows (internal/api/dashsrv/evals_experiments.go:2167). Reached in production on the shadow-duplicate plane, through shadow.duplicate.pairwise
below_conformal	The verdict’s confidence feature fell below the risk-controlled accept threshold	Two sites mint it, and a configuration this build accepts reaches neither. SelectiveGate (internal/judgecore/selective_hook.go:264) needs a fitted accept region, and NewSelectiveGate has one non-test call site, under if region != nil, whose one non-test caller passes nil (internal/shadow/judge_config.go:152-153 and sampler.go:336), pinned by TestShadowSamplerNeverInstallsARegion. The other site reads the token off a container’s self-report, and a sidecar_judge: block is refused at tokentriage validate and again at boot
panel_split	A panel’s members disagreed too evenly for the rule to name a verdict	Minted by panel.go:745. The one of the four that composes from a configuration file, and it needs an even seat count: panelTally decides whenever 2 × top > n over a two-token ballot (panel.go:805-821), so an odd panel always decides. EvalsPanelMinMembers is 3 with no maximum, so four seats is the smallest shape that can split
out_of_scope	The input fell outside the scope the promotion evidence covers	One site mints it, and it reads the token off a container’s self-report (sidecar_transport.go:923-931, used at :732 and :833). An evals.evaluators[].sidecar_judge block is refused at tokentriage validate and again at boot, so nothing a configuration file composes can report one

AbstainPanelSplit’s own doc comment (method.go:104-107) says a split fires when the margin was “under the configured quorum”. There is no quorum leaf: PanelRuleMajority is the single aggregation rule at this freeze (panel.go:167-170), and the comment describes a knob a later card would add.

The judge disclosure block

JudgeDisclosure travels on every judge-produced outcome, so a judge score cannot pass as anonymous ground truth. Three fields are always present; six are pointers, and an absent pointer is a statement about measurement rather than a zero.

Field	Go type	What it says	What absence means
model	string	The judge model that graded the row	Always present
prompt_sha	string	sha7 of the judge template	Always present
validated	bool	A promotion stands for this judge on this metric and segment	Always present. false is a reading, not a gap
kappa	*float64	Judge-to-human Cohen’s κ from the promoting run	Absent means not measured. 0.0 is a measurement, and means agreement at chance level
swap_flip	*float64	Position-swap flip rate	Absent means not measured. 0.0 is a measurement, and is the best result
validated_n	*int64	The validation set’s size	Absent means no validation run stands
iaa	*float64	Inter-annotator κ of the gold set the judge was validated against	Absent means agreement was not measured. Which of four situations applied is the next table
method	string	The method that graded this row, read off the built scorer	Absent means a binary older than the field, or a disclosure rebuilt from the promotion registry rather than emitted by a run. It is not backfilled
abstain_feature	*float64	This verdict’s own scalar abstention feature, an input to a conformal fit	Absent means this verdict’s method published no feature. The shadow producer writes it only at self_consistency_k > 1

IAA reason codes

GoldAdmission.IAAReason says why iaa is null. The closed set has five members: one covers the measured case and four cover the null, so four is the number of ways agreement can go unmeasured.

IAAReason	What happened
measured	κ_h was computed, so iaa carries a number
single_annotator	One annotator over the admitted rows. Admitted, and disclosed
no_annotators	The gold set holds no attributable label
overlap_below_floor	Two or more annotators, and no pair shares IAAOverlapMinN rows
undefined_on_overlap	A pair does share enough rows, and κ_h has a zero denominator on them

Validation refusal codes

Refusal.Code is machine-readable and stable; Refusal.Text beside it is what an operator reads, and it is built so it cannot be constructed without the floor, the need and the have. A floor refusal names the constant it applied in Refusal.Floor; a shape code names none.

Code	What did not pass	Floor cited
gold_set_empty	Every offered row was excluded or unattributable, so no annotator survives	JudgeValidationMinN
iaa_below_floor	The annotators’ κ_h is under the gold-set admission floor. The fix is adjudicating the disagreeing rows, which the refusal names	GoldSetIAAFloor
iaa_unmeasured	κ_h could not be measured and evals.validation.require_iaa is on	IAAOverlapMinN
iaa_undefined	κ_h has a zero denominator on the overlap	(shape)
validation_below_n	Fewer admitted human-labeled rows than the judge-validation floor	JudgeValidationMinN
kappa_below_floor	Judge-to-human κ is under the promotion floor	JudgeKappaFloor
kappa_undefined	κ has a zero denominator	(shape)
swap_below_min_pairs	Fewer pairs run in both orderings than the swap set needs	SwapSetMinPairs
swap_pair_incomplete	A pair is missing one of its two orderings	(shape)
swap_flip_above_ceiling	The measured flip rate is over the position-swap ceiling	SwapFlipCeiling
subject_incomplete	The subject does not name all of B.7’s coordinates	(shape)
rubric_invalid	The rubric’s scale is unusable as a binary verdict	(shape)
drift_upper_bound_below_floor	The post-promotion confidence sequence’s upper bound fell under the transferred floor	DriftTransferredFloor

Twelve of the thirteen are reachable from tokentriage evals validate, and subject_incomplete is not. evaluator.ValidateJudge has one non-test call site (internal/judgeval/runs.go:346), and every gold set that reaches it was built by the assembler, which screens each subject with the same Complete() predicate first (internal/judgeval/assemble.go:638 and :725). An incomplete subject is dropped there and counted, so what an operator meets is incomplete_subject=N on the skipped: line rather than the code.

The floors a validation run applies

All six print on every run, whether or not the run reached them, so floors applied: is the whole ladder and the code names the rung. AppliedFloorNames() (internal/judgeval/runs.go:92-97) returns them in this order.

Identifier	Value	Unit	What it gates
JudgeValidationMinN	50	admitted human-labeled rows	Rows required per subject before a κ may be computed at all
IAAOverlapMinN	20	rows two annotators both labeled	Measured on the overlap sample, and not on the union
SwapSetMinPairs	30	pairs run in both orderings	Pairs required before a flip rate may gate anything
JudgeKappaFloor	0.5	Cohen’s κ	The promotion bar for judge-to-human agreement
GoldSetIAAFloor	0.6	Cohen’s κ_h	The gold set’s own admission bar
SwapFlipCeiling	0.10	flips ÷ pairs	Above this, promotion for pairwise use is refused

The evals.validation.min_kappa leaf moves JudgeKappaFloor, and moves it upward only. The configuration loader turns away a value below 0.5 or above 1 (internal/config/evals.go:1832-1841), while the library clamps instead of refusing (effectiveKappaFloor, internal/evaluator/validate.go:978-988). Two behaviours for the same out-of-range value, at two layers.

A refused run, with these tokens on it

The vocabularies above meet in one place. This is the middle of a refused run, from the harness capture at docs/judge-facts/transcripts/T4-validation-refused.md, with the assembly block and the selective-risk block elided.

$ tokentriage evals validate --config … --state-dir … --log …
promotion registry — D:\tt-jg-scratch\hwork2\run\state\dash\quality
  validation_state: empty
…
validation runs (1)
  [1/1] judge=judge-task-completion model=claude-haiku-4-5 prompt=1fc3923 eval_ver=judge-v1+cfg15acf67
      metric=task_completion segment=task_completion|tier=frontier|model=gpt-5|endpoint=chat
      reason: operator — operator-forced validation of 40 admitted label(s) at revision 8b39113
      gold: admitted=40 annotators=2 licenses=true iaa=unmeasured (overlap_below_floor) revision=8b39113
      kappa: NOT COMPUTED — a gate before it refused (the refusals below name which); validated_n=40
      floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1
      verdict: REFUSED   registry change: refused
      refusal validation_below_n: floor=JudgeValidationMinN need=50 have=40 admitted human-labeled rows
…
      reading: state=never_validated gold_revision=8b39113 auto_validate=false subjects_observed=1

Eight vocabularies are on those lines: validation_state, a run reason, an IAAReason, the six applied floors, a validation code with the floor it cites, the CLI verdict token, a promotion timeline kind and a promotion state. The paths, the digests, the eval_ver suffix and the fixture’s judge and segment are that machine’s; the floor constants, the code and the need/have arithmetic are not.

That capture came from a source build on Windows, and its ledger was produced by the documentation harness. The assembler, Cohen’s κ, the inter-annotator figure, every floor, the promotion registry and the artifact store are the shipped binary’s, while the judge verdicts came from a loopback stub and the human labels were designed by the harness. It shows what a refused run looks like, and not what any judge scores.

Promotion states, timeline kinds and restore states

Three sets answering three different questions: may this subject claim a measured basis, what did the last write do, and what happened to the registry at boot.

PromotionStatus is served as state on a judge reading.

Value	Meaning
promoted	Validated, and the version still matches
never_validated	Nothing was ever stored for this identity. A run that refused reads this too
version_changed	A promotion exists for this identity but was earned under a different judge model, prompt or evaluator version. The stale state is still returned, so a caller can say what was lost

PromotionChange.Kind is what the registry write did, and it is the kind on the timeline.

Kind	Meaning
promoted	A first promotion for this key
repromoted	A promotion that was already standing, renewed by a fresh run
demoted	A standing promotion removed, either because κ fell or because the bar rose
reset_version_change	The subject’s version moved, so the promotion was reset rather than carried
refused	The run refused and nothing was promoted. The record is still written

ValidationState is a fact about the whole registry at boot, so it sits on the collection envelope rather than on a per-subject reading.

Value	Meaning
empty	No jval_ artifact existed at boot. A fresh install, or a store whose artifacts were deleted. Every judge reads reported
restored	The newest artifact restored cleanly
unrestorable	An artifact existed and was turned away: an unversioned entry, an unknown artifact version, or corruption. The registry boots empty and keeps the refusal text for the operator

The CLI prints its own two-token verdict beside these: PROMOTED or REFUSED (cmd/tokentriage/evals_judges.go:775,777).

Validation run reasons

Why a validation run happened. RunReasons() (internal/judgeval/runs.go:87) returns these four, in edge order.

Reason	Fired when
boot	The process started, or a judge tuple’s configuration changed, which is the same edge because every judge leaf is restart-required
new_labels	New admitted human labels crossed the per-subject threshold, default 25
interval	The periodic sweep’s cadence elapsed, default 24h
operator	A human ran tokentriage evals validate

A fifth token, drift_breach, is defined at internal/judgeval/driftwatch.go:813 and reaches a record only inside a formatted detail string (driftwatch.go:601). RunReasons() does not return it, so a consumer that ranges the enumerator cannot see it.

Selective-risk vocabularies

Four sets around the risk-controlled accept region: why a fit declined to publish one, what one threshold candidate did during the scan, what the run did with a sealed sample, and what a region did to a single verdict.

SelectiveRefusalCode is why the fit published no region. Every one carries need, have and the (α, δ) pair.

Code	Meaning
selective_no_feature	The method publishes no scalar confidence feature, so there is nothing to threshold. This is what a pointwise-bool judge at self_consistency_k: 1 produces
selective_cal_n_below_floor	The calibration slice holds fewer rows than SelectiveMinCalN(α, δ)
selective_no_admissible_threshold	The scan ran and no tested candidate’s bound cleared α
selective_accepted_m_below_floor	A threshold was selected, and the holdout’s accepted subset is under SelectiveMinAcceptedM(α, δ)
selective_holdout_verification_failed	A threshold was selected and the accepted subset cleared its floor, and then the holdout’s own upper bound on error exceeded α

SelectiveScanOutcome is what the scan did with one candidate; SelectiveAuditOutcome is what the run did with one sealed sample.

Set	Members
SelectiveScanOutcome	skip (too few accepted rows to compute a bound) · pass (bound computed, cleared α) · fail (bound computed, exceeded α, and the scan stops here) · unreached (the scan stopped before this candidate)
SelectiveAuditOutcome	not_requested · method_undeclared · method_unknown · fitted (first fit of this subject version in this process) · reused (the sealed sample is byte-for-byte the one already fitted) · resampled (new labels moved the split, so a fresh sample and a fresh look)

SelectiveOutcome is what a fitted region did to one verdict. Three of the four mean the region did not act, and they say why.

Value	Meaning
accepted	The feature was at or above the threshold and the verdict stands unchanged
refused	The feature fell below the threshold, so the verdict is replaced by an abstention with reason below_conformal
no_feature	The verdict published no feature, so there was nothing to threshold. The verdict stands
already_abstained	The method had declined for its own reason. The first reason stands, because relabelling it would report the wrong cause

Drift vocabularies

Three sets belonging to the post-promotion agreement monitor. Read them as the shapes a reading can take rather than as output you can produce. Gate.ObserveAgreement has no call site outside its tests (internal/judgeval/driftwatch.go:590), so no shipped binary can fire a judge demotion from this path, and the drift codes below carry the product’s own note that no operator surface renders them today (driftwatch.go:329-330).

DriftOutcome	Meaning
refused	A named limb refused and nothing was evaluated. Not a demotion, and not a fabricated bound
insufficient	The stream is shorter than DriftMinN, so nothing was evaluated
holding	Evaluated, and the upper bound is not below the floor
breached	Evaluated, and the upper bound is below the floor

Drift refusal code	Meaning
p_e0_degenerate	The frozen expected-agreement term leaves no usable denominator
p_e0_out_of_range	The frozen expected-agreement term is outside its legal interval
accumulator_non_finite	The running accumulator went non-finite
derived_disagrees_with_formula	A stored derived value does not match a recomputation
stream_version_moved	The subject’s version moved under the stream
no_lambda_weight	The confidence sequence has no weight to apply
no_promotion_stands	There is no standing promotion to monitor
no_stream_on_state	The promotion carries no drift stream. The accumulator opens only under a single-annotator gold set (internal/evaluator/validate.go:1578), so a promotion earned with two annotators and an overlap has none
duplicate_request_in_batch	The same request id appears twice in one batch
unnamed_request_in_batch	An observation nothing can attribute to a request
p_e0_unit_mismatch	The stream’s frozen term does not certify the request-unit reading
snapshot_write_failed	The write failed after the stream had advanced in memory

DriftDisclosure.Reason carries two further tokens on a reading that was not refused: drift_min_n on insufficient (driftwatch.go:520), and upper_bound_below_floor on breached (driftwatch.go:551).

Anchor diagnosis vocabularies

Three sets belonging to the who-drifted question, which is planned rather than shipped: no shipped binary constructs the monitor that fills them in, so read them as the shape of an answer.

AnchorAttribution	Meaning
judge_drifted	The anchors moved and the live traffic held
traffic_drifted	The anchors held and the live traffic moved
both	Both arms moved
neither	The instrument ran and found nothing. Distinct from undiagnosed, because “nobody ran it” must not serialize the same way
undiagnosed	No diagnosis was produced

AnchorArm	Meaning
moved	The anchor arm fired
held	A negative that could have been a positive: enough discordant items for an outcome to have fired, or an arithmetically identical pair of agreement counts
inconclusive	Between those two, where no outcome could have rejected
refused	The arm did not run, and says so

Thirteen anchor codes name the distinct causes: anchor_rescore_disabled · anchor_egress_not_allowed · anchor_no_rescorer · anchor_no_budget · anchor_no_budget_id · anchor_budget_refused · anchor_no_frozen_set · anchor_below_set_floor · anchor_incomplete_rescore · anchor_rescore_failed · anchor_live_arm_unavailable · anchor_underpowered · anchor_subject_mismatch. Eleven of the thirteen are assigned to a Reason; anchor_below_set_floor and anchor_subject_mismatch reach only an error string, because neither path produces a diagnosis for a reason to sit on (anchor.go:458-474).

Sidecar judge fault reasons

Twelve reasons a judging container’s answer was turned away. The transport discards the third party’s own message in every case and reports only the reason, so a container’s bytes cannot ride onto a row.

Reason	Meaning
unreachable	The request never completed: DNS, connect, TLS, reset or a deadline
status_not_ok	A non-2xx status. The status travels; the body does not
malformed_response	2xx, and the body is not the documented shape
no_verdict_for_item	The response carried no verdict for the id asked about. A different fact from malformed_response
verdict_shape_wrong	The verdict does not match the method asked: a score where a winner belongs, both at once, or neither
score_out_of_range	A score outside [0,1], or NaN, or ±Inf. Refused rather than clamped
winner_unknown	A winner outside the closed A/B/TIE set
abstain_reason_unknown	An abstention whose reason is outside what a container may claim. A container may claim out_of_scope or below_conformal, and neither swap_inconsistent nor panel_split, because those are facts about a trial only the core observes
version_unreported	No model or no version in the response
subject_too_large	An item side exceeds the per-side bound. Skipped, and not truncated
health_unreachable	The /healthz probe never completed
health_status_not_ok	The /healthz probe returned a non-2xx status

The two health reasons are separate from the ten judging reasons so a probe transcript cannot be misread as a judging one.

Gold-set sources, exclusions and skip counters

What the assembler admitted, what it excluded and what it skipped, before any of the above runs.

GoldSource	Meaning
outcomes	Human EvalRow submissions, under a server-derived actor identity
feedback	FeedbackRow rows admitted under the DJ10 mapping, which admits only metrics whose rubric is the binary satisfaction question

ExcludedReason has six members, and the sixth is the empty string.

Value	Meaning
""	Admitted. Absence of an exclusion is the zero value, so a row that says nothing is an admitted row
demo	A tokentriage demo record. Excluded unconditionally
synthetic	A dataset-import or synthetic marker. Excluded unconditionally
both	Demo and synthetic
unattributed	No actor or no request id, so the row cannot join an overlap sample
superseded	Replaced by a later label from the same annotator

The six skip counters print on the skipped: line of every assembly block.

Counter	Meaning
no_metric	An outcome carrying no metric id, so it names no segment
unkeyable	A segment coordinate containing a pipe or an equals sign, which the segment key turns away
no_pass_verdict	The Pass field is nil. An absent pass verdict is not a false, so it is not a label
incomplete_subject	A judge-produced outcome whose subject does not name every B.7 coordinate
pseudo_actor_outcome	A human row carrying the shared bearer pseudo-actor. The shipped write door refuses bearer principals, so a non-zero count here means the row entered by another route
shadow_no_subject	A graded shadow companion row recording no judge id or no metric, so no subject can be built from it

What this page does not cover

The tokens here are the ones the judge track defines. Three neighbouring vocabularies live elsewhere: the calibration and evals enums, including the basis values and the four-state eval verdict, are Calibration & evals vocabularies; the request and response shapes that carry these tokens over HTTP are Judge API; and what each method decides and costs is Judge methods.

This page states what a token means, and not how to act on one. For the symptom-to-fix walk, start at Validate and promote a judge.