Why didn't my judge validate?

You are looking at a screen that is not doing what you expected. Perhaps a check declined and named a count that will not move however much work you throw at it. Perhaps nothing declined at all, and there is nothing there to look at.

Start from what you can see. Find the row in the table that matches your observation, follow it to the entry, and each entry gives you a command that tells the causes apart, the change that fixes it, and the thing to look at afterwards to know it worked.

Every entry here restates, in symptom form, something one of the section’s other pages explains. Reading the section straight through? Skip this page and come back when something breaks.

Find your symptom

What you can see	Go to
Requests are served, no ledger row carries a judge block, and no log line mentions one	Nothing is judged, and nothing is logged
validation runs (0) on a ledger you know has traffic in it	The run reports no validation runs at all
The annotation queue returns 403 and the annotator cannot start	The annotation queue answers 403
The queue returns {"items": []} at HTTP 200	The claim comes back with no items at HTTP 200
Two people have been labelling and the record says agreement was not measured	Two annotators are labelling and agreement is unmeasured
A submission returned {"accepted": 1} and no counter on the next run moved	A label was accepted and no counter moved
floor=JudgeValidationMinN need=50 have=0 on a subject the run lists by name	The label count stays at zero on a subject the run lists
A judge method or a sidecar_judge: block is turned away by name	The config is turned away at validate and again at boot
The same error text appears twice in a row	Every error and warning prints twice
A promotion landed, the floor still admits no candidate, traffic sits on on_empty	The quality floor admits nothing after a promotion landed
Shadow duplication is configured and produces no rows and no warning	Shadow duplication writes nothing and says nothing

Nothing is judged, and nothing is logged

Confirm it. Read the metacost surface, which carries one line per configured evaluator with its counted refusals beside its spend.

$ curl -s localhost:9090/tt/api/v1/metacost | jq '.sources[] | select(.name | startswith("eval:"))'

A judge that is sampling and writing nothing reads there as evaluator_error climbing against a spend that stays flat. Cross-check the ledger: no judge block on any row over the same window.

Why it is quiet. A judge call that cannot be made makes no noise at all. The loop that runs evaluators holds no logger, so a scorer error cannot be written at any level; the error text goes into a counter that carries an evaluator id and a reason and no message field. Two distinct causes land in the same bucket there, which is why the counter alone does not tell you which one you have: a credential variable that holds nothing, and a judge whose model family matches the model that produced the answer.

Fix it. Both at once, since the counter will not separate them for you.

• Export the environment variable named by evals.evaluators[].judge.api_key_env. Config requires the leaf and turns away an empty one, and no check reads whether the variable it names holds anything.
• Set evals.evaluators[].judge.model to a family different from every tier’s model. A judge grading its own family is turned away before the call.

Confirm the fix. A judge block appears on the next sampled row, and the eval:<id> line’s spend moves off zero.

The run reports no validation runs at all

Confirm it. Run the verb with both paths given explicitly, and read the first block.

$ tokentriage evals validate --log ./decisions.jsonl --state-dir ./state
gold-set assembly — ./decisions.jsonl
  sources: outcomes   sets: 0   segments: 0   torn lines: 0   unorderable req_ts: 0
  skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0
  feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0

promotion registry — state\quality
  validation_state: empty
  auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25
    this CLI process does not drive the automatic loop; the SERVER does, from boot
    (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`.

validation runs (0)
  none — the ledger carries no judge subject on any segment with human labels.

Exit status is 0, and a run over nothing is an answer rather than an error. That capture is over a zero-byte ledger on a source build on Windows, which is why the state path renders with a backslash and the store count and the trailing spend note are elided here; three runs of it were byte-identical. The auto_validate values shown are the shipped defaults and were read from no config.

Why it happens. Three things produce the same block, and the header tells them apart. sets: 0 with segments: 0 means the assembler found no judged row carrying a human label to pair against. sets above zero with validation runs (0) means labels exist and no judge subject does. And the header prints the path it read, which catches the third cause.

Fix it. Take them in that order.

• Wrong ledger. The log path resolves as --log, then ledger.decision_log.path, then the compiled default. A relative path in a config resolves against the working directory of this process, which need not be the one the server booted from. Pass --log explicitly and the question disappears.
• No judged rows. How a judge call happens covers what decides whether a row gets graded; the first entry on this page covers a judge that is configured and silent.
• No labels. Collect the labels a judge is measured against is the procedure.

Confirm the fix. sets: and segments: go above zero, and a validation runs (1) block appears with a subject line under it.

The annotation queue answers 403

Confirm it. Read the body, which names the role it wanted.

{ "status": 403, "detail": "role operator is below required admin", "required_role": "admin" }

Why it happens. A claimed item carries the request’s stored prompt and completion verbatim, which sits behind a privileged content scope that only an admin-role credential holds. The write half is different: the outcomes door sits at the operator floor, so a person who can submit labels is not automatically a person who can be offered them.

Fix it. Create each annotator with the admin role, before the server boots.

$ tokentriage user add annotator-a --role admin --state-dir ./state/dash

Confirm the fix. The same claim returns items, and the annotators=[…] list on a later assembly block carries the name the server derived from the session.

The claim comes back with no items at HTTP 200

Confirm it. Check which strategy the request carried. The dashboard’s annotate page preselects hard_cases, so a claim made by clicking rather than by hand is almost always that one.

Why it happens. hard_cases ranks rows the judge found hard, and hardness has two ingredients: a judge that abstained, and a panel that split. A plain pointwise-bool judge that answered every row produces neither, so the queue has nothing to rank and answers with an empty list at HTTP 200. That is a statement about your traffic rather than a fault.

Fix it. Claim with the other member of the closed pair.

{ "strategy": "starved_segments", "k": 60 }

Confirm the fix. Items come back, each carrying a priority_reason naming how far the segment is from its label floor. Reach for hard_cases where you run a panel or a method that abstains, and for starved_segments everywhere else.

Two annotators are labelling and agreement is unmeasured

Confirm it. Read the gold: line on the record.

      gold: admitted=79 annotators=2 licenses=false iaa=unmeasured (overlap_below_floor) revision=5dac3b3
      refusal iaa_unmeasured: floor=IAAOverlapMinN need=20 have=19 overlapping rows

Two annotators, seventy-nine label events, and the widest overlap between any pair of them is nineteen requests against a floor of twenty.

Why it happens. The lease and the measurement pull against each other. A lease exists to stop two annotators labelling one request by accident; agreement is computed on precisely the requests two annotators both labelled. Submitting does not release a lease, only expiry does, so one annotator working the queue takes every available row out of the other’s reach for the lease duration. Used as it comes, the queue cannot hand you an overlap sample.

Fix it. One of three, and only the first is a plan.

• Hand the overlap set to the second annotator out of band. The submission door takes a req_id and does not consult the lease table, so a second annotator can label a request they were not offered. Decide the overlap first, at twenty requests or more, and let the bulk follow.
• Claim in turns, small. With an hour-long lease and a claim of twenty-five, one annotator works a batch while the other waits for the lease to lapse. It is arithmetic you do yourself.
• Restart the server. The lease table is in memory, so a restart releases every lease early. It is a recovery, not a workflow.

Confirm the fix. The gold: line prints a number in place of unmeasured, computed over the shared requests rather than over the union of the two label sets.

A label was accepted and no counter moved

Confirm it. Compare admitted= on the assembly block against the number of labels you sent, then read the two lines under it.

      offered=40 admitted=40 (outcomes=40 feedback=0) requests=30 subjects=1 annotators=[annotator-a annotator-b]
      excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0
  skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0

If admitted= is short and excluded: and skipped: are both all zero, the missing labels are not being turned away anywhere. They are not there.

Why it happens. The outcomes door does not check that the req_id on a submission names a row the ledger holds. A typo returns {"accepted": 1} at HTTP 200 and writes a companion row with nothing to join to, and that row then appears in no counter on the assembly block, not even as a skip.

Fix it. Submit the req_id values the claim handed you rather than any you typed, and check admitted= against the number you sent after every batch. Two other causes shorten the same counter and do announce themselves on the excluded: line: an outcome with no attributed annotator, and one a later label superseded. A second label from the same annotator on the same request replaces the first, last write winning, so a correction lands cleanly and so does a mistake.

Confirm the fix. admitted= matches what you submitted, and requests= matches the number of distinct requests behind it.

The label count stays at zero on a subject the run lists

Confirm it. Read the judge= line on the record, above the line that declines.

  [1/1] judge=judge-groundedness model=… prompt=… eval_ver=…
      refusal validation_below_n: floor=JudgeValidationMinN need=50 have=0 admitted human-labeled rows

The subject is observed, listed and given a run of its own, and the count is zero on a ledger that holds judged rows.

Why it happens. judge-groundedness writes a scalar score and no pass verdict, because the metric behind it publishes no pass floor and the runner forces the verdict field to nil on every row it grades. The gold-set assembler records a verdict only where one is set, so none of that judge’s rows are admitted. The subject is appended to the census before that gate, which is why it appears on the reading at all. So the run tells you to collect labels, and no quantity of labels will move the number.

Fix it. Configure judge-task-completion on the metric task_completion, whose descriptor carries a floor, for any judge that has to reach a promotion.

evals:
  evaluators:
    - id: judge-task-completion
      judge:
        external: allow
        model: gemini-2.5-flash
        api_key_env: JUDGE_API_KEY
      sample_rate: 0.20
      max_usd_per_day: 20.0

Confirm the fix. admitted= on the assembly line moves above zero for the new subject, and have on the refusal climbs with it. judge-groundedness remains usable as a reported signal; its verdict travels losslessly in the score it writes.

The config is turned away at validate and again at boot

Confirm it. Read the prefix, because the body is identical at both gates and the prefix is the only thing that says which one you hit.

$ tokentriage validate --config pairwise.yaml
Error: configuration is invalid: evals.evaluators[1].judge.method: "pairwise" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel

Exit status is 1. Boot prints the same sentence under the prefix Error: evals: instead, after the promotion registry has already opened and the dashboard has already composed, so it arrives late in a boot log rather than at the top of it.

Why it happens. Two judge method configurations compose from a config file: pointwise-bool, which is what an absent leaf reads as, and panel at self_consistency_k: 1. pointwise-rubric, pairwise and reference are turned away over a judge: transport, as is any self_consistency_k above 1, and every sidecar_judge: block.

Fix it. Take the method the message names, or seat a panel. Configure a judge has every key with the wording the binary prints, and Run a cross-family panel is the panel shape.

Confirm the fix. OK: configuration is valid at exit 0, and then a boot that reaches evals: judge evaluator wired. Validating is not the same promise as booting, so do both.

Every error and warning prints twice

Confirm it. Count the copies. Every error and every warning from tokentriage validate and from tokentriage run appears twice, on unrelated configs and for unrelated causes, because this binary writes each one once through its command framework and once from main.

The fix is to change nothing. The repetition belongs to the binary and says nothing about your config. Two copies of a message are one event.

What to do instead. Read the prefix rather than the count, since that is the token that carries information: configuration is invalid: came from the validator and evals: came from boot. And treat a pasted capture showing a single copy as edited, including any you paste yourself.

The quality floor admits nothing after a promotion landed

Confirm it. Read one decision and look at the score table, which carries every candidate’s quality verdict rather than only the admitted ones.

$ tokentriage policy explain --req-id <req-id> --log ./state/decisions.jsonl

A quality_below_min_lcb term means the tier was measured, admitted to the comparison, and its bound fell short. A quality_absent:<reason> term means the quality could not be read at all, with the reason riding the term name. They are different facts and they call for different actions.

Why it happens. Three causes are common and the first is the one nothing warns you about.

• The metric id is misspelled. It is checked against no registry at validate time, so the file passes with OK: configuration is valid at exit 0 and then admits no candidate at runtime. Check the id against the evaluator’s declared metrics by hand.
• The window predates the promotion. A promotion is forward-only, so a report over a window that ends before it stays at the lower ceiling however good the judge turned out to be.
• The promotion is for another segment. A promotion is filed under one judge id, metric and segment. The same judge on a different tier is a different subject.

Fix it. Gate a rollout on a validated judge walks the whole path, including the calibration the floor reads its bound from.

Confirm the fix. The candidate appears on the score table with no quality term against it, and traffic stops landing on on_empty.

Shadow duplication writes nothing and says nothing

Confirm it. Search the boot log for the line that says the feature came up.

$ tokentriage run --config tokentriage.yaml 2>&1 | grep 'shadow duplication enabled'

No match means the feature is off, whatever the config appears to say.

Why it happens. shadow.duplicate.max_usd_per_day is part of the condition that switches the feature on, alongside the enable flag and a non-zero sample rate. Omit it, or write 0, and the whole block is skipped: the file validates clean, the binary boots clean, and no line is written at either gate. Two configs differing by exactly that one leaf both report OK at exit 0, and only the capped one prints the boot line above and the egress warning beside it.

Fix it. Give the leaf a positive dollar figure.

shadow:
  duplicate:
    enabled: true
    sample_rate: 0.05
    max_usd_per_day: 5.0

Confirm the fix. The boot line appears, and the shadow line on the metacost surface starts carrying dollars. What judging costs, and how to cap it has the four planes that can call a grading model and which meter each reports to.

Two traps in the vocabulary itself

Both cost time in the same way: you go looking for a token, find nothing, and conclude the condition did not happen.

The counter and the code spell the same idea in opposite orders. The assembly block’s skipped: line carries incomplete_subject, and the protocol’s closed vocabulary of refusals carries subject_incomplete. Grep for the wrong one and you find nothing at all.

You will meet the counter and not the code. subject_incomplete is a precondition inside the validation routine, and the one caller that reaches that routine is handed subjects the assembler has already screened with the same test. So on this verb the condition surfaces as incomplete_subject=N on the skipped: line and never as a code. Rows counted there left before any gate saw them: no line naming a floor mentions them, and no floor moves when you collect more labels.

What this page does not cover

What each gate measures. The gold set, the overlap sample, the kappa floor and the swap gate, walked in the order the binary walks them, are the walk that promotes a judge.

Every closed vocabulary this track prints. The abstention reasons, the validation floors, the promotion states and the disclosure fields, one plain line each, are Judge vocabularies.

The gaps, as a list rather than as symptoms. Several entries above are the reader-facing face of something this build does not do, and the catalogue of those, grouped by what stands in the way of each, is Air-gap judging, and the known limits.