# Why didn't my judge validate? The judge did not validate, or the number did not move. Symptom, cause and fix for each answer this build prints. Source: https://tokentriage.com/docs/judges/troubleshooting --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; You are looking at a screen that is not doing what you expected. Perhaps a check declined and named a count that will not move however much work you throw at it. Perhaps nothing declined at all, and there is nothing there to look at. Start from what you can see. Find the row in the table that matches your observation, follow it to the entry, and each entry gives you a command that tells the causes apart, the change that fixes it, and the thing to look at afterwards to know it worked. Every entry here restates, in symptom form, something one of the section's other pages explains. Reading the section straight through? Skip this page and come back when something breaks. ## Find your symptom | What you can see | Go to | |---|---| | Requests are served, no ledger row carries a `judge` block, and no log line mentions one | [Nothing is judged, and nothing is logged](/docs/judges/troubleshooting#nothing-is-judged-and-nothing-is-logged) | | `validation runs (0)` on a ledger you know has traffic in it | [The run reports no validation runs at all](/docs/judges/troubleshooting#the-run-reports-no-validation-runs-at-all) | | The annotation queue returns `403` and the annotator cannot start | [The annotation queue answers 403](/docs/judges/troubleshooting#the-annotation-queue-answers-403) | | The queue returns `{"items": []}` at HTTP 200 | [The claim comes back with no items at HTTP 200](/docs/judges/troubleshooting#the-claim-comes-back-with-no-items-at-http-200) | | Two people have been labelling and the record says agreement was not measured | [Two annotators are labelling and agreement is unmeasured](/docs/judges/troubleshooting#two-annotators-are-labelling-and-agreement-is-unmeasured) | | A submission returned `{"accepted": 1}` and no counter on the next run moved | [A label was accepted and no counter moved](/docs/judges/troubleshooting#a-label-was-accepted-and-no-counter-moved) | | `floor=JudgeValidationMinN need=50 have=0` on a subject the run lists by name | [The label count stays at zero on a subject the run lists](/docs/judges/troubleshooting#the-label-count-stays-at-zero-on-a-subject-the-run-lists) | | A judge method or a `sidecar_judge:` block is turned away by name | [The config is turned away at validate and again at boot](/docs/judges/troubleshooting#the-config-is-turned-away-at-validate-and-again-at-boot) | | The same error text appears twice in a row | [Every error and warning prints twice](/docs/judges/troubleshooting#every-error-and-warning-prints-twice) | | A promotion landed, the floor still admits no candidate, traffic sits on `on_empty` | [The quality floor admits nothing after a promotion landed](/docs/judges/troubleshooting#the-quality-floor-admits-nothing-after-a-promotion-landed) | | Shadow duplication is configured and produces no rows and no warning | [Shadow duplication writes nothing and says nothing](/docs/judges/troubleshooting#shadow-duplication-writes-nothing-and-says-nothing) | ## Nothing is judged, and nothing is logged **Confirm it.** Read the metacost surface, which carries one line per configured evaluator with its counted refusals beside its spend. ```console $ curl -s localhost:9090/tt/api/v1/metacost | jq '.sources[] | select(.name | startswith("eval:"))' ``` A judge that is sampling and writing nothing reads there as `evaluator_error` climbing against a spend that stays flat. Cross-check the ledger: no `judge` block on any row over the same window. **Why it is quiet.** A judge call that cannot be made makes no noise at all. The loop that runs evaluators holds no logger, so a scorer error cannot be written at any level; the error text goes into a counter that carries an evaluator id and a reason and no message field. Two distinct causes land in the same bucket there, which is why the counter alone does not tell you which one you have: a credential variable that holds nothing, and a judge whose model family matches the model that produced the answer. **Fix it.** Both at once, since the counter will not separate them for you. - Export the environment variable named by `evals.evaluators[].judge.api_key_env`. Config requires the leaf and turns away an empty one, and no check reads whether the variable it names holds anything. - Set `evals.evaluators[].judge.model` to a family different from every tier's model. A judge grading its own family is turned away before the call. **Confirm the fix.** A `judge` block appears on the next sampled row, and the `eval:` line's spend moves off zero. `tokentriage doctor` will not find this. Its `keys` check walks `upstreams` credential references and stops there; it reads no judge leaf. It is also a soft-tier check, so even a warning from it would leave the exit code alone. The guardrail plane behaves the opposite way and logs `guardrails judge NOT armed: api key unset` at boot, which is worth knowing so you do not conclude from one plane's silence that the other would have spoken. ## The run reports no validation runs at all **Confirm it.** Run the verb with both paths given explicitly, and read the first block. ```console $ tokentriage evals validate --log ./decisions.jsonl --state-dir ./state gold-set assembly — ./decisions.jsonl sources: outcomes sets: 0 segments: 0 torn lines: 0 unorderable req_ts: 0 skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0 feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0 promotion registry — state\quality validation_state: empty auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25 this CLI process does not drive the automatic loop; the SERVER does, from boot (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`. validation runs (0) none — the ledger carries no judge subject on any segment with human labels. ``` Exit status is 0, and a run over nothing is an answer rather than an error. That capture is over a zero-byte ledger on a source build on Windows, which is why the state path renders with a backslash and the store count and the trailing `spend` note are elided here; three runs of it were byte-identical. The `auto_validate` values shown are the shipped defaults and were read from no config. **Why it happens.** Three things produce the same block, and the header tells them apart. `sets: 0` with `segments: 0` means the assembler found no judged row carrying a human label to pair against. `sets` above zero with `validation runs (0)` means labels exist and no judge subject does. And the header prints the path it read, which catches the third cause. **Fix it.** Take them in that order. - **Wrong ledger.** The log path resolves as `--log`, then `ledger.decision_log.path`, then the compiled default. A relative path in a config resolves against the working directory of *this* process, which need not be the one the server booted from. Pass `--log` explicitly and the question disappears. - **No judged rows.** [How a judge call happens](/docs/judges/how-judging-works) covers what decides whether a row gets graded; the first entry on this page covers a judge that is configured and silent. - **No labels.** [Collect the labels a judge is measured against](/docs/judges/collect-labels) is the procedure. **Confirm the fix.** `sets:` and `segments:` go above zero, and a `validation runs (1)` block appears with a subject line under it. ## The annotation queue answers 403 **Confirm it.** Read the body, which names the role it wanted. ```json { "status": 403, "detail": "role operator is below required admin", "required_role": "admin" } ``` **Why it happens.** A claimed item carries the request's stored prompt and completion verbatim, which sits behind a privileged content scope that only an admin-role credential holds. The write half is different: the outcomes door sits at the operator floor, so a person who can submit labels is not automatically a person who can be offered them. **Fix it.** Create each annotator with the admin role, before the server boots. ```console $ tokentriage user add annotator-a --role admin --state-dir ./state/dash ``` **Confirm the fix.** The same claim returns items, and the `annotators=[…]` list on a later assembly block carries the name the server derived from the session. ## The claim comes back with no items at HTTP 200 **Confirm it.** Check which strategy the request carried. The dashboard's annotate page preselects `hard_cases`, so a claim made by clicking rather than by hand is almost always that one. **Why it happens.** `hard_cases` ranks rows the judge found hard, and hardness has two ingredients: a judge that abstained, and a panel that split. A plain `pointwise-bool` judge that answered every row produces neither, so the queue has nothing to rank and answers with an empty list at HTTP 200. That is a statement about your traffic rather than a fault. **Fix it.** Claim with the other member of the closed pair. ```json { "strategy": "starved_segments", "k": 60 } ``` **Confirm the fix.** Items come back, each carrying a `priority_reason` naming how far the segment is from its label floor. Reach for `hard_cases` where you run a panel or a method that abstains, and for `starved_segments` everywhere else. ## Two annotators are labelling and agreement is unmeasured **Confirm it.** Read the `gold:` line on the record. ```console gold: admitted=79 annotators=2 licenses=false iaa=unmeasured (overlap_below_floor) revision=5dac3b3 refusal iaa_unmeasured: floor=IAAOverlapMinN need=20 have=19 overlapping rows ``` Two annotators, seventy-nine label events, and the widest overlap between any pair of them is nineteen requests against a floor of twenty. **Why it happens.** The lease and the measurement pull against each other. A lease exists to stop two annotators labelling one request by accident; agreement is computed on precisely the requests two annotators both labelled. Submitting does not release a lease, only expiry does, so one annotator working the queue takes every available row out of the other's reach for the lease duration. Used as it comes, the queue cannot hand you an overlap sample. **Fix it.** One of three, and only the first is a plan. - **Hand the overlap set to the second annotator out of band.** The submission door takes a `req_id` and does not consult the lease table, so a second annotator can label a request they were not offered. Decide the overlap first, at twenty requests or more, and let the bulk follow. - **Claim in turns, small.** With an hour-long lease and a claim of twenty-five, one annotator works a batch while the other waits for the lease to lapse. It is arithmetic you do yourself. - **Restart the server.** The lease table is in memory, so a restart releases every lease early. It is a recovery, not a workflow. **Confirm the fix.** The `gold:` line prints a number in place of `unmeasured`, computed over the shared requests rather than over the union of the two label sets. A null `iaa` records that agreement was not measured, and this build publishes no field saying which of four situations produced it. The four are one annotator, an empty gold set, no annotator pair sharing enough rows, and the statistic being undefined on the pair that does share them. The text output of `tokentriage evals validate` prints `unmeasured ()` and is the only surface that names it. A kappa of `0.0` is a different thing again: that is a measurement, and it means agreement at the level two coin flips would reach. ## A label was accepted and no counter moved **Confirm it.** Compare `admitted=` on the assembly block against the number of labels you sent, then read the two lines under it. ```console offered=40 admitted=40 (outcomes=40 feedback=0) requests=30 subjects=1 annotators=[annotator-a annotator-b] excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0 skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0 ``` If `admitted=` is short and `excluded:` and `skipped:` are both all zero, the missing labels are not being turned away anywhere. They are not there. **Why it happens.** The outcomes door does not check that the `req_id` on a submission names a row the ledger holds. A typo returns `{"accepted": 1}` at HTTP 200 and writes a companion row with nothing to join to, and that row then appears in no counter on the assembly block, not even as a skip. **Fix it.** Submit the `req_id` values the claim handed you rather than any you typed, and check `admitted=` against the number you sent after every batch. Two other causes shorten the same counter and do announce themselves on the `excluded:` line: an outcome with no attributed annotator, and one a later label superseded. A second label from the same annotator on the same request replaces the first, last write winning, so a correction lands cleanly and so does a mistake. **Confirm the fix.** `admitted=` matches what you submitted, and `requests=` matches the number of distinct requests behind it. ## The label count stays at zero on a subject the run lists **Confirm it.** Read the `judge=` line on the record, above the line that declines. ```console [1/1] judge=judge-groundedness model=… prompt=… eval_ver=… refusal validation_below_n: floor=JudgeValidationMinN need=50 have=0 admitted human-labeled rows ``` The subject is observed, listed and given a run of its own, and the count is zero on a ledger that holds judged rows. **Why it happens.** `judge-groundedness` writes a scalar score and no pass verdict, because the metric behind it publishes no pass floor and the runner forces the verdict field to nil on every row it grades. The gold-set assembler records a verdict only where one is set, so none of that judge's rows are admitted. The subject is appended to the census *before* that gate, which is why it appears on the reading at all. So the run tells you to collect labels, and no quantity of labels will move the number. **Fix it.** Configure `judge-task-completion` on the metric `task_completion`, whose descriptor carries a floor, for any judge that has to reach a promotion. ```yaml evals: evaluators: - id: judge-task-completion judge: external: allow model: gemini-2.5-flash api_key_env: JUDGE_API_KEY sample_rate: 0.20 max_usd_per_day: 20.0 ``` **Confirm the fix.** `admitted=` on the assembly line moves above zero for the new subject, and `have` on the refusal climbs with it. `judge-groundedness` remains usable as a `reported` signal; its verdict travels losslessly in the score it writes. ## The config is turned away at validate and again at boot **Confirm it.** Read the prefix, because the body is identical at both gates and the prefix is the only thing that says which one you hit. ```console $ tokentriage validate --config pairwise.yaml Error: configuration is invalid: evals.evaluators[1].judge.method: "pairwise" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel ``` Exit status is 1. Boot prints the same sentence under the prefix `Error: evals:` instead, after the promotion registry has already opened and the dashboard has already composed, so it arrives late in a boot log rather than at the top of it. **Why it happens.** Two judge method configurations compose from a config file: `pointwise-bool`, which is what an absent leaf reads as, and `panel` at `self_consistency_k: 1`. `pointwise-rubric`, `pairwise` and `reference` are turned away over a `judge:` transport, as is any `self_consistency_k` above 1, and every `sidecar_judge:` block. **Fix it.** Take the method the message names, or seat a panel. [Configure a judge](/docs/judges/configure-a-judge) has every key with the wording the binary prints, and [Run a cross-family panel](/docs/judges/run-a-panel) is the panel shape. **Confirm the fix.** `OK: configuration is valid` at exit 0, and then a boot that reaches `evals: judge evaluator wired`. Validating is not the same promise as booting, so do both. ## Every error and warning prints twice **Confirm it.** Count the copies. Every error and every warning from `tokentriage validate` and from `tokentriage run` appears twice, on unrelated configs and for unrelated causes, because this binary writes each one once through its command framework and once from `main`. **The fix is to change nothing.** The repetition belongs to the binary and says nothing about your config. Two copies of a message are one event. **What to do instead.** Read the *prefix* rather than the count, since that is the token that carries information: `configuration is invalid:` came from the validator and `evals:` came from boot. And treat a pasted capture showing a single copy as edited, including any you paste yourself. ## The quality floor admits nothing after a promotion landed **Confirm it.** Read one decision and look at the score table, which carries every candidate's quality verdict rather than only the admitted ones. ```console $ tokentriage policy explain --req-id --log ./state/decisions.jsonl ``` A `quality_below_min_lcb` term means the tier was measured, admitted to the comparison, and its bound fell short. A `quality_absent:` term means the quality could not be read at all, with the reason riding the term name. They are different facts and they call for different actions. **Why it happens.** Three causes are common and the first is the one nothing warns you about. - **The metric id is misspelled.** It is checked against no registry at validate time, so the file passes with `OK: configuration is valid` at exit 0 and then admits no candidate at runtime. Check the id against the evaluator's declared metrics by hand. - **The window predates the promotion.** A promotion is forward-only, so a report over a window that ends before it stays at the lower ceiling however good the judge turned out to be. - **The promotion is for another segment.** A promotion is filed under one judge id, metric and segment. The same judge on a different tier is a different subject. **Fix it.** [Gate a rollout on a validated judge](/docs/judges/gate-a-rollout) walks the whole path, including the calibration the floor reads its bound from. **Confirm the fix.** The candidate appears on the score table with no quality term against it, and traffic stops landing on `on_empty`. ## Shadow duplication writes nothing and says nothing **Confirm it.** Search the boot log for the line that says the feature came up. ```console $ tokentriage run --config tokentriage.yaml 2>&1 | grep 'shadow duplication enabled' ``` No match means the feature is off, whatever the config appears to say. **Why it happens.** `shadow.duplicate.max_usd_per_day` is part of the condition that switches the feature on, alongside the enable flag and a non-zero sample rate. Omit it, or write `0`, and the whole block is skipped: the file validates clean, the binary boots clean, and no line is written at either gate. Two configs differing by exactly that one leaf both report `OK` at exit 0, and only the capped one prints the boot line above and the egress warning beside it. **Fix it.** Give the leaf a positive dollar figure. ```yaml shadow: duplicate: enabled: true sample_rate: 0.05 max_usd_per_day: 5.0 ``` **Confirm the fix.** The boot line appears, and the shadow line on the metacost surface starts carrying dollars. [What judging costs, and how to cap it](/docs/judges/cost-and-budget) has the four planes that can call a grading model and which meter each reports to. ## Two traps in the vocabulary itself Both cost time in the same way: you go looking for a token, find nothing, and conclude the condition did not happen. **The counter and the code spell the same idea in opposite orders.** The assembly block's `skipped:` line carries `incomplete_subject`, and the protocol's closed vocabulary of refusals carries `subject_incomplete`. Grep for the wrong one and you find nothing at all. **You will meet the counter and not the code.** `subject_incomplete` is a precondition inside the validation routine, and the one caller that reaches that routine is handed subjects the assembler has already screened with the same test. So on this verb the condition surfaces as `incomplete_subject=N` on the `skipped:` line and never as a code. Rows counted there left before any gate saw them: no line naming a floor mentions them, and no floor moves when you collect more labels. ## What this page does not cover **What each gate measures.** The gold set, the overlap sample, the kappa floor and the swap gate, walked in the order the binary walks them, are [the walk that promotes a judge](/docs/judges/agreement). **Every closed vocabulary this track prints.** The abstention reasons, the validation floors, the promotion states and the disclosure fields, one plain line each, are [Judge vocabularies](/docs/reference/judge-vocabularies). **The gaps, as a list rather than as symptoms.** Several entries above are the reader-facing face of something this build does not do, and the catalogue of those, grouped by what stands in the way of each, is [Air-gap judging, and the known limits](/docs/judges/air-gap-and-limits).