When a judge can't tell

A grader that answers every time is the one that looks best on a screen. Every sampled row comes back with a score, the pass rate has a full denominator, and nothing on the page separates the rows the grader understood from the rows it was guessing at. The guesses are the trouble: they arrive in the same shape as the answers it got right and they move the number by the same amount.

So a graded item in this build can come back with a fourth answer beside pass, fail and winner: I decline, and here is why. The vocabulary of reasons is closed at four, and all four are legal on a stored row.

This page is written for whoever puts code in front of those rows. A ledger consumer, a dashboard panel, an export, a report: anything reading a judged row has to handle all four values. The membership check runs at the boundary where a container’s string becomes a stored field, so any of the four can arrive there. What an operator can produce is far narrower. None of the four is reachable from the configuration this section teaches, and The limits takes them one at a time.

How a judge call happens is the call itself, and the walk that promotes a judge is the agreement statistic underneath all of it. There is no procedure on this page.

What declining buys you

Here is the fourth answer doing its job. This build vendors LLMBar, a public 419-pair meta-evaluation corpus, and drives scripted judges over every pair. One of them, first-slot, follows a single rule: whichever response it is shown first is the better one. Under pairwise it is asked about each pair twice, once each way round.

  first-slot/pairwise        pairs=419 graded=  0 abstained=419 undecided=  0 calls= 838  agreement=undefined
  first-slot/pointwise-bool  pairs=419 graded=  0 abstained=  0 undecided=419 calls= 838  agreement=undefined

Shown a pair one way round it picked the first response. Shown the same pair reversed it picked the first one again, which was now the other response. A verdict that moves when only the presentation order did is not a verdict, so all 419 landed as abstentions.

Asked once per pair instead, the same judge would have named a winner on all 419, and nothing in the output would have marked which part of that rate was position. The abstention is where the difference gets written down. Those two lines are quoted out of a longer transcript that Position bias, self-preference and panels runs in full.

An abstention is an answer, not a missing field

A verdict in this build sets one of four things and no more than one: a boolean, a score, a winner, or an abstention. The fourth is not an error slot bolted to the side. It is a verdict-shaped answer whose content is that there is no verdict, and the reason it carries is what makes it something to act on. Two properties follow, and both are enforced rather than advised.

A failure cannot become an abstention. A parse failure, a timeout, a non-200 and a budget stop each return no verdict and an error. None is converted into a declining answer on the way out, because a fabricated abstention would tell an operator the instrument worked and chose to stay quiet when in fact it broke.

An abstention cannot carry a value. A per-item row whose status reads abstained is turned away if it also carries a value or a pass, because a 0 beside an abstention is a fabricated fail. The rule runs the other way too. A row carrying an abstention reason under any other status is turned away, and so is an abstention whose reason went missing, since a reason that vanished cannot be told from a crash.

The four reasons, and what produces each

The four are not interchangeable and they are not equally within reach. Each has one producer, and the producers sit on different planes.

Read it left to right. One graded item on the left sets one answer of the four. The middle box is the abstention itself, an outcome carrying a named reason rather than an empty field. The four boxes on the right are the whole vocabulary, each labelled with its producer and that producer’s plane, and the bottom band is the counting rule. The fourth edge is drawn broken because its producer sits behind a configuration leaf this build turns away.

Reason	What fires it	The one thing that produces it	What it takes to reach
panel_split	the seats disagreed too evenly for a strict majority	the panel method’s own tally	composes from a config file, and needs an even number of seats
swap_inconsistent	a pairwise trial’s two orderings named different winners	the pairwise trial constructor, which takes both orderings and derives the verdict from them	shadow.duplicate.pairwise, on the plane whose judge has no endpoint leaf
below_conformal	the confidence feature fell under the fitted accept threshold	the selective gate	its one construction site outside the tests is called with no fitted region
out_of_scope	the input fell outside the judge’s validated scope	a pass-through of a reason a sidecar judge container reported	sidecar_judge: is refused at tokentriage validate and again at boot

Only panel_split composes from a config file, and it needs an even number of seats. A panel’s ballot vocabulary has two tokens, and every seat votes or the whole invocation is refused. So the tally has at most two buckets over as many ballots as there are seats, and it decides whenever one bucket holds more than half. On an odd panel one always does: three seats fall 2–1 or 3–0. A split needs the buckets level, so the three-seat panel the rest of this section builds cannot produce one, and four seats is the smallest shape that can.

The other three sit behind something a config file does not reach.

• swap_inconsistent comes from a pairwise trial. Pairwise over an evals.evaluators[].judge transport is turned away at tokentriage validate and again at boot, so the trial reaches production only through shadow.duplicate.pairwise, on a plane whose judge takes no endpoint leaf and dials a compiled-in provider address.
• below_conformal comes from the selective gate. Its one construction site outside the tests is handed no fitted region, so the shipped sampler compiles its judging plan with no accept region installed, and a test of the product’s own pins that it does. The stated reason: a gated executor’s declined rows would be censored out of the evidence a future accept region is fitted from.
• out_of_scope has one producer, the sidecar transport’s reason mapper, reached from two call sites on that same transport. It is a pass-through, so a container reports a reason and the mapper admits it into the closed vocabulary. Every sidecar_judge: leaf is refused at both verbs.

Where the count lands

An abstention that vanished would be worse than a wrong answer, because a wrong answer at least leaves a row to argue with. Every one of them is counted, and the shape of the count depends on which surface produced it.

On the evals plane, the count is the whole disclosure. A panel split produces no outcome, so no row is written, and the evaluation is tallied under the named skip reason abstained on that evaluator’s counters. That reason sits beside evaluator_error and cancelled rather than inside them, because an operator who reads evaluator_error goes looking for a bug that is not there. The vote vector is not lost with the row: the split carries the whole panel disclosure out with it, members, votes, rule and margin together.

On a judged row, the reason travels with the row. A shadow grade carries abstain_reason beside the judge disclosure, so the row itself says which of the four applied.

On a per-item reading, the reason is the key. Each row carries a status from a closed set, and abstained is one of five values there, beside graded, skipped_no_body, score_failed and missing_from_ledger. The reading then buckets those rows by reason. A reason nothing produced is absent from that map rather than sitting in it at zero, which is the discipline the rest of this surface follows.

What it does to a rate. An abstention moves neither half of the fraction: out of the numerator because no verdict was written, out of the denominator because the denominator is graded observations. The effect lands on N instead. Your figure keeps its meaning and loses evidence, and a segment that loses enough of it comes back insufficient carrying need and have counts rather than a quiet number. Why your number is reported, not measured reads that one.

What an abstention is not

Not a failure of the instrument. score_failed is a fact about the instrument and abstained is an answer about the item. A seat that cast no ballot, a member that failed and a budget gate that stopped a fan-out part-way all end the invocation with a refusal for the same reason, so panel_split arrives only after every seat has voted.

Not undecided. Look at the first-slot pointwise row again: 419 undecided and nothing abstained. That judge answered every time and graded both arms the same, so the derivation found no preference. The two are counted in different columns.

Not a null on a judges row. iaa: null records that annotator agreement was not measured, and a κ of 0.0 is a measurement at chance level. Neither is an abstention, and the table separating them is on the walk that promotes a judge.

Not this run’s verdict, when it arrives as ledger_abstain_reason. A per-item row carries that field to say what a judged subject had already recorded on the ledger row before this run started. It is provenance about the row rather than this run’s own outcome, and a row a shadow judge declined can still be graded by a different evaluator here.

The limits

None of the four is reachable on the configuration this section teaches. The one that composes will not fire on a three-seat panel, and the other three sit on a plane or behind a leaf a config file does not reach. The preceding section names each blocker. Paying a provider does not close the gap either: below_conformal stays out of reach with money on the table, because the blocker is a missing accept region rather than a missing call. A page that presented the four as four switches would be describing a build nobody has.

The benchmark reading has no operator surface. The two rows quoted here come from the package’s acceptance run. evalprog.RunBenchmark has no call site outside its tests, so the agreement census, its abstained column and its UNDEFINED rate are not served by a verb or an endpoint in this build.

On the experiments surface, an abstention still shares an arm-level counter. When an eval experiment re-scores stored outputs, a declining judge is returned as an error, so the item stays in that arm’s denominator and is counted under the same census field as a scorer fault. The per-item row keeps the two typed apart, which is where to read them. The arm-level census does not yet separate them, and the product records that residual against the file that owns it.

What this page does not cover

Why a judge flips when the candidates swap places. The both-orderings trial, the ceiling its rate is measured against, the same-family rule and the whole LLMBar run those two rows came from are Position bias, self-preference and panels.

Every other closed set, as a lookup. The validation floors, the promotion states and the disclosure fields, one plain line each, are Judge vocabularies.