# When a judge can't tell
Four named reasons a judge can decline, all four legal on a ledger row, and how few of them a configuration can reach.
Source: https://tokentriage.com/docs/judges/abstention
---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import JudgeAbstentionDiagram from '../../../components/JudgeAbstentionDiagram.astro';
A grader that answers every time is the one that looks best on a screen. Every sampled row comes
back with a score, the pass rate has a full denominator, and nothing on the page separates the rows
the grader understood from the rows it was guessing at. The guesses are the trouble: they arrive in
the same shape as the answers it got right and they move the number by the same amount.
So a graded item in this build can come back with a fourth answer beside pass, fail and winner:
*I decline, and here is why*. The vocabulary of reasons is closed at four, and all four are legal
on a stored row.
**This page is written for whoever puts code in front of those rows.** A ledger consumer, a
dashboard panel, an export, a report: anything reading a judged row has to handle all four values.
The membership check runs at the boundary where a container's string becomes a stored field, so any
of the four can arrive there. What an *operator* can produce is far narrower.
**None of the four is reachable from the configuration this section teaches**, and
[The limits](/docs/judges/abstention#the-limits) takes them one at a time.
[How a judge call happens](/docs/judges/how-judging-works) is the call itself, and
[the walk that promotes a judge](/docs/judges/agreement) is the agreement statistic underneath all
of it. There is no procedure on this page.
## What declining buys you
Here is the fourth answer doing its job. This build vendors LLMBar, a public 419-pair
meta-evaluation corpus, and drives scripted judges over every pair. One of them, `first-slot`,
follows a single rule: whichever response it is shown first is the better one. Under `pairwise` it
is asked about each pair twice, once each way round.
```text
first-slot/pairwise pairs=419 graded= 0 abstained=419 undecided= 0 calls= 838 agreement=undefined
first-slot/pointwise-bool pairs=419 graded= 0 abstained= 0 undecided=419 calls= 838 agreement=undefined
```
Shown a pair one way round it picked the first response. Shown the same pair reversed it picked the
first one again, which was now the other response. A verdict that moves when only the presentation
order did is not a verdict, so all 419 landed as abstentions.
Asked once per pair instead, the same judge would have named a winner on all 419, and nothing in the
output would have marked which part of that rate was position. The abstention is where the
difference gets written down. Those two lines are quoted out of a longer transcript that
[Position bias, self-preference and panels](/docs/judges/bias-controls) runs in full.
`rate` on that reading is `agreed` over `graded`, and `graded` is zero, so it comes back `null`,
meaning UNDEFINED rather than `0.0`. A rate of zero is something somebody measured. UNDEFINED is
the absence of a denominator. Collapsing the two publishes a judge that agreed with nobody where
the truth is a judge that answered nobody.
## An abstention is an answer, not a missing field
A verdict in this build sets one of four things and no more than one: a boolean, a score, a winner,
or an abstention. The fourth is not an error slot bolted to the side. It is a verdict-shaped answer
whose content is that there is no verdict, and the reason it carries is what makes it something to
act on. Two properties follow, and both are enforced rather than advised.
**A failure cannot become an abstention.** A parse failure, a timeout, a non-200 and a budget stop
each return no verdict and an error. None is converted into a declining answer on the way out,
because a fabricated abstention would tell an operator the instrument worked and chose to stay
quiet when in fact it broke.
**An abstention cannot carry a value.** A per-item row whose status reads `abstained` is turned
away if it also carries a `value` or a `pass`, because a `0` beside an abstention is a fabricated
fail. The rule runs the other way too. A row carrying an abstention reason under any other status
is turned away, and so is an abstention whose reason went missing, since a reason that vanished
cannot be told from a crash.
## The four reasons, and what produces each
The four are not interchangeable and they are not equally within reach. Each has one producer, and
the producers sit on different planes.
Read it left to right. One graded item on the left sets one answer of the four. The middle box is
the abstention itself, an outcome carrying a named reason rather than an empty field. The four
boxes on the right are the whole vocabulary, each labelled with its producer and that producer's
plane, and the bottom band is the counting rule. The fourth edge is drawn broken because its
producer sits behind a configuration leaf this build turns away.
| Reason | What fires it | The one thing that produces it | What it takes to reach |
|---|---|---|---|
| `panel_split` | the seats disagreed too evenly for a strict majority | the panel method's own tally | composes from a config file, and needs an even number of seats |
| `swap_inconsistent` | a pairwise trial's two orderings named different winners | the pairwise trial constructor, which takes both orderings and derives the verdict from them | `shadow.duplicate.pairwise`, on the plane whose judge has no endpoint leaf |
| `below_conformal` | the confidence feature fell under the fitted accept threshold | the selective gate | its one construction site outside the tests is called with no fitted region |
| `out_of_scope` | the input fell outside the judge's validated scope | a pass-through of a reason a sidecar judge container reported | `sidecar_judge:` is refused at `tokentriage validate` and again at boot |
**Only `panel_split` composes from a config file, and it needs an even number of seats.** A panel's
ballot vocabulary has two tokens, and every seat votes or the whole invocation is refused. So the
tally has at most two buckets over as many ballots as there are seats, and it decides whenever one
bucket holds more than half. On an odd panel one always does: three seats fall 2–1 or 3–0. A split
needs the buckets level, so the three-seat panel
[the rest of this section builds](/docs/judges/run-a-panel) cannot produce one, and four seats is
the smallest shape that can.
**The other three sit behind something a config file does not reach.**
- **`swap_inconsistent`** comes from a pairwise trial. Pairwise over an `evals.evaluators[].judge`
transport is turned away at `tokentriage validate` and again at boot, so the trial reaches
production only through `shadow.duplicate.pairwise`, on a plane whose judge takes no endpoint leaf
and dials a compiled-in provider address.
- **`below_conformal`** comes from the selective gate. Its one construction site outside the tests
is handed no fitted region, so the shipped sampler compiles its judging plan with no accept
region installed, and a test of the product's own pins that it does. The stated reason: a gated
executor's declined rows would be censored out of the evidence a future accept region is fitted
from.
- **`out_of_scope`** has one producer, the sidecar transport's reason mapper, reached from two call
sites on that same transport. It is a pass-through, so a container reports a reason and the
mapper admits it into the closed vocabulary. Every `sidecar_judge:` leaf is refused at both verbs.
## Where the count lands
An abstention that vanished would be worse than a wrong answer, because a wrong answer at least
leaves a row to argue with. Every one of them is counted, and the shape of the count depends on
which surface produced it.
**On the evals plane, the count is the whole disclosure.** A panel split produces no outcome, so no
row is written, and the evaluation is tallied under the named skip reason `abstained` on that
evaluator's counters. That reason sits beside `evaluator_error` and `cancelled` rather than inside
them, because an operator who reads `evaluator_error` goes looking for a bug that is not there. The
vote vector is not lost with the row: the split carries the whole panel disclosure out with it,
members, votes, rule and margin together.
**On a judged row, the reason travels with the row.** A shadow grade carries `abstain_reason`
beside the judge disclosure, so the row itself says which of the four applied.
**On a per-item reading, the reason is the key.** Each row carries a status from a closed set, and
`abstained` is one of five values there, beside `graded`, `skipped_no_body`, `score_failed` and
`missing_from_ledger`. The reading then buckets those rows by reason. A reason nothing produced is
absent from that map rather than sitting in it at zero, which is the discipline the rest of this
surface follows.
**What it does to a rate.** An abstention moves neither half of the fraction: out of the numerator
because no verdict was written, out of the denominator because the denominator is graded
observations. The effect lands on N instead. Your figure keeps its meaning and loses evidence, and a
segment that loses enough of it comes back `insufficient` carrying `need` and `have` counts rather
than a quiet number. [Why your number is reported, not
measured](/docs/judges/basis-ceilings) reads that one.
## What an abstention is not
**Not a failure of the instrument.** `score_failed` is a fact about the instrument and `abstained`
is an answer about the item. A seat that cast no ballot, a member that failed and a budget gate that
stopped a fan-out part-way all end the invocation with a refusal for the same reason, so
`panel_split` arrives only after every seat has voted.
**Not undecided.** Look at the `first-slot` pointwise row again: 419 undecided and nothing
abstained. That judge answered every time and graded both arms the same, so the derivation found no
preference. The two are counted in different columns.
**Not a null on a judges row.** `iaa: null` records that annotator agreement was not measured, and
a κ of `0.0` is a measurement at chance level. Neither is an abstention, and the table separating
them is on [the walk that promotes a judge](/docs/judges/agreement).
**Not this run's verdict, when it arrives as `ledger_abstain_reason`.** A per-item row carries that
field to say what a judged subject had already recorded on the ledger row before this run started.
It is provenance about the row rather than this run's own outcome, and a row a shadow judge
declined can still be graded by a different evaluator here.
## The limits
**None of the four is reachable on the configuration this section teaches.** The one that composes
will not fire on a three-seat panel, and the other three sit on a plane or behind a leaf a config
file does not reach. The preceding section names each blocker. Paying a provider does not close the
gap either: `below_conformal` stays out of reach with money on the table, because the blocker is a
missing accept region rather than a missing call. A page that presented the four as four switches
would be describing a build nobody has.
**The benchmark reading has no operator surface.** The two rows quoted here come from the package's
acceptance run. `evalprog.RunBenchmark` has no call site outside its tests, so the agreement census,
its `abstained` column and its UNDEFINED rate are not served by a verb or an endpoint in this build.
**On the experiments surface, an abstention still shares an arm-level counter.** When an eval
experiment re-scores stored outputs, a declining judge is returned as an error, so the item stays in
that arm's denominator and is counted under the same census field as a scorer fault. The per-item
row keeps the two typed apart, which is where to read them. The arm-level census does not yet
separate them, and the product records that residual against the file that owns it.
## What this page does not cover
**Why a judge flips when the candidates swap places.** The both-orderings trial, the ceiling its
rate is measured against, the same-family rule and the whole LLMBar run those two rows came from are
[Position bias, self-preference and panels](/docs/judges/bias-controls).
**Every other closed set, as a lookup.** The validation floors, the promotion states and the
disclosure fields, one plain line each, are
[Judge vocabularies](/docs/reference/judge-vocabularies).