# LLM judges
Why the grader needs grading too — what an LLM judge scores, and what that score is allowed to license until somebody measures it.
Source: https://tokentriage.com/docs/judges/overview
---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import QualitySubstrateDiagram from '../../../components/QualitySubstrateDiagram.astro';
You want to move a slice of traffic onto a cheaper model, and one question decides whether you
can: did the answers get worse? Nobody is going to read four thousand of them, so you hand the
reading to a model. It scores every output, you average the scores, and you have a quality number.
The number arrives with a problem attached to it. The grader is a language model too. It has its
own preferences, its own blind spots and its own bad days, and it returns a score with the same
confidence on the rows it got wrong as on the rows it got right. Whatever error it carries, your
quality number now carries.
An LLM judge is a measuring instrument, and a measuring instrument's own error is a thing you
measure. That is the sentence this section rests on. Everything in it is either the instrument,
the measurement of the instrument, or the rule that decides what the second one lets the first
one claim.
This page is the map, and there is no configuration on it. If you have read
[Calibration & Evals](/docs/calibration/overview) you already have the vocabulary below. If you
would rather start by doing, the [quickstart](/docs/judges/quickstart) grades one request end to
end, on two provider keys and your own money.
## Where a judge sits
A judge is not a subsystem of its own. It is one kind of evaluator, plugged into machinery that
already exists for turning scored outputs into a number a router can read, and the quickest way
to see what a judge is for is to see that whole path at once. Read the following diagram along
the top row left to right, then along the bottom row right to left.
The decision ledger holds every request. Evaluators score what is in it, and a judge is one of
the kinds named in that middle box, beside the rule checks and the code checks. Out comes a
quality estimate, and the estimate carries a **basis**: a label saying what kind of claim the
number is. Calibration fits and gates those estimates into an artifact, the router consults the
artifact through `min_quality_lcb`, and the dashed line at the bottom is the offline evals track,
which asks the same questions of stored traffic rather than of live traffic.
The basis is the part of that picture this section is about. It is where a judge's own
trustworthiness is recorded, and it is why a judge score cannot walk straight into a routing
decision.
## Reported, and what it takes to move
A quality estimate does not travel as a bare number. It carries its basis wherever it goes, and
three values are worth knowing before you read anything else here.
Two words in the table below do the work, so take them first. A **segment** is one addressed slice
of traffic, written as a single string: `task_completion|tier=frontier|model=gpt-5|endpoint=chat` is
the metric plus the dimensions the request was served under. A **subject** is what this build
measures and promotes: one judge, on one metric, in one segment, pinned to that judge's model, its
prompt and its evaluator version. So there is no validated judge here, only a promoted subject,
and the protocol that promotes one is the thing the corpus and the binary both call **B.7**.
| Basis | What it means |
|---|---|
| `reported` | A judge scored the output, and nobody has measured that judge against human labels on this segment. Displayable and filterable. |
| `measured` | A promotion stands for this row's subject, so the judge's verdicts are admitted where a measurement is required. |
| `insufficient` | Not enough evidence to say anything yet. The entry carries a reason and `need`/`have` counts rather than a quiet `0.0`. |
A judge score starts at `reported`. `min_quality_lcb`, the quality floor a router checks before
it admits a cheaper tier, reads a `measured` basis only, and a judge verdict stays `reported`
until its subject is promoted. An unvalidated judge can fill a dashboard and cannot move a router.
A promotion is only half of what that floor needs. It lifts the ceiling so judged evidence may be
treated as a measurement, and an `active` quality calibration over the same segment is what turns
those measurements into the bound the floor is compared against. Two artefacts, two lifecycles, and
[Gate a rollout on a validated judge](/docs/judges/gate-a-rollout) checks both in order.
The money is a third question, and a promotion does not answer it either. The realized dollar is a
per-row cost counterfactual against the tier that would have served, and it does not depend on
grading. Nothing attaches a quality estimate to a realized record on a default deployment.
`Realizer.QualityEstimateFor` is the only route between the two, and it is nil at its only
non-test construction site, which the product calls the shipped default.
A judge's score says *how good*. Its basis says *how much anybody has checked the judge*. Those
are two different claims, and the second one decides whether the first is allowed anywhere near a
routing decision.
Moving a row from the first line of that table to the second is what
[How a judge reaches a measured basis](/docs/judges/agreement) covers: a gold set of human labels,
Cohen's κ computed against them, and an ordered walk that either promotes the subject or names the
floor it missed. That page also takes the subject address apart, coordinate by coordinate.
A figure can also stay `reported` for a reason that has nothing to do with the judge, because the
read path takes the lowest of several caps and only the judge-tied ones consult the promotion
registry. That is [Why your number is reported, not measured](/docs/judges/basis-ceilings).
## Nothing is being judged here, and that is an answer
Before any of that, it is worth seeing what this machinery does when none of it has been
configured. Run the validation check over an empty ledger and it prints a claim about the ledger
rather than an empty screen.
```console
$ tokentriage evals validate --log ./decisions.jsonl --state-dir ./state
gold-set assembly — ./decisions.jsonl
sources: outcomes sets: 0 segments: 0 torn lines: 0 unorderable req_ts: 0
skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0
feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0
promotion registry — state\quality
validation_state: empty
auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25
this CLI process does not drive the automatic loop; the SERVER does, from boot
(`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`.
validation runs (0)
none — the ledger carries no judge subject on any segment with human labels.
No promotion stands (validation_state: empty), so every judge is `reported` — which is
what no persisted evidence must mean (§2.6).
(`spend` is NOT MEASURED and is reported as nil, not as $0: this loop cannot
spend — no path it executes reaches a judge call, which JG4-06 restated from judgegate's
import posture after an injected re-scorer seam made that argument unsound — and while a
shadow row now records the judge id and metric its judging dollars belong to, no shipped producer SUMS those dollars onto a subject.)
jval_ artifacts in the store: 0 newest: (none)
```
That capture came off a source build on Windows, so `state\quality` is the resolved `--state-dir`
rendered with the platform separator, and a Linux or macOS run prints `state/quality`.
`./decisions.jsonl` is the path this run passed to `--log`, and yours will be your own.
`interval=24h0m0s` and `new_labels_threshold=25` are the shipped defaults for
`evals.validation.auto_validate`, not values a config supplied here. Nothing else in the block is
machine-derived: it carries no timestamp, no port, no run id and no version string. Exit status
was 0, because a validation run over nothing is not an error.
Three things in there are this section in miniature. `validation_state: empty` is a claim about
the ledger, not a rendering failure. The line under `validation runs (0)` gives the cause, which
is that the ledger carries no judge subject on any segment with human labels, so nothing is
promoted and every judge is `reported`. And the parenthesis at the bottom is the binary declining
to publish a figure it has not computed: `spend` is served as nil rather than as `$0`, because
judging costs money per row and no shipped producer sums those dollars onto a subject.
## What this section covers
The explanations build the model: what one judge call does to a request and where its disclosure
lands, how agreement with human labels is computed, what happens when a judge cannot tell, and
why a figure stays `reported`. The how-tos are the loop you run: configure a judge, collect
labels from two annotators, run the validation check, and turn a promoted judge into a routing
decision. The reference pages are the exact surface: the methods this build registers, the closed
vocabularies it prints, and the judge endpoints.
One fact belongs on this page rather than three pages later, because it bounds what all of those
can teach. Two judging setups compose from a config file in this build: `pointwise-bool`, which
is what an absent `method:` reads as, and `panel` at `self_consistency_k: 1`.
The other three methods this build registers are turned away over a `judge:` transport at
`tokentriage validate` and again at boot, and the message names the method it stopped rather than
grading with something else. [Configure a judge](/docs/judges/configure-a-judge) is that surface,
and it doubles as the catalogue of configurations this build turns away.
## The limits, and where the rest of them are
Three limits change what a clean result means, and each is stated below in the terms the binary
uses for it.
**A promoted judge is not monitored.** `Gate.ObserveAgreement` has no call site outside its
tests, so no shipped binary can fire a drift demotion. The arithmetic is demonstrated inside the
test binary against a pinned vector; what is missing is the point where a human label would
arrive. Re-running the validation check on a schedule of your own is the substitute, and a
version move (a new model, a new prompt sha, a new evaluator version) resets a promotion by
itself.
**There is no self-hosted judge path in this build.** `sidecar_judge:` is refused at
`tokentriage validate` and again at boot, in the binary's own words: the leaf is config-complete
and composition-incomplete. The *leaf* is the config key itself, at the end of its path: it parses,
and the composition step behind it is what the refusal names. A judge reaches production over the
provider-API transport, with `external: allow`, or not at all.
**`iaa: null` is not agreement of zero.** It means inter-annotator agreement was not measured. A
κ of `0.0` is a measurement and means agreement at chance level; `kappa: null` means a gate
stopped the run before κ could be computed.
Those three are the shape of it, not the list.
[Air-gap judging, and the known limits](/docs/judges/air-gap-and-limits) is the catalogue. The
capture above is also the shortest demonstration that this build states such things itself: the
note about `spend` is the binary's sentence, not this page's.