LLM judges

You want to move a slice of traffic onto a cheaper model, and one question decides whether you can: did the answers get worse? Nobody is going to read four thousand of them, so you hand the reading to a model. It scores every output, you average the scores, and you have a quality number.

The number arrives with a problem attached to it. The grader is a language model too. It has its own preferences, its own blind spots and its own bad days, and it returns a score with the same confidence on the rows it got wrong as on the rows it got right. Whatever error it carries, your quality number now carries.

An LLM judge is a measuring instrument, and a measuring instrument’s own error is a thing you measure. That is the sentence this section rests on. Everything in it is either the instrument, the measurement of the instrument, or the rule that decides what the second one lets the first one claim.

This page is the map, and there is no configuration on it. If you have read Calibration & Evals you already have the vocabulary below. If you would rather start by doing, the quickstart grades one request end to end, on two provider keys and your own money.

Where a judge sits

A judge is not a subsystem of its own. It is one kind of evaluator, plugged into machinery that already exists for turning scored outputs into a number a router can read, and the quickest way to see what a judge is for is to see that whole path at once. Read the following diagram along the top row left to right, then along the bottom row right to left.

The decision ledger holds every request. Evaluators score what is in it, and a judge is one of the kinds named in that middle box, beside the rule checks and the code checks. Out comes a quality estimate, and the estimate carries a basis: a label saying what kind of claim the number is. Calibration fits and gates those estimates into an artifact, the router consults the artifact through min_quality_lcb, and the dashed line at the bottom is the offline evals track, which asks the same questions of stored traffic rather than of live traffic.

The basis is the part of that picture this section is about. It is where a judge’s own trustworthiness is recorded, and it is why a judge score cannot walk straight into a routing decision.

Reported, and what it takes to move

A quality estimate does not travel as a bare number. It carries its basis wherever it goes, and three values are worth knowing before you read anything else here.

Two words in the table below do the work, so take them first. A segment is one addressed slice of traffic, written as a single string: task_completion|tier=frontier|model=gpt-5|endpoint=chat is the metric plus the dimensions the request was served under. A subject is what this build measures and promotes: one judge, on one metric, in one segment, pinned to that judge’s model, its prompt and its evaluator version. So there is no validated judge here, only a promoted subject, and the protocol that promotes one is the thing the corpus and the binary both call B.7.

Basis	What it means
reported	A judge scored the output, and nobody has measured that judge against human labels on this segment. Displayable and filterable.
measured	A promotion stands for this row’s subject, so the judge’s verdicts are admitted where a measurement is required.
insufficient	Not enough evidence to say anything yet. The entry carries a reason and need/have counts rather than a quiet 0.0.

A judge score starts at reported. min_quality_lcb, the quality floor a router checks before it admits a cheaper tier, reads a measured basis only, and a judge verdict stays reported until its subject is promoted. An unvalidated judge can fill a dashboard and cannot move a router.

A promotion is only half of what that floor needs. It lifts the ceiling so judged evidence may be treated as a measurement, and an active quality calibration over the same segment is what turns those measurements into the bound the floor is compared against. Two artefacts, two lifecycles, and Gate a rollout on a validated judge checks both in order.

The money is a third question, and a promotion does not answer it either. The realized dollar is a per-row cost counterfactual against the tier that would have served, and it does not depend on grading. Nothing attaches a quality estimate to a realized record on a default deployment. Realizer.QualityEstimateFor is the only route between the two, and it is nil at its only non-test construction site, which the product calls the shipped default.

Moving a row from the first line of that table to the second is what How a judge reaches a measured basis covers: a gold set of human labels, Cohen’s κ computed against them, and an ordered walk that either promotes the subject or names the floor it missed. That page also takes the subject address apart, coordinate by coordinate.

A figure can also stay reported for a reason that has nothing to do with the judge, because the read path takes the lowest of several caps and only the judge-tied ones consult the promotion registry. That is Why your number is reported, not measured.

Nothing is being judged here, and that is an answer

Before any of that, it is worth seeing what this machinery does when none of it has been configured. Run the validation check over an empty ledger and it prints a claim about the ledger rather than an empty screen.

$ tokentriage evals validate --log ./decisions.jsonl --state-dir ./state
gold-set assembly — ./decisions.jsonl
  sources: outcomes   sets: 0   segments: 0   torn lines: 0   unorderable req_ts: 0
  skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0
  feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0

promotion registry — state\quality
  validation_state: empty
  auto_validate policy (configured): enabled=false interval=24h0m0s new_labels_threshold=25
    this CLI process does not drive the automatic loop; the SERVER does, from boot
    (`tokentriage run`). THIS run is operator-forced either way and does not consult `enabled`.

validation runs (0)
  none — the ledger carries no judge subject on any segment with human labels.
  No promotion stands (validation_state: empty), so every judge is `reported` — which is
  what no persisted evidence must mean (§2.6).
  (`spend` is NOT MEASURED and is reported as nil, not as $0: this loop cannot
  spend — no path it executes reaches a judge call, which JG4-06 restated from judgegate's
  import posture after an injected re-scorer seam made that argument unsound — and while a
  shadow row now records the judge id and metric its judging dollars belong to, no shipped producer SUMS those dollars onto a subject.)

jval_ artifacts in the store: 0   newest: (none)

That capture came off a source build on Windows, so state\quality is the resolved --state-dir rendered with the platform separator, and a Linux or macOS run prints state/quality. ./decisions.jsonl is the path this run passed to --log, and yours will be your own. interval=24h0m0s and new_labels_threshold=25 are the shipped defaults for evals.validation.auto_validate, not values a config supplied here. Nothing else in the block is machine-derived: it carries no timestamp, no port, no run id and no version string. Exit status was 0, because a validation run over nothing is not an error.

Three things in there are this section in miniature. validation_state: empty is a claim about the ledger, not a rendering failure. The line under validation runs (0) gives the cause, which is that the ledger carries no judge subject on any segment with human labels, so nothing is promoted and every judge is reported. And the parenthesis at the bottom is the binary declining to publish a figure it has not computed: spend is served as nil rather than as $0, because judging costs money per row and no shipped producer sums those dollars onto a subject.

What this section covers

The explanations build the model: what one judge call does to a request and where its disclosure lands, how agreement with human labels is computed, what happens when a judge cannot tell, and why a figure stays reported. The how-tos are the loop you run: configure a judge, collect labels from two annotators, run the validation check, and turn a promoted judge into a routing decision. The reference pages are the exact surface: the methods this build registers, the closed vocabularies it prints, and the judge endpoints.

One fact belongs on this page rather than three pages later, because it bounds what all of those can teach. Two judging setups compose from a config file in this build: pointwise-bool, which is what an absent method: reads as, and panel at self_consistency_k: 1.

The other three methods this build registers are turned away over a judge: transport at tokentriage validate and again at boot, and the message names the method it stopped rather than grading with something else. Configure a judge is that surface, and it doubles as the catalogue of configurations this build turns away.

The limits, and where the rest of them are

Three limits change what a clean result means, and each is stated below in the terms the binary uses for it.

A promoted judge is not monitored. Gate.ObserveAgreement has no call site outside its tests, so no shipped binary can fire a drift demotion. The arithmetic is demonstrated inside the test binary against a pinned vector; what is missing is the point where a human label would arrive. Re-running the validation check on a schedule of your own is the substitute, and a version move (a new model, a new prompt sha, a new evaluator version) resets a promotion by itself.

There is no self-hosted judge path in this build. sidecar_judge: is refused at tokentriage validate and again at boot, in the binary’s own words: the leaf is config-complete and composition-incomplete. The leaf is the config key itself, at the end of its path: it parses, and the composition step behind it is what the refusal names. A judge reaches production over the provider-API transport, with external: allow, or not at all.

iaa: null is not agreement of zero. It means inter-annotator agreement was not measured. A κ of 0.0 is a measurement and means agreement at chance level; kappa: null means a gate stopped the run before κ could be computed.

Those three are the shape of it, not the list. Air-gap judging, and the known limits is the catalogue. The capture above is also the shortest demonstration that this build states such things itself: the note about spend is the binary’s sentence, not this page’s.