# Configure a judge
Write the judge block, get it past validate and past boot, and know which shapes this build turns away before you paste one.
Source: https://tokentriage.com/docs/judges/configure-a-judge
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
You have decided to grade a slice of your traffic with a model, and that decision is now a config
file. Four steps take it from nothing to a running judge: paste the block, check the method you
named against what this build composes, ask the binary, then read one boot line.
The surface is smaller than you would guess. One block under one evaluator entry, five leaves that
carry weight, and a daily dollar cap the loader wants before it will accept any of it.
```yaml
evals:
outcomes:
api_enabled: true
evaluators:
- id: judge-task-completion
judge:
external: allow
model: gemini-2.5-flash
api_key_env: JUDGE_API_KEY
sample_rate: 0.20
max_usd_per_day: 20.0
```
If you have not seen a graded row yet, [How a judge call
happens](/docs/judges/how-judging-works) is the model under all of this, and the
[quickstart](/docs/judges/quickstart) gets you one. The exhaustive per-key table lives in
[Evals & calibration configuration](/docs/reference/evals-config); this page is the block you paste
and the blocks you cannot.
## 1. Paste the block into a config that has a dashboard
That block needs `dashboard.enabled: true` in the same file. The leaf mounts `/tt/api/v1`, and the
endpoint your annotators write labels through is served there. Take the door itself from
[Collect the labels a judge is measured against](/docs/judges/collect-labels).
**Keep the id `judge-task-completion`.** Two judge evaluators are registered and only one of them
can be carried through a validation run. `judge-groundedness` publishes no pass floor, so the runner
leaves every row it grades without a verdict, and no gold set forms for it however long you collect.
Five leaves carry weight, and nothing under `evals:` is filled in behind you. Each absent-reading
here is applied by a named consumer at composition time, which is why an omitted leaf has a meaning
rather than a placeholder.
| Leaf | Required? | What an absent leaf reads as |
|---|---|---|
| `sample_rate` | **yes**, and not defaulted | nothing. Omitting it is refused, because `0` and `1` are both plausible readings and one of them spends money |
| `max_usd_per_day` | **yes** whenever `judge:` is present | nothing. Omitting it is refused. `0` is legal and stops the evaluator at its first priced call |
| `judge.external` | **yes**, the literal token `allow` | closed. Any other value, and an absent leaf, are refused identically |
| `judge.model` | **yes**, on every method including `panel` | nothing. Omitting it is refused |
| `judge.api_key_env` | **yes** | nothing. Omitting it is refused. It is an env-var **name** matching `^[A-Z_][A-Z0-9_]*$`, and a literal secret written here is refused |
Everything else under `judge:` is optional, and every one of those defaults is in
[Evals & calibration configuration](/docs/reference/evals-config). Two of the optional leaves are
worth knowing by name now. `judge.endpoint_env` names a variable holding the URL the judge posts to,
which is how an evals-plane judge is pointed at a listener on your own box. `judge.method` picks the
method step 2 checks.
`evals.outcomes.api_enabled` beside the block is read by the config loader and by nothing else,
which catches one combination: the flag set against a disabled dashboard. It does not open the
door. Measured on this build, with the leaf deleted from the file entirely,
`POST /tt/api/v1/outcomes` answered `{"accepted":1}`. Keep it set, because it is what a reviewer
reads.
`judge.model` and `judge.api_key_env` stay required on `method: panel`, even though no seat reads
either. Both fields are unconditional, so a panel block written without them is stopped twice in one
message:
```console
$ tokentriage validate --config panel-no-model.yaml
Error: configuration is invalid: config panel-no-model.yaml: evals.evaluators[0].judge.model: required (the model id sent to the judge provider). §4 row 6 records what an unset judge model costs: the shipped sampler forwarded its grading-path selector as the model id and every verdict came back nil, for months, invisibly
evals.evaluators[0].judge.api_key_env: required (an env var NAME holding the judge provider's API key, e.g. ANTHROPIC_API_KEY). A judge is not a proxied request: there is no inbound credential to forward, so an absent key env is an evaluator that cannot authenticate
```
Exit status 1, and the binary prints that whole message twice; one copy is shown here, for the
reason step 3 gives. It carries a `config :` segment that the method message in step 2 does
not. That segment is how you tell the two layers apart: the loader raised this one while reading the
file, and the composition planner raises the other after the file has decoded.
The clause about `§4 row 6` is the product quoting its own history at you, and it is worth reading
once. A shipped sampler sent its grading-path selector where the model id belonged, every verdict
came back nil, and no surface said so for months. That is why the leaf is unconditional today rather
than defaulted.
Seats, families, the vote vector and the call multiplier are
[Run a cross-family panel](/docs/judges/run-a-panel).
## 2. Check that the method you named composes
The judge method vocabulary has five tokens and this build composes two of them from a config file.
That gap is the single most expensive thing to discover after you have written the file, so it is a
table rather than a paragraph.
| Composes from a config file | Turned away at `validate` and again at boot |
|---|---|
| `judge.method: pointwise-bool`, and an omitted `method:` | `judge.method: pointwise-rubric` |
| `judge.method: panel` with three seats from two model families, at `self_consistency_k: 1` or absent | `judge.method: pairwise` |
| `shadow.duplicate.pairwise: true` | `judge.method: reference` |
| `shadow.duplicate.self_consistency_k: 3` | any `judge.self_consistency_k` above 1 |
| | every `sidecar_judge:` block |
| | every `sidecar:` block |
| | a `panel.members[]` seat carrying `sidecar_endpoint_env` |
If your entry names nothing in the left column, change it now rather than after the run. Here is the
whole shape of one, so you know what you are looking at when it happens:
```console
$ tokentriage validate --config pairwise.yaml
Error: configuration is invalid: evals.evaluators[0].judge.method: "pairwise" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel
Error: configuration is invalid: evals.evaluators[0].judge.method: "pairwise" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel
```
Exit status 1. `evals.evaluators[0]` is the index of the judge entry in that file; yours is your
own. The message is doubled by the binary rather than by the capture, and step 3 says why.
The same sentence comes back with `"reference"` substituted, and with `"pointwise-rubric"` once that
method's required `rubric:` leaf is present. Without the leaf the loader stops the file one layer
earlier and names `judge.rubric` instead. Either way the answer is the same one: this build does not
fall back to `pointwise-bool` behind your back, so a refused method is a stopped process rather than
a quietly different grade.
Add whichever prefix belongs to the verb you ran, `Error: configuration is invalid: ` at
`tokentriage validate` or `Error: evals: ` at boot.
```text
evals.evaluators[0].judge.self_consistency_k: 3 is not composable by this build — no scorer in internal/evaluator wraps judgecore's k-sample executor, so a k>1 tuple would grade with ONE sample while B.2 was told the row carries a vote-margin feature. Configure k=1 (the default) or take the leaf off
evals.evaluators[0].sidecar_judge: evaluator "judge-task-completion" declares a §2.4 sidecar judge, and a sidecar judge cannot be built from config alone — the endpoint is an ENV REFERENCE, and resolving it inside the evaluator registry would be a second, unaudited configuration path that skipped the air-gap gate. The composition site must build it with evaluator.NewSidecarJudgeScorer and pass it as Instance.Scorer; nothing in this build does yet, so the leaf is config-complete and composition-incomplete
evals.evaluators[0].judge.panel.members[2].sidecar_endpoint_env: a §2.4 sidecar cannot take a panel seat in this build — a seat votes through judgecore.Transport.GradeText and the sidecar client REFUSES GradeText by design (its wire carries a structured item and answers with a structured verdict, so there is no raw text to parse back). A panel seated with one would refuse every row it judged. Use provider members ({model + api_key_env}) for every seat, or take the panel off this evaluator
```
**Where to go instead.** Pairwise reaches production through `shadow.duplicate.pairwise`, a
different leaf on a different plane, and k-sampling reaches it through
`shadow.duplicate.self_consistency_k`. Both compose. Both carry a condition: that plane dials a
compiled-in provider URL and no `shadow.duplicate.*` key moves it, so it needs outbound traffic and
cannot be rehearsed against a listener on your own box. The evals-plane judge can, through
`endpoint_env`. [Air-gap judging, and the known limits](/docs/judges/air-gap-and-limits) has that
asymmetry in full. `pointwise-rubric` and `reference` have no config-reachable alternative here.
## 3. Ask the binary before you start it
```console
$ tokentriage validate --config judge.yaml
warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
mode: ledger
listen: 127.0.0.1:8787 admin: 127.0.0.1:9090
upstreams: 2
- anthropic https://api.anthropic.com
- openai https://api.openai.com
tiers: 2 (cheapest -> most capable)
0. fast -> openai/gpt-5-mini
1. frontier -> anthropic/claude-opus-4-8
rules: 0
OK: configuration is valid
```
That capture came off a source build on 2026-08-24, run from the directory holding `judge.yaml`, so
the file name in the command is the reader's and the tier and upstream lines are read out of the
same file. Exit status was 0. Nothing in the block is machine-derived: no timestamp, no port the
kernel picked, no home directory, no version string. Every line of it, warnings included, arrives on
stdout, so a pipeline that keeps only stderr keeps nothing.
Two things in it are worth reading twice.
**The egress warning is printed twice, and that is the binary, not the capture.** `tokentriage
validate` assembles its printed set from a structural pass and a registry-aware pass, and a warning
both produce is appended once each. A transcript of this surface showing one copy has been edited.
**The warning is a warning rather than an error.** A judge that phones out is a supported
configuration, and `external: allow` is you saying so in the file where a reviewer can grep for it.
On this surface, a config `tokentriage validate` accepts is a config that boots. Every shape in
step 2's table was driven through both verbs, and the body after the prefix is byte-identical;
only the prefix moves, from `Error: configuration is invalid: ` at `validate` to `Error: evals: ` at
boot. Both exit 1. So validating first tells you what starting would have told you, which is not
true of every subsystem and is worth knowing about this one.
## 4. Start it, and confirm the judge was wired
Run the process against the same file. It repeats the config's warnings, and then, where the
evaluator composed, writes one structured record whose `msg` reads:
```text
evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)
```
Beside that message the record carries `evaluator`, the composed `judge_method`, `judge_model`,
`api_key_env`, and your `sample_rate` and `max_usd_per_day` read back at you. A panel prints
`configured_judge_model` in place of `judge_model`, because no seat read the top-level value.
[Grade your first request with a judge](/docs/judges/quickstart) shows the whole boot block around
that record.
Where that record is missing, the evaluator did not compose, and the error preceding it names the
leaf. Where it is present and no row is ever graded, the config was fine and something else is
quiet, and the next section is the first place to look.
## Two leaves nothing checks
Both are shape-checked and neither is looked up, which makes them the two ways a config that
validates green still grades nothing.
**`api_key_env` names a variable whose contents are read at the first judge call.** The loader
requires the leaf, turns away an empty one, and turns away a literal secret written in its place.
Nothing on this plane checks, at load or at boot, whether the variable holds anything. Booting with
it unset logs `evals: judge evaluator wired` and says nothing further, while the two unset upstream
key variables in the same file each get their own boot warning. The failure surfaces later, at the
first grading call, as `judge API key is empty`.
**The guardrail plane does the opposite with the same leaf.** `wireGuardrailJudge` reads the
variable at boot, and where it is unset it leaves the tier disarmed and logs
`guardrails judge NOT armed: api key unset` with the variable name beside it. Both were measured on
one box in one session against the same unset variable. Two planes, one config idiom, two answers,
so a habit learned on either one does not carry to the other.
**`judge.rubric.id` membership is not validated on any path a config file reaches.** The loader
checks the leaf's shape and its coherence with `method:`, and defers membership. The lookup that
would name the registered ids sits inside the rubric method's own runtime path, which a config-file
deployment does not reach, because `pointwise-rubric` is turned away at composition first.
Measured: a config naming `rubric: { id: no-such-rubric }` comes back with step 2's *method*
refusal, word for word, and the id is not mentioned. Filed as a product defect. The two shipped ids
are `task-completion-v1` and `groundedness-v1`.
## What this page does not cover
**Every optional leaf, and what each one defaults to.**
[Evals & calibration configuration](/docs/reference/evals-config) is the reference for the whole
`evals` block; this page is the subset a judge needs to load.
**What each method decides, and what it costs.** The five tokens, their call multipliers and their
named failures are [Judge methods](/docs/reference/judge-methods).
**The other key spelled `max_usd_per_day`.** `shadow.duplicate.max_usd_per_day` sits on the shadow
plane and acts in the opposite direction from the evaluator cap in this page's block: absent or `0`,
it turns shadow duplication off, with no refusal, no warning and no log line.
[What judging costs, and how to cap it](/docs/judges/cost-and-budget) carries the two captures that
establish it.
**The floors a validation run applies.** The `evals.validation` block beside these keys sets them,
and each floor there is raise-only: a file putting `min_kappa` below `0.5` is refused at validate,
naming the frozen floor. The keys are in the reference and the walk that reads them is
[the walk that promotes a judge](/docs/judges/agreement).
**What the numbers mean once rows arrive.** Which requests get sampled, how the draw is made, and
what the graded row then says are [How a judge call happens](/docs/judges/how-judging-works).