Configure a judge — TokenTriage Docs

You have decided to grade a slice of your traffic with a model, and that decision is now a config file. Four steps take it from nothing to a running judge: paste the block, check the method you named against what this build composes, ask the binary, then read one boot line.

The surface is smaller than you would guess. One block under one evaluator entry, five leaves that carry weight, and a daily dollar cap the loader wants before it will accept any of it.

evals:
  outcomes:
    api_enabled: true
  evaluators:
    - id: judge-task-completion
      judge:
        external: allow
        model: gemini-2.5-flash
        api_key_env: JUDGE_API_KEY
      sample_rate: 0.20
      max_usd_per_day: 20.0

If you have not seen a graded row yet, How a judge call happens is the model under all of this, and the quickstart gets you one. The exhaustive per-key table lives in Evals & calibration configuration; this page is the block you paste and the blocks you cannot.

1. Paste the block into a config that has a dashboard

That block needs dashboard.enabled: true in the same file. The leaf mounts /tt/api/v1, and the endpoint your annotators write labels through is served there. Take the door itself from Collect the labels a judge is measured against.

Keep the id judge-task-completion. Two judge evaluators are registered and only one of them can be carried through a validation run. judge-groundedness publishes no pass floor, so the runner leaves every row it grades without a verdict, and no gold set forms for it however long you collect.

Five leaves carry weight, and nothing under evals: is filled in behind you. Each absent-reading here is applied by a named consumer at composition time, which is why an omitted leaf has a meaning rather than a placeholder.

Leaf	Required?	What an absent leaf reads as
`sample_rate`	yes, and not defaulted	nothing. Omitting it is refused, because `0` and `1` are both plausible readings and one of them spends money
`max_usd_per_day`	yes whenever `judge:` is present	nothing. Omitting it is refused. `0` is legal and stops the evaluator at its first priced call
`judge.external`	yes, the literal token `allow`	closed. Any other value, and an absent leaf, are refused identically
`judge.model`	yes, on every method including `panel`	nothing. Omitting it is refused
`judge.api_key_env`	yes	nothing. Omitting it is refused. It is an env-var name matching `^[A-Z_][A-Z0-9_]*$`, and a literal secret written here is refused

Everything else under judge: is optional, and every one of those defaults is in Evals & calibration configuration. Two of the optional leaves are worth knowing by name now. judge.endpoint_env names a variable holding the URL the judge posts to, which is how an evals-plane judge is pointed at a listener on your own box. judge.method picks the method step 2 checks.

evals.outcomes.api_enabled beside the block is read by the config loader and by nothing else, which catches one combination: the flag set against a disabled dashboard. It does not open the door. Measured on this build, with the leaf deleted from the file entirely, POST /tt/api/v1/outcomes answered {"accepted":1}. Keep it set, because it is what a reviewer reads.

If you are seating a panel rather than one model

judge.model and judge.api_key_env stay required on method: panel, even though no seat reads either. Both fields are unconditional, so a panel block written without them is stopped twice in one message:

$ tokentriage validate --config panel-no-model.yaml
Error: configuration is invalid: config panel-no-model.yaml: evals.evaluators[0].judge.model: required (the model id sent to the judge provider). §4 row 6 records what an unset judge model costs: the shipped sampler forwarded its grading-path selector as the model id and every verdict came back nil, for months, invisibly
evals.evaluators[0].judge.api_key_env: required (an env var NAME holding the judge provider's API key, e.g. ANTHROPIC_API_KEY). A judge is not a proxied request: there is no inbound credential to forward, so an absent key env is an evaluator that cannot authenticate

Exit status 1, and the binary prints that whole message twice; one copy is shown here, for the reason step 3 gives. It carries a config <path>: segment that the method message in step 2 does not. That segment is how you tell the two layers apart: the loader raised this one while reading the file, and the composition planner raises the other after the file has decoded.

The clause about §4 row 6 is the product quoting its own history at you, and it is worth reading once. A shipped sampler sent its grading-path selector where the model id belonged, every verdict came back nil, and no surface said so for months. That is why the leaf is unconditional today rather than defaulted.

Seats, families, the vote vector and the call multiplier are Run a cross-family panel.

2. Check that the method you named composes

The judge method vocabulary has five tokens and this build composes two of them from a config file. That gap is the single most expensive thing to discover after you have written the file, so it is a table rather than a paragraph.

Composes from a config file	Turned away at `validate` and again at boot
`judge.method: pointwise-bool`, and an omitted `method:`	`judge.method: pointwise-rubric`
`judge.method: panel` with three seats from two model families, at `self_consistency_k: 1` or absent	`judge.method: pairwise`
`shadow.duplicate.pairwise: true`	`judge.method: reference`
`shadow.duplicate.self_consistency_k: 3`	any `judge.self_consistency_k` above 1
	every `sidecar_judge:` block
	every `sidecar:` block
	a `panel.members[]` seat carrying `sidecar_endpoint_env`

If your entry names nothing in the left column, change it now rather than after the run. Here is the whole shape of one, so you know what you are looking at when it happens:

$ tokentriage validate --config pairwise.yaml
Error: configuration is invalid: evals.evaluators[0].judge.method: "pairwise" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel
Error: configuration is invalid: evals.evaluators[0].judge.method: "pairwise" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel

Exit status 1. evals.evaluators[0] is the index of the judge entry in that file; yours is your own. The message is doubled by the binary rather than by the capture, and step 3 says why.

The same sentence comes back with "reference" substituted, and with "pointwise-rubric" once that method’s required rubric: leaf is present. Without the leaf the loader stops the file one layer earlier and names judge.rubric instead. Either way the answer is the same one: this build does not fall back to pointwise-bool behind your back, so a refused method is a stopped process rather than a quietly different grade.

The other three refusal bodies, with the prefix stripped

Add whichever prefix belongs to the verb you ran, Error: configuration is invalid: at tokentriage validate or Error: evals: at boot.

evals.evaluators[0].judge.self_consistency_k: 3 is not composable by this build — no scorer in internal/evaluator wraps judgecore's k-sample executor, so a k>1 tuple would grade with ONE sample while B.2 was told the row carries a vote-margin feature. Configure k=1 (the default) or take the leaf off

evals.evaluators[0].sidecar_judge: evaluator "judge-task-completion" declares a §2.4 sidecar judge, and a sidecar judge cannot be built from config alone — the endpoint is an ENV REFERENCE, and resolving it inside the evaluator registry would be a second, unaudited configuration path that skipped the air-gap gate. The composition site must build it with evaluator.NewSidecarJudgeScorer and pass it as Instance.Scorer; nothing in this build does yet, so the leaf is config-complete and composition-incomplete

evals.evaluators[0].judge.panel.members[2].sidecar_endpoint_env: a §2.4 sidecar cannot take a panel seat in this build — a seat votes through judgecore.Transport.GradeText and the sidecar client REFUSES GradeText by design (its wire carries a structured item and answers with a structured verdict, so there is no raw text to parse back). A panel seated with one would refuse every row it judged. Use provider members ({model + api_key_env}) for every seat, or take the panel off this evaluator

Where to go instead. Pairwise reaches production through shadow.duplicate.pairwise, a different leaf on a different plane, and k-sampling reaches it through shadow.duplicate.self_consistency_k. Both compose. Both carry a condition: that plane dials a compiled-in provider URL and no shadow.duplicate.* key moves it, so it needs outbound traffic and cannot be rehearsed against a listener on your own box. The evals-plane judge can, through endpoint_env. Air-gap judging, and the known limits has that asymmetry in full. pointwise-rubric and reference have no config-reachable alternative here.

3. Ask the binary before you start it

$ tokentriage validate --config judge.yaml
warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 2
  - anthropic              https://api.anthropic.com
  - openai                 https://api.openai.com
tiers:     2 (cheapest -> most capable)
  0. fast       -> openai/gpt-5-mini
  1. frontier   -> anthropic/claude-opus-4-8
rules:     0
OK: configuration is valid

That capture came off a source build on 2026-08-24, run from the directory holding judge.yaml, so the file name in the command is the reader’s and the tier and upstream lines are read out of the same file. Exit status was 0. Nothing in the block is machine-derived: no timestamp, no port the kernel picked, no home directory, no version string. Every line of it, warnings included, arrives on stdout, so a pipeline that keeps only stderr keeps nothing.

Two things in it are worth reading twice.

The egress warning is printed twice, and that is the binary, not the capture. tokentriage validate assembles its printed set from a structural pass and a registry-aware pass, and a warning both produce is appended once each. A transcript of this surface showing one copy has been edited.

The warning is a warning rather than an error. A judge that phones out is a supported configuration, and external: allow is you saying so in the file where a reviewer can grep for it.

4. Start it, and confirm the judge was wired

Run the process against the same file. It repeats the config’s warnings, and then, where the evaluator composed, writes one structured record whose msg reads:

evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)

Beside that message the record carries evaluator, the composed judge_method, judge_model, api_key_env, and your sample_rate and max_usd_per_day read back at you. A panel prints configured_judge_model in place of judge_model, because no seat read the top-level value. Grade your first request with a judge shows the whole boot block around that record.

Where that record is missing, the evaluator did not compose, and the error preceding it names the leaf. Where it is present and no row is ever graded, the config was fine and something else is quiet, and the next section is the first place to look.

Two leaves nothing checks

Both are shape-checked and neither is looked up, which makes them the two ways a config that validates green still grades nothing.

api_key_env names a variable whose contents are read at the first judge call. The loader requires the leaf, turns away an empty one, and turns away a literal secret written in its place. Nothing on this plane checks, at load or at boot, whether the variable holds anything. Booting with it unset logs evals: judge evaluator wired and says nothing further, while the two unset upstream key variables in the same file each get their own boot warning. The failure surfaces later, at the first grading call, as judge API key is empty.

judge.rubric.id membership is not validated on any path a config file reaches. The loader checks the leaf’s shape and its coherence with method:, and defers membership. The lookup that would name the registered ids sits inside the rubric method’s own runtime path, which a config-file deployment does not reach, because pointwise-rubric is turned away at composition first.

Measured: a config naming rubric: { id: no-such-rubric } comes back with step 2’s method refusal, word for word, and the id is not mentioned. Filed as a product defect. The two shipped ids are task-completion-v1 and groundedness-v1.

What this page does not cover

Every optional leaf, and what each one defaults to. Evals & calibration configuration is the reference for the whole evals block; this page is the subset a judge needs to load.

What each method decides, and what it costs. The five tokens, their call multipliers and their named failures are Judge methods.

The other key spelled max_usd_per_day. shadow.duplicate.max_usd_per_day sits on the shadow plane and acts in the opposite direction from the evaluator cap in this page’s block: absent or 0, it turns shadow duplication off, with no refusal, no warning and no log line. What judging costs, and how to cap it carries the two captures that establish it.

The floors a validation run applies. The evals.validation block beside these keys sets them, and each floor there is raise-only: a file putting min_kappa below 0.5 is refused at validate, naming the frozen floor. The keys are in the reference and the walk that reads them is the walk that promotes a judge.

What the numbers mean once rows arrive. Which requests get sampled, how the draw is made, and what the graded row then says are How a judge call happens.