# Judge methods Every judge method this build registers — what each decides, what it costs, how it fails, and whether it composes from a config file. Source: https://tokentriage.com/docs/reference/judge-methods --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; A judging setup asks a model one question about one of your responses, and which question it asks settles everything after it: the shape of the answer that comes back, how many provider calls it took to get there, and what happens when the reply cannot be read. That question has a name in the config file, and the set of names is closed at five. Two of the five compose from a config file. The other three are turned away at `tokentriage validate` and again at boot, by a message that names the leaf which stopped them. So the first thing to look up about a method is not what it decides but whether a configuration can reach it at all. Every entry below carries the same eight parts: what it decides, a block that was loaded by the binary, the mechanism, the knobs with their absent-readings, the call multiplier, the named failure, the output that shows it running, and when to pick something else. The config surface as a whole is [Configure a judge](/docs/judges/configure-a-judge); this page is its method column. ## The vocabulary The method name is the first thing a judge block is checked on, and it is checked against a closed set: a value outside the five is turned away before anything else in the block is looked at, and the value you wrote is not echoed back into the message. ```console $ tokentriage validate --config typo.yaml Error: configuration is invalid: config typo.yaml: evals.evaluators[1].judge.method: not a §2.1 judge method — the closed vocabulary is pointwise-bool|pointwise-rubric|pairwise|reference|panel (absent reads as pointwise-bool, the shipped grader). The value you wrote is deliberately not repeated back in this message ``` That is one copy of a doubled message, and the doubling is the binary's: every warning and every error on this surface is printed twice, so a capture showing one copy has been edited. The command was run from the directory holding `typo.yaml`, which is why the path in the message is a bare filename; the evaluator index `[1]` is that file's and yours will be your own. Exit status was 1. Booting a copy of the same file printed the same body under an `Error: config : ` prefix and also exited 1. | Token | What it decides | Judge calls per graded row | Composes from a config file | |---|---|---|---| | `pointwise-bool` | acceptable or not, for one response | 1 | **yes**, and an absent `method:` reads as this | | `panel` | acceptable or not, by majority vote of N seats | N | **yes**, at three or more seats from two or more model families | | `pointwise-rubric` | a 1..K rating, scaled to `[0,1]` | 1 | no | | `pairwise` | which of two responses is better, or a tie | 2 | no on `evals.evaluators[].judge`; the leaf that composes it is `shadow.duplicate.pairwise` | | `reference` | acceptable or not, against a supplied reference answer | 1 | no | Two things that table does not say. The multiplier is per **graded** row rather than per served request: what decides how many rows get graded is `sample_rate`, and the arithmetic joining the two is [what judging costs](/docs/judges/cost-and-budget). And `panel` has no fixed multiplier, because its multiplier is its seat count. `tokentriage validate` and `tokentriage run` answer identically about every method on this page. A shape one of them stops, the other stops too, with the same body and exit status 1, and only the prefix moves. A leaf the config loader stops carries a `config : ` segment on both verbs; a method the composition check stops carries no path at all, and reads `Error: configuration is invalid: ` at `validate` against `Error: evals: ` at boot. That segment is how you tell which of the two layers spoke, and validating first tells you what starting would have told you. ## The two that compose These are the shapes a config file can reach. Both grade one response at a time and both answer with a boolean, which is why a panel can be built out of the first one. ### pointwise-bool Decides whether one response is acceptable, from the instruction that produced it and the response itself. ```yaml evals: evaluators: - id: judge-task-completion judge: external: allow # the air-gap gate, and the only value it takes method: pointwise-bool # the absent-reading, written out model: gemini-2.5-flash api_key_env: JUDGE_API_KEY # an env-var NAME, not the key itself sample_rate: 0.20 max_usd_per_day: 20.0 ``` **How it decides.** The judge is asked for one JSON object and a strict parse reads one key from it, `{"score": }`. The object is the span from the first `{` to the last `}`, so a reply wrapped in prose or a code fence still parses. A missing key, a `null`, a quoted `"true"` and a numeric `1` are each read as no verdict rather than as a grade. The boolean lands on the row as `value: 1` or `value: 0`, and the row's `pass` is derived from the metric's own floor, which for `task_completion` is `1.0`. The version stamped beside it is `prompt_sha`, a seven-character digest of the effective prompt, which is an identifier for a template rather than a commitment to one. **Knobs.** | Key under `evals.evaluators[].judge` | Absent-reading | What it does | |---|---|---| | `method` | `pointwise-bool` | selects this method | | `model` | required | the model id sent to the judge provider | | `api_key_env` | required | the env-var name holding that provider's key | | `endpoint_env` | the shipped Anthropic Messages endpoint | the env-var name holding the URL the judge posts to | | `external` | required, the literal `allow` | the air-gap gate | | `self_consistency_k` | `1` | above `1` this leaf is turned away on the evals plane | **What it costs.** One provider call per graded row, the lowest multiplier this build offers. The call is made by the off-path pass after the client's response has been written, so it adds nothing to the served request's latency. Its dollars land on that evaluator's own metacost line and count against `max_usd_per_day`. **How it fails.** A transport error or a reply the parse cannot read returns no verdict and an error, and the grade is left unset rather than guessed. The runner then counts the row under `evaluator_error`, which is a reason and a count with no message field, so the text of the failure does not survive. A missing credential and a same-family judge land in that same bucket and are indistinguishable from outside it. A timeout or a shutdown is counted apart, under `cancelled`, on the grounds that "we stopped it" and "it broke" send you to different places. **How to tell it is working.** The boot log names the composed method and the instrument before any traffic arrives: ```console {"time":"2026-08-24T13:17:09.9297915-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"pointwise-bool","judge_instrument":"gemini-2.5-flash","judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20} ``` One line lifted out of a longer boot log on 2026-08-24, from a source build on Windows booting the shipped `examples/evals/journey-2-validated-judge.yaml` example with the entry above pasted over its judge entry and its two listeners moved to port `0`. The timestamp and its local offset are that machine's; everything else on the line is read from the config. On a graded row the same answer comes back as `judge.method`, which the writer takes from the scorer that produced the outcome rather than from the config text, so a deployment that configured one method and composed another cannot record its intent in place of its behaviour. **Reach for something else when** one model's opinion is not enough evidence, or when the grader shares a family with a tier you route to. A panel is the setup that composes for both, at its seat count in calls. ### panel Decides acceptable or not by strict majority of several models' ballots, from the same instruction and response that every seat is shown. ```yaml evals: evaluators: - id: judge-task-completion judge: external: allow method: panel model: gemini-2.5-flash # required on a panel, and no seat reads it api_key_env: JUDGE_API_KEY # required, for the same reason panel: rule: majority # the whole vocabulary, and the absent-reading members: - { model: gemini-2.5-flash, api_key_env: JUDGE_GEMINI_KEY } - { model: mistral-large-2, api_key_env: JUDGE_MISTRAL_KEY } - { model: llama-3-70b, api_key_env: JUDGE_LLAMA_KEY } sample_rate: 0.20 max_usd_per_day: 20.0 # one cap for the whole fan-out ``` **How it decides.** Each seat grades the item with `pointwise-bool` and its boolean is rendered as one of two ballot tokens, `pass` or `fail`. Every seat votes or the invocation stops, so the ballot count is always the seat count. The tally decides when twice the leading bucket exceeds that count: ```text decided ⟺ 2 × top > n margin = (top − runner-up) / n ``` With two tokens over n ballots the leading bucket holds at least half of them, so an odd panel always decides: three seats fall 2–1 or 3–0 and both clear the test. Only an even seat count can leave the two buckets level, and a level vote is an abstention carrying `panel_split`. Three seats is the floor the config layer applies and there is no ceiling, so four seats validates and boots, and four is the smallest shape that can produce that abstention. **Knobs.** | Key | Absent-reading | What it does | |---|---|---| | `judge.panel.members[]` | required with this method | the seats: at least three, from at least two model families | | `judge.panel.rule` | `majority` | the aggregation rule, closed at that one value | | `…members[].model` | required per seat | the model in that seat; a repeated model is turned away | | `…members[].api_key_env` | required per seat | that seat's provider key, by env-var name | | `…members[].endpoint_env` | that seat's shipped provider endpoint | the URL that seat posts to, by env-var name | | `judge.model`, `judge.api_key_env` | required | still required here, and read by no seat | | `judge.endpoint_env` | turned away on a panel | the seats are the transports, so a top-level endpoint would be dead config | **What it costs.** N provider calls per graded row against the single `max_usd_per_day` on the evaluator entry, not one cap per seat. The cap is consulted before each seat, so a fan-out stopped part-way has already made and settled the calls that came before the stop. The validator states the multiplier before you boot, in a warning that names the seat count and the cap in one sentence. **A seat whose call cannot be priced hides the whole row's spend.** An unpriced total makes the invocation's cost unpriced, and the budget declines to fold an unpriced amount rather than adding a guess, so the earlier seats' priced calls settle as `$0`. The product measures the size of it in `TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed`: 50 rows at 2 priced calls of `$0.0001` is `$0.0100` spent, `SpentToday` reads `$0.0000`, and the declared `$0.05` cap does not move. Watch the unpriced counts on the metacost surface rather than the cap. **How it fails.** Fail-closed. A seat that cannot be reached, a seat that panics, a seat answering with a non-boolean verdict, a budget stop part-way through the fan-out, and a vote vector that does not line up positionally with the members each stop the whole invocation rather than that seat alone. None of them writes a row, and all of them arrive at the runner as `evaluator_error`, because the two finer tallies that would separate them, `judgecore.Panel.Counts` and `PanelScorer.Counts`, have no call site outside their tests. A level vote is different in kind: it is an answer about the item, it is counted under `abstained`, and the whole vote vector travels out with it. **How to tell it is working.** The boot line carries a composite instrument id and the seat count: ```console {"time":"2026-08-24T13:17:22.6485871-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"panel","judge_instrument":"panel(majority:gemini-2.5-flash,mistral-large-2,llama-3-70b)","configured_judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20,"panel_members":3} ``` A second boot of the same example under the same conditions, carrying the entry in this block. A single-model judge prints `judge_model` where this one prints `configured_judge_model`, which is the boot log saying that no seat used the top-level value. On a graded row the ballots come back beside the verdict: ```json "panel": { "members": [ { "model": "claude-haiku-4-5", "family": "claude" }, { "model": "gpt-4o-mini", "family": "gpt" }, { "model": "gemini-2.5-flash", "family": "gemini" } ], "votes": ["pass", "fail", "pass"], "rule": "majority", "margin": 0.3333333333333333 } ``` That is the `panel` object out of the tracked golden fixture the product pins its own wire shape against, with whitespace added and no value changed; `go test ./internal/declog/ -run TestJudgeRowGolden` compares the whole file byte for byte and passes on this tree. It is a shape pin rather than a live capture, and its three seats are the fixture's. `votes` is positionally aligned with `members` and `margin` is derivable from it, so two `pass` against one `fail` over three seats prints `(2 − 1) / 3`. **Reach for something else when** the seat count multiplies a bill you cannot carry, or when one grader is the whole measurement you need. `pointwise-bool` is the same question asked once. ## The three this build turns away All three are registered methods with working implementations, and none of them composes over the `evals.evaluators[].judge` transport, which is a provider Messages API. The composition check reads strings and builds nothing: no client, no credential read, no connection. It produces one sentence per method, and the sentence is the same one with the token substituted. ```console $ tokentriage validate --config m-pointwise-rubric.yaml Error: configuration is invalid: evals.evaluators[1].judge.method: "pointwise-rubric" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel Error: configuration is invalid: evals.evaluators[1].judge.method: "pointwise-rubric" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel ``` Captured on 2026-08-24 from a source build, over a copy of the shipped `journey-2-validated-judge.yaml` example with one line changed. Both copies of the message are the binary's. The same file through `tokentriage run` prints the same body under `Error: evals: ` and also exits 1, and two consecutive runs of each were byte-identical. The `§2.4 wire` the message names is a self-hosted judging container, and every `sidecar_judge:` block that would configure one is turned away at both verbs as well. ### pointwise-rubric Would decide a 1..K rating against a named rubric, scaled to the unit interval, from the instruction, the response and the rubric's anchor text. ```yaml evals: evaluators: - id: judge-task-completion judge: external: allow method: pointwise-rubric rubric: { id: task-completion-v1 } # required with this method, and only with it model: gemini-2.5-flash api_key_env: JUDGE_API_KEY sample_rate: 0.20 max_usd_per_day: 20.0 ``` **How it decides.** The prompt asks for reasoning first and a verdict second, and the parser version records that requirement in its own name, `scalar-v1;cot=required`. The reply is `{"score": }` on the rubric's scale, read from the same object span the boolean parser uses, and the score is rescaled to `(score − 1) / (scale − 1)`. A score outside `1..K`, a float, a string or a `null` is read as no verdict: an out-of-range value is turned away rather than clamped, because clamping a `0` would publish a broken reply as the worst legal grade. Both shipped rubrics carry five anchors, so the scale is 5 and a `3` scales to `0.5`. **Knobs.** | Key | Absent-reading | What it does | |---|---|---| | `judge.rubric.id` | required with this method | names the rubric; the shipped ids are `task-completion-v1` and `groundedness-v1` | | `judge.rubric` under any other method | absent | written under another method it is turned away, because nothing would consume it | **What it costs.** One call per item wherever the method runs. On the `evals.evaluators[].judge` transport it costs nothing at all, because the composition check happens before a client is built, so the process stops without a credential being read or a connection being opened. **How it fails.** Turned away at `tokentriage validate` and again at boot, by the message above, and the composition site does not fall back to `pointwise-bool` behind it. The rubric id itself is checked for shape and for coherence with `method:`, and its membership is looked up on no path a config file reaches, so a rubric id nobody registered produces the method message above with the rubric not mentioned at all. The lookup that would list the registered ids sits inside this method's own runtime path, which a config-file deployment does not get to. **How to tell it is working.** What a config-file deployment sees is the message above with `"pointwise-rubric"` in it, on both verbs. Nothing else about this method reaches a boot log or a row. **Reach for something else when** you wanted a graded scale rather than a boolean. This build has no config-reachable substitute for that: `pointwise-bool` answers the same question with one bit, and a panel answers it with a vote vector and a margin. ### pairwise Decides which of two responses to one instruction is better, or that they tie, and derives that answer from both presentation orderings rather than from one. ```yaml shadow: duplicate: enabled: true sample_rate: 0.05 max_usd_per_day: 5.0 # absent or 0 turns the whole feature off, silently pairwise: true # the both-orderings co-run ``` That block is the leaf that composes. Writing `method: pairwise` under `evals.evaluators[].judge` instead gets the message above with `"pairwise"` in it. **How it decides.** Two calls inside one grading call, with the pair presented once each way. Each reply is `{"winner": "first"|"second"|"tie"}` under parser version `abtie-v1`, in prompt coordinates: the tokens name slots rather than answers, and only the executor resolves a slot into a side, because it is the half that knows which side sat where. Both answers then sit in one frame, where side A is the response your router served. The same side both times gives that side the win, two ties counting as agreement; different sides give no winner at all, and the trial abstains with `swap_inconsistent`. The confidence feature is swap agreement, `1.0` or `0.0`. A pair run in one ordering has no representation: the only constructor takes both winners, so a half-run pair cannot be built. **Knobs.** | Key under `shadow.duplicate` | Absent-reading | What it does | |---|---|---| | `pairwise` | `false` | co-runs this method beside the two pointwise grades on every graded duplicate | | `enabled` | `false` | the duplication feature as a whole | | `sample_rate` | `0` | the share of requests duplicated; `0` composes no sampler | | `max_usd_per_day` | none, and absent or `0` composes no sampler | the cap over the duplicate and every judge call under it | | `self_consistency_k` | `1` | k samples of each pointwise grade; the pairwise co-run is not k-wrapped | | `judge` | absent | closed at the single value `oracle` | **What it costs.** Two judge calls per pair, on top of the two pointwise grades that plane already makes, and on top of the duplicate request itself. The validator does that arithmetic for the config you wrote and prints the total, *"the grader makes 4 Anthropic Messages call(s) per sampled request under this config (2 pointwise grades × self_consistency_k, plus 2 for the pairwise co-run when shadow.duplicate.pairwise is on)"*. A failure on the first ordering ends the trial before the second call is spent. This plane also carries a cost the other one does not: it dials a compiled-in provider URL that no `shadow.duplicate.*` key moves, so it cannot be rehearsed against a listener on your own box, and the first pair you judge is billable. **How it fails.** A transport or parse failure on either ordering writes nothing, not a winner and not an abstention, and is counted. A budget stop writes nothing either, on the grounds that what was stopped is what was not measured. A pair with an empty arm is skipped and counted, because a comparison against an absent answer is not a comparison. The flip itself is not a failure: it is the abstention above. The trials this method records are the swap set a validation run reads, and the share of them that flipped is the rate its ceiling applies to. **How to tell it is working.** The boot log says the co-run composed: ```console {"time":"2026-08-24T13:17:34.7250048-07:00","level":"INFO","msg":"shadow duplication enabled","sample_rate":0.05,"max_usd_per_day":5,"judge_model":"","judge_key_env":"ANTHROPIC_API_KEY","pairwise":true,"self_consistency_k":1} ``` A third boot of the same example under the same conditions, with the block in this entry appended to it. The empty `judge_model` is that config's: this plane takes its model and key from `classifier.oracle.*`, and the key env falls back to `ANTHROPIC_API_KEY`. Delete the `max_usd_per_day` line and the same file still validates at exit 0 and still boots, and its boot output then holds zero lines matching `shadow` against the three the capped one prints. That is the thing to check first. On a graded row the trial comes back as its own block, carrying both orderings and the derived agreement: ```json "pairwise": { "prompt_sha": "e5f6a7b", "winner_ab": "A", "winner_ba": "A", "agree": true } ``` Those four keys are quoted out of the same tracked golden as the panel block above. That golden sets every field it can, so it is a field-by-field shape pin and not a row any run produced: read the keys, not the combination. `prompt_sha` here is the trial's own version rather than the pointwise grades', which answer to a different one. **Reach for something else when** the deployment has no outbound path, or when the question is whether one answer is acceptable rather than which of two is better. [Position bias, self-preference and panels](/docs/judges/bias-controls) is what the two orderings buy and where the method stops helping. ### reference Would decide whether one response is acceptable against a reference answer supplied alongside it. ```yaml evals: evaluators: - id: judge-task-completion judge: external: allow method: reference model: gemini-2.5-flash api_key_env: JUDGE_API_KEY sample_rate: 0.20 max_usd_per_day: 20.0 ``` **How it decides.** The same strict boolean parse `pointwise-bool` uses, `bool-v1`, over a prompt that renders a reference block around the reference text. An item whose reference is empty or blank is turned away before a call is spent, because rendering without that block would send `pointwise-bool`'s own prompt under this method's version string, and the verdict would then be described by a version that did not produce it. **Knobs.** | Key | Absent-reading | What it does | |---|---|---| | `judge.method` | `pointwise-bool` | `reference` selects this method, which takes no leaf of its own | **What it costs.** One call per item wherever the method runs, and nothing on a config-file deployment, for the same reason as `pointwise-rubric`: the process stops before a client exists. **How it fails.** Turned away at `tokentriage validate` and again at boot, by the message above with `"reference"` in it. A second gap sits underneath that one and is the more permanent of the two: outside `judgecore` and its own tests, nothing in this build sets the reference field on a judged item, so the input this method requires has no producer. The one component that would carry a reference onto a wire is the self-hosted judging container, and every config leaf that would configure one is turned away too. **How to tell it is working.** The message above with `"reference"` in it, on both verbs. Nothing about this method reaches a boot log or a row from a config file. **Reach for something else when** you have a known-good answer to compare against. This build's config-reachable graders read the instruction and the response only, so a reference answer belongs in the instruction text or in a human label, and the label route is [Collect the labels a judge is measured against](/docs/judges/collect-labels). ## What each method publishes as a confidence feature A judged verdict can carry one scalar beside it, and that scalar is the input a risk-controlled accept region would threshold. Which methods publish one is a closed table, mirrored into the package holding the quality plane's descriptors, and every cell in it is measured by a test that builds the registered method, runs it at one sample and at three, and reads whether a confidence value came back. | Method | At `self_consistency_k: 1` | The feature's name | Above `k: 1` | |---|---|---|---| | `pairwise` | yes | swap agreement | yes, voted per ordering, reporting the weaker ordering's margin | | `panel` | yes | panel vote margin | no; this build mints no version above `k: 1` for a panel, and the constructor stops one | | `pointwise-bool` | no | none | yes, the k-sample margin | | `pointwise-rubric` | yes | rubric scalar margin | no; a vote over a scalar is a mean rather than a ballot | | `reference` | no | none | yes, the k-sample margin | The `k` in that table is `self_consistency_k`, which is a wrapper rather than a sixth method: it asks the same method k times and takes a strict majority, and a vote with no strict majority is an error rather than an abstention, because the abstention vocabulary has no token for it. On `evals.evaluators[].judge` any value above `1` is turned away at both verbs, so the only k above 1 a config file composes is `shadow.duplicate.self_consistency_k`, on the plane that needs outbound traffic. The feature is published whether or not anything reads it. The shipped sampler builds its judging plan with no accept region installed, which the product pins with a test of its own, so no abstention in this build comes from thresholding one of these numbers. [When a judge can't tell](/docs/judges/abstention) has the four abstention reasons and says which of them an operator can reach. ## What this page does not cover **Where a method's dollars turn up, and the cap that stops them.** The per-method multiplier is above; the price per call, the meters that carry it and the daily cap's post-hoc arithmetic are [What judging costs, and how to cap it](/docs/judges/cost-and-budget). **The rest of the config surface.** The calibration, drift and validation blocks beside `judge:` are [Evals and calibration configuration](/docs/reference/evals-config), and the pasteable walk through the judge block itself is [Configure a judge](/docs/judges/configure-a-judge). **Which of the two registered judge ids to configure.** `judge-groundedness` publishes no pass floor, so the runner leaves every row it grades without a verdict and no gold set forms for it. Configure `judge-task-completion`, whose metric carries a floor of `1.0`. **What a method's verdict is worth once rows arrive.** A judge outcome is `reported` until a validation run promotes its subject, and [the walk that promotes a judge](/docs/judges/agreement) sets out how.