Judge methods — TokenTriage Docs

A judging setup asks a model one question about one of your responses, and which question it asks settles everything after it: the shape of the answer that comes back, how many provider calls it took to get there, and what happens when the reply cannot be read. That question has a name in the config file, and the set of names is closed at five.

Two of the five compose from a config file. The other three are turned away at tokentriage validate and again at boot, by a message that names the leaf which stopped them. So the first thing to look up about a method is not what it decides but whether a configuration can reach it at all.

Every entry below carries the same eight parts: what it decides, a block that was loaded by the binary, the mechanism, the knobs with their absent-readings, the call multiplier, the named failure, the output that shows it running, and when to pick something else. The config surface as a whole is Configure a judge; this page is its method column.

The vocabulary

The method name is the first thing a judge block is checked on, and it is checked against a closed set: a value outside the five is turned away before anything else in the block is looked at, and the value you wrote is not echoed back into the message.

$ tokentriage validate --config typo.yaml
Error: configuration is invalid: config typo.yaml: evals.evaluators[1].judge.method: not a §2.1 judge method — the closed vocabulary is pointwise-bool|pointwise-rubric|pairwise|reference|panel (absent reads as pointwise-bool, the shipped grader). The value you wrote is deliberately not repeated back in this message

That is one copy of a doubled message, and the doubling is the binary’s: every warning and every error on this surface is printed twice, so a capture showing one copy has been edited. The command was run from the directory holding typo.yaml, which is why the path in the message is a bare filename; the evaluator index [1] is that file’s and yours will be your own. Exit status was 1. Booting a copy of the same file printed the same body under an Error: config <path>: prefix and also exited 1.

Token	What it decides	Judge calls per graded row	Composes from a config file
`pointwise-bool`	acceptable or not, for one response	1	yes, and an absent `method:` reads as this
`panel`	acceptable or not, by majority vote of N seats	N	yes, at three or more seats from two or more model families
`pointwise-rubric`	a 1..K rating, scaled to `[0,1]`	1	no
`pairwise`	which of two responses is better, or a tie	2	no on `evals.evaluators[].judge`; the leaf that composes it is `shadow.duplicate.pairwise`
`reference`	acceptable or not, against a supplied reference answer	1	no

Two things that table does not say. The multiplier is per graded row rather than per served request: what decides how many rows get graded is sample_rate, and the arithmetic joining the two is what judging costs. And panel has no fixed multiplier, because its multiplier is its seat count.

The two that compose

These are the shapes a config file can reach. Both grade one response at a time and both answer with a boolean, which is why a panel can be built out of the first one.

pointwise-bool

Decides whether one response is acceptable, from the instruction that produced it and the response itself.

evals:
  evaluators:
    - id: judge-task-completion
      judge:
        external: allow             # the air-gap gate, and the only value it takes
        method: pointwise-bool      # the absent-reading, written out
        model: gemini-2.5-flash
        api_key_env: JUDGE_API_KEY  # an env-var NAME, not the key itself
      sample_rate: 0.20
      max_usd_per_day: 20.0

How it decides. The judge is asked for one JSON object and a strict parse reads one key from it, {"score": <bool>}. The object is the span from the first { to the last }, so a reply wrapped in prose or a code fence still parses. A missing key, a null, a quoted "true" and a numeric 1 are each read as no verdict rather than as a grade. The boolean lands on the row as value: 1 or value: 0, and the row’s pass is derived from the metric’s own floor, which for task_completion is 1.0. The version stamped beside it is prompt_sha, a seven-character digest of the effective prompt, which is an identifier for a template rather than a commitment to one.

Knobs.

Key under `evals.evaluators[].judge`	Absent-reading	What it does
`method`	`pointwise-bool`	selects this method
`model`	required	the model id sent to the judge provider
`api_key_env`	required	the env-var name holding that provider’s key
`endpoint_env`	the shipped Anthropic Messages endpoint	the env-var name holding the URL the judge posts to
`external`	required, the literal `allow`	the air-gap gate
`self_consistency_k`	`1`	above `1` this leaf is turned away on the evals plane

What it costs. One provider call per graded row, the lowest multiplier this build offers. The call is made by the off-path pass after the client’s response has been written, so it adds nothing to the served request’s latency. Its dollars land on that evaluator’s own metacost line and count against max_usd_per_day.

How it fails. A transport error or a reply the parse cannot read returns no verdict and an error, and the grade is left unset rather than guessed. The runner then counts the row under evaluator_error, which is a reason and a count with no message field, so the text of the failure does not survive. A missing credential and a same-family judge land in that same bucket and are indistinguishable from outside it. A timeout or a shutdown is counted apart, under cancelled, on the grounds that “we stopped it” and “it broke” send you to different places.

How to tell it is working. The boot log names the composed method and the instrument before any traffic arrives:

{"time":"2026-08-24T13:17:09.9297915-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"pointwise-bool","judge_instrument":"gemini-2.5-flash","judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20}

One line lifted out of a longer boot log on 2026-08-24, from a source build on Windows booting the shipped examples/evals/journey-2-validated-judge.yaml example with the entry above pasted over its judge entry and its two listeners moved to port 0. The timestamp and its local offset are that machine’s; everything else on the line is read from the config. On a graded row the same answer comes back as judge.method, which the writer takes from the scorer that produced the outcome rather than from the config text, so a deployment that configured one method and composed another cannot record its intent in place of its behaviour.

Reach for something else when one model’s opinion is not enough evidence, or when the grader shares a family with a tier you route to. A panel is the setup that composes for both, at its seat count in calls.

panel

Decides acceptable or not by strict majority of several models’ ballots, from the same instruction and response that every seat is shown.

evals:
  evaluators:
    - id: judge-task-completion
      judge:
        external: allow
        method: panel
        model: gemini-2.5-flash      # required on a panel, and no seat reads it
        api_key_env: JUDGE_API_KEY   # required, for the same reason
        panel:
          rule: majority             # the whole vocabulary, and the absent-reading
          members:
            - { model: gemini-2.5-flash, api_key_env: JUDGE_GEMINI_KEY }
            - { model: mistral-large-2,  api_key_env: JUDGE_MISTRAL_KEY }
            - { model: llama-3-70b,      api_key_env: JUDGE_LLAMA_KEY }
      sample_rate: 0.20
      max_usd_per_day: 20.0          # one cap for the whole fan-out

How it decides. Each seat grades the item with pointwise-bool and its boolean is rendered as one of two ballot tokens, pass or fail. Every seat votes or the invocation stops, so the ballot count is always the seat count. The tally decides when twice the leading bucket exceeds that count:

decided   ⟺   2 × top > n
margin    =   (top − runner-up) / n

With two tokens over n ballots the leading bucket holds at least half of them, so an odd panel always decides: three seats fall 2–1 or 3–0 and both clear the test. Only an even seat count can leave the two buckets level, and a level vote is an abstention carrying panel_split. Three seats is the floor the config layer applies and there is no ceiling, so four seats validates and boots, and four is the smallest shape that can produce that abstention.

Knobs.

Key	Absent-reading	What it does
`judge.panel.members[]`	required with this method	the seats: at least three, from at least two model families
`judge.panel.rule`	`majority`	the aggregation rule, closed at that one value
`…members[].model`	required per seat	the model in that seat; a repeated model is turned away
`…members[].api_key_env`	required per seat	that seat’s provider key, by env-var name
`…members[].endpoint_env`	that seat’s shipped provider endpoint	the URL that seat posts to, by env-var name
`judge.model`, `judge.api_key_env`	required	still required here, and read by no seat
`judge.endpoint_env`	turned away on a panel	the seats are the transports, so a top-level endpoint would be dead config

What it costs. N provider calls per graded row against the single max_usd_per_day on the evaluator entry, not one cap per seat. The cap is consulted before each seat, so a fan-out stopped part-way has already made and settled the calls that came before the stop. The validator states the multiplier before you boot, in a warning that names the seat count and the cap in one sentence.

How it fails. Fail-closed. A seat that cannot be reached, a seat that panics, a seat answering with a non-boolean verdict, a budget stop part-way through the fan-out, and a vote vector that does not line up positionally with the members each stop the whole invocation rather than that seat alone. None of them writes a row, and all of them arrive at the runner as evaluator_error, because the two finer tallies that would separate them, judgecore.Panel.Counts and PanelScorer.Counts, have no call site outside their tests.

A level vote is different in kind: it is an answer about the item, it is counted under abstained, and the whole vote vector travels out with it.

How to tell it is working. The boot line carries a composite instrument id and the seat count:

{"time":"2026-08-24T13:17:22.6485871-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"panel","judge_instrument":"panel(majority:gemini-2.5-flash,mistral-large-2,llama-3-70b)","configured_judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20,"panel_members":3}

A second boot of the same example under the same conditions, carrying the entry in this block. A single-model judge prints judge_model where this one prints configured_judge_model, which is the boot log saying that no seat used the top-level value. On a graded row the ballots come back beside the verdict:

"panel": {
  "members": [
    { "model": "claude-haiku-4-5", "family": "claude" },
    { "model": "gpt-4o-mini",      "family": "gpt" },
    { "model": "gemini-2.5-flash", "family": "gemini" }
  ],
  "votes": ["pass", "fail", "pass"],
  "rule": "majority",
  "margin": 0.3333333333333333
}

That is the panel object out of the tracked golden fixture the product pins its own wire shape against, with whitespace added and no value changed; go test ./internal/declog/ -run TestJudgeRowGolden compares the whole file byte for byte and passes on this tree. It is a shape pin rather than a live capture, and its three seats are the fixture’s. votes is positionally aligned with members and margin is derivable from it, so two pass against one fail over three seats prints (2 − 1) / 3.

Reach for something else when the seat count multiplies a bill you cannot carry, or when one grader is the whole measurement you need. pointwise-bool is the same question asked once.

The three this build turns away

All three are registered methods with working implementations, and none of them composes over the evals.evaluators[].judge transport, which is a provider Messages API. The composition check reads strings and builds nothing: no client, no credential read, no connection. It produces one sentence per method, and the sentence is the same one with the token substituted.

$ tokentriage validate --config m-pointwise-rubric.yaml
Error: configuration is invalid: evals.evaluators[1].judge.method: "pointwise-rubric" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel
Error: configuration is invalid: evals.evaluators[1].judge.method: "pointwise-rubric" is a registered §2.1 method that this build cannot compose over a `judge:` (provider API) transport — internal/evaluator ships no Scorer that drives it there, and the composition site will NOT quietly grade pointwise-bool instead. Over the §2.4 wire the same three methods are served by evaluator.NewSidecarJudgeScorer, which no composition site builds from config yet. Configure pointwise-bool, or configure a panel

Captured on 2026-08-24 from a source build, over a copy of the shipped journey-2-validated-judge.yaml example with one line changed. Both copies of the message are the binary’s. The same file through tokentriage run prints the same body under Error: evals: and also exits 1, and two consecutive runs of each were byte-identical. The §2.4 wire the message names is a self-hosted judging container, and every sidecar_judge: block that would configure one is turned away at both verbs as well.

pointwise-rubric

Would decide a 1..K rating against a named rubric, scaled to the unit interval, from the instruction, the response and the rubric’s anchor text.

evals:
  evaluators:
    - id: judge-task-completion
      judge:
        external: allow
        method: pointwise-rubric
        rubric: { id: task-completion-v1 }   # required with this method, and only with it
        model: gemini-2.5-flash
        api_key_env: JUDGE_API_KEY
      sample_rate: 0.20
      max_usd_per_day: 20.0

How it decides. The prompt asks for reasoning first and a verdict second, and the parser version records that requirement in its own name, scalar-v1;cot=required. The reply is {"score": <integer>} on the rubric’s scale, read from the same object span the boolean parser uses, and the score is rescaled to (score − 1) / (scale − 1). A score outside 1..K, a float, a string or a null is read as no verdict: an out-of-range value is turned away rather than clamped, because clamping a 0 would publish a broken reply as the worst legal grade. Both shipped rubrics carry five anchors, so the scale is 5 and a 3 scales to 0.5.

Knobs.

Key	Absent-reading	What it does
`judge.rubric.id`	required with this method	names the rubric; the shipped ids are `task-completion-v1` and `groundedness-v1`
`judge.rubric` under any other method	absent	written under another method it is turned away, because nothing would consume it

What it costs. One call per item wherever the method runs. On the evals.evaluators[].judge transport it costs nothing at all, because the composition check happens before a client is built, so the process stops without a credential being read or a connection being opened.

How it fails. Turned away at tokentriage validate and again at boot, by the message above, and the composition site does not fall back to pointwise-bool behind it. The rubric id itself is checked for shape and for coherence with method:, and its membership is looked up on no path a config file reaches, so a rubric id nobody registered produces the method message above with the rubric not mentioned at all. The lookup that would list the registered ids sits inside this method’s own runtime path, which a config-file deployment does not get to.

How to tell it is working. What a config-file deployment sees is the message above with "pointwise-rubric" in it, on both verbs. Nothing else about this method reaches a boot log or a row.

Reach for something else when you wanted a graded scale rather than a boolean. This build has no config-reachable substitute for that: pointwise-bool answers the same question with one bit, and a panel answers it with a vote vector and a margin.

pairwise

Decides which of two responses to one instruction is better, or that they tie, and derives that answer from both presentation orderings rather than from one.

shadow:
  duplicate:
    enabled: true
    sample_rate: 0.05
    max_usd_per_day: 5.0     # absent or 0 turns the whole feature off, silently
    pairwise: true           # the both-orderings co-run

That block is the leaf that composes. Writing method: pairwise under evals.evaluators[].judge instead gets the message above with "pairwise" in it.

How it decides. Two calls inside one grading call, with the pair presented once each way. Each reply is {"winner": "first"|"second"|"tie"} under parser version abtie-v1, in prompt coordinates: the tokens name slots rather than answers, and only the executor resolves a slot into a side, because it is the half that knows which side sat where.

Both answers then sit in one frame, where side A is the response your router served. The same side both times gives that side the win, two ties counting as agreement; different sides give no winner at all, and the trial abstains with swap_inconsistent. The confidence feature is swap agreement, 1.0 or 0.0. A pair run in one ordering has no representation: the only constructor takes both winners, so a half-run pair cannot be built.

Knobs.

Key under `shadow.duplicate`	Absent-reading	What it does
`pairwise`	`false`	co-runs this method beside the two pointwise grades on every graded duplicate
`enabled`	`false`	the duplication feature as a whole
`sample_rate`	`0`	the share of requests duplicated; `0` composes no sampler
`max_usd_per_day`	none, and absent or `0` composes no sampler	the cap over the duplicate and every judge call under it
`self_consistency_k`	`1`	k samples of each pointwise grade; the pairwise co-run is not k-wrapped
`judge`	absent	closed at the single value `oracle`

What it costs. Two judge calls per pair, on top of the two pointwise grades that plane already makes, and on top of the duplicate request itself. The validator does that arithmetic for the config you wrote and prints the total, “the grader makes 4 Anthropic Messages call(s) per sampled request under this config (2 pointwise grades × self_consistency_k, plus 2 for the pairwise co-run when shadow.duplicate.pairwise is on)”. A failure on the first ordering ends the trial before the second call is spent.

This plane also carries a cost the other one does not: it dials a compiled-in provider URL that no shadow.duplicate.* key moves, so it cannot be rehearsed against a listener on your own box, and the first pair you judge is billable.

How it fails. A transport or parse failure on either ordering writes nothing, not a winner and not an abstention, and is counted. A budget stop writes nothing either, on the grounds that what was stopped is what was not measured. A pair with an empty arm is skipped and counted, because a comparison against an absent answer is not a comparison. The flip itself is not a failure: it is the abstention above. The trials this method records are the swap set a validation run reads, and the share of them that flipped is the rate its ceiling applies to.

How to tell it is working. The boot log says the co-run composed:

{"time":"2026-08-24T13:17:34.7250048-07:00","level":"INFO","msg":"shadow duplication enabled","sample_rate":0.05,"max_usd_per_day":5,"judge_model":"","judge_key_env":"ANTHROPIC_API_KEY","pairwise":true,"self_consistency_k":1}

A third boot of the same example under the same conditions, with the block in this entry appended to it. The empty judge_model is that config’s: this plane takes its model and key from classifier.oracle.*, and the key env falls back to ANTHROPIC_API_KEY. Delete the max_usd_per_day line and the same file still validates at exit 0 and still boots, and its boot output then holds zero lines matching shadow against the three the capped one prints. That is the thing to check first. On a graded row the trial comes back as its own block, carrying both orderings and the derived agreement:

"pairwise": {
  "prompt_sha": "e5f6a7b",
  "winner_ab": "A",
  "winner_ba": "A",
  "agree": true
}

Those four keys are quoted out of the same tracked golden as the panel block above. That golden sets every field it can, so it is a field-by-field shape pin and not a row any run produced: read the keys, not the combination. prompt_sha here is the trial’s own version rather than the pointwise grades’, which answer to a different one.

Reach for something else when the deployment has no outbound path, or when the question is whether one answer is acceptable rather than which of two is better. Position bias, self-preference and panels is what the two orderings buy and where the method stops helping.

reference

Would decide whether one response is acceptable against a reference answer supplied alongside it.

evals:
  evaluators:
    - id: judge-task-completion
      judge:
        external: allow
        method: reference
        model: gemini-2.5-flash
        api_key_env: JUDGE_API_KEY
      sample_rate: 0.20
      max_usd_per_day: 20.0

How it decides. The same strict boolean parse pointwise-bool uses, bool-v1, over a prompt that renders a reference block around the reference text. An item whose reference is empty or blank is turned away before a call is spent, because rendering without that block would send pointwise-bool’s own prompt under this method’s version string, and the verdict would then be described by a version that did not produce it.

Knobs.

Key	Absent-reading	What it does
`judge.method`	`pointwise-bool`	`reference` selects this method, which takes no leaf of its own

What it costs. One call per item wherever the method runs, and nothing on a config-file deployment, for the same reason as pointwise-rubric: the process stops before a client exists.

How it fails. Turned away at tokentriage validate and again at boot, by the message above with "reference" in it. A second gap sits underneath that one and is the more permanent of the two: outside judgecore and its own tests, nothing in this build sets the reference field on a judged item, so the input this method requires has no producer. The one component that would carry a reference onto a wire is the self-hosted judging container, and every config leaf that would configure one is turned away too.

How to tell it is working. The message above with "reference" in it, on both verbs. Nothing about this method reaches a boot log or a row from a config file.

Reach for something else when you have a known-good answer to compare against. This build’s config-reachable graders read the instruction and the response only, so a reference answer belongs in the instruction text or in a human label, and the label route is Collect the labels a judge is measured against.

What each method publishes as a confidence feature

A judged verdict can carry one scalar beside it, and that scalar is the input a risk-controlled accept region would threshold. Which methods publish one is a closed table, mirrored into the package holding the quality plane’s descriptors, and every cell in it is measured by a test that builds the registered method, runs it at one sample and at three, and reads whether a confidence value came back.

Method	At `self_consistency_k: 1`	The feature’s name	Above `k: 1`
`pairwise`	yes	swap agreement	yes, voted per ordering, reporting the weaker ordering’s margin
`panel`	yes	panel vote margin	no; this build mints no version above `k: 1` for a panel, and the constructor stops one
`pointwise-bool`	no	none	yes, the k-sample margin
`pointwise-rubric`	yes	rubric scalar margin	no; a vote over a scalar is a mean rather than a ballot
`reference`	no	none	yes, the k-sample margin

The k in that table is self_consistency_k, which is a wrapper rather than a sixth method: it asks the same method k times and takes a strict majority, and a vote with no strict majority is an error rather than an abstention, because the abstention vocabulary has no token for it. On evals.evaluators[].judge any value above 1 is turned away at both verbs, so the only k above 1 a config file composes is shadow.duplicate.self_consistency_k, on the plane that needs outbound traffic.

The feature is published whether or not anything reads it. The shipped sampler builds its judging plan with no accept region installed, which the product pins with a test of its own, so no abstention in this build comes from thresholding one of these numbers. When a judge can’t tell has the four abstention reasons and says which of them an operator can reach.

What this page does not cover

Where a method’s dollars turn up, and the cap that stops them. The per-method multiplier is above; the price per call, the meters that carry it and the daily cap’s post-hoc arithmetic are What judging costs, and how to cap it.

The rest of the config surface. The calibration, drift and validation blocks beside judge: are Evals and calibration configuration, and the pasteable walk through the judge block itself is Configure a judge.

Which of the two registered judge ids to configure. judge-groundedness publishes no pass floor, so the runner leaves every row it grades without a verdict and no gold set forms for it. Configure judge-task-completion, whose metric carries a floor of 1.0.

What a method’s verdict is worth once rows arrive. A judge outcome is reported until a validation run promotes its subject, and the walk that promotes a judge sets out how.