Run a cross-family panel — TokenTriage Docs

One model grading another model’s answers gives you one opinion. If that grader has a blind spot, a phrasing it likes or a format it punishes, every score you collect carries the same one, and nothing in the score says so.

Three graders from three vendors have three different blind spots, and the rows where they disagree are the rows worth reading. That is what a panel is for: one judge with several seats, one verdict, and the ballots kept beside it.

It is also the one multi-opinion judging setup that composes from a config file in this build, and it multiplies your judging bill by its seat count. This page seats one, runs it past the validator, boots it, and reads a vote off a row.

You need a single-model judge working first. Configure a judge is that surface and how a judge call happens is what one grading call does. You need no extra subsystem, no new port and no second binary; you do need a credential for each seat’s provider, and a seat count you have priced. That price is what judging costs, and it is worth reading before you widen a panel rather than after.

1. Seat the panel

The seat rules are floors the config layer applies, so a panel that breaks one is stopped before anything boots. Seven of them decide whether your block loads at all.

The rule	What it takes	What a broken one does
seats	at least three `panel.members[]`	stopped, with `need 3, have 2`
families	at least two distinct model families	stopped, and the message calls diversity a floor rather than advice
`judge.model`	required, on a panel too	stopped, even though no seat reads the value
`judge.api_key_env`	required, on a panel too	stopped, for the same reason
`judge.endpoint_env`	not readable on a panel	stopped; name a variable on each seat instead
`judge.self_consistency_k`	`1`, or absent	stopped: a panel mints no version above `k=1`
`panel.rule`	`majority`, which is also the absent-reading	any other value is stopped

The two required leaves read like leftovers from the single-model form. Take either one out and the config is stopped, one message each. A panel’s seats are its own transports, so the top-level judge.model is sent nowhere, and the boot log publishes it under the name configured_judge_model to say so. Write both anyway.

Here is the evaluator entry, as a fragment of a working config:

evals:
  evaluators:
    # …
    - id: judge-task-completion
      sample_rate: 0.20
      max_usd_per_day: 20.0          # one cap for the whole panel
      judge:
        external: allow              # the air-gap gate, on the panel as a whole
        method: panel
        model: gemini-2.5-flash      # required on a panel, and no seat reads it
        api_key_env: JUDGE_API_KEY   # required on a panel, for the same reason
        panel:
          rule: majority             # the whole vocabulary, and the absent-reading
          members:
            - { model: gemini-2.5-flash, api_key_env: JUDGE_GEMINI_KEY }
            - { model: mistral-large-2,  api_key_env: JUDGE_MISTRAL_KEY }
            - { model: llama-3-70b,      api_key_env: JUDGE_LLAMA_KEY }

Three seats, three families, three credentials, one cap. The family is the leading run of letters in the model id, so claude-haiku-4-5 and claude-opus-4-8 are one family and gemini-2.5-flash is another; a bigger model is not a second family. Each seat may also carry endpoint_env, naming a variable that holds the URL that seat posts to, and one seat’s key travels to one seat’s endpoint.

Every seat is checked a second time at grading time, against the model that produced the answer under judgement. That check fails closed: one seat sharing the generator’s family stops the whole invocation rather than that seat alone.

What that block does with one item

Read it left to right. One item fans out to every seat, each seat runs the same boolean method and answers with one ballot, and the tally on the right reduces the ballots to one verdict. The two boxes under the tally are the two answers a panel can give.

Two details in that drawing decide your seat count. Ballots come from a closed pair of tokens, pass and fail, and a decision needs a strict majority: twice the leading count has to exceed the seat count. Three seats therefore always produce one, because the closest vote available to them is two to one. Seat an even number and a tie clears no majority, so the panel abstains with the reason panel_split and the ballots still travel.

2. Validate before you boot

Run the validator over the file. It is the layer that applies every floor in the table above, and it prints two warnings on a panel that loads:

$ tokentriage validate --config tokentriage.yaml
warning: evals.evaluators[1].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[1].judge.model, using the key named by evals.evaluators[1].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
warning: evals.evaluators[1].judge.panel: this judge is a PANEL of 3 members, so a sampled request costs 3 judge calls, not one — every member listed under evals.evaluators[1].judge.panel.members is its own transport and every provider-backed one is its own phone-out, all of them capped by the single evals.evaluators[1].max_usd_per_day. The panel is ONE instrument: it votes with all its members or it does not vote, so a member that cannot be reached refuses the whole invocation rather than being dropped from the vote. No member id is repeated back here — read the lines this message points at
warning: evals.evaluators[1].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[1].judge.model, using the key named by evals.evaluators[1].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
warning: evals.evaluators[1].judge.panel: this judge is a PANEL of 3 members, so a sampled request costs 3 judge calls, not one — every member listed under evals.evaluators[1].judge.panel.members is its own transport and every provider-backed one is its own phone-out, all of them capped by the single evals.evaluators[1].max_usd_per_day. The panel is ONE instrument: it votes with all its members or it does not vote, so a member that cannot be reached refuses the whole invocation rather than being dropped from the vote. No member id is repeated back here — read the lines this message points at
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 2
  - anthropic              https://api.anthropic.com
  - openai                 https://api.openai.com
tiers:     2 (cheapest -> most capable)
  0. cheap      -> openai/gpt-5-mini
  1. frontier   -> anthropic/claude-opus-4-8
rules:     0
OK: configuration is valid

That capture is a scratch copy of the shipped examples/evals/journey-2-validated-judge.yaml example with the judge entry above pasted over its judge entry, run on 2026-08-24 against a source build on Windows. Exit status was 0, and two consecutive runs were byte-identical. Two things in it are that file’s rather than yours: the evaluator index [1], which is where the judge entry sits in this config, and the summary block underneath, which reads back that file’s listeners, upstreams and tiers. Warnings go to stdout and errors go to stderr, so a fence like this one shows both because a terminal does.

Each warning appears twice, and the doubling is the binary’s. Two validation passes each produce it and one print loop emits both, on every message this surface prints. A transcript of this command showing one copy of a warning has been edited.

The panel warning is the sentence to keep. It names the seat count, states the call multiplier in the same breath, and says which cap the extra calls land under: the one max_usd_per_day on the evaluator entry, not one per seat.

3. Boot it, and read the instrument

Start the proxy and read the wiring line. It is written once, at composition, before any traffic:

{"time":"2026-08-24T12:25:30.7715974-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"panel","judge_instrument":"panel(majority:gemini-2.5-flash,mistral-large-2,llama-3-70b)","configured_judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20,"panel_members":3}

One line out of a longer boot log, captured the same day from the same source build, with the listeners moved to port 0 and replica.name pinned so the capture carries no hostname. The timestamp and its local offset are that machine’s. Everything else is read from the config and the shipped defaults, so the same file gives you the same values.

Four fields on it answer four different questions:

Field	What it tells you
`judge_method`	`panel`, read off the composed scorer rather than off the config text
`judge_instrument`	the composite id every row this judge writes will be attributed to
`configured_judge_model`	the top-level `judge.model`, published under a name that says no seat used it
`panel_members`	how many calls one sampled request will make

A single-model judge prints judge_model where this line prints configured_judge_model, and its judge_instrument is the model id. Reading judge_instrument is how you tell from boot output alone that a panel composed rather than quietly grading with one model.

4. Read the vote vector off a row

A panel-graded outcome carries a panel block beside its judge disclosure. This is the panel object from the tracked golden fixture the product pins its own wire shape against:

    "panel": {
      "members": [
        { "model": "claude-haiku-4-5",  "family": "claude" },
        { "model": "gpt-4o-mini",       "family": "gpt" },
        { "model": "gemini-2.5-flash",  "family": "gemini" }
      ],
      "votes": ["pass", "fail", "pass"],
      "rule": "majority",
      "margin": 0.3333333333333333
    }

That is the fixture’s own bytes with whitespace added and no value changed; go test compares the whole file byte for byte on every run, and it passes on this tree. It is a pinned shape rather than a live capture, and its three seats are the fixture’s, not the config above.

Field	What it says
`members[].model`	the model that sat in that seat
`members[].family`	the family resolved for it when the panel was composed
`votes`	the ballots, positionally aligned with `members`
`rule`	the aggregation rule that produced the verdict
`margin`	the lead over the seat count

margin is derivable from votes, which is what makes it checkable: two pass against one fail over three seats is (2 − 1) / 3, the 0.3333… printed above. A 3–0 on the same panel prints 1. Both decide the same way and they are not the same evidence.

The judge disclosure beside it names the panel as one instrument. judge.method reads panel and judge.model reads the composite panel(majority:…) string, the same one the boot line printed as judge_instrument. A promotion is pinned to that composite and to the version the member tuple hashes into, so moving one seat moves both halves of the pin and the subject goes back to a reported basis until it is validated again.

5. What one panel-graded row costs

Three seats is three provider calls for one graded row, and the validator says so before you boot. There is one budget behind them: max_usd_per_day on the evaluator entry covers the whole fan-out rather than a seat each.

The cap is re-read before each seat, which is the part worth planning for. A panel that is stopped at its third seat has made two calls, and those dollars leave your account. They do not all reach the counter the cap is checked against, which is a measured gap and the reason to price a panel before you widen one.

What judging costs, and how to cap it does the arithmetic: the multiplier per method, what a call costs on a measured ledger, and where the resulting dollars show up.

What can go wrong

A seat cannot be reached. A transport failure, an unparseable reply or a non-boolean verdict on any seat stops the whole invocation, which is counted. No row is written, and the seats that already answered are still billed. The panel is validated as one instrument and its version hashes the member list, so a verdict from the surviving seats would be a different instrument answering under this one’s version string.

Diagnosing which seat broke is harder than it should be. A dead seat, a panicking seat, a mid-fan-out budget stop and a malformed vote vector are tallied apart inside the panel and reach an operator as one skip named evaluator_error, so a seat that fails on every row reads as a broken evaluator rather than as a bill.

A seat shares the generator’s family. The same-family check runs per seat against the model that produced the graded answer, and it fails closed. An unreadable family on either side is stopped in the same way, because “we could not tell” is not the same answer as “they differ”. Boot warns when no configured tier is cross-family with the judge, so you find this before your rows do rather than after.

The cap runs out mid-fan-out. Identical shape to a dead seat: the invocation is stopped, the completed calls are billed, and the stop is counted under the budget’s own reason rather than the transport’s.

An even seat count ties. Four seats at 2–2 clear no strict majority, so the outcome is an abstention carrying panel_split, with the vote vector and the margin disclosed. That is an answer about the item rather than a failure, and it is counted apart from the refusals above. When a judge can’t tell has the four abstention reasons.

self_consistency_k above 1. A panel is stopped for it at tokentriage validate and again at boot:

$ tokentriage validate --config tokentriage.yaml
Error: configuration is invalid: config tokentriage.yaml: evals.evaluators[1].judge.self_consistency_k: 3, but evals.evaluators[1].judge.method resolves to panel — judgecore mints NO k>1 MethodVersion for a panel (SelfConsistencyVersion is keyed by method id, while a panel's tuple hashes its member list), so a k-sampled panel would execute and could never be version-disclosed, and a version that cannot move is a promotion B.7 can never reset. Configure the panel at k=1 (§2.1)

One copy of the doubled message, from the same probe set; the config path is this run’s and the evaluator index is this file’s. Sampling the same seat repeatedly is a different leaf on a different plane, and it is shadow.duplicate.self_consistency_k.

What this page does not cover

Whether a panel is the right control for your problem. Position bias, self-preference and the swap trial are position bias, self-preference and panels, which is where the panel sits among the other controls rather than beside its config.

The other four methods. pointwise-bool is the absent-reading default and the only other setup that composes over a judge: transport; judge methods is the catalogue, with what each decides and whether a config file can reach it.

Pairwise judging. It reaches production on shadow.duplicate.pairwise, a different leaf on a different plane, and that plane has no endpoint leaf at all: it dials a compiled-in provider URL, so it needs outbound traffic and cannot be rehearsed against a listener on your own box.

Getting the panel promoted. A panel reaches a measured basis through the same protocol a single judge does, scoped to the composite instrument, and the walk that promotes a judge sets that protocol out gate by gate.