# Run a cross-family panel Compose a panel from two or more model families, read the vote vector, and price the call multiplier before you turn it on. Source: https://tokentriage.com/docs/judges/run-a-panel --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import JudgePanelDiagram from '../../../components/JudgePanelDiagram.astro'; One model grading another model's answers gives you one opinion. If that grader has a blind spot, a phrasing it likes or a format it punishes, every score you collect carries the same one, and nothing in the score says so. Three graders from three vendors have three different blind spots, and the rows where they disagree are the rows worth reading. That is what a panel is for: one judge with several seats, one verdict, and the ballots kept beside it. It is also the one multi-opinion judging setup that composes from a config file in this build, and it multiplies your judging bill by its seat count. This page seats one, runs it past the validator, boots it, and reads a vote off a row. You need a single-model judge working first. [Configure a judge](/docs/judges/configure-a-judge) is that surface and [how a judge call happens](/docs/judges/how-judging-works) is what one grading call does. You need no extra subsystem, no new port and no second binary; you do need a credential for each seat's provider, and a seat count you have priced. That price is [what judging costs](/docs/judges/cost-and-budget), and it is worth reading before you widen a panel rather than after. ## 1. Seat the panel The seat rules are floors the config layer applies, so a panel that breaks one is stopped before anything boots. Seven of them decide whether your block loads at all. | The rule | What it takes | What a broken one does | |---|---|---| | seats | at least three `panel.members[]` | stopped, with `need 3, have 2` | | families | at least two distinct model families | stopped, and the message calls diversity a floor rather than advice | | `judge.model` | required, on a panel too | stopped, even though no seat reads the value | | `judge.api_key_env` | required, on a panel too | stopped, for the same reason | | `judge.endpoint_env` | not readable on a panel | stopped; name a variable on each seat instead | | `judge.self_consistency_k` | `1`, or absent | stopped: a panel mints no version above `k=1` | | `panel.rule` | `majority`, which is also the absent-reading | any other value is stopped | The two required leaves read like leftovers from the single-model form. Take either one out and the config is stopped, one message each. A panel's seats are its own transports, so the top-level `judge.model` is sent nowhere, and the boot log publishes it under the name `configured_judge_model` to say so. Write both anyway. Here is the evaluator entry, as a fragment of a working config: ```yaml evals: evaluators: # … - id: judge-task-completion sample_rate: 0.20 max_usd_per_day: 20.0 # one cap for the whole panel judge: external: allow # the air-gap gate, on the panel as a whole method: panel model: gemini-2.5-flash # required on a panel, and no seat reads it api_key_env: JUDGE_API_KEY # required on a panel, for the same reason panel: rule: majority # the whole vocabulary, and the absent-reading members: - { model: gemini-2.5-flash, api_key_env: JUDGE_GEMINI_KEY } - { model: mistral-large-2, api_key_env: JUDGE_MISTRAL_KEY } - { model: llama-3-70b, api_key_env: JUDGE_LLAMA_KEY } ``` Three seats, three families, three credentials, one cap. The family is the leading run of letters in the model id, so `claude-haiku-4-5` and `claude-opus-4-8` are one family and `gemini-2.5-flash` is another; a bigger model is not a second family. Each seat may also carry `endpoint_env`, naming a variable that holds the URL that seat posts to, and one seat's key travels to one seat's endpoint. Every seat is checked a second time at grading time, against the model that produced the answer under judgement. That check fails closed: one seat sharing the generator's family stops the whole invocation rather than that seat alone. ### What that block does with one item Read it left to right. One item fans out to every seat, each seat runs the same boolean method and answers with one ballot, and the tally on the right reduces the ballots to one verdict. The two boxes under the tally are the two answers a panel can give. Two details in that drawing decide your seat count. Ballots come from a closed pair of tokens, `pass` and `fail`, and a decision needs a strict majority: twice the leading count has to exceed the seat count. Three seats therefore always produce one, because the closest vote available to them is two to one. Seat an even number and a tie clears no majority, so the panel abstains with the reason `panel_split` and the ballots still travel. The vote vector is disclosed whether the panel decided or abstained. A 2–1 and a 3–0 are different facts about how sure your instrument was, and a verdict that arrived without its ballots would let you read the first as the second. ## 2. Validate before you boot Run the validator over the file. It is the layer that applies every floor in the table above, and it prints two warnings on a panel that loads: ```console $ tokentriage validate --config tokentriage.yaml warning: evals.evaluators[1].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[1].judge.model, using the key named by evals.evaluators[1].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them warning: evals.evaluators[1].judge.panel: this judge is a PANEL of 3 members, so a sampled request costs 3 judge calls, not one — every member listed under evals.evaluators[1].judge.panel.members is its own transport and every provider-backed one is its own phone-out, all of them capped by the single evals.evaluators[1].max_usd_per_day. The panel is ONE instrument: it votes with all its members or it does not vote, so a member that cannot be reached refuses the whole invocation rather than being dropped from the vote. No member id is repeated back here — read the lines this message points at warning: evals.evaluators[1].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[1].judge.model, using the key named by evals.evaluators[1].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them warning: evals.evaluators[1].judge.panel: this judge is a PANEL of 3 members, so a sampled request costs 3 judge calls, not one — every member listed under evals.evaluators[1].judge.panel.members is its own transport and every provider-backed one is its own phone-out, all of them capped by the single evals.evaluators[1].max_usd_per_day. The panel is ONE instrument: it votes with all its members or it does not vote, so a member that cannot be reached refuses the whole invocation rather than being dropped from the vote. No member id is repeated back here — read the lines this message points at mode: ledger listen: 127.0.0.1:8787 admin: 127.0.0.1:9090 upstreams: 2 - anthropic https://api.anthropic.com - openai https://api.openai.com tiers: 2 (cheapest -> most capable) 0. cheap -> openai/gpt-5-mini 1. frontier -> anthropic/claude-opus-4-8 rules: 0 OK: configuration is valid ``` That capture is a scratch copy of the shipped `examples/evals/journey-2-validated-judge.yaml` example with the judge entry above pasted over its judge entry, run on 2026-08-24 against a source build on Windows. Exit status was `0`, and two consecutive runs were byte-identical. Two things in it are that file's rather than yours: the evaluator index `[1]`, which is where the judge entry sits in this config, and the summary block underneath, which reads back that file's listeners, upstreams and tiers. Warnings go to stdout and errors go to stderr, so a fence like this one shows both because a terminal does. **Each warning appears twice, and the doubling is the binary's.** Two validation passes each produce it and one print loop emits both, on every message this surface prints. A transcript of this command showing one copy of a warning has been edited. The panel warning is the sentence to keep. It names the seat count, states the call multiplier in the same breath, and says which cap the extra calls land under: the one `max_usd_per_day` on the evaluator entry, not one per seat. ## 3. Boot it, and read the instrument Start the proxy and read the wiring line. It is written once, at composition, before any traffic: ```console {"time":"2026-08-24T12:25:30.7715974-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"panel","judge_instrument":"panel(majority:gemini-2.5-flash,mistral-large-2,llama-3-70b)","configured_judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20,"panel_members":3} ``` One line out of a longer boot log, captured the same day from the same source build, with the listeners moved to port `0` and `replica.name` pinned so the capture carries no hostname. The timestamp and its local offset are that machine's. Everything else is read from the config and the shipped defaults, so the same file gives you the same values. Four fields on it answer four different questions: | Field | What it tells you | |---|---| | `judge_method` | `panel`, read off the composed scorer rather than off the config text | | `judge_instrument` | the composite id every row this judge writes will be attributed to | | `configured_judge_model` | the top-level `judge.model`, published under a name that says no seat used it | | `panel_members` | how many calls one sampled request will make | A single-model judge prints `judge_model` where this line prints `configured_judge_model`, and its `judge_instrument` is the model id. Reading `judge_instrument` is how you tell from boot output alone that a panel composed rather than quietly grading with one model. ## 4. Read the vote vector off a row A panel-graded outcome carries a `panel` block beside its judge disclosure. This is the `panel` object from the tracked golden fixture the product pins its own wire shape against: ```json "panel": { "members": [ { "model": "claude-haiku-4-5", "family": "claude" }, { "model": "gpt-4o-mini", "family": "gpt" }, { "model": "gemini-2.5-flash", "family": "gemini" } ], "votes": ["pass", "fail", "pass"], "rule": "majority", "margin": 0.3333333333333333 } ``` That is the fixture's own bytes with whitespace added and no value changed; `go test` compares the whole file byte for byte on every run, and it passes on this tree. It is a pinned shape rather than a live capture, and its three seats are the fixture's, not the config above. | Field | What it says | |---|---| | `members[].model` | the model that sat in that seat | | `members[].family` | the family resolved for it when the panel was composed | | `votes` | the ballots, positionally aligned with `members` | | `rule` | the aggregation rule that produced the verdict | | `margin` | the lead over the seat count | `margin` is derivable from `votes`, which is what makes it checkable: two `pass` against one `fail` over three seats is `(2 − 1) / 3`, the `0.3333…` printed above. A 3–0 on the same panel prints `1`. Both decide the same way and they are not the same evidence. The judge disclosure beside it names the panel as one instrument. `judge.method` reads `panel` and `judge.model` reads the composite `panel(majority:…)` string, the same one the boot line printed as `judge_instrument`. A promotion is pinned to that composite and to the version the member tuple hashes into, so moving one seat moves both halves of the pin and the subject goes back to a `reported` basis until it is validated again. ## 5. What one panel-graded row costs Three seats is three provider calls for one graded row, and the validator says so before you boot. There is one budget behind them: `max_usd_per_day` on the evaluator entry covers the whole fan-out rather than a seat each. The cap is re-read before each seat, which is the part worth planning for. A panel that is stopped at its third seat has made two calls, and those dollars leave your account. They do not all reach the counter the cap is checked against, which is a measured gap and the reason to price a panel before you widen one. [What judging costs, and how to cap it](/docs/judges/cost-and-budget) does the arithmetic: the multiplier per method, what a call costs on a measured ledger, and where the resulting dollars show up. ## What can go wrong **A seat cannot be reached.** A transport failure, an unparseable reply or a non-boolean verdict on any seat stops the whole invocation, which is counted. No row is written, and the seats that already answered are still billed. The panel is validated as one instrument and its version hashes the member list, so a verdict from the surviving seats would be a different instrument answering under this one's version string. Diagnosing which seat broke is harder than it should be. A dead seat, a panicking seat, a mid-fan-out budget stop and a malformed vote vector are tallied apart inside the panel and reach an operator as one skip named `evaluator_error`, so a seat that fails on every row reads as a broken evaluator rather than as a bill. A stopped invocation still spends, and the daily cap does not always see it. A seat that reports no token usage makes the whole invocation's spend unpriced, and an unpriced settlement is counted and excluded rather than added, so the earlier seats' priced dollars land on your provider bill without advancing the cap. Measured at `$0.0100` spent against a counter reading `$0.0000`. Repair the seat, and until you have, watch the unpriced-settlement disclosure on that evaluator's metacost line rather than the cap; [what judging costs](/docs/judges/cost-and-budget) has the capture. **A seat shares the generator's family.** The same-family check runs per seat against the model that produced the graded answer, and it fails closed. An unreadable family on either side is stopped in the same way, because "we could not tell" is not the same answer as "they differ". Boot warns when no configured tier is cross-family with the judge, so you find this before your rows do rather than after. **The cap runs out mid-fan-out.** Identical shape to a dead seat: the invocation is stopped, the completed calls are billed, and the stop is counted under the budget's own reason rather than the transport's. **An even seat count ties.** Four seats at 2–2 clear no strict majority, so the outcome is an abstention carrying `panel_split`, with the vote vector and the margin disclosed. That is an answer about the item rather than a failure, and it is counted apart from the refusals above. [When a judge can't tell](/docs/judges/abstention) has the four abstention reasons. **`self_consistency_k` above 1.** A panel is stopped for it at `tokentriage validate` and again at boot: ```console $ tokentriage validate --config tokentriage.yaml Error: configuration is invalid: config tokentriage.yaml: evals.evaluators[1].judge.self_consistency_k: 3, but evals.evaluators[1].judge.method resolves to panel — judgecore mints NO k>1 MethodVersion for a panel (SelfConsistencyVersion is keyed by method id, while a panel's tuple hashes its member list), so a k-sampled panel would execute and could never be version-disclosed, and a version that cannot move is a promotion B.7 can never reset. Configure the panel at k=1 (§2.1) ``` One copy of the doubled message, from the same probe set; the config path is this run's and the evaluator index is this file's. Sampling the same seat repeatedly is a different leaf on a different plane, and it is `shadow.duplicate.self_consistency_k`. ## What this page does not cover **Whether a panel is the right control for your problem.** Position bias, self-preference and the swap trial are [position bias, self-preference and panels](/docs/judges/bias-controls), which is where the panel sits among the other controls rather than beside its config. **The other four methods.** `pointwise-bool` is the absent-reading default and the only other setup that composes over a `judge:` transport; [judge methods](/docs/reference/judge-methods) is the catalogue, with what each decides and whether a config file can reach it. **Pairwise judging.** It reaches production on `shadow.duplicate.pairwise`, a different leaf on a different plane, and that plane has no endpoint leaf at all: it dials a compiled-in provider URL, so it needs outbound traffic and cannot be rehearsed against a listener on your own box. **Getting the panel promoted.** A panel reaches a `measured` basis through the same protocol a single judge does, scoped to the composite instrument, and [the walk that promotes a judge](/docs/judges/agreement) sets that protocol out gate by gate.