Position bias, self-preference and panels

Ask a model which of two answers is better and it answers. It will not tell you that it picked the one it saw first, or the one written in a house style it recognises as its own. Those are preferences about presentation rather than about quality, and they come back looking identical to a considered judgement.

Nothing downstream can separate the two. The verdict parses, the row is written, the pass rate moves, and no error is raised for a monitor to catch. Run that judge across four thousand rows and its preference is inside your quality figure, at full strength, carrying the figure’s authority.

Three things in a judging setup work against that, and this page is about what each one costs and where each one stops working. How a judge call happens is the mechanism underneath. There is no configuration on this page; the config surface is Configure a judge.

A judge that got 258 of 314 pairs wrong

Before any mechanism, here is the failure measured rather than described.

This build vendors LLMBar, a public 419-pair meta-evaluation corpus. Each pair carries a human preference between two answers to one instruction, and its four adversarial subsets are built so that the more attractive-looking answer is the wrong one. The product’s benchmark harness drives scripted judges over every pair, through the shipped judge transport, over HTTP to a listener on the same machine.

$ EVALPROG_BENCHMARK_REALRUN_DIR=./llmbar-evidence \
    go test -count=1 ./internal/evalprog/ -run TestRealRunLLMBarAgreement
ok  	github.com/tokentriage/platform/internal/evalprog	2.371s
$ cat ./llmbar-evidence/JG4-03-realrun-transcript.txt
JG4-03 REAL RUN — LLMBar meta-eval agreement, pointwise + pairwise
run clock            2026-08-22T13:00:00Z
corpus               vendored third_party/llmbar (MIT, LLMBar/LICENSE:1,3 read on disk 2026-08-22)
judge transport      REAL *shadow.Judge over REAL HTTP to a loopback httptest endpoint
model behind socket  a scripted RULE, not a vendor model — no egress, no credential
MODE                 FULL: every imported pair judged
[…5 corpus-import lines, 2 lines of ledger and body-store counts, and the 30-row per-subset table elided…]
HAND-DERIVABLE EXPECTATIONS (each a property of the RULE, not of the corpus):
  first-slot/pairwise        pairs=419 graded=  0 abstained=419 undecided=  0 calls= 838  agreement=undefined
  first-slot/pointwise-bool  pairs=419 graded=  0 abstained=  0 undecided=419 calls= 838  agreement=undefined
  gold-reading/pairwise      pairs=419 graded=419 abstained=  0 undecided=  0 calls= 838  agreement=1.0000 (419/419)
  gold-reading/pointwise-bool pairs=419 graded=419 abstained=  0 undecided=  0 calls= 838  agreement=1.0000 (419/419)
  length/pairwise            pairs=419 graded=409 abstained= 10 undecided=  0 calls= 838  agreement=0.2714 (111/409)
  length/pointwise-bool      pairs=419 graded=113 abstained=  0 undecided=306 calls= 838  agreement=0.2124 (24/113)
  first-slot/pairwise abstained on 419/419 — a flip is an abstention, never a coerced winner, and the reading's rate is UNDEFINED rather than 0.0
  first-slot/pointwise-bool undecided on 419/419 — two identical gradings are not a tie and are OUT of the denominator
  gold-reading/pairwise agreed on 419/419 — the plumbing loses nothing and inverts nothing
  length/pairwise natural=0.5789 (55/95) vs adversarial=0.1783 (56/314) — separation +0.4006

EVERY READING ABOVE IS `benchmark_agreement` AND LICENSES NOTHING.
  benchmark_agreement is a PRE-SCREEN and a prompt-regression check over a public meta-evaluation corpus. It is NOT domain validation: it licenses no promotion, no ceiling, no `measured` basis, and no verdict about this deployment's traffic. Leaderboard validity is not your-traffic validity.

The elision is marked and drops only per-subset detail that the totals below it already carry. Two things in that block belong to the run rather than to the corpus: the elapsed 2.371s, and the evidence directory named on the command line, which is where the harness wrote that transcript. Everything else is a constant in the harness or arithmetic over the vendored corpus, so the same two commands give you the same table twice. The whole corpus is scanned only when that directory is set; without it each subset is bounded to 20 pairs and every count is smaller.

Four things in there are worth reading slowly.

The judges are the fixture. The corpus is not. Three scripted rules answered from a loopback socket, so no vendor model was asked, no credential was used, and no rate above is a claim about anybody’s model. LLMBar’s pairs and their human preferences are the published corpus, and the transport, the ledger writes, the body store and the frame resolution are the product’s own.

gold-reading is the control. A rule that answers from the corpus’s own gold agrees on 419 of 419 under both methods. The plumbing loses nothing and inverts nothing, which is what makes the other two rows readable as findings about the rules.

length is the failure, and it is quiet. Its rule is one line: prefer the longer response. On the natural subset it matches the human preference on 55 of 95 decided pairs. On the four adversarial subsets, where the longer answer is built to be the wrong one, it matches on 56 of 314, so 258 pairs came back with a confident winner a person would not have picked. Ten abstained and none failed. There is no error anywhere in that column.

first-slot is the same failure with the preference this page is named for. Its rule is to name whichever response is presented first. Under pairwise the build shows the pair twice, once each way, so the rule names one side the first time and the other side the second, on every pair: abstained=419. Presented once, it would have swept the first slot 419 times and read as a judge with firm opinions.

agreement=undefined on those two rows is the part most often misread. The rate travels as a pointer, and it comes back null when the denominator is zero.

{
  "reading": "benchmark_agreement",
  "benchmark": "llmbar",
  "subset": "natural",
  "method": "pairwise",
  "method_version": "3a35654",
  "model": "loopback-scripted-judge",
  "pairs_scanned": 100,
  "graded": 0,
  "agreed": 0,
  "disagreed": 0,
  "abstained": 100,
  "undecided": 0,
  "skipped_no_body": 0,
  "failed": 0,
  "judge_calls": 200,
  "rate": null
}

That is the first-slot judge’s reading over the natural subset, written by the same run, with the two trailing keys dropped: a long disclaimer string and the ran_at clock. Every key shown is the run’s own and no value was changed. rate: null records that no pair produced a preference, which is UNDEFINED and is not the claim 0.0. abstained: 100 beside it says why, and judge_calls: 200 says the judge was asked twice per pair to get there.

Two orderings, and a verdict derived from both

A positional preference has one property the others lack: it is a function of the order, so reversing the order reverses it. That is what the swap gate uses. The judge is not asked to be fair. It is asked twice, and the two answers are compared with each other.

The drawing has the whole method on it, including the leaf that turns it on.

Read it left to right. One pair goes in, side A being the answer the router served and side B the duplicate’s. Two calls go out inside one grading call, with the presentation order reversed between them, and both answers name the same two sides so they compare directly. The join is the method: the same side both times and the trial has a winner, with two ties counting as agreement; different sides and the trial has no winner at all, abstains with swap_inconsistent, and the flip is counted into the swap set rather than averaged into a rate. The label along the top names the plane this trial runs on, and the following section is about what that plane costs you.

What it costs. Two provider calls for every judged pair, both charged against the one daily cap on shadow.duplicate. And a pair that flips yields no verdict, so a judge with a positional preference answers fewer pairs than you gave it. That is the trade the method makes: a smaller denominator you can describe, instead of a larger one you cannot.

A pair run in one ordering has no representation. The type carrying a pairwise verdict takes both winners or nothing, so a half-run pair cannot be built at all. The validation walk turns away a swap set carrying one-ordering pairs by name, before it looks at any rate, because a flip rate over half-run pairs has a denominator that does not describe the experiment.

What the flips are for. They are the swap set, and the validation walk gates pairwise promotion on it: at least 30 complete pairs, and a flip rate at or under 0.10. Above that ceiling the subject stays reported for pairwise use, and the record names the pairs that flipped rather than only printing the rate. The walk that promotes a judge covers the whole ordered check, and notes that a pointwise-bool judge prints those two floors and reaches neither.

How it fails. It measures position and nothing else. A judge that prefers its own family’s prose prefers it in both orderings, so it agrees with itself, its flip rate reads 0.0, and the gate passes it with the best result available on that axis. A clean swap set is evidence about position and about nothing else. Self-preference is handled somewhere else entirely, which is the section after next.

The plane pairwise runs on

That recommendation arrives with a condition attached, and the condition belongs here rather than in a note at the end.

judge.method: pairwise under evals.evaluators[].judge is turned away at tokentriage validate and again at boot, by name. Two configurations compose over that transport in this build: pointwise-bool, which is what an absent method: reads as, and panel at self_consistency_k: 1. The both-orderings trial reaches production through shadow.duplicate.pairwise, which is a different leaf on a different plane.

The shadow-duplicate judge dials a compiled-in provider URL and no shadow.duplicate.* key moves it, so that plane needs egress. The evals-plane judge takes evals.evaluators[].judge.endpoint_env and can be pointed at a loopback listener, which is the path to use for a rehearsal with no outbound traffic. So the plane that composes the swap trial is the one you cannot rehearse locally, and the plane you can rehearse locally is the one that turns the method away.

That has a consequence for the evidence behind this page, and it is worth stating plainly. runPairwise is called and fed from the both-orderings trials the shadow judge recorded on the ledger; what has been exercised is a planted ledger under test, and no capture drives a live pairwise judge end to end. The missing live pairwise capture is structural in this build rather than merely undone: the shadow plane is the only one that composes pairwise, and its judge transport cannot be pointed at a loopback host. Turning the swap gate on therefore costs provider traffic from the first pair, with no free rehearsal ahead of it.

The benchmark at the top of this page is the nearest thing to a rehearsal that runs with the network unplugged. It exercises the same method against the same transport type, and it runs inside the product’s own test binary rather than from a shipped verb.

The same-family rule

Symmetry cannot cancel a family preference, because a judge that likes its own family’s prose likes it in both orderings. So this one is settled before any traffic arrives, at configuration time.

A judge model’s family must differ from the family of every configured tier’s model. Every tier is a possible generator, because config cannot know which one the router will serve on the row this judge will later grade, so the judge has to differ from all of them. The family is the product line rather than the size, so claude-haiku-4-5 and claude-opus-4-8 are one family. The vendor path segment is stripped first, so anthropic/claude-haiku-4-5 and claude-haiku-4-5 land on the same answer and a spelling difference cannot open a hole.

Here is the shipped examples/evals/journey-2-validated-judge.yaml example with one leaf changed, the judge model moved from gemini-2.5-flash to claude-haiku-4-5, which is the family the frontier tier serves.

$ tokentriage validate --config W26-same-family.yaml
Error: configuration is invalid: config D:/tt-jg-scratch/cfg/W26-same-family.yaml: evals.evaluators[1].judge.model: names a model from the SAME FAMILY as tiers[1].model. B.7 refuses a same-family judge/generator pair — a model scoring its own family's output measures self-preference as well as quality (arXiv 2508.09724), and a judge that shares a generator's failure modes cannot witness them. Every configured tier is a possible generator for this judge, so its family must differ from ALL of them; the family is the product line (haiku and opus are one family), not the size. Name a judge from a different family. Neither value is repeated back here — read the two lines this message points at
Error: configuration is invalid: config D:/tt-jg-scratch/cfg/W26-same-family.yaml: evals.evaluators[1].judge.model: names a model from the SAME FAMILY as tiers[1].model. B.7 refuses a same-family judge/generator pair — a model scoring its own family's output measures self-preference as well as quality (arXiv 2508.09724), and a judge that shares a generator's failure modes cannot witness them. Every configured tier is a possible generator for this judge, so its family must differ from ALL of them; the family is the product line (haiku and opus are one family), not the size. Name a judge from a different family. Neither value is repeated back here — read the two lines this message points at
(exit 1)

The message really does arrive twice. tokentriage validate prints every error twice on stderr and every warning twice on stdout, on this build, and editing one copy out of a capture would be inventing an output the binary does not produce. The sibling verb splits them differently: tokentriage evals validate puts its config warnings on stderr (cmd/tokentriage/evals.go:250), so a pipeline that quiets one verb does not quiet the other.

The scratch path is this run’s, and evals.evaluators[1] is that config’s own index; yours will be your own. Nothing else in the message varies, and neither model id is echoed back, which is why a config carrying a pasted secret in model: can still be debugged from the error. tokentriage run stops the same file at boot with the same sentence, so there is no window where a config validates and then judges anyway.

Every panel seat is checked the same way, and an unreadable family stops the panel. The rule extends to each member of a panel rather than to the entry’s top-level judge model alone. A seat whose family matches a tier gets the same message with the member’s index in it. A seat whose family cannot be read gets a different one, and it goes the other way from the check above.

$ tokentriage validate --config W26-panel-unknown.yaml
Error: configuration is invalid: config D:/tt-jg-scratch/cfg/W26-panel-unknown.yaml: evals.evaluators[1].judge.panel.members[2].model: no readable model family, so this member can never be shown to differ from the generator's — and §2.1 refuses a panel it cannot prove is cross-family. UNKNOWN is refused, not admitted (JH7): "we could not tell" is not "it is fine". The composition site refuses the same member, so this is the boot-time twin of a refusal that would otherwise fire per request. The value you wrote is deliberately not repeated back in this message
[…the same line printed a second time, elided; `validate` doubles every error…]
(exit 1)

The member model in this probe was 4o, which has no leading alphabetic run to read a family from. That id is left out of the message, which is the same disclosure discipline as the capture before it, and it is why both of these can be published with the configs that produced them.

Two layers, and they answer “we could not tell” in opposite directions. Config validation is fail-open on a top-level judge model whose family cannot be read, because validation may not turn away a config the engine would accept. The per-row check that runs before each grading call is fail-closed on the same input: it decides one concrete pair, it cannot see the tier table, and it stops a row whose generator the ledger cannot name. Panel members are fail-closed at both layers, which is the capture above. Rows stopped that way are counted rather than dropped, under same_family_skips on the shadow plane’s periodic summary.

How it fails. It compares model ids as they are written. An upstream that resolves one id to a different model, a gateway alias, or a tier pointed at a router of its own all read as a different family to the config check, and the file cannot see past them. The per-row check is the stronger half, because it reads the generator the ledger recorded rather than the tier table, and it is the half that fires on the traffic you served.

Panels, and why a second family

The two controls above each catch one preference. A panel goes at the problem from the other side: if several judges from unrelated model families reach the same verdict, the errors they share are the ones that survive, and there are fewer of those than any single judge carries alone.

A panel is one instrument with several seats. It votes with all of its members or it does not vote, so a seat that cannot be reached stops the whole invocation rather than being quietly dropped from the count, and a split with no strict majority is an abstention carrying panel_split rather than a coin toss. The floors are three seats and at least two distinct families, and every seat pays its own provider call under the single daily cap the evaluator declares. Composing one, reading the vote vector, and pricing the multiplier before you turn it on are Run a cross-family panel.

What these controls do not cover

A preference that is neither position nor family. The length rows at the top of this page are a third kind, and no floor in the validation walk reads verbosity. What catches it is the same thing that caught it above: human labels on your own traffic, compared against the judge, which is the walk that promotes a judge.

A judge that is consistent and consistently wrong. A flip rate of 0.0 says the two orderings agreed, and a judge can agree with itself all day about the wrong answer. The swap gate is a precondition for a pairwise promotion, not a promotion.

Abstentions, and what a judging setup does with them. A flip produces an outcome with a named reason rather than a missing value, and swap_inconsistent is one of four such reasons with four different producers. When a judge can’t tell has the set, and says which of the four an operator can reach from a config file.

Anything about a judge after it is promoted. No shipped binary can fire a drift demotion, so a swap set that was clean at promotion time stays the last thing anybody measured about that judge’s position sensitivity until you run the check again. Judge drift, with no monitor between checks is that story.