Air-gap judging, and the known limits

An LLM judge grades your traffic by sending it to a model, so turning one on adds a second place your prompts and your responses go. On a deployment with no route off the host, that is the whole question: does any of this work, and if it does, what did you agree to when you switched it on?

Both halves have an answer in this build, and the two are not the same. One of the two judging planes can be driven end to end with the network unplugged. The other dials a URL compiled into the binary, and no config key moves it. The difference between them is a single leaf, and it decides whether an offline rehearsal is possible at all.

The second half of this page is the catalogue. Fifteen things in the judge track are documented here as gaps rather than features, grouped by what stands in the way of each, plus three that a design document called gaps and that turn out to work. Every entry was checked against the running binary rather than against a plan.

If you have not read LLM judges, the vocabulary below assumes it. The product’s wider offline story, the one about installing and verifying with no route out, is Install air-gapped. There is no procedure here.

The air gap is one literal token, and a person types it

You cannot switch on outbound judging traffic by accident, and the mechanism that stops you is worth knowing before anything else on this page, because it is the same mechanism everywhere else in the product.

An evaluator carrying a judge: block is stopped at tokentriage validate unless that block carries external: allow. What it prints instead says why, and names its own siblings:

$ tokentriage validate --config T7-a-air-gap.yaml
Error: configuration is invalid: config D:/tt-jg-scratch/cfg/T7-a-air-gap.yaml: evals.evaluators[1].judge.external: an LLM judge PHONES OUT to its provider on every sampled request and requires external: allow — the air-gap gate (SP-16), the same literal opt-in cache.store.l3, cache.embedders[] and shared_state.{redis,postgres} carry. Without it this evaluator is REFUSED: an air-gapped deployment must not be able to enable a phone-out judge at sample_rate: 1.0, and a reviewer grepping `external:` must be able to enumerate every egress path in the file. The value you wrote is deliberately not repeated back in this message
(exit 1)

That capture is from a source build on Windows, run against a copy of the shipped examples/evals/journey-2-validated-judge.yaml example with the external: allow leaf taken off and nothing else changed. Two tokens in it are that run’s: the config path, which is the scratch directory the capture was produced under, and the index evaluators[1], which is that file’s judge entry.

Nothing else varies between runs, because there is no timestamp, no port and no version string in the message. Exit status was 1. And the binary prints every warning and every error on this surface twice, which is its behaviour rather than this capture’s, so your terminal shows those bytes on two consecutive lines.

Once it is typed, the binary says so at boot rather than leaving you to infer it, and the warning names the leaf that opened the gate and the variable holding the credential:

warning: evals.evaluators[1].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[1].judge.model, using the key named by evals.evaluators[1].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them

A judge pointed at a host of your own rather than at its provider gets a different sentence, naming the endpoint variable instead of the provider. On tokentriage run that bare line goes to stderr and a structured log record carries the same string, so it appears twice for that reason and not for the doubling above.

One plane rehearses with no egress, and one cannot

Four mechanisms in this build can call a grading model, and How a judge call happens has the whole table. For the offline question, only one distinction matters: which of them takes a destination from the config file.

The evals-plane judge takes evals.evaluators[].judge.endpoint_env, an environment-variable name holding the base URL. Point it at a listener on 127.0.0.1 and the shipped judge write path runs with no packet leaving the host. That is how this section’s captures were produced, including the promoted validation run on the walk that promotes a judge: the assembler, Cohen’s κ, the inter-annotator figure, all six floors, the promotion registry and the artifact store are the binary’s, and the grading answers came from a loopback seat.

The shadow-duplication judge takes no destination. The function that builds its config returns a model and a key-variable name and nothing else, the constructor falls back to a compiled-in provider URL when it is handed no endpoint, and the shadow.duplicate config type carries six keys of which none is an endpoint. Five places in this build construct a judge client, counting a panel’s seats separately from the evaluator that holds them, and the asymmetry across the five is the useful part:

Where the judge is built	Takes a destination?	The config leaf
evals.evaluators[].judge	yes	endpoint_env
evals.evaluators[].judge.panel.members[]	yes, one per seat	panel.members[].endpoint_env
shadow.duplicate	no	none
the cascade judge	no	none
the guardrail judge	no	none

Two consequences follow, and the second is the one that catches people out. An air-gapped deployment cannot exercise shadow judging at all, against a loopback listener or against a mock. And shadow.duplicate.pairwise, the path pairwise judging reaches production through, needs outbound traffic and a provider bill. It cannot be rehearsed on your own box, so the missing end-to-end pairwise demonstration in this section is a consequence of the plane it would run on rather than of nobody getting round to it.

The gaps, grouped by what stands in the way

If you are deciding whether to adopt this, read this section before the feature pages. A positive list turns “can I do this?” into a search; a negative list turns it into an answer.

The grouping is by blocker rather than by feature, because the blocker is what tells you whether your case is one function or a subsystem. Fifteen entries in seven groups follow, and each group names the lever that would move it. Nothing here is softened into “may vary”, and nothing is written larger than it is either: where an entry is narrower than its headline suggests, the entry says so, and the three things a design document called gaps and that work get a section at the end.

No composition site, and the binary says so twice

Four config leaves parse, validate their own shape, and reach no code that can build what they name. Each is stopped at tokentriage validate and again at boot, by name, and the composition site does not quietly grade something else instead.

The leaf	The binary’s own words for it
evals.evaluators[].sidecar_judge:	config-complete and composition-incomplete
evals.evaluators[].sidecar:	a SIDECAR scorer, and cannot be built from config alone
judge.method: at pointwise-rubric, pairwise or reference	a registered method this build cannot compose over a judge: (provider API) transport
judge.self_consistency_k above 1	not composable by this build

The lever is a scorer. Three of the four name the constructor a composition site would call, and the two distinct constructors they name occur once each in tracked non-test Go, where the one occurrence is the definition. There is no self-hosted judge path in this build, so a deployment that cannot send text to a provider has no judging path here at all. Two judging setups do compose from a file: pointwise-bool, which is what an absent method: reads as, and panel at self_consistency_k: 1. Configure a judge prints all four refusals in full.

No arrival point, on two mechanisms that are otherwise built

Two pieces of machinery are implemented, tested and pinned to vectors, and nothing in a shipped binary hands either of them an input.

Gate.ObserveAgreement has no call site outside its tests, so no shipped binary can fire a drift demotion. And no composition site constructs the anchor monitor, so the who-drifted diagnosis is served as a present null on every judges row: the null records that no diagnosis was attached, not that the judge was checked and found stable.

The lever is a caller on each, plus a place for a human label to arrive on the first. On the first it is not the whole of it, either. The accumulator a monitor would advance is opened at promotion only for a single-annotator gold set (internal/evaluator/validate.go:1578), so a judge promoted on the cohort the agreement gate asks for carries no stream for a caller to advance. Judge drift, with no monitor between checks is the long version, including the three preconditions the observation type cannot carry and what each costs.

No producer behind a field the wire already carries

Three fields are on the served shape and nothing fills them. The shapes differ, and the difference is one the product built two Go types to preserve, so a page flattening all three into “null” has lost it.

Field	On the wire	What the absence records
spend	present, carrying null	not measured. Judging costs money per row and no shipped producer sums those dollars onto a subject
next_trigger	absent, key and all	nothing assigns it, so the loop’s next run is unknown rather than ruled out
demotions	omitted from every history response	a demotion notice goes to an event sink when it happens and is not retained

The lever is a producer, and on demotions a store to keep the notice in. None of the three is hidden elsewhere: read per-row judge_cost on a shadow-grade row for judging spend, auto_validate and last_run for the loop, and timeline for a demoted entry with its κ and N.

No key on the config type, so a file cannot reach the knob

Two knobs have no leaf. config.EvalEvaluator carries six yaml keys and config: is not among them, so rule-json-schema, whose factory requires rule.family, is refused from every config file this build reads. And the shadow-plane judge’s destination has no leaf, which is the second half of this page’s first section.

The lever is a yaml key on each. One of the two has a way round: drive rule-json-schema from tokentriage evals run --arm or from the experiments API, both of which carry the config fragment it needs. The shadow plane has none, which is why it gets a section of its own above.

The check runs after the config file, on two configurations that validate green

Two setups clear tokentriage validate and then stop later, which makes them harder to diagnose than a leaf that is refused outright.

A misspelled quality_metric is admitted by validate and then admits no candidate at runtime: every candidate’s quality fact is absent, the candidates are excluded and disclosed on the score table, and traffic lands on the selector’s on_empty target. And judge-groundedness, one of the two registered judge ids, publishes no pass floor, so the runner leaves every row it grades without a verdict and the gold-set assembler records only rows that carry one. Its subject is still counted, so a validation run stops on a label count that will not move however many labels arrive.

The lever is a check at validate time on each. Meanwhile, check the metric id against the evaluator’s declared metrics by hand, and configure judge-task-completion for any judge that has to reach a promotion. Why didn’t my judge validate? starts from the symptom.

No identifier to join a label back to the claim that offered it

The annotation queue ships one endpoint, POST /tt/api/v1/evals/annotation-queue/claim; labels come back through POST /tt/api/v1/outcomes, correlated by req_id. No lease identifier travels on either wire, so a submitted label cannot be tied to the claim that offered it.

The lever is a field on both wires. The scope is narrower than it sounds: the row is correlated, because both sides name the same req_id. What a deployment cannot do is measure claim-to-submission completion, tell a re-claimed expiry from a first offer, or attribute a label to the strategy that surfaced it.

Absent because the arithmetic would overstate itself

κ is served beside its validated_n and with no confidence interval. validated_n counts label events, which are (annotator, request) pairs rather than independent draws, and two annotators on one request share a single judge verdict. An interval computed from a clustered count as though it were N independent observations would claim a precision it does not have.

There is no lever wanted here, which is what separates this entry from the six above. It is a position the product holds, and moving it needs a named method with its own test vectors rather than a producer.

Three things on that list that are not gaps

The design document behind this section listed three more. Checking each against the running binary withdrew all three, and they are recorded here rather than dropped, because a reader who meets one of them somewhere else is owed the correction.

starved_segments works on a standard deployment. The segment-label counter is composed by the standard binary wherever it opens the promotion registry, and dashboard.state_dir is defaulted at decode (internal/config/defaults.go:259-260), so a standard deployment has the counter without configuring anything.

The refusal that used to be unconditional now fires only where no counter is wired: an embedded dashsrv composed without Deps.SegmentLabelCounts, or a B.7 registry that failed to open. It names hard_cases as the strategy to claim with meanwhile. The dashboard’s own client contract still asserts the old, unconditional behaviour; the Go server is the authority and the discrepancy is filed.

paired_verdict on an experiment is a client-side type, not a field with no producer. The experiments response carries no such field, and none is in the published schema. The carrier a server fills is paired_verdict on the quality source served by GET /tt/api/v1/evidence.

The pairwise path composes and runs. runPairwise has a non-test call site and is fed from the both-orderings trials the shadow judge recorded on the ledger. What has been exercised is a planted ledger under test, and no capture drives a live pairwise judge end to end. It ships and we have shown it working end to end are two statements, and this build keeps them apart. The second is missing because the plane those trials come from needs egress, which is this page’s first section again.

What this page does not cover

Every refused configuration in full, with the file that produces it. This page groups them; Configure a judge is the surface, and it doubles as the catalogue of configurations this build turns away.

The drift entries at length. Two of the fifteen are the drift monitor and the who-drifted diagnosis, and both have more to say than a group entry allows. Judge drift, with no monitor between checks carries them, together with the one demotion arm an operator can still drive.

The offline install itself. Chart dependencies, the image, signature verification behind the gap, and every other tier gated on the same consent token are Install air-gapped.