# Air-gap judging, and the known limits
What judging does with the network unplugged, and every part of the judge track this section documents as a gap rather than a feature.
Source: https://tokentriage.com/docs/judges/air-gap-and-limits
---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
An LLM judge grades your traffic by sending it to a model, so turning one on adds a second place
your prompts and your responses go. On a deployment with no route off the host, that is the whole
question: does any of this work, and if it does, what did you agree to when you switched it on?
Both halves have an answer in this build, and the two are not the same. One of the two judging
planes can be driven end to end with the network unplugged. The other dials a URL compiled into the
binary, and no config key moves it. The difference between them is a single leaf, and it decides
whether an offline rehearsal is possible at all.
The second half of this page is the catalogue. Fifteen things in the judge track are documented here
as gaps rather than features, grouped by what stands in the way of each, plus three that a design
document called gaps and that turn out to work. Every entry was checked against the running binary
rather than against a plan.
If you have not read [LLM judges](/docs/judges/overview), the vocabulary below assumes it. The
product's wider offline story, the one about installing and verifying with no route out, is
[Install air-gapped](/docs/operate/air-gap). There is no procedure here.
## The air gap is one literal token, and a person types it
You cannot switch on outbound judging traffic by accident, and the mechanism that stops you is worth
knowing before anything else on this page, because it is the same mechanism everywhere else in the
product.
An evaluator carrying a `judge:` block is stopped at `tokentriage validate` unless that block
carries `external: allow`. What it prints instead says why, and names its own siblings:
```console
$ tokentriage validate --config T7-a-air-gap.yaml
Error: configuration is invalid: config D:/tt-jg-scratch/cfg/T7-a-air-gap.yaml: evals.evaluators[1].judge.external: an LLM judge PHONES OUT to its provider on every sampled request and requires external: allow — the air-gap gate (SP-16), the same literal opt-in cache.store.l3, cache.embedders[] and shared_state.{redis,postgres} carry. Without it this evaluator is REFUSED: an air-gapped deployment must not be able to enable a phone-out judge at sample_rate: 1.0, and a reviewer grepping `external:` must be able to enumerate every egress path in the file. The value you wrote is deliberately not repeated back in this message
(exit 1)
```
That capture is from a source build on Windows, run against a copy of the shipped
`examples/evals/journey-2-validated-judge.yaml` example with the `external: allow` leaf taken off
and nothing else changed. Two tokens in it are that run's: the config path, which is the scratch directory the
capture was produced under, and the index `evaluators[1]`, which is that file's judge entry.
Nothing else varies between runs, because there is no timestamp, no port and no version string in
the message. Exit status was 1. And the binary prints every warning and every error on this surface
**twice**, which is its behaviour rather than this capture's, so your terminal shows those bytes on
two consecutive lines.
`external: allow` is a claim about the capability, not about the destination, and it is required
even when the destination is your own machine. The gate's own words for that are that a loopback
address is still egress. So an offline rehearsal still types the token, and a reviewer grepping
`external:` still enumerates every path out of the file in one pass.
Once it is typed, the binary says so at boot rather than leaving you to infer it, and the warning
names the leaf that opened the gate and the variable holding the credential:
```console
warning: evals.evaluators[1].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[1].judge.model, using the key named by evals.evaluators[1].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
```
A judge pointed at a host of your own rather than at its provider gets a different sentence, naming
the endpoint variable instead of the provider. On `tokentriage run` that bare line goes to stderr
and a structured log record carries the same string, so it appears twice for that reason and not for
the doubling above.
## One plane rehearses with no egress, and one cannot
Four mechanisms in this build can call a grading model, and [How a judge call
happens](/docs/judges/how-judging-works) has the whole table. For the offline question, only one
distinction matters: which of them takes a destination from the config file.
The evals-plane judge takes `evals.evaluators[].judge.endpoint_env`, an environment-variable name
holding the base URL. Point it at a listener on `127.0.0.1` and the shipped judge write path runs
with no packet leaving the host. That is how this section's captures were produced, including the
promoted validation run on [the walk that promotes a judge](/docs/judges/agreement): the assembler,
Cohen's κ, the inter-annotator figure, all six floors, the promotion registry and the artifact store
are the binary's, and the grading answers came from a loopback seat.
The shadow-duplication judge takes no destination. The function that builds its config returns a
model and a key-variable name and nothing else, the constructor falls back to a compiled-in provider
URL when it is handed no endpoint, and the `shadow.duplicate` config type carries six keys of which
none is an endpoint. Five places in this build construct a judge client, counting a panel's seats
separately from the evaluator that holds them, and the asymmetry across the five is the useful part:
| Where the judge is built | Takes a destination? | The config leaf |
|---|---|---|
| `evals.evaluators[].judge` | **yes** | `endpoint_env` |
| `evals.evaluators[].judge.panel.members[]` | **yes**, one per seat | `panel.members[].endpoint_env` |
| `shadow.duplicate` | no | none |
| the cascade judge | no | none |
| the guardrail judge | no | none |
Two consequences follow, and the second is the one that catches people out. An air-gapped deployment
cannot exercise shadow judging at all, against a loopback listener or against a mock. And
`shadow.duplicate.pairwise`, the path pairwise judging reaches production through, needs outbound
traffic and a provider bill. It cannot be rehearsed on your own box, so the missing end-to-end
pairwise demonstration in this section is a consequence of the plane it would run on rather than of
nobody getting round to it.
## The gaps, grouped by what stands in the way
If you are deciding whether to adopt this, read this section before the feature pages. A positive
list turns "can I do this?" into a search; a negative list turns it into an answer.
The grouping is by blocker rather than by feature, because the blocker is what tells you whether
your case is one function or a subsystem. Fifteen entries in seven groups follow, and each group
names the lever that would move it. Nothing here is softened into "may vary", and nothing is written
larger than it is either: where an entry is narrower than its headline suggests, the entry says so,
and the three things a design document called gaps and that work get a section at the end.
### No composition site, and the binary says so twice
Four config leaves parse, validate their own shape, and reach no code that can build what they
name. Each is stopped at `tokentriage validate` and again at boot, by name, and the composition site
does not quietly grade something else instead.
| The leaf | The binary's own words for it |
|---|---|
| `evals.evaluators[].sidecar_judge:` | config-complete and composition-incomplete |
| `evals.evaluators[].sidecar:` | a SIDECAR scorer, and cannot be built from config alone |
| `judge.method:` at `pointwise-rubric`, `pairwise` or `reference` | a registered method this build cannot compose over a `judge:` (provider API) transport |
| `judge.self_consistency_k` above 1 | not composable by this build |
**The lever is a scorer.** Three of the four name the constructor a composition site would call, and
the two distinct constructors they name occur once each in tracked non-test Go, where the one
occurrence is the definition. There is no self-hosted judge path in this build, so a deployment that
cannot send text to a provider has no judging path here at all. Two judging setups do compose from a
file: `pointwise-bool`, which is what an absent `method:` reads as, and `panel` at
`self_consistency_k: 1`. [Configure a judge](/docs/judges/configure-a-judge) prints all four
refusals in full.
### No arrival point, on two mechanisms that are otherwise built
Two pieces of machinery are implemented, tested and pinned to vectors, and nothing in a shipped
binary hands either of them an input.
`Gate.ObserveAgreement` has no call site outside its tests, so no shipped binary can fire a drift
demotion. And no composition site constructs the anchor monitor, so the who-drifted diagnosis is
served as a present `null` on every judges row: the null records that no diagnosis was attached,
not that the judge was checked and found stable.
**The lever is a caller** on each, plus a place for a human label to arrive on the first. On the
first it is not the whole of it, either. The accumulator a monitor would advance is opened at
promotion only for a single-annotator gold set (`internal/evaluator/validate.go:1578`), so a judge
promoted on the cohort the agreement gate asks for carries no stream for a caller to advance.
[Judge drift, with no monitor between checks](/docs/judges/drift) is the long version, including
the three preconditions the observation type cannot carry and what each costs.
### No producer behind a field the wire already carries
Three fields are on the served shape and nothing fills them. The shapes differ, and the difference
is one the product built two Go types to preserve, so a page flattening all three into "null" has
lost it.
| Field | On the wire | What the absence records |
|---|---|---|
| `spend` | present, carrying `null` | not measured. Judging costs money per row and no shipped producer sums those dollars onto a subject |
| `next_trigger` | absent, key and all | nothing assigns it, so the loop's next run is unknown rather than ruled out |
| `demotions` | omitted from every history response | a demotion notice goes to an event sink when it happens and is not retained |
**The lever is a producer**, and on `demotions` a store to keep the notice in. None of the three is
hidden elsewhere: read per-row `judge_cost` on a shadow-grade row for judging spend, `auto_validate`
and `last_run` for the loop, and `timeline` for a `demoted` entry with its κ and N.
### No key on the config type, so a file cannot reach the knob
Two knobs have no leaf. `config.EvalEvaluator` carries six yaml keys and `config:` is not among
them, so `rule-json-schema`, whose factory requires `rule.family`, is refused from every config file
this build reads. And the shadow-plane judge's destination has no leaf, which is the second half of
this page's first section.
**The lever is a yaml key** on each. One of the two has a way round: drive `rule-json-schema` from
`tokentriage evals run --arm` or from the experiments API, both of which carry the config fragment
it needs. The shadow plane has none, which is why it gets a section of its own above.
### The check runs after the config file, on two configurations that validate green
Two setups clear `tokentriage validate` and then stop later, which makes them harder to diagnose
than a leaf that is refused outright.
A misspelled `quality_metric` is admitted by `validate` and then admits no candidate at runtime:
every candidate's quality fact is absent, the candidates are excluded and disclosed on the score
table, and traffic lands on the selector's `on_empty` target. And `judge-groundedness`, one of the
two registered judge ids, publishes no pass floor, so the runner leaves every row it grades without
a verdict and the gold-set assembler records only rows that carry one. Its subject is still counted,
so a validation run stops on a label count that will not move however many labels arrive.
**The lever is a check at validate time** on each. Meanwhile, check the metric id against the
evaluator's declared metrics by hand, and configure `judge-task-completion` for any judge that has
to reach a promotion. [Why didn't my judge validate?](/docs/judges/troubleshooting) starts from the
symptom.
### No identifier to join a label back to the claim that offered it
The annotation queue ships one endpoint, `POST /tt/api/v1/evals/annotation-queue/claim`; labels come
back through `POST /tt/api/v1/outcomes`, correlated by `req_id`. No lease identifier travels on
either wire, so a submitted label cannot be tied to the claim that offered it.
**The lever is a field on both wires.** The scope is narrower than it sounds: the *row* is
correlated, because both sides name the same `req_id`. What a deployment cannot do is measure
claim-to-submission completion, tell a re-claimed expiry from a first offer, or attribute a label to
the strategy that surfaced it.
### Absent because the arithmetic would overstate itself
κ is served beside its `validated_n` and with no confidence interval. `validated_n` counts label
events, which are (annotator, request) pairs rather than independent draws, and two annotators on
one request share a single judge verdict. An interval computed from a clustered count as though it
were N independent observations would claim a precision it does not have.
**There is no lever wanted here**, which is what separates this entry from the six above. It is a
position the product holds, and moving it needs a named method with its own test vectors rather than
a producer.
**This list is not a roadmap.** It is a measurement of one build, taken by running that build, and
it carries no dates, no ordering and no commitment that any entry moves. Read it to decide whether
this product fits what you are doing today. Where an entry has a way round, the way round is stated
beside it, and where there is none the entry says so.
## Three things on that list that are not gaps
The design document behind this section listed three more. Checking each against the running binary
withdrew all three, and they are recorded here rather than dropped, because a reader who meets one
of them somewhere else is owed the correction.
**`starved_segments` works on a standard deployment.** The segment-label counter is composed by the
standard binary wherever it opens the promotion registry, and `dashboard.state_dir` is defaulted at
decode (`internal/config/defaults.go:259-260`), so a standard deployment has the counter without
configuring anything.
The refusal that used to be unconditional now fires only where no counter is wired: an embedded
`dashsrv` composed without `Deps.SegmentLabelCounts`, or a B.7 registry that failed to open. It
names `hard_cases` as the strategy to claim with meanwhile. The dashboard's own client contract
still asserts the old, unconditional behaviour; the Go server is the authority and the discrepancy
is filed.
**`paired_verdict` on an experiment is a client-side type, not a field with no producer.** The
experiments response carries no such field, and none is in the published schema. The carrier a
server fills is `paired_verdict` on the quality source served by `GET /tt/api/v1/evidence`.
**The pairwise path composes and runs.** `runPairwise` has a non-test call site and is fed from the
both-orderings trials the shadow judge recorded on the ledger. What has been exercised is a planted
ledger under test, and no capture drives a live pairwise judge end to end. *It ships* and *we have
shown it working end to end* are two statements, and this build keeps them apart. The second is
missing because the plane those trials come from needs egress, which is this page's first section
again.
## What this page does not cover
**Every refused configuration in full, with the file that produces it.** This page groups them;
[Configure a judge](/docs/judges/configure-a-judge) is the surface, and it doubles as the catalogue
of configurations this build turns away.
**The drift entries at length.** Two of the fifteen are the drift monitor and the who-drifted
diagnosis, and both have more to say than a group entry allows.
[Judge drift, with no monitor between checks](/docs/judges/drift) carries them, together with the
one demotion arm an operator can still drive.
**The offline install itself.** Chart dependencies, the image, signature verification behind the
gap, and every other tier gated on the same consent token are
[Install air-gapped](/docs/operate/air-gap).