# How a judge call happens Follow one judged request off the request path, through the extra call it makes, to the disclosure it leaves on the ledger row. Source: https://tokentriage.com/docs/judges/how-judging-works --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; A model that grades your answers is slower than most of what it grades, and it costs money on top. So the thing to settle first, ahead of any statistics about whether the grade is any good, is where that second call sits relative to the person waiting for a response. It sits after them. Your request goes upstream, the answer goes back to your client, and a row lands on the decision ledger. Some minutes later a background pass re-reads that row, works out whether this request is in the graded sample, and if it is, calls the grading model. By then the client has its answer and has gone. That ordering decides most of what follows. Turning a judge on adds nothing to your p99. A grading model that is down, out of credit, or pointed at the wrong host cannot slow, block, or fail a request; the worst it does is leave a row ungraded. And when a grade does arrive, it arrives on its own row, carrying the name of the model that produced it and a digest of the prompt it was sent. This page follows one request through that sequence: what triggers the extra call, what it costs, what it writes, and what happens on each of the three ways it goes wrong. [LLM judges](/docs/judges/overview) is the page about what a judge score is worth once you have one. ## The grading runs after the answer is gone This is the fact the rest of the page hangs off, because it is the difference between a measurement you can switch on during business hours and one you cannot. A running proxy holds an off-path pass that walks the decision ledger on a two-minute timer. Each pass considers only the records that appeared since the previous one, so every served request is offered to each evaluator once per process lifetime. A restart or a hot reload re-offers the history once, which is bounded by your action rather than by the timer. The cadence has no config key. The judge is composed at boot and says so, before any traffic has arrived: ```console {"time":"2026-08-24T10:51:28.2311024-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"pointwise-bool","judge_instrument":"gemini-2.5-flash","judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20} {"time":"2026-08-24T10:51:28.2311024-07:00","level":"INFO","msg":"evals: evaluator runner wired","evaluators":["code-output-nonempty","judge-task-completion"],"backfill_interval":"2m0s","ledger":"D:\\tt-jg-scratch\\home\\.tokentriage\\decisions.jsonl","content_capture":false} ``` Those two lines are an excerpt from a longer boot log, captured on 2026-08-24 from a source build on Windows against the shipped `examples/evals/journey-2-validated-judge.yaml` example. Three tokens are that machine's: the `time` values, the UTC−7 offset they carry, and the backslashed ledger path under a scratch home directory. A release build prints its own version where that build printed `dev`, and a Linux or macOS host prints a `/`-separated path. Everything else on the two lines is read from the config and the shipped defaults, so the same config gives you the same values. Three of those values are worth reading twice. `judge_method` says `pointwise-bool` although that config names no `method:` leaf, because that is the absent-reading, made visible. `backfill_interval` is the pass described above. `content_capture: false` is the third condition on whether anything gets graded at all, and the next section says why. Read the fourth one as a caution rather than a template. `judge_model` is `gemini-2.5-flash`, and the example that capture came from carries no `endpoint_env`, which reads as the shipped Anthropic Messages endpoint. That config therefore puts a Gemini model id in an Anthropic request body. Nothing pairs a judge model with its transport at `tokentriage validate` or at boot, so it validates, it boots, and the lines above are what it prints. Set `endpoint_env` whenever the judge model is not the shipped transport's own. A judge wired this way cannot add latency to a request, and it cannot fail one. The grading call is made after the response has been written, by a loop the client never touches. The same boundary is the one [quality-aware routing](/docs/calibration/quality-aware-routing) already draws for its licensing arithmetic, and judging follows it rather than inventing a second rule. A panic inside that pass is recovered and counted, and the loop carries on. Work that must not affect the client path is kept where it cannot reach it. ## What decides whether a row gets graded Three conditions have to hold, and the first two are the ones an operator sets. **The rate.** `sample_rate` on the evaluator entry is required and has no default. It takes any value in `[0,1]`, and `0` is legal: it means declared, and looking at nothing. **The draw.** The decision is a hash of the request id and the evaluator id compared against the rate, not a coin toss per pass. The same request gets the same answer on every replica, on every pass, and after a restart. Two evaluators configured at `0.20` therefore look at different fifths of your traffic rather than the same fifth, because the evaluator id goes into the hash ahead of the request id. **A body to grade.** A judge reads stored request and response text, so those bytes have to be in the capture store: `dashboard.capture.request` and `dashboard.capture.response`. With capture off, `content_capture` reads `false` on the boot line above, a sampled row has no stored body to grade, and it is skipped under a named reason. A row whose body was not kept is counted, not scored zero. ## Four judges, and they dial separately Four places in this build can call a grading model. They are four mechanisms with four config keys. They switch on separately, they spend separately, and only one of them can be pointed anywhere but the provider. Reading them as one feature is how a deployment ends up air-gapping one and sending traffic out of another. | The judge | The config it comes from | Where the call happens | Destination | |---|---|---|---| | The evals-plane judge | `evals.evaluators[].judge` | off the request path, on the ledger pass above | `endpoint_env`, and one per panel seat | | The shadow-duplication judge | `shadow.duplicate` | off the request path, on the duplication sampler's own goroutine | compiled in | | The cascade judge | `judge: true` on a cascade stage | on the serving path, between the attempt and the check | compiled in | | The guardrail judge | `guardrails.judge` | on the request path, inside the guardrail phase | compiled in | This page, and the rest of the LLM judges section, is about the first row. The other three are documented where they live: the cascade judge on [escalate when the answer falls short](/docs/build/routing-cascades), the guardrail judge on [configuring a guard](/docs/guardrails/configuring-a-guard), and shadow duplication on [prove a cheaper tier with an eval experiment](/docs/calibration/prove-a-cheaper-tier). The cascade judge is the one to keep separate in your head: it is synchronous and on the serving path, so its wall time is inside the row's `total_ms`. **Only the evals-plane judge takes a destination.** The shadow-duplicate judge dials a compiled-in provider URL and no `shadow.duplicate.*` key moves it, so that plane needs egress. The evals-plane judge takes `evals.evaluators[].judge.endpoint_env` and can be pointed at a loopback listener, which is the path to use for a rehearsal with no outbound traffic. The cascade judge and the guardrail judge build their clients the same way the shadow one does. ## What one call costs Every judged row is one extra provider call at the shipped method. A panel multiplies that by its seats, and `shadow.duplicate.pairwise` presents each pair in both orders, which is two. The multiplier belongs to the method; [what judging costs](/docs/judges/cost-and-budget) does the arithmetic and this page does not repeat it. **Latency.** On the evals plane, none your client can observe. The grading call is made by the ledger pass, under its own timeout, after the response has been written. **Spend.** It is recorded on the row, in dollars, against the same pricing snapshot as everything else. `max_usd_per_day` is required on any evaluator entry carrying a `judge:` block, and the gate measures after rather than predicting before: a call is admitted while the day's spend is under the cap, and the call that crosses the cap is allowed to finish. Realized daily spend can therefore pass the figure in the file by at most one in-flight evaluation per worker, which is why the number in the file is not a hard ceiling. A cap of `0` is a stricter setting rather than an absent one, and it stops the evaluator at its first priced call. **Per row, not per subject.** Per-row judging cost is recorded, and no producer sums it onto a subject. The `spend` field on the judges API is served as a present `null` for that reason, which the binary's own note reads as NOT MEASURED rather than as the claim `$0`. ## What the row says afterwards A grade you cannot trace back to a grader is worth very little, so every judge-produced outcome carries a block naming who graded it and on what evidence. Here is one, written by the ledger pass of a running binary on 2026-08-24: ```json { "v": 1, "kind": "eval", "req_id": "tt_hpb3rn2dn5c57xz6wo5c472fzm", "eval": { "metric": "task_completion", "value": 1, "pass": true, "basis": "reported", "evaluator": "judge-task-completion", "eval_ver": "judge-v1+cfg15acf67", "source": "evaluator", "judge": { "model": "claude-haiku-4-5", "prompt_sha": "1fc3923", "validated": false, "method": "pointwise-bool" }, "cost": { "usd": 0.000172, "estimated": false, "unpriced": false }, "ts": "2026-08-24T18:02:46.938Z" } } ``` That is the ledger line with whitespace added and no value changed. The capture-specific values are `req_id`, which the proxy minted, `ts`, which is when the pass judged the row, and `prompt_sha` with `eval_ver`, which identify that fixture's judge tuple. `usd` is arithmetic over token counts reported by a loopback stub standing in for the judge model, priced from the `litellm@2026-07-14#179affb` snapshot, so it is nobody's bill. Yours comes from your own provider and your own traffic. Reading it from the outside in: | Field | What it says | |---|---| | `metric`, `value`, `pass` | the metric the evaluator declares, the score, and the pass verdict derived from that metric's floor | | `basis` | `reported`. A judge's outcome stays `reported` until a validation run promotes its subject | | `evaluator`, `source` | which registered evaluator produced this, and that an evaluator rather than a human did | | `eval_ver` | the version the row is pinned to, which is what a promotion is scoped by | | `judge.model` | the model that produced this grade | | `judge.prompt_sha` | a seven-character digest of the effective prompt that was sent | | `judge.method` | the method that graded this row, read off the composed scorer rather than the config file | | `judge.validated` | whether a promotion stood for this subject at the moment the row was written | | `cost` | the grading call's own dollars, with `estimated` and `unpriced` both false | **Four fields are missing from that disclosure, and their absence is a reading.** `kappa`, `validated_n`, `swap_flip` and `iaa` are omitted when empty, and this writer had nothing to put in them because the judge had not been through a validation run. Absent means not measured. It does not mean zero, and `iaa` in particular does not mean agreement of zero. A validated judge's disclosure carries the ones that were measured. `kappa` is Cohen's kappa (κ), the judge-versus-human agreement statistic a validation run computes, and a κ of zero is among the values it can carry. This fragment comes out of the tracked golden fixture the product pins its own wire shape against, on a shadow-grade row rather than an eval row: ```json "judge": { "model": "claude-haiku-4-5", "prompt_sha": "a1b2c3d", "validated": true, "kappa": 0, "swap_flip": 0.04, "validated_n": 112, "abstain_feature": 0.3333333333333333 }, "pairwise": { "prompt_sha": "e5f6a7b", "winner_ab": "A", "winner_ba": "A", "agree": true }, ``` Nothing there is machine-specific: every value is a constant in the test that marshals it, and the whole file is compared byte for byte on every run. `kappa: 0` is a measurement, and it means agreement at chance level; the fixture carries that value precisely so a marshaler cannot erase it. `swap_flip` is the rate at which a pairwise trial changed its answer when the two candidates swapped places, where `0.0` is the best result rather than a missing one. `validated_n` is the size of the set the κ was computed over. `abstain_feature` is a scalar about this one verdict, not a summary of the subject. `iaa` is absent even here, which reads as inter-annotator agreement not having been measured for that gold set. The `pairwise` block beside it is a two-call trial: the same pair presented in both orders, `winner_ab` and `winner_ba` naming who won each time, and `agree` recording whether the two runs matched. Its `prompt_sha` belongs to the trial and is a different version from `judge.prompt_sha`. Quote fields out of that fixture rather than the row. It sets every field so the shape is pinned, which puts an abstention reason next to an `agree` of `true`, and no run produces that pair. ## The three ways it goes wrong **The judge does not answer.** A transport failure, a non-200, or a 200 whose body carries no readable verdict all end the same way: the scorer returns no verdict, the evaluation is counted under `evaluator_error`, and no row is written. No grade is invented and nothing is scored zero. A timeout is kept apart and counted as `cancelled`, so a deadline set too tight stays distinguishable from a broken judge. Because each record is offered once per process lifetime, that record is not offered again in this process. A restart or a hot reload re-offers history once. **The cap is spent.** The budget gate runs before the call. Once the day's spend has reached `max_usd_per_day` the evaluator stops, and each stop is counted under its own reason rather than passing quietly. Two situations arrive at the same gate and are tallied apart: a cap that is spent, and a metered evaluator for which no cap was declared at all. **The key is unset, and the failure is silent.** `evals.evaluators[].judge.api_key_env` names the environment variable holding the judge's credential. Config requires the leaf, turns away an empty one, and turns away a literal secret written in its place. No check reads whether the variable it names holds anything: not at `tokentriage validate`, not at boot, and not in `tokentriage doctor`, whose `keys` check walks upstream credential references and stops there. Boot prints `evals: judge evaluator wired` either way, while the two unset *upstream* key variables each get a warning on that same boot. After that, nothing is printed at any level. There is no grading call to fail. The client is built with an empty credential and turns the evaluation away before a request is assembled, so the judge seat sees no traffic. The evaluation is counted under `evaluator_error`, no eval row is written, and the error text goes into that counter rather than to an operator, because the loop that runs evaluators holds no logger. A second cause lands in the same counter for the same reason: a judge whose model family matches the model that produced the answer is turned away before the call, which makes a same-family pair and a missing credential indistinguishable from outside. **So the signal is an absence, and you have to know to look for it.** Requests keep being served, and no `judge` block appears on any row. [What the measurement itself costs](/docs/build/cost-metacost) is where the counter surfaces: it carries a line per configured evaluator, and each line lists its counted refusals beside its spend, so a judge that is sampling and writing nothing reads there as `evaluator_error` climbing against a spend that stays flat. The guardrail judge is the contrast worth knowing. Given an `api_key_env` naming an unset variable it leaves that tier disarmed and says so at boot, as `guardrails judge NOT armed: api key unset`. Two judges in one binary, two answers to the same misconfiguration. ## What this page does not cover Two judge methods compose from a config file: `pointwise-bool`, which is the absent-reading default, and `panel` at `self_consistency_k: 1`. `pointwise-rubric`, `pairwise` and `reference` do not compose over a `judge:` transport in this build; `tokentriage validate` stops them by name, and so does boot. The same holds for `self_consistency_k` above 1 and for every `sidecar_judge:` block. [Configure a judge](/docs/judges/configure-a-judge) has each one with the wording the binary prints. The pairwise trial in that fixture reaches production through `shadow.duplicate.pairwise`, on the plane with no endpoint leaf, so it cannot be rehearsed against a listener on your own box. What `validated: true` licenses, how a subject gets there, and what a κ is worth once you have one belong to [the walk that promotes a judge](/docs/judges/agreement) and [why your number is reported, not measured](/docs/judges/basis-ceilings). Panels, position bias and abstentions each have their own page in this section.