How a judge call happens

A model that grades your answers is slower than most of what it grades, and it costs money on top. So the thing to settle first, ahead of any statistics about whether the grade is any good, is where that second call sits relative to the person waiting for a response.

It sits after them. Your request goes upstream, the answer goes back to your client, and a row lands on the decision ledger. Some minutes later a background pass re-reads that row, works out whether this request is in the graded sample, and if it is, calls the grading model. By then the client has its answer and has gone.

That ordering decides most of what follows. Turning a judge on adds nothing to your p99. A grading model that is down, out of credit, or pointed at the wrong host cannot slow, block, or fail a request; the worst it does is leave a row ungraded. And when a grade does arrive, it arrives on its own row, carrying the name of the model that produced it and a digest of the prompt it was sent.

This page follows one request through that sequence: what triggers the extra call, what it costs, what it writes, and what happens on each of the three ways it goes wrong. LLM judges is the page about what a judge score is worth once you have one.

The grading runs after the answer is gone

This is the fact the rest of the page hangs off, because it is the difference between a measurement you can switch on during business hours and one you cannot.

A running proxy holds an off-path pass that walks the decision ledger on a two-minute timer. Each pass considers only the records that appeared since the previous one, so every served request is offered to each evaluator once per process lifetime. A restart or a hot reload re-offers the history once, which is bounded by your action rather than by the timer. The cadence has no config key.

The judge is composed at boot and says so, before any traffic has arrived:

{"time":"2026-08-24T10:51:28.2311024-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"pointwise-bool","judge_instrument":"gemini-2.5-flash","judge_model":"gemini-2.5-flash","api_key_env":"JUDGE_API_KEY","sample_rate":0.2,"max_usd_per_day":20}
{"time":"2026-08-24T10:51:28.2311024-07:00","level":"INFO","msg":"evals: evaluator runner wired","evaluators":["code-output-nonempty","judge-task-completion"],"backfill_interval":"2m0s","ledger":"D:\\tt-jg-scratch\\home\\.tokentriage\\decisions.jsonl","content_capture":false}

Those two lines are an excerpt from a longer boot log, captured on 2026-08-24 from a source build on Windows against the shipped examples/evals/journey-2-validated-judge.yaml example. Three tokens are that machine’s: the time values, the UTC−7 offset they carry, and the backslashed ledger path under a scratch home directory. A release build prints its own version where that build printed dev, and a Linux or macOS host prints a /-separated path. Everything else on the two lines is read from the config and the shipped defaults, so the same config gives you the same values.

Three of those values are worth reading twice. judge_method says pointwise-bool although that config names no method: leaf, because that is the absent-reading, made visible. backfill_interval is the pass described above. content_capture: false is the third condition on whether anything gets graded at all, and the next section says why.

Read the fourth one as a caution rather than a template. judge_model is gemini-2.5-flash, and the example that capture came from carries no endpoint_env, which reads as the shipped Anthropic Messages endpoint. That config therefore puts a Gemini model id in an Anthropic request body. Nothing pairs a judge model with its transport at tokentriage validate or at boot, so it validates, it boots, and the lines above are what it prints. Set endpoint_env whenever the judge model is not the shipped transport’s own.

A panic inside that pass is recovered and counted, and the loop carries on. Work that must not affect the client path is kept where it cannot reach it.

What decides whether a row gets graded

Three conditions have to hold, and the first two are the ones an operator sets.

The rate. sample_rate on the evaluator entry is required and has no default. It takes any value in [0,1], and 0 is legal: it means declared, and looking at nothing.

The draw. The decision is a hash of the request id and the evaluator id compared against the rate, not a coin toss per pass. The same request gets the same answer on every replica, on every pass, and after a restart. Two evaluators configured at 0.20 therefore look at different fifths of your traffic rather than the same fifth, because the evaluator id goes into the hash ahead of the request id.

A body to grade. A judge reads stored request and response text, so those bytes have to be in the capture store: dashboard.capture.request and dashboard.capture.response. With capture off, content_capture reads false on the boot line above, a sampled row has no stored body to grade, and it is skipped under a named reason. A row whose body was not kept is counted, not scored zero.

Four judges, and they dial separately

Four places in this build can call a grading model. They are four mechanisms with four config keys. They switch on separately, they spend separately, and only one of them can be pointed anywhere but the provider. Reading them as one feature is how a deployment ends up air-gapping one and sending traffic out of another.

The judge	The config it comes from	Where the call happens	Destination
The evals-plane judge	evals.evaluators[].judge	off the request path, on the ledger pass above	endpoint_env, and one per panel seat
The shadow-duplication judge	shadow.duplicate	off the request path, on the duplication sampler’s own goroutine	compiled in
The cascade judge	judge: true on a cascade stage	on the serving path, between the attempt and the check	compiled in
The guardrail judge	guardrails.judge	on the request path, inside the guardrail phase	compiled in

This page, and the rest of the LLM judges section, is about the first row. The other three are documented where they live: the cascade judge on escalate when the answer falls short, the guardrail judge on configuring a guard, and shadow duplication on prove a cheaper tier with an eval experiment. The cascade judge is the one to keep separate in your head: it is synchronous and on the serving path, so its wall time is inside the row’s total_ms.

What one call costs

Every judged row is one extra provider call at the shipped method. A panel multiplies that by its seats, and shadow.duplicate.pairwise presents each pair in both orders, which is two. The multiplier belongs to the method; what judging costs does the arithmetic and this page does not repeat it.

Latency. On the evals plane, none your client can observe. The grading call is made by the ledger pass, under its own timeout, after the response has been written.

Spend. It is recorded on the row, in dollars, against the same pricing snapshot as everything else. max_usd_per_day is required on any evaluator entry carrying a judge: block, and the gate measures after rather than predicting before: a call is admitted while the day’s spend is under the cap, and the call that crosses the cap is allowed to finish. Realized daily spend can therefore pass the figure in the file by at most one in-flight evaluation per worker, which is why the number in the file is not a hard ceiling. A cap of 0 is a stricter setting rather than an absent one, and it stops the evaluator at its first priced call.

Per row, not per subject. Per-row judging cost is recorded, and no producer sums it onto a subject. The spend field on the judges API is served as a present null for that reason, which the binary’s own note reads as NOT MEASURED rather than as the claim $0.

What the row says afterwards

A grade you cannot trace back to a grader is worth very little, so every judge-produced outcome carries a block naming who graded it and on what evidence. Here is one, written by the ledger pass of a running binary on 2026-08-24:

{
  "v": 1,
  "kind": "eval",
  "req_id": "tt_hpb3rn2dn5c57xz6wo5c472fzm",
  "eval": {
    "metric": "task_completion",
    "value": 1,
    "pass": true,
    "basis": "reported",
    "evaluator": "judge-task-completion",
    "eval_ver": "judge-v1+cfg15acf67",
    "source": "evaluator",
    "judge": {
      "model": "claude-haiku-4-5",
      "prompt_sha": "1fc3923",
      "validated": false,
      "method": "pointwise-bool"
    },
    "cost": { "usd": 0.000172, "estimated": false, "unpriced": false },
    "ts": "2026-08-24T18:02:46.938Z"
  }
}

That is the ledger line with whitespace added and no value changed. The capture-specific values are req_id, which the proxy minted, ts, which is when the pass judged the row, and prompt_sha with eval_ver, which identify that fixture’s judge tuple. usd is arithmetic over token counts reported by a loopback stub standing in for the judge model, priced from the litellm@2026-07-14#179affb snapshot, so it is nobody’s bill. Yours comes from your own provider and your own traffic.

Reading it from the outside in:

Field	What it says
metric, value, pass	the metric the evaluator declares, the score, and the pass verdict derived from that metric’s floor
basis	reported. A judge’s outcome stays reported until a validation run promotes its subject
evaluator, source	which registered evaluator produced this, and that an evaluator rather than a human did
eval_ver	the version the row is pinned to, which is what a promotion is scoped by
judge.model	the model that produced this grade
judge.prompt_sha	a seven-character digest of the effective prompt that was sent
judge.method	the method that graded this row, read off the composed scorer rather than the config file
judge.validated	whether a promotion stood for this subject at the moment the row was written
cost	the grading call’s own dollars, with estimated and unpriced both false

Four fields are missing from that disclosure, and their absence is a reading. kappa, validated_n, swap_flip and iaa are omitted when empty, and this writer had nothing to put in them because the judge had not been through a validation run. Absent means not measured. It does not mean zero, and iaa in particular does not mean agreement of zero.

A validated judge’s disclosure carries the ones that were measured. kappa is Cohen’s kappa (κ), the judge-versus-human agreement statistic a validation run computes, and a κ of zero is among the values it can carry. This fragment comes out of the tracked golden fixture the product pins its own wire shape against, on a shadow-grade row rather than an eval row:

    "judge": {
      "model": "claude-haiku-4-5",
      "prompt_sha": "a1b2c3d",
      "validated": true,
      "kappa": 0,
      "swap_flip": 0.04,
      "validated_n": 112,
      "abstain_feature": 0.3333333333333333
    },
    "pairwise": {
      "prompt_sha": "e5f6a7b",
      "winner_ab": "A",
      "winner_ba": "A",
      "agree": true
    },

Nothing there is machine-specific: every value is a constant in the test that marshals it, and the whole file is compared byte for byte on every run. kappa: 0 is a measurement, and it means agreement at chance level; the fixture carries that value precisely so a marshaler cannot erase it. swap_flip is the rate at which a pairwise trial changed its answer when the two candidates swapped places, where 0.0 is the best result rather than a missing one.

validated_n is the size of the set the κ was computed over. abstain_feature is a scalar about this one verdict, not a summary of the subject. iaa is absent even here, which reads as inter-annotator agreement not having been measured for that gold set.

The pairwise block beside it is a two-call trial: the same pair presented in both orders, winner_ab and winner_ba naming who won each time, and agree recording whether the two runs matched. Its prompt_sha belongs to the trial and is a different version from judge.prompt_sha.

Quote fields out of that fixture rather than the row. It sets every field so the shape is pinned, which puts an abstention reason next to an agree of true, and no run produces that pair.

The three ways it goes wrong

The judge does not answer. A transport failure, a non-200, or a 200 whose body carries no readable verdict all end the same way: the scorer returns no verdict, the evaluation is counted under evaluator_error, and no row is written. No grade is invented and nothing is scored zero. A timeout is kept apart and counted as cancelled, so a deadline set too tight stays distinguishable from a broken judge. Because each record is offered once per process lifetime, that record is not offered again in this process. A restart or a hot reload re-offers history once.

The cap is spent. The budget gate runs before the call. Once the day’s spend has reached max_usd_per_day the evaluator stops, and each stop is counted under its own reason rather than passing quietly. Two situations arrive at the same gate and are tallied apart: a cap that is spent, and a metered evaluator for which no cap was declared at all.

There is no grading call to fail. The client is built with an empty credential and turns the evaluation away before a request is assembled, so the judge seat sees no traffic. The evaluation is counted under evaluator_error, no eval row is written, and the error text goes into that counter rather than to an operator, because the loop that runs evaluators holds no logger. A second cause lands in the same counter for the same reason: a judge whose model family matches the model that produced the answer is turned away before the call, which makes a same-family pair and a missing credential indistinguishable from outside.

So the signal is an absence, and you have to know to look for it. Requests keep being served, and no judge block appears on any row. What the measurement itself costs is where the counter surfaces: it carries a line per configured evaluator, and each line lists its counted refusals beside its spend, so a judge that is sampling and writing nothing reads there as evaluator_error climbing against a spend that stays flat.

The guardrail judge is the contrast worth knowing. Given an api_key_env naming an unset variable it leaves that tier disarmed and says so at boot, as guardrails judge NOT armed: api key unset. Two judges in one binary, two answers to the same misconfiguration.

What this page does not cover

Two judge methods compose from a config file: pointwise-bool, which is the absent-reading default, and panel at self_consistency_k: 1. pointwise-rubric, pairwise and reference do not compose over a judge: transport in this build; tokentriage validate stops them by name, and so does boot. The same holds for self_consistency_k above 1 and for every sidecar_judge: block. Configure a judge has each one with the wording the binary prints.

The pairwise trial in that fixture reaches production through shadow.duplicate.pairwise, on the plane with no endpoint leaf, so it cannot be rehearsed against a listener on your own box.

What validated: true licenses, how a subject gets there, and what a κ is worth once you have one belong to the walk that promotes a judge and why your number is reported, not measured. Panels, position bias and abstentions each have their own page in this section.