What judging costs, and how to cap it

Turn a judge on and a second model call goes out behind every request it looks at. That call is on your provider bill, and the first thing anybody holding a budget asks is how much it adds.

The answer is a multiplication, and the number doing the multiplying belongs to the judging method rather than to the model you named. Grading one answer once is one call. Asking the same question in both orderings is two. Seating three graders is three.

The second thing they ask is what stops it. That answer is one line in a config file, checked before each call and settled after it. This build has two keys spelled max_usd_per_day, and leaving one out is stopped at the door while leaving the other out turns a whole feature off in silence.

How a judge call happens is where the extra call sits relative to the person waiting for a response, and what the measurement itself costs is the wider surface this page is one slice of. There is no procedure here; the panel you would configure is run a cross-family panel.

The call multiplier

The reason to start here rather than at the price of a token is that the method decides how many calls one graded row makes, and a method change can double a bill without changing a model. The table below covers every judging setup that reaches production in this build, one row per multiplier.

The shadow rows read higher than the others for a reason worth stating once: that plane grades both arms of a comparison, the answer your router chose and the answer the duplicate produced, so its multiplier starts at two before any option is added. The duplicate request itself is a further upstream call on top, and it is billed like any other.

The arithmetic is two lines, and both are checkable against your own ledger:

grading calls  =  sampled rows × calls per graded row
judging spend  =  grading calls × price per grading call

Here it is instantiated. On a ledger this documentation harness produced, one pointwise-bool grading cost $0.000172, priced against claude-haiku-4-5 from the litellm@2026-07-14#179affb snapshot. Eighty rows were graded, so eighty calls were made, and they sum to $0.013760. Those token counts came from a loopback stub standing in for the judge model, which makes the dollar arithmetic you can follow rather than anybody’s bill; your own price per call is your provider’s.

Now hold the traffic still and change only the method. The same eighty rows through the three-seat panel is 240 calls, which is $0.041280 at the same price per call. Nothing about the traffic moved. The multiplier is the entire difference, which is why it belongs in the sizing decision rather than in the surprise at the end of the month.

Where the dollars turn up

Four mechanisms in this build can call a grading model, and they do not all report to the same place. Knowing which meter carries which is what stops a reader concluding that judging is free because the surface they happened to open shows nothing.

The evals plane is the one this section is about, and the metacost surface is where its bill lands. This is that surface on a running instance whose ledger holds the eighty graded rows above:

{
  "sources": [
    {
      "name": "eval:annotation-queue",
      "spend_usd": { "usd": 0, "estimated": false, "unpriced": false },
      "period": "day"
    },
    {
      "name": "eval:judge-task-completion",
      "spend_usd": { "usd": 0.01376, "estimated": false, "unpriced": false },
      "cap_usd": 20,
      "period": "day",
      "burn_pct": 0.0688
    }
  ],
  "provenance": {
    "pricing_snapshots": [],
    "rows": 0,
    "estimated_rows": 0,
    "unpriced_rows": 0,
    "demo_rows": 0,
    "truncated": false,
    "source": "scan",
    "indexed_through": { "file": "decisions.jsonl", "offset": 214190, "ts": "2026-08-24T18:05:05.557Z" },
    "window": { "since": "2026-08-24T00:00:00.000Z", "until": "2026-08-25T00:00:00.000Z", "tz": "UTC" }
  }
}

That is the response bytes of GET /tt/api/v1/metacost, pretty-printed with no value changed, captured on 2026-08-24 from a source build serving the harness ledger described above. Two consecutive requests returned identical bytes. The window is the current UTC day and moves; the offset and ts under indexed_through belong to that ledger file. The dollars are the same loopback-stub arithmetic, so quote the shape rather than the figure.

Three readings, in the order they matter. eval:judge-task-completion carries the judging bill for the day with the cap and the burn against it on the same object, and 0.01376 against a cap of 20 is the 0.0688 percent shown. eval:annotation-queue reports $0 and is not flagged unpriced, which is a measured zero: collecting human labels makes no model call, and free is a measurement. And the provenance envelope underneath counts shadow-duplication rows rather than evaluator outcomes, so its zeroes describe a population this ledger has none of. An evaluator line’s own exclusions ride on that line, which the next section shows.

The four planes also meter differently in a way a cost report can miss. The cascade judge runs on the serving path, so its dollars sit inside a served request’s row, and the row’s top-level cost prices the accepted attempt alone.

Read the ladder from the bottom. Every rung is an upstream call the provider bills, the inline judge beside each rung is another, and only the accepted attempt is priced into the row’s headline figure. Two consequences follow for anyone summing costs: a report that adds up cost.usd across rows understates a cascade, and a governance budget, which burns that same top-level figure, does not see the cascade judge’s dollars either. Escalate when the answer falls short has the break-even arithmetic and the per-stage disclosure that carries the missing money.

The null that must not become a zero

Every figure above came from a row-level record. The per-subject figure is a different story, and it is the one that gets misread, because the field is present and its value looks like a number that got lost.

A judges reading serves spend as a present null on every row this build serves. Here is one, lifted from a subject that has been through the whole validation protocol and been promoted:

      "state": "promoted",
      "kappa": 0.6761023407729995,
      "validated_n": 105,
      "iaa": 0.8397435897435899,
      "gold_revision": "93f8396",

      … the floors_applied and floors blocks, elided …

      "spend": null,
      "selective_risk": null,
      "anchor": null,
      "last_run": "2026-08-24T18:08:58.333Z"

A fragment of one subject out of GET /tt/api/v1/evals/judges, captured from a booted instance on 2026-08-24; the elision is marked and the fragment is therefore not parseable on its own. The judge verdicts behind that κ came from a loopback stub answering a canned score, and the two annotators’ labels were designed by the harness. So κ, validated_n, iaa and gold_revision are what a promoted subject looks like rather than what any judge scores. The assembler, the κ arithmetic, the IAA, the floors and the registry are the binary’s, and the key set, state and the three nulls are not fixture values.

The field is a pointer with no omitempty, so the key is served rather than dropped, and a measured zero and an unmeasured figure cannot serialize the same way. The binary states the distinction itself, in a note it serves beside the reading and prints in the CLI’s text output. It reads: “spend is NOT MEASURED and is served null on every row, which is not the same claim as $0. … The JUDGING spend is real money and is recorded per ROW rather than summed per subject.”

Promotion does not change it. The capture above is a promoted subject with a measured κ, a measured inter-annotator agreement and a null spend, because the protocol that promotes a judge measures agreement and does not add up dollars.

So read the dollars where something adds them up. The metacost line in the previous section totals them per evaluator per period. Underneath it, each judge-written eval outcome carries its own cost object, and on the shadow plane a grade row carries judge_cost beside the judge id, the metric, the judge model and the prompt sha. All three are checkable. What no surface answers is “what did this subject cost”, and filling that in with $0.00 would publish a confident free over calls that were billed.

A call that could not be priced is not a free call

The price table does not cover every model, and a judge pointed at one it cannot price produces a dollar figure nobody can compute. The rule is the same one the rest of this product’s cost surfaces follow: the figure is withheld, the row is counted, and the total says how many it left out.

Here is the same metacost surface over a ledger whose judge outcomes are unpriced:

    {
      "name": "eval:judge-task-completion",
      "spend_usd": { "usd": null, "estimated": false, "unpriced": true },
      "cap_usd": 20,
      "period": "day",
      "disclosures": [
        "all 80 eval outcome(s) in this window are UNPRICED and excluded — the spend is UNKNOWN, not $0.00",
        "spend is UNPRICED, so no burn percentage against this cap is computable"
      ]
    }

One source line out of the same endpoint, pretty-printed. The rendering is the shipped server’s; the input is not. Every judge model within reach of this capture is one the price table covers, so the run used a copy of the harness ledger with the cost object on its eighty judge outcomes rewritten to the unpriced shape. The count 80, the cap and the two sentences are the server’s own.

Four things in that line are the pattern to recognise. usd is null rather than 0. unpriced is true, which is what separates an unknown from a measured free. burn_pct is absent, because a percentage of an unknown is not a number. And the disclosure rides on the same object as the figure it qualifies, carrying the count it excluded rather than leaving it to a footnote. A CLI rendering the same money prints the literal UNPRICED where a dollar would go. How to read a cost figure is the rule in general.

The cap, and what happens when it is spent

max_usd_per_day on an evaluator entry is required whenever that entry carries a judge: block, and the message says why: a judge spends money on every sampled request, and a metered evaluator without a declared daily cap is stopped at tokentriage validate. That is the cap doing its job before anything boots.

At runtime it measures after rather than predicting before. A call is admitted while the day’s spend is under the cap, and the call that crosses it is allowed to finish, so realized daily spend can pass the figure in the file by at most one in-flight evaluation per worker. The number in the file is a stopping point rather than a ceiling, and the product says so in the text it prints rather than leaving you to discover it.

Every stop is counted under a named reason rather than passing quietly, and the vocabulary separates two situations an operator has to tell apart: a cap that is declared and spent, and an evaluator for which no cap was declared at all. Both surface on what the measurement itself costs, which carries a line per configured evaluator with its counted refusals beside its spend.

A panel spends against that one cap and re-reads it before each seat. A panel stopped at its third seat has already made two calls, and those dollars leave your account with no row to show for them, because a panel votes with all its seats or it does not vote. Whether they also reach the counter the cap is checked against is the next section, and the answer is not the one the arithmetic so far would lead you to.

A panel seat that reports no usage stops the cap counting

A panel folds its seats into one figure for the invocation, and the fold follows the house rule for an unknown: if any part could not be priced, the whole is unpriced, because summing the rest would claim the unpriceable part was free. An unpriced settlement is then counted and excluded rather than added, so the day’s accumulator does not move.

On a single-model judge that is the right answer, because there is no priced part to keep. On a panel there is one, and it goes with the rest. A seat that reports no token usage, which includes any seat whose call failed, makes the whole invocation unpriced and takes its siblings’ priced dollars down with it.

The size of that hole is measured in the product’s own suite, and pinned by a test so the figure is a contract rather than a paragraph:

$ go test ./internal/evaluator/ -run TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed -v
=== RUN   TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed
    desc_panel_test.go:767: MEASURED AMPLIFICATION: 50 rows x 2 PRICED calls at $0.0001 = $0.0100 genuinely spent; budget.SpentToday = $0.0000; declared cap $0.05 never moved. UNPRICED contagion + Settle's early return on an unpriced fold. A seat that fails on every row is an unbounded spender under a declared cap — escalated, NOT fixed here (a house-wide convention change).
--- PASS: TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed (0.00s)
PASS
ok  	github.com/tokentriage/platform/internal/evaluator	0.021s

Fifty rows, two priced seats each at $0.0001, one seat failing every row. $0.0100 of priced spend was measured, SpentToday read $0.0000, and the declared $0.05 cap did not move. It errs towards admitting, which is the direction that costs you money: a panel with one consistently broken seat spends without a bound under a cap that looks untouched. A failing seat is also hard to see from outside, because a dead seat, a panicking seat, a mid-fan-out budget stop and a malformed vote vector all reach an operator as one skip named evaluator_error.

The settlement is counted even where the dollars are not, and that count is the thread to pull. The metacost line for that evaluator grows a disclosure reading N settlement(s) UNPRICED: … this cap is not advancing, ranked ahead of everything else on the line so a reader who stops after one disclosure stops on that one. What the count cannot tell you is how much of the spend behind it was priced before the fold zeroed it.

Until the fold changes, read that disclosure as the working cap for a panel, and repair the failing seat rather than lowering the figure in the file. It is escalated in the product’s source rather than fixed, and the test above is what holds the escalation to a number.

The cap that turns the feature off instead

The second key spelled max_usd_per_day sits under shadow.duplicate, and it acts in the opposite direction from the first. Absent or 0, it turns the whole of shadow duplication off: no refusal, no warning, no log line, and no rows.

Two configs establish it, and they differ by one line:

   duplicate:
     enabled: true
     sample_rate: 0.05
-    max_usd_per_day: 5.0
     pairwise: true

With the cap, the validator prints the plane’s egress warning and its own call arithmetic:

$ tokentriage validate --config with-cap.yaml
warning: shadow.duplicate: sampled shadow duplication PHONES OUT — the duplicate runs against the configured upstream and the grader makes 4 Anthropic Messages call(s) per sampled request under this config (2 pointwise grades × self_consistency_k, plus 2 for the pairwise co-run when shadow.duplicate.pairwise is on) with the model named by classifier.oracle.model (falling back to the built-in cheap judge model) and the key named by classifier.oracle.api_key_env. The single-binary air-gap posture is broken for this path. BOTH the duplicate's and the grader's measured dollars are capped POST-HOC by shadow.duplicate.max_usd_per_day (measured, not predicted), and a judge call whose price cannot be resolved is EXCLUDED from that cap and counted — never folded in as $0. Unlike evals.evaluators[].judge this leaf carries no external: allow gate — it shipped before the gate existed and adding requiredness here would refuse configs that are valid today
OK: configuration is valid

Take the max_usd_per_day line out and leave enabled: true, sample_rate: 0.05 and pairwise: true where they are:

$ tokentriage validate --config no-cap.yaml
OK: configuration is valid

Boot the second file and the log carries no shadow line of any kind. The sampler is not composed at all, so the ledger collects no duplicate rows and no grades, for as long as the config stands. Both probes were scratch copies of the shipped examples/evals/journey-2-validated-judge.yaml example with the shadow.duplicate block above appended, run on 2026-08-24 against a source build on Windows. The first block shows one copy of a warning this command prints twice, and the boot runs had their listeners on port 0 and replica.name pinned so no hostname reached the capture.

Two keys spell the same words and take opposite actions on the same missing value. An operator who carries the evaluator key’s behaviour across to this one collects nothing for weeks and then finds the routing claim it was funding has no evidence under it.

What this page does not cover

Sizing shadow duplication against what it buys. The spend-versus-saving ratio, the healthy spike-and-decline shape and the two figures called metacost that must not be subtracted are what the measurement itself costs, which is the surface this page reads from.

Writing the block the cap sits in. The cap is one leaf on a block that has several, and the steps that get the whole of it past validate and past boot are configure a judge. Every optional leaf beside it, with the default the code applies, is Evals & calibration configuration.

Latency. On the evals plane there is none your client can observe, because the grading call is made after the response has been written. The cascade judge is the exception, and its wall time is inside the served row’s total_ms.

Rehearsing a judge without paying for it. The evals-plane judge takes endpoint_env and can be pointed at a loopback listener. The shadow-duplication judge has no endpoint leaf at all: it dials a compiled-in provider URL, so shadow.duplicate.pairwise needs outbound traffic and real credentials. Air-gap judging, and the known limits is that story.