# What judging costs, and how to cap it What a judged request adds to the bill, how the call multiplier scales it, and the daily cap that stops the spending. Source: https://tokentriage.com/docs/judges/cost-and-budget --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; import CascadeLadderDiagram from '../../../components/CascadeLadderDiagram.astro'; Turn a judge on and a second model call goes out behind every request it looks at. That call is on your provider bill, and the first thing anybody holding a budget asks is how much it adds. The answer is a multiplication, and the number doing the multiplying belongs to the judging method rather than to the model you named. Grading one answer once is one call. Asking the same question in both orderings is two. Seating three graders is three. The second thing they ask is what stops it. That answer is one line in a config file, checked before each call and settled after it. This build has two keys spelled `max_usd_per_day`, and leaving one out is stopped at the door while leaving the other out turns a whole feature off in silence. [How a judge call happens](/docs/judges/how-judging-works) is where the extra call sits relative to the person waiting for a response, and [what the measurement itself costs](/docs/build/cost-metacost) is the wider surface this page is one slice of. There is no procedure here; the panel you would configure is [run a cross-family panel](/docs/judges/run-a-panel). ## The call multiplier The reason to start here rather than at the price of a token is that the method decides how many calls one graded row makes, and a method change can double a bill without changing a model. The table below covers every judging setup that reaches production in this build, one row per multiplier. | The setup | Grading calls per graded row | Where it composes | |---|---|---| | `pointwise-bool` | 1 | `evals.evaluators[].judge`, and it is what an absent `method:` reads as | | `panel` with N seats | N, at three seats or more | `evals.evaluators[].judge.method: panel` | | shadow duplication, pointwise | 2 × `self_consistency_k` | `shadow.duplicate` | | shadow duplication, pairwise co-run | 2 more on top | `shadow.duplicate.pairwise` | The shadow rows read higher than the others for a reason worth stating once: that plane grades both arms of a comparison, the answer your router chose and the answer the duplicate produced, so its multiplier starts at two before any option is added. The duplicate request itself is a further upstream call on top, and it is billed like any other. The arithmetic is two lines, and both are checkable against your own ledger: ```text grading calls = sampled rows × calls per graded row judging spend = grading calls × price per grading call ``` Here it is instantiated. On a ledger this documentation harness produced, one `pointwise-bool` grading cost `$0.000172`, priced against `claude-haiku-4-5` from the `litellm@2026-07-14#179affb` snapshot. Eighty rows were graded, so eighty calls were made, and they sum to `$0.013760`. Those token counts came from a loopback stub standing in for the judge model, which makes the dollar arithmetic you can follow rather than anybody's bill; your own price per call is your provider's. Now hold the traffic still and change only the method. The same eighty rows through the three-seat panel is 240 calls, which is `$0.041280` at the same price per call. Nothing about the traffic moved. The multiplier is the entire difference, which is why it belongs in the sizing decision rather than in the surprise at the end of the month. Price the multiplier before the sample rate, because the sample rate is the knob you will reach for afterwards. Halving `sample_rate` halves the bill and halves the evidence at the same time; dropping a seat changes what the instrument is. Those are different trades and only one of them is reversible without re-validating the judge. ## Where the dollars turn up Four mechanisms in this build can call a grading model, and they do not all report to the same place. Knowing which meter carries which is what stops a reader concluding that judging is free because the surface they happened to open shows nothing. | The judge | Where its dollars are reported | Its cap | |---|---|---| | the evals-plane judge | one `eval:` line on the metacost surface | `evals.evaluators[].max_usd_per_day`, required whenever a `judge:` block is present | | the shadow-duplication judge | the `shadow` line, beside the duplicate's own dollars | `shadow.duplicate.max_usd_per_day` | | the cascade judge | inside the served request's row, broken out as `judge_cost` | none of its own | | the guardrail judge | on the guardrail disclosure for that request | `guardrails.judge.max_usd_per_request`, which is required and [declared rather than enforced](/docs/guardrails/configuring-a-guard) | The evals plane is the one this section is about, and the metacost surface is where its bill lands. This is that surface on a running instance whose ledger holds the eighty graded rows above: ```json { "sources": [ { "name": "eval:annotation-queue", "spend_usd": { "usd": 0, "estimated": false, "unpriced": false }, "period": "day" }, { "name": "eval:judge-task-completion", "spend_usd": { "usd": 0.01376, "estimated": false, "unpriced": false }, "cap_usd": 20, "period": "day", "burn_pct": 0.0688 } ], "provenance": { "pricing_snapshots": [], "rows": 0, "estimated_rows": 0, "unpriced_rows": 0, "demo_rows": 0, "truncated": false, "source": "scan", "indexed_through": { "file": "decisions.jsonl", "offset": 214190, "ts": "2026-08-24T18:05:05.557Z" }, "window": { "since": "2026-08-24T00:00:00.000Z", "until": "2026-08-25T00:00:00.000Z", "tz": "UTC" } } } ``` That is the response bytes of `GET /tt/api/v1/metacost`, pretty-printed with no value changed, captured on 2026-08-24 from a source build serving the harness ledger described above. Two consecutive requests returned identical bytes. The `window` is the current UTC day and moves; the `offset` and `ts` under `indexed_through` belong to that ledger file. The dollars are the same loopback-stub arithmetic, so quote the shape rather than the figure. Three readings, in the order they matter. `eval:judge-task-completion` carries the judging bill for the day with the cap and the burn against it on the same object, and `0.01376` against a cap of `20` is the `0.0688` percent shown. `eval:annotation-queue` reports `$0` and is not flagged unpriced, which is a measured zero: collecting human labels makes no model call, and free is a measurement. And the `provenance` envelope underneath counts shadow-duplication rows rather than evaluator outcomes, so its zeroes describe a population this ledger has none of. An evaluator line's own exclusions ride on that line, which the next section shows. The four planes also meter differently in a way a cost report can miss. The cascade judge runs on the serving path, so its dollars sit inside a served request's row, and the row's top-level `cost` prices the accepted attempt alone. Read the ladder from the bottom. Every rung is an upstream call the provider bills, the inline judge beside each rung is another, and only the accepted attempt is priced into the row's headline figure. Two consequences follow for anyone summing costs: a report that adds up `cost.usd` across rows understates a cascade, and a governance budget, which burns that same top-level figure, does not see the cascade judge's dollars either. [Escalate when the answer falls short](/docs/build/routing-cascades) has the break-even arithmetic and the per-stage disclosure that carries the missing money. ## The null that must not become a zero Every figure above came from a row-level record. The per-subject figure is a different story, and it is the one that gets misread, because the field is present and its value looks like a number that got lost. A judges reading serves `spend` as a present `null` on every row this build serves. Here is one, lifted from a subject that has been through the whole validation protocol and been promoted: ```json "state": "promoted", "kappa": 0.6761023407729995, "validated_n": 105, "iaa": 0.8397435897435899, "gold_revision": "93f8396", … the floors_applied and floors blocks, elided … "spend": null, "selective_risk": null, "anchor": null, "last_run": "2026-08-24T18:08:58.333Z" ``` A fragment of one subject out of `GET /tt/api/v1/evals/judges`, captured from a booted instance on 2026-08-24; the elision is marked and the fragment is therefore not parseable on its own. **The judge verdicts behind that κ came from a loopback stub answering a canned score, and the two annotators' labels were designed by the harness.** So κ, `validated_n`, `iaa` and `gold_revision` are what a promoted subject looks like rather than what any judge scores. The assembler, the κ arithmetic, the IAA, the floors and the registry are the binary's, and the key set, `state` and the three nulls are not fixture values. `spend: null` means NOT MEASURED, and that is a different claim from `$0`. Judging costs money and is recorded per row; what is missing is the addition. No shipped producer sums a subject's judging dollars, and the one function that would (`AnchorMonitor.SpendFor`) has no call site outside its tests. The field is a pointer with no `omitempty`, so the key is served rather than dropped, and a measured zero and an unmeasured figure cannot serialize the same way. The binary states the distinction itself, in a note it serves beside the reading and prints in the CLI's text output. It reads: *"spend is NOT MEASURED and is served null on every row, which is not the same claim as $0. … The JUDGING spend is real money and is recorded per ROW rather than summed per subject."* Promotion does not change it. The capture above is a promoted subject with a measured κ, a measured inter-annotator agreement and a `null` spend, because the protocol that promotes a judge measures agreement and does not add up dollars. **So read the dollars where something adds them up.** The metacost line in the previous section totals them per evaluator per period. Underneath it, each judge-written eval outcome carries its own `cost` object, and on the shadow plane a grade row carries `judge_cost` beside the judge id, the metric, the judge model and the prompt sha. All three are checkable. What no surface answers is "what did *this subject* cost", and filling that in with `$0.00` would publish a confident free over calls that were billed. ## A call that could not be priced is not a free call The price table does not cover every model, and a judge pointed at one it cannot price produces a dollar figure nobody can compute. The rule is the same one the rest of this product's cost surfaces follow: the figure is withheld, the row is counted, and the total says how many it left out. Here is the same metacost surface over a ledger whose judge outcomes are unpriced: ```json { "name": "eval:judge-task-completion", "spend_usd": { "usd": null, "estimated": false, "unpriced": true }, "cap_usd": 20, "period": "day", "disclosures": [ "all 80 eval outcome(s) in this window are UNPRICED and excluded — the spend is UNKNOWN, not $0.00", "spend is UNPRICED, so no burn percentage against this cap is computable" ] } ``` One source line out of the same endpoint, pretty-printed. The rendering is the shipped server's; the input is not. Every judge model within reach of this capture is one the price table covers, so the run used a copy of the harness ledger with the `cost` object on its eighty judge outcomes rewritten to the unpriced shape. The count `80`, the cap and the two sentences are the server's own. Four things in that line are the pattern to recognise. `usd` is `null` rather than `0`. `unpriced` is `true`, which is what separates an unknown from a measured free. `burn_pct` is absent, because a percentage of an unknown is not a number. And the disclosure rides on the same object as the figure it qualifies, carrying the count it excluded rather than leaving it to a footnote. A CLI rendering the same money prints the literal `UNPRICED` where a dollar would go. [How to read a cost figure](/docs/build/cost-figures) is the rule in general. ## The cap, and what happens when it is spent `max_usd_per_day` on an evaluator entry is required whenever that entry carries a `judge:` block, and the message says why: a judge spends money on every sampled request, and a metered evaluator without a declared daily cap is stopped at `tokentriage validate`. That is the cap doing its job before anything boots. At runtime it measures after rather than predicting before. A call is admitted while the day's spend is under the cap, and the call that crosses it is allowed to finish, so realized daily spend can pass the figure in the file by at most one in-flight evaluation per worker. The number in the file is a stopping point rather than a ceiling, and the product says so in the text it prints rather than leaving you to discover it. | The value you write | What it does | |---|---| | a positive figure | admits calls while the UTC day's settled spend is below it | | `0` | a stricter setting, not an absent one: no call is admitted and settled spend stays at zero | | absent, beside a `judge:` block | stopped at `tokentriage validate`, by name | | `.inf` | stopped, at the config layer and again at the gate | Every stop is counted under a named reason rather than passing quietly, and the vocabulary separates two situations an operator has to tell apart: a cap that is declared and spent, and an evaluator for which no cap was declared at all. Both surface on [what the measurement itself costs](/docs/build/cost-metacost), which carries a line per configured evaluator with its counted refusals beside its spend. A panel spends against that one cap and re-reads it before each seat. A panel stopped at its third seat has already made two calls, and those dollars leave your account with no row to show for them, because a panel votes with all its seats or it does not vote. Whether they also reach the counter the cap is checked against is the next section, and the answer is not the one the arithmetic so far would lead you to. ### A panel seat that reports no usage stops the cap counting A panel folds its seats into one figure for the invocation, and the fold follows the house rule for an unknown: if any part could not be priced, the whole is unpriced, because summing the rest would claim the unpriceable part was free. An unpriced settlement is then counted and excluded rather than added, so the day's accumulator does not move. On a single-model judge that is the right answer, because there is no priced part to keep. On a panel there is one, and it goes with the rest. A seat that reports no token usage, which includes any seat whose call failed, makes the whole invocation unpriced and takes its siblings' priced dollars down with it. The size of that hole is measured in the product's own suite, and pinned by a test so the figure is a contract rather than a paragraph:
```console $ go test ./internal/evaluator/ -run TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed -v === RUN TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed desc_panel_test.go:767: MEASURED AMPLIFICATION: 50 rows x 2 PRICED calls at $0.0001 = $0.0100 genuinely spent; budget.SpentToday = $0.0000; declared cap $0.05 never moved. UNPRICED contagion + Settle's early return on an unpriced fold. A seat that fails on every row is an unbounded spender under a declared cap — escalated, NOT fixed here (a house-wide convention change). --- PASS: TestPanelPricedSpendBeforeAFailingSeatIsDisclosedNotSummed (0.00s) PASS ok github.com/tokentriage/platform/internal/evaluator 0.021s ``` Run from a checkout of the product on 2026-08-24. Nothing in it is machine-specific; the three dollar figures and the row count are the test's own constants, and it is the failure mode's own alarm: the day the fold is repaired, this test goes red and somebody has to come back and rewrite it.
Fifty rows, two priced seats each at `$0.0001`, one seat failing every row. `$0.0100` of priced spend was measured, `SpentToday` read `$0.0000`, and the declared `$0.05` cap did not move. **It errs towards admitting**, which is the direction that costs you money: a panel with one consistently broken seat spends without a bound under a cap that looks untouched. A failing seat is also hard to see from outside, because a dead seat, a panicking seat, a mid-fan-out budget stop and a malformed vote vector all reach an operator as one skip named `evaluator_error`. The settlement is counted even where the dollars are not, and that count is the thread to pull. The metacost line for that evaluator grows a disclosure reading `N settlement(s) UNPRICED: … this cap is not advancing`, ranked ahead of everything else on the line so a reader who stops after one disclosure stops on that one. What the count cannot tell you is how much of the spend behind it was priced before the fold zeroed it. Until the fold changes, read that disclosure as the working cap for a panel, and repair the failing seat rather than lowering the figure in the file. It is escalated in the product's source rather than fixed, and the test above is what holds the escalation to a number. ## The cap that turns the feature off instead The second key spelled `max_usd_per_day` sits under `shadow.duplicate`, and it acts in the opposite direction from the first. Absent or `0`, it turns the whole of shadow duplication off: no refusal, no warning, no log line, and no rows. Two configs establish it, and they differ by one line: ```diff duplicate: enabled: true sample_rate: 0.05 - max_usd_per_day: 5.0 pairwise: true ``` With the cap, the validator prints the plane's egress warning and its own call arithmetic: ```console $ tokentriage validate --config with-cap.yaml warning: shadow.duplicate: sampled shadow duplication PHONES OUT — the duplicate runs against the configured upstream and the grader makes 4 Anthropic Messages call(s) per sampled request under this config (2 pointwise grades × self_consistency_k, plus 2 for the pairwise co-run when shadow.duplicate.pairwise is on) with the model named by classifier.oracle.model (falling back to the built-in cheap judge model) and the key named by classifier.oracle.api_key_env. The single-binary air-gap posture is broken for this path. BOTH the duplicate's and the grader's measured dollars are capped POST-HOC by shadow.duplicate.max_usd_per_day (measured, not predicted), and a judge call whose price cannot be resolved is EXCLUDED from that cap and counted — never folded in as $0. Unlike evals.evaluators[].judge this leaf carries no external: allow gate — it shipped before the gate existed and adding requiredness here would refuse configs that are valid today OK: configuration is valid ``` Take the `max_usd_per_day` line out and leave `enabled: true`, `sample_rate: 0.05` and `pairwise: true` where they are: ```console $ tokentriage validate --config no-cap.yaml OK: configuration is valid ``` Boot the second file and the log carries no `shadow` line of any kind. The sampler is not composed at all, so the ledger collects no duplicate rows and no grades, for as long as the config stands. Both probes were scratch copies of the shipped `examples/evals/journey-2-validated-judge.yaml` example with the `shadow.duplicate` block above appended, run on 2026-08-24 against a source build on Windows. The first block shows one copy of a warning this command prints twice, and the boot runs had their listeners on port `0` and `replica.name` pinned so no hostname reached the capture. An absent or zero `shadow.duplicate.max_usd_per_day` disables shadow duplication silently: the whole shadow path is gated on `enabled`, a positive `sample_rate` and a positive `max_usd_per_day` together in one condition (`cmd/tokentriage/run.go`, the `cfg.Shadow.Duplicate` guard), so a zero in any of the three composes nothing. The symptom is an empty result set nobody warned you about, weeks after you turned sampling on. Set a positive figure, and confirm it by grepping the boot log for `shadow duplication enabled` before you wait for rows. This one is filed, and the contrast is worth carrying: the evaluator key stops a config that omits it, and this one stops the feature. Two keys spell the same words and take opposite actions on the same missing value. An operator who carries the evaluator key's behaviour across to this one collects nothing for weeks and then finds the routing claim it was funding has no evidence under it. ## What this page does not cover **Sizing shadow duplication against what it buys.** The spend-versus-saving ratio, the healthy spike-and-decline shape and the two figures called metacost that must not be subtracted are [what the measurement itself costs](/docs/build/cost-metacost), which is the surface this page reads from. **Writing the block the cap sits in.** The cap is one leaf on a block that has several, and the steps that get the whole of it past validate and past boot are [configure a judge](/docs/judges/configure-a-judge). Every optional leaf beside it, with the default the code applies, is [Evals & calibration configuration](/docs/reference/evals-config). **Latency.** On the evals plane there is none your client can observe, because the grading call is made after the response has been written. The cascade judge is the exception, and its wall time is inside the served row's `total_ms`. **Rehearsing a judge without paying for it.** The evals-plane judge takes `endpoint_env` and can be pointed at a loopback listener. The shadow-duplication judge has no endpoint leaf at all: it dials a compiled-in provider URL, so `shadow.duplicate.pairwise` needs outbound traffic and real credentials. [Air-gap judging, and the known limits](/docs/judges/air-gap-and-limits) is that story.