# Grade your first request with a judge Send one request, grade it with a model, read the graded row back. It needs two keys from different model families and spends money. Source: https://tokentriage.com/docs/judges/quickstart --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; A score a model hands you is worth what you can find out about it afterwards. This page ends with that record on your screen: one line saying which model graded one of your requests, on which prompt, what the grading call cost, and how much anybody has checked the grader. In this tutorial we'll set two keys, write a short config, check it before anything spends, start the proxy, send one request through it, and read the graded row back off the decision ledger. That ledger is the JSON-lines file this tool writes one row per request into. The [Quickstart](/docs/get-started/quickstart) needed no credentials, because nothing in it talked to a provider. This page is the other kind. Two calls leave your machine and both are billed to you. ## What you need - **The `tokentriage` binary on your PATH.** The [Quickstart](/docs/get-started/quickstart) puts one there in about a minute. - **An OpenAI key, in `OPENAI_API_KEY`.** It pays for the one request you send. - **An Anthropic key, in `JUDGE_API_KEY`.** It pays for the one grading call that follows. The judge transport in this build posts to the Anthropic Messages API, so this key is an Anthropic one. - **A fraction of a cent, twice.** One request and one grading call. `max_usd_per_day` in the config below stops the judge at a dollar a day whatever else happens. What it does **not** need: - **No account with us**, no signup and no licence key. - **No human labels and no annotators.** Grading a request and measuring the grader are two different jobs, and this page does the first. - **No database, no container and no second machine.** - **No routing.** `mode: ledger` prices traffic and leaves your responses alone. **Two vendors rather than one, and that is a rule rather than a preference.** A judge is turned away from grading a model of its own family, because a model scoring its own family's output measures self-preference as well as quality. So the model that answers and the model that grades cannot both be `claude`, and they cannot both be `gpt`. **How long it takes.** The wall clock here is one wait you cannot shorten. Grading runs off the request path, on a pass that fires every two minutes, and a request is graded on the next pass after it lands. So the graded row arrives anywhere from a few seconds to two minutes later; on the runs behind this page it ranged from 8 seconds to 1 minute 56 seconds. Everything else is four commands. Getting the binary and getting the two keys sit outside that budget. ## 1. Put the two keys in your environment TokenTriage reads credentials from environment variables by name and holds none of its own. ```console $ export OPENAI_API_KEY=sk-... $ export JUDGE_API_KEY=sk-ant-... ``` In PowerShell those two lines are `$env:OPENAI_API_KEY = "sk-..."` and `$env:JUDGE_API_KEY = "sk-ant-..."`. Step 3 tells you whether the first one took. Nothing tells you whether the second one did, and the callout under step 3 says what to do about that. ## 2. Write the config Save this as `judge-quickstart.yaml`, beside wherever you are working. ```yaml version: 1 mode: ledger upstreams: openai: provider: openai base_url: https://api.openai.com # host only — your client already sends /v1 api_key_env: OPENAI_API_KEY ledger: decision_log: path: ./decisions.jsonl dashboard: enabled: true capture: request: true response: true evals: evaluators: - id: judge-task-completion sample_rate: 1.0 max_usd_per_day: 1.00 judge: external: allow model: claude-haiku-4-5 api_key_env: JUDGE_API_KEY ``` Four things in that file decide whether anything gets graded. - **`id: judge-task-completion`** names one of the two judge evaluators this build registers, and it is the one to pick. `judge-groundedness` carries no pass floor, so its rows reach validation with no verdict on them; `judge-task-completion`'s metric carries a floor of 1.0. - **`sample_rate: 1.0`** grades every request. The leaf is required and has no default. - **`max_usd_per_day: 1.00`** is required wherever a `judge:` block is present, and it is the stop on this tutorial's spending. - **`dashboard.capture`** keeps the request and response bytes. A judge reads stored text, so with capture off a sampled row has nothing on it to grade. `external: allow` is the air-gap gate: the literal opt-in a config carries wherever a leaf reaches the network, and without it the evaluator is refused rather than composed. There is no `method:` leaf here, and step 4 shows what an absent one reads as. ## 3. Check it before anything spends `doctor` is read-only. It boots nothing, writes nothing and prints no secret, so it is safe to run against a live setup as well as this one. ```console $ tokentriage doctor --config judge-quickstart.yaml ✓ binary — tokentriage dev (none), embedded pricing snapshot litellm@2026-07-14#179affb — clean ✓ config — judge-quickstart.yaml — strict decode clean: mode ledger, 1 upstream(s), 0 tier(s), 0 rule(s); 1 warning(s) warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them ✓ compile — provider registry + routing policy compiled — 1 upstream(s) wired; 1 warning(s) — see below warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them ✓ ports — listen 127.0.0.1:8787 bindable, admin 127.0.0.1:9090 bindable — clean ✓ ledger — ./decisions.jsonl — not yet created; parent dir exists, created on first request — clean ○ index — D:\tt-w21\home\.tokentriage\index\decisions.sqlite — not built yet; the dashboard read-model builds on first run — clean run: tokentriage index rebuild (optional; also auto-built when the dashboard starts) ✓ pricing — embedded table litellm@2026-07-14#179affb loaded, no overrides configured — clean ✓ keys — 1 upstream(s), 1 *_env reference(s) all set — clean ✓ dashboard — enabled — auth off, state D:\tt-w21\home\.tokentriage\dash, index D:\tt-w21\home\.tokentriage\index\decisions.sqlite, demo false — would compose (config valid; stores open at boot) ○ instance — not probed — pass --probe to check a running instance's admin plane over HTTP run: tokentriage doctor --probe (probes the config's admin address) run: tokentriage doctor --probe --probe-url http://127.0.0.1:9090 ✓ features — 31 capabilities — 21 shipped, 10 announced-not-shipped; counts agree with this binary (21 detectors / 7 categories, 92 providers, 8 recommendation types); since unreleased ok — all checks passed (exit 0) ``` Read the glyphs: `✓` passed, `!` warned, `○` skipped and said why, `✗` failed. Two lines carry `○` here and neither is a fault of yours. `instance` is skipped because nothing is running yet. `index` is skipped because nothing has built the dashboard read model on this machine, which is where a first run starts. Once something has, that line reads `✓ index — … open, no rows indexed yet` and the rebuild hint under it goes away. That capture came off a source build on Windows under a neutral home directory, so a release prints its version where this one prints `dev`, and the three backslashed store paths will be your own home directory, with `/` separators on Linux and macOS. The `✓ ports` line is what you get with nothing started yet. Everything else on those lines is read from the config and the compiled defaults. Exit status was 0. The judge warning prints twice, under `config` and again under `compile`. Reading it once is enough. It is the binary saying that this evaluator sends your captured request and response text to a provider on every sampled row, and that its verdicts stay `reported` until somebody measures the judge. **The `keys` line does not cover the judge key.** It counts upstream credential references, so it reads `all set — clean` whether or not `JUDGE_API_KEY` holds anything, and boot says nothing about it either. With that variable empty the grading call is not made, no row is written, and the log is silent about all of it. Check its length yourself before you go on: `printf '%s' "$JUDGE_API_KEY" | wc -c` in a POSIX shell, `$env:JUDGE_API_KEY.Length` in PowerShell. Anything but a number above zero and step 6 finds nothing.
The judge block has two leaves that a config is refused without, and both refusals name the leaf. These are the config lines out of two `doctor` runs, one per broken file: ```console ✗ config — probe-noexternal.yaml — strict decode failed: config probe-noexternal.yaml: evals.evaluators[0].judge.external: an LLM judge PHONES OUT to its provider on every sampled request and requires external: allow — the air-gap gate (SP-16), the same literal opt-in cache.store.l3, cache.embedders[] and shared_state.{redis,postgres} carry. Without it this evaluator is REFUSED: an air-gapped deployment must not be able to enable a phone-out judge at sample_rate: 1.0, and a reviewer grepping `external:` must be able to enumerate every egress path in the file. The value you wrote is deliberately not repeated back in this message ``` ```console ✗ config — probe-nocap.yaml — strict decode failed: config probe-nocap.yaml: evals.evaluators[0].max_usd_per_day: required whenever evals.evaluators[0].judge is present — an LLM judge SPENDS MONEY on every sampled request, and DQ13 refuses a metered evaluator without a declared daily cap. Set an explicit dollar figure (0 is legal and means "stop at the first priced call") ``` Each is one line lifted out of a longer report; both runs exited 1. The file names are the two probes this page was checked with, and yours will read `judge-quickstart.yaml`. `[0]` is the index of the judge entry in the file, so a config with evaluators above it prints a higher number. Put the named leaf back and run `doctor` again.
## 4. Start it ```console $ tokentriage run --config judge-quickstart.yaml {"time":"2026-08-24T12:23:35.9786582-07:00","level":"INFO","msg":"evals: B.7 promotion registry opened","dir":"D:\\tt-w21\\home\\.tokentriage\\dash\\quality","validation_state":"empty","detail":"","standing_promotions":0,"auto_validate":false,"interval":"24h0m0s","new_labels_threshold":25} {"time":"2026-08-24T12:23:35.9908707-07:00","level":"INFO","msg":"dashboard composed","index":"D:\\tt-w21\\home\\.tokentriage\\index\\decisions.sqlite","log":"./decisions.jsonl","auth":"off","demo":false} {"time":"2026-08-24T12:23:35.9908707-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"pointwise-bool","judge_instrument":"claude-haiku-4-5","judge_model":"claude-haiku-4-5","api_key_env":"JUDGE_API_KEY","sample_rate":1,"max_usd_per_day":1} {"time":"2026-08-24T12:23:35.9913874-07:00","level":"INFO","msg":"evals: evaluator runner wired","evaluators":["judge-task-completion"],"backfill_interval":"2m0s","ledger":"./decisions.jsonl","content_capture":true} {"time":"2026-08-24T12:23:35.9913874-07:00","level":"INFO","msg":"evals: the B.7 auto-validation loop is OFF (evals.validation.auto_validate.enabled is false); promotions restored from the artifact store still stand and still license `measured`, and `tokentriage evals validate` remains available as an operator-forced run"} {"time":"2026-08-24T12:23:35.9913874-07:00","level":"INFO","msg":"tokentriage starting","listen":"127.0.0.1:8787","admin":"127.0.0.1:9090","mode":"ledger","upstreams":1,"tiers":0} tokentriage dev · mode: ledger proxy: http://127.0.0.1:8787 (point your app's base_url here) dashboard: http://127.0.0.1:9090/ui/ ledger: ./decisions.jsonl next: $env:ANTHROPIC_BASE_URL = "http://127.0.0.1:8787" tokentriage wrap -- (Anthropic clients; no env changes) tokentriage doctor (check the setup) tokentriage demo (no traffic yet? the whole journey on synthetic data) ``` Three lines are cut from the top of that block: the judge egress warning you already read at step 3, once as a plain line and once as a structured record, and a `replica identity resolved` line carrying the capture host's name. Nothing else is edited. The timestamps and their `-07:00` offset belong to the capture box, as do the two backslashed store paths under a neutral home directory, and the `next:` block is shell-aware, so a POSIX shell gets `export ANTHROPIC_BASE_URL=...` where this one shows the PowerShell form. Four things on those lines are worth checking, and none of them is machine-specific. - **`judge_method: pointwise-bool`** although the config names no `method:` leaf. That is the absent-reading, made visible. - **`content_capture: true`** says the bytes a judge grades are being kept. If this reads `false`, step 6 will find nothing. - **`sample_rate: 1` and `max_usd_per_day: 1`** are your two leaves, read back. - **`backfill_interval: 2m0s`** is the pass that will do the grading, and the wait in step 6. Leave it running and open a second terminal.
```console Error: bind data plane "127.0.0.1:8787": listen tcp 127.0.0.1:8787: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. ``` Something else holds the port. That message is the Windows wording; Linux and macOS say `address already in use`. It prints twice, which is the shipped behaviour on this surface rather than two attempts. Either stop what is holding the port, or add a `server:` block to the config with `listen: "127.0.0.1:8788"` and `admin: "127.0.0.1:9091"`, and use the new port in step 5. `tokentriage doctor --config judge-quickstart.yaml` tells you which of the two planes is taken before you try again.
## 5. Send one request through it The OpenAI dialect lives at `/v1/chat/completions`. This is the same call you would make against `api.openai.com`, with the host swapped. The proxy supplies the upstream credential from `OPENAI_API_KEY`, so the request carries no key of its own. ```console $ curl -s http://127.0.0.1:8787/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"gpt-4o-mini", "messages":[{"role":"user","content":"Write one sentence about a decision ledger."}]}' {"id": "chatcmpl-w21-1", "object": "chat.completion", "created": 1750000000, "model": "gpt-4o-mini", "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "A canned completion from a loopback stub, not a model output."}}], "usage": {"prompt_tokens": 29, "completion_tokens": 17, "total_tokens": 46}} ``` The body in that capture came back from a loopback stub standing in for OpenAI, and it says so in its own text, because the captures on this page were produced with no traffic leaving the box. Yours will be your model's answer, with your own `id` and your own token counts. Everything the proxy did around it is the production path: the dialect parsing, the token accounting, the price lookup against the compiled-in snapshot, and the ledger write. One row landed on the decision ledger, and the part of it that matters here is the `content` block at the end, which reads `{"request": true, "response": true}`. Those are the stored bytes the judge will grade.
The upstream returned it, not the proxy. A `ledger`-mode proxy forwards whatever credential is available and hands the provider's answer back unchanged. Boot warns about the case where the variable is empty: ```console warning: upstream "openai" references OPENAI_API_KEY, which is not set — requests use the credential your client sends, and get the provider's 401 if it sends none ``` That line comes from a boot with the variable unset, which is why the step 4 capture does not carry it. If you saw it there, redo step 1 and restart. If you did not, the variable holds something the provider declined, and the fix is a working key rather than a config change.
## 6. Wait two minutes, then read the graded row Nothing more to type for a moment. The grading call is made after the response has been written, by a pass that walks the ledger on its own timer, so the request you sent is graded up to two minutes later. Then read the row back: ```console $ grep '"kind":"eval"' ./decisions.jsonl {"v":1,"kind":"eval","req_id":"tt_epxotefn4sh3a73brldo4p5j2u","eval":{"metric":"task_completion","value":1,"pass":true,"basis":"reported","evaluator":"judge-task-completion","eval_ver":"judge-v1+cfg15acf67","source":"evaluator","judge":{"model":"claude-haiku-4-5","prompt_sha":"1fc3923","validated":false,"method":"pointwise-bool"},"cost":{"usd":0.000177,"estimated":false,"unpriced":false},"ts":"2026-08-24T19:38:10.581Z"}} ``` That is one line of the ledger, as the off-path pass wrote it, 1 minute 10 seconds after the request above it and carrying the same `req_id`. Four values are that capture's own: `req_id` and `ts` move with every run, and `prompt_sha` with `eval_ver` identify this judge tuple, so yours follow your own config. `cost.usd` needs its own sentence. It is arithmetic over the token counts a loopback judge stub reported, priced from the `litellm@2026-07-14#179affb` snapshot compiled into this binary, so it is nobody's bill; yours comes from Anthropic. Everything else on the line was identical across four runs, including `metric`, `value`, `pass`, `basis`, `evaluator`, `source` and the whole `judge` object. That command is POSIX shell. In PowerShell it is `Select-String '"kind":"eval"' .\decisions.jsonl`.
Four things stop one from arriving, and the first is the common one. **The pass has not run yet.** It fires every two minutes and grades what appeared since the previous one, so a request that lands a moment after a pass waits nearly the full interval. Give it another two minutes before you go looking for a cause. **`JUDGE_API_KEY` is empty.** This is the one nothing reports. Running this page's setup with that variable unset, the judge seat behind the capture received no call at all, no eval row was written, and the running binary printed nothing about it. Set the variable and restart. **Capture is off.** If step 4's boot line read `content_capture: false`, a sampled row has no stored body to grade and is skipped under a named reason. Put `dashboard.capture.request` and `dashboard.capture.response` back, restart, and send step 5 again. The bytes are kept as the row is written, so the row you already have will not gain them. **The judge and the model that answered share a family.** The gate is consulted before the call, so a `claude` judge asked to grade a `claude` answer is turned away before it costs anything. Sending step 5's request with `"model":"claude-haiku-4-5"` against this page's judge, the judge seat behind the capture received no call, no eval row was written, and the log said nothing about that either. Point the request at a model from a different family, or point `judge.model` at one, and send step 5 again.
## What the row says That is the artefact this page exists to put in front of you, and three parts of it carry the answer. **Who graded it, and on what.** `judge.model` is the model that produced the grade and `judge.prompt_sha` is a seven-character digest of the prompt it was sent. `judge.method` is the method the composed scorer ran, read off the scorer rather than off your config file. **What the grading cost.** `cost.usd` is the grading call's own dollars, with `estimated` and `unpriced` both `false`. That figure is the judge's spend on this one row, and it sits beside the request's own cost on the decision row above it. **What the grade is allowed to claim.** `basis` reads `reported` and `judge.validated` reads `false`, and those two are the same fact from two directions: a model graded this row, and nobody has measured that model against human labels on this traffic. Four fields a validated judge's disclosure carries are absent here, `kappa`, `validated_n`, `swap_flip` and `iaa`, because this writer had nothing to put in them. Absent means not measured. It does not mean zero. You produced a grade. You did not produce evidence that the grade is any good, and the row says so on its face. Moving `reported` to `measured` takes human labels on the same traffic, Cohen's kappa (κ, the judge-versus-human agreement statistic) computed against them, and a validation run that clears four floors. That is a separate loop and a separate page. ## What this page did not cover **Measuring the judge.** Everything on this page produces one judge's opinion. What that opinion is worth is [the walk that promotes a judge](/docs/judges/agreement), and the labels it is measured against come through [the annotation queue](/docs/judges/collect-labels). **Every knob but four.** This config set a rate, a cap, a model and a key. The rest of the surface, including the panel form and every configuration this build turns away, is [configure a judge](/docs/judges/configure-a-judge), and what a judged workload costs at a sample rate you would run in production is [what judging costs](/docs/judges/cost-and-budget). **Why the figure stays where it is.** A promoted judge does not lift every cap on a quality number, and [why your number is reported, not measured](/docs/judges/basis-ceilings) is the one to read when a promotion lands and the screen does not move. **Reading it anywhere but the ledger.** The same disclosure reaches the dashboard and `GET /tt/api/v1/evals/judges`. This page used `grep` because it needs nothing else to be running.