Grade your first request with a judge

A score a model hands you is worth what you can find out about it afterwards. This page ends with that record on your screen: one line saying which model graded one of your requests, on which prompt, what the grading call cost, and how much anybody has checked the grader.

In this tutorial we’ll set two keys, write a short config, check it before anything spends, start the proxy, send one request through it, and read the graded row back off the decision ledger. That ledger is the JSON-lines file this tool writes one row per request into.

The Quickstart needed no credentials, because nothing in it talked to a provider. This page is the other kind. Two calls leave your machine and both are billed to you.

What you need

• The tokentriage binary on your PATH. The Quickstart puts one there in about a minute.
• An OpenAI key, in OPENAI_API_KEY. It pays for the one request you send.
• An Anthropic key, in JUDGE_API_KEY. It pays for the one grading call that follows. The judge transport in this build posts to the Anthropic Messages API, so this key is an Anthropic one.
• A fraction of a cent, twice. One request and one grading call. max_usd_per_day in the config below stops the judge at a dollar a day whatever else happens.

What it does not need:

• No account with us, no signup and no licence key.
• No human labels and no annotators. Grading a request and measuring the grader are two different jobs, and this page does the first.
• No database, no container and no second machine.
• No routing. mode: ledger prices traffic and leaves your responses alone.

Two vendors rather than one, and that is a rule rather than a preference. A judge is turned away from grading a model of its own family, because a model scoring its own family’s output measures self-preference as well as quality. So the model that answers and the model that grades cannot both be claude, and they cannot both be gpt.

How long it takes. The wall clock here is one wait you cannot shorten. Grading runs off the request path, on a pass that fires every two minutes, and a request is graded on the next pass after it lands. So the graded row arrives anywhere from a few seconds to two minutes later; on the runs behind this page it ranged from 8 seconds to 1 minute 56 seconds. Everything else is four commands. Getting the binary and getting the two keys sit outside that budget.

1. Put the two keys in your environment

TokenTriage reads credentials from environment variables by name and holds none of its own.

$ export OPENAI_API_KEY=sk-...
$ export JUDGE_API_KEY=sk-ant-...

In PowerShell those two lines are $env:OPENAI_API_KEY = "sk-..." and $env:JUDGE_API_KEY = "sk-ant-...". Step 3 tells you whether the first one took. Nothing tells you whether the second one did, and the callout under step 3 says what to do about that.

2. Write the config

Save this as judge-quickstart.yaml, beside wherever you are working.

version: 1

mode: ledger

upstreams:
  openai:
    provider: openai
    base_url: https://api.openai.com   # host only — your client already sends /v1
    api_key_env: OPENAI_API_KEY

ledger:
  decision_log:
    path: ./decisions.jsonl

dashboard:
  enabled: true
  capture:
    request: true
    response: true

evals:
  evaluators:
    - id: judge-task-completion
      sample_rate: 1.0
      max_usd_per_day: 1.00
      judge:
        external: allow
        model: claude-haiku-4-5
        api_key_env: JUDGE_API_KEY

Four things in that file decide whether anything gets graded.

• id: judge-task-completion names one of the two judge evaluators this build registers, and it is the one to pick. judge-groundedness carries no pass floor, so its rows reach validation with no verdict on them; judge-task-completion’s metric carries a floor of 1.0.
• sample_rate: 1.0 grades every request. The leaf is required and has no default.
• max_usd_per_day: 1.00 is required wherever a judge: block is present, and it is the stop on this tutorial’s spending.
• dashboard.capture keeps the request and response bytes. A judge reads stored text, so with capture off a sampled row has nothing on it to grade.

external: allow is the air-gap gate: the literal opt-in a config carries wherever a leaf reaches the network, and without it the evaluator is refused rather than composed. There is no method: leaf here, and step 4 shows what an absent one reads as.

3. Check it before anything spends

doctor is read-only. It boots nothing, writes nothing and prints no secret, so it is safe to run against a live setup as well as this one.

$ tokentriage doctor --config judge-quickstart.yaml
✓ binary — tokentriage dev (none), embedded pricing snapshot litellm@2026-07-14#179affb — clean
✓ config — judge-quickstart.yaml — strict decode clean: mode ledger, 1 upstream(s), 0 tier(s), 0 rule(s); 1 warning(s)
    warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
✓ compile — provider registry + routing policy compiled — 1 upstream(s) wired; 1 warning(s) — see below
    warning: evals.evaluators[0].judge: an LLM judge PHONES OUT to its provider on every sampled request (the model named by evals.evaluators[0].judge.model, using the key named by evals.evaluators[0].judge.api_key_env) — the single-binary air-gap posture is broken for this evaluator, and its outcomes are Basis: reported until the B.7 validation protocol promotes them
✓ ports — listen 127.0.0.1:8787 bindable, admin 127.0.0.1:9090 bindable — clean
✓ ledger — ./decisions.jsonl — not yet created; parent dir exists, created on first request — clean
○ index — D:\tt-w21\home\.tokentriage\index\decisions.sqlite — not built yet; the dashboard read-model builds on first run — clean
    run: tokentriage index rebuild   (optional; also auto-built when the dashboard starts)
✓ pricing — embedded table litellm@2026-07-14#179affb loaded, no overrides configured — clean
✓ keys — 1 upstream(s), 1 *_env reference(s) all set — clean
✓ dashboard — enabled — auth off, state D:\tt-w21\home\.tokentriage\dash, index D:\tt-w21\home\.tokentriage\index\decisions.sqlite, demo false — would compose (config valid; stores open at boot)
○ instance — not probed — pass --probe to check a running instance's admin plane over HTTP
    run: tokentriage doctor --probe                                   (probes the config's admin address)
    run: tokentriage doctor --probe --probe-url http://127.0.0.1:9090
✓ features — 31 capabilities — 21 shipped, 10 announced-not-shipped; counts agree with this binary (21 detectors / 7 categories, 92 providers, 8 recommendation types); since unreleased

ok — all checks passed (exit 0)

Read the glyphs: ✓ passed, ! warned, ○ skipped and said why, ✗ failed. Two lines carry ○ here and neither is a fault of yours. instance is skipped because nothing is running yet. index is skipped because nothing has built the dashboard read model on this machine, which is where a first run starts. Once something has, that line reads ✓ index — … open, no rows indexed yet and the rebuild hint under it goes away.

That capture came off a source build on Windows under a neutral home directory, so a release prints its version where this one prints dev, and the three backslashed store paths will be your own home directory, with / separators on Linux and macOS. The ✓ ports line is what you get with nothing started yet. Everything else on those lines is read from the config and the compiled defaults. Exit status was 0.

The judge warning prints twice, under config and again under compile. Reading it once is enough. It is the binary saying that this evaluator sends your captured request and response text to a provider on every sampled row, and that its verdicts stay reported until somebody measures the judge.

✗ config — probe-noexternal.yaml — strict decode failed: config probe-noexternal.yaml: evals.evaluators[0].judge.external: an LLM judge PHONES OUT to its provider on every sampled request and requires external: allow — the air-gap gate (SP-16), the same literal opt-in cache.store.l3, cache.embedders[] and shared_state.{redis,postgres} carry. Without it this evaluator is REFUSED: an air-gapped deployment must not be able to enable a phone-out judge at sample_rate: 1.0, and a reviewer grepping `external:` must be able to enumerate every egress path in the file. The value you wrote is deliberately not repeated back in this message

✗ config — probe-nocap.yaml — strict decode failed: config probe-nocap.yaml: evals.evaluators[0].max_usd_per_day: required whenever evals.evaluators[0].judge is present — an LLM judge SPENDS MONEY on every sampled request, and DQ13 refuses a metered evaluator without a declared daily cap. Set an explicit dollar figure (0 is legal and means "stop at the first priced call")

4. Start it

$ tokentriage run --config judge-quickstart.yaml
{"time":"2026-08-24T12:23:35.9786582-07:00","level":"INFO","msg":"evals: B.7 promotion registry opened","dir":"D:\\tt-w21\\home\\.tokentriage\\dash\\quality","validation_state":"empty","detail":"","standing_promotions":0,"auto_validate":false,"interval":"24h0m0s","new_labels_threshold":25}
{"time":"2026-08-24T12:23:35.9908707-07:00","level":"INFO","msg":"dashboard composed","index":"D:\\tt-w21\\home\\.tokentriage\\index\\decisions.sqlite","log":"./decisions.jsonl","auth":"off","demo":false}
{"time":"2026-08-24T12:23:35.9908707-07:00","level":"INFO","msg":"evals: judge evaluator wired (air-gap gate OPEN — this deployment PHONES OUT)","evaluator":"judge-task-completion","judge_method":"pointwise-bool","judge_instrument":"claude-haiku-4-5","judge_model":"claude-haiku-4-5","api_key_env":"JUDGE_API_KEY","sample_rate":1,"max_usd_per_day":1}
{"time":"2026-08-24T12:23:35.9913874-07:00","level":"INFO","msg":"evals: evaluator runner wired","evaluators":["judge-task-completion"],"backfill_interval":"2m0s","ledger":"./decisions.jsonl","content_capture":true}
{"time":"2026-08-24T12:23:35.9913874-07:00","level":"INFO","msg":"evals: the B.7 auto-validation loop is OFF (evals.validation.auto_validate.enabled is false); promotions restored from the artifact store still stand and still license `measured`, and `tokentriage evals validate` remains available as an operator-forced run"}
{"time":"2026-08-24T12:23:35.9913874-07:00","level":"INFO","msg":"tokentriage starting","listen":"127.0.0.1:8787","admin":"127.0.0.1:9090","mode":"ledger","upstreams":1,"tiers":0}
tokentriage dev · mode: ledger
proxy: http://127.0.0.1:8787   (point your app's base_url here)
dashboard: http://127.0.0.1:9090/ui/
ledger: ./decisions.jsonl
next:
  $env:ANTHROPIC_BASE_URL = "http://127.0.0.1:8787"
  tokentriage wrap -- <your app>        (Anthropic clients; no env changes)
  tokentriage doctor                    (check the setup)
  tokentriage demo                      (no traffic yet? the whole journey on synthetic data)

Three lines are cut from the top of that block: the judge egress warning you already read at step 3, once as a plain line and once as a structured record, and a replica identity resolved line carrying the capture host’s name. Nothing else is edited. The timestamps and their -07:00 offset belong to the capture box, as do the two backslashed store paths under a neutral home directory, and the next: block is shell-aware, so a POSIX shell gets export ANTHROPIC_BASE_URL=... where this one shows the PowerShell form.

Four things on those lines are worth checking, and none of them is machine-specific.

• judge_method: pointwise-bool although the config names no method: leaf. That is the absent-reading, made visible.
• content_capture: true says the bytes a judge grades are being kept. If this reads false, step 6 will find nothing.
• sample_rate: 1 and max_usd_per_day: 1 are your two leaves, read back.
• backfill_interval: 2m0s is the pass that will do the grading, and the wait in step 6.

Leave it running and open a second terminal.

Error: bind data plane "127.0.0.1:8787": listen tcp 127.0.0.1:8787: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

5. Send one request through it

The OpenAI dialect lives at /v1/chat/completions. This is the same call you would make against api.openai.com, with the host swapped. The proxy supplies the upstream credential from OPENAI_API_KEY, so the request carries no key of its own.

$ curl -s http://127.0.0.1:8787/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini",
       "messages":[{"role":"user","content":"Write one sentence about a decision ledger."}]}'
{"id": "chatcmpl-w21-1", "object": "chat.completion", "created": 1750000000, "model": "gpt-4o-mini", "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "A canned completion from a loopback stub, not a model output."}}], "usage": {"prompt_tokens": 29, "completion_tokens": 17, "total_tokens": 46}}

The body in that capture came back from a loopback stub standing in for OpenAI, and it says so in its own text, because the captures on this page were produced with no traffic leaving the box. Yours will be your model’s answer, with your own id and your own token counts. Everything the proxy did around it is the production path: the dialect parsing, the token accounting, the price lookup against the compiled-in snapshot, and the ledger write.

One row landed on the decision ledger, and the part of it that matters here is the content block at the end, which reads {"request": true, "response": true}. Those are the stored bytes the judge will grade.

warning: upstream "openai" references OPENAI_API_KEY, which is not set — requests use the credential your client sends, and get the provider's 401 if it sends none

6. Wait two minutes, then read the graded row

Nothing more to type for a moment. The grading call is made after the response has been written, by a pass that walks the ledger on its own timer, so the request you sent is graded up to two minutes later. Then read the row back:

$ grep '"kind":"eval"' ./decisions.jsonl
{"v":1,"kind":"eval","req_id":"tt_epxotefn4sh3a73brldo4p5j2u","eval":{"metric":"task_completion","value":1,"pass":true,"basis":"reported","evaluator":"judge-task-completion","eval_ver":"judge-v1+cfg15acf67","source":"evaluator","judge":{"model":"claude-haiku-4-5","prompt_sha":"1fc3923","validated":false,"method":"pointwise-bool"},"cost":{"usd":0.000177,"estimated":false,"unpriced":false},"ts":"2026-08-24T19:38:10.581Z"}}

That is one line of the ledger, as the off-path pass wrote it, 1 minute 10 seconds after the request above it and carrying the same req_id. Four values are that capture’s own: req_id and ts move with every run, and prompt_sha with eval_ver identify this judge tuple, so yours follow your own config.

cost.usd needs its own sentence. It is arithmetic over the token counts a loopback judge stub reported, priced from the litellm@2026-07-14#179affb snapshot compiled into this binary, so it is nobody’s bill; yours comes from Anthropic. Everything else on the line was identical across four runs, including metric, value, pass, basis, evaluator, source and the whole judge object.

That command is POSIX shell. In PowerShell it is Select-String '"kind":"eval"' .\decisions.jsonl.

What the row says

That is the artefact this page exists to put in front of you, and three parts of it carry the answer.

Who graded it, and on what. judge.model is the model that produced the grade and judge.prompt_sha is a seven-character digest of the prompt it was sent. judge.method is the method the composed scorer ran, read off the scorer rather than off your config file.

What the grading cost. cost.usd is the grading call’s own dollars, with estimated and unpriced both false. That figure is the judge’s spend on this one row, and it sits beside the request’s own cost on the decision row above it.

What the grade is allowed to claim. basis reads reported and judge.validated reads false, and those two are the same fact from two directions: a model graded this row, and nobody has measured that model against human labels on this traffic. Four fields a validated judge’s disclosure carries are absent here, kappa, validated_n, swap_flip and iaa, because this writer had nothing to put in them. Absent means not measured. It does not mean zero.

What this page did not cover

Measuring the judge. Everything on this page produces one judge’s opinion. What that opinion is worth is the walk that promotes a judge, and the labels it is measured against come through the annotation queue.

Every knob but four. This config set a rate, a cap, a model and a key. The rest of the surface, including the panel form and every configuration this build turns away, is configure a judge, and what a judged workload costs at a sample rate you would run in production is what judging costs.

Why the figure stays where it is. A promoted judge does not lift every cap on a quality number, and why your number is reported, not measured is the one to read when a promotion lands and the screen does not move.

Reading it anywhere but the ledger. The same disclosure reaches the dashboard and GET /tt/api/v1/evals/judges. This page used grep because it needs nothing else to be running.