# Check a gateway judge before a comparison Prepare four authored controls, inspect their separate spend ceiling, and compare the judge's observed verdicts with your expected labels. Source: https://tokentriage.com/docs/build/calibrate-gateway-judge --- import NextSteps from '../../../components/NextSteps.astro'; Before trusting a judge's comparison scores, check whether it distinguishes faithful answers from defective ones. This task uses four answers you write: two expected passes and two expected failures. The judge sees their source notes and required facts, but not your expected labels. You need a saved **judged** comparison with a published sample, readable reference notes and required facts on its source cases, and a saved gateway draft or applied release. The judge needs a qualified model and a saved connection authorized for judging in this scope. Your account also needs the comparison, test and content permissions; an owner role alone does not grant content access. Start with [the comparison guide](/docs/build/compare-cost-and-quality) if you have not created the comparison and published its sample. ## Prepare four controls Open the gateway's **Compare** tab and select the judged comparison. In **Judge calibration**, expand **Prepare four calibration examples**. Enter the exact judge model, connection, rubric digest and judge prompt version. Set the maximum characters per side, judge output token limit and a positive **Calibration spend ceiling (USD)**. For each example, select a **Published source case**. Read its reference notes and required facts before writing the **Authored answer**. Examples 1 and 2 should preserve those facts; examples 3 and 4 should contain a deliberate contradiction or omission you can identify. The source case version and notes remain pinned to the published sample. Choose **Save examples and check cost**. This saves four authored answer cases under the workspace's retention rules, then prepares a cost plan. It calls no provider. These controls are authored inputs, not outputs generated by either comparison arm. ## Review the separate cost plan The prepared plan should show **4 paid judge calls**, their conservative maximum and your ceiling, followed by the pinned judge model, prompt, connection, limits and rubric. Missing qualification, a missing grant or an insufficient ceiling must be resolved before starting. Do not interpret an unavailable amount as zero. This allocation is separate from the comparison's arm and judge calls. For example, a 20-case judged comparison plans 40 arm calls and 40 judge calls; these four calibration calls are an additional allocation, not part of those 80 calls. The cost check can supply the pinned scorer for a new comparison plan. That reuse means only that its configuration was prepared. It does not mean that calibration ran or passed. Editing the examples or judge settings invalidates the prepared plan; changing the scope or published sample also requires another check. ## Start once and read what happened Choose **Start four paid judge calls** only after reviewing that exact plan and its separate budget. Keep the returned calibration ID. **Load calibration** reopens its saved state and results without starting another run. If a start response is lost, **Retry the same calibration start** retains the original idempotency key and plan. Do not create a new attempt to fill a missing result. A call whose dispatch was recorded but whose outcome cannot be recovered stays unknown; it is not automatically sent again. Read **observed results of planned calls**, then compare each **Observed** verdict with **Expected**. A completed run is a lifecycle state, not a claim that all four verdicts agree. Keep unknown and not-executed rows visible. A missing cost remains not measured or unpriced; content and costs can also be withheld by your current permissions. ## Reopen through a supported client The same customer comparison scope contains three operations. Paths below are relative to the customer API's comparison resource, not the self-hosted admin API. | Operation | Relative route | TypeScript / Python method | |---|---|---| | Prepare without a provider call | `POST /calibrations/preflight` | `comparisons.calibrationPreflight` / `comparisons.calibration_preflight` | | Start the approved plan | `POST /calibrations` | `comparisons.calibrationStart` / `comparisons.calibration_start` | | Read a saved calibration | `GET /calibrations/{cal}` | `comparisons.calibration` / `comparisons.calibration` | Start carries the same subject, scorer, ceiling and four examples as preflight, plus its returned authority epoch and digest and a retained idempotency key. Read uses the saved calibration ID. Credential restrictions still apply: a management key cannot select a saved draft. Use the signed-in interface for that subject; do not substitute a runtime key. ## Keep a judge profile {/* Provenance of this section's captures: go test ./internal/cli/ -run TestJudgeProfileDocsTranscripts -update-judge-profile-transcripts then paste the named block from internal/cli/testdata/judge-profile-transcripts.txt (=== profile-create, === profile-create-invalid, === profile-get, === calibration-preflight, === calibration-list, === calibration-get). The command lines are the block's command re-wrapped with backslashes; the output is verbatim, including the preflight block's first line. controls.json is the file that test writes. The calibration-start blocks aren't shown: the loopback server doesn't apply the management-key rule, and a deployment answers preflight and start app_key_cannot_select_subject for any management key (internal/control/foundation/httpapi/judge_profiles.go). The portal labels are ui/src/features/foundation/quality/judgeprofiles at 5ae9488d3. The contract is docs/capabilities/quality-evaluation/q04-judge-profiles.md. */} If you check the same judge more than once, save it as a judge profile. A profile version pins the one judge template the product serves, `judge_faithfulness_v1`, with the judge model, the saved connection's current version, the two bounds, and four controls taken from the judged comparison's current published sample. Each calibration of a version is the paid check this page already describes: the same plan, pricing, ceiling below USD 10, separate budget and handling of unknown costs. The profile adds a name, versions, a pin state and a summary. [Judge profiles](/docs/reference/managed-evaluations#judge-profiles) lists every field, bound and answer. A control is two IDs and an expected label. The **source case** is a published case of the comparison; it gives the judge the reference notes and required facts. The **answer case** is an authored test case holding the answer the judge grades. Two controls expect a pass and two expect a fail, and the judge doesn't see those labels. A profile stores the IDs only; the text stays on the cases. | You need | For | |---|---| | `metadata.read` in the gateway environment | reading profiles, their versions and their list of calibrations | | `judge_profiles.write` on a binding issued at `role_version: 2`, and `comparisons.run` | saving a profile or a version | | a judge grant on the connection for this application and environment | saving, planning and starting | | `comparisons.run` | reading one calibration and its summary | | `tests.run`, `cost.read` and `content.read`, and a signed-in session | planning and starting a calibration | ### Save the profile In the portal, open the judged comparison's **Judge calibration** panel, expand **Judge profiles: save this judge and calibrate it again** and choose **New judge profile**. Enter the **Profile name**, **Judge model**, **Judge connection**, **Maximum characters per side** and **Judge output token limit**. For each control, enter an ID, choose a published source case, read its required facts and write the authored answer. Controls 1 and 2 expect a pass, 3 and 4 a fail. **Save profile** first saves each answer as an authored test case under the workspace's retention rules, then saves the profile. It calls no provider. From the CLI, the answers already exist as authored test cases, and a file names the four controls. `controls.json`: ```json [ {"control_id":"faithful-refund","source":{"case_id":"tcase_7f3k9qp2mxaz","case_version":1},"answer":{"case_id":"tcase_2c8v5hj9wpde","case_version":1},"expected_outcome":"pass"}, {"control_id":"faithful-vpn","source":{"case_id":"tcase_8g4m2rq3nyb2","case_version":1},"answer":{"case_id":"tcase_3d9w6kk2xqef","case_version":1},"expected_outcome":"pass"}, {"control_id":"invented-fee","source":{"case_id":"tcase_7f3k9qp2mxaz","case_version":1},"answer":{"case_id":"tcase_4e2x7mm3yrfg","case_version":1},"expected_outcome":"fail"}, {"control_id":"wrong-window","source":{"case_id":"tcase_8g4m2rq3nyb2","case_version":1},"answer":{"case_id":"tcase_5f3y8nn4zsgh","case_version":1},"expected_outcome":"fail"} ] ``` Each of the two published cases anchors one faithful answer and one defective answer. Then create the profile: ```console $ tokentriage cloud judge-profile create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --name "B2 faithfulness" \ --judge-model claude-haiku-4-5-20251001 --judge-connection pc_7f3k9qp2mxaz \ --max-chars 4096 --max-output-tokens 512 \ --controls-comparison cmp_7f3k9qp2mxaz --controls-sample-set-version 1 \ --controls-file controls.json --idempotency-key q-b2-judge-1 \ --base-url "$TOKENTRIAGE_BASE_URL" profile jprof_7f3k9qp2mxaz name B2 faithfulness state active version 1 profile_digest sha256:3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b template judge_faithfulness_v1 v1 (prompt 4e1f0a2, rubric sha256:2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f80912) judge claude-haiku-4-5-20251001 (thinking none) connection pc_7f3k9qp2mxaz v3 bounds max_chars 4096, max_output_tokens 512 controls_source cmp_7f3k9qp2mxaz sample set v1 sha256:6e5d4c3b2a19087f6e5d4c3b2a19087f6e5d4c3b2a19087f6e5d4c3b2a19087f CONTROL EXPECTED SOURCE ANSWER faithful-refund pass tcase_7f3k9qp2mxaz v1 tcase_2c8v5hj9wpde v1 faithful-vpn pass tcase_8g4m2rq3nyb2 v1 tcase_3d9w6kk2xqef v1 invented-fee fail tcase_7f3k9qp2mxaz v1 tcase_4e2x7mm3yrfg v1 wrong-window fail tcase_8g4m2rq3nyb2 v1 tcase_5f3y8nn4zsgh v1 ``` **How to tell it worked.** `state` is `active` at `version` 1. The server filled in what you didn't send: the build's prompt and rubric identities, the thinking mode for the model, the connection's current version (`v3`) and the sample set's digest. The four rows are two `pass` and two `fail`. Keep the `profile` ID. Sending the same command with the same key answers this profile again rather than creating a second one. The IDs and digests in these captures are fixture values from a local loopback server. If the file holds three controls, nothing is saved and the answer names the problem: ```console $ tokentriage cloud judge-profile create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --name "B2 faithfulness" \ --judge-model claude-haiku-4-5-20251001 --judge-connection pc_7f3k9qp2mxaz \ --max-chars 4096 --max-output-tokens 512 \ --controls-comparison cmp_7f3k9qp2mxaz --controls-sample-set-version 1 \ --controls-file three-controls.json --idempotency-key q-b2-judge-2 \ --base-url "$TOKENTRIAGE_BASE_URL" Error: customer.judge_profiles.create was refused (HTTP 422); detail: controls: controls_count; a profile has exactly four controls, two pass and two fail; type=judge_profile_invalid; request_id=req_0000000000000001 ``` Add the fourth control and send it again. The other reasons, such as an answer case that isn't authored or a source case outside the sample set, are listed under [Judge profile errors](/docs/reference/managed-evaluations#judge-profile-errors). ### Check its pin state A profile read answers `pin_state`, which the server works out on every read: whether the template, model price, connection version and sample set the version pins are still the ones this environment has. ```console $ tokentriage cloud judge-profile get --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz \ --base-url "$TOKENTRIAGE_BASE_URL" profile jprof_7f3k9qp2mxaz name B2 faithfulness state active version 1 profile_digest sha256:3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b template judge_faithfulness_v1 v1 (prompt 4e1f0a2, rubric sha256:2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f80912) judge claude-haiku-4-5-20251001 (thinking none) connection pc_7f3k9qp2mxaz v3 bounds max_chars 4096, max_output_tokens 512 controls_source cmp_7f3k9qp2mxaz sample set v1 sha256:6e5d4c3b2a19087f6e5d4c3b2a19087f6e5d4c3b2a19087f pin_state current CONTROL EXPECTED SOURCE ANSWER faithful-refund pass tcase_7f3k9qp2mxaz v1 tcase_2c8v5hj9wpde v1 faithful-vpn pass tcase_8g4m2rq3nyb2 v1 tcase_3d9w6kk2xqef v1 invented-fee fail tcase_7f3k9qp2mxaz v1 tcase_4e2x7mm3yrfg v1 wrong-window fail tcase_8g4m2rq3nyb2 v1 tcase_5f3y8nn4zsgh v1 ``` `current` means a calibration of this version can be planned. Editing the judge connection makes the state `stale` with reason `connection_version_moved`; publishing a newer sample on the comparison makes it `stale` with `controls_source_moved`. A plan for a stale version is answered `judge_profile_stale` with that reason, and nothing starts. The portal shows the same state as a chip beside the version. ### Add a version When a pin moves, publish a new version. In the portal, choose **New version** on the profile; the editor starts from the current version, and **Save version 2** publishes it. From the CLI, `cloud judge-profile version create` takes the same flags as `create` without `--name`, plus `--expected-version` (the version you're replacing, as `get` shows it) and a new `--idempotency-key`. If somebody published first, the answer is `judge_profile_version_conflict`; read the profile again and build on its current version. Versions don't change once published. Calibrations of an earlier version stay linked, and their summaries read `applicable: false` with reason `version_superseded`. ### Plan and start a calibration in the portal Planning and starting both need a signed-in session. A calibration always names the draft or release it runs against, and a management key, which the CLI and both SDKs send, may not select one. From those clients the two calls are answered `app_key_cannot_select_subject` (HTTP 403). On the profile, choose the **Profile version**. Under **Calibrate version 1**, leave **Spend limit (micro-USD)** empty for the default or enter your own, then choose **Check cost**. The card names the saved draft or applied release it runs against, the same subject the one-off calibration above uses. The plan appears under **Before you start: what this calibration will do**: four paid judge calls, the spend limit with the per-call maximum and the total reserved, the judge and its bounds, the profile pin, and the plan digest. Checking writes nothing and calls no provider. The CLI's preflight prints the same plan. This capture comes from the loopback server, which doesn't apply the management-key rule; the block's first line records that a deployment answers this command `app_key_cannot_select_subject`. ```console # needs a signed-in session: a deployment answers this verb app_key_cannot_select_subject (HTTP 403) for a management key $ tokentriage cloud judge-profile calibration preflight --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz --version 1 \ --subject draft:7 --base-url "$TOKENTRIAGE_BASE_URL" profile jprof_7f3k9qp2mxaz v1 comparison cmp_7f3k9qp2mxaz judge judge_faithfulness_v1 claude-haiku-4-5-20251001 planned_judge_calls 4 per_call_max 12000 micro-USD total_reserved 48000 micro-USD ceiling 500000 micro-USD authority_epoch 42 plan_digest sha256:4f5e6d7c8b9a0f1e2d3c4b5a69788f9e0d1c2b3a4f5e6d7c8b9a0f1e2d3c4b5a ``` The amounts are fixture values, not the model's price. Four calls at most 12,000 micro-USD each reserve 48,000 micro-USD under a ceiling of 500,000 micro-USD, which is USD 0.50. **Start calibration** approves that plan: the profile's `plan_digest` under that `authority_epoch`, with a request key made for this plan and shown beside it. The server plans again before it starts. If the connection version, the sample set or the plan moved since your check, nothing starts or is charged, and the answer is `judge_profile_stale` naming what moved. Check the cost again and approve what it shows now. If the answer to a start is lost, press **Start calibration** again. It sends the same key, so the server answers the same calibration instead of paying for a second one. A new **Check cost** is a new plan with a new key. ### Read the summary Listing a profile's calibrations needs only `metadata.read`, and reading one needs `comparisons.run`. Neither names a subject, so a management key can do both: ```console $ tokentriage cloud judge-profile calibration list --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz \ --base-url "$TOKENTRIAGE_BASE_URL" CALIBRATION COMPARISON VERSION IDEMPOTENCY_KEY STARTED_BY CREATED_AT jcal_7f3k9qp2mxaz cmp_7f3k9qp2mxaz 1 q-b2-cal-1 ada 2026-09-24T12:01:00Z ``` ```console $ tokentriage cloud judge-profile calibration get --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz \ --calibration jcal_7f3k9qp2mxaz --base-url "$TOKENTRIAGE_BASE_URL" calibration jcal_7f3k9qp2mxaz comparison cmp_7f3k9qp2mxaz profile jprof_7f3k9qp2mxaz v1 idempotency_key q-b2-cal-1 state complete controls 4: 4 observed, 3 agree, 1 disagree, 0 unknown, 0 not executed applicable true (pins_current) statistics none (four_controls_are_diagnostic) started_by ada created_at 2026-09-24T12:01:00Z Read it again with: tokentriage cloud judge-profile calibration get --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz --calibration jcal_7f3k9qp2mxaz ``` **How to tell it worked.** `state` is `complete`, and all four controls were observed: three verdicts matched their expected label and one didn't. `applicable` is `true` because version 1's pins are still current. In the portal, **Calibrations of version 1** lists the same run; open it to see each control's expected and observed verdict and which one disagreed. `statistics` is `none` in every summary. Four controls are a diagnostic, not a certification, so no agreement rate is computed; three of four is a count to investigate, not 75% accuracy. Unknown and not-executed controls are counted apart from disagreements, and neither is a pass. ## Keep the acceptance decision separate Four controls test the pinned judge on those inputs. They establish neither comparison quality nor the correctness of future answers. Review actual comparison outputs separately, record genuine human judgment where required, and make the candidate decision in the comparison workflow. Calibration cannot approve or deploy a candidate. This guide describes the shipped workflow and its locally verified states. A prepared plan or a mock screenshot is not evidence of a paid calibration run; this documentation does not claim that live calibration or human acceptance has completed.