Check a gateway judge before a comparison

Before trusting a judge’s comparison scores, check whether it distinguishes faithful answers from defective ones. This task uses four answers you write: two expected passes and two expected failures. The judge sees their source notes and required facts, but not your expected labels.

You need a saved judged comparison with a published sample, readable reference notes and required facts on its source cases, and a saved gateway draft or applied release. The judge needs a qualified model and a saved connection authorized for judging in this scope. Your account also needs the comparison, test and content permissions; an owner role alone does not grant content access. Start with the comparison guide if you have not created the comparison and published its sample.

Prepare four controls

Open the gateway’s Compare tab and select the judged comparison. In Judge calibration, expand Prepare four calibration examples. Enter the exact judge model, connection, rubric digest and judge prompt version. Set the maximum characters per side, judge output token limit and a positive Calibration spend ceiling (USD).

For each example, select a Published source case. Read its reference notes and required facts before writing the Authored answer. Examples 1 and 2 should preserve those facts; examples 3 and 4 should contain a deliberate contradiction or omission you can identify. The source case version and notes remain pinned to the published sample.

Choose Save examples and check cost. This saves four authored answer cases under the workspace’s retention rules, then prepares a cost plan. It calls no provider. These controls are authored inputs, not outputs generated by either comparison arm.

Review the separate cost plan

The prepared plan should show 4 paid judge calls, their conservative maximum and your ceiling, followed by the pinned judge model, prompt, connection, limits and rubric. Missing qualification, a missing grant or an insufficient ceiling must be resolved before starting. Do not interpret an unavailable amount as zero.

This allocation is separate from the comparison’s arm and judge calls. For example, a 20-case judged comparison plans 40 arm calls and 40 judge calls; these four calibration calls are an additional allocation, not part of those 80 calls.

The cost check can supply the pinned scorer for a new comparison plan. That reuse means only that its configuration was prepared. It does not mean that calibration ran or passed. Editing the examples or judge settings invalidates the prepared plan; changing the scope or published sample also requires another check.

Start once and read what happened

Choose Start four paid judge calls only after reviewing that exact plan and its separate budget. Keep the returned calibration ID. Load calibration reopens its saved state and results without starting another run.

If a start response is lost, Retry the same calibration start retains the original idempotency key and plan. Do not create a new attempt to fill a missing result. A call whose dispatch was recorded but whose outcome cannot be recovered stays unknown; it is not automatically sent again.

Read observed results of planned calls, then compare each Observed verdict with Expected. A completed run is a lifecycle state, not a claim that all four verdicts agree. Keep unknown and not-executed rows visible. A missing cost remains not measured or unpriced; content and costs can also be withheld by your current permissions.

Reopen through a supported client

The same customer comparison scope contains three operations. Paths below are relative to the customer API’s comparison resource, not the self-hosted admin API.

Operation	Relative route	TypeScript / Python method
Prepare without a provider call	`POST /calibrations/preflight`	`comparisons.calibrationPreflight` / `comparisons.calibration_preflight`
Start the approved plan	`POST /calibrations`	`comparisons.calibrationStart` / `comparisons.calibration_start`
Read a saved calibration	`GET /calibrations/{cal}`	`comparisons.calibration` / `comparisons.calibration`

Start carries the same subject, scorer, ceiling and four examples as preflight, plus its returned authority epoch and digest and a retained idempotency key. Read uses the saved calibration ID. Credential restrictions still apply: a management key cannot select a saved draft. Use the signed-in interface for that subject; do not substitute a runtime key.

Keep a judge profile

If you check the same judge more than once, save it as a judge profile. A profile version pins the one judge template the product serves, judge_faithfulness_v1, with the judge model, the saved connection’s current version, the two bounds, and four controls taken from the judged comparison’s current published sample. Each calibration of a version is the paid check this page already describes: the same plan, pricing, ceiling below USD 10, separate budget and handling of unknown costs. The profile adds a name, versions, a pin state and a summary. Judge profiles lists every field, bound and answer.

A control is two IDs and an expected label. The source case is a published case of the comparison; it gives the judge the reference notes and required facts. The answer case is an authored test case holding the answer the judge grades. Two controls expect a pass and two expect a fail, and the judge doesn’t see those labels. A profile stores the IDs only; the text stays on the cases.

You need	For
`metadata.read` in the gateway environment	reading profiles, their versions and their list of calibrations
`judge_profiles.write` on a binding issued at `role_version: 2`, and `comparisons.run`	saving a profile or a version
a judge grant on the connection for this application and environment	saving, planning and starting
`comparisons.run`	reading one calibration and its summary
`tests.run`, `cost.read` and `content.read`, and a signed-in session	planning and starting a calibration

Save the profile

In the portal, open the judged comparison’s Judge calibration panel, expand Judge profiles: save this judge and calibrate it again and choose New judge profile. Enter the Profile name, Judge model, Judge connection, Maximum characters per side and Judge output token limit. For each control, enter an ID, choose a published source case, read its required facts and write the authored answer. Controls 1 and 2 expect a pass, 3 and 4 a fail. Save profile first saves each answer as an authored test case under the workspace’s retention rules, then saves the profile. It calls no provider.

From the CLI, the answers already exist as authored test cases, and a file names the four controls. controls.json:

[
 {"control_id":"faithful-refund","source":{"case_id":"tcase_7f3k9qp2mxaz","case_version":1},"answer":{"case_id":"tcase_2c8v5hj9wpde","case_version":1},"expected_outcome":"pass"},
 {"control_id":"faithful-vpn","source":{"case_id":"tcase_8g4m2rq3nyb2","case_version":1},"answer":{"case_id":"tcase_3d9w6kk2xqef","case_version":1},"expected_outcome":"pass"},
 {"control_id":"invented-fee","source":{"case_id":"tcase_7f3k9qp2mxaz","case_version":1},"answer":{"case_id":"tcase_4e2x7mm3yrfg","case_version":1},"expected_outcome":"fail"},
 {"control_id":"wrong-window","source":{"case_id":"tcase_8g4m2rq3nyb2","case_version":1},"answer":{"case_id":"tcase_5f3y8nn4zsgh","case_version":1},"expected_outcome":"fail"}
]

Each of the two published cases anchors one faithful answer and one defective answer. Then create the profile:

$ tokentriage cloud judge-profile create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --name "B2 faithfulness" \
    --judge-model claude-haiku-4-5-20251001 --judge-connection pc_7f3k9qp2mxaz \
    --max-chars 4096 --max-output-tokens 512 \
    --controls-comparison cmp_7f3k9qp2mxaz --controls-sample-set-version 1 \
    --controls-file controls.json --idempotency-key q-b2-judge-1 \
    --base-url "$TOKENTRIAGE_BASE_URL"
profile            jprof_7f3k9qp2mxaz
name               B2 faithfulness
state              active
version            1
profile_digest     sha256:3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b
template           judge_faithfulness_v1 v1 (prompt 4e1f0a2, rubric sha256:2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f80912)
judge              claude-haiku-4-5-20251001 (thinking none)
connection         pc_7f3k9qp2mxaz v3
bounds             max_chars 4096, max_output_tokens 512
controls_source    cmp_7f3k9qp2mxaz sample set v1 sha256:6e5d4c3b2a19087f6e5d4c3b2a19087f6e5d4c3b2a19087f6e5d4c3b2a19087f

CONTROL          EXPECTED  SOURCE                 ANSWER
faithful-refund  pass      tcase_7f3k9qp2mxaz v1  tcase_2c8v5hj9wpde v1
faithful-vpn     pass      tcase_8g4m2rq3nyb2 v1  tcase_3d9w6kk2xqef v1
invented-fee     fail      tcase_7f3k9qp2mxaz v1  tcase_4e2x7mm3yrfg v1
wrong-window     fail      tcase_8g4m2rq3nyb2 v1  tcase_5f3y8nn4zsgh v1

How to tell it worked. state is active at version 1. The server filled in what you didn’t send: the build’s prompt and rubric identities, the thinking mode for the model, the connection’s current version (v3) and the sample set’s digest. The four rows are two pass and two fail. Keep the profile ID. Sending the same command with the same key answers this profile again rather than creating a second one.

The IDs and digests in these captures are fixture values from a local loopback server.

If the file holds three controls, nothing is saved and the answer names the problem:

$ tokentriage cloud judge-profile create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --name "B2 faithfulness" \
    --judge-model claude-haiku-4-5-20251001 --judge-connection pc_7f3k9qp2mxaz \
    --max-chars 4096 --max-output-tokens 512 \
    --controls-comparison cmp_7f3k9qp2mxaz --controls-sample-set-version 1 \
    --controls-file three-controls.json --idempotency-key q-b2-judge-2 \
    --base-url "$TOKENTRIAGE_BASE_URL"
Error: customer.judge_profiles.create was refused (HTTP 422); detail: controls: controls_count; a profile has exactly four controls, two pass and two fail; type=judge_profile_invalid; request_id=req_0000000000000001

Add the fourth control and send it again. The other reasons, such as an answer case that isn’t authored or a source case outside the sample set, are listed under Judge profile errors.

Check its pin state

A profile read answers pin_state, which the server works out on every read: whether the template, model price, connection version and sample set the version pins are still the ones this environment has.

$ tokentriage cloud judge-profile get --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz \
    --base-url "$TOKENTRIAGE_BASE_URL"
profile            jprof_7f3k9qp2mxaz
name               B2 faithfulness
state              active
version            1
profile_digest     sha256:3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b
template           judge_faithfulness_v1 v1 (prompt 4e1f0a2, rubric sha256:2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f80912)
judge              claude-haiku-4-5-20251001 (thinking none)
connection         pc_7f3k9qp2mxaz v3
bounds             max_chars 4096, max_output_tokens 512
controls_source    cmp_7f3k9qp2mxaz sample set v1 sha256:6e5d4c3b2a19087f6e5d4c3b2a19087f6e5d4c3b2a19087f
pin_state          current

CONTROL          EXPECTED  SOURCE                 ANSWER
faithful-refund  pass      tcase_7f3k9qp2mxaz v1  tcase_2c8v5hj9wpde v1
faithful-vpn     pass      tcase_8g4m2rq3nyb2 v1  tcase_3d9w6kk2xqef v1
invented-fee     fail      tcase_7f3k9qp2mxaz v1  tcase_4e2x7mm3yrfg v1
wrong-window     fail      tcase_8g4m2rq3nyb2 v1  tcase_5f3y8nn4zsgh v1

current means a calibration of this version can be planned. Editing the judge connection makes the state stale with reason connection_version_moved; publishing a newer sample on the comparison makes it stale with controls_source_moved. A plan for a stale version is answered judge_profile_stale with that reason, and nothing starts. The portal shows the same state as a chip beside the version.

Add a version

When a pin moves, publish a new version. In the portal, choose New version on the profile; the editor starts from the current version, and Save version 2 publishes it. From the CLI, cloud judge-profile version create takes the same flags as create without --name, plus --expected-version (the version you’re replacing, as get shows it) and a new --idempotency-key. If somebody published first, the answer is judge_profile_version_conflict; read the profile again and build on its current version.

Versions don’t change once published. Calibrations of an earlier version stay linked, and their summaries read applicable: false with reason version_superseded.

Plan and start a calibration in the portal

Planning and starting both need a signed-in session. A calibration always names the draft or release it runs against, and a management key, which the CLI and both SDKs send, may not select one. From those clients the two calls are answered app_key_cannot_select_subject (HTTP 403).

On the profile, choose the Profile version. Under Calibrate version 1, leave Spend limit (micro-USD) empty for the default or enter your own, then choose Check cost. The card names the saved draft or applied release it runs against, the same subject the one-off calibration above uses. The plan appears under Before you start: what this calibration will do: four paid judge calls, the spend limit with the per-call maximum and the total reserved, the judge and its bounds, the profile pin, and the plan digest. Checking writes nothing and calls no provider.

The CLI’s preflight prints the same plan. This capture comes from the loopback server, which doesn’t apply the management-key rule; the block’s first line records that a deployment answers this command app_key_cannot_select_subject.

# needs a signed-in session: a deployment answers this verb app_key_cannot_select_subject (HTTP 403) for a management key
$ tokentriage cloud judge-profile calibration preflight --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz --version 1 \
    --subject draft:7 --base-url "$TOKENTRIAGE_BASE_URL"
profile                   jprof_7f3k9qp2mxaz v1
comparison                cmp_7f3k9qp2mxaz
judge                     judge_faithfulness_v1 claude-haiku-4-5-20251001
planned_judge_calls       4
per_call_max              12000 micro-USD
total_reserved            48000 micro-USD
ceiling                   500000 micro-USD
authority_epoch           42
plan_digest               sha256:4f5e6d7c8b9a0f1e2d3c4b5a69788f9e0d1c2b3a4f5e6d7c8b9a0f1e2d3c4b5a

The amounts are fixture values, not the model’s price. Four calls at most 12,000 micro-USD each reserve 48,000 micro-USD under a ceiling of 500,000 micro-USD, which is USD 0.50.

Start calibration approves that plan: the profile’s plan_digest under that authority_epoch, with a request key made for this plan and shown beside it. The server plans again before it starts. If the connection version, the sample set or the plan moved since your check, nothing starts or is charged, and the answer is judge_profile_stale naming what moved. Check the cost again and approve what it shows now.

If the answer to a start is lost, press Start calibration again. It sends the same key, so the server answers the same calibration instead of paying for a second one. A new Check cost is a new plan with a new key.

Read the summary

Listing a profile’s calibrations needs only metadata.read, and reading one needs comparisons.run. Neither names a subject, so a management key can do both:

$ tokentriage cloud judge-profile calibration list --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz \
    --base-url "$TOKENTRIAGE_BASE_URL"
CALIBRATION        COMPARISON        VERSION  IDEMPOTENCY_KEY  STARTED_BY  CREATED_AT
jcal_7f3k9qp2mxaz  cmp_7f3k9qp2mxaz  1        q-b2-cal-1       ada         2026-09-24T12:01:00Z

$ tokentriage cloud judge-profile calibration get --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz \
    --calibration jcal_7f3k9qp2mxaz --base-url "$TOKENTRIAGE_BASE_URL"
calibration        jcal_7f3k9qp2mxaz
comparison         cmp_7f3k9qp2mxaz
profile            jprof_7f3k9qp2mxaz v1
idempotency_key    q-b2-cal-1
state              complete
controls           4: 4 observed, 3 agree, 1 disagree, 0 unknown, 0 not executed
applicable         true (pins_current)
statistics         none (four_controls_are_diagnostic)
started_by         ada
created_at         2026-09-24T12:01:00Z

Read it again with: tokentriage cloud judge-profile calibration get --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM --profile jprof_7f3k9qp2mxaz --calibration jcal_7f3k9qp2mxaz

How to tell it worked. state is complete, and all four controls were observed: three verdicts matched their expected label and one didn’t. applicable is true because version 1’s pins are still current. In the portal, Calibrations of version 1 lists the same run; open it to see each control’s expected and observed verdict and which one disagreed.

statistics is none in every summary. Four controls are a diagnostic, not a certification, so no agreement rate is computed; three of four is a count to investigate, not 75% accuracy. Unknown and not-executed controls are counted apart from disagreements, and neither is a pass.

Keep the acceptance decision separate

Four controls test the pinned judge on those inputs. They establish neither comparison quality nor the correctness of future answers. Review actual comparison outputs separately, record genuine human judgment where required, and make the candidate decision in the comparison workflow. Calibration cannot approve or deploy a candidate.

This guide describes the shipped workflow and its locally verified states. A prepared plan or a mock screenshot is not evidence of a paid calibration run; this documentation does not claim that live calibration or human acceptance has completed.