# Compare cost and quality before you switch models Run your own saved tests against two configurations, read the evidence with its cost, and record what you decided, from the CLI or either SDK. Source: https://tokentriage.com/docs/build/compare-cost-and-quality --- import Callout from '../../../components/Callout.astro'; import CodeTabs from '../../../components/CodeTabs.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; {/* Provenance of every transcript below, so a later editor regenerates rather than guesses: the fixture JSON excerpts — go test ./internal/cli/ -run TestComparisonDocsTranscripts -update-comparison-transcripts then paste the named block from internal/cli/testdata/comparison-transcripts.txt the operation ids, methods and paths — internal/control/foundation/httpapi/testdata/contract.golden the refusal vocabulary — internal/control/foundation/comparison/refusals.go The lane answering those transcripts is a loopback fixture, so the money in them is arithmetic over a fixture and is nobody's bill. */} A cheaper model is a guess until you have run your own examples through both configurations and looked at what came back. This page is how you do that against a hosted gateway: pick the saved tests you already trust, pin the configuration you are serving today against the one you are considering, and get back one sentence about quality with the numbers under it, and the cost of each side kept apart. You will get more from this page if the gateway already has saved test cases and you have sent traffic through it: see [Read your gateway's traffic and spend](/docs/build/gateway-insights). ## What a comparison is Two arms, your cases, one scorer. - **The baseline** is what the environment is serving now. **The candidate** is a saved draft revision. Both are pinned by the server at their effective digest, so what ran is checkable afterwards. - **The sample** is between two and fifty of the gateway's saved test case versions. You mark the ones where a failure is disqualifying whatever the counts say. - **The scorer** is `labeled` (each answer is checked against the expected label on the case) or `judged` (one model from a different family grades both answers against the case's reference notes). The kind is fixed when you create the comparison. - **The bar** is declared before the run: how many cases must pass, whether the candidate must match the baseline, how many ungraded cases you will tolerate. Starting a run makes one paid provider call per case per arm, and a judged run adds one judge call per case per arm. Twenty cases judged is eighty calls against your own connections, inside your own budget windows. Run the preflight first. An owner or administrator does not automatically have `content.read`. Starting a comparison requires an explicit content grant for its saved inputs, along with the comparison and test permissions. Each arm connection also needs a current `comparison` grant for this application and environment; an `application` grant alone is insufficient. Reading cost and metadata does not grant access to prompts, reference notes or answers. The JSON response excerpts below are synthetic examples of the response shape. They are not a production acceptance result or a bill. ## Create the comparison One name, one kind. It reaches no provider and costs nothing. ```text $ tokentriage cloud comparison create --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \ --name "cheaper checkout" --kind judged --json { "schema": "customer/v1", "request_id": "req_0000000000000000", "kind": "comparison", "data": { "comparison_id": "cmp_ABCDEFGHJKLM", "gateway_id": "gw_ABCDEFGHJKLM", "environment_id": "env_ABCDEFGHJKLM", "name": "cheaper checkout", "kind": "judged", "current_sample_version": 3, "created_by": "ada", "created_at": "2026-09-22T12:00:00Z", "revision": 1 } } ``` Then publish the examples. A judged comparison needs reference notes on every case; a labeled one needs an expected label. Publishing the same selection twice returns the version that already holds it, so a retry does not fork your evidence. ```text $ tokentriage cloud comparison samples publish --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --comparison cmp_ABCDEFGHJKLM \ --case tcase_ABCDEFGHJKLM:1:critical:notes=cobj_ABCDEFGHJKLM \ --case tcase_BCDEFGHJKLMN:2:notes=cobj_BCDEFGHJKLMN ``` ## Read the plan before you spend For a judged comparison, [prepare four judge controls](/docs/build/calibrate-gateway-judge) and check their separate cost plan first. That free provider-cost check pins the scorer for a new comparison plan. Running those controls makes four separate paid judge calls; merely preparing them does not establish calibrated results or human acceptance. In the signed-in builder, select the currently applied baseline, saved candidate revision, published sample version, scorer and acceptance rule. Review preflight before starting. It resolves the subjects again and reports the exact dispatch count, current qualified maximum for each model, and the total allocation. The allocations can differ between arms; a cheap model does not give the expensive arm a larger limit. These are spend ceilings, not measured costs. A missing content grant, unknown remaining allowance, unqualified or expired model bound, underfunded plan, same-family judge or missing judge grant prevents a valid plan. Twenty judged cases require 40 arm calls and 40 judge calls. A plan that cannot fit the configured ceiling must be revised or explicitly authorized through the operator's supported policy process before it can start. Management API keys cannot select a saved draft. The CLI and management-key SDKs preserve this restriction and return `app_key_cannot_select_subject`; use the signed-in owner interface for that comparison. Never substitute a runtime key or borrow another person's session to get around the refusal. For subjects your API credential may select, the CLI accepts the complete API plan: ```console tokentriage cloud comparison run preflight --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" --comparison "$CMP" \ --plan-file comparison-plan.json --json ``` The file must contain `sample_set_version`, explicit `baseline` and `candidate` selectors, the complete `scorer`, the complete acceptance `rule`, and a positive `ceiling_micro_usd`. A scorer includes its ID, version and rubric digest; a judged scorer also includes its pinned judge model, prompt digest, bounds and saved judge connection. Use the exact plan you reviewed. Unknown fields, duplicate JSON members and mixing `--plan-file` with individual plan flags are refused. The server remains the authority for subject selection, permissions, qualification and spend. ## Start it, and follow it Starting spends real provider money. Keep the plan unchanged, use the authority epoch returned by preflight, and choose one idempotency key for this attempt. A repeated start with that key and plan returns the same run. A different key buys a new attempt; it is not a recovery procedure for an uncertain result. ```console tokentriage cloud comparison run start --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" --comparison "$CMP" \ --plan-file comparison-plan.json --approved-epoch "$EPOCH" \ --idempotency-key "$ATTEMPT_KEY" --json ``` The management-key subject restrictions above also apply to start. For a saved-draft comparison, start in the signed-in builder and use its returned run ID for CLI or SDK reads. Wait for a terminal state before requesting results. A partial or abandoned run retains its gaps and uncertain charges; it is not a successful run and should not be replayed automatically. ## Read the evidence ```text $ tokentriage cloud comparison results --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \ --comparison cmp_ABCDEFGHJKLM --run crun_ABCDEFGHJKLM --json { "kind": "comparison-results", "data": { "run": { "state": "partial", "terminal_state": "partial" }, "outcome": "meets_bar", "sentence": "Meets your bar on these examples", "pass_count": 18, "case_count": 20, "baseline_pass_count": 18, "unknown_count": 1, "cost": { "per_arm": [ { "arm_ordinal": 1, "priced_nano_usd": 412000, "priced_cases": 19, "pending_cases": 1, "unpriced_reasons": ["price_pending"] }, { "arm_ordinal": 2, "priced_nano_usd": 126000, "priced_cases": 19, "pending_cases": 1, "unpriced_reasons": ["price_pending"] } ], "judge_spend_nano_usd": 84000, "total_run_spend_nano_usd": 622000, "comparable_population": { "count": 19, "total_cases": 20 }, "sample_serving_cost_reduction": { "numerator_nano_usd": 286000, "denominator_nano_usd": 412000 }, "reduction_state": "available", "coverage": "partial", "coverage_reason": "unpriced_population" }, "cost_state": "available", "gaps": [ { "ordinal": 7, "arm_ordinal": 2, "critical": false, "reason": "verdict_unknown", "detail": "judge_no_verdict" } ], "content_withheld": true } } ``` Read it in this order: 1. **The sentence, with its counts.** 18 of 20, against the baseline's 18. The denominator is every case you asked for, not the ones that answered, so a case that failed cheaply cannot lift the headline. 2. **What is missing.** One case is ungraded and `gaps` says which and why. An ungraded case is not a failure: the candidate was not tried and found wanting there, it was not graded at all. 3. **The money, apart from the quality.** Each arm's priced spend, the judge's spend, the total, and the population the comparison is over: 19 of 20, because one case is not priced yet. The reduction is a ratio of two integers over that population. 4. **What you cannot see.** `content_withheld` means prompts, completions, judge answers and notes were removed because you hold no `content.read`. Without `cost.read` the cost block is absent and `cost_state` reads `unavailable` with a named reason, rather than showing a zero. ### Project only over a declared application cohort To ask what a saved difference would imply for matching application traffic, add a UTC window to the results read. These parameters make no provider call and do not start or retry a comparison. You need a terminal run and access to the same gateway environment; a monetary projection also requires `cost.read`. ```console tokentriage cloud comparison results --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --comparison "$CMP" --run "$RUN" --cohort-from "$FROM" --cohort-to "$TO" \ --cohort-release "$RELEASE" --json ``` Omit `--cohort-release` to include all releases in the declared window. To continue a previous insight read, pass its complete `read_context` JSON to `--cohort-read-context`. The server checks that context; an expired or unreadable context does not authorize a new population silently. ```typescript const evidence = await client.comparisons.results(scope, comparison, run, { cohort: { from, to, release_id: releaseId, read_context: overview.read_context }, }); console.log(evidence.data?.cohort_projection); ``` ```python evidence = client.comparisons.results(scope, comparison, run, cohort={ "from": start, "to": end, "release_id": release_id, "read_context": overview["read_context"], }) print(evidence["data"].get("cohort_projection")) ``` The direct API uses the existing `GET .../comparisons/{comparison}/runs/{run}/results` with `cohort_from`, `cohort_to`, optional `cohort_release_id` and optional `cohort_read_context` query parameters. URL-encode the read context as JSON. These parameters describe application traffic only. Read `cohort_projection.state` before its amount. An available result includes a signed `projected_reduction_nano_usd`, the covered and uncovered request counts, tested pairs, rule identity, evidence digest, price basis and assumptions. Negative amounts remain negative. An unavailable result carries a reason and omits the amount; uncovered traffic has no assumed saving. The server derives the projection from qualified paired deltas and matching request counts. The clients print that result without recomputing it. The shipped `sdk/ts/examples/read_comparison.ts` and `sdk/python/examples/read_comparison.py` accept optional `TOKENTRIAGE_FROM` and `TOKENTRIAGE_TO` together, plus optional `TOKENTRIAGE_RELEASE`. They preserve `cohort_projection` alongside the saved sample evidence. Focused tests execute these examples and the CLI against local contract responses, including negative and unavailable results. That verification is not a live cohort measurement. A deployment without this producer can still return saved sample results without a cohort projection.
A reviewer can replace one cell's verdict. It appends a revision and edits nothing, so the banner recomputes and the verdict it replaced stays readable beside it. ```text $ tokentriage cloud comparison review --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \ --comparison cmp_ABCDEFGHJKLM --run crun_ABCDEFGHJKLM \ --case tcase_ABCDEFGHJKLM --case-version 1 --arm 2 --outcome fail ```
## Record what you decided The outcome is worked out from the verdicts and the bar you set; the action is yours. The outcome you claim is checked against the computed one, and a disagreement is `decision_outcome_mismatch`. ```text $ tokentriage cloud comparison decide --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \ --comparison cmp_ABCDEFGHJKLM --run crun_ABCDEFGHJKLM \ --claimed-outcome meets_bar --action change_route --arm 2 --json { "kind": "comparison-decision", "data": { "decision": { "decision_id": "cdec_ABCDEFGHJKLM", "outcome": "meets_bar", "action": "change_route", "evidence_digest": "sha256:5e6f...", "decided_by": "ada", "revision": 1, "decided_at": "2026-09-22T12:30:00Z" } } } ``` Recording a decision deploys nothing. It is the evidence a release cites, and the release path checks again that the comparison still describes what you are about to ship. If the setup moved you get `comparison_evidence_stale`; if the configuration you measured against is no longer the one being served you get `baseline_release_changed`. Two words, because the fixes differ: re-run against the current setup, or re-run against what is live now. `change_route` also needs permission to deploy and an outcome of `meets_bar`. `keep_route` needs neither and touches no release. ### Review the evidence yourself Open a run's review panel from its comparison page in the portal. It lists every cell, filtered by **Needs attention** (the default), **Not reviewed by a person**, **Reviewed**, or **All**. There, a person reads the source and both arms' outputs, then records a verdict with a note. The verdict carries the digests of what was on screen for that reading. The panel labels a verdict backed by those digests **attested**, not "verified": the server checks that a digest names the stored bytes, not that the sentence in the note is correct. Without a content grant the panel withholds the source and the outputs, and the verdict it records reads **not attested**. The CLI and both SDKs carry the same attestation. `--read-source-sha256` and `--read-output-sha256` send the digest of the source and, for a cell that produced one, the output the reviewer read; a digest that names bytes other than the cell's is `review_reading_mismatch`. Omit both flags to record a verdict as it was recorded before this attestation existed, and it reads not attested. ```console $ tokentriage cloud comparison review --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --comparison "$CMP" --run "$RUN" --case "$CASE" --case-version 1 --arm 2 \ --outcome fail --note-file note.txt \ --read-source-sha256 "$SOURCE_SHA256" --read-output-sha256 "$OUTPUT_SHA256" ``` A decision also carries the person's reasons. `--rationale-file` sends them from a file, unchanged, stored the same way as a review note: ```console $ tokentriage cloud comparison decide --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --comparison "$CMP" --run "$RUN" \ --claimed-outcome meets_bar --action change_route --arm 2 \ --rationale-file rationale.txt ``` **A review recorded after a release is created does not stop the deploy.** The digest a release names is checked once, at creation: `releases.create` is refused with `decision_evidence_changed` when a decision's verdicts have since been revised, but nothing re-checks the digest at deploy time. Create the release and deploy it soon after you record the decision it cites, while its evidence is still what you reviewed. ### Apply the saved candidate deliberately After recording `change_route`, create a release from the exact saved draft revision that the candidate arm tested. The server reads the document, current controls and connection bindings. Supply the comparison, run and decision IDs so creation checks that the resulting configuration matches the accepted evidence. If the run's evidence has moved since this decision, the portal shows "Evidence changed since this decision" instead of the adoption button; record a new decision revision on the current evidence before it will offer one again. ```console tokentriage cloud release create --org "$ORG" --workspace "$WS" --app "$APP" \ --gateway "$GW" --env "$ENV" --draft-revision 2 \ --comparison "$CMP" --run "$RUN" --decision "$DECISION" \ --base-url "$BASE_URL" --json ``` Creation leaves traffic unchanged. Read `cloud release head` with the same scope, retain its applied release and revision, then use `cloud release deploy --release "$NEW_RELEASE" --expected-revision "$HEAD_REVISION"` with that scope. Wait until the head reports the new release applied before checking fresh attributed traffic. Rollback is a separate `cloud release rollback --release "$PRIOR_RELEASE" --expected-revision "$CURRENT_HEAD_REVISION"` operation; read the head again first. Protected-environment and fresh-provider-test requirements still apply. A stale comparison or a changed head is a result to investigate, not a reason to retry an old request automatically. TypeScript accepts `releases.create(org, ws, app, gw, env, { draft_revision: 2, comparison: { comparison_id, run_id, decision_id } })`. Python accepts the same selector as `releases.create(org, ws, app, gw, env, draft_revision=2, comparison={"comparison_id": cmp, "run_id": run, "decision_id": decision})`. Do not combine `draft_revision` with direct revision pins or connection bindings. A create response lost in transit must be reconciled by listing releases before trying another create. ## From either SDK These readers use a management API key and the retained comparison/run IDs. They make no provider call, create no run and record no review. The repository examples `sdk/ts/examples/read_comparison.ts` and `sdk/python/examples/read_comparison.py` exercise the same reads and preserve absent cost as `null`/`None`. ```ts title="TypeScript" const scope = { org, ws, app, gw, env }; const status = (await client.comparisons.run(scope, cmp, runId)).data; if (!status) throw new Error("comparison status was absent"); if (["complete", "partial", "abandoned"].includes(String(status.state))) { const answer = (await client.comparisons.results(scope, cmp, runId)).data; if (!answer) throw new Error("comparison evidence was absent"); console.log(answer.outcome, answer.unknown_count, answer.gaps); console.log(answer.cost_state, answer.cost_reason ?? null, answer.cost ?? null); } else { console.log(status.state, status.observed_spend ?? null); } ``` ```python title="Python" scope = ComparisonScope(org=org, ws=ws, app=app, gw=gw, env=env) status = client.comparisons.run(scope, cmp, run_id)["data"] if status["state"] in ("complete", "partial", "abandoned"): answer = client.comparisons.results(scope, cmp, run_id)["data"] print(answer["outcome"], answer["unknown_count"], answer.get("gaps", [])) print(answer["cost_state"], answer.get("cost_reason"), answer.get("cost")) else: print(status["state"], status.get("observed_spend")) ``` Local verification runs these actual SDK programs and the CLI against the production customer handler, management-key authentication and a real PostgreSQL comparison/run ledger. A queued run canceled before dispatch remains two unknown cases with no invented pass or price. This verifies the readers and the permission restriction; it does not establish paid comparison quality or human acceptance. ## Repeat a comparison from a dataset {/* Provenance of this section's captures: go test ./internal/cli/ -run TestComparisonDefinitionDocsTranscripts -update-comparison-definition-transcripts then paste the named block from internal/cli/testdata/comparison-definition-transcripts.txt (=== definition-create, === definition-create-ineligible, === run-list). The command lines are the block's command re-wrapped with backslashes; the output is verbatim. scorer.json and rule.json are the files that test writes. The run-preflight, run-start and run-start-explicit blocks aren't shown: the loopback server doesn't apply the management-key rule, and a deployment answers all three verbs app_key_cannot_select_subject for any management key (internal/control/foundation/httpapi/comparison_definitions.go). The contract is docs/capabilities/quality-evaluation/q03-comparison-definitions.md. */} If you compare the same examples each time the candidate changes, save the setup once as a comparison definition. It pins one version of a [hosted dataset](/docs/build/curate-evaluation-datasets), the release the environment serves now, the candidate draft revision, the scorer, the rule and a spend ceiling. Each start is an ordinary comparison run on the definition's comparison, so you follow it and read its evidence with the steps above. [Comparison definitions](/docs/reference/managed-evaluations#comparison-definitions) lists every field, bound and answer. Only saved test cases can be compared. Every example of the pinned dataset version must be `saved_case`, because a saved test case is a request the gateway already holds and each arm can send again. An example you typed into the dataset (`authored`) has no such request. Save it as a test case first, as in [Choose scratch or keep, before anything runs](/docs/build/test-and-share-gateway-runs#choose-scratch-or-keep-before-anything-runs), then publish a dataset version that refers to it as `saved_case`. | You need | For | |---|---| | a dataset version of 2 to 50 `saved_case` examples; for a labeled definition, each with an `expected_label` of at most 120 bytes | saving the definition | | `comparisons.run`, `tests.run` and `metadata.read` in the gateway environment | saving it, planning a run and starting one | | `content.read`, a current `comparison` grant on each arm's connection, and a signed-in session | planning a run and starting one | ### Save the definition The scorer and the rule are the ones a comparison plan takes. Here they sit in two files. `scorer.json`: ```json {"scorer_id":"label_exact_v1","scorer_version":1,"rubric_digest":"sha256:1f2e3d4c5b6a7980f1e2d3c4b5a69788f1e2d3c4b5a69788f1e2d3c4b5a69788"} ``` `rule.json`: ```json {"kind":"labeled","at_least_baseline_pass_count":true,"no_failure_on_critical_cases":true,"max_unknown_cases":0} ``` Then create the definition. It publishes the selection as the sample set of a new comparison, and it calls no provider. ```console $ tokentriage cloud comparison definition create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --name "A2 cheaper route" --kind labeled \ --dataset ds_ABCDEFGHJKLM --dataset-version 4 \ --dataset-digest sha256:8b9c0d1e2f3a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4 \ --baseline-release rel_7f3k9qp2mxaz --candidate-draft-revision 7 \ --scorer-file scorer.json --rule-file rule.json --critical refund-2 \ --ceiling-micro-usd 2000000 --idempotency-key q-a2-def-1 \ --base-url "$TOKENTRIAGE_BASE_URL" definition cdef_7f3k9qp2mxaz name A2 cheaper route kind labeled state active comparison cmp_7f3k9qp2mxaz version 1 definition_digest sha256:5d2f0c1b6a7e8d9c0b1a2f3e4d5c6b7a8f9e0d1c2b3a4f5e6d7c8b9a0f1e2d3c dataset ds_ABCDEFGHJKLM v4 sha256:8b9c0d1e2f3a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4 sample_set v1 sha256:7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d9e8f7a6b (2 cases) baseline applied_release rel_7f3k9qp2mxaz candidate saved_draft_revision 7 scorer label_exact_v1 v1 rule labeled ceiling 2000000 micro-USD ORDINAL ITEM_ID CASE CASE_VERSION CRITICAL 1 refund-2 tcase_7f3k9qp2mxaz 1 true 2 vpn-1 tcase_8g4m2rq3nyb2 1 false ``` **How to tell it worked.** `state` is `active` at `version` 1, and `sample_set` holds 2 cases, one per example. Each row names the saved test case version the example points at, in `item_id` order, and `refund-2` is critical because `--critical` named it. Keep the `definition` ID. Sending the same command with the same key answers this definition again rather than creating a second one. The IDs and digests in these captures are fixture values from a local loopback server, whose dataset holds two saved test cases at version 4. Versions 1 and 2 of this same `ds_ABCDEFGHJKLM`, as published in the datasets tutorial, hold `authored` examples, so either would be turned away until you publish a version of saved cases, as version 4 does here.
At least one example of the pinned version can't be compared, and nothing was saved. The answer names each example and the reason. Here version 3 of the dataset holds an `authored` example: ```console $ tokentriage cloud comparison definition create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --name "A2 cheaper route" --kind labeled \ --dataset ds_ABCDEFGHJKLM --dataset-version 3 \ --dataset-digest sha256:0c1d2e3f405162738495a6b7c8d9eafb0c1d2e3f405162738495a6b7c8d9eafb \ --baseline-release rel_7f3k9qp2mxaz --candidate-draft-revision 7 \ --scorer-file scorer.json --rule-file rule.json --critical refund-2 \ --ceiling-micro-usd 2000000 --idempotency-key q-a2-def-2 \ --base-url "$TOKENTRIAGE_BASE_URL" Error: customer.comparison_definitions.create was refused (HTTP 422); detail: faq-3.origin: authored_example; save it as a test case and add it as saved_case; type=comparison_definition_ineligible; request_id=req_0000000000000001 ``` Save `faq-3` as a test case, publish a dataset version that refers to it as `saved_case`, and create the definition against that version with a new idempotency key. The other reasons, such as a missing or over-long expected label, are listed under [Comparison definition errors](/docs/reference/managed-evaluations#comparison-definition-errors).
### Plan the run, approve the plan, start it The preflight writes nothing and calls no provider. It answers the plan for one definition version: 2N arm dispatches for N cases, 2N judge calls for a judged definition or 0 for a labeled one, the ceiling with its arm and judge halves, an `authority_epoch` and a `plan_digest` that names this plan. The start carries three values back, and each one matters: - **`approved_plan_digest`** is the preflight's `plan_digest`. The server plans again before it commits. If the baseline, the sample set, the scorer, the ceiling or a grant moved since your preflight, or a pinned judge profile is no longer current, nothing starts and the answer is `comparison_definition_plan_stale`. Ask for a new plan and approve what it says now. - **`approved_epoch`** is the preflight's `authority_epoch`. - **`idempotency_key`** is yours, and you choose a new one for every preflight you approve. The same key with the same approval answers the same run, so a retry after a lost response doesn't pay twice. A key you already sent can't carry a different approval: that's `idempotency_payload_mismatch`. Both calls need a signed-in session. A management key, which the CLI and both SDKs send, is answered `app_key_cannot_select_subject` (HTTP 403) on the preflight and on the start, because a definition's candidate is a saved draft revision. That's the rule described in [Read the plan before you spend](#read-the-plan-before-you-spend), and it applies to every definition. The managed deployment doesn't serve these operations yet; [Availability](/docs/reference/managed-evaluations#availability) says what changes that. ### Find the runs it started Listing a definition's runs needs only `comparisons.run`, so a management key can do it. The newest run comes first: ```console $ tokentriage cloud comparison definition run list --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --definition cdef_7f3k9qp2mxaz \ --base-url "$TOKENTRIAGE_BASE_URL" RUN COMPARISON VERSION IDEMPOTENCY_KEY STARTED_BY STARTED_AT crun_7f3k9qp2mxaz cmp_7f3k9qp2mxaz 1 q-a2-start-1 ada 2026-09-24T12:01:00Z ``` Each row links a run to the definition version and the idempotency key that started it. Pass the `COMPARISON` and `RUN` values to `cloud comparison run get` to follow progress, then read the evidence and record a decision as in [Read the evidence](#read-the-evidence). When the candidate or the dataset moves on, publish a new definition version with `cloud comparison definition version create`. It takes the same selection flags without `--name` and `--kind`, plus `--expected-version` and a new `--idempotency-key`. Then plan again. ## What this does not establish Twenty to fifty examples is a bounded demonstration, not statistical sufficiency. A judge supplies evidence, not proof. A sample serving-cost reduction is a reduction over the cases where both sides were priced, not a monthly saving. What the comparison shows is that on these examples, under this bar, the candidate did or did not hold up.