Compare cost and quality before you switch models

A cheaper model is a guess until you have run your own examples through both configurations and looked at what came back. This page is how you do that against a hosted gateway: pick the saved tests you already trust, pin the configuration you are serving today against the one you are considering, and get back one sentence about quality with the numbers under it, and the cost of each side kept apart.

You will get more from this page if the gateway already has saved test cases and you have sent traffic through it: see Read your gateway’s traffic and spend.

What a comparison is

Two arms, your cases, one scorer.

• The baseline is what the environment is serving now. The candidate is a saved draft revision. Both are pinned by the server at their effective digest, so what ran is checkable afterwards.
• The sample is between two and fifty of the gateway’s saved test case versions. You mark the ones where a failure is disqualifying whatever the counts say.
• The scorer is labeled (each answer is checked against the expected label on the case) or judged (one model from a different family grades both answers against the case’s reference notes). The kind is fixed when you create the comparison.
• The bar is declared before the run: how many cases must pass, whether the candidate must match the baseline, how many ungraded cases you will tolerate.

An owner or administrator does not automatically have content.read. Starting a comparison requires an explicit content grant for its saved inputs, along with the comparison and test permissions. Each arm connection also needs a current comparison grant for this application and environment; an application grant alone is insufficient. Reading cost and metadata does not grant access to prompts, reference notes or answers.

The JSON response excerpts below are synthetic examples of the response shape. They are not a production acceptance result or a bill.

Create the comparison

One name, one kind. It reaches no provider and costs nothing.

$ tokentriage cloud comparison create --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \
    --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \
    --name "cheaper checkout" --kind judged --json
{
  "schema": "customer/v1",
  "request_id": "req_0000000000000000",
  "kind": "comparison",
  "data": {
    "comparison_id": "cmp_ABCDEFGHJKLM",
    "gateway_id": "gw_ABCDEFGHJKLM",
    "environment_id": "env_ABCDEFGHJKLM",
    "name": "cheaper checkout",
    "kind": "judged",
    "current_sample_version": 3,
    "created_by": "ada",
    "created_at": "2026-09-22T12:00:00Z",
    "revision": 1
  }
}

Then publish the examples. A judged comparison needs reference notes on every case; a labeled one needs an expected label. Publishing the same selection twice returns the version that already holds it, so a retry does not fork your evidence.

$ tokentriage cloud comparison samples publish --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --comparison cmp_ABCDEFGHJKLM \
    --case tcase_ABCDEFGHJKLM:1:critical:notes=cobj_ABCDEFGHJKLM \
    --case tcase_BCDEFGHJKLMN:2:notes=cobj_BCDEFGHJKLMN

Read the plan before you spend

For a judged comparison, prepare four judge controls and check their separate cost plan first. That free provider-cost check pins the scorer for a new comparison plan. Running those controls makes four separate paid judge calls; merely preparing them does not establish calibrated results or human acceptance.

In the signed-in builder, select the currently applied baseline, saved candidate revision, published sample version, scorer and acceptance rule. Review preflight before starting. It resolves the subjects again and reports the exact dispatch count, current qualified maximum for each model, and the total allocation. The allocations can differ between arms; a cheap model does not give the expensive arm a larger limit. These are spend ceilings, not measured costs.

A missing content grant, unknown remaining allowance, unqualified or expired model bound, underfunded plan, same-family judge or missing judge grant prevents a valid plan. Twenty judged cases require 40 arm calls and 40 judge calls. A plan that cannot fit the configured ceiling must be revised or explicitly authorized through the operator’s supported policy process before it can start.

Management API keys cannot select a saved draft. The CLI and management-key SDKs preserve this restriction and return app_key_cannot_select_subject; use the signed-in owner interface for that comparison. Never substitute a runtime key or borrow another person’s session to get around the refusal.

For subjects your API credential may select, the CLI accepts the complete API plan:

 tokentriage cloud comparison run preflight --org "$ORG" --workspace "$WS" \
   --app "$APP" --gateway "$GW" --environment "$ENV" --comparison "$CMP" \
   --plan-file comparison-plan.json --json

The file must contain sample_set_version, explicit baseline and candidate selectors, the complete scorer, the complete acceptance rule, and a positive ceiling_micro_usd. A scorer includes its ID, version and rubric digest; a judged scorer also includes its pinned judge model, prompt digest, bounds and saved judge connection. Use the exact plan you reviewed. Unknown fields, duplicate JSON members and mixing --plan-file with individual plan flags are refused. The server remains the authority for subject selection, permissions, qualification and spend.

Start it, and follow it

Starting spends real provider money. Keep the plan unchanged, use the authority epoch returned by preflight, and choose one idempotency key for this attempt. A repeated start with that key and plan returns the same run. A different key buys a new attempt; it is not a recovery procedure for an uncertain result.

 tokentriage cloud comparison run start --org "$ORG" --workspace "$WS" \
   --app "$APP" --gateway "$GW" --environment "$ENV" --comparison "$CMP" \
   --plan-file comparison-plan.json --approved-epoch "$EPOCH" \
   --idempotency-key "$ATTEMPT_KEY" --json

The management-key subject restrictions above also apply to start. For a saved-draft comparison, start in the signed-in builder and use its returned run ID for CLI or SDK reads. Wait for a terminal state before requesting results. A partial or abandoned run retains its gaps and uncertain charges; it is not a successful run and should not be replayed automatically.

Read the evidence

$ tokentriage cloud comparison results --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \
    --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \
    --comparison cmp_ABCDEFGHJKLM --run crun_ABCDEFGHJKLM --json
{
  "kind": "comparison-results",
  "data": {
    "run": { "state": "partial", "terminal_state": "partial" },
    "outcome": "meets_bar",
    "sentence": "Meets your bar on these examples",
    "pass_count": 18,
    "case_count": 20,
    "baseline_pass_count": 18,
    "unknown_count": 1,
    "cost": {
      "per_arm": [
        { "arm_ordinal": 1, "priced_nano_usd": 412000, "priced_cases": 19, "pending_cases": 1,
          "unpriced_reasons": ["price_pending"] },
        { "arm_ordinal": 2, "priced_nano_usd": 126000, "priced_cases": 19, "pending_cases": 1,
          "unpriced_reasons": ["price_pending"] }
      ],
      "judge_spend_nano_usd": 84000,
      "total_run_spend_nano_usd": 622000,
      "comparable_population": { "count": 19, "total_cases": 20 },
      "sample_serving_cost_reduction": { "numerator_nano_usd": 286000, "denominator_nano_usd": 412000 },
      "reduction_state": "available",
      "coverage": "partial",
      "coverage_reason": "unpriced_population"
    },
    "cost_state": "available",
    "gaps": [
      { "ordinal": 7, "arm_ordinal": 2, "critical": false,
        "reason": "verdict_unknown", "detail": "judge_no_verdict" }
    ],
    "content_withheld": true
  }
}

Read it in this order:

1. The sentence, with its counts. 18 of 20, against the baseline’s 18. The denominator is every case you asked for, not the ones that answered, so a case that failed cheaply cannot lift the headline.
2. What is missing. One case is ungraded and gaps says which and why. An ungraded case is not a failure: the candidate was not tried and found wanting there, it was not graded at all.
3. The money, apart from the quality. Each arm’s priced spend, the judge’s spend, the total, and the population the comparison is over: 19 of 20, because one case is not priced yet. The reduction is a ratio of two integers over that population.
4. What you cannot see. content_withheld means prompts, completions, judge answers and notes were removed because you hold no content.read. Without cost.read the cost block is absent and cost_state reads unavailable with a named reason, rather than showing a zero.

Project only over a declared application cohort

To ask what a saved difference would imply for matching application traffic, add a UTC window to the results read. These parameters make no provider call and do not start or retry a comparison. You need a terminal run and access to the same gateway environment; a monetary projection also requires cost.read.

tokentriage cloud comparison results --org "$ORG" --workspace "$WS" \
  --app "$APP" --gateway "$GW" --environment "$ENV" \
  --comparison "$CMP" --run "$RUN" --cohort-from "$FROM" --cohort-to "$TO" \
  --cohort-release "$RELEASE" --json

Omit --cohort-release to include all releases in the declared window. To continue a previous insight read, pass its complete read_context JSON to --cohort-read-context. The server checks that context; an expired or unreadable context does not authorize a new population silently.

evidence = client.comparisons.results(scope, comparison, run, cohort={
    "from": start, "to": end, "release_id": release_id,
    "read_context": overview["read_context"],
})
print(evidence["data"].get("cohort_projection"))

const evidence = await client.comparisons.results(scope, comparison, run, {
  cohort: { from, to, release_id: releaseId, read_context: overview.read_context },
});
console.log(evidence.data?.cohort_projection);

The direct API uses the existing GET .../comparisons/{comparison}/runs/{run}/results with cohort_from, cohort_to, optional cohort_release_id and optional cohort_read_context query parameters. URL-encode the read context as JSON. These parameters describe application traffic only.

Read cohort_projection.state before its amount. An available result includes a signed projected_reduction_nano_usd, the covered and uncovered request counts, tested pairs, rule identity, evidence digest, price basis and assumptions. Negative amounts remain negative. An unavailable result carries a reason and omits the amount; uncovered traffic has no assumed saving. The server derives the projection from qualified paired deltas and matching request counts. The clients print that result without recomputing it.

The shipped sdk/ts/examples/read_comparison.ts and sdk/python/examples/read_comparison.py accept optional TOKENTRIAGE_FROM and TOKENTRIAGE_TO together, plus optional TOKENTRIAGE_RELEASE. They preserve cohort_projection alongside the saved sample evidence. Focused tests execute these examples and the CLI against local contract responses, including negative and unavailable results. That verification is not a live cohort measurement. A deployment without this producer can still return saved sample results without a cohort projection.

$ tokentriage cloud comparison review --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \
    --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \
    --comparison cmp_ABCDEFGHJKLM --run crun_ABCDEFGHJKLM \
    --case tcase_ABCDEFGHJKLM --case-version 1 --arm 2 --outcome fail

Record what you decided

The outcome is worked out from the verdicts and the bar you set; the action is yours. The outcome you claim is checked against the computed one, and a disagreement is decision_outcome_mismatch.

$ tokentriage cloud comparison decide --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \
    --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \
    --comparison cmp_ABCDEFGHJKLM --run crun_ABCDEFGHJKLM \
    --claimed-outcome meets_bar --action change_route --arm 2 --json
{
  "kind": "comparison-decision",
  "data": {
    "decision": {
      "decision_id": "cdec_ABCDEFGHJKLM",
      "outcome": "meets_bar",
      "action": "change_route",
      "evidence_digest": "sha256:5e6f...",
      "decided_by": "ada",
      "revision": 1,
      "decided_at": "2026-09-22T12:30:00Z"
    }
  }
}

Recording a decision deploys nothing. It is the evidence a release cites, and the release path checks again that the comparison still describes what you are about to ship. If the setup moved you get comparison_evidence_stale; if the configuration you measured against is no longer the one being served you get baseline_release_changed. Two words, because the fixes differ: re-run against the current setup, or re-run against what is live now.

change_route also needs permission to deploy and an outcome of meets_bar. keep_route needs neither and touches no release.

Review the evidence yourself

Open a run’s review panel from its comparison page in the portal. It lists every cell, filtered by Needs attention (the default), Not reviewed by a person, Reviewed, or All. There, a person reads the source and both arms’ outputs, then records a verdict with a note.

The verdict carries the digests of what was on screen for that reading. The panel labels a verdict backed by those digests attested, not “verified”: the server checks that a digest names the stored bytes, not that the sentence in the note is correct. Without a content grant the panel withholds the source and the outputs, and the verdict it records reads not attested.

The CLI and both SDKs carry the same attestation. --read-source-sha256 and --read-output-sha256 send the digest of the source and, for a cell that produced one, the output the reviewer read; a digest that names bytes other than the cell’s is review_reading_mismatch. Omit both flags to record a verdict as it was recorded before this attestation existed, and it reads not attested.

$ tokentriage cloud comparison review --org "$ORG" --workspace "$WS" \
    --app "$APP" --gateway "$GW" --environment "$ENV" \
    --comparison "$CMP" --run "$RUN" --case "$CASE" --case-version 1 --arm 2 \
    --outcome fail --note-file note.txt \
    --read-source-sha256 "$SOURCE_SHA256" --read-output-sha256 "$OUTPUT_SHA256"

A decision also carries the person’s reasons. --rationale-file sends them from a file, unchanged, stored the same way as a review note:

$ tokentriage cloud comparison decide --org "$ORG" --workspace "$WS" \
    --app "$APP" --gateway "$GW" --environment "$ENV" \
    --comparison "$CMP" --run "$RUN" \
    --claimed-outcome meets_bar --action change_route --arm 2 \
    --rationale-file rationale.txt

A review recorded after a release is created does not stop the deploy. The digest a release names is checked once, at creation: releases.create is refused with decision_evidence_changed when a decision’s verdicts have since been revised, but nothing re-checks the digest at deploy time. Create the release and deploy it soon after you record the decision it cites, while its evidence is still what you reviewed.

Apply the saved candidate deliberately

After recording change_route, create a release from the exact saved draft revision that the candidate arm tested. The server reads the document, current controls and connection bindings. Supply the comparison, run and decision IDs so creation checks that the resulting configuration matches the accepted evidence. If the run’s evidence has moved since this decision, the portal shows “Evidence changed since this decision” instead of the adoption button; record a new decision revision on the current evidence before it will offer one again.

tokentriage cloud release create --org "$ORG" --workspace "$WS" --app "$APP" \
  --gateway "$GW" --env "$ENV" --draft-revision 2 \
  --comparison "$CMP" --run "$RUN" --decision "$DECISION" \
  --base-url "$BASE_URL" --json

Creation leaves traffic unchanged. Read cloud release head with the same scope, retain its applied release and revision, then use cloud release deploy --release "$NEW_RELEASE" --expected-revision "$HEAD_REVISION" with that scope. Wait until the head reports the new release applied before checking fresh attributed traffic. Rollback is a separate cloud release rollback --release "$PRIOR_RELEASE" --expected-revision "$CURRENT_HEAD_REVISION" operation; read the head again first. Protected-environment and fresh-provider-test requirements still apply. A stale comparison or a changed head is a result to investigate, not a reason to retry an old request automatically.

TypeScript accepts releases.create(org, ws, app, gw, env, { draft_revision: 2, comparison: { comparison_id, run_id, decision_id } }). Python accepts the same selector as releases.create(org, ws, app, gw, env, draft_revision=2, comparison={"comparison_id": cmp, "run_id": run, "decision_id": decision}). Do not combine draft_revision with direct revision pins or connection bindings. A create response lost in transit must be reconciled by listing releases before trying another create.

From either SDK

These readers use a management API key and the retained comparison/run IDs. They make no provider call, create no run and record no review. The repository examples sdk/ts/examples/read_comparison.ts and sdk/python/examples/read_comparison.py exercise the same reads and preserve absent cost as null/None.

Local verification runs these actual SDK programs and the CLI against the production customer handler, management-key authentication and a real PostgreSQL comparison/run ledger. A queued run canceled before dispatch remains two unknown cases with no invented pass or price. This verifies the readers and the permission restriction; it does not establish paid comparison quality or human acceptance.

Repeat a comparison from a dataset

If you compare the same examples each time the candidate changes, save the setup once as a comparison definition. It pins one version of a hosted dataset, the release the environment serves now, the candidate draft revision, the scorer, the rule and a spend ceiling. Each start is an ordinary comparison run on the definition’s comparison, so you follow it and read its evidence with the steps above. Comparison definitions lists every field, bound and answer.

Only saved test cases can be compared. Every example of the pinned dataset version must be saved_case, because a saved test case is a request the gateway already holds and each arm can send again. An example you typed into the dataset (authored) has no such request. Save it as a test case first, as in Choose scratch or keep, before anything runs, then publish a dataset version that refers to it as saved_case.

Save the definition

The scorer and the rule are the ones a comparison plan takes. Here they sit in two files. scorer.json:

{"scorer_id":"label_exact_v1","scorer_version":1,"rubric_digest":"sha256:1f2e3d4c5b6a7980f1e2d3c4b5a69788f1e2d3c4b5a69788f1e2d3c4b5a69788"}

rule.json:

{"kind":"labeled","at_least_baseline_pass_count":true,"no_failure_on_critical_cases":true,"max_unknown_cases":0}

Then create the definition. It publishes the selection as the sample set of a new comparison, and it calls no provider.

$ tokentriage cloud comparison definition create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --name "A2 cheaper route" --kind labeled \
    --dataset ds_ABCDEFGHJKLM --dataset-version 4 \
    --dataset-digest sha256:8b9c0d1e2f3a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4 \
    --baseline-release rel_7f3k9qp2mxaz --candidate-draft-revision 7 \
    --scorer-file scorer.json --rule-file rule.json --critical refund-2 \
    --ceiling-micro-usd 2000000 --idempotency-key q-a2-def-1 \
    --base-url "$TOKENTRIAGE_BASE_URL"
definition         cdef_7f3k9qp2mxaz
name               A2 cheaper route
kind               labeled
state              active
comparison         cmp_7f3k9qp2mxaz
version            1
definition_digest  sha256:5d2f0c1b6a7e8d9c0b1a2f3e4d5c6b7a8f9e0d1c2b3a4f5e6d7c8b9a0f1e2d3c
dataset            ds_ABCDEFGHJKLM v4 sha256:8b9c0d1e2f3a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4
sample_set         v1 sha256:7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d9e8f7a6b (2 cases)
baseline           applied_release rel_7f3k9qp2mxaz
candidate          saved_draft_revision 7
scorer             label_exact_v1 v1
rule               labeled
ceiling            2000000 micro-USD

ORDINAL  ITEM_ID   CASE                CASE_VERSION  CRITICAL
1        refund-2  tcase_7f3k9qp2mxaz  1             true
2        vpn-1     tcase_8g4m2rq3nyb2  1             false

How to tell it worked. state is active at version 1, and sample_set holds 2 cases, one per example. Each row names the saved test case version the example points at, in item_id order, and refund-2 is critical because --critical named it. Keep the definition ID. Sending the same command with the same key answers this definition again rather than creating a second one.

The IDs and digests in these captures are fixture values from a local loopback server, whose dataset holds two saved test cases at version 4. Versions 1 and 2 of this same ds_ABCDEFGHJKLM, as published in the datasets tutorial, hold authored examples, so either would be turned away until you publish a version of saved cases, as version 4 does here.

$ tokentriage cloud comparison definition create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --name "A2 cheaper route" --kind labeled \
    --dataset ds_ABCDEFGHJKLM --dataset-version 3 \
    --dataset-digest sha256:0c1d2e3f405162738495a6b7c8d9eafb0c1d2e3f405162738495a6b7c8d9eafb \
    --baseline-release rel_7f3k9qp2mxaz --candidate-draft-revision 7 \
    --scorer-file scorer.json --rule-file rule.json --critical refund-2 \
    --ceiling-micro-usd 2000000 --idempotency-key q-a2-def-2 \
    --base-url "$TOKENTRIAGE_BASE_URL"
Error: customer.comparison_definitions.create was refused (HTTP 422); detail: faq-3.origin: authored_example; save it as a test case and add it as saved_case; type=comparison_definition_ineligible; request_id=req_0000000000000001

Plan the run, approve the plan, start it

The preflight writes nothing and calls no provider. It answers the plan for one definition version: 2N arm dispatches for N cases, 2N judge calls for a judged definition or 0 for a labeled one, the ceiling with its arm and judge halves, an authority_epoch and a plan_digest that names this plan.

The start carries three values back, and each one matters:

• approved_plan_digest is the preflight’s plan_digest. The server plans again before it commits. If the baseline, the sample set, the scorer, the ceiling or a grant moved since your preflight, or a pinned judge profile is no longer current, nothing starts and the answer is comparison_definition_plan_stale. Ask for a new plan and approve what it says now.
• approved_epoch is the preflight’s authority_epoch.
• idempotency_key is yours, and you choose a new one for every preflight you approve. The same key with the same approval answers the same run, so a retry after a lost response doesn’t pay twice. A key you already sent can’t carry a different approval: that’s idempotency_payload_mismatch.

Both calls need a signed-in session. A management key, which the CLI and both SDKs send, is answered app_key_cannot_select_subject (HTTP 403) on the preflight and on the start, because a definition’s candidate is a saved draft revision. That’s the rule described in Read the plan before you spend, and it applies to every definition. The managed deployment doesn’t serve these operations yet; Availability says what changes that.

Find the runs it started

Listing a definition’s runs needs only comparisons.run, so a management key can do it. The newest run comes first:

$ tokentriage cloud comparison definition run list --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --definition cdef_7f3k9qp2mxaz \
    --base-url "$TOKENTRIAGE_BASE_URL"
RUN                COMPARISON        VERSION  IDEMPOTENCY_KEY  STARTED_BY  STARTED_AT
crun_7f3k9qp2mxaz  cmp_7f3k9qp2mxaz  1        q-a2-start-1     ada         2026-09-24T12:01:00Z

Each row links a run to the definition version and the idempotency key that started it. Pass the COMPARISON and RUN values to cloud comparison run get to follow progress, then read the evidence and record a decision as in Read the evidence. When the candidate or the dataset moves on, publish a new definition version with cloud comparison definition version create. It takes the same selection flags without --name and --kind, plus --expected-version and a new --idempotency-key. Then plan again.

What this does not establish

Twenty to fifty examples is a bounded demonstration, not statistical sufficiency. A judge supplies evidence, not proof. A sample serving-cost reduction is a reduction over the cases where both sides were priced, not a monthly saving. What the comparison shows is that on these examples, under this bar, the candidate did or did not hold up.