# Check a dataset's supplied outputs against its labels Score the outputs a dataset already holds against its expected labels, find the wrong one, and rerun on a new version, with no provider calls. Source: https://tokentriage.com/docs/build/run-offline-evaluation --- import Callout from '../../../components/Callout.astro'; import CodeTabs from '../../../components/CodeTabs.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; {/* Provenance of every transcript below, so a later editor regenerates rather than guesses: the CLI captures (definition create, run preflight, run start, the quota answer, run get and run items) — go test ./internal/cli/ -run TestEvaluationDocsTranscripts -update-evaluation-transcripts then paste the named block from internal/cli/testdata/evaluation-transcripts.txt (=== definition-create, === run-preflight, === run-start, === run-start-quota, === run-get, === run-items). The command lines on this page are the block's command re-wrapped with backslashes; the output is verbatim. The rerun step has no capture: the transcript suite doesn't drive definition version create. the Python and TypeScript calls — sdk/python/examples/quality_offline_evaluation.py and sdk/ts/examples/quality_offline_evaluation.ts, exercised by sdk/python/tests/test_customer_quality_offline_evaluation_example.py and sdk/ts/test/unit/customer_evaluations_example.test.ts against fake transports. Those programs set max_unscored 0; this page uses 1 in every tab so each channel publishes the spec the CLI capture shows. the operations, bounds, states, reasons and error words — docs/capabilities/quality-evaluation/q02-offline-evaluation.md and internal/control/foundation/evaluation/{types,refusals}.go The lane answering the CLI captures is a loopback test server that answers the contract's fixture shapes. It stores nothing but whether a run is active, calls no provider and spends nobody's money. Its digests are computed with the contract's rule over the dataset tutorial's version 2, so they reproduce. A later receipt replaces these captures with a run against a live deployment. */} A label mistake is cheapest to find before you pay a model to answer again. If your dataset already holds the outputs your application returned, you can score those outputs against the expected labels without sending a single request to a provider. In this tutorial we'll score version 2 of the dataset from [Curate an evaluation dataset your team can reopen](/docs/build/curate-evaluation-datasets), open the one example whose output carries the wrong label, see why the other wasn't scored, and then point the check at a newer dataset version. At the end you'll hold a definition ID, a run ID and a `result_digest`, and you'll know which example needs attention and why. | You need | You don't need | |---|---| | the dataset from the datasets tutorial, with its ID and version 2's `items_digest` | a provider connection or model credentials | | a management key bound at `role_version: 2` with `evaluations.write` and `evaluations.run` in that gateway environment, plus `metadata.read` and `content.read` there | a budget: the plan names 0 provider calls | | one of: `curl` and `jq`, the Python SDK, the TypeScript SDK, or the `tokentriage` CLI | any spend: nothing on this page calls a model | The path is four requests: define the check, plan it, start it, and read its items. [Evaluation actions](/docs/reference/managed-evaluations#evaluation-actions) explains the binding. The managed deployment doesn't serve these operations yet, so the captures on this page come from a local loopback server; [Availability](/docs/reference/managed-evaluations#availability) says what changes that. ## Set up once Every channel reads the same values from your shell. The IDs and the digest shown are the example values the captures on this page use; put yours in their place. ```console $ export TOKENTRIAGE_BASE_URL=https://app.example.com $ export TOKENTRIAGE_MANAGEMENT_KEY='' $ export ORG=org_ABCDEFGHJKLM WS=t_ABCDEFGHJKLM APP=app_ABCDEFGHJKLM $ export GW=gw_ABCDEFGHJKLM ENV=env_ABCDEFGHJKLM $ export DATASET=ds_ABCDEFGHJKLM $ export DS_DIGEST=sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 ``` `DS_DIGEST` is version 2's `items_digest`, the string the datasets tutorial had you write down. Then build the client for the channel you'll use. The CLI needs nothing more. ```console $ LANE="$TOKENTRIAGE_BASE_URL/portal/api/v1/customer/v1" $ SCOPE="organizations/$ORG/workspaces/$WS/applications/$APP" $ EVALS="$LANE/$SCOPE/gateways/$GW/environments/$ENV/evaluations" $ AUTH="Authorization: Bearer $TOKENTRIAGE_MANAGEMENT_KEY" ``` ```python import os, time from tokentriage.customer import CustomerClient, EvaluationScope client = CustomerClient(os.environ["TOKENTRIAGE_BASE_URL"], os.environ["TOKENTRIAGE_MANAGEMENT_KEY"]) scope = EvaluationScope(org=os.environ["ORG"], ws=os.environ["WS"], app=os.environ["APP"], gw=os.environ["GW"], env=os.environ["ENV"]) dataset = {"dataset_id": os.environ["DATASET"], "version": 2, "items_digest": os.environ["DS_DIGEST"]} ``` ```typescript import { CustomerClient } from "tokentriage/customer"; const env = (k: string) => process.env[k] ?? ""; const client = new CustomerClient({ baseUrl: env("TOKENTRIAGE_BASE_URL"), managementApiKey: env("TOKENTRIAGE_MANAGEMENT_KEY"), }); const scope = { org: env("ORG"), ws: env("WS"), app: env("APP"), gw: env("GW"), env: env("ENV") }; const dataset = { dataset_id: env("DATASET"), version: 2, items_digest: env("DS_DIGEST") }; ``` ## Define the check A definition says which dataset version to read, where the label sits in each supplied output, and what result you'll accept. Here the whole output is the label (`plain_text`), case matters, and you accept no mismatches and at most one example that can't be scored. The `idempotency_key` is yours: sending it again with the same definition returns this one instead of making a second. ```console $ REF=$(jq -n --arg ds "$DATASET" --arg digest "$DS_DIGEST" \ '{idempotency_key: "q-demo-definition-1", name: "Support exact labels", mode: "offline_exact_label", dataset: {dataset_id: $ds, version: 2, items_digest: $digest}, label_source: {kind: "plain_text"}, case_sensitive: true, acceptance: {max_mismatched: 0, max_unscored: 1}}' | curl -sS -X POST "$EVALS/definitions" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq -c '.data.version.ref') $ echo "$REF" ``` ```python created = client.evaluations.definitions_create( scope, idempotency_key="q-demo-definition-1", name="Support exact labels", dataset=dataset, label_source={"kind": "plain_text"}, case_sensitive=True, acceptance={"max_mismatched": 0, "max_unscored": 1}, )["data"] ref = created["version"]["ref"] print(ref["definition_id"], ref["version"], ref["spec_digest"]) ``` ```typescript const created = (await client.evaluations.definitionsCreate(scope, { idempotency_key: "q-demo-definition-1", name: "Support exact labels", mode: "offline_exact_label", dataset, label_source: { kind: "plain_text" }, case_sensitive: true, acceptance: { max_mismatched: 0, max_unscored: 1 }, })).data; const ref = created?.version?.ref; if (!ref) throw new Error("definition create returned no ref"); console.log(ref.definition_id, ref.version, ref.spec_digest); ``` ```console $ tokentriage cloud evaluation definition create --org "$ORG" \ --workspace "$WS" --app "$APP" --gateway "$GW" --environment "$ENV" \ --name "Support exact labels" --dataset "$DATASET" --dataset-version 2 \ --dataset-digest "$DS_DIGEST" --label-source plain_text \ --case-sensitive=true --max-mismatched 0 --max-unscored 1 \ --idempotency-key q-demo-definition-1 --base-url "$TOKENTRIAGE_BASE_URL" ``` Every channel publishes version 1 of the same definition. This is the CLI tab's capture; the HTTP body and the SDKs' `data` carry the same fields as JSON. ```console $ tokentriage cloud evaluation definition create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --name "Support exact labels" \ --dataset ds_ABCDEFGHJKLM --dataset-version 2 \ --dataset-digest sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 \ --label-source plain_text --case-sensitive=true --max-mismatched 0 \ --max-unscored 1 --idempotency-key q-demo-definition-1 \ --base-url "$TOKENTRIAGE_BASE_URL" definition edef_7f3k9qp2mxaz name Support exact labels state active version 1 spec_digest sha256:c04709d8d560854eb49c6dee1b062f23b1d468ed59fb3b0bf62b0307dbb61360 mode offline_exact_label dataset ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 label_source plain_text rule label_exact/v1 (case_sensitive true) acceptance max_mismatched 0, max_unscored 1 ``` The capture ran against a local loopback server, so `--base-url` pointed there and the line shows your variable instead. The CLI also writes a one-line request trace to standard error; it's left out of every capture here because its URL is the loopback's. **How to tell it worked.** `version` is `1` and the `dataset` line names version 2 with the digest you exported. The rule `label_exact/v1` trims white space from both ends of each label and then compares bytes. The definition's `ref` is its ID, its version and its `spec_digest`; every later step passes that `ref` along. ## Plan the run Before anything starts, ask for the plan. It reads the definition and the dataset, counts what would be scored, and returns a `digest` that names this exact plan. Nothing is written. ```console $ PLAN=$(jq -n --argjson ref "$REF" '{definition: $ref}' | curl -sS -X POST "$EVALS/runs/preflight" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq -c '.data') $ echo "$PLAN" | jq '{selected, expected_label_absent, planned_provider_calls, provider_spend, content_read, active_run_id, digest}' ``` ```python plan = client.evaluations.runs_preflight(scope, definition=ref)["data"] print(plan["selected"], plan["planned_provider_calls"], plan["provider_spend"], plan["digest"]) ``` ```typescript const plan = (await client.evaluations.runsPreflight(scope, { definition: ref })).data; if (!plan?.digest) throw new Error("preflight returned no digest"); console.log(plan.selected, plan.planned_provider_calls, plan.provider_spend, plan.digest); ``` ```console $ DEF=edef_7f3k9qp2mxaz # the definition ID from the previous step $ tokentriage cloud evaluation run preflight --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --definition "$DEF" --version 1 --base-url "$TOKENTRIAGE_BASE_URL" ``` ```console $ tokentriage cloud evaluation run preflight --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --definition edef_7f3k9qp2mxaz --version 1 \ --base-url "$TOKENTRIAGE_BASE_URL" definition edef_7f3k9qp2mxaz v1 dataset ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 selected 2 expected_label_absent 0 planned_provider_calls 0 provider_spend No provider calls (basis: no_dispatch) content_read true active_run_id - digest sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9 ``` **How to tell it worked.** `selected` is `2`, one per example, and `planned_provider_calls` is `0`. `provider_spend` reads `no_dispatch`: the run will read outputs the dataset already holds, so there is no request to pay for and no amount to show. `content_read` is `true` because your key can open the example bodies, which the run needs. `active_run_id` is empty, so nothing else is running here. Keep the `digest`. ## Start the run and wait for it The start carries the plan's digest as `approved_digest`. If the dataset or the definition moved since you read the plan, the start is turned away instead of scoring something you didn't look at. It also needs its own `idempotency_key`: a retry with the same key answers the same run. ```console $ RUN=$(jq -n --argjson ref "$REF" --arg d "$(echo "$PLAN" | jq -r .digest)" \ '{idempotency_key: "q-demo-run-1", definition: $ref, approved_digest: $d}' | curl -sS -X POST "$EVALS/runs" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq -r '.data.run_id') $ for i in $(seq 30); do state=$(curl -sS "$EVALS/runs/$RUN" -H "$AUTH" | jq -r .data.state) case "$state" in complete|partial|failed|cancelled) break ;; esac sleep 2 done $ curl -sS "$EVALS/runs/$RUN" -H "$AUTH" | jq '.data | {state, counts, unscored_reasons, acceptance, provider_spend, result_digest}' ``` ```python run = client.evaluations.runs_start( scope, idempotency_key="q-demo-run-1", definition=ref, approved_digest=plan["digest"], )["data"] run_id = run["run_id"] terminal = {"complete", "partial", "failed", "cancelled"} for _ in range(30): if run["state"] in terminal: break time.sleep(1) run = client.evaluations.runs_get(scope, run_id)["data"] print(run["state"], run["counts"], run["acceptance"]["outcome"], run["provider_spend"]) ``` ```typescript let run = (await client.evaluations.runsStart(scope, { idempotency_key: "q-demo-run-1", definition: ref, approved_digest: plan.digest, })).data; const runId = run?.run_id ?? ""; const terminal = new Set(["complete", "partial", "failed", "cancelled"]); for (let i = 0; !terminal.has(run?.state ?? "") && i < 30; i++) { await new Promise((r) => setTimeout(r, 1000)); run = (await client.evaluations.runsGet(scope, runId)).data; } console.log(run?.state, run?.counts, run?.acceptance?.outcome, run?.provider_spend); ``` ```console $ tokentriage cloud evaluation run start --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --definition "$DEF" --version 1 \ --approved-digest sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9 \ --idempotency-key q-demo-run-1 --base-url "$TOKENTRIAGE_BASE_URL" $ RUN=erun_7f3k9qp2mxaz # the run ID the start printed $ tokentriage cloud evaluation run wait --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --run "$RUN" --base-url "$TOKENTRIAGE_BASE_URL" ``` The start answers at once with the run `queued` and every example not yet evaluated: ```console $ tokentriage cloud evaluation run start --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --definition edef_7f3k9qp2mxaz --version 1 \ --approved-digest sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9 \ --idempotency-key q-demo-run-1 --base-url "$TOKENTRIAGE_BASE_URL" run erun_7f3k9qp2mxaz state queued failure - definition edef_7f3k9qp2mxaz v1 dataset ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 acceptance undetermined (max_mismatched 0, max_unscored 1) provider_spend No provider calls (basis: no_dispatch) result_digest - SELECTED SCORED MATCHED MISMATCHED UNSCORED NOT_EVALUATED 2 0 0 0 0 2 ``` `run wait` reads the run every two seconds and prints it once it finishes, in the same form `run get` uses. This is that read, after the run finished: ```console $ tokentriage cloud evaluation run get --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --run erun_7f3k9qp2mxaz \ --base-url "$TOKENTRIAGE_BASE_URL" run erun_7f3k9qp2mxaz state partial failure - definition edef_7f3k9qp2mxaz v1 dataset ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 acceptance not_met (max_mismatched 0, max_unscored 1) provider_spend No provider calls (basis: no_dispatch) result_digest sha256:7b8d3f8cc2a649018c048b8d6cecaff5f4e1e9673bfc09db6aeefe350b1b354e SELECTED SCORED MATCHED MISMATCHED UNSCORED NOT_EVALUATED 2 1 0 1 1 0 UNSCORED_REASON ITEMS output_absent 1 ``` **How to tell it worked.** The counts add up: 2 selected is 0 matched, 1 mismatched, 1 unscored and 0 not evaluated. `result_digest` is filled in, which happens only when the run finishes. Write it down; the last step checks it again. `partial` and `not_met` are the check doing its job, not a fault. `partial` means every example was recorded and at least one couldn't be scored. `not_met` means the result broke your acceptance bar: one mismatch against a limit of zero. The one unscored example is within the limit of one you set. If the wait runs out first, the run carries on. Read it again with the same run ID rather than starting another; a second start would be turned away while the first is still going. ## Open the mismatched and the unscored examples The run's items say, example by example, what was expected, what the supplied output said, and why an example wasn't scored. ```console $ curl -sS "$EVALS/runs/$RUN/items" -H "$AUTH" | jq '.data[] | {item_id, outcome, reason, expected_label, observed_label, content_state}' ``` ```python for item in client.evaluations.items_list(scope, run_id)["data"]: print(item["item_id"], item["outcome"], item.get("reason"), item.get("expected_label"), item.get("observed_label"), item["content_state"]) ``` ```typescript const items = (await client.evaluations.itemsList(scope, runId)).data ?? []; for (const item of items) console.log(item.item_id, item.outcome, item.reason ?? "-", item.expected_label, item.observed_label ?? "-", item.content_state); ``` ```console $ tokentriage cloud evaluation run items --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --run "$RUN" --base-url "$TOKENTRIAGE_BASE_URL" ``` ```console $ tokentriage cloud evaluation run items --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --run erun_7f3k9qp2mxaz \ --base-url "$TOKENTRIAGE_BASE_URL" ITEM_ID OUTCOME REASON EXPECTED_LABEL OBSERVED_LABEL CONTENT_STATE refund-2 unscored output_absent billing - available vpn-1 mismatched - network billing available ``` **How to tell it worked.** Two rows, one per example, in `item_id` order. `vpn-1` is the wrong-label row. The dataset expects `network`, the label you corrected in version 2, and the output supplied with the example still says `billing`. The check has found an answer that disagrees with the corrected expectation, which is the thing to investigate. `refund-2` wasn't scored, and the reason says why: `output_absent`. The dataset holds no supplied output for it, so there's nothing to compare. It counts as unscored, not as a mismatch or a zero. To list only one kind of row, add `--outcome mismatched` (or `unscored`), the `outcome` argument in Python and TypeScript, or `?outcome=mismatched` on the HTTP path. `observed_label` is read from the example body each time you list the items, under your own `content.read`. The run doesn't keep a copy. A teammate without that grant sees the same outcomes and expected labels, with `content_state` `withheld` and no observed label. ## Rerun on a new dataset version A run is fixed to the definition version it started from, and a definition version is fixed to one dataset version. So when the dataset changes, you publish its next version (as the datasets tutorial shows), then publish the definition's next version pinned to it, and start a new run. The first run and its `result_digest` stay as they are. Put the new dataset version's number and `items_digest` in two variables. They come from the answer that published it. ```console $ export DS_VERSION=3 # the version number you published $ export DS_DIGEST_NEW='' ``` ```console $ DEF=$(echo "$REF" | jq -r .definition_id) $ REF2=$(jq -n --arg ds "$DATASET" --argjson v "$DS_VERSION" \ --arg digest "$DS_DIGEST_NEW" \ '{idempotency_key: "q-demo-definition-1-v2", expected_version: 1, mode: "offline_exact_label", dataset: {dataset_id: $ds, version: $v, items_digest: $digest}, label_source: {kind: "plain_text"}, case_sensitive: true, acceptance: {max_mismatched: 0, max_unscored: 1}}' | curl -sS -X POST "$EVALS/definitions/$DEF/versions" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq -c '.data.version.ref') $ PLAN2=$(jq -n --argjson ref "$REF2" '{definition: $ref}' | curl -sS -X POST "$EVALS/runs/preflight" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq -c '.data') $ jq -n --argjson ref "$REF2" --arg d "$(echo "$PLAN2" | jq -r .digest)" \ '{idempotency_key: "q-demo-run-2", definition: $ref, approved_digest: $d}' | curl -sS -X POST "$EVALS/runs" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq '.data | {run_id, state, definition, dataset}' ``` ```python dataset_new = {"dataset_id": os.environ["DATASET"], "version": int(os.environ["DS_VERSION"]), "items_digest": os.environ["DS_DIGEST_NEW"]} v2 = client.evaluations.definition_versions_create( scope, ref["definition_id"], idempotency_key="q-demo-definition-1-v2", expected_version=1, dataset=dataset_new, label_source={"kind": "plain_text"}, case_sensitive=True, acceptance={"max_mismatched": 0, "max_unscored": 1}, )["data"] ref2 = v2["version"]["ref"] plan2 = client.evaluations.runs_preflight(scope, definition=ref2)["data"] run2 = client.evaluations.runs_start( scope, idempotency_key="q-demo-run-2", definition=ref2, approved_digest=plan2["digest"], )["data"] print(run2["run_id"], run2["state"]) ``` ```typescript const datasetNew = { dataset_id: env("DATASET"), version: Number(env("DS_VERSION")), items_digest: env("DS_DIGEST_NEW") }; const next = (await client.evaluations.definitionVersionsCreate( scope, ref.definition_id, { idempotency_key: "q-demo-definition-1-v2", expected_version: 1, mode: "offline_exact_label", dataset: datasetNew, label_source: { kind: "plain_text" }, case_sensitive: true, acceptance: { max_mismatched: 0, max_unscored: 1 }, })).data; const ref2 = next?.version?.ref; if (!ref2) throw new Error("definition version create returned no ref"); const plan2 = (await client.evaluations.runsPreflight(scope, { definition: ref2 })).data; const run2 = (await client.evaluations.runsStart(scope, { idempotency_key: "q-demo-run-2", definition: ref2, approved_digest: plan2?.digest ?? "", })).data; console.log(run2?.run_id, run2?.state); ``` ```console $ tokentriage cloud evaluation definition version create --org "$ORG" \ --workspace "$WS" --app "$APP" --gateway "$GW" --environment "$ENV" \ --definition "$DEF" --expected-version 1 --dataset "$DATASET" \ --dataset-version "$DS_VERSION" --dataset-digest "$DS_DIGEST_NEW" \ --label-source plain_text --case-sensitive=true --max-mismatched 0 \ --max-unscored 1 --idempotency-key q-demo-definition-1-v2 \ --base-url "$TOKENTRIAGE_BASE_URL" $ tokentriage cloud evaluation run start --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --definition "$DEF" --version 2 --approve-current-plan \ --idempotency-key q-demo-run-2 --base-url "$TOKENTRIAGE_BASE_URL" ``` This step has no capture on this page. Its answers have the same shape as the earlier steps, with the new values in them. `--approve-current-plan` asks for the plan and approves it in one command, and prints the plan it approved to standard error; use `run preflight` and `--approved-digest` as before if you want to read the plan first. **How to tell it worked.** The definition answers `version 2`, and its `dataset` line names the new dataset version and digest. The new run has a new `erun_` ID. Wait for it and read its items as before. Then read the first run again with `run get` (or `runs_get`, `runsGet`, or `GET …/runs/{run_id}`): it still names dataset version 2, and its `result_digest` is the string you wrote down. ## When a step goes wrong These are the answers you're most likely to meet on this path, with what each means and what to do.
One run at a time may be queued or running in a gateway environment, and another one is. The answer names it. Here a second start arrives while the first run is still queued: ```console $ tokentriage cloud evaluation run start --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --definition edef_7f3k9qp2mxaz --version 1 \ --approved-digest sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9 \ --idempotency-key q-demo-run-2 --base-url "$TOKENTRIAGE_BASE_URL" Error: customer.evaluations.runs.start was refused (HTTP 429); detail: run erun_7f3k9qp2mxaz of this gateway environment is queued or running; active_run_id erun_7f3k9qp2mxaz; type=evaluation_quota_exhausted; request_id=req_0000000000000001 ``` Nothing was started, and the key you sent wasn't used up. Wait for the active run with `run wait --run` and the ID in `active_run_id`, or stop it with `run cancel` if you started it by mistake, then send your start again. A start that repeats the active run's own request and idempotency key isn't turned away: it answers that run.
The dataset, the definition or the plan changed after you read the plan, so the digest you approved no longer names what would run. Nothing was started. Ask for the plan again with `run preflight` (or `runs_preflight`, `runsPreflight`), read what it says now, and start with its new `digest` and a new idempotency key.
The deployment isn't running the worker that scores runs, so it won't queue one it can't finish. Definitions, plans, runs and items are all still readable. Nothing was started; try again later, and if the answer persists, ask the deployment's operator whether the evaluation worker is running. A deployment that doesn't run offline evaluations at all answers `evaluations_not_enabled` (HTTP 501) on every call instead.
Starting a run needs `content.read` in the gateway environment, because the run opens each example's supplied output under your authority. The plan's `content_read` line tells you in advance: `false` means this start will be turned away. Ask for a content grant in that environment. Reading runs and items needs only `metadata.read`; without `content.read` the observed labels are left out.
An unscored example has one reason, taken from this list in this order: - `expected_label_absent`: the dataset example has no `expected_label`. Add one in the next dataset version. The plan's `expected_label_absent` counts these in advance. - `content_excluded`: the workspace is excluded while an export or deletion is in progress, so the body can't be opened. - `content_purged`: the example's body was deleted. Its identity and digest remain. - `content_not_retained`: this deployment holds no body it can open for the example. - `output_absent`: the example has no supplied output. - `label_absent`: the output is blank once trimmed (`plain_text`), or the JSON object has no member at the pointer (`json_pointer`). - `label_malformed`: with `json_pointer`, the output isn't one JSON object, the member isn't a string, or the label is over 256 bytes. An unscored example counts against `max_unscored`, never as a mismatch.
A `failed` run carries `failure.code`: `authority_revoked` (the starter's grant changed mid-run), `workspace_excluded`, `dataset_unavailable`, `digest_mismatch`, `attempts_exhausted` or `worker_fault`. Items recorded before the failure stay, and the definition is still usable. Fix the cause the code names, ask for a new plan and start a new run.
## What an offline check tells you An exact-label check tells you whether each supplied output carries the expected label, after trimming, with the case rule you chose. It doesn't judge whether an answer was good in any other way, and it doesn't compare two configurations. For that, use [Compare cost and quality before you switch models](/docs/build/compare-cost-and-quality). A supplied output is declared, not observed. Somebody put it in the dataset; TokenTriage didn't call a model to produce it, and the run calls no provider. A match says the declared output carries the expected label, not that a model produced that output. `provider_spend` is `{"basis": "no_dispatch", "dispatches": 0}` with no amount, because nothing was dispatched; what the outputs cost when they were first generated isn't part of this run. If you run TokenTriage yourself, the self-hosted [eval programs](/docs/calibration/eval-programs) are a separate mechanism with their own page. Nothing on this page applies to them. The CLI captures on this page ran the shipped `tokentriage` binary against a local loopback server answering the evaluation contract's fixture shapes; its digests were computed with the contract's rule over the dataset tutorial's version 2. The Python and TypeScript calls come from the repository's example programs, whose tests drive them through fake transports. That local verification checks the flags, the request bodies, the digests and how each client shows a run and its items. It doesn't show a live deployment's answers, its worker or its timing, it doesn't cover the rerun step, and it spent nobody's money; a run against a live deployment will replace these captures.