Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Check a dataset's supplied outputs against its labels

Score the outputs a dataset already holds against its expected labels, find the wrong one, and rerun on a new version, with no provider calls.

TutorialLast updated

On this page

A label mistake is cheapest to find before you pay a model to answer again. If your dataset already holds the outputs your application returned, you can score those outputs against the expected labels without sending a single request to a provider.

In this tutorial we’ll score version 2 of the dataset from Curate an evaluation dataset your team can reopen, open the one example whose output carries the wrong label, see why the other wasn’t scored, and then point the check at a newer dataset version. At the end you’ll hold a definition ID, a run ID and a result_digest, and you’ll know which example needs attention and why.

You need You don’t need
the dataset from the datasets tutorial, with its ID and version 2’s items_digest a provider connection or model credentials
a management key bound at role_version: 2 with evaluations.write and evaluations.run in that gateway environment, plus metadata.read and content.read there a budget: the plan names 0 provider calls
one of: curl and jq, the Python SDK, the TypeScript SDK, or the tokentriage CLI any spend: nothing on this page calls a model

The path is four requests: define the check, plan it, start it, and read its items. Evaluation actions explains the binding. The managed deployment doesn’t serve these operations yet, so the captures on this page come from a local loopback server; Availability says what changes that.

Set up once

Every channel reads the same values from your shell. The IDs and the digest shown are the example values the captures on this page use; put yours in their place.

$ export TOKENTRIAGE_BASE_URL=https://app.example.com
$ export TOKENTRIAGE_MANAGEMENT_KEY='<your management key>'
$ export ORG=org_ABCDEFGHJKLM WS=t_ABCDEFGHJKLM APP=app_ABCDEFGHJKLM
$ export GW=gw_ABCDEFGHJKLM ENV=env_ABCDEFGHJKLM
$ export DATASET=ds_ABCDEFGHJKLM
$ export DS_DIGEST=sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348

DS_DIGEST is version 2’s items_digest, the string the datasets tutorial had you write down. Then build the client for the channel you’ll use. The CLI needs nothing more.

$ LANE="$TOKENTRIAGE_BASE_URL/portal/api/v1/customer/v1"
$ SCOPE="organizations/$ORG/workspaces/$WS/applications/$APP"
$ EVALS="$LANE/$SCOPE/gateways/$GW/environments/$ENV/evaluations"
$ AUTH="Authorization: Bearer $TOKENTRIAGE_MANAGEMENT_KEY"

Define the check

A definition says which dataset version to read, where the label sits in each supplied output, and what result you’ll accept. Here the whole output is the label (plain_text), case matters, and you accept no mismatches and at most one example that can’t be scored. The idempotency_key is yours: sending it again with the same definition returns this one instead of making a second.

$ REF=$(jq -n --arg ds "$DATASET" --arg digest "$DS_DIGEST" \
  '{idempotency_key: "q-demo-definition-1", name: "Support exact labels",
    mode: "offline_exact_label",
    dataset: {dataset_id: $ds, version: 2, items_digest: $digest},
    label_source: {kind: "plain_text"}, case_sensitive: true,
    acceptance: {max_mismatched: 0, max_unscored: 1}}' |
  curl -sS -X POST "$EVALS/definitions" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- |
  jq -c '.data.version.ref')
$ echo "$REF"

Every channel publishes version 1 of the same definition. This is the CLI tab’s capture; the HTTP body and the SDKs’ data carry the same fields as JSON.

$ tokentriage cloud evaluation definition create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --name "Support exact labels" \
    --dataset ds_ABCDEFGHJKLM --dataset-version 2 \
    --dataset-digest sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 \
    --label-source plain_text --case-sensitive=true --max-mismatched 0 \
    --max-unscored 1 --idempotency-key q-demo-definition-1 \
    --base-url "$TOKENTRIAGE_BASE_URL"
definition        edef_7f3k9qp2mxaz
name              Support exact labels
state             active
version           1
spec_digest       sha256:c04709d8d560854eb49c6dee1b062f23b1d468ed59fb3b0bf62b0307dbb61360
mode              offline_exact_label
dataset           ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348
label_source      plain_text
rule              label_exact/v1 (case_sensitive true)
acceptance        max_mismatched 0, max_unscored 1

The capture ran against a local loopback server, so --base-url pointed there and the line shows your variable instead. The CLI also writes a one-line request trace to standard error; it’s left out of every capture here because its URL is the loopback’s.

How to tell it worked. version is 1 and the dataset line names version 2 with the digest you exported. The rule label_exact/v1 trims white space from both ends of each label and then compares bytes. The definition’s ref is its ID, its version and its spec_digest; every later step passes that ref along.

Plan the run

Before anything starts, ask for the plan. It reads the definition and the dataset, counts what would be scored, and returns a digest that names this exact plan. Nothing is written.

$ PLAN=$(jq -n --argjson ref "$REF" '{definition: $ref}' |
  curl -sS -X POST "$EVALS/runs/preflight" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- | jq -c '.data')
$ echo "$PLAN" | jq '{selected, expected_label_absent, planned_provider_calls,
    provider_spend, content_read, active_run_id, digest}'
$ tokentriage cloud evaluation run preflight --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --definition edef_7f3k9qp2mxaz --version 1 \
    --base-url "$TOKENTRIAGE_BASE_URL"
definition              edef_7f3k9qp2mxaz v1
dataset                 ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348
selected                2
expected_label_absent   0
planned_provider_calls  0
provider_spend          No provider calls (basis: no_dispatch)
content_read            true
active_run_id           -
digest                  sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9

How to tell it worked. selected is 2, one per example, and planned_provider_calls is 0. provider_spend reads no_dispatch: the run will read outputs the dataset already holds, so there is no request to pay for and no amount to show. content_read is true because your key can open the example bodies, which the run needs. active_run_id is empty, so nothing else is running here. Keep the digest.

Start the run and wait for it

The start carries the plan’s digest as approved_digest. If the dataset or the definition moved since you read the plan, the start is turned away instead of scoring something you didn’t look at. It also needs its own idempotency_key: a retry with the same key answers the same run.

$ RUN=$(jq -n --argjson ref "$REF" --arg d "$(echo "$PLAN" | jq -r .digest)" \
  '{idempotency_key: "q-demo-run-1", definition: $ref, approved_digest: $d}' |
  curl -sS -X POST "$EVALS/runs" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- | jq -r '.data.run_id')
$ for i in $(seq 30); do
    state=$(curl -sS "$EVALS/runs/$RUN" -H "$AUTH" | jq -r .data.state)
    case "$state" in complete|partial|failed|cancelled) break ;; esac
    sleep 2
  done
$ curl -sS "$EVALS/runs/$RUN" -H "$AUTH" | jq '.data | {state, counts,
    unscored_reasons, acceptance, provider_spend, result_digest}'

The start answers at once with the run queued and every example not yet evaluated:

$ tokentriage cloud evaluation run start --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --definition edef_7f3k9qp2mxaz --version 1 \
    --approved-digest sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9 \
    --idempotency-key q-demo-run-1 --base-url "$TOKENTRIAGE_BASE_URL"
run               erun_7f3k9qp2mxaz
state             queued
failure           -
definition        edef_7f3k9qp2mxaz v1
dataset           ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348
acceptance        undetermined (max_mismatched 0, max_unscored 1)
provider_spend    No provider calls (basis: no_dispatch)
result_digest     -

SELECTED  SCORED  MATCHED  MISMATCHED  UNSCORED  NOT_EVALUATED
2         0       0        0           0         2

run wait reads the run every two seconds and prints it once it finishes, in the same form run get uses. This is that read, after the run finished:

$ tokentriage cloud evaluation run get --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --run erun_7f3k9qp2mxaz \
    --base-url "$TOKENTRIAGE_BASE_URL"
run               erun_7f3k9qp2mxaz
state             partial
failure           -
definition        edef_7f3k9qp2mxaz v1
dataset           ds_ABCDEFGHJKLM v2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348
acceptance        not_met (max_mismatched 0, max_unscored 1)
provider_spend    No provider calls (basis: no_dispatch)
result_digest     sha256:7b8d3f8cc2a649018c048b8d6cecaff5f4e1e9673bfc09db6aeefe350b1b354e

SELECTED  SCORED  MATCHED  MISMATCHED  UNSCORED  NOT_EVALUATED
2         1       0        1           1         0

UNSCORED_REASON  ITEMS
output_absent    1

How to tell it worked. The counts add up: 2 selected is 0 matched, 1 mismatched, 1 unscored and 0 not evaluated. result_digest is filled in, which happens only when the run finishes. Write it down; the last step checks it again.

partial and not_met are the check doing its job, not a fault. partial means every example was recorded and at least one couldn’t be scored. not_met means the result broke your acceptance bar: one mismatch against a limit of zero. The one unscored example is within the limit of one you set.

If the wait runs out first, the run carries on. Read it again with the same run ID rather than starting another; a second start would be turned away while the first is still going.

Open the mismatched and the unscored examples

The run’s items say, example by example, what was expected, what the supplied output said, and why an example wasn’t scored.

$ curl -sS "$EVALS/runs/$RUN/items" -H "$AUTH" |
    jq '.data[] | {item_id, outcome, reason, expected_label,
        observed_label, content_state}'
$ tokentriage cloud evaluation run items --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --run erun_7f3k9qp2mxaz \
    --base-url "$TOKENTRIAGE_BASE_URL"
ITEM_ID   OUTCOME     REASON         EXPECTED_LABEL  OBSERVED_LABEL  CONTENT_STATE
refund-2  unscored    output_absent  billing         -               available
vpn-1     mismatched  -              network         billing         available

How to tell it worked. Two rows, one per example, in item_id order.

vpn-1 is the wrong-label row. The dataset expects network, the label you corrected in version 2, and the output supplied with the example still says billing. The check has found an answer that disagrees with the corrected expectation, which is the thing to investigate.

refund-2 wasn’t scored, and the reason says why: output_absent. The dataset holds no supplied output for it, so there’s nothing to compare. It counts as unscored, not as a mismatch or a zero.

To list only one kind of row, add --outcome mismatched (or unscored), the outcome argument in Python and TypeScript, or ?outcome=mismatched on the HTTP path.

observed_label is read from the example body each time you list the items, under your own content.read. The run doesn’t keep a copy. A teammate without that grant sees the same outcomes and expected labels, with content_state withheld and no observed label.

Rerun on a new dataset version

A run is fixed to the definition version it started from, and a definition version is fixed to one dataset version. So when the dataset changes, you publish its next version (as the datasets tutorial shows), then publish the definition’s next version pinned to it, and start a new run. The first run and its result_digest stay as they are.

Put the new dataset version’s number and items_digest in two variables. They come from the answer that published it.

$ export DS_VERSION=3   # the version number you published
$ export DS_DIGEST_NEW='<its items_digest>'
$ DEF=$(echo "$REF" | jq -r .definition_id)
$ REF2=$(jq -n --arg ds "$DATASET" --argjson v "$DS_VERSION" \
    --arg digest "$DS_DIGEST_NEW" \
  '{idempotency_key: "q-demo-definition-1-v2", expected_version: 1,
    mode: "offline_exact_label",
    dataset: {dataset_id: $ds, version: $v, items_digest: $digest},
    label_source: {kind: "plain_text"}, case_sensitive: true,
    acceptance: {max_mismatched: 0, max_unscored: 1}}' |
  curl -sS -X POST "$EVALS/definitions/$DEF/versions" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- |
  jq -c '.data.version.ref')
$ PLAN2=$(jq -n --argjson ref "$REF2" '{definition: $ref}' |
  curl -sS -X POST "$EVALS/runs/preflight" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- | jq -c '.data')
$ jq -n --argjson ref "$REF2" --arg d "$(echo "$PLAN2" | jq -r .digest)" \
  '{idempotency_key: "q-demo-run-2", definition: $ref, approved_digest: $d}' |
  curl -sS -X POST "$EVALS/runs" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- |
  jq '.data | {run_id, state, definition, dataset}'

This step has no capture on this page. Its answers have the same shape as the earlier steps, with the new values in them. --approve-current-plan asks for the plan and approves it in one command, and prints the plan it approved to standard error; use run preflight and --approved-digest as before if you want to read the plan first.

How to tell it worked. The definition answers version 2, and its dataset line names the new dataset version and digest. The new run has a new erun_ ID. Wait for it and read its items as before. Then read the first run again with run get (or runs_get, runsGet, or GET …/runs/{run_id}): it still names dataset version 2, and its result_digest is the string you wrote down.

When a step goes wrong

These are the answers you’re most likely to meet on this path, with what each means and what to do.

The start answers evaluation_quota_exhausted (HTTP 429)

One run at a time may be queued or running in a gateway environment, and another one is. The answer names it. Here a second start arrives while the first run is still queued:

$ tokentriage cloud evaluation run start --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --definition edef_7f3k9qp2mxaz --version 1 \
    --approved-digest sha256:434d3ad99f7b1316e8b4118a6931628023ac5adcb149667e20cde79bfd4ee2a9 \
    --idempotency-key q-demo-run-2 --base-url "$TOKENTRIAGE_BASE_URL"
Error: customer.evaluations.runs.start was refused (HTTP 429); detail: run erun_7f3k9qp2mxaz of this gateway environment is queued or running; active_run_id erun_7f3k9qp2mxaz; type=evaluation_quota_exhausted; request_id=req_0000000000000001

Nothing was started, and the key you sent wasn’t used up. Wait for the active run with run wait --run and the ID in active_run_id, or stop it with run cancel if you started it by mistake, then send your start again. A start that repeats the active run’s own request and idempotency key isn’t turned away: it answers that run.

The start answers evaluation_subject_stale (HTTP 409)

The dataset, the definition or the plan changed after you read the plan, so the digest you approved no longer names what would run. Nothing was started. Ask for the plan again with run preflight (or runs_preflight, runsPreflight), read what it says now, and start with its new digest and a new idempotency key.

The start answers evaluation_worker_unavailable (HTTP 503)

The deployment isn’t running the worker that scores runs, so it won’t queue one it can’t finish. Definitions, plans, runs and items are all still readable. Nothing was started; try again later, and if the answer persists, ask the deployment’s operator whether the evaluation worker is running. A deployment that doesn’t run offline evaluations at all answers evaluations_not_enabled (HTTP 501) on every call instead.

The start answers content_read_denied (HTTP 403)

Starting a run needs content.read in the gateway environment, because the run opens each example’s supplied output under your authority. The plan’s content_read line tells you in advance: false means this start will be turned away. Ask for a content grant in that environment. Reading runs and items needs only metadata.read; without content.read the observed labels are left out.

An example is unscored, and the reason isn't output_absent

An unscored example has one reason, taken from this list in this order:

  • expected_label_absent: the dataset example has no expected_label. Add one in the next dataset version. The plan’s expected_label_absent counts these in advance.
  • content_excluded: the workspace is excluded while an export or deletion is in progress, so the body can’t be opened.
  • content_purged: the example’s body was deleted. Its identity and digest remain.
  • content_not_retained: this deployment holds no body it can open for the example.
  • output_absent: the example has no supplied output.
  • label_absent: the output is blank once trimmed (plain_text), or the JSON object has no member at the pointer (json_pointer).
  • label_malformed: with json_pointer, the output isn’t one JSON object, the member isn’t a string, or the label is over 256 bytes.

An unscored example counts against max_unscored, never as a mismatch.

The run ends failed

A failed run carries failure.code: authority_revoked (the starter’s grant changed mid-run), workspace_excluded, dataset_unavailable, digest_mismatch, attempts_exhausted or worker_fault. Items recorded before the failure stay, and the definition is still usable. Fix the cause the code names, ask for a new plan and start a new run.

What an offline check tells you

An exact-label check tells you whether each supplied output carries the expected label, after trimming, with the case rule you chose. It doesn’t judge whether an answer was good in any other way, and it doesn’t compare two configurations. For that, use Compare cost and quality before you switch models.

If you run TokenTriage yourself, the self-hosted eval programs are a separate mechanism with their own page. Nothing on this page applies to them.

The CLI captures on this page ran the shipped tokentriage binary against a local loopback server answering the evaluation contract’s fixture shapes; its digests were computed with the contract’s rule over the dataset tutorial’s version 2. The Python and TypeScript calls come from the repository’s example programs, whose tests drive them through fake transports. That local verification checks the flags, the request bodies, the digests and how each client shows a run and its items. It doesn’t show a live deployment’s answers, its worker or its timing, it doesn’t cover the rerun step, and it spent nobody’s money; a run against a live deployment will replace these captures.