Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Curate an evaluation dataset your team can reopen

Save synthetic examples, publish a corrected version, and reopen the original as a teammate would, from the API, either SDK or the CLI.

TutorialLast updated

On this page

An evaluation result is only as repeatable as the examples under it. If those examples live in a spreadsheet somebody edited last Tuesday, nobody can say which set a score was measured on, and a teammate who reruns it may be scoring different rows.

In this tutorial we’ll save two synthetic support tickets as a hosted dataset, fix a wrong label by publishing version 2, and then reopen version 1 as a teammate to check that it hasn’t moved. At the end you’ll hold a dataset ID, two version numbers and two items_digest strings, and you’ll know which one names which set of examples.

You need You don’t need
a hosted gateway environment, and its organization, workspace, application, gateway and environment IDs a provider connection or model credentials
a management key whose role grants datasets.write there (and content.read if you want to see example bodies) saved test cases, traffic, or a budget
one of: curl and jq, the Python SDK, the TypeScript SDK, or the tokentriage CLI any spend: nothing on this page calls a model

The path is four requests: create, publish a correction, list, and read an old version. If you still need a key, a management key carries a person explains how to get one and what it can reach.

Set up once

Every channel reads the same six values from your shell. The IDs shown are the example values the captures on this page use; put yours in their place.

$ export TOKENTRIAGE_BASE_URL=https://app.example.com
$ export TOKENTRIAGE_MANAGEMENT_KEY='<your management key>'
$ export ORG=org_ABCDEFGHJKLM WS=t_ABCDEFGHJKLM APP=app_ABCDEFGHJKLM
$ export GW=gw_ABCDEFGHJKLM ENV=env_ABCDEFGHJKLM

Then build the client for the channel you’ll use. The CLI needs nothing more: it reads the key from TOKENTRIAGE_MANAGEMENT_KEY and takes --base-url on every command.

$ LANE="$TOKENTRIAGE_BASE_URL/portal/api/v1/customer/v1"
$ SCOPE="organizations/$ORG/workspaces/$WS/applications/$APP"
$ DATASETS="$LANE/$SCOPE/gateways/$GW/environments/$ENV/datasets"
$ AUTH="Authorization: Bearer $TOKENTRIAGE_MANAGEMENT_KEY"

Write two examples

A dataset version is a list of examples. Each one has an item_id you choose, an origin, and optionally an expected_label and a supplied output. Save this as examples-v1.json. The label on vpn-1 is a mistake that version 2 corrects.

[
  {"item_id": "vpn-1", "origin": "authored", "input": "Cannot connect to VPN",
   "expected_label": "billing", "output": "billing"},
  {"item_id": "refund-2", "origin": "authored",
   "input": "I was charged twice this month", "expected_label": "billing"}
]

authored means you typed the input yourself. The output on vpn-1 is an answer you are declaring, as if a model had said it. The provenance section near the end explains why that matters.

Publish version 1

Creating the dataset publishes its first version in the same request. The idempotency_key is yours to choose and to keep: sending the same key with the same examples again returns this dataset instead of making a second one.

$ jq -n --slurpfile ex examples-v1.json \
  '{idempotency_key: "q-demo-dataset-1", name: "A2 support labels",
    examples: $ex[0]}' |
  curl -sS -X POST "$DATASETS" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- |
  jq '{dataset_id: .data.dataset.dataset_id, version: .data.version.version,
       items_digest: .data.version.items_digest}'

Whichever channel you used, the answer is one dataset document. This is the CLI tab’s capture; the HTTP body is the same document and the SDKs return it as data.

$ tokentriage cloud dataset create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --name "A2 support labels" \
    --examples-file examples-v1.json --idempotency-key q-demo-dataset-1 \
    --json --base-url "$TOKENTRIAGE_BASE_URL"
{
  "schema": "customer/v1",
  "request_id": "req_0000000000000000",
  "kind": "dataset",
  "data": {
    "dataset": {
      "dataset_id": "ds_ABCDEFGHJKLM",
      "organization_id": "org_ABCDEFGHJKLM",
      "workspace_id": "t_ABCDEFGHJKLM",
      "application_id": "app_ABCDEFGHJKLM",
      "gateway_id": "gw_ABCDEFGHJKLM",
      "environment_id": "env_ABCDEFGHJKLM",
      "name": "A2 support labels",
      "current_version": 1,
      "state": "active",
      "created_by": "ada",
      "created_at": "2026-09-24T10:00:00Z",
      "updated_at": "2026-09-24T10:00:00Z"
    },
    "version": {
      "dataset_id": "ds_ABCDEFGHJKLM",
      "version": 1,
      "ref": {
        "dataset_id": "ds_ABCDEFGHJKLM",
        "version": 1,
        "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7"
      },
      "items_count": 2,
      "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7",
      "sources": {
        "authored": 2,
        "saved_case": 0
      },
      "published_by": "ada",
      "published_at": "2026-09-24T10:00:00Z",
      "examples": [
        {
          "item_id": "refund-2",
          "origin": "authored",
          "expected_label": "billing",
          "saved_case": null,
          "body_digest": "sha256:bda05c455c521252441a71370507b1c635e3be0eb4ebac6645a76c9934eaddfd",
          "content_state": "available",
          "input": "I was charged twice this month"
        },
        {
          "item_id": "vpn-1",
          "origin": "authored",
          "expected_label": "billing",
          "saved_case": null,
          "body_digest": "sha256:f383a1660a905d0e0682bf57111d366f60b09c3f397fea0b7574da49e46fa3f6",
          "content_state": "available",
          "input": "Cannot connect to VPN",
          "output": "billing"
        }
      ]
    }
  }
}

The capture ran against a local loopback server, so --base-url pointed there and the line shows your variable instead. The CLI also writes a one-line request trace to standard error, naming the operation, the URL and the idempotency key; it’s left out of every capture here because its URL is the loopback’s.

How to tell it worked. version.version is 1, items_count is 2, and both examples read content_state: "available" because the capture’s key holds content.read. Write down dataset_id and items_digest. That digest is the fingerprint of version 1, and the teammate’s check at the end compares against it.

Correct the label as version 2

The ticket about a VPN is a network problem, not a billing one. A published version can’t be edited, so you publish the whole corrected list as the next version. Save examples-v2.json with one change, vpn-1’s expected_label:

[
  {"item_id": "vpn-1", "origin": "authored", "input": "Cannot connect to VPN",
   "expected_label": "network", "output": "billing"},
  {"item_id": "refund-2", "origin": "authored",
   "input": "I was charged twice this month", "expected_label": "billing"}
]

A new version names the version it replaces as expected_version. If somebody else published in the meantime, your write is turned away instead of overwriting theirs.

$ DATASET=ds_ABCDEFGHJKLM   # the dataset_id from the previous step
$ jq -n --slurpfile ex examples-v2.json \
  '{idempotency_key: "q-demo-dataset-1-v2", expected_version: 1,
    examples: $ex[0]}' |
  curl -sS -X POST "$DATASETS/$DATASET/versions" -H "$AUTH" \
    -H "Content-Type: application/json" --data-binary @- |
  jq '.data.version | {version, items_digest}'

Without --json the CLI prints a summary. It names each example’s content state and whether its body came back, and it doesn’t print the body itself:

$ tokentriage cloud dataset version create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --expected-version 1 \
    --examples-file examples-v2.json --idempotency-key q-demo-dataset-1-v2 \
    --base-url "$TOKENTRIAGE_BASE_URL"
dataset           ds_ABCDEFGHJKLM
name              A2 support labels
state             active
version           2
items_count       2
items_digest      sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348
content_withheld  false

ITEM_ID   ORIGIN    EXPECTED_LABEL  CONTENT_STATE  BODY
refund-2  authored  billing         available      present
vpn-1     authored  network         available      present

How to tell it worked. version is 2 and items_digest is a different string from the one you wrote down. The label is part of what the digest covers, so correcting it moved the digest. The bodies didn’t change, so each example’s body_digest is the same as in version 1. A retry with the same key and the same file answers version 2 again.

Reopen version 1 as a teammate

Now switch seats. A teammate whose role grants metadata.read but not content.read wants the set you started with, perhaps because an earlier result was measured on it. Run this with the teammate’s key in TOKENTRIAGE_MANAGEMENT_KEY. They find the dataset by listing, then read version 1 by number.

$ curl -sS "$DATASETS" -H "$AUTH" |
    jq '.data[] | {name, dataset_id, current: .current.version}'
$ curl -sS "$DATASETS/$DATASET/versions/1" -H "$AUTH" |
  jq '.data.version | {version, items_digest, examples: [.examples[] |
      {item_id, expected_label, content_state}]}'

The list shows each dataset at its current version, without examples:

$ tokentriage cloud dataset list --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \
    --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \
    --base-url "$TOKENTRIAGE_BASE_URL"
DATASET          NAME               VERSION  ITEMS  ITEMS_DIGEST                                                             STATE
ds_ABCDEFGHJKLM  A2 support labels  2        2      sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348  active

Version 1 comes back as it was published:

$ tokentriage cloud dataset version get --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --version 1 \
    --base-url "$TOKENTRIAGE_BASE_URL"
dataset           ds_ABCDEFGHJKLM
name              A2 support labels
state             active
version           1
items_count       2
items_digest      sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7
content_withheld  true

ITEM_ID   ORIGIN    EXPECTED_LABEL  CONTENT_STATE  BODY
refund-2  authored  billing         withheld       withheld
vpn-1     authored  billing         withheld       withheld

How to tell it worked. items_digest is character for character the string you wrote down at version 1, and vpn-1 still carries its old billing label. The teammate can confirm the version without seeing a single ticket.

The bodies read withheld because this key holds no content.read. That’s the grant working, not a fault: the labels, digests and counts are metadata, and the ticket text and the declared output are customer content with their own permission.

What a withheld version looks like as JSON

The body fields input, output and required_facts are absent rather than empty, each example names why in content_state, and the version carries content_withheld: true.

$ tokentriage cloud dataset version get --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --version 1 \
    --json --base-url "$TOKENTRIAGE_BASE_URL"
{
  "schema": "customer/v1",
  "request_id": "req_0000000000000000",
  "kind": "dataset",
  "data": {
    "dataset": {
      "dataset_id": "ds_ABCDEFGHJKLM",
      "organization_id": "org_ABCDEFGHJKLM",
      "workspace_id": "t_ABCDEFGHJKLM",
      "application_id": "app_ABCDEFGHJKLM",
      "gateway_id": "gw_ABCDEFGHJKLM",
      "environment_id": "env_ABCDEFGHJKLM",
      "name": "A2 support labels",
      "current_version": 2,
      "state": "active",
      "created_by": "ada",
      "created_at": "2026-09-24T10:00:00Z",
      "updated_at": "2026-09-24T10:05:00Z"
    },
    "version": {
      "dataset_id": "ds_ABCDEFGHJKLM",
      "version": 1,
      "ref": {
        "dataset_id": "ds_ABCDEFGHJKLM",
        "version": 1,
        "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7"
      },
      "items_count": 2,
      "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7",
      "sources": {
        "authored": 2,
        "saved_case": 0
      },
      "published_by": "ada",
      "published_at": "2026-09-24T10:00:00Z",
      "examples": [
        {
          "item_id": "refund-2",
          "origin": "authored",
          "expected_label": "billing",
          "saved_case": null,
          "body_digest": "sha256:bda05c455c521252441a71370507b1c635e3be0eb4ebac6645a76c9934eaddfd",
          "content_state": "withheld"
        },
        {
          "item_id": "vpn-1",
          "origin": "authored",
          "expected_label": "billing",
          "saved_case": null,
          "body_digest": "sha256:f383a1660a905d0e0682bf57111d366f60b09c3f397fea0b7574da49e46fa3f6",
          "content_state": "withheld"
        }
      ],
      "content_withheld": true
    }
  }
}

When a step goes wrong

These are the four answers you’re most likely to meet on this path, each with what it means and what to do.

The write answers dataset_version_conflict (HTTP 409)

Somebody published a version after you read the dataset, so the expected_version you sent is no longer the current one. Here a second writer, still holding version 1 as the head, tries to publish after version 2 went in:

$ tokentriage cloud dataset version create --org org_ABCDEFGHJKLM \
    --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \
    --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --expected-version 1 \
    --examples-file examples-v2.json --idempotency-key q-demo-dataset-1-v2-retry \
    --base-url "$TOKENTRIAGE_BASE_URL"
Error: customer.datasets.versions.create was refused (HTTP 409); detail: expected_version 1 is not the current version 2; type=dataset_version_conflict; request_id=req_0000000000000001

Nothing was published. Read the current version with cloud dataset get (or datasets.get in either SDK), check whether it already holds your change, and if it doesn’t, apply your change to its examples and publish again with its version number as --expected-version and a new idempotency key. In Python and TypeScript the same answer arrives as a CustomerProblemError whose type is dataset_version_conflict.

A related 409, idempotency_payload_mismatch, means you reused an idempotency key with different examples. Keep one key per intended write: the same key only ever replays the same request.

An example's content_state is withheld, excluded, purged or not_retained

The body was left out of the answer, and the word says why:

  • withheld: your key has no content.read here. Ask for a content grant if you need the text; the metadata is readable without it.
  • excluded: the workspace has been excluded while an export or deletion is in progress, so its content isn’t served.
  • purged: the content was deleted. The version’s identity, digests and counts remain.
  • not_retained: this deployment keeps no body it can open for the example.

None of these changes items_digest, so a version stays verifiable whatever you can read.

A dataset you know exists answers dataset_not_found (HTTP 404)

A dataset lives at one gateway in one environment, and an ID from anywhere else answers exactly as an ID that doesn’t exist. Check that all five of your IDs match where the dataset was created: the same ds_ ID under a different environment is not found. If they match, your key may have no binding there; forbidden (HTTP 403) is the answer only when you can see the scope and lack metadata.read or datasets.write.

Every call answers datasets_not_enabled (HTTP 501)

The deployment you’re talking to doesn’t run hosted datasets. The CLI reports it as “the server has not implemented customer.datasets.create yet (HTTP 501)”. Retrying won’t change it; confirm TOKENTRIAGE_BASE_URL points at the deployment you meant, and ask its operator whether hosted datasets are enabled.

Where the examples came from, and how long they stay

A version records provenance you can read back without opening a body. sources counts examples by origin, published_by names the person or key that published it, and published_at says when.

An example’s origin is one of two things. authored means somebody typed the input into the request, as you did here. saved_case means the example points at a saved test case version in the same gateway, and its input is read from that case rather than copied.

The bodies are stored as customer content in your workspace, so they follow the rules the rest of your content follows. Reading them needs content.read. Excluding the workspace withdraws them from every read and write at once. An export includes each body as a content object, listed under content_inventory and content_bodies; in this release it doesn’t include dataset or version metadata such as names, version numbers, labels and digests. A purge deletes every dataset in the workspace whole: its name, its versions, its examples and their bodies, with no record kept behind. Export and delete your data covers those steps.

What this doesn’t cover

A dataset in this release is a curated, versioned list of up to 50 examples at one gateway environment. You can’t share it across workspaces or gateways, build it from a window of production traffic, or delete a single dataset on its own.

Scoring a dataset is a later step and isn’t available yet. That includes evaluation definitions, evaluation runs and reusable judge profiles, so nothing on this page grades the examples. Existing comparisons still use their own published samples, as Compare cost and quality before you switch models describes. If you run TokenTriage yourself, the self-hosted eval programs and judge configuration are separate mechanisms with their own pages.

The CLI captures on this page ran the shipped tokentriage binary against a local loopback server answering the dataset contract’s fixture shapes. The Python and TypeScript calls come from the repository’s example programs, whose tests drive them through fake transports. That local verification checks the flags, the request bodies, the digests and how each client shows a withheld body. It doesn’t show a live deployment’s answer or its timing, and it spent nobody’s money; a run against a live deployment will replace these captures.