# Curate an evaluation dataset your team can reopen Save synthetic examples, publish a corrected version, and reopen the original as a teammate would, from the API, either SDK or the CLI. Source: https://tokentriage.com/docs/build/curate-evaluation-datasets --- import Callout from '../../../components/Callout.astro'; import CodeTabs from '../../../components/CodeTabs.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; {/* Provenance of every transcript below, so a later editor regenerates rather than guesses: the CLI captures (create, version create, the stale-edit conflict, list, version get and its --json form) — go test ./internal/cli/ -run TestDatasetDocsTranscripts -update-dataset-transcripts then paste the named block from internal/cli/testdata/dataset-transcripts.txt (=== create, === version-create, === version-conflict, === list, === version-get, === version-get-json). The command lines on this page are the block's command re-wrapped with backslashes; the output is verbatim. the Python and TypeScript calls — sdk/python/examples/quality_datasets.py and sdk/ts/examples/quality_datasets.ts, exercised by sdk/python/tests/test_customer_quality_datasets_example.py and sdk/ts/test/unit/customer_datasets_example.test.ts against fake transports the operations, bounds, digest rule and refusal words — docs/capabilities/quality-evaluation/q01-hosted-datasets.md and internal/control/foundation/dataset/{types,digest,refusals}.go The lane answering the CLI captures is a loopback test server that answers the contract's fixture shapes. It stores nothing, calls no provider and spends nobody's money. The digests in it are computed over the fixture examples with the D5 rule, so they reproduce. A later receipt replaces these captures with a run against a live deployment. */} An evaluation result is only as repeatable as the examples under it. If those examples live in a spreadsheet somebody edited last Tuesday, nobody can say which set a score was measured on, and a teammate who reruns it may be scoring different rows. In this tutorial we'll save two synthetic support tickets as a hosted dataset, fix a wrong label by publishing version 2, and then reopen version 1 as a teammate to check that it hasn't moved. At the end you'll hold a dataset ID, two version numbers and two `items_digest` strings, and you'll know which one names which set of examples. | You need | You don't need | |---|---| | a hosted gateway environment, and its organization, workspace, application, gateway and environment IDs | a provider connection or model credentials | | a management key whose role grants `datasets.write` there (and `content.read` if you want to see example bodies) | saved test cases, traffic, or a budget | | one of: `curl` and `jq`, the Python SDK, the TypeScript SDK, or the `tokentriage` CLI | any spend: nothing on this page calls a model | The path is four requests: create, publish a correction, list, and read an old version. If you still need a key, [a management key carries a person](/docs/build/organizations-and-access#a-management-key-carries-a-person-and-the-list-says-which) explains how to get one and what it can reach. ## Set up once Every channel reads the same six values from your shell. The IDs shown are the example values the captures on this page use; put yours in their place. ```console $ export TOKENTRIAGE_BASE_URL=https://app.example.com $ export TOKENTRIAGE_MANAGEMENT_KEY='' $ export ORG=org_ABCDEFGHJKLM WS=t_ABCDEFGHJKLM APP=app_ABCDEFGHJKLM $ export GW=gw_ABCDEFGHJKLM ENV=env_ABCDEFGHJKLM ``` Then build the client for the channel you'll use. The CLI needs nothing more: it reads the key from `TOKENTRIAGE_MANAGEMENT_KEY` and takes `--base-url` on every command. ```console $ LANE="$TOKENTRIAGE_BASE_URL/portal/api/v1/customer/v1" $ SCOPE="organizations/$ORG/workspaces/$WS/applications/$APP" $ DATASETS="$LANE/$SCOPE/gateways/$GW/environments/$ENV/datasets" $ AUTH="Authorization: Bearer $TOKENTRIAGE_MANAGEMENT_KEY" ``` ```python import json, os from tokentriage.customer import CustomerClient, DatasetScope client = CustomerClient(os.environ["TOKENTRIAGE_BASE_URL"], os.environ["TOKENTRIAGE_MANAGEMENT_KEY"]) scope = DatasetScope(org=os.environ["ORG"], ws=os.environ["WS"], app=os.environ["APP"], gw=os.environ["GW"], env=os.environ["ENV"]) ``` ```typescript import { readFileSync } from "node:fs"; import { CustomerClient } from "tokentriage/customer"; const env = (k: string) => process.env[k] ?? ""; const client = new CustomerClient({ baseUrl: env("TOKENTRIAGE_BASE_URL"), managementApiKey: env("TOKENTRIAGE_MANAGEMENT_KEY"), }); const scope = { org: env("ORG"), ws: env("WS"), app: env("APP"), gw: env("GW"), env: env("ENV") }; ``` ## Write two examples A dataset version is a list of examples. Each one has an `item_id` you choose, an `origin`, and optionally an `expected_label` and a supplied `output`. Save this as `examples-v1.json`. The label on `vpn-1` is a mistake that version 2 corrects. ```json [ {"item_id": "vpn-1", "origin": "authored", "input": "Cannot connect to VPN", "expected_label": "billing", "output": "billing"}, {"item_id": "refund-2", "origin": "authored", "input": "I was charged twice this month", "expected_label": "billing"} ] ``` `authored` means you typed the input yourself. The `output` on `vpn-1` is an answer you are declaring, as if a model had said it. The provenance section near the end explains why that matters. ## Publish version 1 Creating the dataset publishes its first version in the same request. The `idempotency_key` is yours to choose and to keep: sending the same key with the same examples again returns this dataset instead of making a second one. ```console $ jq -n --slurpfile ex examples-v1.json \ '{idempotency_key: "q-demo-dataset-1", name: "A2 support labels", examples: $ex[0]}' | curl -sS -X POST "$DATASETS" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq '{dataset_id: .data.dataset.dataset_id, version: .data.version.version, items_digest: .data.version.items_digest}' ``` ```python with open("examples-v1.json") as f: examples_v1 = json.load(f) created = client.datasets.create( scope, idempotency_key="q-demo-dataset-1", name="A2 support labels", examples=examples_v1, )["data"] dataset_id = created["dataset"]["dataset_id"] print(dataset_id, created["version"]["version"], created["version"]["items_digest"]) ``` ```typescript const v1 = JSON.parse(readFileSync("examples-v1.json", "utf8")); const created = (await client.datasets.create(scope, { idempotency_key: "q-demo-dataset-1", name: "A2 support labels", examples: v1, })).data; const datasetId = created?.dataset?.dataset_id ?? ""; console.log(datasetId, created?.version?.version, created?.version?.items_digest); ``` ```console $ tokentriage cloud dataset create --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --name "A2 support labels" --examples-file examples-v1.json \ --idempotency-key q-demo-dataset-1 --json --base-url "$TOKENTRIAGE_BASE_URL" ``` Whichever channel you used, the answer is one `dataset` document. This is the CLI tab's capture; the HTTP body is the same document and the SDKs return it as `data`. ```console $ tokentriage cloud dataset create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --name "A2 support labels" \ --examples-file examples-v1.json --idempotency-key q-demo-dataset-1 \ --json --base-url "$TOKENTRIAGE_BASE_URL" { "schema": "customer/v1", "request_id": "req_0000000000000000", "kind": "dataset", "data": { "dataset": { "dataset_id": "ds_ABCDEFGHJKLM", "organization_id": "org_ABCDEFGHJKLM", "workspace_id": "t_ABCDEFGHJKLM", "application_id": "app_ABCDEFGHJKLM", "gateway_id": "gw_ABCDEFGHJKLM", "environment_id": "env_ABCDEFGHJKLM", "name": "A2 support labels", "current_version": 1, "state": "active", "created_by": "ada", "created_at": "2026-09-24T10:00:00Z", "updated_at": "2026-09-24T10:00:00Z" }, "version": { "dataset_id": "ds_ABCDEFGHJKLM", "version": 1, "ref": { "dataset_id": "ds_ABCDEFGHJKLM", "version": 1, "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7" }, "items_count": 2, "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7", "sources": { "authored": 2, "saved_case": 0 }, "published_by": "ada", "published_at": "2026-09-24T10:00:00Z", "examples": [ { "item_id": "refund-2", "origin": "authored", "expected_label": "billing", "saved_case": null, "body_digest": "sha256:bda05c455c521252441a71370507b1c635e3be0eb4ebac6645a76c9934eaddfd", "content_state": "available", "input": "I was charged twice this month" }, { "item_id": "vpn-1", "origin": "authored", "expected_label": "billing", "saved_case": null, "body_digest": "sha256:f383a1660a905d0e0682bf57111d366f60b09c3f397fea0b7574da49e46fa3f6", "content_state": "available", "input": "Cannot connect to VPN", "output": "billing" } ] } } } ``` The capture ran against a local loopback server, so `--base-url` pointed there and the line shows your variable instead. The CLI also writes a one-line request trace to standard error, naming the operation, the URL and the idempotency key; it's left out of every capture here because its URL is the loopback's. **How to tell it worked.** `version.version` is `1`, `items_count` is `2`, and both examples read `content_state: "available"` because the capture's key holds `content.read`. Write down `dataset_id` and `items_digest`. That digest is the fingerprint of version 1, and the teammate's check at the end compares against it. ## Correct the label as version 2 The ticket about a VPN is a network problem, not a billing one. A published version can't be edited, so you publish the whole corrected list as the next version. Save `examples-v2.json` with one change, `vpn-1`'s `expected_label`: ```json [ {"item_id": "vpn-1", "origin": "authored", "input": "Cannot connect to VPN", "expected_label": "network", "output": "billing"}, {"item_id": "refund-2", "origin": "authored", "input": "I was charged twice this month", "expected_label": "billing"} ] ``` A new version names the version it replaces as `expected_version`. If somebody else published in the meantime, your write is turned away instead of overwriting theirs. ```console $ DATASET=ds_ABCDEFGHJKLM # the dataset_id from the previous step $ jq -n --slurpfile ex examples-v2.json \ '{idempotency_key: "q-demo-dataset-1-v2", expected_version: 1, examples: $ex[0]}' | curl -sS -X POST "$DATASETS/$DATASET/versions" -H "$AUTH" \ -H "Content-Type: application/json" --data-binary @- | jq '.data.version | {version, items_digest}' ``` ```python with open("examples-v2.json") as f: examples_v2 = json.load(f) v2 = client.datasets.versions_create( scope, dataset_id, idempotency_key="q-demo-dataset-1-v2", expected_version=1, examples=examples_v2, )["data"]["version"] print(v2["version"], v2["items_digest"]) ``` ```typescript const v2 = (await client.datasets.versionsCreate(scope, datasetId, { idempotency_key: "q-demo-dataset-1-v2", expected_version: 1, examples: JSON.parse(readFileSync("examples-v2.json", "utf8")), })).data?.version; console.log(v2?.version, v2?.items_digest); ``` ```console $ tokentriage cloud dataset version create --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --dataset "$DATASET" --expected-version 1 \ --examples-file examples-v2.json --idempotency-key q-demo-dataset-1-v2 \ --base-url "$TOKENTRIAGE_BASE_URL" ``` Without `--json` the CLI prints a summary. It names each example's content state and whether its body came back, and it doesn't print the body itself: ```console $ tokentriage cloud dataset version create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --expected-version 1 \ --examples-file examples-v2.json --idempotency-key q-demo-dataset-1-v2 \ --base-url "$TOKENTRIAGE_BASE_URL" dataset ds_ABCDEFGHJKLM name A2 support labels state active version 2 items_count 2 items_digest sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 content_withheld false ITEM_ID ORIGIN EXPECTED_LABEL CONTENT_STATE BODY refund-2 authored billing available present vpn-1 authored network available present ``` **How to tell it worked.** `version` is `2` and `items_digest` is a different string from the one you wrote down. The label is part of what the digest covers, so correcting it moved the digest. The bodies didn't change, so each example's `body_digest` is the same as in version 1. A retry with the same key and the same file answers version 2 again. ## Reopen version 1 as a teammate Now switch seats. A teammate whose role grants `metadata.read` but not `content.read` wants the set you started with, perhaps because an earlier result was measured on it. Run this with the teammate's key in `TOKENTRIAGE_MANAGEMENT_KEY`. They find the dataset by listing, then read version 1 by number. ```console $ curl -sS "$DATASETS" -H "$AUTH" | jq '.data[] | {name, dataset_id, current: .current.version}' $ curl -sS "$DATASETS/$DATASET/versions/1" -H "$AUTH" | jq '.data.version | {version, items_digest, examples: [.examples[] | {item_id, expected_label, content_state}]}' ``` ```python for row in client.datasets.list(scope)["data"]: print(row["name"], row["dataset_id"], row["current"]["version"]) v1 = client.datasets.versions_get(scope, dataset_id, 1)["data"]["version"] print(v1["items_digest"], [(e["item_id"], e["content_state"]) for e in v1["examples"]]) ``` ```typescript const heads = (await client.datasets.list(scope)).data ?? []; for (const row of heads) console.log(row.name, row.current?.version); const old = (await client.datasets.versionsGet(scope, datasetId, 1)).data; console.log(old?.version?.items_digest, old?.version?.examples?.map( (e) => [e.item_id, e.content_state])); ``` ```console $ tokentriage cloud dataset list --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --base-url "$TOKENTRIAGE_BASE_URL" $ tokentriage cloud dataset version get --org "$ORG" --workspace "$WS" \ --app "$APP" --gateway "$GW" --environment "$ENV" \ --dataset "$DATASET" --version 1 --base-url "$TOKENTRIAGE_BASE_URL" ``` The list shows each dataset at its current version, without examples: ```console $ tokentriage cloud dataset list --org org_ABCDEFGHJKLM --workspace t_ABCDEFGHJKLM \ --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM --environment env_ABCDEFGHJKLM \ --base-url "$TOKENTRIAGE_BASE_URL" DATASET NAME VERSION ITEMS ITEMS_DIGEST STATE ds_ABCDEFGHJKLM A2 support labels 2 2 sha256:92372a66f14f1871e1eb83c5e96b26a90dcbed38892f0b60410949c6d7cf6348 active ``` Version 1 comes back as it was published: ```console $ tokentriage cloud dataset version get --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --version 1 \ --base-url "$TOKENTRIAGE_BASE_URL" dataset ds_ABCDEFGHJKLM name A2 support labels state active version 1 items_count 2 items_digest sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7 content_withheld true ITEM_ID ORIGIN EXPECTED_LABEL CONTENT_STATE BODY refund-2 authored billing withheld withheld vpn-1 authored billing withheld withheld ``` **How to tell it worked.** `items_digest` is character for character the string you wrote down at version 1, and `vpn-1` still carries its old `billing` label. The teammate can confirm the version without seeing a single ticket. The bodies read `withheld` because this key holds no `content.read`. That's the grant working, not a fault: the labels, digests and counts are metadata, and the ticket text and the declared output are customer content with their own permission.
The body fields `input`, `output` and `required_facts` are absent rather than empty, each example names why in `content_state`, and the version carries `content_withheld: true`. ```console $ tokentriage cloud dataset version get --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --version 1 \ --json --base-url "$TOKENTRIAGE_BASE_URL" { "schema": "customer/v1", "request_id": "req_0000000000000000", "kind": "dataset", "data": { "dataset": { "dataset_id": "ds_ABCDEFGHJKLM", "organization_id": "org_ABCDEFGHJKLM", "workspace_id": "t_ABCDEFGHJKLM", "application_id": "app_ABCDEFGHJKLM", "gateway_id": "gw_ABCDEFGHJKLM", "environment_id": "env_ABCDEFGHJKLM", "name": "A2 support labels", "current_version": 2, "state": "active", "created_by": "ada", "created_at": "2026-09-24T10:00:00Z", "updated_at": "2026-09-24T10:05:00Z" }, "version": { "dataset_id": "ds_ABCDEFGHJKLM", "version": 1, "ref": { "dataset_id": "ds_ABCDEFGHJKLM", "version": 1, "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7" }, "items_count": 2, "items_digest": "sha256:ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7", "sources": { "authored": 2, "saved_case": 0 }, "published_by": "ada", "published_at": "2026-09-24T10:00:00Z", "examples": [ { "item_id": "refund-2", "origin": "authored", "expected_label": "billing", "saved_case": null, "body_digest": "sha256:bda05c455c521252441a71370507b1c635e3be0eb4ebac6645a76c9934eaddfd", "content_state": "withheld" }, { "item_id": "vpn-1", "origin": "authored", "expected_label": "billing", "saved_case": null, "body_digest": "sha256:f383a1660a905d0e0682bf57111d366f60b09c3f397fea0b7574da49e46fa3f6", "content_state": "withheld" } ], "content_withheld": true } } } ```
## When a step goes wrong These are the four answers you're most likely to meet on this path, each with what it means and what to do.
Somebody published a version after you read the dataset, so the `expected_version` you sent is no longer the current one. Here a second writer, still holding version 1 as the head, tries to publish after version 2 went in: ```console $ tokentriage cloud dataset version create --org org_ABCDEFGHJKLM \ --workspace t_ABCDEFGHJKLM --app app_ABCDEFGHJKLM --gateway gw_ABCDEFGHJKLM \ --environment env_ABCDEFGHJKLM --dataset ds_ABCDEFGHJKLM --expected-version 1 \ --examples-file examples-v2.json --idempotency-key q-demo-dataset-1-v2-retry \ --base-url "$TOKENTRIAGE_BASE_URL" Error: customer.datasets.versions.create was refused (HTTP 409); detail: expected_version 1 is not the current version 2; type=dataset_version_conflict; request_id=req_0000000000000001 ``` Nothing was published. Read the current version with `cloud dataset get` (or `datasets.get` in either SDK), check whether it already holds your change, and if it doesn't, apply your change to its examples and publish again with its version number as `--expected-version` and a new idempotency key. In Python and TypeScript the same answer arrives as a `CustomerProblemError` whose `type` is `dataset_version_conflict`. A related 409, `idempotency_payload_mismatch`, means you reused an idempotency key with different examples. Keep one key per intended write: the same key only ever replays the same request.
The body was left out of the answer, and the word says why: - `withheld`: your key has no `content.read` here. Ask for a content grant if you need the text; the metadata is readable without it. - `excluded`: the workspace has been excluded while an export or deletion is in progress, so its content isn't served. - `purged`: the content was deleted. The version's identity, digests and counts remain. - `not_retained`: this deployment keeps no body it can open for the example. None of these changes `items_digest`, so a version stays verifiable whatever you can read.
A dataset lives at one gateway in one environment, and an ID from anywhere else answers exactly as an ID that doesn't exist. Check that all five of your IDs match where the dataset was created: the same `ds_` ID under a different environment is not found. If they match, your key may have no binding there; `forbidden` (HTTP 403) is the answer only when you can see the scope and lack `metadata.read` or `datasets.write`.
The deployment you're talking to doesn't run hosted datasets. The CLI reports it as "the server has not implemented customer.datasets.create yet (HTTP 501)". Retrying won't change it; confirm `TOKENTRIAGE_BASE_URL` points at the deployment you meant, and ask its operator whether hosted datasets are enabled.
## Where the examples came from, and how long they stay A version records provenance you can read back without opening a body. `sources` counts examples by origin, `published_by` names the person or key that published it, and `published_at` says when. An example's origin is one of two things. `authored` means somebody typed the input into the request, as you did here. `saved_case` means the example points at a saved test case version in the same gateway, and its input is read from that case rather than copied. A supplied `output` is declared, never observed. Somebody wrote it into the request; TokenTriage didn't call a model to produce it, and saving a dataset calls no provider at all. Treat it as a claim about what a model said, to be scored as such. The bodies are stored as customer content in your workspace, so they follow the rules the rest of your content follows. Reading them needs `content.read`. Excluding the workspace withdraws them from every read and write at once. An export includes each body as a content object, listed under `content_inventory` and `content_bodies`; in this release it doesn't include dataset or version metadata such as names, version numbers, labels and digests. A purge deletes every dataset in the workspace whole: its name, its versions, its examples and their bodies, with no record kept behind. [Export and delete your data](/docs/build/export-and-delete) covers those steps. ## What this doesn't cover A dataset in this release is a curated, versioned list of up to 50 examples at one gateway environment. You can't share it across workspaces or gateways, build it from a window of production traffic, or delete a single dataset on its own. Scoring a dataset is a later step and isn't available yet. That includes evaluation definitions, evaluation runs and reusable judge profiles, so nothing on this page grades the examples. Existing comparisons still use their own published samples, as [Compare cost and quality before you switch models](/docs/build/compare-cost-and-quality) describes. If you run TokenTriage yourself, the self-hosted [eval programs](/docs/calibration/eval-programs) and [judge configuration](/docs/judges/configure-a-judge) are separate mechanisms with their own pages. The CLI captures on this page ran the shipped `tokentriage` binary against a local loopback server answering the dataset contract's fixture shapes. The Python and TypeScript calls come from the repository's example programs, whose tests drive them through fake transports. That local verification checks the flags, the request bodies, the digests and how each client shows a withheld body. It doesn't show a live deployment's answer or its timing, and it spent nobody's money; a run against a live deployment will replace these captures.