# Export and remove evaluation evidence Take datasets, evaluations, comparison definitions and judge profiles out in a workspace bundle, read their rows by kind, and see what deletion keeps. Source: https://tokentriage.com/docs/build/manage-evaluation-evidence --- import Callout from '../../../components/Callout.astro'; import CodeTabs from '../../../components/CodeTabs.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; {/* Provenance, so a later editor re-derives rather than guesses: the six kinds, their sections, the row envelope and the row rules are docs/capabilities/quality-evaluation/q06-evidence-exit.md §4.1, with session ruling D1 placing `dataset` in `runs`; the legacy row keys and the section verbs are §2 of the same card (adapters_sections.go, adapters.go). the export, download and deletion journey, the coverage vocabulary, the retained reason codes and the restore sequence are the workspace page, build/export-and-delete.mdx, which cites its own sources; this page links to it rather than restating it. the retention classes are internal/control/foundation/scopedcontent/content.go (RetentionClasses); the read slugs content_excluded and content_purged are the ones reference/managed-evaluations.mdx and build/run-offline-evaluation.mdx already document. the two readers were run on 2026-09-24 over a hand-built bundle (one `runs` part of four rows and a withheld `comparisons` section): Python 3.14.5 and Node 24.13.0 (type stripping) on win32. No row on this page is a capture from a shipped bundle. */} Your team built datasets, ran offline evaluations against them and set up comparison definitions and judge profiles. Now somebody wants a copy of all of it, or wants the workspace that holds it gone. This page covers both, for evaluation evidence in particular. By the end you'll have a bundle on disk with its evaluation rows sorted by kind, and you'll know which bodies stayed behind, what a deletion removes, and what it keeps. The journey runs on the workspace's own data exit, so you need what [export and delete your data](/docs/build/export-and-delete) needs: a management key, the organization and workspace ids, and the `lifecycle.export` grant, plus `lifecycle.delete` to remove anything. Read that page first if you haven't; this one adds only what changes for evaluation evidence. ## Where evaluation evidence lands in a bundle Evaluation evidence doesn't get a section of its own. A bundle still has the same fourteen sections, and evaluation rows join two of them: `runs` and `comparisons`. Each evaluation row carries a `kind` member that says what it is. - **`runs`** holds `evaluation_definition`, `evaluation_run` and `dataset` rows, beside the test run records it always held. - **`comparisons`** holds `comparison_definition`, `judge_profile` and `decision_provenance` rows, beside the comparison records it always held. A versioned resource exports one row per version, so a dataset with three versions is three `dataset` rows. The full table, with what each row's `document` holds, is in [the managed evaluations reference](/docs/reference/managed-evaluations#evaluation-evidence-in-an-export-bundle). Every evaluation row has the same envelope. This is its shape, not a captured row: ```json { "kind": "dataset", "id": "…", "version": 1, "organization_id": "…", "workspace_id": "…", "created_at": "RFC 3339, UTC", "document": {} } ``` `version` is left out for kinds that have no versions (`evaluation_run` and `decision_provenance`), and for a resource that was created but never given a version. Each section keeps its own grant. The collector re-checks your grants when it packages the bundle, so `runs` needs `metadata.read` and `comparisons` needs `comparisons.run` at that moment. Judge profiles travel in `comparisons`, so a bundle only carries them for someone holding `comparisons.run`, even though the API reads them under `metadata.read`. ## Export and download the bundle Ask for the export. It answers 202 with the export in `collecting`: ```console $ tokentriage cloud export create \ --base-url https://app.example.com \ --org org_7f3k9qp2mxaz --workspace t_7f3k9qp2mxaz ``` Read it back until the state is `ready`, `expired` or `failed`. From a program, this is `exports.get` in a loop, the way `sdk/python/examples/export_and_delete.py` and `sdk/ts/examples/export_and_delete.ts` do it: ```console $ tokentriage cloud export get \ --base-url https://app.example.com \ --org org_7f3k9qp2mxaz --workspace t_7f3k9qp2mxaz \ --export dex_ABCDEFGHJKLM --json ``` Then take the bytes. `download` checks the archive's digest and every part the manifest names before it reports success: ```console $ tokentriage cloud export download \ --base-url https://app.example.com \ --org org_7f3k9qp2mxaz --workspace t_7f3k9qp2mxaz \ --export dex_ABCDEFGHJKLM --out bundle.tar.gz $ mkdir bundle && tar -xzf bundle.tar.gz -C bundle ``` These are shown without their answers, because the answer comes from your deployment. The manifest the `get` prints and the refusals a download can meet are on [the workspace export page](/docs/build/export-and-delete#ask-for-a-bundle-and-take-the-bytes). Before you read any rows, look at `runs` and `comparisons` in the manifest. A section that isn't `included` has no rows at all, neither the old records nor the evaluation ones: - `withheld` means the service held the section and left it out for the stated reason. For these two sections, the usual reason is a grant you didn't hold when the bundle was packaged. - `unavailable` with `bound_exceeded` means the workspace holds more records than one page of that store, and the service left the whole section out rather than hand you part of it. Any other `unavailable` reason, such as `source_unavailable`, means the deployment couldn't read that store. ## Read the rows by kind A section now mixes rows of different shapes, so read each one by looking at `kind` before anything else. Three rules cover every row: 1. **No `kind` member**: a row from before evaluation kinds existed. In `runs` it's a test run, keyed by `run_id`. In `comparisons` it's a comparison record, keyed by `comparison_id`. 2. **A `kind` you handle**: decode its `document` for that kind. 3. **A `kind` you don't know**: skip it and count it. A later release can add kinds, and a reader that fails on one would fail on a bundle that's otherwise fine. The count matters because the manifest's `rows` for a section includes every row of every kind. Legacy rows, read rows and skipped rows add up to it, and a reader can check that. This reader does all three and checks the total: ```python import json from collections import Counter def read_section(root, manifest, name, kinds): section = next(s for s in manifest["sections"] if s["section"] == name) if section["coverage"] != "included": why = section.get("reason") raise RuntimeError(f"{name} is {section['coverage']}: {why}") legacy, rows, skipped = [], [], Counter() for part in section["parts"]: with open(f"{root}/{part['path']}", encoding="utf-8") as lines: for row in map(json.loads, lines): kind = row.get("kind") if kind is None: legacy.append(row) # a row from before kinds existed elif kind in kinds: rows.append(row) else: skipped[kind] += 1 # a kind newer than this reader if len(legacy) + len(rows) + sum(skipped.values()) != section["rows"]: raise RuntimeError(f"{name}: rows read differ from the manifest") return legacy, rows, skipped ``` ```typescript import { readFileSync } from "node:fs"; type Row = { kind?: string; [member: string]: unknown }; export function readSection( root: string, manifest: any, name: string, kinds: Set, ) { const section = manifest.sections.find((s: any) => s.section === name); if (section.coverage !== "included") { throw new Error(`${name} is ${section.coverage}: ${section.reason}`); } const legacy: Row[] = []; const rows: Row[] = []; const skipped = new Map(); for (const part of section.parts) { const text = readFileSync(`${root}/${part.path}`, "utf8"); for (const line of text.split("\n").filter(Boolean)) { const row: Row = JSON.parse(line); if (row.kind === undefined) legacy.push(row); // from before kinds else if (kinds.has(row.kind)) rows.push(row); else skipped.set(row.kind, (skipped.get(row.kind) ?? 0) + 1); } } let read = legacy.length + rows.length; for (const n of skipped.values()) read += n; if (read !== section.rows) { throw new Error(`${name}: rows read differ from the manifest`); } return { legacy, rows, skipped }; } ``` Call it once per section with the kinds you handle there, for example `read_section("bundle", manifest, "runs", {"evaluation_run", "dataset"})`. Both versions were run over a hand-built `runs` part of four rows: one test run, one `evaluation_run`, one `dataset` and one row of a made-up kind. Each returned one legacy row, two read rows and one skipped row under the made-up kind, and each stopped with `comparisons is withheld` on a withheld section. That fixture is ours, not a bundle a deployment wrote.
An importer that recognises a row by `run_id` or `comparison_id` and ignores other rows keeps working unchanged: evaluation rows share no top-level member with the old rows, and every old row is written the same as before. An importer that decodes every line strictly, and fails on a member it doesn't know, stops at the first evaluation row. That failure is loud, not silent. Change the importer to check `kind` first, as the preceding reader does.
To join a dataset example to its body, match the example's `body_ref` in a `dataset` row with the `ref` of a row in `content_inventory`. The inventory row tells you the body's digest, size, retention class and state. ## What stays out of the bundle, and why An evaluation row describes evidence; it doesn't carry the evidence's text. No row holds an example body, an input, an output, the required facts or an observed label. A dataset example's `expected_label` is metadata, so it's in the `dataset` row. The bodies are content objects, and they follow the content rules on the workspace page. The `content_bodies` section answers `withheld` with `retrieved_separately` for every export, and you fetch each body through the content read, which re-checks `content.read` per object. That verb is granted to nobody by default. A body you try to fetch can answer one of two states that aren't a permission problem: | Answer | What it means for the body | |---|---| | `content_excluded` | the workspace, or that object, was excluded; the bytes may still exist, and the read returns none of them | | `content_purged` | the bytes are gone; the inventory row keeps the identity, digest and size | How long a body is kept is its retention class: `short`, `standard` or `extended`. Your workspace declares the length of each, so the class is the fact the service stores and the duration is your declaration. The inventory row carries it as `retention_class`. Two more things about the rows themselves. A `decision_provenance` row whose comparison definition version can't be found is still exported, with its `provenance.state` set to `unavailable`, so the entry stays visible. And a `ceiling_micro_usd` inside a `document` is a spend ceiling somebody configured, as the API returns it. It's not spend. An offline evaluation run always reports its provider spend as `no_dispatch`, because it calls no provider. ## Remove evaluation evidence This release has no per-dataset, per-evaluation, per-definition or per-profile delete. You remove evaluation evidence by deleting the workspace that holds it, with the two-call challenge described in [what "deleted" means, store by store](/docs/build/export-and-delete#what-deleted-means-store-by-store). Take the export first, and check its digest, because the deletion can't be taken back once its fence commits. From the moment the fence commits, every dataset, evaluation, comparison definition and judge profile read and write in the workspace answers `workspace_excluded`, and the example bodies read `excluded`. The purge then deletes, for that workspace: - datasets, their versions and their examples, names included - evaluation definitions, their versions, runs, run items and run jobs - comparison definitions, their versions and their run links - judge profiles, their versions and their calibration links - the example bodies (`dataset_example`) and review notes (`review_note`) in object storage `excluded` is unreadable, not erased. If what you owe someone is that the bytes are gone, wait until every leg of the deletion reports `purged`, and treat a leg at `in_progress` as work still outstanding. `tokentriage cloud workspace deletion get` shows each leg. ## What a deletion keeps Two kinds of record outlive the evaluation evidence, and both are listed, with a sentence each, in the deletion's progress document: - **The content inventory receipt** for each body. Its state moves to `purged` and it keeps the digest, the byte count and the instant the state changed. It's the record that a body existed and is gone, not a way to read it. - **Financial evidence**: settled spend, statements and ledger references, and the records that the export and the deletion happened. The full list and its reason codes are under [what is kept, and why](/docs/build/export-and-delete#what-is-kept-and-why). ## A restore doesn't bring evidence back A restore isn't something you ask for. It's an operator procedure run against a whole deployment from its backups, and no customer API call, SDK method or `tokentriage` command starts one. A backup can predate your deletion, so restoring it would bring the workspace's evaluation rows back on its own. The workspace's fence and exclusions are also kept in a journal outside the database. The operator replays that journal before reads reopen, so the fence is declared again and every content object is excluded again. After that, the workspace answers `workspace_excluded` and its evaluation bodies answer `content_excluded`, as they did before the restore. Reads stay closed until the replay has run. A body that was still stored when the backup was taken, and purged since, has no bytes left to return in any case: the restored inventory row points at bytes that are gone, and the read fails closed. The measured restore figures, and the conditions they were measured under, are on [the workspace page](/docs/build/export-and-delete#a-restore-does-not-reinstate-a-deleted-workspace).