Export and remove evaluation evidence

Your team built datasets, ran offline evaluations against them and set up comparison definitions and judge profiles. Now somebody wants a copy of all of it, or wants the workspace that holds it gone. This page covers both, for evaluation evidence in particular.

By the end you’ll have a bundle on disk with its evaluation rows sorted by kind, and you’ll know which bodies stayed behind, what a deletion removes, and what it keeps.

The journey runs on the workspace’s own data exit, so you need what export and delete your data needs: a management key, the organization and workspace ids, and the lifecycle.export grant, plus lifecycle.delete to remove anything. Read that page first if you haven’t; this one adds only what changes for evaluation evidence.

Where evaluation evidence lands in a bundle

Evaluation evidence doesn’t get a section of its own. A bundle still has the same fourteen sections, and evaluation rows join two of them: runs and comparisons. Each evaluation row carries a kind member that says what it is.

• runs holds evaluation_definition, evaluation_run and dataset rows, beside the test run records it always held.
• comparisons holds comparison_definition, judge_profile and decision_provenance rows, beside the comparison records it always held.

A versioned resource exports one row per version, so a dataset with three versions is three dataset rows. The full table, with what each row’s document holds, is in the managed evaluations reference.

Every evaluation row has the same envelope. This is its shape, not a captured row:

{
  "kind": "dataset",
  "id": "…",
  "version": 1,
  "organization_id": "…",
  "workspace_id": "…",
  "created_at": "RFC 3339, UTC",
  "document": {}
}

version is left out for kinds that have no versions (evaluation_run and decision_provenance), and for a resource that was created but never given a version.

Each section keeps its own grant. The collector re-checks your grants when it packages the bundle, so runs needs metadata.read and comparisons needs comparisons.run at that moment. Judge profiles travel in comparisons, so a bundle only carries them for someone holding comparisons.run, even though the API reads them under metadata.read.

Export and download the bundle

Ask for the export. It answers 202 with the export in collecting:

$ tokentriage cloud export create \
    --base-url https://app.example.com \
    --org org_7f3k9qp2mxaz --workspace t_7f3k9qp2mxaz

Read it back until the state is ready, expired or failed. From a program, this is exports.get in a loop, the way sdk/python/examples/export_and_delete.py and sdk/ts/examples/export_and_delete.ts do it:

$ tokentriage cloud export get \
    --base-url https://app.example.com \
    --org org_7f3k9qp2mxaz --workspace t_7f3k9qp2mxaz \
    --export dex_ABCDEFGHJKLM --json

Then take the bytes. download checks the archive’s digest and every part the manifest names before it reports success:

$ tokentriage cloud export download \
    --base-url https://app.example.com \
    --org org_7f3k9qp2mxaz --workspace t_7f3k9qp2mxaz \
    --export dex_ABCDEFGHJKLM --out bundle.tar.gz
$ mkdir bundle && tar -xzf bundle.tar.gz -C bundle

These are shown without their answers, because the answer comes from your deployment. The manifest the get prints and the refusals a download can meet are on the workspace export page.

Before you read any rows, look at runs and comparisons in the manifest. A section that isn’t included has no rows at all, neither the old records nor the evaluation ones:

• withheld means the service held the section and left it out for the stated reason. For these two sections, the usual reason is a grant you didn’t hold when the bundle was packaged.
• unavailable with bound_exceeded means the workspace holds more records than one page of that store, and the service left the whole section out rather than hand you part of it. Any other unavailable reason, such as source_unavailable, means the deployment couldn’t read that store.

Read the rows by kind

A section now mixes rows of different shapes, so read each one by looking at kind before anything else. Three rules cover every row:

1. No kind member: a row from before evaluation kinds existed. In runs it’s a test run, keyed by run_id. In comparisons it’s a comparison record, keyed by comparison_id.
2. A kind you handle: decode its document for that kind.
3. A kind you don’t know: skip it and count it. A later release can add kinds, and a reader that fails on one would fail on a bundle that’s otherwise fine.

The count matters because the manifest’s rows for a section includes every row of every kind. Legacy rows, read rows and skipped rows add up to it, and a reader can check that. This reader does all three and checks the total:

import json
from collections import Counter


def read_section(root, manifest, name, kinds):
    section = next(s for s in manifest["sections"] if s["section"] == name)
    if section["coverage"] != "included":
        why = section.get("reason")
        raise RuntimeError(f"{name} is {section['coverage']}: {why}")
    legacy, rows, skipped = [], [], Counter()
    for part in section["parts"]:
        with open(f"{root}/{part['path']}", encoding="utf-8") as lines:
            for row in map(json.loads, lines):
                kind = row.get("kind")
                if kind is None:
                    legacy.append(row)  # a row from before kinds existed
                elif kind in kinds:
                    rows.append(row)
                else:
                    skipped[kind] += 1  # a kind newer than this reader
    if len(legacy) + len(rows) + sum(skipped.values()) != section["rows"]:
        raise RuntimeError(f"{name}: rows read differ from the manifest")
    return legacy, rows, skipped

import { readFileSync } from "node:fs";

type Row = { kind?: string; [member: string]: unknown };

export function readSection(
  root: string, manifest: any, name: string, kinds: Set<string>,
) {
  const section = manifest.sections.find((s: any) => s.section === name);
  if (section.coverage !== "included") {
    throw new Error(`${name} is ${section.coverage}: ${section.reason}`);
  }
  const legacy: Row[] = [];
  const rows: Row[] = [];
  const skipped = new Map<string, number>();
  for (const part of section.parts) {
    const text = readFileSync(`${root}/${part.path}`, "utf8");
    for (const line of text.split("\n").filter(Boolean)) {
      const row: Row = JSON.parse(line);
      if (row.kind === undefined) legacy.push(row); // from before kinds
      else if (kinds.has(row.kind)) rows.push(row);
      else skipped.set(row.kind, (skipped.get(row.kind) ?? 0) + 1);
    }
  }
  let read = legacy.length + rows.length;
  for (const n of skipped.values()) read += n;
  if (read !== section.rows) {
    throw new Error(`${name}: rows read differ from the manifest`);
  }
  return { legacy, rows, skipped };
}

Call it once per section with the kinds you handle there, for example read_section("bundle", manifest, "runs", {"evaluation_run", "dataset"}). Both versions were run over a hand-built runs part of four rows: one test run, one evaluation_run, one dataset and one row of a made-up kind. Each returned one legacy row, two read rows and one skipped row under the made-up kind, and each stopped with comparisons is withheld on a withheld section. That fixture is ours, not a bundle a deployment wrote.

To join a dataset example to its body, match the example’s body_ref in a dataset row with the ref of a row in content_inventory. The inventory row tells you the body’s digest, size, retention class and state.

What stays out of the bundle, and why

An evaluation row describes evidence; it doesn’t carry the evidence’s text. No row holds an example body, an input, an output, the required facts or an observed label. A dataset example’s expected_label is metadata, so it’s in the dataset row.

The bodies are content objects, and they follow the content rules on the workspace page. The content_bodies section answers withheld with retrieved_separately for every export, and you fetch each body through the content read, which re-checks content.read per object. That verb is granted to nobody by default.

A body you try to fetch can answer one of two states that aren’t a permission problem:

How long a body is kept is its retention class: short, standard or extended. Your workspace declares the length of each, so the class is the fact the service stores and the duration is your declaration. The inventory row carries it as retention_class.

Two more things about the rows themselves. A decision_provenance row whose comparison definition version can’t be found is still exported, with its provenance.state set to unavailable, so the entry stays visible. And a ceiling_micro_usd inside a document is a spend ceiling somebody configured, as the API returns it. It’s not spend. An offline evaluation run always reports its provider spend as no_dispatch, because it calls no provider.

Remove evaluation evidence

This release has no per-dataset, per-evaluation, per-definition or per-profile delete. You remove evaluation evidence by deleting the workspace that holds it, with the two-call challenge described in what “deleted” means, store by store. Take the export first, and check its digest, because the deletion can’t be taken back once its fence commits.

From the moment the fence commits, every dataset, evaluation, comparison definition and judge profile read and write in the workspace answers workspace_excluded, and the example bodies read excluded. The purge then deletes, for that workspace:

• datasets, their versions and their examples, names included
• evaluation definitions, their versions, runs, run items and run jobs
• comparison definitions, their versions and their run links
• judge profiles, their versions and their calibration links
• the example bodies (dataset_example) and review notes (review_note) in object storage

What a deletion keeps

Two kinds of record outlive the evaluation evidence, and both are listed, with a sentence each, in the deletion’s progress document:

• The content inventory receipt for each body. Its state moves to purged and it keeps the digest, the byte count and the instant the state changed. It’s the record that a body existed and is gone, not a way to read it.
• Financial evidence: settled spend, statements and ledger references, and the records that the export and the deletion happened.

The full list and its reason codes are under what is kept, and why.

A restore doesn’t bring evidence back

A restore isn’t something you ask for. It’s an operator procedure run against a whole deployment from its backups, and no customer API call, SDK method or tokentriage command starts one.

A backup can predate your deletion, so restoring it would bring the workspace’s evaluation rows back on its own. The workspace’s fence and exclusions are also kept in a journal outside the database. The operator replays that journal before reads reopen, so the fence is declared again and every content object is excluded again. After that, the workspace answers workspace_excluded and its evaluation bodies answer content_excluded, as they did before the restore.

Reads stay closed until the replay has run. A body that was still stored when the backup was taken, and purged since, has no bytes left to return in any case: the restored inventory row points at bytes that are gone, and the read fails closed. The measured restore figures, and the conditions they were measured under, are on the workspace page.