# Managed evaluations reference

Hosted datasets, offline evaluations, comparison definitions and judge profiles on a managed gateway, listed whole, with actions, bounds, identities, operations, errors and lifecycle.

Source: https://tokentriage.com/docs/reference/managed-evaluations

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

{/* Provenance: the operations, bounds, slugs and statuses are
    docs/capabilities/quality-evaluation/q01-hosted-datasets.md (decisions D5, D6,
    D10, D11) and internal/control/foundation/dataset/{types,digest,refusals}.go;
    the CLI's operation table is internal/cli/cloud.go (pinned by
    cloud_contract_test.go); the role rows are rolesV2 in
    internal/control/foundation/roles.go; the shared statuses (forbidden 403,
    validation_failed 422, workspace_excluded 409) are
    internal/control/foundation/httpapi/registry.go. The worked digest is the fixture
    in internal/cli/cloud_datasets_transcripts_test.go, recomputed with the jq recipe
    below.

    The offline-evaluation section: operations, actions, bounds, digests, states,
    unscored reasons, provider_spend and refusal slugs with their statuses are
    docs/capabilities/quality-evaluation/q02-offline-evaluation.md (decisions
    E2, E5-E11) and internal/control/foundation/evaluation/{types,digest,
    refusals,extract}.go; the client method names are
    sdk/python/src/tokentriage/customer/_client.py (_Evaluations),
    sdk/ts/src/customer/client.ts (EvaluationsApi) and
    internal/cli/cloud_evaluations.go. The Q02 operations and the two new verbs
    are registered by integration with the Q02 release; until then they are not
    on the managed target, and Q01's dataset operations likewise until their
    rollout. The Availability section says so and must change when they land.

    The comparison-definitions section: operations, kinds, bounds, digests and
    refusal slugs are docs/capabilities/quality-evaluation/q03-comparison-definitions.md
    (D3, D5-D8, D10 as amended by DEC-017, the T6 adapter decisions and the T7
    deviation) and internal/control/foundation/comparisondef/{types,refusals}.go;
    the management-key fence and the response kinds are
    internal/control/foundation/httpapi/comparison_definitions.go; the rows are
    registry_comparison_definitions.go; the content.read and comparison-grant
    checks are internal/control/cmd/controlplaned/foundation_comparisons_customer.go;
    the client method names are sdk/python/src/tokentriage/customer/_client.py
    (_ComparisonDefinitions), sdk/ts/src/customer/client.ts
    (ComparisonDefinitionsApi) and internal/cli/cloud_comparison_definitions.go.
    The lifecycle paragraph states the DEC-017 ruling (purge only, no export
    section); re-read it when Q03 T8 lands.

    The review-attestation, rationale and provenance additions to comparisons
    (`reviewed_evidence`, `acting_credential_kind`, `evidence_digest`,
    `rationale`, `provenance`, the "Provenance of a run" note) are
    docs/capabilities/quality-evaluation/q05-review-and-decision.md (D2-D8) and
    internal/control/foundation/comparison/{reading,types}.go; the wire shapes
    and refusals are internal/control/foundation/httpapi/{comparisons,
    decisions,registry}.go; the provenance read side is
    internal/control/foundation/comparisondef/provenance.go; the CLI flags are
    internal/cli/cloud_comparisons.go; the client method names are
    sdk/python/src/tokentriage/customer/_client.py (_Comparisons.review,
    .decide) and sdk/ts/src/customer/client.ts (ComparisonsApi). The full
    verdict and decision shapes, and the whole error catalogue, stay on
    [Gateway quality evidence](/docs/reference/gateway-quality-evidence); this
    page states only what a comparison-definition run adds on top.

    The judge-profiles section: operations, bounds, digests, pin states,
    applicability reasons and error slugs are
    docs/capabilities/quality-evaluation/q04-judge-profiles.md (D1-D9, D11 as
    amended by DEC-017, and the T6-T13 landed notes) and
    internal/control/foundation/judgeprofile/{types,digest,service}.go; the
    management-key fence, the strict decoder and the response kinds are
    internal/control/foundation/httpapi/judge_profiles.go; the rows are
    registry_judge_profiles.go; the lane's tests.run, content.read and cost.read
    checks and the USD 10 plan bound are
    internal/control/cmd/controlplaned/foundation_calibration.go and
    internal/control/foundation/calibration/types.go; the role rows are rolesV2 in
    internal/control/foundation/roles.go; the client method names are
    sdk/python/src/tokentriage/customer/_client.py (_JudgeProfiles),
    sdk/ts/src/customer/client.ts (JudgeProfilesApi) and
    internal/cli/cloud_judge_profiles.go. The lifecycle paragraph states the
    DEC-017 ruling (purge only, no export section). */}

This page lists what a managed deployment exposes for evaluation material, in the order
the product is built: who may act, what the limits are, how a version is identified,
which operations exist, what each error means, and what happens to the material over
its life. [Curate an evaluation dataset your team can reopen](/docs/build/curate-evaluation-datasets)
walks through the same operations as a tutorial.

That material comes in four parts. **Hosted datasets** are named, versioned lists of
examples at one gateway environment. **Offline evaluations** score the outputs a dataset
version already holds against its expected labels, without calling a provider; they have
[their own section](#offline-evaluations) after the dataset sections, and
[Check a dataset's supplied outputs against its labels](/docs/build/run-offline-evaluation)
is their tutorial. **Comparison definitions** save a provider comparison over one dataset
version so you can plan and start it again; they have
[their own section](#comparison-definitions), and
[Repeat a comparison from a dataset](/docs/build/compare-cost-and-quality#repeat-a-comparison-from-a-dataset)
walks through one. **Judge profiles** save a gateway judge with four controls so you can
calibrate it again; they have [the last section](#judge-profiles), and
[Keep a judge profile](/docs/build/calibrate-gateway-judge#keep-a-judge-profile) walks
through one.

<Callout type="note">
Evaluations that generate new outputs are a later step and aren't available yet. An
offline evaluation scores only outputs a dataset already holds.
To score two configurations on new outputs today, use
[Compare cost and quality before you switch models](/docs/build/compare-cost-and-quality).
</Callout>

## Availability

The managed deployment doesn't serve these operations yet. The offline-evaluation
operations and their two actions are registered with the release that ships them, and
hosted datasets reach the managed deployment with their own rollout. Until a deployment
runs the release in question, the operations on this page aren't part of what it serves,
and its answer to these paths isn't the contract described here. The tutorials' captures
come from a local loopback server for that reason.

Comparison definitions reach it with a release of their own, in the same way, and so do
judge profiles.

A deployment that runs the release either composes the service or it doesn't. Where it
doesn't, every dataset operation answers `datasets_not_enabled`, every evaluation
operation answers `evaluations_not_enabled`, every comparison definition operation
answers `comparison_definitions_not_enabled`, and every judge profile operation answers
`judge_profiles_not_enabled`, all with HTTP 501, and the CLI reports it
as "the server has not implemented `<operation>` yet (HTTP 501)". Nothing about the
request can change that answer. A deployment that composes evaluations but runs no
evaluation worker answers every read and turns away a run start with
`evaluation_worker_unavailable` (HTTP 503).

Self-hosted TokenTriage has its own, separate mechanisms:
[eval programs](/docs/calibration/eval-programs) and
[judge configuration](/docs/judges/configure-a-judge). Nothing on this page applies to
them.

## Actions

Dataset reads are governed by the existing `metadata.read` action, resolved at the
gateway environment the dataset belongs to; there is no `datasets.read` action. Writes
add one new action, `datasets.write`, and bodies still need the existing `content.read`.

| Action | Allows |
|---|---|
| `metadata.read` | list datasets, read a dataset or any version, and read metadata: names, labels, digests, counts, provenance |
| `datasets.write` | create a dataset and publish new versions |
| `content.read` | see example bodies (`input`, `output`, `required_facts`) in a read; without it, bodies are withheld and the metadata is still returned |

`metadata.read` is granted the same way it always was. `datasets.write` is different: it
is published in role version 2, but the default role version stays 1 in this release, so
`datasets.write` is resolvable and grantable only by an explicit `role_version: 2` on
`customer.bindings.create`, never the default stamp. A binding issued under an earlier
role version doesn't gain it; it has to be reissued at `role_version: 2`.

| Role | `metadata.read` | `datasets.write` (explicit `role_version: 2`) |
|---|---|---|
| `workspace_admin` | yes | yes |
| `app_developer` | yes | yes |
| `org_owner`, `org_admin`, `analyst` | yes | no |
| `billing_admin`, `env_deployer`, `content_reader`, `credential_manager`, `connection_manager` | no | no |
| `runtime_service`, `test_executor` | no | no |

`content.read` is a separate grant in every case. A runtime key reaches none of these
operations. [Content, cost, billing and metadata are four separate grants](/docs/build/organizations-and-access#content-cost-billing-and-metadata-are-four-separate-grants)
explains why.

## Bounds

A request outside these bounds is answered with `dataset_bounds_exceeded` (counts and the
request size) or `dataset_example_invalid` (a field), and nothing is stored.

| Item | Bound |
|---|---|
| examples per version | 1 to 50 |
| request body | at most 8 MiB |
| `name` | 1 to 120 characters after trimming; no control characters |
| `item_id` | 1 to 64 characters of `A–Z a–z 0–9 . _ -`; unique within a version |
| `origin` | `authored` or `saved_case` |
| `input` | required for `authored`, at most 32 KiB; not accepted for `saved_case` |
| `output` | optional, at most 32 KiB, any origin |
| `expected_label` | optional, at most 256 bytes after trimming; a blank label is treated as absent |
| `required_facts` | optional, 1 to 16 strings of at most 1 KiB each |
| one example's stored body | at most 100 KiB |
| `saved_case` | `{case_id, case_version}` naming a saved test case version in the same gateway |
| `idempotency_key` | 1 to 128 characters of `A–Z a–z 0–9 . _ : -` |

Unknown fields in an example are rejected with reason `unknown_field`, so an upload can't
carry run, price or source fields beside its text.

## Identities and digests

Every published version is immutable. Four values identify what a version holds.

| Value | What it is |
|---|---|
| `dataset_id` | `ds_` followed by 12 base58 characters, minted by the server |
| `version` | a whole number from 1, contiguous per dataset; publishing adds the next one |
| `items_digest` | `sha256:` and 64 hex digits over every example's metadata and body digest |
| `body_digest` | `sha256:` and 64 hex digits over one example's stored body |

A version's `ref` is `{dataset_id, version, items_digest}`, the triple anything that pins
a version records.

**`body_digest`** is the SHA-256 of the example's body object: one JSON document with the
keys `input`, `output` and `required_facts` in that order, each omitted when absent, no
insignificant whitespace, UTF-8, no HTML escaping. For a `saved_case` example the body
holds no `input`, because the input is read from the case.

**`items_digest`** is computed without any body, so a reader without `content.read` can
check it:

1. Take each example's `body_digest`, `expected_label` (or `null`), `item_id`, `origin`
   and `saved_case` (or `null`).
2. Write each as one compact JSON object with those keys in that byte order.
3. Sort the lines by `item_id` bytes and end each with a newline.
4. Hash the result with SHA-256 and prefix `sha256:`.

A label change moves `items_digest` and leaves `body_digest` alone; a text change moves
both. Reordering the examples in a request changes neither.

<Details summary="Check an items_digest yourself with jq">

Pipe a version read (`GET …/versions/{version}`, or `tokentriage cloud dataset version
get … --json`) through this. It rebuilds the lines from the response and hashes them.

```console
$ jq -c -S '.data.version.examples | sort_by(.item_id)[] |
    {body_digest, expected_label, item_id, origin, saved_case}' v1.json |
    sha256sum
```

For the tutorial's version 1, saved as `v1.json`, the hash it prints is
`ca4558a01fe7bb553f959cd9bcb55d5e1b5c47d5c295a90328a09c95d0f5ccb7`: the version's
`items_digest` without its `sha256:` prefix.
On Windows, add `-b` to the first `jq` so it writes `\n` rather than `\r\n` line endings.
The recipe assumes plain-text labels: the server's encoder escapes U+2028 and U+2029,
which jq writes raw.

</Details>

## Operations

All paths are relative to
`/portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}`.
Every response is the lane's envelope,
`{"schema":"customer/v1","request_id":…,"kind":…,"data":…}`.

| Operation | Method | Path | Action | Success |
|---|---|---|---|---|
| `customer.datasets.create` | POST | `/datasets` | `datasets.write` | 201 `dataset`; 200 on replay |
| `customer.datasets.list` | GET | `/datasets` | `metadata.read` | 200, heads with a `current` summary, no examples |
| `customer.datasets.get` | GET | `/datasets/{dataset_id}` | `metadata.read` | 200 `dataset` at its current version, with examples |
| `customer.datasets.versions.create` | POST | `/datasets/{dataset_id}/versions` | `datasets.write` | 201 `dataset`; 200 on replay |
| `customer.datasets.versions.list` | GET | `/datasets/{dataset_id}/versions` | `metadata.read` | 200, version summaries, no examples |
| `customer.datasets.versions.get` | GET | `/datasets/{dataset_id}/versions/{version}` | `metadata.read` | 200 `dataset` at that version, with examples |

The two lists page with `limit` and `page_token`, and return `next_page_token` while more
remain. The single-resource answers and both writes carry `kind: "dataset"`.

The same six operations in each client:

| Operation | Python | TypeScript | CLI |
|---|---|---|---|
| create | `datasets.create(scope, *, idempotency_key, name, examples)` | `datasets.create(scope, body)` | `cloud dataset create` |
| list | `datasets.list(scope, *, limit, page_token)` | `datasets.list(scope, options)` | `cloud dataset list` |
| get | `datasets.get(scope, dataset_id)` | `datasets.get(scope, datasetId)` | `cloud dataset get` |
| versions create | `datasets.versions_create(scope, dataset_id, *, idempotency_key, expected_version, examples)` | `datasets.versionsCreate(scope, datasetId, body)` | `cloud dataset version create` |
| versions list | `datasets.versions_list(scope, dataset_id, *, limit, page_token)` | `datasets.versionsList(scope, datasetId, options)` | `cloud dataset version list` |
| versions get | `datasets.versions_get(scope, dataset_id, version)` | `datasets.versionsGet(scope, datasetId, version)` | `cloud dataset version get` |

In Python, `scope` is `DatasetScope(org, ws, app, gw, env)`; in TypeScript it's
`{ org, ws, app, gw, env }`.

### Write requests

A create carries `idempotency_key`, `name` and `examples`. A new version carries
`idempotency_key`, `expected_version` and `examples`, and no name. A new version replaces
the whole example list; there's no patch.

- **`idempotency_key`** is required and chosen by the caller. The body field is primary;
  an `Idempotency-Key` header is accepted when the field is absent. The same key with the
  same request returns the original resource with HTTP 200. The comparison covers the
  name (for a create) or `expected_version` (for a version) plus the `items_digest` of the
  examples, so reordering them is still the same request. None of the clients generates
  a key.
- **`expected_version`** must equal the dataset's current version. Anything else is
  `dataset_version_conflict` and nothing is published.

### Response fields

A `dataset` document has two parts. `data.dataset` is the head: `dataset_id`, the five
scope IDs, `name`, `current_version`, `state`, `created_by`, `created_at`, `updated_at`.
`data.version` is one version:

| Field | Meaning |
|---|---|
| `version`, `ref`, `items_count`, `items_digest` | the version's identity, as described in the preceding section |
| `sources` | example counts by origin: `{authored, saved_case}` |
| `published_by`, `published_at` | the person or key ID that published the version, and when |
| `examples` | the examples in `item_id` byte order |
| `content_withheld` | `true` when any example's body was left out of this answer |

Each example carries `item_id`, `origin`, `expected_label` (absent when none),
`saved_case` (or `null`), `body_digest` and `content_state`. The body fields `input`,
`output` and `required_facts` are present only when `content_state` is `available`.
`created_by` and `published_by` carry an actor ID, not an email address.

| `content_state` | The body is… |
|---|---|
| `available` | in the answer |
| `withheld` | left out because the caller has no `content.read` here |
| `excluded` | left out because the workspace is excluded |
| `purged` | deleted; metadata remains |
| `not_retained` | not held in a form this deployment can open |
| `unreadable` | left out because its stored record can't be read (for example, the record is missing); an operator should look at it |
| `digest_mismatch` | left out because the stored bytes don't match the recorded digest; an operator should look at it |

When a limitation applies to both a saved case's input and the example's own body, the
answer names the more final one, in the order `withheld`, `not_retained`, `excluded`,
`unreadable`, `digest_mismatch`, `purged`.

## Errors

Every error is an `application/problem+json` document with `type` (the code below),
`title`, `status`, `detail`, `instance` and `request_id`. Quote `request_id` to support.

| Code | Status | When |
|---|---|---|
| `datasets_not_enabled` | 501 | the deployment doesn't run hosted datasets |
| `dataset_not_found` | 404 | the dataset doesn't exist, or exists outside the caller's scope; the two are the same answer |
| `dataset_version_not_found` | 404 | the dataset is visible and has no such version |
| `dataset_version_conflict` | 409 | `expected_version` isn't the current version |
| `idempotency_payload_mismatch` | 409 | the idempotency key was used before with a different request |
| `dataset_example_invalid` | 422 | one or more examples break a field rule; see `problems` |
| `dataset_bounds_exceeded` | 422 | fewer than 1 or more than 50 examples, or a request over 8 MiB |
| `forbidden` | 403 | the caller can see the scope and lacks the action |
| `workspace_excluded` | 409 | the workspace is excluded; reads and writes both stop |
| `validation_failed` | 422 | a malformed request outside the examples: a scope, the key, the name, an unknown top-level field |
| `content_capture_not_permitted` | 403 | create or publish only: the workspace doesn't retain dataset examples |
| `retention_ceiling_exceeded` | 422 | create or publish only: the retention asked for is longer than the workspace allows |
| `dataset_content_digest_mismatch` | 409 | create or publish only: stored content didn't match its recorded digest; nothing is published. Quote `request_id` to support |

A `dataset_example_invalid` answer carries `problems`, a list of
`{item_id, field, reason}`. `item_id` is empty for a problem with the list as a whole or
with an `item_id` that is itself invalid. No problem echoes a body.

| `reason` | Meaning |
|---|---|
| `duplicate_item_id` | two examples share an `item_id` |
| `invalid_item_id` | the `item_id` breaks the grammar |
| `invalid_origin` | the origin isn't `authored` or `saved_case`, or an `authored` example names a saved case |
| `input_required` | an `authored` example has no `input` |
| `input_forbidden_for_saved_case` | a `saved_case` example carries its own `input` |
| `saved_case_not_found` | the saved case reference is malformed, unknown, or in another gateway |
| `too_long` | a field, the fact list or the stored body is over its bound |
| `unknown_field` | the example carries a field this contract doesn't define |

`saved_case_not_found` covers every way a reference can fail, so it can't be used to learn
whether a case exists in another gateway.

## Lifecycle

A dataset's `state` is `active`, `deleting` or `purged`, and moves forward only. The
workspace's data-exit process drives the last two; there is no per-dataset delete in
this release.

- **Workspace exclusion.** Once a workspace is excluded, every dataset read and write
  in it answers `workspace_excluded`, and example bodies read `excluded`.
- **Export.** An export bundle carries one `dataset` row per dataset version in the
  `runs` section, listed with the other evaluation kinds in the following table. Each
  example's body is a content object: it's listed under `content_inventory` and
  fetched separately under `content.read`, and `content_bodies` answers `withheld`.
- **Purge.** A purge deletes every dataset in the workspace whole: its name, its
  versions, its examples and their bodies. No record of a version is kept behind.

[Export and delete your data](/docs/build/export-and-delete) describes the bundle,
exclusion and purge for the workspace as a whole.
[Export and remove evaluation evidence](/docs/build/manage-evaluation-evidence) walks
through reading these rows.

### Evaluation evidence in an export bundle

Evaluation evidence has no export section of its own. Its rows join two existing
sections, `runs` and `comparisons`, so both sections are heterogeneous: each holds
rows of more than one shape.

- A row with no `kind` member is a row those sections carried before evaluation kinds
  existed: a test run keyed by `run_id` in `runs`, or a comparison record keyed by
  `comparison_id` in `comparisons`. Those rows are written unchanged.
- A row with a `kind` member is an evaluation row. Its top-level members share no name
  with the older rows.
- A section's manifest `rows` counts every row of every kind. A reader dispatches on
  `kind`, and skips and counts a kind it doesn't know rather than failing, because a
  later release can add kinds.

Every evaluation row has the same envelope:

```json
{
  "kind": "…",
  "id": "…",
  "version": 1,
  "organization_id": "…",
  "workspace_id": "…",
  "created_at": "…",
  "document": {}
}
```

`created_at` is an RFC 3339 instant in UTC. `version` is omitted for the kinds with no
versions. The kinds form a closed set in this release, in this order:

| Section | `kind` | One row is | `id` / `version` / `created_at` | `document` |
|---|---|---|---|---|
| `comparisons` | `comparison_definition` | one comparison definition version | the definition's id / its version number / when the version was published | `head` (the definition) and `version` |
| `comparisons` | `judge_profile` | one judge profile version | the profile's id / its version number / when the version was published | `head`, `version`, and `calibrations`: the calibration links of this version |
| `comparisons` | `decision_provenance` | one definition run link that names a comparison run | the comparison run's id / none / when the run started | `comparison_id`, `comparison_run_id`, `link` (the run link) and `provenance` |
| `runs` | `evaluation_definition` | one evaluation definition version | the definition's id / its version number / when the version was published | `head` (the definition) and `version` |
| `runs` | `evaluation_run` | one offline run | the run's id / none / when the run was created | `run`, `items` (the run's dataset items) and `results` (its per-item records) |
| `runs` | `dataset` | one dataset version | the dataset's id / its version number / when the version was published | `head`, `version` without its examples, and `examples`: each example's `item_id`, `origin`, `expected_label`, `saved_case`, `body_digest` and `body_ref` |

The rows follow five rules:

- A resource with no versions yet exports one row whose `document` holds only `head`,
  with no `version`.
- No row carries an example body, an input, an output, the required facts or an
  observed label. `expected_label` is metadata and is exported.
- An example's `body_ref` equals the `ref` of its body's `content_inventory` row, which
  is how a reader joins the two.
- A `decision_provenance` row whose definition version can't be found is still
  exported, with `provenance.state` set to `unavailable`.
- A `ceiling_micro_usd` is exported as the API returns it. It's a configured ceiling,
  not spend.

Each section keeps its own grant, re-checked when the bundle is packaged: `runs` needs
`metadata.read` and `comparisons` needs `comparisons.run`. Judge profiles are exported
in `comparisons`, so a bundle carries them only for a holder of `comparisons.run`,
although the API reads them under `metadata.read`. A section past its store's page
bound answers `unavailable` with `bound_exceeded` and carries no rows of any kind.

### Rolling back below the datasets release

The dataset schema is additive; there's no schema rewind. The supported rollback keeps a
datasets-aware binary in place and disables the dataset plane instead: omit the content
bucket configuration, and every dataset operation answers `datasets_not_enabled` (501).

Before rolling the control plane binary back below the datasets release, run
`select count(*) from foundation_content_objects where kind = 'dataset_example'`. A
non-zero count means don't roll the binary back — disable the plane instead. A
pre-datasets binary stalls that workspace's purge and fails its export until the binary
rolls forward again.

A Foundation change (maintenance FMT-045, decision DEC-022) that tolerates unknown
content kinds is scheduled to land before the datasets release. Once it ships, that
release becomes the rollback floor and this check is no longer needed.

## Provenance of an example

An `authored` example's input was typed into the request. A `saved_case` example refers
to a saved test case version in the same gateway, and its input is read from that case
rather than copied. A supplied `output` is declared by whoever wrote the request and is
never observed: creating or versioning a dataset calls no provider and costs nothing.

## Provenance of a run

A comparison run started from a definition (`runs.start` above) carries a `provenance`
alongside its evidence, on `results.get` and beside each `decisions.create` and
`decisions.get` answer: `{state, definition?, judge_profile?}`.

| `state` | Meaning |
|---|---|
| `definition` | the run is linked to the comparison-definition version above; `definition` carries `{definition_id, version, definition_digest}`, plus `drifted_plan_digest` when the run committed a plan other than the one that version approved |
| `direct` | no comparison-definition names this run in the caller's scope; it may still be a plain comparison run started without one |
| `unavailable` | the store or the workspace fence couldn't answer the question |

`judge_profile` (`{profile_id, version, profile_digest}`) is present only alongside
`state: "definition"`, and only when that definition version pinned one. Nothing here is
copied from the definition: every field is read from the link and the version at answer
time, so a caller compares a run's `provenance.definition.definition_digest` against the
definition's own current digest to see whether the version it ran against is still the
one published.

## Offline evaluations

An offline evaluation checks the outputs a dataset version already holds against that
version's expected labels. A **definition** pins one dataset version and says where the
label sits in each output, how to compare it, and what result is acceptable. A **run**
scores every example of the pinned version once and records one outcome per example.
The only mode is `offline_exact_label`: no provider is called, and nothing is generated.

### Evaluation actions

Reads use the existing `metadata.read`, resolved at the gateway environment the
definition or run belongs to; there is no `evaluations.read` action. Two new actions
cover writing definitions and running them, and the example bodies stay behind the
existing `content.read`.

| Action | Allows |
|---|---|
| `metadata.read` | list and read definitions, definition versions, runs and items: specs, counts, outcomes, reasons, expected labels, digests |
| `evaluations.write` | create a definition and publish its next version (with `metadata.read`) |
| `evaluations.run` | plan a run (preflight), start one (with `metadata.read` and `content.read`), and cancel one |
| `content.read` | required to start a run, because the run opens each example's supplied output under the starter's authority; in an item read, it adds `observed_label` |

`evaluations.write` and `evaluations.run` behave like `datasets.write`: they're granted
only by an explicit `role_version: 2` binding, not by the default stamp, and they join
role version 2 with the release that registers the evaluation operations. A binding
issued under an earlier role version has to be reissued at `role_version: 2`. Readers
need no new binding.

| Role | `metadata.read` | `evaluations.write`, `evaluations.run` (explicit `role_version: 2`) |
|---|---|---|
| `workspace_admin` | yes | yes |
| `app_developer` | yes | yes |
| `org_owner`, `org_admin`, `analyst` | yes | no |
| every other role, including `runtime_service` and `test_executor` | no | no |

A run works under the authority of the person or key that started it. Before each
example it checks that the starter still holds `evaluations.run`, `metadata.read` and
`content.read` in the environment; if not, the run ends `failed` with
`authority_revoked`. It doesn't fall back to a service identity.

### Evaluation bounds

A request outside these bounds is answered with `evaluation_definition_invalid` (a
field, listed in `problems`), `evaluation_mode_not_supported` (a mode or label source
kind), or `validation_failed` (the request as a whole), and nothing is stored.

| Item | Bound |
|---|---|
| `mode` | `offline_exact_label` only |
| `name` | as a dataset name: 1 to 120 characters after trimming, no control characters; on create only |
| `dataset` | `{dataset_id, version, items_digest}` of a version in the same gateway environment; the digest must equal the stored one |
| `label_source` | `{"kind":"plain_text"}`, or `{"kind":"json_pointer","pointer":"/label"}` with one RFC 6901 token of at most 64 bytes |
| `case_sensitive` | required, `true` or `false`; there is no default |
| `acceptance` | required, `{"max_mismatched": 0..50, "max_unscored": 0..50}` |
| examples per run | every example of the pinned version, so 1 to 50 |
| items per page | at most 50 |
| an extracted label | at most 256 bytes |
| `idempotency_key` | as for datasets: 1 to 128 characters of `A–Z a–z 0–9 . _ : -` |

The comparison rule is `label_exact/v1`. It trims Unicode white space from both ends of
the extracted label and of the expected label, then compares bytes. With
`case_sensitive: false` it compares them under Unicode case folding instead.

Extraction depends on `label_source`:

- **`plain_text`**: the whole output, trimmed, is the label. A blank output has no label.
- **`json_pointer`**: the output must be one JSON object with no trailing data and no
  duplicate member. The label is the string member the pointer names. A missing member
  means no label. Invalid JSON, a value that isn't an object, a member that isn't a
  string, or a label over 256 bytes makes the label malformed.

### Evaluation identities and digests

Every definition version, and every finished run, is immutable.

| Value | What it is |
|---|---|
| `definition_id` | `edef_` followed by 12 base58 characters, minted by the server |
| `version` | a definition version: a whole number from 1, contiguous; publishing adds the next one |
| `spec_digest` | `sha256:` and 64 hex digits over the version's specification |
| `run_id` | `erun_` followed by 12 base58 characters, minted by the server |
| plan `digest` | `sha256:` and 64 hex digits over the plan a preflight answers |
| `result_digest` | `sha256:` and 64 hex digits over a finished run's result; `null` until the run finishes |

A definition version's `ref` is `{definition_id, version, spec_digest}`; runs, plans and
starts name a definition by it. Each digest is the SHA-256 of one canonical JSON object,
written as the dataset digests are: keys in byte order, no insignificant white space,
UTF-8, no HTML escaping, no trailing newline.

| Digest | Covers |
|---|---|
| `spec_digest` | `acceptance`, `case_sensitive`, `dataset`, `label_source`, `mode`, `rule` |
| plan `digest` | `dataset`, `definition` (the `ref`), `rule`, `selected` |
| `result_digest` | `dataset`, `definition`, `items` (each `{body_digest, item_id, outcome, reason}`, in `item_id` order, including `not_evaluated`), `rule`, `state` |

A finished run also carries `ref`: the operation `customer.evaluations.runs.get`, the
five scope IDs, `run_id`, `definition`, `dataset` and `result_digest`. It is `null` until
the run finishes, and it grants nothing to whoever holds it.

### Evaluation operations

All paths are relative to
`/portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}/evaluations`,
and every response is the lane's envelope.

| Operation | Method | Path | Action | Success |
|---|---|---|---|---|
| `customer.evaluations.definitions.create` | POST | `/definitions` | `evaluations.write`, `metadata.read` | 201 `evaluation-definition`; 200 on replay |
| `customer.evaluations.definitions.list` | GET | `/definitions?dataset_id=` | `metadata.read` | 200 |
| `customer.evaluations.definitions.get` | GET | `/definitions/{definition_id}` | `metadata.read` | 200 |
| `customer.evaluations.definition_versions.create` | POST | `/definitions/{definition_id}/versions` | `evaluations.write`, `metadata.read` | 201; 200 on replay |
| `customer.evaluations.definition_versions.list` | GET | `/definitions/{definition_id}/versions` | `metadata.read` | 200 |
| `customer.evaluations.definition_versions.get` | GET | `/definitions/{definition_id}/versions/{version}` | `metadata.read` | 200 |
| `customer.evaluations.runs.preflight` | POST | `/runs/preflight` | `evaluations.run` | 200 `evaluation-run-plan`; writes nothing |
| `customer.evaluations.runs.start` | POST | `/runs` | `evaluations.run`, `metadata.read`, `content.read` | 202 `evaluation-run`; 200 on replay |
| `customer.evaluations.runs.list` | GET | `/runs?definition_id=&state=` | `metadata.read` | 200 |
| `customer.evaluations.runs.get` | GET | `/runs/{run_id}` | `metadata.read` | 200 `evaluation-run` |
| `customer.evaluations.runs.cancel` | POST | `/runs/{run_id}/cancel` | `evaluations.run` | 200 `evaluation-run` |
| `customer.evaluations.items.list` | GET | `/runs/{run_id}/items?outcome=` | `metadata.read` | 200, at most 50 items |

The lists page with `limit` and `page_token`. The same twelve operations in each client:

| Operation | Python | TypeScript | CLI |
|---|---|---|---|
| definitions create | `evaluations.definitions_create` | `evaluations.definitionsCreate` | `cloud evaluation definition create` |
| definitions list | `evaluations.definitions_list` | `evaluations.definitionsList` | `cloud evaluation definition list` |
| definitions get | `evaluations.definitions_get` | `evaluations.definitionsGet` | `cloud evaluation definition get` |
| definition versions create | `evaluations.definition_versions_create` | `evaluations.definitionVersionsCreate` | `cloud evaluation definition version create` |
| definition versions list | `evaluations.definition_versions_list` | `evaluations.definitionVersionsList` | `cloud evaluation definition version list` |
| definition versions get | `evaluations.definition_versions_get` | `evaluations.definitionVersionsGet` | `cloud evaluation definition version get` |
| runs preflight | `evaluations.runs_preflight` | `evaluations.runsPreflight` | `cloud evaluation run preflight` |
| runs start | `evaluations.runs_start` | `evaluations.runsStart` | `cloud evaluation run start` |
| runs list | `evaluations.runs_list` | `evaluations.runsList` | `cloud evaluation run list` |
| runs get | `evaluations.runs_get` | `evaluations.runsGet` | `cloud evaluation run get`, or `run wait` to poll until it finishes |
| runs cancel | `evaluations.runs_cancel` | `evaluations.runsCancel` | `cloud evaluation run cancel` |
| items list | `evaluations.items_list` | `evaluations.itemsList` | `cloud evaluation run items` |

In Python, `scope` is `EvaluationScope(org, ws, app, gw, env)`; in TypeScript it's
`{ org, ws, app, gw, env }`. The CLI's `run preflight` and `run start` take
`--definition` and `--version` and read the `ref` for you.

**Write requests.** A definition create carries `idempotency_key`, `name`, `mode`,
`dataset`, `label_source`, `case_sensitive` and `acceptance`. A new version carries the
whole specification again without `name`, plus `expected_version`, which must equal the
current head. A preflight carries `{definition: ref}`. A start carries
`idempotency_key`, `definition` and `approved_digest`, the plan `digest` the caller
read. Cancel takes no key and no body fields. Idempotency works as for datasets: the
same key with the same request answers the original with HTTP 200, and none of the
clients generates a key.

**A plan** answers `digest`, `definition`, `dataset`, `selected`,
`expected_label_absent`, `planned_provider_calls` (always 0), `provider_spend`,
`content_read` (whether the caller holds `content.read` there) and `active_run_id` (the
run already queued or running here, or `null`).

**A run** answers `run_id`, the five scope IDs, `definition`, `dataset`, `mode`, `state`,
`failure`, `counts`, `unscored_reasons` (a count per reason), `acceptance`
(`{rule, outcome}`), `provider_spend`, `initiated_by`, `cancelled_by`, `created_at`,
`started_at`, `finished_at`, `result_digest` and `ref`.

| Count | Meaning |
|---|---|
| `selected` | examples in the pinned dataset version; `matched + mismatched + unscored + not_evaluated` |
| `scored` | `matched + mismatched` |
| `matched`, `mismatched` | the extracted label equals, or differs from, the expected label under the rule |
| `unscored` | recorded without a comparison; the reason is counted in `unscored_reasons` |
| `not_evaluated` | not yet recorded, or not recorded because the run stopped |

`acceptance.outcome` is `met` or `not_met` for a `complete` or `partial` run, judged
against `max_mismatched` and `max_unscored`, and `undetermined` in every other state.

**An item** answers `item_id`, `origin`, `outcome`, `reason` (for `unscored` only),
`expected_label`, `body_digest`, `content_state`, `observed_label` and `recorded_at`.
`observed_label` is present only when `content_state` is `available` and a label could
be extracted. It isn't stored: each read opens the body again under the reader's
`content.read`, so exclusion and purge reach it. A `not_evaluated` item carries no
`reason` and no `recorded_at`.

### Provider spend

Every plan and every run carries `"provider_spend": {"basis": "no_dispatch",
"dispatches": 0}`. It has no amount field, and the missing amount is a statement rather
than a gap: an offline run composes no dispatcher, provider connection or price lookup,
so nothing is sent and nothing is priced. It isn't a zero-dollar charge, and it says
nothing about what the supplied outputs cost when they were first produced.

### Unscored reasons

An example that can't be compared is recorded as `unscored` with one reason. The set is
closed, and the first reason that applies, in this order, is the one recorded.

| `reason` | The example… |
|---|---|
| `expected_label_absent` | has no `expected_label` in the dataset version |
| `content_excluded` | has a body the workspace's exclusion withholds |
| `content_purged` | has a body that was purged |
| `content_not_retained` | has no body this deployment can open |
| `output_absent` | has no supplied output |
| `label_absent` | has an output with no label: blank once trimmed, or no member at the pointer |
| `label_malformed` | has an output the pointer can't read: not one JSON object, a non-string member, or a label over 256 bytes |

An unscored example counts toward `max_unscored`, not as a mismatch.

### Run states

| `state` | Meaning |
|---|---|
| `queued` | admitted and waiting for the worker; every example is `not_evaluated` |
| `running` | the worker is recording outcomes |
| `complete` | every example recorded, none unscored |
| `partial` | every example recorded, at least one unscored |
| `failed` | stopped early; `failure.code` says why |
| `cancelled` | stopped by `runs.cancel` |

A run moves `queued → running → complete | partial | failed | cancelled`, or straight
from `queued` to `cancelled` or `failed`. The last four are final. One run at a time may
be `queued` or `running` per gateway environment.

`failure.code` is one of `authority_revoked` (the starter's grant changed),
`workspace_excluded`, `dataset_unavailable`, `digest_mismatch` (a pinned digest no
longer matches), `attempts_exhausted` (three worker attempts didn't finish it) or `worker_fault`.
Items recorded before a run fails or is cancelled stay. A cancel ends the run at once,
and no item is written after it. A second cancel answers the cancelled run again.

A run that is interrupted, such as by a worker restart, is resumed by the next claim,
which scores only the examples not yet recorded; the `result_digest` doesn't depend on
how many times that happened.

### Evaluation errors

The problem document has the dataset errors' shape. An `evaluation_definition_invalid`
answer also carries `problems`, a list of `{field, reason}` that doesn't echo a value,
and `evaluation_quota_exhausted` names the active run.

| Code | Status | When |
|---|---|---|
| `evaluations_not_enabled` | 501 | the deployment doesn't run offline evaluations |
| `evaluation_worker_unavailable` | 503 | no evaluation worker runs; a start is turned away, reads answer |
| `evaluation_definition_not_found` | 404 | the definition doesn't exist, or exists outside the caller's scope; the same answer |
| `evaluation_definition_version_not_found` | 404 | the definition is visible and has no such version |
| `evaluation_run_not_found` | 404 | the run doesn't exist, or exists outside the caller's scope; the same answer |
| `evaluation_definition_version_conflict` | 409 | `expected_version` isn't the current head |
| `evaluation_subject_stale` | 409 | the dataset, definition or plan digest differs from the stored or recomputed one; plan again |
| `evaluation_run_finished` | 409 | a cancel of a `complete`, `partial` or `failed` run |
| `evaluation_mode_not_supported` | 422 | any mode but `offline_exact_label`, or any `label_source.kind` but the two listed |
| `evaluation_definition_invalid` | 422 | one or more fields break a rule; see `problems` |
| `evaluation_quota_exhausted` | 429 | a run of this gateway environment is queued or running; the answer names it as `active_run_id` |
| `idempotency_payload_mismatch` | 409 | the idempotency key was used before with a different request |
| `dataset_not_found` | 404 | the pinned dataset doesn't exist in scope, or is being deleted or purged |
| `dataset_version_not_found` | 404 | the pinned dataset has no such version |
| `forbidden` | 403 | the caller can see the scope and lacks the action |
| `content_read_denied` | 403 | a start by a caller without `content.read` in the environment |
| `workspace_excluded` | 409 | the workspace is excluded; reads and writes stop, and a running run fails |
| `validation_failed` | 422 | a malformed request outside the definition fields: a scope, the key, a filter |

| `problems[].reason` | Meaning |
|---|---|
| `unknown_field` | the request carries a field this contract doesn't define |
| `pointer_invalid` | the pointer isn't one RFC 6901 token of at most 64 bytes |
| `case_sensitive_required` | `case_sensitive` is missing |
| `acceptance_invalid` | `acceptance` is missing, or a limit is outside 0 to 50 |
| `name_invalid` | the name is blank, too long or holds a control character |

### Evaluation lifecycle

Definitions, runs and items follow the workspace's data-exit process, as datasets do.
An exclusion stops every read and write and fails a running run with
`workspace_excluded`. An export carries definitions as `evaluation_definition` rows
and runs, with their items, as `evaluation_run` rows in the `runs` section, as listed
under [evaluation evidence in an export bundle](/docs/reference/managed-evaluations#evaluation-evidence-in-an-export-bundle).
A purge deletes them with the rest of the workspace. After a dataset's bodies
are purged, an item read has no `observed_label` to show, and a finished run's
recorded outcomes and `result_digest` stay as they were.

## Comparison definitions

A comparison definition saves the setup of a
[provider comparison](/docs/build/compare-cost-and-quality) over one dataset version, so
you can plan and start the same comparison again. A **definition** has a name and a kind,
`labeled` or `judged`, and owns one comparison (`cmp_…`). Each **definition version** pins
a dataset version, the applied baseline release, the candidate draft revision, the scorer,
the acceptance rule and an optional spend ceiling, and publishes that selection as the
comparison's sample set. A **run** is an ordinary comparison run (`crun_…`) on that
comparison, and the definition keeps a link to it.

Unlike an offline evaluation, a run calls providers. It makes one paid dispatch per
example per arm, and a judged definition adds one judge call per example per arm. Saving
a definition or a version, and asking for a plan, call no provider and cost nothing.

### Comparison definition actions

There is no new action. Every operation declares the existing `comparisons.run`, resolved
at the gateway environment the definition belongs to. The other requirements are checked
in the same environment:

| Requirement | Needed for |
|---|---|
| `comparisons.run` | every operation, reads included |
| `tests.run` and `metadata.read` | create, version create, preflight and start; the pinned dataset version is read as the caller |
| `content.read` | preflight and start, for the saved test cases' inputs; without it the answer is `content_read_denied` |
| a current `comparison` grant on each arm's provider connection | preflight and start, for this application and environment; an `application` grant alone isn't enough, and the answer is `forbidden` |
| a signed-in session | preflight and start; a management key is answered `app_key_cannot_select_subject` (HTTP 403) |

The last row means the CLI and both SDKs, which authenticate with a management key, can
save, list and read definitions and list their runs, but can't plan or start one. A
definition always aims its candidate arm at a saved draft revision, and a management key
may not select a comparison subject.

| Role | Reads | Writes, preflight and start |
|---|---|---|
| `workspace_admin`, `app_developer` | yes | yes |
| `org_owner`, `org_admin` | yes | no: they hold no `tests.run` |
| every other role, including `analyst`, `runtime_service` and `test_executor` | no | no |

`content.read` and the connection grant are separate in every case. Reading a run's
evidence afterwards uses the comparison reads, which keep their own `cost.read` and
`content.read` rules.

### Comparison definition bounds

A request outside these bounds is answered with `comparison_definition_ineligible` (an
example of the pinned version, listed in `problems`) or `validation_failed` (the request
as a whole), and nothing is stored.

| Item | Bound |
|---|---|
| `name` | 1 to 120 bytes after trimming; on create only |
| `kind` | `labeled` or `judged`; on create only, and fixed for the definition |
| `dataset` | `{dataset_id, version, items_digest}` of a version in the same gateway environment; the digest must equal the stored one |
| examples | 2 to 50, every one `saved_case`, and no saved test case version twice |
| `expected_label` | labeled only: every example needs one, at most 120 bytes, with no CR, LF or NUL |
| `critical_item_ids` | optional; unique item IDs of the pinned version |
| `required_facts` | judged only, optional: `{item_id: [fact, …]}`, at most 20 facts of at most 200 bytes each |
| `baseline` | `{"kind":"applied_release","release_id":"rel_…"}` |
| `candidate` | `{"kind":"saved_draft_revision","draft_revision":7}` |
| `scorer`, `rule` | the comparison's scorer and acceptance rule, verbatim; `rule.kind` equals `kind` |
| `judge_profile` | judged only, optional: `{profile_id, version, profile_digest}` of a current judge profile version, in place of `scorer`; giving both is `validation_failed` |
| `ceiling_micro_usd` | optional, a positive whole number; absent means the comparison default |
| `idempotency_key` | 1 to 128 characters of `A–Z a–z 0–9 . _ : -` |
| request body | at most 512 KiB; an unknown field is `validation_failed`, naming it |

Only `saved_case` examples can be compared in this release. A saved test case is a
request the gateway already holds, so each arm can send it again. An `authored` example's
text was typed into the dataset, and the comparison has no request to replay for it. To
compare it, save it as a test case first (`customer.cases.create`), then publish a dataset
version that refers to it as `saved_case`.

The server doesn't read example bodies to build the sample set. For a labeled definition,
each example's dataset `expected_label` becomes the case's expected label. For a judged
definition, each case's reference notes are the saved test case's own input, and the
required facts are the ones the request declares per item. Cases are numbered in
`item_id` byte order.

### Comparison definition identities and digests

| Value | What it is |
|---|---|
| `definition_id` | `cdef_` followed by 12 base58 characters, minted by the server |
| `version` | a whole number from 1, contiguous per definition; publishing adds the next one |
| `definition_digest` | `sha256:` and 64 hex digits over the version: its dataset `ref`, items, subjects, scorer, rule, ceiling, comparison and sample set, plus the judge profile pin when one was given |
| `comparison_id` | the definition's one comparison, `cmp_…`, created with the definition |
| `plan_digest` | `sha256:` and 64 hex digits over a preflight's plan, answered by each preflight |

A definition version's `ref` is `{definition_id, version, definition_digest}`. Every
version is immutable. Publishing a selection that a previous version already published
answers the same sample set version.

`plan_digest` covers the definition `ref`, the comparison, the sample set version and
digest, both arms, the scorer, the rule, the planned arm and judge calls, the three spend
ceilings and the budget plan. If any of those moves, the next preflight answers a
different digest.

### Comparison definition operations

All paths are relative to
`/portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}`,
and every response is the lane's envelope. Every row declares `comparisons.run`; the
section on actions lists what else each one needs.

| Operation | Method | Path | Success |
|---|---|---|---|
| `customer.comparison_definitions.create` | POST | `/comparison-definitions` | 201 `comparison-definition`; 200 on replay |
| `customer.comparison_definitions.list` | GET | `/comparison-definitions` | 200 `comparison-definitions`, heads with a current version summary |
| `customer.comparison_definitions.get` | GET | `/comparison-definitions/{definition_id}` | 200 `comparison-definition` at its current version |
| `customer.comparison_definitions.versions.create` | POST | `/comparison-definitions/{definition_id}/versions` | 201 `comparison-definition`; 200 on replay |
| `customer.comparison_definitions.versions.list` | GET | `/comparison-definitions/{definition_id}/versions` | 200 `comparison-definition-versions` |
| `customer.comparison_definitions.versions.get` | GET | `/comparison-definitions/{definition_id}/versions/{version}` | 200 `comparison-definition-version` |
| `customer.comparison_definitions.runs.preflight` | POST | `/comparison-definitions/{definition_id}/runs/preflight` | 200 `comparison-definition-preflight`; writes nothing, calls no provider |
| `customer.comparison_definitions.runs.start` | POST | `/comparison-definitions/{definition_id}/runs` | 202 `comparison-definition-run`, also on replay |
| `customer.comparison_definitions.runs.list` | GET | `/comparison-definitions/{definition_id}/runs` | 200 `comparison-definition-runs`, newest first |

The lists page with `limit` and `page_token`. The same nine operations in each client:

| Operation | Python | TypeScript | CLI |
|---|---|---|---|
| create | `comparison_definitions.create` | `comparisonDefinitions.create` | `cloud comparison definition create` |
| list | `comparison_definitions.list` | `comparisonDefinitions.list` | `cloud comparison definition list` |
| get | `comparison_definitions.get` | `comparisonDefinitions.get` | `cloud comparison definition get` |
| versions create | `comparison_definitions.versions_create` | `comparisonDefinitions.versionsCreate` | `cloud comparison definition version create` |
| versions list | `comparison_definitions.versions_list` | `comparisonDefinitions.versionsList` | `cloud comparison definition version list` |
| versions get | `comparison_definitions.versions_get` | `comparisonDefinitions.versionsGet` | `cloud comparison definition version get` |
| runs preflight | `comparison_definitions.runs_preflight` | `comparisonDefinitions.runsPreflight` | `cloud comparison definition run preflight` |
| runs start | `comparison_definitions.runs_start` | `comparisonDefinitions.runsStart` | `cloud comparison definition run start` |
| runs list | `comparison_definitions.runs_list` | `comparisonDefinitions.runsList` | `cloud comparison definition run list` |

In Python, `scope` is `ComparisonScope(org, ws, app, gw, env)`; in TypeScript it's
`{ org, ws, app, gw, env }`. The preflight and start methods exist in every client, and
each of them sends a management key, so against a deployment those two answer
`app_key_cannot_select_subject`.

**Write requests.** A create carries `idempotency_key`, `name`, `kind`, `dataset`,
`baseline`, `candidate`, `scorer` and `rule`, and optionally `critical_item_ids`,
`required_facts` and `ceiling_micro_usd`. A judged definition may give `judge_profile`
instead of `scorer`; the server derives the scorer from the pinned profile version:

```json
{"idempotency_key": "q-a2-def-1", "name": "A2 cheaper route", "kind": "labeled",
 "dataset": {"dataset_id": "ds_…", "version": 2, "items_digest": "sha256:…"},
 "critical_item_ids": ["refund-2"],
 "baseline": {"kind": "applied_release", "release_id": "rel_…"},
 "candidate": {"kind": "saved_draft_revision", "draft_revision": 7},
 "scorer": {"scorer_id": "label_exact_v1", "scorer_version": 1, "rubric_digest": "sha256:…"},
 "rule": {"kind": "labeled", "at_least_baseline_pass_count": true,
          "no_failure_on_critical_cases": true, "max_unknown_cases": 0},
 "ceiling_micro_usd": 2000000}
```

A new version carries the same fields without `name` and `kind`, plus `expected_version`,
which must equal the current version. Idempotency works as for datasets: the body field
is primary, an `Idempotency-Key` header is accepted when it's absent, the same key with
the same request answers the original, and none of the clients generates a key.

**A definition** answers `data.definition`, the head (`definition_id`, the five scope IDs,
`name`, `kind`, `comparison_id`, `current_version`, `state`, `created_by`, `created_at`,
`updated_at`), and `data.version`: `version`, `ref`, `dataset`, `sample_set`
(`{comparison_id, version, digest, case_count}`), `items` (each `{ordinal, item_id,
case_id, case_version, critical}`), `baseline`, `candidate`, `scorer`, `rule`,
`ceiling_micro_usd`, `published_by` and `published_at`. It carries IDs, pins and links
only; no example body, output, cost or verdict is stored in a definition. A version
that pinned a judge profile also carries `judge_profile: {profile_id, version,
profile_digest}`, alongside the `scorer` that profile derived.

**A preflight** carries `{"version": 1}` and answers `definition` (the `ref`),
`comparison_id`, `plan_digest` and `preflight`. `preflight` is the answer of the
comparison's own preflight: the arms, the sample set, the plan with 2N arm dispatches and
2N or 0 judge calls, the spend ceilings split between arms and judge, the budget plan and
`authority_epoch`.

**A start** carries `version`, `idempotency_key`, `approved_epoch` and
`approved_plan_digest`:

```json
{"version": 1, "idempotency_key": "q-a2-start-1", "approved_epoch": 42,
 "approved_plan_digest": "sha256:…"}
```

`approved_epoch` is the preflight's `authority_epoch` and `approved_plan_digest` its
`plan_digest`. The server plans again before it commits. If the digest it computes isn't
the approved one, nothing starts and the answer is `comparison_definition_plan_stale`.
Use a new idempotency key for every preflight you approve: the same key with a different
approval is a different request, answered `idempotency_payload_mismatch`. The same key
with the same request answers the same run.

A start answers `link` (`definition_id`, `definition_version`, `comparison_id`, `run_id`,
`idempotency_key`, `started_by`, `started_at`) and `run`, the comparison run as
`customer.comparisons.runs.start` answers it. Progress, spend, cancellation, results,
review and decisions use the existing `customer.comparisons.*` operations on the link's
`comparison_id` and `run_id`, and every field below applies to them the same as to a
comparison started without a definition.

**Review and decisions.** `verdicts.review` accepts an optional `reviewed_evidence:
{source_sha256, output_sha256}`, the digest of the source and, for a cell that produced
one, the output the reviewer read; its answer carries `acting_credential_kind`
(`portal_session` or `management_key`) and the two `reviewed_source_sha256` /
`reviewed_output_sha256` digests, kept even when the content itself is withheld. A
digest that names bytes other than the cell's is `review_reading_mismatch` (409).
`decisions.create` accepts an optional `rationale`, the person's words, stored the same
way as a review note; its answer carries `acting_credential_kind` and this run's
`provenance` beside the decision, not inside it. `results.get` carries the run's current
`evidence_digest`, and `releases.create` refuses a decision whose digest no longer
matches with `decision_evidence_changed` (409); record a new decision revision on the
current evidence before naming it in a release. The full verdict and decision shapes,
and the complete error catalogue, are listed at
[Gateway quality evidence](/docs/reference/gateway-quality-evidence).

CLI: `cloud comparison review --read-source-sha256 --read-output-sha256 --note-file` and
`cloud comparison decide --rationale-file --supersedes`. Python:
`comparisons.review(scope, cmp, crun, ..., reviewed_evidence={"source_sha256": ...,
"output_sha256": ...})` and `comparisons.decide(scope, cmp, crun, ..., rationale=b"...")`.
TypeScript: `client.comparisons.review(scope, cmp, crun, { ..., reviewed_evidence })` and
`client.comparisons.decide(scope, cmp, crun, { ..., rationale })`.

### Comparison definition errors

The problem document has the dataset errors' shape. A `comparison_definition_ineligible`
answer also carries `problems`, a list of `{item_id, field, reason}`.

| Code | Status | When |
|---|---|---|
| `comparison_definitions_not_enabled` | 501 | the deployment doesn't compose comparison definitions |
| `judge_profiles_not_enabled` | 501 | a body or version pins a `judge_profile` and the deployment composes no judge profiles |
| `comparison_definition_not_found` | 404 | the definition doesn't exist, is still being created, or exists outside the caller's scope; the same answer |
| `comparison_definition_version_not_found` | 404 | the definition is visible and has no such version |
| `comparison_definition_version_conflict` | 409 | `expected_version` isn't the current version |
| `comparison_definition_ineligible` | 422 | one or more examples of the pinned version can't be compared; see `problems` |
| `comparison_definition_dataset_unavailable` | 409 | the pinned dataset version is being deleted or was purged |
| `comparison_definition_plan_stale` | 409 | start: the approved digest isn't the current plan's, or the committed run differed from the approval (it's cancelled and named in `detail`); preflight and start: the version's pinned judge profile is no longer current |
| `idempotency_payload_mismatch` | 409 | the idempotency key was used before with a different request |
| `app_key_cannot_select_subject` | 403 | preflight or start with a management key |
| `content_read_denied` | 403 | preflight or start by a caller without `content.read` in the environment |
| `release_not_applied` | 409 | preflight or start: the pinned baseline release is no longer the applied one |
| `dataset_not_found`, `dataset_version_not_found` | 404 | the pinned dataset or version isn't visible to the caller |
| `forbidden` | 403 | the caller can see the scope and lacks an action or a connection grant |
| `workspace_excluded` | 409 | the workspace is excluded; reads and writes stop |
| `validation_failed` | 422 | a malformed request: a scope, the key, the name, an unknown field |

The comparison's own answers pass through unchanged: `sample_set_empty` and
`sample_set_too_large` on a write, and on a preflight or start the comparison run set,
including `spend_ceiling_above_remaining`, the `judge_*` codes, `duplicate_comparison_run`,
`authority_revoked` and `environment_not_active`. Review and decisions pass through
`review_reading_mismatch` (409, an attested reading names bytes other than the cell's)
and `decision_evidence_changed` (409, a release names a decision whose verdicts were
revised since), alongside the existing `content_read_denied`, `content_excluded`,
`content_purged` and `decision_superseded`.
[Gateway quality evidence](/docs/reference/gateway-quality-evidence#refusal-codes) lists
each code with its status.

| `problems[].reason` | Meaning |
|---|---|
| `authored_example` | the example is `authored`; save it as a test case and add it as `saved_case` |
| `duplicate_saved_case` | two examples name the same saved test case version |
| `expected_label_required` | a labeled definition's example has no `expected_label` |
| `expected_label_too_long` | the label is over 120 bytes |
| `expected_label_invalid` | the label holds a CR, LF or NUL |
| `required_facts_forbidden` | `required_facts` on a labeled definition |
| `required_facts_too_many` | more than 20 facts for one item |
| `required_fact_invalid` | a fact is blank, over 200 bytes, or holds a CR, LF or NUL |
| `unknown_item_id` | `critical_item_ids` or `required_facts` names an item the version doesn't hold |
| `digest_mismatch` | the request's `items_digest` isn't the stored version's, or (`field: judge_profile`, no `item_id`) the pinned `profile_digest` isn't the version's |
| `profile_not_found` (`field: judge_profile`) | the pinned judge profile or version is unknown, outside the caller's scope, or removed; one concealed answer |
| `profile_not_current` (`field: judge_profile`) | the pinned judge profile version's `pin_state` is no longer `current` |

### Comparison definition lifecycle

A definition's `state` is `active`, `deleting` or `purged`, and moves forward only. The
workspace's data-exit process drives the last two; there's no per-definition delete in
this release.

- **Workspace exclusion.** Once a workspace is excluded, every definition read and write
  in it answers `workspace_excluded`.
- **Export.** An export carries each definition version as a `comparison_definition`
  row and each run link that names a comparison run as a `decision_provenance` row in
  the `comparisons` section, as listed under
  [evaluation evidence in an export bundle](/docs/reference/managed-evaluations#evaluation-evidence-in-an-export-bundle).
- **Purge.** A purge removes the workspace's definitions along with the rest of its data.
  The comparisons and runs a definition started follow the comparison lifecycle.

## Judge profiles

A judge profile saves one [gateway judge](/docs/build/calibrate-gateway-judge) so you can
calibrate it, and cite it, without entering it again. A **profile** has a name. Each
**profile version** pins the judge template, the judge model and its saved connection
version, two bounds, and four controls drawn from one judged comparison's sample set. A
**calibration** is the existing judge calibration (`jcal_…`) on that comparison, started
from a version; the profile keeps a link to it.

The only template is `judge_faithfulness_v1`, scorer version 1. Its prompt and rubric come
from the running build, and a version records their identities; you can't supply your own
prompt or rubric text. A control is two IDs and a label: a published **source case** of
the pinned sample set, which gives the reference notes and required facts, and an
**answer case**, an `authored` saved test case whose input is the answer the judge grades.
Two controls expect `pass` and two expect `fail`. The judge doesn't see the expected labels.

Saving a profile or a version, and asking for a plan, call no provider and cost nothing. A
calibration is four paid judge calls through the same plan, pricing, spend ceiling,
budget and recovery rules as a calibration started from the comparison itself. Four
controls are a diagnostic, not a certification: a calibration reports how many controls
agreed and computes no agreement rate.

### Judge profile actions

Reads use the existing `metadata.read`. Writes add one new action, `judge_profiles.write`,
and every requirement is resolved at the gateway environment the profile belongs to:

| Requirement | Needed for |
|---|---|
| `metadata.read` | list and get profiles, list and get versions, list a profile's calibrations |
| `judge_profiles.write` and `comparisons.run` | create a profile and publish a version |
| a current judge grant on the pinned connection | create, version create, preflight and start; without it the answer is `judge_connection_not_granted` |
| `comparisons.run` | preflight, start, and read one calibration |
| `tests.run`, `content.read` and `cost.read` | preflight and start; the calibration lane checks them in the same environment |
| a signed-in session | preflight and start; a management key is answered `app_key_cannot_select_subject` (HTTP 403) |

`judge_profiles.write` is published in role version 2, as `datasets.write` is: it's
resolvable only on a binding issued with an explicit `role_version: 2`. The last row means
the CLI and both SDKs, which authenticate with a management key, can save, version, list
and read profiles and read their calibrations, but can't plan or start one. A calibration
always names a subject, a saved draft revision or an applied release, and a management key
may not select one.

| Role | Reads | Create and version (`role_version: 2`) | Preflight and start |
|---|---|---|---|
| `workspace_admin`, `app_developer` | yes | yes | yes, with a `content.read` grant |
| `org_owner`, `org_admin` | yes | no: they hold no `judge_profiles.write` | no: they hold no `tests.run` |
| `analyst` | yes, except one calibration's read | no | no |
| every other role, including `runtime_service` and `test_executor` | no | no | no |

A calibration's read redacts the run with your `content.read` and `cost.read`, as the
comparison's calibration read does. The summary's counts are returned either way.

### Judge profile bounds

A request outside these bounds is answered with `judge_profile_invalid` and a list of
`problems`, or with `validation_failed` for a malformed request, and nothing is stored.

| Item | Bound |
|---|---|
| `name` | 1 to 120 bytes after trimming; on create only |
| `template` | `{"judge_id":"judge_faithfulness_v1","scorer_version":1}`; a `prompt_sha` or `rubric_digest` you echo must equal the build's |
| `judge` | `{model, connection_id}`; the model needs a thinking row and a judge price in this build; the qualified model is `claude-haiku-4-5-20251001` |
| `bounds.max_chars` | 1 to 6000 characters per side |
| `bounds.max_output_tokens` | 1 to 512 |
| `controls_source` | `{comparison_id, sample_set_version}` of a judged comparison in the same environment, at its current sample set; an optional `sample_set_digest` must equal the stored one |
| `controls` | four, two with `expected_outcome` `pass` and two with `fail` |
| `control_id` | 1 to 80 bytes, no CR, LF or NUL, unique within the version |
| `source` | `{case_id, case_version}` of a case in the pinned sample set |
| `answer` | `{case_id, case_version}` of an `authored` saved test case visible in the environment |
| `idempotency_key` | 1 to 128 characters of `A–Z a–z 0–9 . _ : -`; at most 100 on a calibration start |
| request body | at most 64 KiB; an unknown field at any depth is `judge_profile_invalid` with `unknown_field` |

A version stores IDs only. The reference notes, required facts and answer text stay on the
cases, under their own retention and `content.read` rules.

### Judge profile identities and digests

| Value | What it is |
|---|---|
| `id` (the profile ID) | `jprof_` followed by 12 base58 characters, minted by the server |
| `version` | a whole number from 1, contiguous per profile; publishing adds the next one |
| `profile_digest` | `sha256:` and 64 hex digits over the canonical JSON of the version's pins: schema `judge-profile/v1`, template with prompt and rubric identities, model, thinking mode, connection and connection version, bounds, controls source with its digest, and the controls in `control_id` order |
| `plan_digest` | `sha256:` and 64 hex digits over a preflight's plan: the profile `ref`, the comparison and the calibration plan's own fingerprint |
| `calibration_id` | the calibration lane's `jcal_…`, linked to one start |

A version's `ref` is `{profile_id, version, profile_digest}`. Every version is immutable.
The digest leaves out the name, the ID, the version number, the actors and the times, so
two versions that pin the same things carry the same `profile_digest`.

`plan_digest` covers the caller, the subject's resolved configuration, the four examples,
the judge, its connection version, the per-call maximum, the reservation, the ceiling and
the `authority_epoch`. It's not the calibration plan's inner `plan.digest`; a start approves
`plan_digest`.

A version's `pin_state` is computed on every get: whether its pins still hold in this build,
for these grants, against the comparison's current sample set.

| `pin_state.state` | Meaning |
|---|---|
| `current` | the template, model price, connection version and sample set it pins are current |
| `stale` | a pin moved; `pin_state.reason` is `connection_version_moved`, `controls_source_moved` or `profile_removed` |
| `template_unavailable` | this build can't serve the pinned template, model or price |
| `connection_not_granted` | the pinned connection isn't granted for judging here |
| `unknown` | the pins couldn't be checked on this read; nothing is assumed current |

### Judge profile operations

All paths are relative to
`/portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}`,
and every response is the lane's envelope. The section on actions lists what each one
needs.

| Operation | Method | Path | Success |
|---|---|---|---|
| `customer.judge_profiles.create` | POST | `/judge-profiles` | 201 `judge-profile`; 200 on replay |
| `customer.judge_profiles.list` | GET | `/judge-profiles` | 200 `judge-profiles`, heads with their current version, newest first |
| `customer.judge_profiles.get` | GET | `/judge-profiles/{profile_id}` | 200 `judge-profile` at its current version, with `pin_state` |
| `customer.judge_profiles.versions.create` | POST | `/judge-profiles/{profile_id}/versions` | 201 `judge-profile`; 200 on replay |
| `customer.judge_profiles.versions.list` | GET | `/judge-profiles/{profile_id}/versions` | 200 `judge-profile-versions`, newest first |
| `customer.judge_profiles.versions.get` | GET | `/judge-profiles/{profile_id}/versions/{version}` | 200 `judge-profile-version`, with `pin_state` |
| `customer.judge_profiles.calibrations.preflight` | POST | `/judge-profiles/{profile_id}/calibrations/preflight` | 200 `judge-profile-calibration-plan`; writes nothing, calls no provider |
| `customer.judge_profiles.calibrations.start` | POST | `/judge-profiles/{profile_id}/calibrations` | 202 `judge-profile-calibration`, also on replay |
| `customer.judge_profiles.calibrations.list` | GET | `/judge-profiles/{profile_id}/calibrations` | 200 `judge-profile-calibrations`, links newest first; `?version=` keeps one version's |
| `customer.judge_profiles.calibrations.get` | GET | `/judge-profiles/{profile_id}/calibrations/{calibration_id}` | 200 `judge-profile-calibration`: link, run and summary |

The lists page with `limit` and `page_token`. The same ten operations in each client:

| Operation | Python | TypeScript | CLI |
|---|---|---|---|
| create | `judge_profiles.create` | `judgeProfiles.create` | `cloud judge-profile create` |
| list | `judge_profiles.list` | `judgeProfiles.list` | `cloud judge-profile list` |
| get | `judge_profiles.get` | `judgeProfiles.get` | `cloud judge-profile get` |
| versions create | `judge_profiles.versions_create` | `judgeProfiles.versionsCreate` | `cloud judge-profile version create` |
| versions list | `judge_profiles.versions_list` | `judgeProfiles.versionsList` | `cloud judge-profile version list` |
| versions get | `judge_profiles.versions_get` | `judgeProfiles.versionsGet` | `cloud judge-profile version get` |
| calibrations preflight | `judge_profiles.calibrations_preflight` | `judgeProfiles.calibrationsPreflight` | `cloud judge-profile calibration preflight` |
| calibrations start | `judge_profiles.calibrations_start` | `judgeProfiles.calibrationsStart` | `cloud judge-profile calibration start` |
| calibrations list | `judge_profiles.calibrations_list` | `judgeProfiles.calibrationsList` | `cloud judge-profile calibration list` |
| calibrations get | `judge_profiles.calibrations_get` | `judgeProfiles.calibrationsGet` | `cloud judge-profile calibration get` |

The preflight and start methods exist in every client, and each of them sends a management
key, so against a deployment those two answer `app_key_cannot_select_subject`. The portal's
**Judge profiles** section, inside a judged comparison's **Judge calibration** panel,
plans and starts with your signed-in session.

**Write requests.** A create carries `idempotency_key`, `name`, `template`, `judge`,
`bounds`, `controls_source` and four `controls`:

```json
{"idempotency_key": "q-b2-judge-1", "name": "B2 faithfulness",
 "template": {"judge_id": "judge_faithfulness_v1", "scorer_version": 1},
 "judge": {"model": "claude-haiku-4-5-20251001", "connection_id": "pc_…"},
 "bounds": {"max_chars": 4096, "max_output_tokens": 512},
 "controls_source": {"comparison_id": "cmp_…", "sample_set_version": 1},
 "controls": [{"control_id": "faithful-refund",
               "source": {"case_id": "tcase_…", "case_version": 1},
               "answer": {"case_id": "tcase_…", "case_version": 1},
               "expected_outcome": "pass"}, …]}
```

A new version carries the same fields without `name`, plus `expected_version`, which must
equal the current version. Idempotency works as for datasets: the body field is primary,
an `Idempotency-Key` header is accepted when it's absent, the same key with the same
request answers the original, and none of the clients generates a key.

**A profile** answers `data.profile`, the head (`id`, the five scope IDs, `name`,
`current_version`, `state`, `created_by`, `created_at`, `updated_at`), and `data.version`:
`profile_id`, `version`, `ref`, `template` (with `prompt_sha` and `rubric_digest`), `judge`
(with the derived `thinking_mode` and the pinned `connection_version`), `bounds`,
`controls_source` (with `sample_set_digest`), `controls`, the derived `scorer`,
`published_by` and `published_at`. A get adds `data.pin_state`, `{state, reason}`. A
version get answers the same shape for one version.

**A preflight** carries `version`, `subject` and optionally `ceiling_micro_usd`:

```json
{"version": 1, "subject": {"kind": "saved_draft_revision", "draft_revision": 7},
 "ceiling_micro_usd": 500000}
```

`subject` is `{"kind":"saved_draft_revision","draft_revision":N}` or
`{"kind":"applied_release","release_id":"rel_…"}`. An absent ceiling takes the calibration
lane's default. The answer is `profile` (the `ref`), `comparison_id`, `plan_digest` and
`plan`, the judge calibration plan the comparison's own preflight answers: four planned
judge calls, the per-call maximum, the total reserved, the ceiling and `authority_epoch`.
Four priced calls must fit a ceiling below USD 10. The server checks the version's pins
first and answers `judge_profile_stale` if one moved.

**A start** carries the preflight's fields plus `idempotency_key`, `approved_epoch` and
`approved_plan_digest`:

```json
{"version": 1, "subject": {"kind": "saved_draft_revision", "draft_revision": 7},
 "ceiling_micro_usd": 500000, "idempotency_key": "q-b2-cal-1",
 "approved_epoch": 42, "approved_plan_digest": "sha256:…"}
```

`approved_epoch` is the preflight's `plan.authority_epoch` and `approved_plan_digest` its
top-level `plan_digest`. The server plans again before it starts. If the digest it computes
isn't the approved one, nothing starts and the answer is `judge_profile_stale` with reason
`plan_changed`. Use a new idempotency key for every preflight you approve: the same key
with a different approval is `idempotency_payload_mismatch`. The same key with the same
request answers the same calibration, including after a lost response, so a retry doesn't
pay twice.

A start and a calibration get answer `link` (`profile_id`, `profile_version`,
`comparison_id`, `idempotency_key`, `calibration_id`, `started_by`, `created_at`,
`linked_at`), `calibration`, the run redacted with your `content.read` and `cost.read` as
the comparison's calibration read redacts it, and `summary`:

| `summary` field | Meaning |
|---|---|
| `controls` | 4 |
| `observed`, `unknown`, `not_executed` | how the four calls ended; the three add up to `controls` |
| `agreements`, `disagreements` | observed verdicts that match, or don't match, the expected outcome |
| `applicable` | whether this calibration still speaks for the profile's current pins |
| `applicability_reason` | `pins_current`, `version_superseded`, `plan_pins_differ`, `pins_stale`, `template_unavailable`, `connection_not_granted` or `pin_state_unknown` |
| `agreement_statistics` | `null` in every summary |
| `agreement_statistics_reason` | `four_controls_are_diagnostic` |

A calibration list answers the links alone. Cost, verdicts and state stay on the
calibration; the profile stores none of them.

### Judge profile errors

The problem document has the dataset errors' shape. A `judge_profile_invalid` answer also
carries `problems`, a list of `{control_id, field, reason}`, and a `judge_profile_stale`
answer carries `reason`.

| Code | Status | When |
|---|---|---|
| `judge_profiles_not_enabled` | 501 | the deployment doesn't compose judge profiles |
| `judge_profile_not_found` | 404 | the profile doesn't exist, exists outside the caller's scope, or never linked the named calibration; the same answer |
| `judge_profile_version_not_found` | 404 | the profile is visible and has no such version |
| `judge_profile_version_conflict` | 409 | `expected_version` isn't the current version |
| `judge_profile_invalid` | 422 | one or more fields or controls can't be used; see `problems` |
| `judge_profile_template_unavailable` | 409 | a `scorer_version` other than 1, a prompt or rubric identity that isn't the build's, or a model without a thinking row or judge price |
| `judge_profile_stale` | 409 | a pin moved: `reason` is `connection_version_moved`, `controls_source_moved`, `plan_changed` or `profile_removed` |
| `idempotency_payload_mismatch` | 409 | the idempotency key was used before with a different request |
| `judge_connection_not_granted` | 403 | the connection isn't granted for judging in this application and environment |
| `app_key_cannot_select_subject` | 403 | preflight or start with a management key |
| `content_read_denied` | 403 | preflight or start by a caller without `content.read` in the environment |
| `forbidden` | 403 | the caller can see the scope and lacks an action |
| `workspace_excluded` | 409 | the workspace is excluded; reads and writes stop |
| `validation_failed` | 422 | a malformed request: a scope, a key, a version, a subject, a page token |

The calibration lane's own answers pass through unchanged on a preflight or start,
including the spend and budget codes, `dependency_unavailable` and the `judge_*` codes.
[Gateway quality evidence](/docs/reference/gateway-quality-evidence#refusal-codes) lists
each code with its status.

| `problems[].reason` | Meaning |
|---|---|
| `unknown_field` | a field the server doesn't accept; `field` names its path |
| `invalid_name` | the name is empty or over 120 bytes after trimming |
| `max_chars_out_of_range` | `max_chars` isn't 1 to 6000 |
| `max_output_tokens_out_of_range` | `max_output_tokens` isn't 1 to 512 |
| `controls_count` | not four controls |
| `expected_outcome_split` | not two `pass` and two `fail` |
| `invalid_expected_outcome` | an outcome other than `pass` or `fail` |
| `duplicate_control_id` | two controls share an ID |
| `invalid_control_id` | an ID that's empty, over 80 bytes, or holds a CR, LF or NUL |
| `controls_source_not_found` | the comparison or sample set isn't visible here |
| `controls_source_not_judged` | the comparison isn't a judged one |
| `sample_set_digest_mismatch` | the request's `sample_set_digest` isn't the stored one |
| `source_case_not_in_sample_set` | a source case isn't in the pinned sample set |
| `answer_case_not_found` | an answer case isn't a saved test case visible here |
| `answer_case_not_authored` | an answer case isn't `authored` |

### Judge profile lifecycle

A profile's `state` is `active`, `deleting` or `purged`, and moves forward only. The
workspace's data-exit process drives the last two; there's no per-profile delete in this
release.

- **Workspace exclusion.** Once a workspace is excluded, every profile read and write in it
  answers `workspace_excluded`.
- **Export.** An export carries each profile version, with its calibration links, as a
  `judge_profile` row in the `comparisons` section, as listed under
  [evaluation evidence in an export bundle](/docs/reference/managed-evaluations#evaluation-evidence-in-an-export-bundle).
- **Purge.** A purge deletes the workspace's calibration links, profile versions and
  profiles, names included, along with the rest of its data. The calibrations themselves
  follow the comparison lifecycle.

## How this page was checked

The field names, bounds, codes and the digest rule were read from the dataset contract
and the service code. The worked digest was computed by the service's own digest
function over the tutorial's examples and recomputed with the jq recipe. The CLI's
behaviour for these operations is pinned by transcripts captured against a loopback
server that answers the contract's fixture shapes. That local verification does not show
a live deployment's answers, and it spent nobody's money.

The offline-evaluation section was read from the evaluation contract, the service's
types, digest and error code, and the three clients. Its CLI behaviour is pinned by
transcripts from the same kind of loopback server, whose digests are computed with the
contract's rule. None of that shows a deployment's worker or its timing.

The comparison-definitions section was read from the definition contract, the service's
types and error codes, the HTTP adapter and the three clients. Its CLI behaviour is
pinned by transcripts from a loopback server that answers the contract's fixture shapes
and doesn't apply the management-key rule. No preflight or start on this page has run
against a deployment, and no provider was called.

The judge-profiles section was read from the judge profile contract, the service's types,
digests and error codes, the HTTP adapter, the calibration lane's checks and the three
clients. Its CLI behaviour is pinned by transcripts from a loopback server that answers
the contract's fixture shapes and doesn't apply the management-key rule. No calibration on
this page has run against a deployment, and no judge was called.

<NextSteps
  items={[
    { title: 'Curate a dataset step by step', href: '/docs/build/curate-evaluation-datasets', description: 'Create, correct and reopen a two-example dataset from the API, either SDK or the CLI.' },
    { title: 'Run an offline check step by step', href: '/docs/build/run-offline-evaluation', description: 'Score version 2 of the tutorial dataset, open the wrong label and rerun on a newer version.' },
    { title: 'Who can read what', href: '/docs/build/organizations-and-access', description: 'Roles, bindings, and why content is its own grant.' },
    { title: 'Workspace export and deletion', href: '/docs/build/export-and-delete', description: 'What the bundle, the exclusion and the purge do across all of a workspace.' },
  ]}
/>