Managed evaluations reference

This page lists what a managed deployment exposes for evaluation material, in the order the product is built: who may act, what the limits are, how a version is identified, which operations exist, what each error means, and what happens to the material over its life. Curate an evaluation dataset your team can reopen walks through the same operations as a tutorial.

That material comes in four parts. Hosted datasets are named, versioned lists of examples at one gateway environment. Offline evaluations score the outputs a dataset version already holds against its expected labels, without calling a provider; they have their own section after the dataset sections, and Check a dataset’s supplied outputs against its labels is their tutorial. Comparison definitions save a provider comparison over one dataset version so you can plan and start it again; they have their own section, and Repeat a comparison from a dataset walks through one. Judge profiles save a gateway judge with four controls so you can calibrate it again; they have the last section, and Keep a judge profile walks through one.

Availability

The managed deployment doesn’t serve these operations yet. The offline-evaluation operations and their two actions are registered with the release that ships them, and hosted datasets reach the managed deployment with their own rollout. Until a deployment runs the release in question, the operations on this page aren’t part of what it serves, and its answer to these paths isn’t the contract described here. The tutorials’ captures come from a local loopback server for that reason.

Comparison definitions reach it with a release of their own, in the same way, and so do judge profiles.

A deployment that runs the release either composes the service or it doesn’t. Where it doesn’t, every dataset operation answers datasets_not_enabled, every evaluation operation answers evaluations_not_enabled, every comparison definition operation answers comparison_definitions_not_enabled, and every judge profile operation answers judge_profiles_not_enabled, all with HTTP 501, and the CLI reports it as “the server has not implemented <operation> yet (HTTP 501)”. Nothing about the request can change that answer. A deployment that composes evaluations but runs no evaluation worker answers every read and turns away a run start with evaluation_worker_unavailable (HTTP 503).

Self-hosted TokenTriage has its own, separate mechanisms: eval programs and judge configuration. Nothing on this page applies to them.

Actions

Dataset reads are governed by the existing metadata.read action, resolved at the gateway environment the dataset belongs to; there is no datasets.read action. Writes add one new action, datasets.write, and bodies still need the existing content.read.

metadata.read is granted the same way it always was. datasets.write is different: it is published in role version 2, but the default role version stays 1 in this release, so datasets.write is resolvable and grantable only by an explicit role_version: 2 on customer.bindings.create, never the default stamp. A binding issued under an earlier role version doesn’t gain it; it has to be reissued at role_version: 2.

content.read is a separate grant in every case. A runtime key reaches none of these operations. Content, cost, billing and metadata are four separate grants explains why.

Bounds

A request outside these bounds is answered with dataset_bounds_exceeded (counts and the request size) or dataset_example_invalid (a field), and nothing is stored.

Unknown fields in an example are rejected with reason unknown_field, so an upload can’t carry run, price or source fields beside its text.

Identities and digests

Every published version is immutable. Four values identify what a version holds.

A version’s ref is {dataset_id, version, items_digest}, the triple anything that pins a version records.

body_digest is the SHA-256 of the example’s body object: one JSON document with the keys input, output and required_facts in that order, each omitted when absent, no insignificant whitespace, UTF-8, no HTML escaping. For a saved_case example the body holds no input, because the input is read from the case.

items_digest is computed without any body, so a reader without content.read can check it:

1. Take each example’s body_digest, expected_label (or null), item_id, origin and saved_case (or null).
2. Write each as one compact JSON object with those keys in that byte order.
3. Sort the lines by item_id bytes and end each with a newline.
4. Hash the result with SHA-256 and prefix sha256:.

A label change moves items_digest and leaves body_digest alone; a text change moves both. Reordering the examples in a request changes neither.

$ jq -c -S '.data.version.examples | sort_by(.item_id)[] |
    {body_digest, expected_label, item_id, origin, saved_case}' v1.json |
    sha256sum

Operations

All paths are relative to /portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}. Every response is the lane’s envelope, {"schema":"customer/v1","request_id":…,"kind":…,"data":…}.

The two lists page with limit and page_token, and return next_page_token while more remain. The single-resource answers and both writes carry kind: "dataset".

The same six operations in each client:

In Python, scope is DatasetScope(org, ws, app, gw, env); in TypeScript it’s { org, ws, app, gw, env }.

Write requests

A create carries idempotency_key, name and examples. A new version carries idempotency_key, expected_version and examples, and no name. A new version replaces the whole example list; there’s no patch.

• idempotency_key is required and chosen by the caller. The body field is primary; an Idempotency-Key header is accepted when the field is absent. The same key with the same request returns the original resource with HTTP 200. The comparison covers the name (for a create) or expected_version (for a version) plus the items_digest of the examples, so reordering them is still the same request. None of the clients generates a key.
• expected_version must equal the dataset’s current version. Anything else is dataset_version_conflict and nothing is published.

Response fields

A dataset document has two parts. data.dataset is the head: dataset_id, the five scope IDs, name, current_version, state, created_by, created_at, updated_at. data.version is one version:

Each example carries item_id, origin, expected_label (absent when none), saved_case (or null), body_digest and content_state. The body fields input, output and required_facts are present only when content_state is available. created_by and published_by carry an actor ID, not an email address.

When a limitation applies to both a saved case’s input and the example’s own body, the answer names the more final one, in the order withheld, not_retained, excluded, unreadable, digest_mismatch, purged.

Errors

Every error is an application/problem+json document with type (the code below), title, status, detail, instance and request_id. Quote request_id to support.

A dataset_example_invalid answer carries problems, a list of {item_id, field, reason}. item_id is empty for a problem with the list as a whole or with an item_id that is itself invalid. No problem echoes a body.

saved_case_not_found covers every way a reference can fail, so it can’t be used to learn whether a case exists in another gateway.

Lifecycle

A dataset’s state is active, deleting or purged, and moves forward only. The workspace’s data-exit process drives the last two; there is no per-dataset delete in this release.

• Workspace exclusion. Once a workspace is excluded, every dataset read and write in it answers workspace_excluded, and example bodies read excluded.
• Export. An export bundle carries one dataset row per dataset version in the runs section, listed with the other evaluation kinds in the following table. Each example’s body is a content object: it’s listed under content_inventory and fetched separately under content.read, and content_bodies answers withheld.
• Purge. A purge deletes every dataset in the workspace whole: its name, its versions, its examples and their bodies. No record of a version is kept behind.

Export and delete your data describes the bundle, exclusion and purge for the workspace as a whole. Export and remove evaluation evidence walks through reading these rows.

Evaluation evidence in an export bundle

Evaluation evidence has no export section of its own. Its rows join two existing sections, runs and comparisons, so both sections are heterogeneous: each holds rows of more than one shape.

• A row with no kind member is a row those sections carried before evaluation kinds existed: a test run keyed by run_id in runs, or a comparison record keyed by comparison_id in comparisons. Those rows are written unchanged.
• A row with a kind member is an evaluation row. Its top-level members share no name with the older rows.
• A section’s manifest rows counts every row of every kind. A reader dispatches on kind, and skips and counts a kind it doesn’t know rather than failing, because a later release can add kinds.

Every evaluation row has the same envelope:

{
  "kind": "…",
  "id": "…",
  "version": 1,
  "organization_id": "…",
  "workspace_id": "…",
  "created_at": "…",
  "document": {}
}

created_at is an RFC 3339 instant in UTC. version is omitted for the kinds with no versions. The kinds form a closed set in this release, in this order:

The rows follow five rules:

• A resource with no versions yet exports one row whose document holds only head, with no version.
• No row carries an example body, an input, an output, the required facts or an observed label. expected_label is metadata and is exported.
• An example’s body_ref equals the ref of its body’s content_inventory row, which is how a reader joins the two.
• A decision_provenance row whose definition version can’t be found is still exported, with provenance.state set to unavailable.
• A ceiling_micro_usd is exported as the API returns it. It’s a configured ceiling, not spend.

Each section keeps its own grant, re-checked when the bundle is packaged: runs needs metadata.read and comparisons needs comparisons.run. Judge profiles are exported in comparisons, so a bundle carries them only for a holder of comparisons.run, although the API reads them under metadata.read. A section past its store’s page bound answers unavailable with bound_exceeded and carries no rows of any kind.

Rolling back below the datasets release

The dataset schema is additive; there’s no schema rewind. The supported rollback keeps a datasets-aware binary in place and disables the dataset plane instead: omit the content bucket configuration, and every dataset operation answers datasets_not_enabled (501).

Before rolling the control plane binary back below the datasets release, run select count(*) from foundation_content_objects where kind = 'dataset_example'. A non-zero count means don’t roll the binary back — disable the plane instead. A pre-datasets binary stalls that workspace’s purge and fails its export until the binary rolls forward again.

A Foundation change (maintenance FMT-045, decision DEC-022) that tolerates unknown content kinds is scheduled to land before the datasets release. Once it ships, that release becomes the rollback floor and this check is no longer needed.

Provenance of an example

An authored example’s input was typed into the request. A saved_case example refers to a saved test case version in the same gateway, and its input is read from that case rather than copied. A supplied output is declared by whoever wrote the request and is never observed: creating or versioning a dataset calls no provider and costs nothing.

Provenance of a run

A comparison run started from a definition (runs.start above) carries a provenance alongside its evidence, on results.get and beside each decisions.create and decisions.get answer: {state, definition?, judge_profile?}.

judge_profile ({profile_id, version, profile_digest}) is present only alongside state: "definition", and only when that definition version pinned one. Nothing here is copied from the definition: every field is read from the link and the version at answer time, so a caller compares a run’s provenance.definition.definition_digest against the definition’s own current digest to see whether the version it ran against is still the one published.

Offline evaluations

An offline evaluation checks the outputs a dataset version already holds against that version’s expected labels. A definition pins one dataset version and says where the label sits in each output, how to compare it, and what result is acceptable. A run scores every example of the pinned version once and records one outcome per example. The only mode is offline_exact_label: no provider is called, and nothing is generated.

Evaluation actions

Reads use the existing metadata.read, resolved at the gateway environment the definition or run belongs to; there is no evaluations.read action. Two new actions cover writing definitions and running them, and the example bodies stay behind the existing content.read.

evaluations.write and evaluations.run behave like datasets.write: they’re granted only by an explicit role_version: 2 binding, not by the default stamp, and they join role version 2 with the release that registers the evaluation operations. A binding issued under an earlier role version has to be reissued at role_version: 2. Readers need no new binding.

A run works under the authority of the person or key that started it. Before each example it checks that the starter still holds evaluations.run, metadata.read and content.read in the environment; if not, the run ends failed with authority_revoked. It doesn’t fall back to a service identity.

Evaluation bounds

A request outside these bounds is answered with evaluation_definition_invalid (a field, listed in problems), evaluation_mode_not_supported (a mode or label source kind), or validation_failed (the request as a whole), and nothing is stored.

The comparison rule is label_exact/v1. It trims Unicode white space from both ends of the extracted label and of the expected label, then compares bytes. With case_sensitive: false it compares them under Unicode case folding instead.

Extraction depends on label_source:

• plain_text: the whole output, trimmed, is the label. A blank output has no label.
• json_pointer: the output must be one JSON object with no trailing data and no duplicate member. The label is the string member the pointer names. A missing member means no label. Invalid JSON, a value that isn’t an object, a member that isn’t a string, or a label over 256 bytes makes the label malformed.

Evaluation identities and digests

Every definition version, and every finished run, is immutable.

A definition version’s ref is {definition_id, version, spec_digest}; runs, plans and starts name a definition by it. Each digest is the SHA-256 of one canonical JSON object, written as the dataset digests are: keys in byte order, no insignificant white space, UTF-8, no HTML escaping, no trailing newline.

A finished run also carries ref: the operation customer.evaluations.runs.get, the five scope IDs, run_id, definition, dataset and result_digest. It is null until the run finishes, and it grants nothing to whoever holds it.

Evaluation operations

All paths are relative to /portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}/evaluations, and every response is the lane’s envelope.

The lists page with limit and page_token. The same twelve operations in each client:

In Python, scope is EvaluationScope(org, ws, app, gw, env); in TypeScript it’s { org, ws, app, gw, env }. The CLI’s run preflight and run start take --definition and --version and read the ref for you.

Write requests. A definition create carries idempotency_key, name, mode, dataset, label_source, case_sensitive and acceptance. A new version carries the whole specification again without name, plus expected_version, which must equal the current head. A preflight carries {definition: ref}. A start carries idempotency_key, definition and approved_digest, the plan digest the caller read. Cancel takes no key and no body fields. Idempotency works as for datasets: the same key with the same request answers the original with HTTP 200, and none of the clients generates a key.

A plan answers digest, definition, dataset, selected, expected_label_absent, planned_provider_calls (always 0), provider_spend, content_read (whether the caller holds content.read there) and active_run_id (the run already queued or running here, or null).

A run answers run_id, the five scope IDs, definition, dataset, mode, state, failure, counts, unscored_reasons (a count per reason), acceptance ({rule, outcome}), provider_spend, initiated_by, cancelled_by, created_at, started_at, finished_at, result_digest and ref.

acceptance.outcome is met or not_met for a complete or partial run, judged against max_mismatched and max_unscored, and undetermined in every other state.

An item answers item_id, origin, outcome, reason (for unscored only), expected_label, body_digest, content_state, observed_label and recorded_at. observed_label is present only when content_state is available and a label could be extracted. It isn’t stored: each read opens the body again under the reader’s content.read, so exclusion and purge reach it. A not_evaluated item carries no reason and no recorded_at.

Provider spend

Every plan and every run carries "provider_spend": {"basis": "no_dispatch", "dispatches": 0}. It has no amount field, and the missing amount is a statement rather than a gap: an offline run composes no dispatcher, provider connection or price lookup, so nothing is sent and nothing is priced. It isn’t a zero-dollar charge, and it says nothing about what the supplied outputs cost when they were first produced.

Unscored reasons

An example that can’t be compared is recorded as unscored with one reason. The set is closed, and the first reason that applies, in this order, is the one recorded.

An unscored example counts toward max_unscored, not as a mismatch.

Run states

A run moves queued → running → complete | partial | failed | cancelled, or straight from queued to cancelled or failed. The last four are final. One run at a time may be queued or running per gateway environment.

failure.code is one of authority_revoked (the starter’s grant changed), workspace_excluded, dataset_unavailable, digest_mismatch (a pinned digest no longer matches), attempts_exhausted (three worker attempts didn’t finish it) or worker_fault. Items recorded before a run fails or is cancelled stay. A cancel ends the run at once, and no item is written after it. A second cancel answers the cancelled run again.

A run that is interrupted, such as by a worker restart, is resumed by the next claim, which scores only the examples not yet recorded; the result_digest doesn’t depend on how many times that happened.

Evaluation errors

The problem document has the dataset errors’ shape. An evaluation_definition_invalid answer also carries problems, a list of {field, reason} that doesn’t echo a value, and evaluation_quota_exhausted names the active run.

Evaluation lifecycle

Definitions, runs and items follow the workspace’s data-exit process, as datasets do. An exclusion stops every read and write and fails a running run with workspace_excluded. An export carries definitions as evaluation_definition rows and runs, with their items, as evaluation_run rows in the runs section, as listed under evaluation evidence in an export bundle. A purge deletes them with the rest of the workspace. After a dataset’s bodies are purged, an item read has no observed_label to show, and a finished run’s recorded outcomes and result_digest stay as they were.

Comparison definitions

A comparison definition saves the setup of a provider comparison over one dataset version, so you can plan and start the same comparison again. A definition has a name and a kind, labeled or judged, and owns one comparison (cmp_…). Each definition version pins a dataset version, the applied baseline release, the candidate draft revision, the scorer, the acceptance rule and an optional spend ceiling, and publishes that selection as the comparison’s sample set. A run is an ordinary comparison run (crun_…) on that comparison, and the definition keeps a link to it.

Unlike an offline evaluation, a run calls providers. It makes one paid dispatch per example per arm, and a judged definition adds one judge call per example per arm. Saving a definition or a version, and asking for a plan, call no provider and cost nothing.

Comparison definition actions

There is no new action. Every operation declares the existing comparisons.run, resolved at the gateway environment the definition belongs to. The other requirements are checked in the same environment:

The last row means the CLI and both SDKs, which authenticate with a management key, can save, list and read definitions and list their runs, but can’t plan or start one. A definition always aims its candidate arm at a saved draft revision, and a management key may not select a comparison subject.

content.read and the connection grant are separate in every case. Reading a run’s evidence afterwards uses the comparison reads, which keep their own cost.read and content.read rules.

Comparison definition bounds

A request outside these bounds is answered with comparison_definition_ineligible (an example of the pinned version, listed in problems) or validation_failed (the request as a whole), and nothing is stored.

Only saved_case examples can be compared in this release. A saved test case is a request the gateway already holds, so each arm can send it again. An authored example’s text was typed into the dataset, and the comparison has no request to replay for it. To compare it, save it as a test case first (customer.cases.create), then publish a dataset version that refers to it as saved_case.

The server doesn’t read example bodies to build the sample set. For a labeled definition, each example’s dataset expected_label becomes the case’s expected label. For a judged definition, each case’s reference notes are the saved test case’s own input, and the required facts are the ones the request declares per item. Cases are numbered in item_id byte order.

Comparison definition identities and digests

A definition version’s ref is {definition_id, version, definition_digest}. Every version is immutable. Publishing a selection that a previous version already published answers the same sample set version.

plan_digest covers the definition ref, the comparison, the sample set version and digest, both arms, the scorer, the rule, the planned arm and judge calls, the three spend ceilings and the budget plan. If any of those moves, the next preflight answers a different digest.

Comparison definition operations

All paths are relative to /portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}, and every response is the lane’s envelope. Every row declares comparisons.run; the section on actions lists what else each one needs.

The lists page with limit and page_token. The same nine operations in each client:

In Python, scope is ComparisonScope(org, ws, app, gw, env); in TypeScript it’s { org, ws, app, gw, env }. The preflight and start methods exist in every client, and each of them sends a management key, so against a deployment those two answer app_key_cannot_select_subject.

Write requests. A create carries idempotency_key, name, kind, dataset, baseline, candidate, scorer and rule, and optionally critical_item_ids, required_facts and ceiling_micro_usd. A judged definition may give judge_profile instead of scorer; the server derives the scorer from the pinned profile version:

{"idempotency_key": "q-a2-def-1", "name": "A2 cheaper route", "kind": "labeled",
 "dataset": {"dataset_id": "ds_…", "version": 2, "items_digest": "sha256:…"},
 "critical_item_ids": ["refund-2"],
 "baseline": {"kind": "applied_release", "release_id": "rel_…"},
 "candidate": {"kind": "saved_draft_revision", "draft_revision": 7},
 "scorer": {"scorer_id": "label_exact_v1", "scorer_version": 1, "rubric_digest": "sha256:…"},
 "rule": {"kind": "labeled", "at_least_baseline_pass_count": true,
          "no_failure_on_critical_cases": true, "max_unknown_cases": 0},
 "ceiling_micro_usd": 2000000}

A new version carries the same fields without name and kind, plus expected_version, which must equal the current version. Idempotency works as for datasets: the body field is primary, an Idempotency-Key header is accepted when it’s absent, the same key with the same request answers the original, and none of the clients generates a key.

A definition answers data.definition, the head (definition_id, the five scope IDs, name, kind, comparison_id, current_version, state, created_by, created_at, updated_at), and data.version: version, ref, dataset, sample_set ({comparison_id, version, digest, case_count}), items (each {ordinal, item_id, case_id, case_version, critical}), baseline, candidate, scorer, rule, ceiling_micro_usd, published_by and published_at. It carries IDs, pins and links only; no example body, output, cost or verdict is stored in a definition. A version that pinned a judge profile also carries judge_profile: {profile_id, version, profile_digest}, alongside the scorer that profile derived.

A preflight carries {"version": 1} and answers definition (the ref), comparison_id, plan_digest and preflight. preflight is the answer of the comparison’s own preflight: the arms, the sample set, the plan with 2N arm dispatches and 2N or 0 judge calls, the spend ceilings split between arms and judge, the budget plan and authority_epoch.

A start carries version, idempotency_key, approved_epoch and approved_plan_digest:

{"version": 1, "idempotency_key": "q-a2-start-1", "approved_epoch": 42,
 "approved_plan_digest": "sha256:…"}

approved_epoch is the preflight’s authority_epoch and approved_plan_digest its plan_digest. The server plans again before it commits. If the digest it computes isn’t the approved one, nothing starts and the answer is comparison_definition_plan_stale. Use a new idempotency key for every preflight you approve: the same key with a different approval is a different request, answered idempotency_payload_mismatch. The same key with the same request answers the same run.

A start answers link (definition_id, definition_version, comparison_id, run_id, idempotency_key, started_by, started_at) and run, the comparison run as customer.comparisons.runs.start answers it. Progress, spend, cancellation, results, review and decisions use the existing customer.comparisons.* operations on the link’s comparison_id and run_id, and every field below applies to them the same as to a comparison started without a definition.

Review and decisions. verdicts.review accepts an optional reviewed_evidence: {source_sha256, output_sha256}, the digest of the source and, for a cell that produced one, the output the reviewer read; its answer carries acting_credential_kind (portal_session or management_key) and the two reviewed_source_sha256 / reviewed_output_sha256 digests, kept even when the content itself is withheld. A digest that names bytes other than the cell’s is review_reading_mismatch (409). decisions.create accepts an optional rationale, the person’s words, stored the same way as a review note; its answer carries acting_credential_kind and this run’s provenance beside the decision, not inside it. results.get carries the run’s current evidence_digest, and releases.create refuses a decision whose digest no longer matches with decision_evidence_changed (409); record a new decision revision on the current evidence before naming it in a release. The full verdict and decision shapes, and the complete error catalogue, are listed at Gateway quality evidence.

CLI: cloud comparison review --read-source-sha256 --read-output-sha256 --note-file and cloud comparison decide --rationale-file --supersedes. Python: comparisons.review(scope, cmp, crun, ..., reviewed_evidence={"source_sha256": ..., "output_sha256": ...}) and comparisons.decide(scope, cmp, crun, ..., rationale=b"..."). TypeScript: client.comparisons.review(scope, cmp, crun, { ..., reviewed_evidence }) and client.comparisons.decide(scope, cmp, crun, { ..., rationale }).

Comparison definition errors

The problem document has the dataset errors’ shape. A comparison_definition_ineligible answer also carries problems, a list of {item_id, field, reason}.

The comparison’s own answers pass through unchanged: sample_set_empty and sample_set_too_large on a write, and on a preflight or start the comparison run set, including spend_ceiling_above_remaining, the judge_* codes, duplicate_comparison_run, authority_revoked and environment_not_active. Review and decisions pass through review_reading_mismatch (409, an attested reading names bytes other than the cell’s) and decision_evidence_changed (409, a release names a decision whose verdicts were revised since), alongside the existing content_read_denied, content_excluded, content_purged and decision_superseded. Gateway quality evidence lists each code with its status.

Comparison definition lifecycle

A definition’s state is active, deleting or purged, and moves forward only. The workspace’s data-exit process drives the last two; there’s no per-definition delete in this release.

• Workspace exclusion. Once a workspace is excluded, every definition read and write in it answers workspace_excluded.
• Export. An export carries each definition version as a comparison_definition row and each run link that names a comparison run as a decision_provenance row in the comparisons section, as listed under evaluation evidence in an export bundle.
• Purge. A purge removes the workspace’s definitions along with the rest of its data. The comparisons and runs a definition started follow the comparison lifecycle.

Judge profiles

A judge profile saves one gateway judge so you can calibrate it, and cite it, without entering it again. A profile has a name. Each profile version pins the judge template, the judge model and its saved connection version, two bounds, and four controls drawn from one judged comparison’s sample set. A calibration is the existing judge calibration (jcal_…) on that comparison, started from a version; the profile keeps a link to it.

The only template is judge_faithfulness_v1, scorer version 1. Its prompt and rubric come from the running build, and a version records their identities; you can’t supply your own prompt or rubric text. A control is two IDs and a label: a published source case of the pinned sample set, which gives the reference notes and required facts, and an answer case, an authored saved test case whose input is the answer the judge grades. Two controls expect pass and two expect fail. The judge doesn’t see the expected labels.

Saving a profile or a version, and asking for a plan, call no provider and cost nothing. A calibration is four paid judge calls through the same plan, pricing, spend ceiling, budget and recovery rules as a calibration started from the comparison itself. Four controls are a diagnostic, not a certification: a calibration reports how many controls agreed and computes no agreement rate.

Judge profile actions

Reads use the existing metadata.read. Writes add one new action, judge_profiles.write, and every requirement is resolved at the gateway environment the profile belongs to:

judge_profiles.write is published in role version 2, as datasets.write is: it’s resolvable only on a binding issued with an explicit role_version: 2. The last row means the CLI and both SDKs, which authenticate with a management key, can save, version, list and read profiles and read their calibrations, but can’t plan or start one. A calibration always names a subject, a saved draft revision or an applied release, and a management key may not select one.

A calibration’s read redacts the run with your content.read and cost.read, as the comparison’s calibration read does. The summary’s counts are returned either way.

Judge profile bounds

A request outside these bounds is answered with judge_profile_invalid and a list of problems, or with validation_failed for a malformed request, and nothing is stored.

A version stores IDs only. The reference notes, required facts and answer text stay on the cases, under their own retention and content.read rules.

Judge profile identities and digests

A version’s ref is {profile_id, version, profile_digest}. Every version is immutable. The digest leaves out the name, the ID, the version number, the actors and the times, so two versions that pin the same things carry the same profile_digest.

plan_digest covers the caller, the subject’s resolved configuration, the four examples, the judge, its connection version, the per-call maximum, the reservation, the ceiling and the authority_epoch. It’s not the calibration plan’s inner plan.digest; a start approves plan_digest.

A version’s pin_state is computed on every get: whether its pins still hold in this build, for these grants, against the comparison’s current sample set.

Judge profile operations

All paths are relative to /portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}, and every response is the lane’s envelope. The section on actions lists what each one needs.

The lists page with limit and page_token. The same ten operations in each client:

The preflight and start methods exist in every client, and each of them sends a management key, so against a deployment those two answer app_key_cannot_select_subject. The portal’s Judge profiles section, inside a judged comparison’s Judge calibration panel, plans and starts with your signed-in session.

Write requests. A create carries idempotency_key, name, template, judge, bounds, controls_source and four controls:

{"idempotency_key": "q-b2-judge-1", "name": "B2 faithfulness",
 "template": {"judge_id": "judge_faithfulness_v1", "scorer_version": 1},
 "judge": {"model": "claude-haiku-4-5-20251001", "connection_id": "pc_…"},
 "bounds": {"max_chars": 4096, "max_output_tokens": 512},
 "controls_source": {"comparison_id": "cmp_…", "sample_set_version": 1},
 "controls": [{"control_id": "faithful-refund",
               "source": {"case_id": "tcase_…", "case_version": 1},
               "answer": {"case_id": "tcase_…", "case_version": 1},
               "expected_outcome": "pass"}, …]}

A new version carries the same fields without name, plus expected_version, which must equal the current version. Idempotency works as for datasets: the body field is primary, an Idempotency-Key header is accepted when it’s absent, the same key with the same request answers the original, and none of the clients generates a key.

A profile answers data.profile, the head (id, the five scope IDs, name, current_version, state, created_by, created_at, updated_at), and data.version: profile_id, version, ref, template (with prompt_sha and rubric_digest), judge (with the derived thinking_mode and the pinned connection_version), bounds, controls_source (with sample_set_digest), controls, the derived scorer, published_by and published_at. A get adds data.pin_state, {state, reason}. A version get answers the same shape for one version.

A preflight carries version, subject and optionally ceiling_micro_usd:

{"version": 1, "subject": {"kind": "saved_draft_revision", "draft_revision": 7},
 "ceiling_micro_usd": 500000}

subject is {"kind":"saved_draft_revision","draft_revision":N} or {"kind":"applied_release","release_id":"rel_…"}. An absent ceiling takes the calibration lane’s default. The answer is profile (the ref), comparison_id, plan_digest and plan, the judge calibration plan the comparison’s own preflight answers: four planned judge calls, the per-call maximum, the total reserved, the ceiling and authority_epoch. Four priced calls must fit a ceiling below USD 10. The server checks the version’s pins first and answers judge_profile_stale if one moved.

A start carries the preflight’s fields plus idempotency_key, approved_epoch and approved_plan_digest:

{"version": 1, "subject": {"kind": "saved_draft_revision", "draft_revision": 7},
 "ceiling_micro_usd": 500000, "idempotency_key": "q-b2-cal-1",
 "approved_epoch": 42, "approved_plan_digest": "sha256:…"}

approved_epoch is the preflight’s plan.authority_epoch and approved_plan_digest its top-level plan_digest. The server plans again before it starts. If the digest it computes isn’t the approved one, nothing starts and the answer is judge_profile_stale with reason plan_changed. Use a new idempotency key for every preflight you approve: the same key with a different approval is idempotency_payload_mismatch. The same key with the same request answers the same calibration, including after a lost response, so a retry doesn’t pay twice.

A start and a calibration get answer link (profile_id, profile_version, comparison_id, idempotency_key, calibration_id, started_by, created_at, linked_at), calibration, the run redacted with your content.read and cost.read as the comparison’s calibration read redacts it, and summary:

A calibration list answers the links alone. Cost, verdicts and state stay on the calibration; the profile stores none of them.

Judge profile errors

The problem document has the dataset errors’ shape. A judge_profile_invalid answer also carries problems, a list of {control_id, field, reason}, and a judge_profile_stale answer carries reason.

The calibration lane’s own answers pass through unchanged on a preflight or start, including the spend and budget codes, dependency_unavailable and the judge_* codes. Gateway quality evidence lists each code with its status.

Judge profile lifecycle

A profile’s state is active, deleting or purged, and moves forward only. The workspace’s data-exit process drives the last two; there’s no per-profile delete in this release.

• Workspace exclusion. Once a workspace is excluded, every profile read and write in it answers workspace_excluded.
• Export. An export carries each profile version, with its calibration links, as a judge_profile row in the comparisons section, as listed under evaluation evidence in an export bundle.
• Purge. A purge deletes the workspace’s calibration links, profile versions and profiles, names included, along with the rest of its data. The calibrations themselves follow the comparison lifecycle.

How this page was checked

The field names, bounds, codes and the digest rule were read from the dataset contract and the service code. The worked digest was computed by the service’s own digest function over the tutorial’s examples and recomputed with the jq recipe. The CLI’s behaviour for these operations is pinned by transcripts captured against a loopback server that answers the contract’s fixture shapes. That local verification does not show a live deployment’s answers, and it spent nobody’s money.

The offline-evaluation section was read from the evaluation contract, the service’s types, digest and error code, and the three clients. Its CLI behaviour is pinned by transcripts from the same kind of loopback server, whose digests are computed with the contract’s rule. None of that shows a deployment’s worker or its timing.

The comparison-definitions section was read from the definition contract, the service’s types and error codes, the HTTP adapter and the three clients. Its CLI behaviour is pinned by transcripts from a loopback server that answers the contract’s fixture shapes and doesn’t apply the management-key rule. No preflight or start on this page has run against a deployment, and no provider was called.

The judge-profiles section was read from the judge profile contract, the service’s types, digests and error codes, the HTTP adapter, the calibration lane’s checks and the three clients. Its CLI behaviour is pinned by transcripts from a loopback server that answers the contract’s fixture shapes and doesn’t apply the management-key rule. No calibration on this page has run against a deployment, and no judge was called.