# Gateway quality evidence vocabulary

The sixteen comparison operations, the outcome and verdict words, the gap reasons, and every refusal code with its status.

Source: https://tokentriage.com/docs/reference/gateway-quality-evidence

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

{/* Provenance: the operations are internal/control/foundation/httpapi/testdata/contract.golden;
    the words are internal/control/foundation/comparison/{types,refusals}.go and
    internal/control/foundation/costquality/gaps.go; the statuses are
    internal/control/foundation/httpapi/registry.go. Regenerate the first table with
    `go test ./internal/control/foundation/httpapi/ -run TestFoundationContractMatchesTheGolden
    -update-contract-golden`. */}

This is the closed vocabulary behind a hosted gateway's cost and quality comparisons: the
operations, the words an answer can carry, and the codes a request can be refused with.
[What a cost and quality comparison proves](/docs/build/cost-quality-evidence) explains
what the words mean; this page is the list.

Every operation below hangs under one gateway environment, because a comparison is
evidence about what one environment is serving. Every one of them needs `comparisons.run`
in that environment, including the reads.

## Operations

All paths are relative to `/portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}`.

| Operation | Method | Path |
|---|---|---|
| `customer.comparisons.create` | POST | `/comparisons` |
| `customer.comparisons.list` | GET | `/comparisons` |
| `customer.comparisons.get` | GET | `/comparisons/{cmp}` |
| `customer.comparisons.samples.publish` | POST | `/comparisons/{cmp}/samples` |
| `customer.comparisons.samples.get` | GET | `/comparisons/{cmp}/samples/{ver}` |
| `customer.comparisons.runs.preflight` | POST | `/comparisons/{cmp}/runs/preflight` |
| `customer.comparisons.runs.start` | POST | `/comparisons/{cmp}/runs` |
| `customer.comparisons.runs.list` | GET | `/comparisons/{cmp}/runs` |
| `customer.comparisons.runs.get` | GET | `/comparisons/{cmp}/runs/{crun}` |
| `customer.comparisons.runs.cancel` | POST | `/comparisons/{cmp}/runs/{crun}/cancel` |
| `customer.comparisons.results.get` | GET | `/comparisons/{cmp}/runs/{crun}/results` |
| `customer.comparisons.verdicts.list` | GET | `/comparisons/{cmp}/runs/{crun}/verdicts` |
| `customer.comparisons.verdicts.review` | POST | `/comparisons/{cmp}/runs/{crun}/verdicts` |
| `customer.comparisons.decisions.create` | POST | `/comparisons/{cmp}/runs/{crun}/decisions` |
| `customer.comparisons.decisions.list` | GET | `/comparisons/{cmp}/runs/{crun}/decisions` |
| `customer.comparisons.decisions.get` | GET | `/comparisons/{cmp}/runs/{crun}/decisions/{cdec}` |

`runs.start` is the only one of the sixteen that carries an `Idempotency-Key`, and the
key is the caller's to choose: the same key with the same plan answers with the same run,
and the same key with a different ceiling or scorer is `duplicate_comparison_run`.

## The words an answer carries

**Comparison kind** is fixed at creation: `labeled` (each answer is checked against the
expected label on the case) or `judged` (a model from another family grades both answers
against the case's reference notes).

**Run state** moves forward only.

| State | Meaning |
|---|---|
| `preflight` | The plan and the pins exist; nothing is queued |
| `queued` | The run is committed; no arm has dispatched and nothing can have been spent |
| `arms_running` | Cells are dispatching, so provider calls may have happened |
| `scoring` | Every cell is terminal and verdicts are being written; judge calls happen here |
| `complete` | Every intended cell is terminal and every intended verdict exists |
| `partial` | The run finished with some cases missing, each listed with its reason |
| `abandoned` | Cancelled, or gave up before it could produce evidence |

`partial` is a truthful end and not a failure. The missing cases still count in the
quality denominator.

**Verdict source** is `label_check`, `judge` or `human`. A human verdict supersedes a
machine one for the decision and deletes nothing.

**Verdict outcome** is `pass`, `fail` or `unknown`. An `unknown` always names a reason
and is counted against the tolerance the bar declared, rather than as a failure.

**Decision outcome** is `meets_bar`, `does_not_meet_bar` or `not_enough_evidence`. It is
computed from the verdicts and the bar; a request that claims a different one is refused.

**Decision action** is `keep_route` or `change_route`. Neither deploys anything.

## Figure states

A figure that cannot be computed is absent with a named reason. There is no state that
means zero.

| Reason | Meaning |
|---|---|
| `denominator_zero` | The baseline's comparable cost is zero, so there is no ratio |
| `unpriced_population` | Some comparable cell is pending or declined a price |
| `no_comparable_cases` | No case has both arms terminal and both priced |
| `scope_not_authorized` | The reader holds no `cost.read`; the quality evidence is still served |

## Gap reasons

One case can produce several gaps and all of them are reported, with the sharper code in
`detail` where one exists.

| Reason | Meaning |
|---|---|
| `cell_missing` | No run was recorded for that case on that arm |
| `cell_not_terminal` | The run for that cell had not finished |
| `cell_failed` | The cell finished and did not succeed; `detail` carries the failure code |
| `cell_unpriced` | The cell is terminal and its charge is pending or unknown |
| `verdict_missing` | The cell produced an answer and nothing graded it |
| `verdict_unknown` | The cell was graded `unknown`; `detail` carries the verdict's reason |
| `verdict_failed` | The cell was graded `fail`; a failure on a critical case sorts first |

## Refusal codes

Statuses are the ones this lane publishes. A refusal always carries a sentence a person
can act on.

| Code | Status | What happened |
|---|---|---|
| `comparisons_not_enabled` | 501 | This deployment composes no comparison backend |
| `comparison_not_found` | 404 | No such comparison, run, sample version or decision here |
| `comparison_kind_mismatch` | 409 | Labelled material on a judged comparison, or the reverse |
| `sample_set_empty` | 422 | Fewer than two examples were chosen |
| `sample_set_too_large` | 422 | More than fifty were |
| `sample_case_not_found` | 404 | A chosen case is not a case of this gateway |
| `sample_case_version_mismatch` | 409 | The version named is not one of that case's versions |
| `expected_label_required` | 422 | A labeled set left a case with no answer key |
| `reference_notes_required` | 422 | A judged set left a case with nothing to grade against |
| `acceptance_rule_required` | 422 | The bar was not declared, or asks for nothing |
| `acceptance_rule_immutable` | 409 | The bar was set before the run and cannot change now |
| `spend_ceiling_above_remaining` | 409 | The ceiling is more than the workspace has left |
| `judge_connection_not_granted` | 403 | No connection here is allowed to be used for judging |
| `judge_same_family_refused` | 422 | Judge and arm share a product line; nothing was sent |
| `comparison_run_not_terminal` | 409 | The run is still going, so there is nothing to decide on |
| `comparison_evidence_incomplete` | 409 | It finished with examples missing |
| `comparison_evidence_stale` | 409 | A pin moved; re-run against the current setup |
| `baseline_release_changed` | 409 | What you measured against is no longer what is served |
| `comparison_outcome_does_not_support_change` | 409 | The evidence does not say `meets_bar` |
| `decision_outcome_mismatch` | 409 | The outcome claimed is not the one the verdicts compute |
| `decision_requires_release_authority` | 403 | Changing the route needs permission to deploy |
| `decision_superseded` | 409 | A newer decision was recorded for this run |
| `duplicate_comparison_run` | 409 | That key was used with a different ceiling or scorer |
| `comparison_canceled` | 409 | The run was canceled; what ran is kept with its cost |

<Details summary="Six codes that arrive as a verdict reason, not as an HTTP refusal">

These are decided while the run is going, long after the start was accepted. Each appears
as the `reason_code` on an `unknown` verdict and in the run's gaps.

| Code | What happened |
|---|---|
| `spend_ceiling_exceeded` | The run reached the limit it was given and stopped |
| `judge_ceiling_reached` | The judge half of the limit ran out; the rest is ungraded |
| `judge_no_verdict` | The judge answered unreadably; the call may still be charged |
| `judge_unavailable` | The judge could not be reached for that example |
| `judge_paid_effect_unknown` | A judge call was sent and its completion cannot be established |
| `judge_connection_not_pinned` | The pinned configuration binds no connection for judging |

</Details>

<Callout type="key">
A refusal naming a comparison, run, sample version or decision that belongs to another
company answers `comparison_not_found`, the same as one nobody created. The lane does not
confirm that another company's row exists.
</Callout>

<NextSteps
  items={[
    { title: 'Run a comparison', href: '/docs/build/compare-cost-and-quality', description: 'The commands, the plan before you spend, and the decision record.' },
    { title: 'What the evidence proves', href: '/docs/build/cost-quality-evidence', description: 'How the outcome is computed and why a sample reduction is not a saving.' },
    { title: 'Gateway insight reads', href: '/docs/reference/gateway-insights', description: 'The read contract for what the gateway actually served and spent.' },
  ]}
/>