Gateway quality evidence vocabulary

This is the closed vocabulary behind a hosted gateway’s cost and quality comparisons: the operations, the words an answer can carry, and the codes a request can be refused with. What a cost and quality comparison proves explains what the words mean; this page is the list.

Every operation below hangs under one gateway environment, because a comparison is evidence about what one environment is serving. Every one of them needs comparisons.run in that environment, including the reads.

Operations

All paths are relative to /portal/api/v1/customer/v1/organizations/{org}/workspaces/{ws}/applications/{app}/gateways/{gw}/environments/{env}.

Operation	Method	Path
customer.comparisons.create	POST	/comparisons
customer.comparisons.list	GET	/comparisons
customer.comparisons.get	GET	/comparisons/{cmp}
customer.comparisons.samples.publish	POST	/comparisons/{cmp}/samples
customer.comparisons.samples.get	GET	/comparisons/{cmp}/samples/{ver}
customer.comparisons.runs.preflight	POST	/comparisons/{cmp}/runs/preflight
customer.comparisons.runs.start	POST	/comparisons/{cmp}/runs
customer.comparisons.runs.list	GET	/comparisons/{cmp}/runs
customer.comparisons.runs.get	GET	/comparisons/{cmp}/runs/{crun}
customer.comparisons.runs.cancel	POST	/comparisons/{cmp}/runs/{crun}/cancel
customer.comparisons.results.get	GET	/comparisons/{cmp}/runs/{crun}/results
customer.comparisons.verdicts.list	GET	/comparisons/{cmp}/runs/{crun}/verdicts
customer.comparisons.verdicts.review	POST	/comparisons/{cmp}/runs/{crun}/verdicts
customer.comparisons.decisions.create	POST	/comparisons/{cmp}/runs/{crun}/decisions
customer.comparisons.decisions.list	GET	/comparisons/{cmp}/runs/{crun}/decisions
customer.comparisons.decisions.get	GET	/comparisons/{cmp}/runs/{crun}/decisions/{cdec}

runs.start is the only one of the sixteen that carries an Idempotency-Key, and the key is the caller’s to choose: the same key with the same plan answers with the same run, and the same key with a different ceiling or scorer is duplicate_comparison_run.

The words an answer carries

Comparison kind is fixed at creation: labeled (each answer is checked against the expected label on the case) or judged (a model from another family grades both answers against the case’s reference notes).

Run state moves forward only.

State	Meaning
preflight	The plan and the pins exist; nothing is queued
queued	The run is committed; no arm has dispatched and nothing can have been spent
arms_running	Cells are dispatching, so provider calls may have happened
scoring	Every cell is terminal and verdicts are being written; judge calls happen here
complete	Every intended cell is terminal and every intended verdict exists
partial	The run finished with some cases missing, each listed with its reason
abandoned	Cancelled, or gave up before it could produce evidence

partial is a truthful end and not a failure. The missing cases still count in the quality denominator.

Verdict source is label_check, judge or human. A human verdict supersedes a machine one for the decision and deletes nothing.

Verdict outcome is pass, fail or unknown. An unknown always names a reason and is counted against the tolerance the bar declared, rather than as a failure.

Decision outcome is meets_bar, does_not_meet_bar or not_enough_evidence. It is computed from the verdicts and the bar; a request that claims a different one is refused.

Decision action is keep_route or change_route. Neither deploys anything.

Figure states

A figure that cannot be computed is absent with a named reason. There is no state that means zero.

Reason	Meaning
denominator_zero	The baseline’s comparable cost is zero, so there is no ratio
unpriced_population	Some comparable cell is pending or declined a price
no_comparable_cases	No case has both arms terminal and both priced
scope_not_authorized	The reader holds no cost.read; the quality evidence is still served

Gap reasons

One case can produce several gaps and all of them are reported, with the sharper code in detail where one exists.

Reason	Meaning
cell_missing	No run was recorded for that case on that arm
cell_not_terminal	The run for that cell had not finished
cell_failed	The cell finished and did not succeed; detail carries the failure code
cell_unpriced	The cell is terminal and its charge is pending or unknown
verdict_missing	The cell produced an answer and nothing graded it
verdict_unknown	The cell was graded unknown; detail carries the verdict’s reason
verdict_failed	The cell was graded fail; a failure on a critical case sorts first

Refusal codes

Statuses are the ones this lane publishes. A refusal always carries a sentence a person can act on.

Code	Status	What happened
comparisons_not_enabled	501	This deployment composes no comparison backend
comparison_not_found	404	No such comparison, run, sample version or decision here
comparison_kind_mismatch	409	Labelled material on a judged comparison, or the reverse
sample_set_empty	422	Fewer than two examples were chosen
sample_set_too_large	422	More than fifty were
sample_case_not_found	404	A chosen case is not a case of this gateway
sample_case_version_mismatch	409	The version named is not one of that case’s versions
expected_label_required	422	A labeled set left a case with no answer key
reference_notes_required	422	A judged set left a case with nothing to grade against
acceptance_rule_required	422	The bar was not declared, or asks for nothing
acceptance_rule_immutable	409	The bar was set before the run and cannot change now
spend_ceiling_above_remaining	409	The ceiling is more than the workspace has left
judge_connection_not_granted	403	No connection here is allowed to be used for judging
judge_same_family_refused	422	Judge and arm share a product line; nothing was sent
comparison_run_not_terminal	409	The run is still going, so there is nothing to decide on
comparison_evidence_incomplete	409	It finished with examples missing
comparison_evidence_stale	409	A pin moved; re-run against the current setup
baseline_release_changed	409	What you measured against is no longer what is served
comparison_outcome_does_not_support_change	409	The evidence does not say meets_bar
decision_outcome_mismatch	409	The outcome claimed is not the one the verdicts compute
decision_requires_release_authority	403	Changing the route needs permission to deploy
decision_superseded	409	A newer decision was recorded for this run
duplicate_comparison_run	409	That key was used with a different ceiling or scorer
comparison_canceled	409	The run was canceled; what ran is kept with its cost

Six codes that arrive as a verdict reason, not as an HTTP refusal

These are decided while the run is going, long after the start was accepted. Each appears as the reason_code on an unknown verdict and in the run’s gaps.

Code	What happened
spend_ceiling_exceeded	The run reached the limit it was given and stopped
judge_ceiling_reached	The judge half of the limit ran out; the rest is ungraded
judge_no_verdict	The judge answered unreadably; the call may still be charged
judge_unavailable	The judge could not be reached for that example
judge_paid_effect_unknown	A judge call was sent and its completion cannot be established
judge_connection_not_pinned	The pinned configuration binds no connection for judging