What a cost and quality comparison proves

A comparison answers a narrow question: on these examples, under the bar this team wrote down beforehand, did the candidate hold up, and what did each side cost? Everything on this page is about keeping that question narrow, because the failure mode of model evaluation is a single number that sounds like a promise.

Compare cost and quality before you switch models is the how-to. This page is why the answer is shaped the way it is.

One sentence, and the counts that produced it

The banner is one of three sentences, and it is a count over cases rather than a score:

Outcome	Sentence	What it means
meets_bar	Meets your bar on these examples	Every criterion you declared held
does_not_meet_bar	Does not meet your bar	The candidate was graded and a criterion did not hold
not_enough_evidence	Not enough evidence	Too many cases lack verdicts, or the saved rule requires human review that is still missing

The checks run in a fixed order. Sufficiency comes first: counting over cells that never answered is how a run with half its cases missing reads as a pass. Then the criteria, together, so a failure on a case you marked critical disqualifies the candidate however the counts fall. Only then does the run get to say the bar was met.

A judged run can have a machine verdict for every answer and still need human review. The banner shows that pending review separately from unknown verdicts. The saved rule and the machine’s recorded failures remain visible while review is pending.

There is no blended quality score anywhere in the answer. A single number mixing accuracy and price is the one figure nobody can check against anything.

An ungraded case is not a failure

Three things can leave a cell without a verdict, and each carries its own reason:

• The judge would grade its own family. The check runs before the call, so the pair costs nothing: judge_same_family_refused.
• The judge answered and the answer was unreadable, or the provider could not be reached: judge_no_verdict, judge_unavailable. The first may still have been charged, and the run says so.
• The money set aside for judging ran out, or a call was sent and this build cannot tell whether it completed: judge_ceiling_reached, judge_paid_effect_unknown. Neither is retried.

Counting any of those as a failure would tell you the candidate was tried and found wanting where it was not graded at all. They count against the tolerance you declared; over that tolerance the run is not_enough_evidence.

What “cheaper” measures

The cost block is beside the quality and not inside it.

• Per arm, the priced spend and how many cases are behind it.
• The judge’s spend, separately, because it is the cost of measuring rather than the cost of serving.
• The comparable population, as k of N: the cases where both arms finished and both are priced. The reduction is computed over those and nothing else.
• The reduction, as a ratio of two integers rather than a percentage the client rounds.

Unpriced judge calls have unknown cost even when their quality verdicts are available. When any generation or judge call is unpriced, the displayed run amount is a known subtotal. Total run cost remains unknown; the subtotal excludes those calls. A complete serving-cost comparison therefore does not imply a complete experiment bill.

Worked example: 19 of 20, and why the figure shrank

Twenty cases. One case is still awaiting a price on both arms, so the comparable population is 19. Baseline priced spend over those 19 is 412,000 nano-USD; the candidate is 126,000. The reduction is 286000 / 412000 over 19 of 20 cases, and coverage reads partial with the reason unpriced_population.

If the last price never settles, the figure stays at 19 of 20. If it settles tomorrow, the summary is recomputed on the next read, because nothing here is stored. A decision already recorded is not invalidated by that: its digest covers the sample, both pins, the scorer, the bar and the verdicts, and not the cost.

A sample serving-cost reduction is not a monthly saving. It is what these examples would have cost on each side. Traffic mix, prompt length, cache behaviour and retries all move the monthly figure, and the way to learn the monthly figure is to change the route and measure it.

Projected saving at this volume

The gateway overview can estimate how the tested cost difference applies to comparable application requests in your selected window. Expand the comparison details to see how many requests qualified, how many remain uncovered, and which saved comparison supplies the evidence.

For the supported single cohort, the estimate is the covered request count × mean paired baseline-minus-candidate cost. All intended test pairs must be priced and the candidate must meet its quality bar. A negative result stays negative. The displayed amount is rounded toward zero to whole nano-USD only after multiplication.

The selection rule is saved with the comparison before an estimate is shown. Requests must match one tested configuration’s model, effective controls, connection version and price basis, as well as its route, text-input size bucket, output limit, generation settings and system/developer instructions. The first version supports text-only chat with those observations available; unsupported inputs and requests whose capture policy withholds the required metadata remain uncovered. Historical requests without that metadata are not retroactively classified.

This is a provider-cost estimate for the covered population. Similar request metadata does not prove that future user questions will have the same difficulty or output length. It is not realized savings, an invoice prediction, or a net-return calculation. Test and judge charges and platform fees remain separate. Changed settings, missing prices, failed quality or no comparable application requests give an unavailable reason instead of a zero estimate.

The comparison uses its saved test evidence, while the application count uses the same bounded report context as the overview. Refresh an expired report context. Opening an old saved view still checks your current access; the view cannot grant access to the comparison or its costs.

Why two words for stale evidence

A comparison is evidence about one pinned pair of configurations in one environment, over one set of connections. When any of that moves, the evidence stops describing what you would ship, and the answer says which thing moved:

• comparison_evidence_stale: a pin moved. A new draft revision, a mandatory-controls bump, a different environment, a rotated credential. Re-run against the current setup.
• baseline_release_changed: somebody deployed. What you measured against is no longer what the gateway serves. Re-run against what is live now.

Both are checked when the decision is recorded and again immediately before a release is created, because the gap between deciding and deploying is where a stale deploy happens. Old evidence stays pinned to what it tested and is not re-labelled.

Who sees what

Reading a comparison needs comparisons.run in that environment: the evidence is what a route change rests on, so reading a gateway’s configuration is not enough. Two finer grants narrow what comes back rather than refusing the read:

• Without cost.read, the quality evidence arrives and the cost block does not. The answer reads unavailable with scope_not_authorized. It does not show a zero.
• Without content.read, the outcomes arrive and the prompts, completions, judge answers and reviewer notes do not. The answer says content was withheld, because “nothing was attached” and “you may not see it” are different facts.

What a judge is, and is not

A judge is one model call per answer, from a family different from both arms, grading against reference notes you wrote. Its verdict is evidence a person can read and override: a human verdict supersedes a machine one for the decision and deletes nothing, so both stay visible and the paid call behind the superseded verdict is still counted.

Judges agree with careful human raters often enough to be useful and not often enough to be authority. Treat a judged comparison as a structured second opinion over your own examples, and keep the cases where it disagreed with you.