Why your number is reported, not measured

You collected the labels, ran the validation, and the figure on the screen still reads reported. Nothing is broken and no switch was missed. reported is an answer to a question about what may be claimed, and that answer has a cause with a name you can look up.

A quality basis is not a label somebody picked. It is the minimum of every cap that applies to one observation, and min() is the operator: adding a cap can only lower the result, and the one lowest cap settles the figure however many caps sit above it.

effective quality basis = min( metric ground-truth class, evaluator kind, outcome source, judge marker )

Three caps apply to every quality outcome. A judge-graded row carries a fourth, and that fourth is the one your figure has probably met. This page is about telling which cap is holding a number down, and what a promotion moves when it lands. If this is your first page in the section, LLM judges introduces the vocabulary; the walk that produces a promotion is How a judge reaches a measured basis, and there is no procedure here.

The caps, and the operator between them

Before you can move a number you have to know which constraint owns it, and a quality basis has more than one owner. Each cap answers a different question about the same observation, and each one can say only measured or reported.

The cap	The question it answers	When it says `reported`
Metric ground-truth class	Is there a ground truth on this property to observe at all?	on a `judge`-class or `heuristic`-class metric
Evaluator kind	Did this instrument observe the fact, or estimate it?	on a `judge`-kind evaluator
Outcome source	May the door this observation arrived through assert a measurement?	on the `otel` source
The judge marker on the row	Did a model grade this row?	wherever a judge disclosure is present and no promotion stands

The execution, reference and human ground-truth classes license measured. Four of the five registered outcome sources license it too (evaluator, human, business and feedback), with otel the one that caps: an evaluation that ran in somebody else’s process is a claim, not an observation this deployment made. Evaluators & metrics carries the per-evaluator table.

The fourth cap is the one worth understanding, because it is the one your figure has met. It needs no registry lookup to bind. Any observation arriving with a judge disclosure on it caps at reported until the B.7 validation protocol promotes that observation’s own subject. The row states on its face that a model graded it, so the cap is checkable from the bytes alone, and a judge score cannot travel as anonymous ground truth even where the line that wrote it stamped basis: measured.

That last clause is not hypothetical, and it is the second way a figure comes back reported. More than one writer appends to the ledger, so a line asserting basis: measured reaches a dashboard unchallenged. The read surface therefore re-derives the ceiling instead of trusting the row, and publishes the minimum of what the row declares and what the caps allow. A row claiming more than its caps support is published at its caps, not turned away.

No cap raises another. min() runs over whichever caps apply, so the only way a figure goes up is for a cap to stop applying, and a standing promotion is the one thing that stops one. There is no tiebreak to state, and none is wanted: two caps at reported give the same answer as one, and when several tie at the binding value the disclosure names all of them rather than picking a winner.

Four caps converging on one node is easier to see than to read, so here is the arithmetic with its values on it, worked on a groundedness row.

Read it down the left column and then across. The writer’s claim enters at the top, each cap below it carries the ceiling it sets on this row, and all five meet at the minimum. The outcome source is marked as the one that does not lift, and it holds the same value before a promotion and after it. The claim is forced down rather than turned away, so the observation is kept and the coercion travels beside it.

Three of the four caps are marked as lifting there, which is the arithmetic maximum on a groundedness row rather than a state this build’s metrics reach. The worked example below lands on two, and the paragraph after it says what holds the third down.

Which cap is holding your number down

Naming the caps is not the same as knowing which one bound, so walk one figure end to end. This time take a task_completion outcome on the segment task_completion|tier=frontier|model=gpt-5|endpoint=chat, produced by the built-in judge-task-completion evaluator, written through the evaluator outcome source, on a row carrying a judge disclosure that names the judge model and the prompt sha.

The cap	Its value on this row	The ceiling it sets
Metric ground-truth class	`human` — a person’s verdict on task completion is the ground truth	`measured`
Evaluator kind	`judge` — `judge-task-completion` calls a model to decide	`reported`
Outcome source	`evaluator` — a registered evaluator ran and wrote the row	`measured`
The judge marker on the row	present, and no promotion stands for this subject	`reported`

min(measured, reported, measured, reported) is reported, and that is the figure you are looking at. The metric could have supported a measurement. The door could have. The instrument could not, and the row’s own disclosure said as much.

Two caps bind here, and both get stated back where the claim arrived over the wire. POST /tt/api/v1/outcomes accepts such a row, stores it at the ceiling, and returns a basis_ceiling_coerced disclosure carrying from, to, and a detail sentence that names every cap which held rather than the first one it found. That door accepts no judge disclosure, so the three registry caps are the whole arithmetic there.

A row the evaluator runner writes has no submitted claim to disclose against, so it is capped as it is written and derived again on the way out. On the aggregate surfaces the same fact appears under a different name: a segment whose evidence was all capped down comes back insufficient with the reason basis_ceiling_reported.

Now promote that subject. Two of the four caps move and two do not.

The evaluator-kind cap lifts to measured. A judge kind is one of the arms that yields to a standing promotion for the row’s own subject.
The judge marker stops applying. It exists to hold a judge that has no promotion behind it, so a promotion is the condition under which it is not added at all.
The metric class does not move. It was already at measured, and human is not a judge-tied class.
The outcome source does not move. No promotion reaches a source ceiling, on any row.

min() over what is left is measured, and the figure changes on the rows written from that instant onward.

Change the metric and the arithmetic changes with it. groundedness is a judge-class metric, so its class cap starts at reported and a promotion would lift that one too, which is why the diagram above, worked on a groundedness row, shows three caps lifting where this row shows two.

That third lift is arithmetic rather than a route. groundedness is the only judge-class metric this build registers, and its descriptor sets PassFloor: nil, so the runner leaves every groundedness row without a pass verdict. The gold-set assembler records a verdict only where one is set, so a judge-groundedness subject accumulates none and its run stops at the first gate. Configure judge-task-completion on task_completion for anything that has to reach a promotion, and Why didn’t my judge validate? has the symptom.

answer_relevance is heuristic-class, and no promotion reaches it at all: whether a heuristic can observe ground truth is not a question the validation protocol answers, so a heuristic-class figure stays reported however good your judge turns out to be.

The other place a number is held back

The basis ceiling is not the only refusal on a quality screen, and the second one speaks a different vocabulary. A quality basis says what one observation may claim. A routing verdict says whether routing beat its baseline, over a set of them. Those two are the pair most often conflated on a quality screen, and the two verdict vocabularies note keeps them apart.

What is worth carrying away from the second one, while you are on this page, is that it declines in the same shape as the first. It names the floor it fell short of. It prints the bound it computed anyway. And it gives the outcome a name from a closed set, so CANNOT_MEASURE is a state rather than an error and the verb exits 0. Where that verdict comes from is Eval programs and experiments; none of it moves a cap.

One declined routing verdict, printed in full

This is tokentriage eval --quality shadow over a sixteen-line ledger. Six labels in it belong to the eval track rather than to this page. BEAT is the verdict that routing beat its baseline. CP-LCB is a Clopper–Pearson lower confidence bound. AIQ, PGR, VB and Ep are descriptive router-economics figures, and the block marks each of them as not the licensing basis.

$ tokentriage eval --quality shadow --log ./decisions.jsonl
=== TokenTriage honesty eval ===
verdict: CANNOT_MEASURE the paired BEAT — 8 decided pair(s) below the floor of 30 (report.BeatMinDecidedPairs, raise-only; need 30, have 8) — too few shadow-vs-chosen pairs with both sides decided to license BEAT — computed and refused by the floor: CP-LCB 0.135 <= 0.50 (½−m, m=0; one-sided 0.95) — estimate 0.667, n₊=2 of 3 discordant; 8 decided pairs, 5 ties, 0 abstained; method pointwise-pair
quality source: shadow (labeled iff judged-shadow grade present (chosen_ok != nil); counterfactual per-cost verdict from both tiers' grades)
labeled rows:   8 of 8 (floor 8)

TIER POINTS  (mean cost, mean quality over that tier's labeled observations)
TIER                  OBS  MEAN-COST  MEAN-QUAL  SHARE
frontier              8    $0.000240  0.7500     50.0%
premium               8    $0.002100  0.8750     50.0%
ROUTER (all labeled)  8    $0.000240  0.7500     100.0%

PAIRED BEAT BOUND  (B.1 licensing basis: shadow vs chosen, pointwise pairs — the only estimand that licenses BEAT)
  decided pairs:  8  (floor 30, report.BeatMinDecidedPairs; 8 graded rows)
  ties:           5  (support the claim only through this count)
  discordant:     3  (floor 10, report.BeatMinDiscordant): n₊=2 shadow wins, n₋=1 chosen wins
  abstained:      0  (excluded-and-counted; never a candidate failure — JH12)
  estimate:       0.6667  = n₊/discordant
  CP-LCB:         0.1354  (one-sided 0.95, clopper-pearson-v1) vs threshold 0.50 (½−m, m=0)
  licensing:      REFUSED — 8 decided pair(s) below the floor of 30 (report.BeatMinDecidedPairs, raise-only; need 30, have 8) — too few shadow-vs-chosen pairs with both sides decided to license BEAT — computed and refused by the floor: CP-LCB 0.135 <= 0.50 (½−m, m=0; one-sided 0.95) — estimate 0.667, n₊=2 of 3 discordant; 8 decided pairs, 5 ties, 0 abstained; method pointwise-pair

FRONTIER  (descriptive router-vs-hull comparison at the router's own cost — ESTIMATED, never the BEAT license; see PAIRED BEAT BOUND)
  router quality:        0.7500  @ cost $0.000240
  zero-router @ router:  0.7500
  delta:                 0.0000

AIQ  (descriptive shadow-economics, NOT the verdict basis; shared cost domain [$0.000240, $0.002100])
  AIQ(router):      0.7500
  AIQ(zero-router): 0.8125
  delta:            -0.0625

PGR  (binary strong/weak: frontier → premium)
  strong-weak gap:  0.1250
  PGR(router):      0.0000
  APGR/CPT:         n/a — a single realized router has no call-rate sweep (use calibrate to sweep α)

RouterEval
  VB (vs best single):  0.8571  (>1 means the router beats every single model)
  Ep (selection bits):  1.0000
  Ep normalized:        1.0000  (low means effectively a single-model policy)

Exit status is 0. Both halves of that ledger need naming before you quote a number out of it. The eight primary decision rows were minted and priced by the binary itself, routing to a loopback upstream stub, so frontier and $0.000240 are its own readings over that traffic.

The eight companion shadow grades were planted by the documentation harness in the shipped golden’s field shape, because no config key points the shadow-plane judge at a loopback host. So 8 decided pairs, 5 ties and n₊=2 are that plant, and every number under them moves if you change it. The floors, the Clopper–Pearson bound, the verdict and every word of the refusal are the binary’s. The capture was run twice against the same ledger, and the two outputs are byte-identical.

Four things in that output are worth keeping.

The floor is named with need and have, twice. need 30, have 8 appears in the header verdict and again under licensing:. A declined licence that hides its own arithmetic cannot be argued with; this one can.
The bound prints even though it lost. CP-LCB: 0.1354 sits under an estimate: 0.6667, because three discordant pairs cannot support a bound and the output says so instead of promoting the estimate into a claim.
abstained: 0 prints at zero. No abstentions is a measurement, and it is written excluded-and-counted rather than dropped from the block.
The descriptive numbers still render, and are labelled. FRONTIER, AIQ, PGR and RouterEval each carry a caption saying they are not the licensing basis, so delta: 0.0000 is not an answer to the question the verdict declined.

What a promotion buys, and what it does not

A promotion is narrower than the word suggests, and knowing its shape is what stops you expecting the wrong number to move.

It is scoped to a subject, not to a judge. The subject is the judge id, the judge model, the prompt sha, the evaluator version, the metric and the segment. The promotion is filed under the identity {judge id, metric, segment} and pinned to a version built from the other three, so moving the judge model, the prompt or the evaluator version leaves the pin unmatched: the reading comes back version_changed, and the subject’s rows ride the reported ceiling again.

It does not travel across a segment. The same judge on a different tier is a different subject, and an unvalidated one.

It is forward-only, so it does not rewrite the rows you already have. The licence compares a row’s own timestamp against the instant the promotion was recorded, and a row written before that instant stays reported permanently. A window that ends before the promotion stays reported however well the judge validated, which is the most common reason an operator promotes, re-reads the same window, and concludes the promotion failed.

It does not lift a cap that was not about the judge. A heuristic-class metric and the otel source both cap at reported, and neither yields to a promotion. Three of the four caps consult the promotion registry; the source ceiling is the one that does not. Two of those three move on any row this build can promote, which is what the worked example above shows.

It does not make judging spend measurable. spend is served as a present null on every judges row, which the binary’s own note reads as NOT MEASURED: judging costs money and is recorded per row, and no shipped producer sums those dollars onto a subject.

How this goes wrong

Reading reported as a fault. It is one of three values in a closed vocabulary, and it means a proxy nobody has measured yet. A screen showing it is working.

Reading the wrong reported. Guardrail confidence carries its own five-value basis vocabulary (exact, fixture, calibrated, reported, insufficient) and quality basis has three. Both spell one value reported, and they are different claims about different things. On a judge page, write quality basis and mean it.

Reading the wrong source. Two five-value vocabularies here are called sources, and they share one member. An outcome source says which door an observation came through: evaluator, human, business, otel, feedback. A quality source says which labelled signal an eval reads for a row’s quality: oracle, outcome, shadow, feedback, and the eval:<metric> family. The routing verdict on this page prints quality source: shadow, which is not one of the four caps.

Promoting, then re-reading the window you were already looking at. Forward-only means the figure moves on rows written afterwards. Re-run the report over a window that starts after the promotion instant before you conclude anything went wrong.

What this page does not cover

Which situation produced your reported. The basis_ceiling_reported reading on a quality summary counts how many of a segment’s observations were capped, and states outright that it cannot tell you why. The registry holds promotions only, so three different histories read the same there: a judge nobody has validated, one that was measured and declined below the kappa floor of 0.5, and one that was promoted and later demoted. The record that carries a named floor with its need and have counts comes from tokentriage evals validate and GET /tt/api/v1/evals/judges, and Why didn’t my judge validate? starts from there.

What every null on a judges row means. iaa: null is not agreement of zero, and a κ of 0.0 is a measurement at chance level rather than a missing value. The table that separates them is on the walk that promotes a judge, which also says why κ travels without a confidence interval.

What happens to a promotion afterwards. No shipped binary can fire a drift demotion; the four-piece triad is demonstrated inside the test binary against a pinned vector. Judge drift, with no monitor between checks has the rest.

Turning a measured basis into a routing decision. A quality floor reads a measured basis only, so the ceiling is the precondition rather than the mechanism. Gate a rollout on a validated judge is the mechanism.