# Why your number is reported, not measured
A judge-graded figure is the minimum of four caps. Find which one is holding your number down, and which two a promotion lifts.
Source: https://tokentriage.com/docs/judges/basis-ceilings
---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';
import Details from '../../../components/Details.astro';
import JudgeCeilingDiagram from '../../../components/JudgeCeilingDiagram.astro';
You collected the labels, ran the validation, and the figure on the screen still reads
`reported`. Nothing is broken and no switch was missed. `reported` is an answer to a question
about what may be claimed, and that answer has a cause with a name you can look up.
A quality basis is not a label somebody picked. It is the **minimum** of every cap that applies to
one observation, and `min()` is the operator: adding a cap can only lower the result, and the one
lowest cap settles the figure however many caps sit above it.
```text
effective quality basis = min( metric ground-truth class, evaluator kind, outcome source, judge marker )
```
Three caps apply to every quality outcome. A judge-graded row carries a fourth, and that fourth is
the one your figure has probably met. This page is about telling which cap is holding a number
down, and what a promotion moves when it lands. If this is your first page in the section,
[LLM judges](/docs/judges/overview) introduces the vocabulary; the walk that produces a promotion
is [How a judge reaches a measured basis](/docs/judges/agreement), and there is no procedure here.
## The caps, and the operator between them
Before you can move a number you have to know which constraint owns it, and a quality basis has
more than one owner. Each cap answers a different question about the same observation, and each
one can say only `measured` or `reported`.
| The cap | The question it answers | When it says `reported` |
|---|---|---|
| Metric ground-truth class | Is there a ground truth on this property to observe at all? | on a `judge`-class or `heuristic`-class metric |
| Evaluator kind | Did this instrument observe the fact, or estimate it? | on a `judge`-kind evaluator |
| Outcome source | May the door this observation arrived through assert a measurement? | on the `otel` source |
| The judge marker on the row | Did a model grade this row? | wherever a judge disclosure is present and no promotion stands |
The `execution`, `reference` and `human` ground-truth classes license `measured`. Four of the five
registered outcome sources license it too (`evaluator`, `human`, `business` and `feedback`), with
`otel` the one that caps: an evaluation that ran in somebody else's process is a claim, not an
observation this deployment made. [Evaluators & metrics](/docs/calibration/evaluators) carries the
per-evaluator table.
The fourth cap is the one worth understanding, because it is the one your figure has met. It needs
no registry lookup to bind. Any observation arriving with a judge disclosure on it caps at
`reported` until the B.7 validation protocol promotes that observation's own subject. The row
states on its face that a model graded it, so the cap is checkable from the bytes alone, and a
judge score cannot travel as anonymous ground truth even where the line that wrote it stamped
`basis: measured`.
That last clause is not hypothetical, and it is the second way a figure comes back `reported`.
More than one writer appends to the ledger, so a line asserting `basis: measured` reaches a
dashboard unchallenged. The read surface therefore re-derives the ceiling instead of trusting the
row, and publishes the minimum of what the row declares and what the caps allow. A row claiming
more than its caps support is published at its caps, not turned away.
No cap raises another. `min()` runs over whichever caps apply, so the only way a figure goes up is
for a cap to stop applying, and a standing promotion is the one thing that stops one. There is no
tiebreak to state, and none is wanted: two caps at `reported` give the same answer as one, and
when several tie at the binding value the disclosure names all of them rather than picking a
winner.
`insufficient` is the third value in the quality basis vocabulary, and it is not a rung on this
ladder. It is a statement about a *set* of observations being too thin, and it carries `need` and
`have` counts. One row is `measured` or `reported`; a segment is what comes back `insufficient`.
Four caps converging on one node is easier to see than to read, so here is the arithmetic with its
values on it, worked on a `groundedness` row.
Read it down the left column and then across. The writer's claim enters at the top, each cap below
it carries the ceiling it sets on this row, and all five meet at the minimum. The outcome source is
marked as the one that does not lift, and it holds the same value before a promotion and after it.
The claim is forced down rather than turned away, so the observation is kept and the coercion
travels beside it.
Three of the four caps are marked as lifting there, which is the arithmetic maximum on a
`groundedness` row rather than a state this build's metrics reach. The worked example below lands
on two, and the paragraph after it says what holds the third down.
## Which cap is holding your number down
Naming the caps is not the same as knowing which one bound, so walk one figure end to end. This
time take a `task_completion` outcome on the segment
`task_completion|tier=frontier|model=gpt-5|endpoint=chat`, produced by the built-in
`judge-task-completion` evaluator, written through the `evaluator` outcome source, on a row
carrying a judge disclosure that names the judge model and the prompt sha.
| The cap | Its value on this row | The ceiling it sets |
|---|---|---|
| Metric ground-truth class | `human` — a person's verdict on task completion *is* the ground truth | `measured` |
| Evaluator kind | `judge` — `judge-task-completion` calls a model to decide | `reported` |
| Outcome source | `evaluator` — a registered evaluator ran and wrote the row | `measured` |
| The judge marker on the row | present, and no promotion stands for this subject | `reported` |
`min(measured, reported, measured, reported)` is `reported`, and that is the figure you are looking
at. The metric could have supported a measurement. The door could have. The instrument could not,
and the row's own disclosure said as much.
Two caps bind here, and both get stated back where the claim arrived over the wire.
`POST /tt/api/v1/outcomes` accepts such a row, stores it at the ceiling, and returns a
`basis_ceiling_coerced` disclosure carrying `from`, `to`, and a detail sentence that names every
cap which held rather than the first one it found. That door accepts no judge disclosure, so the
three registry caps are the whole arithmetic there.
A row the evaluator runner writes has no submitted claim to disclose against, so it is capped as it
is written and derived again on the way out. On the aggregate surfaces the same fact appears under a
different name: a segment whose evidence was all capped down comes back `insufficient` with the
reason `basis_ceiling_reported`.
Now promote that subject. Two of the four caps move and two do not.
- **The evaluator-kind cap lifts to `measured`.** A `judge` kind is one of the arms that yields to
a standing promotion for the row's own subject.
- **The judge marker stops applying.** It exists to hold a judge that has no promotion behind it,
so a promotion is the condition under which it is not added at all.
- **The metric class does not move.** It was already at `measured`, and `human` is not a judge-tied
class.
- **The outcome source does not move.** No promotion reaches a source ceiling, on any row.
`min()` over what is left is `measured`, and the figure changes on the rows written from that
instant onward.
Change the metric and the arithmetic changes with it. `groundedness` is a `judge`-class metric, so
its class cap starts at `reported` and a promotion would lift that one too, which is why the
diagram above, worked on a `groundedness` row, shows three caps lifting where this row shows two.
That third lift is arithmetic rather than a route. `groundedness` is the only `judge`-class metric
this build registers, and its descriptor sets `PassFloor: nil`, so the runner leaves every
groundedness row without a pass verdict. The gold-set assembler records a verdict only where one is
set, so a `judge-groundedness` subject accumulates none and its run stops at the first gate.
Configure `judge-task-completion` on `task_completion` for anything that has to reach a promotion,
and [Why didn't my judge validate?](/docs/judges/troubleshooting) has the symptom.
`answer_relevance` is `heuristic`-class, and no promotion reaches it at all: whether a heuristic can
observe ground truth is not a question the validation protocol answers, so a heuristic-class figure
stays `reported` however good your judge turns out to be.
## The other place a number is held back
The basis ceiling is not the only refusal on a quality screen, and the second one speaks a
different vocabulary. A quality basis says what **one observation** may claim. A routing verdict
says whether routing beat its baseline, over a **set** of them. Those two are the pair most often
conflated on a quality screen, and the
[two verdict vocabularies](/docs/concepts/honesty-model#two-verdict-vocabularies--dont-conflate-them)
note keeps them apart.
What is worth carrying away from the second one, while you are on this page, is that it declines in
the same shape as the first. It names the floor it fell short of. It prints the bound it computed
anyway. And it gives the outcome a name from a closed set, so `CANNOT_MEASURE` is a state rather
than an error and the verb exits 0. Where that verdict comes from is
[Eval programs and experiments](/docs/calibration/eval-programs); none of it moves a cap.
This is `tokentriage eval --quality shadow` over a sixteen-line ledger. Six labels in it belong to
the eval track rather than to this page. **BEAT** is the verdict that routing beat its baseline.
**CP-LCB** is a Clopper–Pearson lower confidence bound. **AIQ**, **PGR**, **VB** and **Ep** are
descriptive router-economics figures, and the block marks each of them as not the licensing basis.
```console
$ tokentriage eval --quality shadow --log ./decisions.jsonl
=== TokenTriage honesty eval ===
verdict: CANNOT_MEASURE the paired BEAT — 8 decided pair(s) below the floor of 30 (report.BeatMinDecidedPairs, raise-only; need 30, have 8) — too few shadow-vs-chosen pairs with both sides decided to license BEAT — computed and refused by the floor: CP-LCB 0.135 <= 0.50 (½−m, m=0; one-sided 0.95) — estimate 0.667, n₊=2 of 3 discordant; 8 decided pairs, 5 ties, 0 abstained; method pointwise-pair
quality source: shadow (labeled iff judged-shadow grade present (chosen_ok != nil); counterfactual per-cost verdict from both tiers' grades)
labeled rows: 8 of 8 (floor 8)
TIER POINTS (mean cost, mean quality over that tier's labeled observations)
TIER OBS MEAN-COST MEAN-QUAL SHARE
frontier 8 $0.000240 0.7500 50.0%
premium 8 $0.002100 0.8750 50.0%
ROUTER (all labeled) 8 $0.000240 0.7500 100.0%
PAIRED BEAT BOUND (B.1 licensing basis: shadow vs chosen, pointwise pairs — the only estimand that licenses BEAT)
decided pairs: 8 (floor 30, report.BeatMinDecidedPairs; 8 graded rows)
ties: 5 (support the claim only through this count)
discordant: 3 (floor 10, report.BeatMinDiscordant): n₊=2 shadow wins, n₋=1 chosen wins
abstained: 0 (excluded-and-counted; never a candidate failure — JH12)
estimate: 0.6667 = n₊/discordant
CP-LCB: 0.1354 (one-sided 0.95, clopper-pearson-v1) vs threshold 0.50 (½−m, m=0)
licensing: REFUSED — 8 decided pair(s) below the floor of 30 (report.BeatMinDecidedPairs, raise-only; need 30, have 8) — too few shadow-vs-chosen pairs with both sides decided to license BEAT — computed and refused by the floor: CP-LCB 0.135 <= 0.50 (½−m, m=0; one-sided 0.95) — estimate 0.667, n₊=2 of 3 discordant; 8 decided pairs, 5 ties, 0 abstained; method pointwise-pair
FRONTIER (descriptive router-vs-hull comparison at the router's own cost — ESTIMATED, never the BEAT license; see PAIRED BEAT BOUND)
router quality: 0.7500 @ cost $0.000240
zero-router @ router: 0.7500
delta: 0.0000
AIQ (descriptive shadow-economics, NOT the verdict basis; shared cost domain [$0.000240, $0.002100])
AIQ(router): 0.7500
AIQ(zero-router): 0.8125
delta: -0.0625
PGR (binary strong/weak: frontier → premium)
strong-weak gap: 0.1250
PGR(router): 0.0000
APGR/CPT: n/a — a single realized router has no call-rate sweep (use calibrate to sweep α)
RouterEval
VB (vs best single): 0.8571 (>1 means the router beats every single model)
Ep (selection bits): 1.0000
Ep normalized: 1.0000 (low means effectively a single-model policy)
```
Exit status is `0`. Both halves of that ledger need naming before you quote a number out of it.
The eight primary decision rows were minted and priced by the binary itself, routing to a loopback
upstream stub, so `frontier` and `$0.000240` are its own readings over that traffic.
The eight companion shadow grades were planted by the documentation harness in the shipped golden's
field shape, because no config key points the shadow-plane judge at a loopback host. So `8` decided
pairs, `5` ties and `n₊=2` are that plant, and every number under them moves if you change it. The
floors, the Clopper–Pearson bound, the verdict and every word of the refusal are the binary's. The
capture was run twice against the same ledger, and the two outputs are byte-identical.
Four things in that output are worth keeping.
- **The floor is named with need and have, twice.** `need 30, have 8` appears in the header verdict
and again under `licensing:`. A declined licence that hides its own arithmetic cannot be argued
with; this one can.
- **The bound prints even though it lost.** `CP-LCB: 0.1354` sits under an `estimate: 0.6667`,
because three discordant pairs cannot support a bound and the output says so instead of promoting
the estimate into a claim.
- **`abstained: 0` prints at zero.** No abstentions is a measurement, and it is written
`excluded-and-counted` rather than dropped from the block.
- **The descriptive numbers still render, and are labelled.** `FRONTIER`, `AIQ`, `PGR` and
`RouterEval` each carry a caption saying they are not the licensing basis, so `delta: 0.0000` is
not an answer to the question the verdict declined.
## What a promotion buys, and what it does not
A promotion is narrower than the word suggests, and knowing its shape is what stops you expecting
the wrong number to move.
**It is scoped to a subject, not to a judge.** The subject is the judge id, the judge model, the
prompt sha, the evaluator version, the metric and the segment. The promotion is filed under the
identity `{judge id, metric, segment}` and pinned to a version built from the other three, so
moving the judge model, the prompt or the evaluator version leaves the pin unmatched: the reading
comes back `version_changed`, and the subject's rows ride the `reported` ceiling again.
**It does not travel across a segment.** The same judge on a different tier is a different
subject, and an unvalidated one.
**It is forward-only, so it does not rewrite the rows you already have.** The licence compares a
row's own timestamp against the instant the promotion was recorded, and a row written before that
instant stays `reported` permanently. A window that ends before the promotion stays `reported`
however well the judge validated, which is the most common reason an operator promotes, re-reads
the same window, and concludes the promotion failed.
**It does not lift a cap that was not about the judge.** A `heuristic`-class metric and the `otel`
source both cap at `reported`, and neither yields to a promotion. Three of the four caps consult
the promotion registry; the source ceiling is the one that does not. Two of those three move on any
row this build can promote, which is what the worked example above shows.
**It does not make judging spend measurable.** `spend` is served as a present `null` on every
judges row, which the binary's own note reads as NOT MEASURED: judging costs money and is recorded
per row, and no shipped producer sums those dollars onto a subject.
## How this goes wrong
**Reading `reported` as a fault.** It is one of three values in a closed vocabulary, and it means
a proxy nobody has measured yet. A screen showing it is working.
**Reading the wrong `reported`.** Guardrail confidence carries its own five-value basis vocabulary
(`exact`, `fixture`, `calibrated`, `reported`, `insufficient`) and quality basis has three. Both
spell one value `reported`, and they are different claims about different things. On a judge page,
write *quality basis* and mean it.
**Reading the wrong `source`.** Two five-value vocabularies here are called sources, and they share
one member. An **outcome source** says which door an observation came through: `evaluator`,
`human`, `business`, `otel`, `feedback`. A **quality source** says which labelled signal an eval
reads for a row's quality: `oracle`, `outcome`, `shadow`, `feedback`, and the `eval:`
family. The routing verdict on this page prints `quality source: shadow`, which is not one of the
four caps.
**Promoting, then re-reading the window you were already looking at.** Forward-only means the
figure moves on rows written afterwards. Re-run the report over a window that starts after the
promotion instant before you conclude anything went wrong.
## What this page does not cover
**Which situation produced your `reported`.** The `basis_ceiling_reported` reading on a quality
summary counts how many of a segment's observations were capped, and states outright that it cannot
tell you why. The registry holds promotions only, so three different histories read the same there:
a judge nobody has validated, one that was measured and declined below the kappa floor of 0.5, and
one that was promoted and later demoted. The record that carries a named floor with its need and
have counts comes from `tokentriage evals validate` and `GET /tt/api/v1/evals/judges`, and
[Why didn't my judge validate?](/docs/judges/troubleshooting) starts from there.
**What every null on a judges row means.** `iaa: null` is not agreement of zero, and a κ of `0.0`
is a measurement at chance level rather than a missing value. The table that separates them is on
[the walk that promotes a judge](/docs/judges/agreement), which also says why κ travels without a
confidence interval.
**What happens to a promotion afterwards.** No shipped binary can fire a drift demotion; the
four-piece triad is demonstrated inside the test binary against a pinned vector.
[Judge drift, with no monitor between checks](/docs/judges/drift) has the rest.
**Turning a `measured` basis into a routing decision.** A quality floor reads a `measured` basis
only, so the ceiling is the precondition rather than the mechanism.
[Gate a rollout on a validated judge](/docs/judges/gate-a-rollout) is the mechanism.