Gate a rollout on a validated judge

A cheaper model would save you money on half your traffic, and you would like to send it there only while somebody can show the answers are still good enough. You have the grading in place and a promotion behind it. What is left is the step that turns that record into an admission rule the router applies to every request.

That step is a floor on a selector. A candidate tier becomes eligible only while its lower confidence bound clears the number you set, and a tier that cannot clear it stops being an option rather than becoming a cheap one. This page sets the floor, reloads, and reads the decision back.

Two pages come first and this one assumes both. Getting a subject through the validation walk is Validate a judge and promote it; what a promotion moves and what it leaves alone is Why your number is reported, not measured. The floor’s own grammar, its selector parameters and the score-table terms it writes are Gate routing on a proven quality floor, which covers the same mechanism for every metric rather than for a judged one.

What has to be true before the floor admits anything

Four conditions, and the first two are the judge half. Check them in this order, because each one makes the next readable.

Condition	How you check it	If it does not hold
A promotion stands for the subject	tokentriage evals validate prints reading: state=promoted	the walk declined; read the floor it named
The rows you want to gate on were written after the promotion instant	compare the promotion time against your window	re-read a later window
An active quality calibration covers the segment	tokentriage quality reliability	fit one, then activate it
The selector is a cost-aware one	cheapest_meeting, epsilon_greedy or weighted_objective	the floor is a parameter of those three

What you do not need: a running server for the check itself, a judge credential to read the record back, or any change to the tiers themselves. The floor is added to a select node and nothing else moves.

1. Confirm the promotion stands, and note when it landed

Run the validation verb and read one line off the record for your subject.

$ tokentriage evals validate --config ./tokentriage.yaml --state-dir ./state/dash

The line that answers this question is reading: state=promoted, printed under the subject’s block beside gold_revision and subjects_observed. A subject that has been through a run that declined reads never_validated instead, and the refusal lines above it name the floor with what it wanted beside what it had. Validate a judge and promote it reads a whole record of each kind.

Write down when the promotion landed. A promotion is forward-only: the quality basis on a row is derived at read time from the row’s own timestamp, so every judged row written before that instant keeps its reported basis permanently. A window that ends before the promotion licenses nothing, however well the judge validated.

2. Fit a calibration on the segment you want to gate

The floor reads candidate.quality.lcb, which comes off an active calibration artifact for that segment. Without one there is no bound to compare, and the selector’s refusal says so in those words when you try to set a metric on its own.

Fit and activate as you would for any metric, over a window that starts after the promotion instant from step 1. Calibrate quality from your logs is the procedure; the only judge-specific part is the window.

Offline proof and the online gate are two different jobs, and it is worth seeing where each one sits before you wire the second.

Read that left to right. A dataset of ledger references drives one instrument; the fold keeps arms of different bases apart instead of blending them; only priced points reach the frontier; and the answer is one of four states. Of the three instruments, only the counterfactual one can reach BEAT, so an offline run tells you where a tier sits and a shadow dual-run tells you it would have won. That is the evidence you bring to the floor. The floor itself is the next diagram, and it runs on every request rather than on a dataset.

3. Put the floor on the selector

min_quality_lcb and quality_metric live together in a select node’s by.params. This admits the cheapest tier whose bound clears 0.90 on task_completion, and sends everything else to the frontier when none qualifies.

routing:
  default_tier: frontier
  policy:
    version: 1
    root:
      rules:
        - id: task-completion-floor
          to:
            select:
              from: { tiers: all }
              by:
                selector: cheapest_meeting
                params:
                  min_quality_lcb: 0.90
                  quality_metric: task_completion
              on_empty: { tier: frontier }
      default: { tier: frontier }

task_completion is the metric judge-task-completion scores, so the floor and the judge are talking about the same property. The on_empty target is where traffic goes when no candidate clears the floor, and it is a routing decision the ledger records rather than a silent fallback.

4. Validate, then reload

Validate the whole file before anything reloads. The block above was loaded by the binary as part of the config this page was written against.

$ tokentriage validate --config tokentriage.yaml
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 1
  - openai                 https://api.openai.com
tiers:     2 (cheapest -> most capable)
  0. cheap      -> openai/gpt-5-mini
  1. frontier   -> openai/gpt-5
rules:     0
OK: configuration is valid

Exit status is 0. That capture is from a source build on Windows, run from the config’s own directory so no path appears; two egress warnings for the judge evaluator were printed above it on stdout, twice each, and are elided here. Two lines are worth pre-empting because they read like faults and are not. upstreams: 1 counts upstreams and not tiers, and rules: 0 counts the legacy routing.rules list rather than the policy tree — the shipped examples/evals/journey-3-hard-floor-airgapped.yaml, whose whole subject is a floor, prints the same 0.

The router reads calibration off an immutable snapshot published away from the request path, so the floor takes effect on the next reload rather than part-way through a request in flight.

Above the divider, off the request path, the active artifact is folded in the background and published as a snapshot. Below it, on the request path, each candidate tier does one map probe of that snapshot, and the gate admits a tier whose lower bound clears the floor and excludes one whose bound sits below it. The dashed connector crossing the divider is the only traffic between the two halves, which is why a reload rather than a request is what moves the floor.

5. Read a decision back and find the term that explains it

Send traffic, then read one decision. The selector’s score table carries every candidate’s quality verdict rather than only the admitted ones.

$ tokentriage policy explain --req-id <req-id> --log ./state/decisions.jsonl

Three outcomes are possible for a candidate, and they are three different facts.

What you see on the score table	What happened	What to change
the tier was chosen, no quality term	its bound cleared the floor	nothing
quality_below_min_lcb	the tier was measured, admitted to the comparison, and its bound fell short	lower the floor, or improve the tier
quality_absent:<reason>	the tier’s quality could not be read at all, and the reason rides the term name	the reason names it: fit a calibration, wait for evidence, or promote the judge

The distinction in the last two rows is the one worth holding on to. Absence is disclosure, not a penalty: a tier nobody could measure is excluded and disclosed rather than scored as having failed.

Three ways the floor admits nothing, and how to tell them apart

Each of these ends with the same observation — traffic on the on_empty target and a cheaper tier taking none of it. They have three different causes and three different fixes.

The judged evidence is capped, so the segment has no measured figure. Ask the quality surface for a measured figure on that segment and it answers basis_ceiling_reported, with need and have counts and a hint naming the remedy. A judge signal is promoted only by the validation protocol, per judge tuple per metric per segment, and no promotion stands for these rows’ subject. The same hint says the surface cannot tell you which of three reasons applies. The registry holds promotions only, so an unvalidated judge, a declined one and a demoted one all read alike there. tokentriage evals validate serves the record with its named floor and its need and have.

The promotion is for a different segment. A promotion is filed under one judge id, metric and segment, and pinned to the judge model, the prompt sha and the evaluator version. The same judge on another tier is another subject and an unvalidated one, so a floor on the second segment admits nothing while the first is fine.

The metric id is misspelled. This one clears every check you would think to run.

$ tokentriage validate --config typo.yaml
…
rules:     0
OK: configuration is valid

Exit status 0, on a file whose quality_metric reads task_completeion. The id is not checked against any registry at validate time, so the mistake surfaces at runtime as a metric that joins no segment: every candidate’s quality fact is absent under a closed token, every candidate is excluded and disclosed, and traffic lands on on_empty. Check the id against the evaluator’s declared metrics by hand, and read the score table after the first request rather than trusting a green validate.

Keep the floor current after it is live

A promotion is a measurement of a judge against labels collected at one moment, and three things under it can move afterwards: the grading model, the prompt it is sent, and the mix of requests it grades. The first two reset the promotion by themselves, because the pin stops matching and the subject reads version_changed. The third moves nothing you can see.

So re-run the validation check on a cadence you pick, and treat a prompt edit as a re-validation. A run that falls under the floor demotes at the same version; a run that clears it re-promotes and re-opens the window. Judge drift, with no monitor between checks has the full account of which part of that loop you own.

What this page does not cover

The full selector grammar. The three cost-aware selectors, where a select node sits in a policy tree, and why naming candidate.quality.mean is a compile error are the policy language and Gate routing on a proven quality floor.

Fitting and activating a calibration. The B.5 gate, the holdout split and the reliability report belong to the calibration track. How calibration works is the engine and Calibrate quality from your logs is the procedure.

Why a figure reads reported when a promotion did land. A quality basis is the lowest of several caps, and only the judge-tied ones yield to a promotion. Why your number is reported, not measured finds the one that binds.

Every other symptom. A floor that admits nothing is one observation among several. Why didn’t my judge validate? indexes the rest by what you can see.