Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Judge drift, with no monitor between checks

What happens when the judge changes under you, what this build measures about it, and which part of the demotion loop nothing can fire.

ExplanationLast updated

On this page

A judge you validated in March is still grading traffic in August. The provider has shipped two point releases of the model since then, somebody tightened the grading prompt in April, and the requests it reads today are not the requests it was measured on. The agreement figure on the screen is from March, and nothing on the screen says so.

That is drift. Three things sit underneath a promotion and any of them can move: the model doing the grading, the prompt it is sent, and the population of requests it grades. The first two move because a person changed them. The third moves because your product got popular in a market nobody measured a judge on.

This page says what happens to a promotion in each of those three cases. Two of them reset it without anybody asking, which is a mechanism you can lean on. The third is the subject of a monitor that is built, tested, and pinned to a hand-checkable vector, and that nothing in a shipped binary feeds.

The walk that promotes a judge comes first; this page assumes it. Drift on a calibration is a different quantity with its own detectors and its own cadence, and it is Keep calibrations fresh. There is no procedure here.

Two of the three moves reset the promotion by themselves

The cheapest protection you have against a judge changing under you is the one you get without configuring anything, so it is worth knowing precisely what it covers.

A promotion is filed under three coordinates and pinned to three others. judge_id, metric and segment are the key the registry files it under. model, prompt_sha and eval_ver compose the version it is pinned to. A judged row’s quality basis is re-derived at read time against that registry rather than trusted from the row, so the pin is consulted on every read.

A promotion licenses forward from its own instant, and only at its own versionReading left to right along one judge subject's life. In the first window no promotion stands for the subject, so every row written there carries a reported basis — and it keeps it permanently, because a promotion arriving later never re-licenses rows already on the ledger. The middle window opens at the promotion instant: a validation run promoted this subject at this version, so rows written from that instant onward may carry a measured basis, stamped by the registry with the agreement figure, the row count, the inter-annotator figure and the swap flip rate. The third window opens when the version moves — a different judge model, a different prompt sha, or a different self-reported evaluator version. The registry then answers version_changed: the earlier promotion is still stored and still readable, but it does not travel, and rows written after the move are reported again until a fresh validation run promotes the new version. Identity and version are separate keys. A promotion is stored under the judge id, the metric and the segment, and pinned to the model, the prompt sha and the evaluator version, so any one of those three moving is a reset; and promotion never travels across a segment, so the same judge on another tier is a different subject with no licence at all. Two residuals are stated rather than hidden: the licensed window is half-open forward, so a row dated after the promotion is licensed with nothing bounding the far end, and a row from an earlier epoch that was demoted and later promoted again falls before the standing instant and stays reported.one subject, over timethe promotion instanta version movebeforeno promotion stands forthis subject yetbasis: reportedthese rows keep it FOREVER —a promotion never re-licensesrows already on the ledgerlicenseda validation run promotedthis subject at this versionbasis: measuredthe registry stamps the row:validated, with κ, the row count,the iaa figure and the flip rateafter the movethe model, the prompt sha orthe evaluator version changedstate: version_changedthe earlier promotion is stillstored and still readable — itdoes not travel, and the clock resetsA promotion is stored under judge id · metric · segment, and pinned to model · prompt_sha · eval_ver. Any one ofthose three moving is a reset. Promotion never travels across a segment either: the same judge on another tier isa different subject, with no licence at all.Two residuals the build states rather than hides. The licensed window is half-open forward, so a row dated after thepromotion is licensed and nothing bounds the far end; and a row from an earlier epoch, demoted and later promotedagain, falls before the standing instant and stays reported.

Read it left to right along one subject’s life. The middle window is the licensed one, and the two instants that bound it are the parts that matter here: a promotion licenses forward from its own instant, and the licence stops travelling the moment the version moves. Rows on either side carry reported, permanently.

So change the judge’s model, edit its prompt, or bump the evaluator version, and the pin no longer matches. The reading comes back version_changed, the old record is still served so you can see what was lost, and rows written after the move ride the reported ceiling until a fresh validation run promotes the new version. A prompt edit costs the gold set over again, which is worth knowing before you make one.

What none of that catches is the third move. Your traffic can turn over completely without a single coordinate changing, and the promotion stands the whole time. That is the case the rest of this page is about.

The contract names two demotion arms, and only one has a producer

A demotion is a promotion withdrawn. This build publishes a field saying which measurement decided one, and the published description for it names two. cohen_kappa is where judge-versus-human agreement was re-measured and compared against the κ floor. drift_agreement_upper_bound is Appendix B.3’s post-promotion monitor, comparing a one-sided upper bound on agreement against a bar transferred from that same floor. The description calls those two the vocabulary this build produces.

One of them has a producer an operator can reach. The other does not, and this is the case where the product and its own published contract disagree.

The re-validation arm works, and you can drive it. Re-run tokentriage evals validate against a subject whose promotion still stands at the same version, and a record that falls under the κ floor demotes it. The registry deletes the standing state and revokes B.2’s accept region for that subject in the same write. A notice carrying the previous κ, the new κ, the floor and the refusals that decided it goes to the event sink, and the entry lands on the subject’s timeline as demoted. That is the arm that publishes cohen_kappa.

The monitor arm has no caller. Gate.ObserveAgreement has no call site outside its tests, so no shipped binary can fire a drift demotion. The refusal code that would make a demotion publish drift_agreement_upper_bound is minted at one site in the tree, and that site is inside the gate whose entry point nothing calls. So the published OpenAPI description is the source that is wrong here and the running binary is the authority, which is the rule everywhere in this corpus. The discrepancy is filed.

What is built, and what is missing

Reading “no producer” as “nothing was written” gets the shape wrong, and the shape is what tells you how far away the feature is. Almost the whole loop is here.

The arithmetic is built and pinned to a hand-checkable vector inside the test binary: an empirical Bernstein confidence sequence whose first breach on an alternating stream is asserted at t = 44, against five upper-bound values held to sixteen digits. The same vector is driven a second time through the shipped evaluation path, so it is pinned on the path a demotion would take. The demotion sink is composed at boot by a running proxy and handed to the gate. An end-to-end test drives the shipped gate into the shipped sink and reads the resulting payload back off the events API.

What is missing is the point where a human label would arrive. And the type that point would feed carries a request id and a boolean and nothing else, which puts three of the monitor’s own preconditions out of reach at that type:

The precondition Why it cannot be checked there
the outcomes are fresh, and post-promotion the observation carries no timestamp
no row comes from the gold revision that promoted the judge the observation carries no gold-revision id
no request is counted twice the request id de-duplicates within one batch, and cannot see across batches

The third one is the one most often written too large. De-duplication inside a batch is enforced, and the request id exists for that and nothing else: a batch naming one request twice, or naming none, is refused. It is across batches that nothing can check.

All three fail in the same direction, and the product says so in its own comment: they “make the sequence look longer and surer than it is”. The header measures the cost. A duplicated observation raises the false-demotion rate on a judge sitting at the floor from 0.003 at k=1 to between 0.096 and 0.120 at k=3, against a published δ of 0.05, and delays detection of a judge that did drift by roughly 100 requests.

One further gate sits on the input side, and it is worth knowing because it narrows the feature even where a caller lands. The accumulator a monitor advances is opened at promotion only where the promoting gold set had at most one annotator, because Cohen’s κ over label events has a chance term that is not the request-unit quantity this bound needs.

A judge promoted on the two-annotator gold set the agreement gate asks for therefore carries no stream, and the monitor declines by name on that cohort when it is asked. So the narrowing is not a corner: it lands on the cohort Collect the labels plans an overlap for, and a caller landing tomorrow would not reach it.

What a judges row says about any of this

Here is one element of the subjects array from GET /tt/api/v1/evals/judges, served by a booted instance over a state where a validation run had promoted the subject. Three values are elided with a marker; nothing else is edited.

{
  "subject": { "…": "the six coordinates, elided" },
  "state": "promoted",
  "kappa": 0.6761023407729995,
  "validated_n": 105,
  "iaa": 0.8397435897435899,
  "gold_revision": "93f8396",
  "floors_applied": ["…the six floor names, elided…"],
  "floors": { "…": "the six values, elided" },
  "spend": null,
  "selective_risk": null,
  "anchor": null,
  "last_run": "2026-08-24T18:08:58.333Z"
}

That capture is from a source build on Windows against a harness ledger whose judge verdicts came from a loopback stub and whose human labels the harness designed, so κ, iaa, validated_n, gold_revision and last_run are what a promoted run looks like rather than what any judge scores. The key set, state and the three nulls are not fixture values, and two boots of the same state serve this body byte for byte.

Three fields answer the drift question, and two of them answer it by being missing from that block:

  • last_run is persisted, not re-stamped on read. It is when the validation that promoted this subject ran, and it stays there because nothing re-reads the judge in between. A judges screen that looks current is showing you a figure from whenever you last ran the check.
  • anchor is null on every judges row this build serves. That is the who-drifted diagnosis: an anchor set that would be re-scored to say whether the judge moved or the traffic did. The statistics ship and are advisory, and nothing joins a diagnosis to a reading, so the null records that no diagnosis was attached rather than that the judge was checked and found stable.
  • next_trigger is absent from every judges row this build serves. Nothing assigns it, so its absence means the loop’s next run is unknown rather than ruled out. Read auto_validate for whether the loop is running at all, and last_run for when it last did.

One more absence lives on the neighbouring resource. demotions is omitted on every history response this build serves, because a demotion notice is handed to an event sink at the moment it happens and is not retained. A demotion still appears on timeline as a demoted entry with its κ and N, and the notice’s full detail travels on the quality.judge.demoted event and the recalibration log. So the dashboard’s Demotions heading has nothing to render, for two reasons at once, and only the outer one is about that field.

What to do instead

There is no drop-in replacement for a monitor. What there is, is a schedule you own, plus two habits that make the version pin cover more of the ground.

Re-run the validation check on a cadence you choose. It is the demotion arm that works. Pick the cadence from how fast your traffic mix moves rather than from a default, budget the labels, and read κ beside its validated_n each time. A run that falls under the floor demotes at the same version, and a run that clears it re-promotes and re-opens the window.

Move the version when you change the judge. This one is free, and it is a reason to prefer an explicit prompt edit over a silent provider upgrade. An edit you make moves prompt_sha and resets the promotion; a provider swapping a model behind a stable name moves nothing you can see. Pin the judge model to a dated identifier where the provider offers one.

Watch the population, not the judge. The move nothing catches is the traffic one, so the signal worth alerting on is your segment mix rather than any judge field. A segment that grew from 2% to 30% of traffic since the promotion is a subject that was measured somewhere else.

What this page does not cover

Which cap is holding a figure down. A judged row that reads reported after a version move is one case of a wider arithmetic, and the other caps have nothing to do with drift. Why your number is reported, not measured walks all four.

Calibration drift. A calibration’s bound going stale is a different quantity, with detectors and a refit cadence this build does ship. Keep calibrations fresh is that track, and the two should not be read as one feature.

The rest of the gaps. The monitor and the diagnosis are two entries on a longer list, grouped by what stands in the way of each. Air-gap judging, and the known limits is the catalogue.