# Judge drift, with no monitor between checks What happens when the judge changes under you, what this build measures about it, and which part of the demotion loop nothing can fire. Source: https://tokentriage.com/docs/judges/drift --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import JudgePromotionTimelineDiagram from '../../../components/JudgePromotionTimelineDiagram.astro'; A judge you validated in March is still grading traffic in August. The provider has shipped two point releases of the model since then, somebody tightened the grading prompt in April, and the requests it reads today are not the requests it was measured on. The agreement figure on the screen is from March, and nothing on the screen says so. That is drift. Three things sit underneath a promotion and any of them can move: the model doing the grading, the prompt it is sent, and the population of requests it grades. The first two move because a person changed them. The third moves because your product got popular in a market nobody measured a judge on. This page says what happens to a promotion in each of those three cases. Two of them reset it without anybody asking, which is a mechanism you can lean on. The third is the subject of a monitor that is built, tested, and pinned to a hand-checkable vector, and that nothing in a shipped binary feeds. [The walk that promotes a judge](/docs/judges/agreement) comes first; this page assumes it. Drift on a *calibration* is a different quantity with its own detectors and its own cadence, and it is [Keep calibrations fresh](/docs/calibration/drift-and-refit). There is no procedure here. ## Two of the three moves reset the promotion by themselves The cheapest protection you have against a judge changing under you is the one you get without configuring anything, so it is worth knowing precisely what it covers. A promotion is filed under three coordinates and pinned to three others. `judge_id`, `metric` and `segment` are the key the registry files it under. `model`, `prompt_sha` and `eval_ver` compose the version it is pinned to. A judged row's quality basis is re-derived at read time against that registry rather than trusted from the row, so the pin is consulted on every read. Read it left to right along one subject's life. The middle window is the licensed one, and the two instants that bound it are the parts that matter here: a promotion licenses forward from its own instant, and the licence stops travelling the moment the version moves. Rows on either side carry `reported`, permanently. So change the judge's model, edit its prompt, or bump the evaluator version, and the pin no longer matches. The reading comes back `version_changed`, the old record is still served so you can see what was lost, and rows written after the move ride the `reported` ceiling until a fresh validation run promotes the new version. A prompt edit costs the gold set over again, which is worth knowing before you make one. What none of that catches is the third move. Your traffic can turn over completely without a single coordinate changing, and the promotion stands the whole time. That is the case the rest of this page is about. ## The contract names two demotion arms, and only one has a producer A demotion is a promotion withdrawn. This build publishes a field saying which measurement decided one, and the published description for it names two. `cohen_kappa` is where judge-versus-human agreement was re-measured and compared against the κ floor. `drift_agreement_upper_bound` is Appendix B.3's post-promotion monitor, comparing a one-sided upper bound on agreement against a bar transferred from that same floor. The description calls those two the vocabulary this build produces. One of them has a producer an operator can reach. The other does not, and this is the case where the product and its own published contract disagree. **The re-validation arm works, and you can drive it.** Re-run `tokentriage evals validate` against a subject whose promotion still stands at the same version, and a record that falls under the κ floor demotes it. The registry deletes the standing state and revokes B.2's accept region for that subject in the same write. A notice carrying the previous κ, the new κ, the floor and the refusals that decided it goes to the event sink, and the entry lands on the subject's timeline as `demoted`. That is the arm that publishes `cohen_kappa`. **The monitor arm has no caller.** `Gate.ObserveAgreement` has no call site outside its tests, so no shipped binary can fire a drift demotion. The refusal code that would make a demotion publish `drift_agreement_upper_bound` is minted at one site in the tree, and that site is inside the gate whose entry point nothing calls. So the published OpenAPI description is the source that is wrong here and the running binary is the authority, which is the rule everywhere in this corpus. The discrepancy is filed. Automatic drift demotion is the design intent, and the design intent is what the contract describes. No shipped binary can fire one. A promoted judge in this build is not monitored between validation runs, and the substitute is to run the check yourself on a schedule of your own. ## What is built, and what is missing Reading "no producer" as "nothing was written" gets the shape wrong, and the shape is what tells you how far away the feature is. Almost the whole loop is here. The arithmetic is built and pinned to a hand-checkable vector inside the test binary: an empirical Bernstein confidence sequence whose first breach on an alternating stream is asserted at t = 44, against five upper-bound values held to sixteen digits. The same vector is driven a second time through the shipped evaluation path, so it is pinned on the path a demotion would take. The demotion sink is composed at boot by a running proxy and handed to the gate. An end-to-end test drives the shipped gate into the shipped sink and reads the resulting payload back off the events API. What is missing is the point where a human label would arrive. And the type that point would feed carries a request id and a boolean and nothing else, which puts three of the monitor's own preconditions out of reach at that type: | The precondition | Why it cannot be checked there | |---|---| | the outcomes are fresh, and post-promotion | the observation carries no timestamp | | no row comes from the gold revision that promoted the judge | the observation carries no gold-revision id | | no request is counted twice | the request id de-duplicates *within* one batch, and cannot see across batches | The third one is the one most often written too large. De-duplication inside a batch is enforced, and the request id exists for that and nothing else: a batch naming one request twice, or naming none, is refused. It is across batches that nothing can check. All three fail in the same direction, and the product says so in its own comment: they *"make the sequence look longer and surer than it is"*. The header measures the cost. A duplicated observation raises the false-demotion rate on a judge sitting at the floor from 0.003 at k=1 to between 0.096 and 0.120 at k=3, against a published δ of 0.05, and delays detection of a judge that did drift by roughly 100 requests. One further gate sits on the input side, and it is worth knowing because it narrows the feature even where a caller lands. The accumulator a monitor advances is opened at promotion only where the promoting gold set had at most one annotator, because Cohen's κ over label events has a chance term that is not the request-unit quantity this bound needs. A judge promoted on the two-annotator gold set the agreement gate asks for therefore carries no stream, and the monitor declines by name on that cohort when it is asked. So the narrowing is not a corner: it lands on the cohort [Collect the labels](/docs/judges/collect-labels) plans an overlap for, and a caller landing tomorrow would not reach it. ## What a judges row says about any of this Here is one element of the `subjects` array from `GET /tt/api/v1/evals/judges`, served by a booted instance over a state where a validation run had promoted the subject. Three values are elided with a marker; nothing else is edited. ```json { "subject": { "…": "the six coordinates, elided" }, "state": "promoted", "kappa": 0.6761023407729995, "validated_n": 105, "iaa": 0.8397435897435899, "gold_revision": "93f8396", "floors_applied": ["…the six floor names, elided…"], "floors": { "…": "the six values, elided" }, "spend": null, "selective_risk": null, "anchor": null, "last_run": "2026-08-24T18:08:58.333Z" } ``` That capture is from a source build on Windows against a harness ledger whose judge verdicts came from a loopback stub and whose human labels the harness designed, so κ, `iaa`, `validated_n`, `gold_revision` and `last_run` are what a promoted run looks like rather than what any judge scores. The key set, `state` and the three nulls are not fixture values, and two boots of the same state serve this body byte for byte. Three fields answer the drift question, and two of them answer it by being missing from that block: - **`last_run` is persisted, not re-stamped on read.** It is when the validation that promoted this subject ran, and it stays there because nothing re-reads the judge in between. A judges screen that looks current is showing you a figure from whenever you last ran the check. - **`anchor` is null on every judges row this build serves.** That is the who-drifted diagnosis: an anchor set that would be re-scored to say whether the judge moved or the traffic did. The statistics ship and are advisory, and nothing joins a diagnosis to a reading, so the null records that no diagnosis was attached rather than that the judge was checked and found stable. - **`next_trigger` is absent from every judges row this build serves.** Nothing assigns it, so its absence means the loop's next run is unknown rather than ruled out. Read `auto_validate` for whether the loop is running at all, and `last_run` for when it last did. One more absence lives on the neighbouring resource. `demotions` is omitted on every history response this build serves, because a demotion notice is handed to an event sink at the moment it happens and is not retained. A demotion still appears on `timeline` as a `demoted` entry with its κ and N, and the notice's full detail travels on the `quality.judge.demoted` event and the recalibration log. So the dashboard's Demotions heading has nothing to render, for two reasons at once, and only the outer one is about that field. ## What to do instead There is no drop-in replacement for a monitor. What there is, is a schedule you own, plus two habits that make the version pin cover more of the ground. **Re-run the validation check on a cadence you choose.** It is the demotion arm that works. Pick the cadence from how fast your traffic mix moves rather than from a default, budget the labels, and read κ beside its `validated_n` each time. A run that falls under the floor demotes at the same version, and a run that clears it re-promotes and re-opens the window. **Move the version when you change the judge.** This one is free, and it is a reason to prefer an explicit prompt edit over a silent provider upgrade. An edit you make moves `prompt_sha` and resets the promotion; a provider swapping a model behind a stable name moves nothing you can see. Pin the judge model to a dated identifier where the provider offers one. **Watch the population, not the judge.** The move nothing catches is the traffic one, so the signal worth alerting on is your segment mix rather than any judge field. A segment that grew from 2% to 30% of traffic since the promotion is a subject that was measured somewhere else. ## What this page does not cover **Which cap is holding a figure down.** A judged row that reads `reported` after a version move is one case of a wider arithmetic, and the other caps have nothing to do with drift. [Why your number is reported, not measured](/docs/judges/basis-ceilings) walks all four. **Calibration drift.** A calibration's bound going stale is a different quantity, with detectors and a refit cadence this build does ship. [Keep calibrations fresh](/docs/calibration/drift-and-refit) is that track, and the two should not be read as one feature. **The rest of the gaps.** The monitor and the diagnosis are two entries on a longer list, grouped by what stands in the way of each. [Air-gap judging, and the known limits](/docs/judges/air-gap-and-limits) is the catalogue.