Keep calibrations fresh: drift & refit

A calibration is a claim about a window of traffic. When the traffic shifts, because a provider silently updated a model behind a stable name, that claim can stop holding. This page wires drift detection and a refit cadence so the shift is detected, named, and recalibrated, with no step where a stale calibration keeps serving a bound it can no longer support.

This is drift on a fitted calibration, and it is not the same quantity as drift on a judge. The detectors below, the five refit triggers and the read-time invalidation all belong to a qcal_ artifact. A judge that changes under you moves a different record with a different cadence and different levers, and that story is Judge drift, with no monitor between checks. Reading the two as one feature is the mistake worth spending a sentence to head off.

Prerequisites

• An active quality calibration (see Calibrate quality from your logs).
• evals.calibration.quality.enabled: true.

1. Configure detectors and a refit cadence

Register drift detectors and set the scheduled refit interval:

evals:
  calibration:
    quality:
      enabled: true
      confidence: 0.90
      min_n: 50
      refit_interval: 12h            # the scheduled cadence (omit / 0s ⇒ no periodic refit)
  drift:
    detectors: [robust-seasonal-v1, psi-window-v1]   # registered ids; an unknown id is refused

2. Know the five refit triggers

The refit scheduler runs off the request path and fires each trigger as an edge — once per (artifact, epoch), as a named recalibration-log event. Only one of the five needs configuration:

Trigger	Fires when
drift_finding	a detector lands a finding on the segment
policy_epoch_change	the routing config’s fingerprint changes
staleness	the fit crosses the 30-day horizon
operator_request	an operator asks for a refit
scheduled_interval	the refit_interval cadence ticks

3. Read the drift timeline

A drift finding is only representable with its method and window, since “quality dropped” is not a finding. Read the timeline over the API or the studio’s drift surface:

$ curl -s localhost:9090/tt/api/v1/quality/drift | jq

A finding carries the detector method, the window it compared, and how to reproduce it — for example psi-window-v1 measuring a population-stability shift over a 7d-vs-24h window on one segment.

4. Understand read-time invalidation

When a finding (or staleness, or an epoch change) invalidates a segment, the affected entry flips to an insufficient refusal (drift_invalidated / calibration_stale) at read time, and it is never written back into the content-addressed artifact. The bytes of a qcal_ artifact are immutable; a successor is always a new id rather than a rewrite.

5. Activate the refit from shadow

An automatic refit does not promote itself into the serving slot. The default mode is PromoteShadow: the scheduler lands the fresh fit in shadow (fitted and evaluated, but serving nothing), and an operator activates it. That activation is audited:

$ tokentriage quality reliability --config tokentriage.yaml     # inspect the shadow fit
$ tokentriage quality activate <new-qcal-id> --config tokentriage.yaml

Each artifact in the chain links its lineage to its parent, so there is no silent decay — only a versioned, readable succession.