Collect the labels a judge is measured against

Cohen’s κ compares a judge’s verdicts against people’s, so somebody has to be the people. This page is the work of getting their answers onto your rows in a shape the validation run will admit.

Two things about that shape cost more than they look. Agreement between the people is one of the numbers the run measures, so wherever you want that figure the verdicts have to come from two annotators; one person agrees with themselves at a rate nothing can compute. And the two have to overlap: a set of requests they both looked at, independently, so there is something to compare them on. Twenty shared requests is a floor rather than a target. A gold set that lands on nineteen has its agreement recorded as unmeasured, and a deployment that raised the bar to require agreement gets a run that declines and hands the whole labelling effort back.

One annotator is admitted too. evals.validation.require_iaa defaults to false (internal/config/evals.go:693-698), so a single-annotator gold set licenses a promotion with its agreement figure recorded as unmeasured. That cohort is not the lesser option in every direction: a promotion from one annotator is the only kind that opens a drift stream, because the accumulator opens only under len(Annotators) <= 1 (internal/evaluator/validate.go:1578). Two annotators buy an agreement figure and cost the post-promotion monitoring; one buys the monitoring and leaves iaa unmeasured. Choose before you recruit: who labels decides it, not a leaf you change later.

So the ordering below is not decoration. You turn the queue on, give two people an identity the server derives rather than accepts, claim work, submit a verdict against a row, and then read the assembly line that says whether any of it counted.

Every block on this page came off a run of the shipped binary against a ledger it minted itself, and the conditions are named under each one. If you have no judged rows yet, Configure a judge is the config surface and How a judge call happens is what it does with them. What these labels are then measured into, and why κ is the statistic, is the walk that promotes a judge.

Turn the annotation queue on

The queue is off in a config that does not mention it, and both body-capture leaves have to be on before it can serve a single row. Set all three at once, because each missing one produces its own refusal and you would otherwise meet them in series.

dashboard:
  enabled: true
  state_dir: ./state/dash
  auth:
    mode: users
  capture:
    request: true
    response: true
evals:
  outcomes:
    api_enabled: true
  annotation_queue:
    enabled: true

That block goes beside the evals.evaluators[] entry carrying your judge: block, and the whole file was loaded with tokentriage validate before it was published here. capture is the pair that stores the prompt and the completion; an annotator reads those bytes, so a deployment keeping hashes only has nothing to offer. dashboard.enabled is what mounts both the queue and the door a label comes back through, which is why it is in the block at all.

outcomes.api_enabled needs one sentence of its own, because it does less than it looks like it does. The config loader reads it and requires dashboard.enabled beside it; nothing in this build reads it at request time. Measured on a live instance: POST /tt/api/v1/outcomes accepted a label with the leaf absent from the config entirely. Set it anyway, because it is what a reader of your config will look for, and do not reach for it as an off switch.

Leave the queue leaf out and a claim comes back like this:

{
  "type": "/tt/api/problems/feature_absent",
  "title": "Feature not wired in this deployment",
  "status": 404,
  "detail": "the annotation queue is not enabled on this deployment (evals.annotation_queue.enabled is off, which is the default)",
  "code": "feature_absent",
  "request_id": "req_157c1a18b96f32e5",
  "enable_hint": "The annotation queue is OFF by default. Set `evals.annotation_queue.enabled: true` and restart. The queue serves EXISTING ledger rows for human labelling and opens no second write path: a submitted label still travels through POST /outcomes with its actor server-derived. ★ BEFORE YOU HAND THIS TO ANNOTATORS: a claim serves the row's stored prompt and completion verbatim, so it requires the privileged `content:read` scope, which only an ADMIN-role credential can hold — an operator-role token cannot even be minted with it, and an operator dashboard session does not carry it. Turning this leaf on does not by itself make the queue reachable for a non-admin annotator. Body capture must also be on (see the capture refusal this route reports separately)."
}

Those are the response bytes with whitespace added and no value changed. request_id is the one that run minted and yours will differ; everything else is a constant in the binary. The refusal for capture is a different one, and it carries the trap worth knowing before you plan a labelling session: capture is prospective. The binary’s own words are “only exchanges recorded after the restart have bodies, so rows already on the ledger stay unclaimable”. Turning capture on today does not make yesterday’s traffic labellable.

One boot line says the rest of the machinery came up:

{"time":"2026-08-24T12:14:40.7698743-07:00","level":"INFO","msg":"evals: B.7 promotion registry opened","dir":"state\\dash\\quality","validation_state":"empty","detail":"","standing_promotions":0,"auto_validate":false,"interval":"24h0m0s","new_labels_threshold":25}

That line is the precondition for one of the two claim strategies below. It came from a source build on Windows, so the dir is backslashed and a Linux or macOS host prints a /-separated path; time is that machine’s, with its UTC−7 offset. validation_state: empty is a claim about the registry rather than a failure, and the three policy values are the shipped defaults.

Give each annotator an identity the server derives

An annotator’s name is not a field on the submission. The server takes it from the authenticated principal and stamps it on the row, which is what makes two annotators countable as two rather than as whoever the client said it was. That needs named users:

$ tokentriage user add annotator-a --role admin --state-dir ./state/dash
Password:
added user "annotator-a" with role admin

Password: is a prompt on stderr and the confirmation is on stdout, so a terminal interleaves them in that order. Run it once per annotator, before the server boots.

That is a real constraint on who you can hand this to, and it is worth settling before you recruit two people. The write half is different: POST /tt/api/v1/outcomes sits at the operator floor, so a person who can submit labels is not automatically a person who can be offered them.

Claim work, and read why the row was offered

A claim offers work. It picks rows a human should look at, leases them so two annotators are not sent the same request by accident, and hands back the text to read. It writes nothing to the ledger.

The dashboard serves this at Evals · annotate, where claiming is a button rather than something the page does when it loads. Over the API it is one call, POST /tt/api/v1/evals/annotation-queue/claim, carrying the annotator’s dashboard session:

{ "strategy": "starved_segments", "k": 60 }

Two strategies exist and they answer different questions.

starved_segments ranks coordinates by how few human labels their gold set already holds, thinnest first. It needs the label counter the judge runtime supplies, which the standard binary wires wherever it opens the B.7 promotion registry, so the boot line above is its precondition. Here is one item off a claim on a ledger with judged rows and no labels yet:

{
  "items": [
    {
      "req_id": "tt_56a7tyvh3kbmb3npdh4jcecsqe",
      "req_ts": "2026-08-24T19:14:41.716Z",
      "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat",
      "instruction": "{\"model\": \"gpt-5\", \"messages\": [{\"role\": \"user\", \"content\": \"w23 item 0000 W23-VERDICT-PASS\"}]}",
      "output": "[… 340 bytes of stored response body, elided …]",
      "priority_reason": "starved_segments: this segment's gold set holds 0 of the 50 human labels validation needs (50 short); the gold set is projected and empty, which is not the same fact as no gold set"
    }
  ],
  "lease_expires": "2026-08-24T20:16:42.395Z"
}

That claim asked for sixty and was granted sixty; one item is shown and the single elision is marked. instruction is the stored request body verbatim, and W23-VERDICT-PASS inside it is a marker the capture harness planted so a loopback listener could stand in for the judge model. Yours carries your own traffic. req_id, req_ts and lease_expires move on every run; segment is this fixture’s routing; priority_reason is the server’s own sentence, and the 50 in it is JudgeValidationMinN, supplied by the counter rather than spelled a second time by the queue.

hard_cases ranks rows the judge found hard, and hardness has two ingredients: a judge that abstained, and a panel that split. A row carrying neither is not a hard case and is not offered. The consequence is worth stating flatly, because it looks like breakage: a plain pointwise-bool judge that answered every row produces no hard cases at all, and the claim comes back {"items": [], "lease_expires": ""} at HTTP 200. That is an answer about your traffic, not a fault. Reach for hard_cases where you run a panel or a method that abstains, and for starved_segments everywhere else.

A malformed claim is refused by name. A strategy outside the pair gets 400 bad_request reading “strategy is outside the closed §2.5 vocabulary (starved_segments, hard_cases)”, and a k below 1 gets “k must be at least 1, a claim for nothing is not a claim”. A k above 200 is trimmed to the cap and answered at HTTP 200 rather than turned away.

Submit the verdict

The annotation queue ships one endpoint, POST /tt/api/v1/evals/annotation-queue/claim; labels come back through POST /tt/api/v1/outcomes, correlated by req_id. That shared outcomes door is where a human verdict has always gone, and the queue stratifies which rows get looked at without opening a second way to write one.

One submission per batch of rows, each carrying the req_id the claim offered:

{
  "outcomes": [
    { "req_id": "tt_56a7tyvh3kbmb3npdh4jcecsqe", "metric": "task_completion",
      "evaluator": "annotation-queue", "eval_ver": "queue-v1",
      "source": "human", "basis": "measured", "value": 1.0, "pass": true }
  ]
}

evaluator and metric have to be a pair the registry declares, or that item is refused with evaluator_metric_mismatch while the rest of the batch lands. The response is an envelope with one result per item; this is the envelope from a batch of thirty, with its first result and the other twenty-nine elided:

{
  "accepted": 30,
  "refused": 0,
  "coerced": 0,
  "results": [
    {
      "index": 0,
      "accepted": true,
      "basis": "measured",
      "actor": "annotator-a",
      "disclosures": [
        {
          "code": "actor_derived",
          "field": "/outcomes/0/source",
          "detail": "the annotator identity was derived from the authenticated principal; it is not an input, and a client-supplied actor is never trusted"
        },
        {
          "code": "eval_cost_derived",
          "field": "/outcomes/0/eval_cost",
          "detail": "no eval_cost was supplied; resolved to a MEASURED $0 from the outcome source — an annotation or a downstream business signal costs this deployment nothing, and free is a measurement rather than an absence"
        }
      ]
    }
  ]
}

actor: "annotator-a" is why the users block is in the config: the server put that name there, and a client-supplied one would not have survived the trip. basis: "measured" is what this door stored for the label; the judge-graded row beside it is capped elsewhere, and Why your number is reported, not measured is which cap.

Submitting does not release the lease either. Measured: annotator-a submitted thirty labels and re-claimed immediately, and the queue answered {"items": [], "lease_expires": ""}, because the rows were still leased to that same annotator. Only expiry releases a lease, and the default lease runs an hour.

The overlap is the part the queue will not hand you

Here is where the two mechanics collide, and it is the one thing on this page that will cost you a day if you meet it after the labelling rather than before.

The lease exists to stop two annotators labelling one request by accident. Inter-annotator agreement is computed on precisely the requests two annotators both labelled. So the queue’s default behaviour is the opposite of what the measurement needs, and it is not subtle. Measured on a live instance: annotator-a claimed all sixty available rows, and annotator-b claiming the same queue one second later was served this.

{ "items": [], "lease_expires": "" }

Three ways out, and only one of them is a plan.

• Hand the overlap set to the second annotator directly. The submission door takes a req_id and does not consult the lease table, so annotator-b can label a request annotator-b was not offered. This is what the capture behind this page did: annotator-a worked the queue, and the first twenty of those request ids went to annotator-b out of band.
• Claim in turns, small. With a one-hour lease and a claim of k: 25, one annotator can work a batch and the other can pick up the same rows after the lease lapses. It is slow and it is arithmetic you have to do yourself.
• Restart the server. The lease table is in memory, so a restart releases every lease early. Measured, and it is benign in the only direction it can be wrong: a row is offered again sooner than promised rather than withheld longer. It is a recovery, not a workflow.

Now the sizes, and one piece of vocabulary that decides them. A label event is one (annotator, request) pair. Two annotators on one request are two label events and one request, and the two floors count different things:

Floor	What it counts	Value
JudgeValidationMinN	label events on the segment, after a last-write-wins dedupe	50
IAAOverlapMinN	requests a pair of annotators both labelled	20
GoldSetIAAFloor	κ_h over those shared requests	0.6

So a sixty-request run where one annotator labels all sixty and the other labels twenty is eighty label events and twenty shared requests, and it clears both counts with nothing spare on the second one. Plan the overlap first and let the bulk follow it, rather than the other way round.

How to tell it worked

Run the validation verb and read its first block. The assembly header prints on every run, before any statistic, and it is the count of what your labels turned into:

$ tokentriage evals validate --config … --state-dir … --log …
gold-set assembly — D:/tt-jg-scratch/w23work/run/state/decisions.jsonl
  sources: outcomes   sets: 1   segments: 1   torn lines: 0   unorderable req_ts: 0
  task_completion | task_completion|tier=frontier|model=gpt-5|endpoint=chat
      offered=80 admitted=80 (outcomes=80 feedback=0) requests=60 subjects=1 annotators=[annotator-a annotator-b]
      excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0
  skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0
  feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0

The path is that capture’s --log and yours will be your own. Every counter on those lines is a measurement and is stable for a given ledger. The four to read:

• annotators=[annotator-a annotator-b] is the server-derived list, and it is the check that your two identities landed as two. One name here means one annotator whatever your rota said.
• admitted=80 beside requests=60. Eighty label events over sixty requests: one annotator labelled all sixty and the other labelled twenty of them. JudgeValidationMinN is measured against the eighty.
• excluded: counts what the admission gate turned away by provenance, and total=0 here because that ledger carries no demo, synthetic or superseded rows. A non-zero number is labelling effort that went nowhere.
• skipped: counts rows that never became an offer because a coordinate was missing. All zero on a clean run.

Two lines further down, the run says what the overlap bought:

      gold: admitted=80 annotators=2 licenses=true iaa=0.8000 revision=4587539
      kappa: 0.6500 over n=80 (p_o=0.8250 p_e=0.5000)
      floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1
      verdict: PROMOTED   registry change: promoted

iaa=0.8000 is κ_h over the twenty shared requests, and clearing GoldSetIAAFloor is what admitted the gold set. Read the conditions before quoting either number. The assembler, Cohen’s κ, the inter-annotator agreement, every floor and the promotion registry are the shipped binary’s and ran end to end against a ledger the proxy minted. The judge verdicts came from a loopback listener answering a canned score, and the two annotators’ labels were designed by the harness that drove the run. So 0.6500 is the shape of a promoted run rather than a measurement of any judge, and revision is a digest over minted request ids and moves every time.

How this goes wrong

The overlap lands one row short. This is the failure this page exists for, and it is a cliff rather than a slope. On a deployment that set evals.validation.require_iaa: true, a gold set with nineteen shared requests instead of twenty gets this:

      gold: admitted=79 annotators=2 licenses=false iaa=unmeasured (overlap_below_floor) revision=5dac3b3
      kappa: NOT COMPUTED — a gate before it refused (the refusals below name which); validated_n=0
      floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1
      verdict: REFUSED   registry change: refused
      refusal iaa_unmeasured: floor=IAAOverlapMinN need=20 have=19 overlapping rows
        iaa_unmeasured: inter-annotator κ_h is UNMEASURED (overlap_below_floor) — 2
        annotator(s), widest overlap 19 rows, IAAOverlapMinN (20) required.
        `evals.validation.require_iaa: true` upgrades B.7's admitted-but-disclosed default (iaa:
        null) to a REFUSAL, so this gold set cannot license judge promotion. Assign a ≥20-row
        overlap sample to a second annotator, or clear the knob and accept the disclosure

Seventy-nine admitted label events, two annotators, and the run declines because one request left the overlap. need=20 have=19 is the arithmetic, and the last sentence is the binary telling you which of two things to change. Exit status was 0, because a decline is an answer.

Leave require_iaa at its default and the same gold set promotes instead, with iaa=unmeasured (overlap_below_floor) on the gold line and nothing else to warn you. That reason string reaches the text output of this verb and no other surface, so a null iaa read over the API alone does not say which of four situations produced it.

A second label from the same annotator replaces the first. The dedupe is last-write-wins per (annotator, request), so a correction lands cleanly and a mistake lands as cleanly. In the run above, annotator-a submitted a second outcome on a request already inside the overlap, and that outcome carried no pass verdict: the later write won, the row was counted under no_pass_verdict=1, and the request dropped out of the overlap entirely. One stray submission moved have from 20 to 19.

A label on a request the ledger does not carry is accepted. The outcomes door does not check that req_id names a row it has, so a typo returns accepted: 1 and writes a companion row with nothing to join to. It then appears in no counter on the assembly block at all, not even a skip. Check admitted= against the number you submitted.

A row the queue offered can still be excluded at admission. The queue drops demo and synthetic rows before it ranks, so those two rarely reach an annotator. The other two exclusion families are properties of the label rather than of the row, and no claim can see them coming: an outcome with no attributed annotator, and one a later label superseded. The excluded: line counts all four separately, and it is worth reading before you conclude the labels were lost.

The bodies aged out. Capture is swept at dashboard.capture.ttl. A row whose prompt and completion are gone is skipped by the claim rather than offered blank, so an old segment can quietly stop producing work while the counter still says it needs labels.

What this page does not cover

One of the two registered judge ids cannot be labelled toward a validation at all. judge-groundedness publishes no pass floor, so the runner leaves every row it grades without a verdict and the assembler records a subject’s verdict only where one is set. Labels collected against it accumulate no gold set. Use judge-task-completion, whose metric carries a floor.

Running the check and reading a decline. The verb, its flags, and the floor a stopped run names are Validate a judge and promote it.

What the numbers mean once they exist. κ, κ_h, the four gates and what a promotion changes are on the walk that promotes a judge, which also has the table separating a null iaa from an agreement of zero.

Every field on the claim and the submission. The request and response shapes, and what each field carries when it comes back null, are Judge API.