# Collect the labels a judge is measured against Get human labels onto ledger rows through the annotation queue the dashboard and the API both serve, and plan the overlap agreement needs. Source: https://tokentriage.com/docs/judges/collect-labels --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; Cohen's κ compares a judge's verdicts against people's, so somebody has to be the people. This page is the work of getting their answers onto your rows in a shape the validation run will admit. Two things about that shape cost more than they look. Agreement between the people is one of the numbers the run measures, so wherever you want that figure the verdicts have to come from **two** annotators; one person agrees with themselves at a rate nothing can compute. And the two have to overlap: a set of requests they both looked at, independently, so there is something to compare them on. Twenty shared requests is a floor rather than a target. A gold set that lands on nineteen has its agreement recorded as unmeasured, and a deployment that raised the bar to require agreement gets a run that declines and hands the whole labelling effort back. One annotator is admitted too. `evals.validation.require_iaa` defaults to `false` (`internal/config/evals.go:693-698`), so a single-annotator gold set licenses a promotion with its agreement figure recorded as unmeasured. That cohort is not the lesser option in every direction: a promotion from one annotator is the only kind that opens a drift stream, because the accumulator opens only under `len(Annotators) <= 1` (`internal/evaluator/validate.go:1578`). Two annotators buy an agreement figure and cost the post-promotion monitoring; one buys the monitoring and leaves `iaa` unmeasured. Choose before you recruit: who labels decides it, not a leaf you change later. So the ordering below is not decoration. You turn the queue on, give two people an identity the server derives rather than accepts, claim work, submit a verdict against a row, and then read the assembly line that says whether any of it counted. Every block on this page came off a run of the shipped binary against a ledger it minted itself, and the conditions are named under each one. If you have no judged rows yet, [Configure a judge](/docs/judges/configure-a-judge) is the config surface and [How a judge call happens](/docs/judges/how-judging-works) is what it does with them. What these labels are then measured into, and why κ is the statistic, is [the walk that promotes a judge](/docs/judges/agreement). ## Turn the annotation queue on The queue is off in a config that does not mention it, and both body-capture leaves have to be on before it can serve a single row. Set all three at once, because each missing one produces its own refusal and you would otherwise meet them in series. ```yaml dashboard: enabled: true state_dir: ./state/dash auth: mode: users capture: request: true response: true evals: outcomes: api_enabled: true annotation_queue: enabled: true ``` That block goes beside the `evals.evaluators[]` entry carrying your `judge:` block, and the whole file was loaded with `tokentriage validate` before it was published here. `capture` is the pair that stores the prompt and the completion; an annotator reads those bytes, so a deployment keeping hashes only has nothing to offer. `dashboard.enabled` is what mounts both the queue and the door a label comes back through, which is why it is in the block at all. `outcomes.api_enabled` needs one sentence of its own, because it does less than it looks like it does. The config loader reads it and requires `dashboard.enabled` beside it; nothing in this build reads it at request time. Measured on a live instance: `POST /tt/api/v1/outcomes` accepted a label with the leaf absent from the config entirely. Set it anyway, because it is what a reader of your config will look for, and do not reach for it as an off switch. Leave the queue leaf out and a claim comes back like this: ```json { "type": "/tt/api/problems/feature_absent", "title": "Feature not wired in this deployment", "status": 404, "detail": "the annotation queue is not enabled on this deployment (evals.annotation_queue.enabled is off, which is the default)", "code": "feature_absent", "request_id": "req_157c1a18b96f32e5", "enable_hint": "The annotation queue is OFF by default. Set `evals.annotation_queue.enabled: true` and restart. The queue serves EXISTING ledger rows for human labelling and opens no second write path: a submitted label still travels through POST /outcomes with its actor server-derived. ★ BEFORE YOU HAND THIS TO ANNOTATORS: a claim serves the row's stored prompt and completion verbatim, so it requires the privileged `content:read` scope, which only an ADMIN-role credential can hold — an operator-role token cannot even be minted with it, and an operator dashboard session does not carry it. Turning this leaf on does not by itself make the queue reachable for a non-admin annotator. Body capture must also be on (see the capture refusal this route reports separately)." } ``` Those are the response bytes with whitespace added and no value changed. `request_id` is the one that run minted and yours will differ; everything else is a constant in the binary. The refusal for capture is a different one, and it carries the trap worth knowing before you plan a labelling session: **capture is prospective.** The binary's own words are *"only exchanges recorded after the restart have bodies, so rows already on the ledger stay unclaimable"*. Turning capture on today does not make yesterday's traffic labellable. One boot line says the rest of the machinery came up: ```json {"time":"2026-08-24T12:14:40.7698743-07:00","level":"INFO","msg":"evals: B.7 promotion registry opened","dir":"state\\dash\\quality","validation_state":"empty","detail":"","standing_promotions":0,"auto_validate":false,"interval":"24h0m0s","new_labels_threshold":25} ``` That line is the precondition for one of the two claim strategies below. It came from a source build on Windows, so the `dir` is backslashed and a Linux or macOS host prints a `/`-separated path; `time` is that machine's, with its UTC−7 offset. `validation_state: empty` is a claim about the registry rather than a failure, and the three policy values are the shipped defaults. ## Give each annotator an identity the server derives An annotator's name is not a field on the submission. The server takes it from the authenticated principal and stamps it on the row, which is what makes two annotators countable as two rather than as whoever the client said it was. That needs named users: ```console $ tokentriage user add annotator-a --role admin --state-dir ./state/dash Password: added user "annotator-a" with role admin ``` `Password:` is a prompt on stderr and the confirmation is on stdout, so a terminal interleaves them in that order. Run it once per annotator, before the server boots. **Annotators need the admin role, and an operator session is turned away at the door.** A claimed item carries the request's stored prompt and completion verbatim, which sits behind the privileged `content:read` scope, and only an admin-role credential holds that scope. An operator-role session carries `*:read`, `*:write` and `config:apply`, none of which grant it. Measured on a live instance, an operator session claiming work gets `{"status": 403, "detail": "role operator is below required admin", "required_role": "admin"}` before the handler runs. That is a real constraint on who you can hand this to, and it is worth settling before you recruit two people. The write half is different: `POST /tt/api/v1/outcomes` sits at the operator floor, so a person who can submit labels is not automatically a person who can be offered them. ## Claim work, and read why the row was offered A claim offers work. It picks rows a human should look at, leases them so two annotators are not sent the same request by accident, and hands back the text to read. It writes nothing to the ledger. The dashboard serves this at **Evals · annotate**, where claiming is a button rather than something the page does when it loads. Over the API it is one call, `POST /tt/api/v1/evals/annotation-queue/claim`, carrying the annotator's dashboard session: ```json { "strategy": "starved_segments", "k": 60 } ``` Two strategies exist and they answer different questions. **`starved_segments`** ranks coordinates by how few human labels their gold set already holds, thinnest first. It needs the label counter the judge runtime supplies, which the standard binary wires wherever it opens the B.7 promotion registry, so the boot line above is its precondition. Here is one item off a claim on a ledger with judged rows and no labels yet: ```json { "items": [ { "req_id": "tt_56a7tyvh3kbmb3npdh4jcecsqe", "req_ts": "2026-08-24T19:14:41.716Z", "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat", "instruction": "{\"model\": \"gpt-5\", \"messages\": [{\"role\": \"user\", \"content\": \"w23 item 0000 W23-VERDICT-PASS\"}]}", "output": "[… 340 bytes of stored response body, elided …]", "priority_reason": "starved_segments: this segment's gold set holds 0 of the 50 human labels validation needs (50 short); the gold set is projected and empty, which is not the same fact as no gold set" } ], "lease_expires": "2026-08-24T20:16:42.395Z" } ``` That claim asked for sixty and was granted sixty; one item is shown and the single elision is marked. `instruction` is the stored request body verbatim, and `W23-VERDICT-PASS` inside it is a marker the capture harness planted so a loopback listener could stand in for the judge model. Yours carries your own traffic. `req_id`, `req_ts` and `lease_expires` move on every run; `segment` is this fixture's routing; `priority_reason` is the server's own sentence, and the `50` in it is `JudgeValidationMinN`, supplied by the counter rather than spelled a second time by the queue. **`hard_cases`** ranks rows the judge found hard, and hardness has two ingredients: a judge that abstained, and a panel that split. A row carrying neither is not a hard case and is not offered. The consequence is worth stating flatly, because it looks like breakage: a plain `pointwise-bool` judge that answered every row produces no hard cases at all, and the claim comes back `{"items": [], "lease_expires": ""}` at HTTP 200. That is an answer about your traffic, not a fault. Reach for `hard_cases` where you run a panel or a method that abstains, and for `starved_segments` everywhere else. A malformed claim is refused by name. A strategy outside the pair gets `400 bad_request` reading *"strategy is outside the closed §2.5 vocabulary (starved_segments, hard_cases)"*, and a `k` below 1 gets *"k must be at least 1, a claim for nothing is not a claim"*. A `k` above 200 is trimmed to the cap and answered at HTTP 200 rather than turned away. ## Submit the verdict The annotation queue ships one endpoint, `POST /tt/api/v1/evals/annotation-queue/claim`; labels come back through `POST /tt/api/v1/outcomes`, correlated by `req_id`. That shared outcomes door is where a human verdict has always gone, and the queue stratifies which rows get looked at without opening a second way to write one. One submission per batch of rows, each carrying the `req_id` the claim offered: ```json { "outcomes": [ { "req_id": "tt_56a7tyvh3kbmb3npdh4jcecsqe", "metric": "task_completion", "evaluator": "annotation-queue", "eval_ver": "queue-v1", "source": "human", "basis": "measured", "value": 1.0, "pass": true } ] } ``` `evaluator` and `metric` have to be a pair the registry declares, or that item is refused with `evaluator_metric_mismatch` while the rest of the batch lands. The response is an envelope with one result per item; this is the envelope from a batch of thirty, with its first result and the other twenty-nine elided: ```json { "accepted": 30, "refused": 0, "coerced": 0, "results": [ { "index": 0, "accepted": true, "basis": "measured", "actor": "annotator-a", "disclosures": [ { "code": "actor_derived", "field": "/outcomes/0/source", "detail": "the annotator identity was derived from the authenticated principal; it is not an input, and a client-supplied actor is never trusted" }, { "code": "eval_cost_derived", "field": "/outcomes/0/eval_cost", "detail": "no eval_cost was supplied; resolved to a MEASURED $0 from the outcome source — an annotation or a downstream business signal costs this deployment nothing, and free is a measurement rather than an absence" } ] } ] } ``` `actor: "annotator-a"` is why the users block is in the config: the server put that name there, and a client-supplied one would not have survived the trip. `basis: "measured"` is what this door stored for the label; the judge-graded row beside it is capped elsewhere, and [Why your number is reported, not measured](/docs/judges/basis-ceilings) is which cap. **No lease identifier travels on either wire, so a submitted label cannot be tied to the claim that offered it.** The claim response carries `items` and one `lease_expires` instant for the whole batch. The submission names `req_id`, which is on both wires and is what you correlate on. What no deployment can measure from these two shapes is claim-to-submission completion, a re-claimed expiry against a first offer, or which strategy surfaced a given label. For lease accounting there is `lease_expires` and the server's per-claim audit row, and that is the whole of it. Submitting does not release the lease either. Measured: annotator-a submitted thirty labels and re-claimed immediately, and the queue answered `{"items": [], "lease_expires": ""}`, because the rows were still leased to that same annotator. Only expiry releases a lease, and the default lease runs an hour. ## The overlap is the part the queue will not hand you Here is where the two mechanics collide, and it is the one thing on this page that will cost you a day if you meet it after the labelling rather than before. The lease exists to stop two annotators labelling one request by accident. Inter-annotator agreement is computed on precisely the requests two annotators both labelled. So the queue's default behaviour is the opposite of what the measurement needs, and it is not subtle. Measured on a live instance: annotator-a claimed all sixty available rows, and annotator-b claiming the same queue one second later was served this. ```json { "items": [], "lease_expires": "" } ``` Three ways out, and only one of them is a plan. - **Hand the overlap set to the second annotator directly.** The submission door takes a `req_id` and does not consult the lease table, so annotator-b can label a request annotator-b was not offered. This is what the capture behind this page did: annotator-a worked the queue, and the first twenty of those request ids went to annotator-b out of band. - **Claim in turns, small.** With a one-hour lease and a claim of `k: 25`, one annotator can work a batch and the other can pick up the same rows after the lease lapses. It is slow and it is arithmetic you have to do yourself. - **Restart the server.** The lease table is in memory, so a restart releases every lease early. Measured, and it is benign in the only direction it can be wrong: a row is offered again sooner than promised rather than withheld longer. It is a recovery, not a workflow. Now the sizes, and one piece of vocabulary that decides them. A **label event** is one (annotator, request) pair. Two annotators on one request are two label events and one request, and the two floors count different things: | Floor | What it counts | Value | |---|---|---| | `JudgeValidationMinN` | label events on the segment, after a last-write-wins dedupe | 50 | | `IAAOverlapMinN` | requests a pair of annotators both labelled | 20 | | `GoldSetIAAFloor` | κ_h over those shared requests | 0.6 | So a sixty-request run where one annotator labels all sixty and the other labels twenty is eighty label events and twenty shared requests, and it clears both counts with nothing spare on the second one. Plan the overlap first and let the bulk follow it, rather than the other way round. An overlap sample is the only thing on this page that a second annotator can produce and a first cannot. Everything else scales by adding hours; this one needs a second person looking at requests somebody has already looked at, which feels like waste right up until the run declines for want of one more of them. ## How to tell it worked Run the validation verb and read its first block. The assembly header prints on every run, before any statistic, and it is the count of what your labels turned into: ```console $ tokentriage evals validate --config … --state-dir … --log … gold-set assembly — D:/tt-jg-scratch/w23work/run/state/decisions.jsonl sources: outcomes sets: 1 segments: 1 torn lines: 0 unorderable req_ts: 0 task_completion | task_completion|tier=frontier|model=gpt-5|endpoint=chat offered=80 admitted=80 (outcomes=80 feedback=0) requests=60 subjects=1 annotators=[annotator-a annotator-b] excluded: demo=0 synthetic=0 both=0 unattributed=0 superseded=0 total=0 skipped: no_metric=0 unkeyable=0 no_pass_verdict=0 incomplete_subject=0 pseudo_actor_outcome=0 shadow_no_subject=0 feedback (DJ10 dj10-none): enabled=false offered=0 admitted=0 refused: api_source=0 source_not_human=0 pseudo_actor=0 verdict_not_binary=0 ``` The path is that capture's `--log` and yours will be your own. Every counter on those lines is a measurement and is stable for a given ledger. The four to read: - **`annotators=[annotator-a annotator-b]`** is the server-derived list, and it is the check that your two identities landed as two. One name here means one annotator whatever your rota said. - **`admitted=80` beside `requests=60`.** Eighty label events over sixty requests: one annotator labelled all sixty and the other labelled twenty of them. `JudgeValidationMinN` is measured against the eighty. - **`excluded:` counts what the admission gate turned away** by provenance, and `total=0` here because that ledger carries no demo, synthetic or superseded rows. A non-zero number is labelling effort that went nowhere. - **`skipped:` counts rows that never became an offer** because a coordinate was missing. All zero on a clean run. Two lines further down, the run says what the overlap bought: ```console gold: admitted=80 annotators=2 licenses=true iaa=0.8000 revision=4587539 kappa: 0.6500 over n=80 (p_o=0.8250 p_e=0.5000) floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1 verdict: PROMOTED registry change: promoted ``` `iaa=0.8000` is κ_h over the twenty shared requests, and clearing `GoldSetIAAFloor` is what admitted the gold set. **Read the conditions before quoting either number.** The assembler, Cohen's κ, the inter-annotator agreement, every floor and the promotion registry are the shipped binary's and ran end to end against a ledger the proxy minted. The judge verdicts came from a loopback listener answering a canned score, and the two annotators' labels were designed by the harness that drove the run. So `0.6500` is the shape of a promoted run rather than a measurement of any judge, and `revision` is a digest over minted request ids and moves every time. ## How this goes wrong **The overlap lands one row short.** This is the failure this page exists for, and it is a cliff rather than a slope. On a deployment that set `evals.validation.require_iaa: true`, a gold set with nineteen shared requests instead of twenty gets this: ```console gold: admitted=79 annotators=2 licenses=false iaa=unmeasured (overlap_below_floor) revision=5dac3b3 kappa: NOT COMPUTED — a gate before it refused (the refusals below name which); validated_n=0 floors applied: JudgeValidationMinN=50 IAAOverlapMinN=20 SwapSetMinPairs=30 JudgeKappaFloor=0.5 GoldSetIAAFloor=0.6 SwapFlipCeiling=0.1 verdict: REFUSED registry change: refused refusal iaa_unmeasured: floor=IAAOverlapMinN need=20 have=19 overlapping rows iaa_unmeasured: inter-annotator κ_h is UNMEASURED (overlap_below_floor) — 2 annotator(s), widest overlap 19 rows, IAAOverlapMinN (20) required. `evals.validation.require_iaa: true` upgrades B.7's admitted-but-disclosed default (iaa: null) to a REFUSAL, so this gold set cannot license judge promotion. Assign a ≥20-row overlap sample to a second annotator, or clear the knob and accept the disclosure ``` Seventy-nine admitted label events, two annotators, and the run declines because one request left the overlap. `need=20 have=19` is the arithmetic, and the last sentence is the binary telling you which of two things to change. Exit status was 0, because a decline is an answer. Leave `require_iaa` at its default and the same gold set promotes instead, with `iaa=unmeasured (overlap_below_floor)` on the gold line and nothing else to warn you. That reason string reaches the text output of this verb and no other surface, so a null `iaa` read over the API alone does not say which of four situations produced it. **A second label from the same annotator replaces the first.** The dedupe is last-write-wins per (annotator, request), so a correction lands cleanly and a mistake lands as cleanly. In the run above, annotator-a submitted a second outcome on a request already inside the overlap, and that outcome carried no pass verdict: the later write won, the row was counted under `no_pass_verdict=1`, and the request dropped out of the overlap entirely. One stray submission moved `have` from 20 to 19. **A label on a request the ledger does not carry is accepted.** The outcomes door does not check that `req_id` names a row it has, so a typo returns `accepted: 1` and writes a companion row with nothing to join to. It then appears in no counter on the assembly block at all, not even a skip. Check `admitted=` against the number you submitted. **A row the queue offered can still be excluded at admission.** The queue drops demo and synthetic rows before it ranks, so those two rarely reach an annotator. The other two exclusion families are properties of the label rather than of the row, and no claim can see them coming: an outcome with no attributed annotator, and one a later label superseded. The `excluded:` line counts all four separately, and it is worth reading before you conclude the labels were lost. **The bodies aged out.** Capture is swept at `dashboard.capture.ttl`. A row whose prompt and completion are gone is skipped by the claim rather than offered blank, so an old segment can quietly stop producing work while the counter still says it needs labels. ## What this page does not cover **One of the two registered judge ids cannot be labelled toward a validation at all.** `judge-groundedness` publishes no pass floor, so the runner leaves every row it grades without a verdict and the assembler records a subject's verdict only where one is set. Labels collected against it accumulate no gold set. Use `judge-task-completion`, whose metric carries a floor. **Running the check and reading a decline.** The verb, its flags, and the floor a stopped run names are [Validate a judge and promote it](/docs/judges/validate-and-promote). **What the numbers mean once they exist.** κ, κ_h, the four gates and what a promotion changes are on [the walk that promotes a judge](/docs/judges/agreement), which also has the table separating a null `iaa` from an agreement of zero. **Every field on the claim and the submission.** The request and response shapes, and what each field carries when it comes back null, are [Judge API](/docs/reference/judge-api).