Judge API

Three endpoints answer questions about the models grading your traffic: which judge subjects this deployment has observed and what B.7’s validation protocol established about each, what one subject’s history holds, and which rows a human should label next. All three sit on the admin plane under the /tt/api/v1 prefix served by the dashboard server.

The site renders the full typed spec at /api (Scalar). This page is the operator’s reading of it, and every field meaning below was read off the Go type that produces the bytes, with that producer named beside it. Where the generated description and the producer disagree, the producer is what your client will meet.

This page gives the shapes, the field semantics and the problem types. The closed enums those payloads carry, and the wider evals surface these three routes sit inside, are elsewhere; both are linked at the foot.

Method & path	Role · scope	What it answers
GET /tt/api/v1/evals/judges	viewer · evals:read	Every judge subject this process has observed, with the validation reading that decides whether its scores may claim measured.
GET /tt/api/v1/evals/judges/{subject}/history	viewer · evals:read	One subject’s promotion timeline and the validation record the registry holds for it, refusals included.
POST /tt/api/v1/evals/annotation-queue/claim	admin · quality:write and content:read	A batch of ledger rows leased to the caller for human labelling, each carrying why the strategy offered it.

Route floors come from the matrix at internal/authn/roles.go:503,516,517; the two read routes are registered at internal/api/contract/ops_judges.go:95,139 and the claim at internal/api/contract/ops_annotation.go:204.

GET /evals/judges

The collection. No parameters, no paging. One row per observed subject, ordered by judge_id, metric, segment, model, prompt_sha, eval_ver so the wire order is stable across processes (internal/judgeval/registry.go:738-760).

{
  "auto_validate": false,
  "subjects_observed": 1,
  "subjects": [
    {
      "subject": {
        "id": "1d7a361adad5",
        "judge_id": "judge-task-completion",
        "model": "claude-haiku-4-5",
        "prompt_sha": "1fc3923",
        "eval_ver": "judge-v1+cfg15acf67",
        "metric": "task_completion",
        "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat"
      },
      "state": "promoted",
      "kappa": 0.6761023407729995,
      "validated_n": 105,
      "iaa": 0.8397435897435899,
      "gold_revision": "93f8396",
      "floors_applied": [
        "JudgeValidationMinN", "IAAOverlapMinN", "SwapSetMinPairs",
        "JudgeKappaFloor", "GoldSetIAAFloor", "SwapFlipCeiling"
      ],
      "floors": {
        "judge_validation_min_n": 50,
        "iaa_overlap_min_n": 20,
        "swap_set_min_pairs": 30,
        "judge_kappa_floor": 0.5,
        "gold_set_iaa_floor": 0.6,
        "swap_flip_ceiling": 0.1
      },
      "spend": null,
      "selective_risk": null,
      "anchor": null,
      "last_run": "2026-08-24T18:08:58.333Z"
    }
  ],
  "validation_state": "restored",
  "notes": [
    "spend is NOT MEASURED and is served null on every row … [elided, ~1,700 chars]",
    "selective_risk carries Appendix B.2's certificate … [elided, ~2,400 chars]"
  ]
}

HTTP 200, 4,715 bytes on the wire. The server sends compact JSON; the block above is that body with whitespace added and the two notes[] strings replaced by a marked elision, and nothing else was touched. It came from a source build on Windows serving a harness ledger whose judge verdicts came from a loopback stub and whose human labels were designed by that harness, so κ, N and iaa are what a promoted subject looks like rather than what any judge scores. The key set, floors_applied, the six values under floors, the three nulls and validation_state are not machine-specific; the subject id, gold_revision and last_run are.

The envelope

Field	What it carries
auto_validate	Whether the automatic validation loop is being driven against this gate in this process. Behaviour, not the config leaf: a run that asks by hand reads false (internal/judgeval/runs.go:79, registry.go:664-669).
subjects_observed	Distinct judge subjects this process has seen. A process-lifetime figure, not a deployment history, and a restart lowers it (internal/judgeval/registry.go:753).
subjects[]	One JudgeReading per observed subject. Normalised to [] on the route rather than trusted from the provider, because null and [] render differently in every client and only one of them reads as a list that was computed (internal/api/dashsrv/judges.go:63-68).
validation_state	The registry’s boot disclosure, one of empty, restored, unrestorable (internal/judgeval/registry.go:82-96). unrestorable means an artifact existed and was turned away, so the registry booted empty and every subject rides the reported ceiling.
validation_state_detail	The store’s own words about that state, clipped at 4,096 bytes with the number of dropped bytes named in the string (internal/api/contract/judges_types.go:600-620). It is not restricted to boot facts.
notes[]	Two server disclosures about fields it serves and does not produce, one about spend and one about selective_risk (cmd/tokentriage/judges_api.go:32,55,87). Both run past a thousand characters. Inside the document rather than beside it, so --json output stays parseable.

subjects: [] beside subjects_observed: 0 is a claim, not an empty answer: it says no judge has produced an outcome on this deployment. A client that renders it as a blank table has thrown away the only fact on the page.

One row

Field	What it carries
subject	The six coordinates a promotion is scoped by, plus id. prompt_sha is the judgecore method version, not a hash of your prompt file (internal/api/contract/judges_types.go:46-63).
subject.id	An opaque, path-safe handle: netstring-framed SHA-256 over the six coordinates, truncated to 12 hex characters (internal/api/contract/ops_judges.go:35-42). Opaque because the model coordinate contains /, and a percent-encoded key presents to the authorization layer as several path segments. Read it; do not construct it and do not parse it back.
state	promoted, never_validated or version_changed (internal/evaluator/validate.go:1362-1371). There is no refused member: a run that was turned away leaves never_validated with a record behind it. version_changed means a promotion exists but was earned under a different judge model, prompt or evaluator version.
kappa, validated_n	The judge-versus-human agreement coefficient and the admitted row count it was computed over. They travel together, gated on the record having measured a κ (internal/judgeval/registry.go:702-706). Absent means a floor before κ stopped the run. 0 is a measurement and means agreement at chance.
iaa	Agreement between the human annotators. A present null, never omitted. See the census below before you read a cause into it.
swap_flip	The share of position-swap trials whose two orderings named different winners. Absent means no swap set ran; 0.0 is measured and is the best result available (registry.go:713-716).
gold_revision	The gold-set digest the latest run was measured against. Empty is ordinary rather than exceptional: a restart re-seeds only the newest artifact’s records (registry.go:684).
floors_applied[]	The floor slots a run applied, as a constant list of Go identifiers. This is the vocabulary refusals cite.
floors	The values those slots held on that run, copied off the record and never re-derived (registry.go:722-723). Only judge_kappa_floor moves, and evals.validation.min_kappa can only raise it. Without these numbers a subject validated at 0.7 is byte-identical to one at 0.5.
spend, selective_risk, anchor	Present nulls. Each null carries a different reading; the census below gives all three.
last_run	When the validation that produced the standing record ran. Persisted, not stamped on read: two boots of the same state serve the same instant.
next_trigger	Absent. Nothing in this build assigns it.

The CLI envelope is a different document

tokentriage evals validate --json prints the same projections through the same marshaler, so a field the verb prints and the field this route serves are the same field. The envelopes differ, and a script that treats one as the other will read a false zero:

	evals validate --json	GET /evals/judges
Shape	{judges, history[], notes[]}	a bare JudgesResponse
Population	the subjects this run produced results for (cmd/tokentriage/evals_judges.go:451)	every observed subject (cmd/tokentriage/judges_api.go:105)
validation_state	absent: never assigned (evals_judges.go:443-462)	served
auto_validate	always the Go zero false	read off the gate
Per-subject history	history[] carries this run’s record	on the history route, and at most one record

Two of those rows are product defects rather than contract choices, and both under-report: an unrestorable registry is invisible to --json, and a deployment that is running the automatic loop is reported as not running it. The human transcript prints both correctly.

GET /evals/judges/{subject}/history

{subject} is the opaque id from subjects[].subject.id on the collection, and the route resolves it by minting an id for each observed subject through the same function that published it (cmd/tokentriage/judges_api.go:132-140). A client-constructed id resolves to nothing.

{
  "subject": {
    "id": "1d7a361adad5",
    "judge_id": "judge-task-completion",
    "model": "claude-haiku-4-5",
    "prompt_sha": "1fc3923",
    "eval_ver": "judge-v1+cfg15acf67",
    "metric": "task_completion",
    "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat"
  },
  "timeline": [],
  "records": [
    {
      "at": "2026-08-24T18:08:58.333Z",
      "version": "model=claude-haiku-4-5\u001fprompt=1fc3923\u001fver=judge-v1+cfg15acf67",
      "promoted": true,
      "kappa": 0.6761023407729995,
      "validated_n": 105,
      "iaa": 0.8397435897435899,
      "floors_applied": [
        "JudgeValidationMinN", "IAAOverlapMinN", "SwapSetMinPairs",
        "JudgeKappaFloor", "GoldSetIAAFloor", "SwapFlipCeiling"
      ],
      "floors": {
        "judge_validation_min_n": 50,
        "iaa_overlap_min_n": 20,
        "swap_set_min_pairs": 30,
        "judge_kappa_floor": 0.5,
        "gold_set_iaa_floor": 0.6,
        "swap_flip_ceiling": 0.1
      },
      "gold_revision": "93f8396"
    }
  ]
}

HTTP 200, 791 bytes, from the same instance and the same subject as the capture above.

That timeline: [] beside a promoted record is the shape a restarted server serves. The timeline is the promotion history this process observed. It is not reconstructed from artifacts at boot, because re-deriving each change kind from stored records would mean replaying the promotion state machine a second time, free to disagree with the first (internal/judgeval/registry.go:230-238).

The promotion above came from a CLI run in an earlier process, so the record survived the restart and the timeline entry did not. A client that treats an empty timeline as “this subject was never promoted” will contradict the records[] array sitting beside it.

Field	What it carries
subject	The same JudgeSubjectRef the collection serves.
records[].version, timeline[].version	The subject version a run was measured under, as model=…, prompt=… and ver=… joined by U+001F, which the server escapes as the six characters \u001f. It is a display string; the coordinates travel separately on subject.
timeline[]	Promotion changes this process observed, oldest first. kind is one of promoted, repromoted, demoted, reset_version_change, refused (internal/evaluator/validate.go:1423-1427). Empty on a process that has applied no change, including one that restored a standing promotion at boot.
records[]	At most one, and it is the record the registry currently holds (cmd/tokentriage/judges_api.go:147-150). A subject whose record did not survive the boot restore gets an empty list rather than a fabricated one.
records[].promoted	Whether that run promoted. A run that was turned away is a first-class record with promoted: false and its refusals[] populated, not a gap.
records[].refusals[]	One entry per named floor that fired. floor, not the numbers, is the discriminator: a shape refusal names no floor and carries zeroes in need and have (internal/api/contract/judges_types.go:620-635). text is the operator sentence the gate wrote, and unit names what need and have are counted in.
demotions[]	Omitted from every response this build serves. See the census.

POST /evals/annotation-queue/claim

The request is {k, segment?, strategy} (internal/api/contract/annotation_types.go:67-80).

{ "k": 60, "strategy": "starved_segments" }

Field	What it carries
k	How many items to lease. Below 1 is a 400. Above 200 is capped, not refused: the server trims to MaxOutcomesPerBatch and answers with fewer, because leasing more work than POST /outcomes accepts in one call would hand an annotator rows they cannot answer (internal/api/dashsrv/annotation_queue.go:374-378, internal/api/contract/ops_outcomes.go:110).
segment	Optional. Absent and empty are one state on this wire; either means every segment, and the strategy chooses across all of them.
strategy	A closed two-member vocabulary, starved_segments and hard_cases (internal/api/contract/annotation_types.go:46-56). Anything else is a 400 naming both members.

starved_segments ranks coordinates by how few human labels their gold set holds, thinnest first. It needs the label counter the judge runtime supplies, which the standard binary wires wherever it opens the promotion registry (cmd/tokentriage/run.go:1112,2132). dashboard.state_dir is defaulted at decode (internal/config/defaults.go:259-260), so omitting that leaf is not how a deployment loses the counter; 404 feature_absent naming the unwired counter comes back where the counter was not composed at all — an embedded dashboard server built without Deps.SegmentLabelCounts, or a B.7 registry that failed to open.

hard_cases needs no gold set. It ranks on two ledger signals, an abstention on the shadow-grade companion row and a split panel on an eval companion row, weighted 2 and 1 (internal/api/dashsrv/annotation_queue.go:866-867,1213-1240). A row carrying neither scores zero and is not offered, so a deployment whose judge neither abstains nor votes on a panel gets an empty claim back from this strategy.

The response is {items, lease_expires} and nothing else. A claim that leased nothing is a 200 carrying the affirmative negative, captured here against a ledger whose gold set is already full:

{ "items": [], "lease_expires": "" }

items is never null, and an absent lease is the empty string rather than an instant the server did not grant (internal/api/contract/annotation_types.go:113-126). A populated claim carries one entry per leased row. The block below is one item from a sixty-row claim captured against a ledger with judged rows and no labels yet, with the stored bodies shortened and marked; the priority_reason is the server’s own sentence, untouched:

{
  "items": [
    {
      "req_id": "tt_56a7tyvh3kbmb3npdh4jcecsqe",
      "req_ts": "2026-08-24T19:14:41.716Z",
      "segment": "task_completion|tier=frontier|model=gpt-5|endpoint=chat",
      "instruction": "{\"model\": \"gpt-5\", \"messages\": [ … ]}",
      "output": "[… stored response body …]",
      "priority_reason": "starved_segments: this segment's gold set holds 0 of the 50 human labels validation needs (50 short); the gold set is projected and empty, which is not the same fact as no gold set"
    }
  ],
  "lease_expires": "2026-08-24T20:16:42.395Z"
}

Field	What it carries
req_id	The ledger reference. This is the correlation key: the label that answers this item names the same req_id on POST /outcomes.
req_ts	The request’s arrival time, RFC3339 with milliseconds, UTC.
segment	Always emitted, empty included, because which segment an item belongs to is the disclosure a stratified claim rests on.
instruction, output	The row’s stored prompt and completion, verbatim. These are why the route carries a content fence.
priority_reason	The server’s own sentence about why this item was offered under the requested strategy. Free text. The strategy-prefixed spelling in the goldens is illustrative and is not a format contract; no client may parse it (internal/api/contract/annotation_types.go:102-108).
lease_expires	One instant covering the whole batch, default one hour (internal/api/dashsrv/annotation_queue.go:103).

A claimed row is offered to nobody else until the lease expires, and submitting does not release it. Only expiry does. Claim what you will label.

Reading a null on a judges row

Every quantity on these payloads is a pointer, so null means unmeasured and 0 means measured at zero. Those are different facts and a client that coerces one into the other publishes a number nobody computed. This table is the whole discipline.

Field	Non-null in this build?	What null means	What it does not mean
spend	No	Not measured. readingLocked (internal/judgeval/registry.go:657-724) is the one projection both Gate.Reading and Gate.Readings use, and it assigns no Spend. AnchorMonitor.SpendFor (internal/judgeval/anchor.go:965) has no call site outside its tests.	$0. The grading calls cost money and each call’s cost is on its own ledger row (declog.ShadowGrade.JudgeCost, internal/declog/record.go:936); no shipped producer adds those dollars up onto a subject.
anchor	No	No who-drifted diagnosis has been attached to this reading. AnchorMonitor.Annotate (internal/judgeval/anchor.go:1017-1027) is the one function that sets it, and its two callers are both tests.	“the judge was checked and held”. A re-score that ran and found nothing carries diagnosis: neither with its counts.
next_trigger	No	The key is absent, not null: it is a string with omitempty and nothing in this build assigns it (internal/judgeval/status.go:232). Absence reads as unknown, not as never-again.	that validation will not run again.
iaa	Yes	Inter-annotator agreement was not measured. Four situations produce it and this wire carries none of them.	one annotator. That is one of the four. iaa: 0.0 is a measurement and means the annotators agreed at chance.
selective_risk	Yes	No accept region was fitted for this subject in this process. A refused fit is not null: it is a populated report carrying a refusal_code. An absent key means a server predating the field.	that this judge’s verdicts are risk-filtered at runtime, or that risk control looked and found nothing.
kappa, validated_n	Yes	Absent, not null: κ was never computed, because a floor before it stopped the run (registry.go:702-706).	agreement of zero. kappa: 0 is the most damning value the protocol can produce.
swap_flip	Yes	Absent: no swap set ran (registry.go:713-716).	a perfect flip rate. 0.0 is measured and is perfect.
floors	Yes	The registry holds no record for this subject. Nothing is re-derived here (registry.go:722-723).	that default floors applied.
gold_revision	Yes	Empty: the record was applied without a gold set, or a restart re-seeded only the newest artifact’s records (registry.go:684).	that the gold set was empty.
demotions	No	Omitted from every history response this build serves, so the dashboard heading that reads it has nothing to render (cmd/tokentriage/judges_api.go:153-156).	that no demotion happened. A demotion appears on timeline as a demoted entry, and the notice’s detail travels on the quality.judge.demoted event and the recalibration log.

The problem types each endpoint returns

Problems are RFC 9457 documents from a closed registry of 22 codes (internal/api/contract/problem.go:87-131). All three operations declare the framework set under default: bad_request, unauthenticated, forbidden, not_found, method_not_allowed, payload_too_large, rate_limited, internal.

Endpoint	Status · code	When
both reads	404 · feature_absent	No promotion registry is wired. The enable_hint names dashboard.state_dir and the boot line evals: B.7 promotion registry opened (internal/api/dashsrv/judges.go:48-51,54,74).
judgeHistory	404 · not_found	No subject with that id has been observed. Undeclared: the operation declares only feature_absent, so a client switching on code meets an undocumented value on this route’s most ordinary error path (internal/api/dashsrv/judges.go:92).
judgeHistory	400 · bad_request	An empty {subject} segment (judges.go:82). Defensive: the mux cleans // before matching, so no HTTP request has been constructed that reaches it.
claim	404 · feature_absent	Seven branches: no audit writer, evals.annotation_queue.enabled off (the default), body capture missing, a starved_segments counter that is not wired, no gold set to rank, a starved coordinate with no offerable row, and no canonical decision log (annotation_queue.go:337,401,416,441,489,533,620).
claim	400 · bad_request	Malformed body, a strategy outside the closed vocabulary, or k < 1 (:345,356,363).
claim	403 · forbidden	The role floor, or the content:read fence (:390).
claim	503 · budget_exceeded	The query budget: deadline, row cap or concurrency (:632).
claim	500 · internal	The gold-set label count failed, or the decision log could not be read (:473,639).

Two captured refusals, from a running instance:

{
  "type": "/tt/api/problems/not_found",
  "title": "Not found",
  "status": 404,
  "detail": "evals/judges: no judge subject with id 000000000000 has been observed on this deployment. Read the id from `subjects[].subject.id` on GET /tt/api/v1/evals/judges rather than constructing it",
  "code": "not_found",
  "request_id": "req_7d7303b3336f19e9"
}

{
  "type": "/tt/api/problems/bad_request",
  "title": "Bad request",
  "status": 400,
  "detail": "strategy is outside the closed §2.5 vocabulary (starved_segments, hard_cases)",
  "code": "bad_request",
  "request_id": "req_c482be09ed0cf83b"
}

request_id is per-request and yours will differ. Every other token in both bodies is the server’s.

What these three endpoints do not carry

• A judge’s dollars, summed. spend is a present null on every row. Per-call cost is on the ledger row; the addition onto a subject has no producer.
• A drift diagnosis. anchor is a present null on every row, and the judges dashboard does not render a named absence for it the way it does for spend.
• A schedule. next_trigger is absent from every row.
• A demotion notice. demotions is omitted from every history response.
• A paired verdict on an experiment. EvalExperiment carries no paired_verdict property on this contract, on the Go type or in the generated spec; the field exists in the dashboard’s own TypeScript. The carrier a server fills is QualitySource.paired_verdict on GET /evidence and GET /guardrails/evidence (internal/api/contract/evidence.go:152).
• A lease identifier. See the callout at the top of this page.
• A confidence interval on κ. κ travels with its n and no interval.

The rest of the calibration and evals surface, including the experiments and outcomes routes a judge’s rows travel through, is at Quality & evals API.