Decision ledger fields
Every field of one decision-ledger row, with JSON name, type, and meaning.
On this page
The decision ledger is an append-only JSONL file: one JSON object per line per proxied request.
This page documents the wire schema of that object field-by-field. The schema is v: 1 and evolves
additive-only — a breaking change bumps the top-level v.
Top-level record
| Field | Type | Meaning |
|---|---|---|
v |
int | Schema version (currently 1). |
ts |
string | Request arrival, RFC3339 with ms, UTC. |
req_id |
string | Server-minted request id; equals the x-tt-request-id response header. |
trace |
object | Explicit-header trace grouping. See trace. |
http |
object | Request/response envelope. See http. |
request |
object | Parsed-request summary (hash + counted features). See request. |
signals |
object | Map of every registered provider signal’s output (heterogeneous values). See signals. |
decision |
object | The routing decision and its disclosure blocks. See decision. |
usage |
object | Normalized token accounting. See usage. |
cost |
object | Priced outcome with provenance. See cost. |
latency |
object | Timing breakdown. See latency. |
outcome |
object | Response-side result. See outcome. |
mcp |
array | Per-tool MCP attribution; [] when none. See mcp. |
dims |
object | Business-dimension attribution (explicit headers only). Omitempty. See dims. |
content |
object | Capture-state flags for request/response bodies. Omitempty. |
demo |
bool | Synthetic-row watermark; present only on demo data. Omitempty. |
feedback |
array | Populated by the reader from feedback companion rows; never on the primary row. Omitempty. |
settlement |
object | Populated by the reader from the settlement companion row; never written on the primary row. Omitempty. See the settlement row. |
settlement_duplicates |
int | Further terminal settlement rows the reader saw for this req_id beyond the first, counted rather than discarded. Omitempty. |
units |
object | Non-token billable units (images/audio/…). Omitempty; token rows omit it. See units. |
disclosures |
object | Row-level honesty ledger (e.g. cross-dialect translation loss). Omitempty. See disclosures. |
replica |
string | Resolved writing-replica name in a scale-out fleet. Omitempty; absent = single-node/pre-scale-out. |
request
This primary prompt-metadata block stores a hash and counted features, not prompt text. Separate content capture follows its own configured retention and access policy.
| Field | Type | Meaning |
|---|---|---|
model |
string | The model the client requested. |
prompt_sha256 |
string | SHA-256 of the prompt. Not the prompt text — a fingerprint only. |
prompt_bytes |
int | Byte length of the assembled prompt. |
messages |
int | Message count. |
system_bytes |
int | Byte length of the system prompt. |
has_tools |
bool | Whether tools were declared. |
tool_count |
int | Number of declared tools. |
tool_result_count |
int | Number of tool-result messages present. |
max_tokens |
int | Requested max output tokens. |
reasoning |
object | Dialect-mapped reasoning knobs. Omitempty. { "effort": "high" }. |
trace
| Field | Type | Meaning |
|---|---|---|
trace_id |
string | Trace grouping id. |
step_id |
string | Step id within the trace. |
parent_id |
string | Parent step id. |
agent |
string | Agent name. |
step_class |
string | user_turn | tool_loop | provider_state | "". |
step_class_source |
string | header | derived | "". |
w3c |
string | Raw traceparent when present. |
source |
string | x-tt | traceparent | none. |
http
| Field | Type | Meaning |
|---|---|---|
path |
string | Request path, e.g. /v1/chat/completions. |
dialect |
string | Wire dialect, e.g. openai. |
stream |
bool | Whether the response was streamed. |
status |
int | HTTP status code. |
endpoint |
string | Endpoint kind (e.g. chat). Omitempty; absent = chat-or-passthrough. |
provider |
string | The provider that served the request — its own identity, decoupled from dialect. Omitempty. |
signals
signals is an open map of every registered signal provider’s output. Values are heterogeneous:
numeric sub-scores (float64), a token estimate (int under token_est), and the string
complexity_tier. Example:
{
"code": 1,
"complexity": 0.42,
"complexity_tier": "MEDIUM",
"token_est": 4605
}
The decision block
The wire-shaped view of the routing decision, plus its optional disclosure blocks.
| Field | Type | Meaning |
|---|---|---|
mode |
string | Operating mode, e.g. route | shadow | ledger. |
tier |
string | Chosen tier; "" when no tier was selected (e.g. a governance-refused row). |
upstream |
string | Chosen upstream; "" when none. |
model_rewrite |
object | {from, to} rewrite, or null if none. See below. |
rule |
string | Deciding rule: a rule id | default | override | pin. |
candidates |
array | Considered candidates with scores. See below. |
shadow_tier |
string | Shadow-mode would-route tier; null otherwise. |
override |
bool | A terminal override fired (e.g. x-tt-tier). |
fail_safe |
bool | The decision fell back to a fail-safe. |
affinity_pin |
bool | An affinity pin held the route. |
cooldown_skips |
array | Tiers skipped due to cooldown. |
refused_by_budget |
string | Flat-budget that refused the request (HTTP 429). Omitempty. |
refused_by_policy |
string | Typed reason of a policy refuse target (e.g. budget_exhausted). Omitempty. |
policy_rev |
string | Content hash of the active policy (v2 policy language). Omitempty. |
explain |
object | The decision-explain block (v2 rows). Omitempty. |
cascade |
object | Cascade block; present iff the decision staged a cascade. Omitempty. |
response_cache |
object | Response-cache disclosure. Omitempty; absent = cache did not engage. See decision.response_cache. |
governance |
object | Governance disclosure. Omitempty; absent = governance did not engage. See decision.governance. |
refused_by_limit |
object | Typed governance refusal (HTTP 401/403/429/503). Omitempty. See decision.refused_by_limit. |
predicted |
object | The calibrated quality the router consulted for the chosen tier. Omitempty; absent = no quality consult. See decision.predicted. |
decision.model_rewrite
| Field | Type | Meaning |
|---|---|---|
from |
string | The client-requested model. |
to |
string | The model actually served. |
decision.candidates[]
Each element (core.Candidate):
| Field | Type | Meaning |
|---|---|---|
tier |
string | Candidate tier. |
eligible |
bool | Whether the candidate passed filters. |
score |
float | Score, or null when unscored. |
decision.predicted
The quality-consult witness: the calibrated quality figure the router read for the tier it
chose, written only when a selector actually consulted candidate.quality.* at decision time.
An absent block is the fact “this row was decided without quality evidence” — it shows as the
quality_absent term in decision.explain, and it is never backfilled (backfilling would
retroactively claim the router saw a number it never saw). Because the block is written only on a
real consult, every field is always emitted: a lcb of 0 here is a real consulted zero, not
an absence. See Quality-aware routing for how the figure
changes a route.
| Field | Type | Meaning |
|---|---|---|
metric |
string | The metric consulted, e.g. groundedness. |
tier |
string | The chosen tier this prediction is for. |
lcb |
float | The decision value — the Clopper–Pearson lower bound the router gated on (not the mean; routing on a point estimate is the overconfidence anti-pattern). |
mean |
float | The segment’s calibrated mean. Disclosure only — never the decision value. |
confidence |
float | The artifact’s nominal coverage (1 − α). |
n |
int | Observations behind the prediction. |
segment |
string | The Mondrian segment key metric | tier | model | endpoint — the model is the one served, not requested. |
artifact_id |
string | The active qcal_ artifact consulted (qcal_<compactTS>_<sha8>). |
quality_ver |
string | The quality-view version that was consulted. |
decision.governance
Present iff governance engaged (a key resolved, limits were evaluated, shadow verdicts were produced, or a suspension was in effect). Absent means governance did not touch the request.
| Field | Type | Meaning |
|---|---|---|
kid |
string | Resolved key id. |
node |
string | Resolved tenancy node (e.g. org:acme/team:ml/user:dev1). |
verdict |
string | allow | would_refuse (shadow). |
would_refuse |
array | Shadow mode: limit ids that would have fired. Omitempty. |
suspended |
array | Limits for which fail-open engaged this request. Omitempty. |
user_asserted |
bool | The §2.2 dims-authority marker. Always present (a disclosed false is meaningful). |
rotation_grace |
bool | The request rode a key-rotation grace window. Always present. |
reserved_tokens |
int | TPM reservation taken, when engaged. Omitempty. |
serving_key |
object | BYO row-attribution: whose key served this row. Names only, never secrets. Omitempty. See below. |
gate_us |
object | Per-stage gate wall time in µs: { "a": int, "b": int }. |
decision.governance.serving_key
Names which tenant key/node served the row (BYO upstream-key attribution). Names only — never secret material.
| Field | Type | Meaning |
|---|---|---|
kid |
string | Public key id. Omitempty. |
node |
string | Tenancy node string. Omitempty. |
rotated_from |
string | Rotation-lineage predecessor kid; "" when none. Omitempty. |
decision.refused_by_limit
The full typed governance refusal, minus its human message. Present on every governance-refused row; the accountant charges it zero provider cost.
| Field | Type | Meaning |
|---|---|---|
type |
string | budget_exceeded | rate_limited | quota_exhausted | concurrency_exceeded | quota_cost_unfittable | scope_violation | invalid_key | enforcement_unavailable. |
refused_by |
object | The firing limit and the holder’s node. Omitted on a 401. See below. |
enforcement |
object | Posture and disclosed bound. Omitted on 401/403. See below. |
decision.refused_by_limit.refused_by
| Field | Type | Meaning |
|---|---|---|
limit_id |
string | Id of the firing limit. Omitempty. |
limit_type |
string | Limit type, e.g. budget_usd. Omitempty. |
unit |
string | Limit unit, e.g. usd. Omitempty. |
node |
string | The holder’s own node. Always present. |
period |
string | Reset period, e.g. month. Omitempty. |
resets_at |
string | RFC3339 reset time. Omitempty. |
retry_after_s |
int | Retry-after seconds. Omitempty. |
observed |
float | Observed value at breach; omitted (not 0) when the store cannot report it. |
limit |
float | The configured limit; omitted when unavailable. |
decision.refused_by_limit.enforcement
| Field | Type | Meaning |
|---|---|---|
posture |
string | enforcing | fail_closed. |
approximate |
bool | The refusal used a disclosed approximate bound. |
bound |
object | Quantitative bound terms. Omitempty. See below. |
note |
string | Human bound caveat. Omitempty. |
enforcement.bound
| Field | Type | Meaning |
|---|---|---|
tee_lag_rows |
int | Rows the accounting tee lags by. Always emitted (a 0 is meaningful). |
dropped_rows |
int | Rows dropped from accounting. Always emitted. |
in_flight_reserved |
int | Current TPM/lease reservation view. Omitempty. |
estimate_skew_p95 |
float | Rolling reconcile-delta p95. Omitempty. |
convergence_lag_rows |
int | Convergence lag in rows. Omitempty. |
decision.response_cache
The response-cache honesty ledger. Written unconditionally whenever the cache engaged (ran a
lookup or a store); absent means the cache did not engage. outcome and stored are always
present; every other field is omitempty and carried only when its class of row bears it (e.g.
kind/similarity/verified on a hit, bypass_reason on a bypass). A present block with
stored: false is the meaningful fact “the cache engaged but this row’s response was not written.”
| Field | Type | Meaning |
|---|---|---|
outcome |
string | hit | would_hit | miss | bypass | error. Always present. |
kind |
string | Matcher kind on a hit/would_hit, e.g. exact | semantic. Omitempty. |
rule |
string | Cache rule id that governed the request. Omitempty. |
strategy |
string | Matcher name. Omitempty. |
entry_sha7 |
string | Stored entry id (short hash). Omitempty. |
source_req_id |
string | Provenance: the req_id of the original request that produced the entry. Omitempty. |
age_ms |
int | Age of the stored entry at serve time. Omitempty. |
similarity |
float | Semantic match score. Omitempty (semantic only). |
threshold |
float | Match threshold (the calibrated T* when a verified policy is active). Omitempty. |
verified |
bool | Verifier verdict: true | false | absent (no verifier). Omitempty. |
scope_sha7 |
string | The scope key (short hash). Omitempty. |
stored_provider |
string | Provider that produced the stored response. Omitempty. |
stored_model |
string | Model that produced the stored response. Omitempty. |
source_usage |
object | The source row’s usage block. Omitempty (a bypass/miss omits it). |
saved |
object | Typed counterfactual savings (StageCost shape). Omitempty. See below. |
saved_basis |
string | Audit string: how saved was computed. Omitempty. |
embed_cost |
object | Embedding spend when metered (StageCost shape). Omitempty. See below. |
stored |
bool | This row’s response was written to cache. Always present (false is meaningful). |
bypass_reason |
string | On a bypass: client_off | stream_binary | tool_calls | over_budget | …. Omitempty. |
lookup_us |
int | Lookup wall time (µs). Omitempty. |
embed_us |
int | Embedding wall time (µs). Omitempty. |
serving_tier |
string | Cross-tier provenance: l1 | l2 | l3 — which tier served the hit. Omitempty. |
recall_exact |
bool | Semantic hit: true = exact-KNN recall; false = ANN (a disclosed approximation). Omitempty. |
recall_config |
string | Semantic ANN hit: the recall backend’s config, e.g. hnsw(m=16,ef_search=200). Omitempty. |
verified_active |
bool | A verified-semantic policy governed this decision. Omitempty. |
verified_calibrated |
bool | A calibrated T* replaced the static threshold (licenses the saving). Omitempty. |
verified_samples |
int | Labeled observations the calibration considered. Omitempty. |
verified_bound |
float | Wilson lower bound on the correct-serve rate at T*. Omitempty. |
saved / embed_cost (StageCost)
Both saved and embed_cost use the typed money shape — never a bare float. An unpriced value
sets unpriced: true rather than a silent $0.
| Field | Type | Meaning |
|---|---|---|
usd |
float | Dollar amount. |
estimated |
bool | The amount is an estimate. |
unpriced |
bool | The stage could not be priced; usd is not a real dollar amount. |
usage
Normalized token accounting (core.Usage), reused directly by the ledger.
| Field | Type | Meaning |
|---|---|---|
input_tokens |
int | Prompt tokens. |
output_tokens |
int | Completion tokens. |
cache_read_tokens |
int | Provider prompt-cache read tokens. |
cache_write_tokens |
int | Provider prompt-cache write tokens. |
cache_write_detail |
object | TTL split of the cache-write tokens: { ephemeral_5m_tokens, ephemeral_1h_tokens }. Omitted when absent. |
reasoning_tokens |
int | Reasoning/thinking tokens. |
total_tokens |
int | Total tokens. |
source |
string | Provenance of the counts: provider | estimated | absent | replay (a cache-served row). |
cost
The priced outcome with provenance. Only ledger.CostFrom can construct a priced cost, so
usd never carries a dollar figure without a resolved key.
| Field | Type | Meaning |
|---|---|---|
usd |
float | Dollar cost. Not a real amount when unpriced is true. |
pricing_key |
string | The catalog key the price resolved under, e.g. anthropic/claude-haiku-4-5. |
pricing_snapshot |
string | Price provenance, source@date#sha7, e.g. litellm@2026-07-14#179affb. The sha7 is a pinned constant, not a hash of the vendored catalog. |
estimated |
bool | The cost is an estimate. |
would_usd |
object | Per-tier shadow economics: {tier: usd}. null when not computed. |
unpriced |
bool | The pricing lookup missed. When true, aggregates must exclude-and-disclose the row, never sum its zero. Omitempty. |
unpriced_reason |
string | Why the dollar is not known, from a closed six-value vocabulary in two families. Pricing miss: model_unknown | region_unknown | endpoint_unpriced | unit_missing. Async settlement: settlement_pending | settlement_terminal. Omitempty. See the two families. |
region |
string | Region the price resolved under, or the region attempted on a region_unknown miss. Omitempty. |
would_unpriced |
array | Tiers excluded from would_usd because their counterfactual could not be priced. Omitempty. |
The two unpriced-reason families
unpriced_reason is a closed vocabulary of six values in two families, listed here in the order
the source declares them. It has widened twice, both times for async settlement, and it is closed
again.
| Value | Family | What it says |
|---|---|---|
model_unknown |
pricing miss | No catalog entry for the model under the provider prefix. |
region_unknown |
pricing miss | Region-qualified entries exist, but not for this region. |
endpoint_unpriced |
pricing miss | The endpoint-mode filter eliminated every candidate key. |
unit_missing |
pricing miss | A key resolved, but it lacks the rate units this cost shape requires. This is the reason a succeeded job settles under when its terminal object carried no billable quantity. |
settlement_pending |
async settlement | A job was submitted, relayed and ledgered; the provider has not determined the money yet. It arrives later on a settlement companion row joined by req_id. |
settlement_terminal |
async settlement | The job reached a terminal, non-succeeded state, so there is no money to resolve. |
The four pricing misses each say a price lookup was attempted and failed, which is what makes them
an operator’s work queue. The two settlement reasons say there is no price to look up: not yet, or
not at all. No edit to a price table clears either one, so a surface that folds them into the
needs-pricing cohort reports a catalog gap that is not there. GET /pricing/unpriced excludes both
and counts them separately under settlement_excluded.
Four things follow for anything that reads these fields.
| Do not | Because |
|---|---|
| Fold the two settlement reasons into the needs-pricing cohort | They are a second family, and no pricing action clears either one. Treat membership as a test on the value, not as a substring of the word settlement. |
Add pending_settlement_rows to unpriced_rows |
On the analytics envelope pending_settlement_rows is a subset of unpriced_rows, not a sibling: one pending row increments both counters. Summing them double-counts it. |
Derive a pending count by matching unpriced_reason == "settlement_pending" |
An abandoned job keeps that reason after it has settled, so the string match climbs monotonically under a name that promises it comes down. The product’s own test is structural instead: no settlement row, and unpriced, and that reason. |
| Read the submit row as corrected in place | The ledger is append-only. A settlement is a second line joined on req_id; the submit row keeps the cost cell it was written with, and readers that show both put them side by side. |
Adding a settled dollar up, and reconciling a month that contains one, is a procedure rather than a field definition. Reading a settled cost walks it.
latency
| Field | Type | Meaning |
|---|---|---|
ttft_ms |
int | Time to first token (ms). |
total_ms |
int | Total wall time (ms). |
overhead_us |
int | TokenTriage in-proxy overhead (µs). |
outcome
| Field | Type | Meaning |
|---|---|---|
finish_reason |
string | stop | length | tool_use | content_filter | error | budget_refused | limit_refused | "". |
error |
string | Upstream error string, truncated. |
client_disconnect |
bool | The client disconnected before completion. |
signals |
object | Turn-level outcome signals ({name: bool}). |
oracle |
object | Off-path oracle label {score, label}, filled async when sampled; else null. |
shadow_grade |
object | Shadow-judge verdict, joined by the reader from a companion row; else null. Omitempty. |
mcp
Per-tool MCP attribution; [] before any tool ran. Each element:
| Field | Type | Meaning |
|---|---|---|
tool |
string | Tool name. |
server |
string | MCP server name. |
calls |
int | Number of calls to this tool. |
result_bytes |
int | Total result bytes returned. |
input_tokens_attributed |
int | Input tokens attributed to this tool. |
estimated |
bool | Always true — the attribution is a char/4 estimate, never a provider figure. |
dims
Business-dimension attribution — explicit headers only, never inferred from content. Omitempty.
user/tenant/feature/tags come from x-tt-* headers; team/key are the server-stamped
principal dims when governance resolves an authenticated key.
| Field | Type | Meaning |
|---|---|---|
user |
string | From x-tt-user. Omitempty. |
tenant |
string | From x-tt-tenant. Omitempty. |
feature |
string | From x-tt-feature. Omitempty. |
tags |
object | Bounded free-form {k: v} from x-tt-tag-<k>. Omitempty. |
team |
string | Stamped principal dim: resolved team node. Omitempty. |
key |
string | Stamped principal dim: resolved kid. Omitempty. |
units
Non-token billable units, present only on non-token endpoints (images/audio/…). When present, all four keys are emitted — it is a complete unit statement.
| Field | Type | Meaning |
|---|---|---|
images |
int | Billable images. |
seconds |
float | Billable seconds. |
characters |
int | Billable characters. |
requests |
int | Billable request units. |
disclosures
The row-level honesty ledger. Omitempty.
| Field | Type | Meaning |
|---|---|---|
translation |
object | Cross-dialect translation loss. Written on every translated row (an empty {from,to} is itself the fact “translated, nothing lost”). |
guardrails |
object | What the configured guardrails did on this request. Written on every guarded row. |
disclosures.translation
| Field | Type | Meaning |
|---|---|---|
from |
string | Source dialect. |
to |
string | Target dialect. |
dropped_params |
array | Parameters dropped in translation. Omitempty. |
approximations |
array | Approximations made in translation. Omitempty. |
disclosures.guardrails
| Field | Type | Meaning |
|---|---|---|
policy_ref |
string | The auditable policy pointer, <profile>@cfg<sha7>. |
engaged |
array | Legacy selected/evaluated binding summary; inspect versioned receipt executions for actual completion. |
fired |
array | Observed findings and selected actions; a fire does not prove successful mutation or delivery. |
suspended |
array | Incomplete protection under either posture: {binding, phase, reason, detector}. Read execution stage and delivery separately. |
receipt |
object | Versioned generation, executions, independent external invocations and delivery; absent historical receipt means unknown coverage. |
input_ms / output_ms |
float | Added latency per gate. |
confidence_refused |
int | Confidence envelopes this layer declined to record. Omitempty. |
Each fired[] entry carries binding, category, phase, detector, detector_version,
action (the verb), effect (what happened to the bytes), a content-free entities census,
reversible, an optional confidence envelope, and your rule_ref.
effect is not the same as the verb: a redact in monitor mode records annotated. Stock
activation rejects spanless mutation bindings. Read execution stage, transformation and delivery
together, because a committed transformation can still be withheld from the client. reversible
is derived from the action table. See Verbs & effects and
the vocabularies.
Companion rows
The ledger stays append-only: async labels are appended as separate rows keyed on req_id, not
mutated in place. Readers join them onto the primary row. A companion row carries a kind
discriminator; a primary record has no kind field at all, so the two can be told apart without
guessing. This build writes seven kinds.
kind |
Field | Meaning |
|---|---|---|
oracle |
oracle |
Off-path oracle label {score, label}. |
shadow_grade |
grade |
Shadow-judge verdict for a duplicated request. See the shadow-grade row. |
eval |
eval |
One evaluator, human, business or OTel observation of one metric. Carries the judge disclosure. See the eval row. |
feedback |
feedback |
One actor’s verdict on a request: {verdict, note, actor, source, ts}, where source is human or api and the two are held apart rather than blended. |
guardrail |
(flat) | What the configured guardrails did, in the shape disclosures.guardrails documents. |
beside |
(flat) | Beside-mode observation: what the incumbent gateway did for a relayed request, and what this router would have decided. |
settlement |
settlement |
The money and usage of an async job, appended by the poller once the job reaches a terminal state. See the settlement row. |
The settlement row
Written when a job whose submit row was UNPRICED(settlement_pending) reaches a terminal state at
the provider. The jobs poller appends it, minutes to hours after the submit exchange finished, and
readers join it onto the primary row by req_id. Its cost is built through the same
ledger.CostFrom chokepoint as every other dollar on the ledger, so a terminal object carrying no
usage settles unpriced with a typed reason rather than a confident $0.00.
| Field | Type | Meaning |
|---|---|---|
v |
int | Schema version (currently 1). |
kind |
string | settlement. |
req_id |
string | The submit exchange’s req_id. The whole join key. |
ts |
string | Settlement time, RFC3339 with ms, UTC. Not the request time: the two are minutes to hours apart, so folding one into the other’s window backdates the dollar. |
provider |
string | Descriptor id; the twin of http.provider on the submit row. |
endpoint |
string | Endpoint kind; the twin of http.endpoint. |
job_id |
string | The provider’s job identity. An id, not a credential. |
status |
string | succeeded | failed | canceled | expired | abandoned. Every value is terminal. abandoned is the one TokenTriage writes itself, when the poll-attempt cap runs out. |
attempts |
int | Poll attempts consumed before this row was written. |
error |
string | Provider-reported failure detail, or the abandonment reason. Omitempty; truncated at 1024 bytes. |
usage |
object | Provider-reported token accounting, in the usage shape. source: "absent" when the terminal wire carried none. |
units |
object | Non-token billable units settled from the terminal object. Omitempty; when present, all four keys of units are emitted. |
cost |
object | The settled money with provenance, in the cost shape, or an unpriced cell carrying one of the two settlement reasons. |
A reader takes this row’s cost in place of the submit row’s rather than adding the two, and counts
any further terminal row for the same req_id under settlement_duplicates instead of folding it
in.
The eval row
An eval outcome is a companion row rather than a field on the decision record because a delayed
outcome is the normal case: a label can land seconds or weeks after the request it describes. The
join key is the tuple (metric, evaluator, source, actor), and the actor component is what lets two
annotators label the same request without one overwriting the other.
Field (under eval) |
Type | Meaning |
|---|---|---|
metric |
string | Registered metric id. |
value |
float | The metric-scaled observation. A bare float here is correct: a single observation is not an estimate, so it has no n, no interval and no calibration of its own. |
pass |
bool | The binary projection. Omitempty; absent means the metric has no pass floor, which is not the same as a false. |
basis |
string | measured | reported. insufficient is an aggregate state and appears on no row. Capped at write time from the metric’s ground-truth class. |
evaluator |
string | Registered evaluator id: the measurement’s provenance. |
eval_ver |
string | Evaluator version (prompt/model/code hash). |
source |
string | evaluator | human | business | otel. feedback is a registry-level source mapped at read time from a feedback row and is not a legal value here. |
actor |
string | Server-derived, and required when source is human. Omitempty otherwise. |
judge |
object | Present iff an LLM judge produced this outcome. See the judge disclosure. Omitempty; absent means no judge graded the row. |
panel |
object | Present iff that judge ran as a panel. See the panel disclosure. Omitempty; absent means the judge was not a panel, and is not an empty panel. |
cost |
object | The observation’s own eval spend, in the StageCost shape. An unpriced judging call sets unpriced: true and is counted, not summed as a confident $0. The cost of the graded output stays on its own row. |
ts |
string | The outcome time, RFC3339 ms UTC. Not the request time. |
labels |
object | Bounded {k: v} (experiment id, dataset id, drift probe). Omitempty. |
The judge disclosure
The same judge object travels on an eval row and on a shadow_grade row, so the two carriers
cannot drift apart. It exists so a judge score can be traced to the model that produced it and to
whatever validation stands behind that model, which is what stops a judge score reading as anonymous
ground truth. How a judge call happens walks one of these rows
from the request that produced it.
Four of these fields are pointers, and that is the whole reading. kappa, swap_flip,
validated_n and iaa are absent when unmeasured and present when measured, including when the
measurement is 0. A κ of 0.0 is agreement at chance level, which is the worst result the
validation protocol can return; an omitempty value type would have deleted it. So on this object,
absent and zero are different facts, and the difference matters in both directions.
Field (under judge) |
Type | Meaning |
|---|---|---|
model |
string | The judge model that graded this row. Always present. |
prompt_sha |
string | sha7 of the judge template: the pointwise method’s own version. Always present. |
validated |
bool | Whether the κ gate had passed for this metric and segment when the row was written. Always present, and a disclosed false is meaningful. Promotion is forward-only, so a row written before a promotion keeps validated: false afterwards; that row and a later promoted state are both true at once. |
kappa |
float | Judge-to-human agreement. Omitempty pointer: absent ⇒ unmeasured; 0 ⇒ measured, at chance level. |
swap_flip |
float | Position-swap flip rate. Absent ⇒ unmeasured; 0 ⇒ measured, and perfect. |
validated_n |
int | Size of the validation set behind validated. Absent ⇒ no validation run stands behind this row. |
iaa |
float | Inter-annotator κ of the gold set the judge was validated against. Absent ⇒ agreement was not measured, and it is not agreement of zero. Four situations produce an absent value: one annotator, an empty gold set, no annotator pair sharing enough rows, or κ undefined on the pair that does. This build carries the reason on no wire and on no dashboard panel, so an absent iaa alone does not say which of the four applied; tokentriage evals validate prints unmeasured (<reason>) in its text output. |
method |
string | The judging method that actually graded this row, from the closed vocabulary pointwise-bool | pointwise-rubric | pairwise | reference | panel. Read off the built scorer, not off the config leaf, so a deployment that configured one method and composed another is visible here. Omitempty; absent ⇒ an older binary, or a disclosure rebuilt from the promotion registry rather than emitted by a grading run. |
abstain_feature |
float | This verdict’s own abstention feature: the method’s raw scalar (swap agreement, vote margin, rubric margin). It is an input to conformal calibration and is not a probability about the output, which is why it is not called confidence. Per-verdict, not per-subject. Omitempty; absent ⇒ this verdict’s method published no feature. |
The panel disclosure
Present on an eval outcome only when the judge ran as a panel. A panel is one instrument, so this block records how the one verdict was reached rather than promoting its seats individually. The vote vector always travels: a 2-1 and a 3-0 are different facts, and a margin with no votes behind it would be a number nobody can re-derive.
Field (under panel) |
Type | Meaning |
|---|---|---|
members |
array | The member transports that voted. Each is {model, family}, the family resolved at composition and recorded here as a plain string. |
votes |
array | Aligned with members by position: votes[i] is members[i]’s vote. The alignment is positional and load-bearing. |
rule |
string | The aggregation rule that produced the verdict. majority is the one value this build writes. |
margin |
float | abs(lead) / len(members), the vote margin the abstention feature reads. Derivable from votes, so a 0 here is checkable against the vector. |
The shadow-grade row
Written when the shadow judge grades a duplicated request. It carries two dollars and up to two
subjects: the duplicate request’s own cost and the grader’s cost, and a pointwise verdict beside an
optional pairwise trial. The two subjects are held apart: chosen_ok and shadow_ok belong to the
pointwise judge, the trial belongs to the pairwise one, and the two prompt_sha values on the row
are the two methods’ own versions and are not crossed.
Field (under grade) |
Type | Meaning |
|---|---|---|
chosen_ok |
bool | The pointwise judge’s verdict on the response that was served. null when the judge was undecided or failed. |
shadow_ok |
bool | The same verdict on the shadow duplicate. null on the same terms. |
judge_model |
string | The grading model. |
duplicate_cost_usd |
float | What the duplicate request cost. Not the grader’s cost. |
duplicate_tier |
string | The tier the duplicate was sent to. |
duplicate_unpriced |
bool | The duplicate could not be priced. Omitempty. |
judge_cost |
object | The grader’s own spend, in the StageCost shape, summed across both grading calls. Omitempty; absent ⇒ no judging call was made, which differs from a call that cost nothing measurable. Unpriced is contagious through the sum. |
judge |
object | The same judge disclosure an eval row carries, for the pointwise grading of this duplicate. Omitempty. |
judge_id |
string | The instrument that graded the duplicate. Omitempty; with metric, judge.model and judge.prompt_sha it names four of the six coordinates a promotion is scoped by. A graded row missing it is counted at assembly, not dropped. |
metric |
string | The metric the duplicate was graded on. Omitempty. The segment is deliberately not a field: the read side derives it from the primary record, so the row cannot file a subject under a spelling that holds none of its labels. |
pairwise |
object | Present iff the pairwise method ran on this duplicate. See the pairwise trial. Omitempty; absent is an older binary or a method that did not run, and is not a zero-valued trial. |
abstain_reason |
string | Set iff a judged subject on this row abstained, from the closed vocabulary swap_inconsistent (the both-orders trial flipped), below_conformal (the accept region turned the verdict away at the configured risk), panel_split (the panel vote fell short of the rule) and out_of_scope (input outside the judge’s validated scope). Omitempty; empty ⇒ no subject abstained, or the row predates the field. The reason is subject-scoped, so it can belong to the pairwise trial alone while chosen_ok and shadow_ok remain the pointwise subject’s answers. Every vocabulary the judge track prints is enumerated in the judge vocabularies. |
The pairwise trial
One A/B comparison of the chosen response against the shadow duplicate, run in both presentation orderings. A single-ordering trial cannot be represented at the source, so this block carries what a two-ordering trial produced.
Field (under pairwise) |
Type | Meaning |
|---|---|---|
prompt_sha |
string | The pairwise trial’s own method version. The sibling judge.prompt_sha on the same row belongs to the pointwise subject. |
winner_ab |
string | A | B | TIE: the winner when the chosen response was presented first. |
winner_ba |
string | The winner when the shadow was presented first, reported in the same frame, where A still means the chosen response. So winner_ab == winner_ba means the judge preferred the same side under both orderings. |
agree |
bool | false ⇒ the trial flipped when the order reversed, and the pairwise result abstains with abstain_reason: swap_inconsistent. |