# Evals & calibration configuration

Every key under the evals block, with the default the code applies, the leaves that are required, and the configurations this build turns away.

Source: https://tokentriage.com/docs/reference/evals-config

---
import Callout from '../../../components/Callout.astro';
import NextSteps from '../../../components/NextSteps.astro';

The `evals:` block (`internal/config/evals.go`, §2.11) configures calibration & evals:
outcome ingestion, the declared evaluator instances, the judge-validation rules, quality
calibration, the annotation queue, and the drift detectors. It is additive over every prior schema:
an absent block compiles byte-identically to a pre-feature config.

<Callout type="key">
**There is no master `evals.enabled`, and one sub-block gate does less than its name suggests.**
`calibration.quality.enabled`, `outcomes.otel_ingest`, `annotation_queue.enabled` and
`validation.auto_validate.enabled` each gate their own subsystem when the binary runs.
**`outcomes.api_enabled` does not.** It is read once, at config load, to catch a single
combination, and nothing reads it when a request arrives. A zero-valued `evals:` asserts *"the
operator did not configure evals"*, never *"evals are off"*.
</Callout>

**This block fills no defaults.** `ApplyDefaults` leaves an `Evals` value byte-identical to
what was decoded. The **Default** column below names the value the *consuming* code applies to an
absent leaf, and the constant that spells it; nothing is written back into the decoded config. The
leaves are the parameters of a statistical claim (`confidence` **is** the claim, not an operational
knob), so a leaf a claim depends on is *required* when its block is on, and a leaf whose absence has
a true reading (`min_n` ⇒ the named floor; `refit_interval` ⇒ no scheduled refit) is left absent and
read that way.

## evals.outcomes

The two ingestion doors, and they do not behave alike. Both default to `false`, and both are refused
at config load when `dashboard.enabled` is false, because the management API that serves them is
mounted only when the dashboard is on. Only `otel_ingest` is then consulted again when a request
arrives.

| Field | Type | Default | Meaning |
|---|---|---|---|
| `api_enabled` | bool | `false` | Your declaration that typed batch ingestion on `POST /tt/api/v1/outcomes` is wanted. **It is read by the config loader and by nothing else** (`internal/config/evals.go:899`), where it catches one combination: the flag set against a disabled dashboard. The handler is registered unconditionally (`internal/api/dashsrv/outcomes.go:118-122`); what actually decides whether a call lands is the mounted dashboard, the management API's own role and scope check, and a wired decision-log sink. Measured on this build: with the whole `evals.outcomes` block deleted, `POST /tt/api/v1/outcomes` answered `{"accepted":1,"refused":0}` at HTTP 200. Keep it set, because it is what a reviewer of your config reads; do not reach for it as an off switch. |
| `otel_ingest` | bool | `false` | Serves `POST /tt/api/v1/outcomes/otel` (OTLP/HTTP JSON `gen_ai.evaluation.result`). Consulted again at request time (`internal/api/dashsrv/outcomes_otel.go:441`) before a byte of the body is read, which is the difference between this row and the one above it: off, the route answers `feature_absent` and names the leaf. Outcomes arriving this way are **`basis: reported`** by definition, so enabling it widens what may be *recorded*, not what may be *claimed*. |

## evals.evaluators[]

The declared scorer instances, in declaration order. Each names a **registered** evaluator id
(§2.3) and carries its own sampling and budget discipline. An instance is **one** scorer:
declaring two transport blocks under one id is refused.

| Field | Type | Required | Meaning |
|---|---|---|---|
| `id` | string | yes | Names a registered evaluator descriptor. Unique within the list. |
| `sample_rate` | float | **yes** | Fraction of eligible requests this evaluator scores, in `[0,1]`. Not defaulted: `0` (never) and `1` (every eligible request) are both plausible readings of an omission, so it is refused rather than guessed. `0` is legal and means "declared but not sampling". |
| `max_usd_per_day` | float | when `judge:` present | Per-evaluator daily spend cap (USD). **Mandatory whenever `judge:` is present**; optional otherwise. Must be **finite** and `>= 0` (`0` stops the evaluator at its first priced call; `.inf` is refused, since an infinite cap is not a cap). Enforced post-hoc. On a panel it caps the whole fan-out, not one seat. |
| `judge` | object | — | Configures the LLM-judge kind. Its presence is what makes the instance *metered*. See [judge](#evalsevaluatorsjudge). |
| `sidecar` | object | — | Configures the operator-run scorer-service client. See [sidecar](#evalsevaluatorssidecar). |
| `sidecar_judge` | object | — | Configures a judge served over the §2.4 sidecar wire. Config-complete and composition-incomplete: see [sidecar_judge](#evalsevaluatorssidecar_judge). |

<Callout type="warning">
**`sample_rate` is required, and it is the mistake that spends money silently.** An omitted rate
on a `judge:` entry is refused, not defaulted: one plausible reading scores nothing, the other
is an undeclared bill.
</Callout>

### evals.evaluators[].judge

An LLM judge phones out on every sampled request. Its outcomes are `basis: reported` until the B.7
validation protocol promotes them. The first three leaves are required on **every** judge block,
including one that sets `method: panel`, where the top-level `model` and `api_key_env` are still
refused when absent even though the seats carry their own.

| Field | Type | Required | Meaning |
|---|---|---|---|
| `external` | string | **yes** | The air-gap gate (SP-16). Must be the literal `allow` or the whole evaluator entry is **refused** at config validation. Any other value, including a typo or `true`, leaves the gate closed. The comparison is `!= "allow"`, never `== ""`. |
| `model` | string | **yes** | The judge model id sent to the provider. An empty value is refused; so is the literal `oracle` (the shadow block's grading-path selector, and sending it as a model id is a recorded wiring bug). Use a **different model family** than the tiers being judged: B.7 refuses a same-family judge, and on a panel that refusal extends to every seat against every configured tier. |
| `api_key_env` | string | **yes** | **Names** the environment variable holding the judge provider's API key (uppercase, e.g. `ANTHROPIC_API_KEY`). It must never *contain* the key: the running config is served verbatim to any principal holding `config:read`. |
| `endpoint_env` | string | — | **Names** the environment variable holding the URL the judge POSTs to. Absent ⇒ the shipped Anthropic Messages endpoint, so a config without this leaf composes the client it composed before the leaf existed. A literal URL is refused at validation. **The `api_key_env` key travels to whatever this resolves to, as a request header.** Refused beside `method: panel`, where the seats are the transports. |
| `method` | string | — | The §2.1 judging method. Absent reads as **`pointwise-bool`** (`EvalsJudgeMethodDefault`), the shipped grader. The closed vocabulary is `pointwise-bool` \| `pointwise-rubric` \| `pairwise` \| `reference` \| `panel`; anything else is refused by config, which does not repeat your value back. See [Judge methods](/docs/reference/judge-methods) for which of the five compose here. |
| `self_consistency_k` | int | — | The self-consistency sample count. Absent reads as **`1`** (`EvalsSelfConsistencyKDefault`), a single sample with no vote-margin feature. Present ⇒ `>= 1`. A value above 1 is refused on this leaf by `tokentriage validate` and again at boot: no scorer in `internal/evaluator` wraps judgecore's k-sample executor, so a k>1 tuple would grade with one sample while B.2 was told the row carries a vote-margin feature. `shadow.duplicate.self_consistency_k` is a different leaf on a different plane and does compose. |
| `rubric` | object | iff `pointwise-rubric` | `{id: <rubric-id>}`. Required when `method: pointwise-rubric` and refused on every other method, both directions. `id` must be non-empty; **membership is not resolved on any path a config file reaches**, because the only method that reads a rubric is itself refused over this transport. |
| `panel` | object | iff `panel` | The panel composite. Required when `method: panel` and refused on every other method, both directions. See [panel](#evalsevaluatorsjudgepanel). |

**Two tuples compose from a config file, and the rest are refused twice.** `pointwise-bool`, which
is the absent-reading default, and `panel` at `self_consistency_k: 1`. `pointwise-rubric`,
`pairwise` and `reference` are refused over a `judge:` (provider API) transport at
`tokentriage validate` and again at boot, by name; the message says that nothing in
`internal/evaluator` drives them there, and that the composition site will not quietly grade
`pointwise-bool` instead.

#### evals.evaluators[].judge.panel

A panel is **one instrument**, not several: its `MethodVersion` hashes the member tuple, so B.7
validates and promotes the committee rather than its seats. It votes with all its members or it does
not vote, so a member that cannot be reached refuses the whole invocation.

| Field | Type | Required | Meaning |
|---|---|---|---|
| `members` | array | **yes** | The panel's transports. `>= 3` (`EvalsPanelMinMembers`) from `>= 2` distinct model families (`EvalsPanelMinFamilies`). **There is no maximum.** The floors are not restated at boot: `judgecore.NewPanel` is dry-built over name-only seats, so `validate` and boot cannot disagree about them. |
| `rule` | string | — | The aggregation rule. Absent reads as **`majority`** (`EvalsPanelRuleMajority`); present must equal it, because the vocabulary is closed at one value today. |

#### evals.evaluators[].judge.panel.members[]

Each member is **exactly one** transport kind. A member naming both is two scorers in one seat; a
member naming neither is an empty chair. Both are refused.

| Field | Type | Required | Meaning |
|---|---|---|---|
| `model` | string | with `api_key_env` | The provider-API seat's model id. Its family must differ from every other seat's and from **every configured tier's**, since every tier is a possible generator. A key with no model authenticates nothing, so the pair is required together. |
| `api_key_env` | string | with `model` | **Names** this seat's key variable. One seat, one credential, one destination. |
| `endpoint_env` | string | — | This seat's optional transport override. Absent ⇒ the shipped Anthropic Messages endpoint. Refused on a `sidecar_endpoint_env` member, which already names its endpoint. |
| `sidecar_endpoint_env` | string | the other kind | **Names** a §2.4 sidecar seat's base URL. A literal URL is refused. **This build refuses such a seat at construction**: a seat votes through `judgecore.Transport.GradeText`, and the sidecar client declines `GradeText`, so a panel seated with one would turn away every row it judged. Use provider members for every seat. |

### evals.evaluators[].sidecar

An operator-run scorer service (e.g. an NLI/HHEM endpoint). It POSTs captured request and
response bodies to the service, so it is egress and is gated like the judge. The block validates in
full and reaches no composition site: `tokentriage validate` and boot both stop it at
`evaluator.NewScorer`, on the ground that resolving the endpoint variable there would be a second
configuration path skipping the `external: allow` gate. That is why the worked example below
declares no `sidecar:` entry.

| Field | Type | Required | Meaning |
|---|---|---|---|
| `external` | string | **yes** | The air-gap gate. Must be the literal `allow` or the entry is refused. |
| `endpoint_env` | string | **yes** | **Names** the environment variable holding the sidecar base URL (e.g. `HHEM_ENDPOINT`). A literal URL is refused at config validation, because an endpoint routinely carries its own authorization in host or path. |

### evals.evaluators[].sidecar_judge

A judge served over the §2.4 sidecar wire: a sibling of the NLI sidecar contract, inheriting its
discipline. The whole block validates in full and is then refused, in its own words, as
*config-complete and composition-incomplete*. `tokentriage validate` stops it and so does boot,
naming `evaluator.NewSidecarJudgeScorer` as the constructor the composition site would have to call.
Declaring it beside a `judge:` block on one entry is refused separately.

| Field | Type | Required | Meaning |
|---|---|---|---|
| `external` | string | **yes** | The third air-gap gate, the same literal `allow`. A loopback address is still egress: this client POSTs captured request and response bodies to the endpoint. |
| `endpoint_env` | string | **yes** | **Names** the environment variable holding the sidecar judge's base URL. A literal URL is refused. |
| `method` | string | **yes** | The method served over the wire, from the closed set `pointwise-rubric` \| `pairwise` \| `reference`. `pointwise-bool` never crosses this wire (there is no boolean item shape) and a panel is a composite over transports rather than a transport. |
| `rubric` | object | iff `pointwise-rubric` | `{id: <rubric-id>}`, under the same both-directions coherence rule as the judge's. |

<Callout type="note">
**The literal `external: allow` is the egress opt-in**, on `judge:`, `sidecar:` and
`sidecar_judge:` alike. It is the same literal the four other air-gap gates carry
(`cache.store.l3`, `cache.embedders[]`, `shared_state.{redis,postgres}`, `analytics.export.sink`),
so a reviewer grepping `external:` enumerates every egress path in the file. Air-gapped deployments
omit it, and the evaluator is refused. Every credential and endpoint field on these blocks is an
env-var **name, never a value**.
</Callout>

## evals.validation

The judge-validation rules (B.7, DQ16). Four leaves plus a scheduling block. Every one of them is
restart-required, because a judge-tuple change is a B.7 version boundary.

| Field | Type | Default | Meaning |
|---|---|---|---|
| `require_iaa` | bool | `false` | A **raise-only** knob. `false` admits a single-annotator gold set *with its disclosure* (`iaa: null`, unmeasured). `true` upgrades that disclosure to a refusal: such a gold set then cannot license judge promotion. Neither value weakens the κ floor that already applies wherever agreement is measured. |
| `gold_sources` | string[] | `[outcomes]` | Where gold labels may come from. Closed vocabulary `outcomes` \| `feedback`, no duplicates, and an out-of-vocabulary entry is refused without your value being repeated back. `feedback` admits only metrics whose rubric **is** the binary satisfaction question, disclosed through the mapping id every gold-set revision carries. |
| `min_kappa` | float | `0.5` (`EvalsMinKappaFloor`) | **Raise-only** over B.7's frozen promotion floor, which this constant mirrors from `evaluator.JudgeKappaFloor`. Absent ⇒ the frozen floor applies unchanged. A value **below** the floor is refused, naming the floor and your value; a value **above 1** is refused too, because Cohen's κ never exceeds 1 and such a bar could promote nothing. Every validation record prints the floor its own run applied. |
| `selective_risk` | float | `0.05` (`EvalsSelectiveRiskDefault`) | The selective-judging risk α: the accept threshold is chosen so the exact upper bound on error-among-accepted is `<= α`, holdout-verified. Must be **strictly** between `0` and `1`; both endpoints are refused, since α = 0 accepts nothing and α = 1 certifies nothing. |
| `auto_validate` | object | the loop off | Schedules the runtime B.7 loop. Its zero value is the whole loop off. See [auto_validate](#evalsvalidationauto_validate). |

### evals.validation.auto_validate

The loop is edge-triggered on a config change and on new admitted labels, with a periodic sweep
underneath. With it off, judges stay `reported` unless an operator asks for a run with
[`tokentriage evals validate`](/docs/reference/cli#evals).

| Field | Type | Default | Meaning |
|---|---|---|---|
| `enabled` | bool | `false` | Gates the whole loop. Off, no validation run is ever scheduled. The API's affirmative negative (`auto_validate: false` beside `subjects_observed: N`) is what makes the off state readable rather than an empty list. |
| `interval` | duration | `24h` (`EvalsAutoValidateIntervalDefault`) | The periodic sweep cadence. A negative value is refused. `0s` and an omitted key are the same value (the type is not a pointer) and both read as the default sweep. Edge triggers are free and unaffected by this leaf. |
| `new_labels_threshold` | int | `25` (`EvalsAutoValidateLabelsDefault`) | Re-run a subject when this many **new admitted** labels arrive for it. Present ⇒ `>= 1`. |

## evals.annotation_queue

| Field | Type | Default | Meaning |
|---|---|---|---|
| `enabled` | bool | `false` | Serves the annotation queue, a curation front-end over the **existing** `POST /outcomes` write path rather than a second one. Off, `POST /tt/api/v1/evals/annotation-queue/claim` answers `feature_absent` and names the leaf. Turning it on is not sufficient on its own: a claim serves the row's stored prompt and completion verbatim, so it also needs body capture and an admin-role credential. |

## evals.calibration.quality

Configures the quality-calibration fitter (§2.4) that turns raw evaluator scores into
probabilities whose error is measured on a held-out slice. See
[How calibration works](/docs/calibration/how-calibration-works).

| Field | Type | Required | Meaning |
|---|---|---|---|
| `enabled` | bool | — | Gates fitting. `false` ⇒ no artifact is written and no calibrated claim can be emitted at all. |
| `confidence` | float | **when `enabled`** | The nominal coverage level 1−α of the intervals the fitter produces (`0.90` ⇒ α = 0.10). **No default**: it *is* the claim, and this package will not choose an operator's α. Must be strictly between `0` and `1` whenever present. |
| `min_n` | int | — | The per-segment refuse-below-N floor. **Raise-only**: absent ⇒ the named floor **`50`** (`calib.DefaultMinSegmentN`) applies; a configured value **below** `50` is refused. It floors the *calibration split*, so a segment needs ≈ 4× this many fit-eligible rows (~200) before any claim is licensable. |
| `refit_interval` | duration | — | The **scheduled** refit cadence (e.g. `24h`). Omit (or `0s`) ⇒ no periodic refit; the drift, policy-epoch, and staleness triggers still fire. A negative value is refused. |

<Callout type="warning">
**Do not confuse this with `tokentriage calibrate`.** `evals.calibration.quality` configures
**probability calibration**, the `qcal_` machinery that gates quality-aware routing. The
threshold `calibrate` command, which routes N% of traffic to the frontier tier, writes a
*different* artifact kind, documented in
[Routing](/docs/build/routing#calibrate-from-your-own-logs) and the
[CLI reference](/docs/reference/cli). Same lifecycle chassis, different claim.
</Callout>

## evals.drift

| Field | Type | Default | Meaning |
|---|---|---|---|
| `detectors` | string[] | `[]` | **Registered** drift-detector ids (e.g. `robust-seasonal-v1`, `psi-window-v1`), each non-empty and unique. Config checks **shape only**; membership is resolved on the read path, where an id the registry does not carry is dropped from the scan and the response publishes the detector list it actually ran. **Empty ⇒ drift is UNMEASURED**, not "no drift": the drift-clean precondition of a calibrated claim (DQ4) is then satisfied *vacuously* rather than checked. Configuring an empty list while `calibration.quality.enabled` is on emits a validation warning. |

## The dashboard.capture dependency

`evals run` and sidecar evaluators re-score **stored** request/response bodies. Capture defaults
to **hash-only**, so a default deployment's `evals run` finds no bodies and returns a *counted
refusal* naming `config.DashboardCapture.response`, which is a documented default state rather than
an error. Turn on capture to make eval runs executable:

```yaml
dashboard:
  enabled: true
  capture:
    request: true      # default false (hash-only)
    response: true     # default false (hash-only)
    ttl: 720h          # default 720h; 0 ⇒ until retention sweeps
    path: ~/.tokentriage/content   # default ~/.tokentriage/content
```

| Field | Type | Default | Meaning |
|---|---|---|---|
| `request` | bool | `false` | Capture request bodies to the content store. |
| `response` | bool | `false` | Capture response bodies, the ones an eval run re-scores. |
| `ttl` | duration | `720h` | Retention of captured bodies. `0` ⇒ kept until retention sweeps. |
| `path` | string | `~/.tokentriage/content` | Content-store directory. |

One neighbouring leaf is worth knowing about here: `dashboard.state_dir` is **defaulted at decode**
to `~/.tokentriage/dash` (`internal/config/dashboard.go:22`), so deleting it from a file does not
leave the judge runtime without a state directory. The B.7 promotion registry opens under it.

## A worked example

The single-judge shape, with the ingestion door, the fitter and the validation block:

```yaml
evals:
  outcomes:
    api_enabled: true        # read at config load; the door is dashboard.enabled
    otel_ingest: false       # POST /tt/api/v1/outcomes/otel (basis: reported)
  annotation_queue:
    enabled: true            # needs capture + an admin-role credential besides
  evaluators:
    - id: code-output-nonempty     # CODE-kind: local, free — only id + sample_rate
      sample_rate: 1.0
    - id: judge-task-completion    # JUDGE (external LLM): metered ⇒ budget mandatory
      judge:
        external: allow
        model: claude-haiku-4-5    # a DIFFERENT family than the gpt tiers judged
        api_key_env: JUDGE_API_KEY
        # method: omitted ⇒ pointwise-bool; self_consistency_k: omitted ⇒ 1
      sample_rate: 0.05
      max_usd_per_day: 5.0
  validation:
    require_iaa: false             # raise-only judge-validation gate
    gold_sources: [outcomes]       # the default reading, written out
    min_kappa: 0.6                 # raise-only over the frozen 0.5
    selective_risk: 0.05           # the default reading, written out
    auto_validate:
      enabled: true
      interval: 24h
      new_labels_threshold: 25
  calibration:
    quality:
      enabled: true
      confidence: 0.90             # REQUIRED when enabled; no default — it IS the claim
      min_n: 50                    # raise-only; 50 is the floor and the default
      refit_interval: 24h          # omit/0s ⇒ no periodic refit
  drift:
    detectors: [robust-seasonal-v1, psi-window-v1]   # empty ⇒ drift UNMEASURED
```

And the panel shape. The top-level `model` and `api_key_env` are **still required** here, even
though the three seats carry their own, and `endpoint_env` is refused at this level:

```yaml
evals:
  evaluators:
    - id: judge-task-completion
      sample_rate: 0.05
      max_usd_per_day: 5.0         # ONE cap for the whole fan-out, not per seat
      judge:
        external: allow
        model: claude-haiku-4-5    # required even on method: panel
        api_key_env: JUDGE_API_KEY # required even on method: panel
        method: panel
        panel:
          rule: majority           # the one legal value; omit for the same reading
          members:                 # >= 3 seats from >= 2 families, no maximum
            - model: claude-haiku-4-5
              api_key_env: ANTHROPIC_API_KEY
            - model: gemini-2.5-flash
              api_key_env: GEMINI_API_KEY
            - model: mistral-large-2411
              api_key_env: MISTRAL_API_KEY
```

Both blocks were loaded with `tokentriage validate` before they were published here. Every seat's
family has to differ from every **configured tier's** family too, so the models above are the ones
that pass beside a `gpt`-family tier pair; swap in whatever differs from yours.
[Configure a judge](/docs/judges/configure-a-judge) walks the same keys as a task, with the
text each wrong value prints. Six end-to-end configurations ship as `examples/evals/journey-{1..6}-*.yaml`,
walked through in `examples/evals/README.md`.

<NextSteps items={[
  { title: "Configure a judge", href: "/docs/judges/configure-a-judge", description: "The same keys as a task, with the refusal each wrong value prints and the boot log that confirms the right one." },
  { title: "Configuration reference", href: "/docs/reference/configuration", description: "The full config schema this block is additive over." },
  { title: "Quality & evals API", href: "/docs/reference/quality-api", description: "The management endpoints these config gates enable." },
  { title: "Calibrate quality from your logs", href: "/docs/calibration/calibrate-from-logs", description: "Fit and activate a quality calibration once the block is configured." }
]} />