Evals & calibration configuration

The evals: block (internal/config/evals.go, §2.11) configures the measured-quality substrate: outcome ingestion, the declared evaluator instances, the judge-validation posture, quality calibration, and the drift detectors. It is additive over every prior schema — an absent block compiles byte-identically to a pre-feature config.

This block fills no defaults. ApplyDefaults leaves an Evals value byte-identical to what was decoded. The leaves are the parameters of a statistical claim — confidence is the claim, not an operational knob — so a leaf a claim depends on is required when its block is on, and a leaf whose absence has a true reading (min_n ⇒ the named floor; refit_interval ⇒ no scheduled refit) is left absent and read that way. Nothing is ever filled in behind the operator.

evals.outcomes

The two ingestion doors. Both are off by default (an ingestion endpoint that exists is one that can be written to), and both require dashboard.enabled — the management API that serves them is mounted only when the dashboard is on; a flag set against a disabled dashboard is refused by name.

Field	Type	Default	Meaning
api_enabled	bool	false	Serves POST /tt/api/v1/outcomes (typed batch ingestion). Requires dashboard.enabled.
otel_ingest	bool	false	Serves POST /tt/api/v1/outcomes/otel (OTLP/HTTP JSON gen_ai.evaluation.result). Outcomes arriving this way are basis: reported by definition — enabling it never widens what may be claimed, only what may be recorded. Requires dashboard.enabled.

evals.evaluators[]

The declared scorer instances, in declaration order. Each names a registered evaluator id (§2.3) and carries its own sampling and budget discipline. An instance is one scorer: declaring both a judge: and a sidecar: block under one id is refused.

Field	Type	Required	Meaning
id	string	yes	Names a registered evaluator descriptor. Unique within the list.
sample_rate	float	yes	Fraction of eligible requests this evaluator scores, in [0,1]. Not defaulted — 0 (never) and 1 (every eligible request) are both plausible readings of an omission, so it is refused rather than guessed. 0 is legal and means “declared but not sampling”.
max_usd_per_day	float	when judge: present	Per-evaluator daily spend cap (USD). Mandatory whenever judge: is present; optional otherwise. Must be finite and >= 0 (0 stops the evaluator at its first priced call; .inf is refused — an infinite cap is not a cap). Enforced post-hoc.
judge	object	—	Configures the LLM-judge kind. Its presence is what makes the instance metered. See judge.
sidecar	object	—	Configures the operator-run scorer-service client. See sidecar.

evals.evaluators[].judge

An LLM judge phones out to its provider on every sampled request. Its outcomes are basis: reported until the B.7 validation protocol promotes them.

Field	Type	Required	Meaning
external	string	yes	The air-gap gate (SP-16). Must be the literal allow or the whole evaluator entry is refused at config validation. Any other value — including a typo or true — leaves the gate closed. The comparison is != "allow", never == "".
model	string	yes	The judge model id sent to the provider. An empty value is refused; so is the literal oracle (the shadow block’s grading-path selector — sending it as a model id is a recorded wiring bug). Use a different model family than the tiers being judged (B.7 refuses a same-family judge).
api_key_env	string	yes	Names the environment variable holding the judge provider’s API key (uppercase, e.g. ANTHROPIC_API_KEY). It must never contain the key — the running config is served verbatim to any principal holding config:read.

evals.evaluators[].sidecar

An operator-run scorer service (e.g. an NLI/HHEM endpoint). It POSTs captured request and response bodies to the service, so it is egress and is gated like the judge.

Field	Type	Required	Meaning
external	string	yes	The air-gap gate. Must be the literal allow or the entry is refused.
endpoint_env	string	yes	Names the environment variable holding the sidecar base URL (e.g. HHEM_ENDPOINT). A literal URL is refused at config validation — an endpoint routinely carries its own authorization in host or path.

evals.validation

Field	Type	Default	Meaning
require_iaa	bool	false	A raise-only judge-validation knob (B.7). false admits a single-annotator gold set with its disclosure (iaa: null, unmeasured). true upgrades that disclosure to a refusal: a single-annotator gold set then cannot license judge promotion. Neither value weakens the κ ≥ 0.6 floor that already applies wherever IAA is measured.

evals.calibration.quality

Configures the quality-calibration fitter (§2.4) that turns raw evaluator scores into probabilities whose error is measured on a held-out slice. See How calibration works.

Field	Type	Required	Meaning
enabled	bool	—	Gates fitting. false ⇒ no artifact is written and no calibrated claim can be emitted at all.
confidence	float	when enabled	The nominal coverage level 1−α of the intervals the fitter produces (0.90 ⇒ α = 0.10). No default — it is the claim, and this package will not choose an operator’s α. Must be strictly between 0 and 1 whenever present.
min_n	int	—	The per-segment refuse-below-N floor. Raise-only: absent ⇒ the named floor 50 (calib.DefaultMinSegmentN) applies; a configured value below 50 is refused. It floors the calibration split, so a segment needs ≈ 4× this many fit-eligible rows (~200) before any claim is licensable.
refit_interval	duration	—	The scheduled refit cadence (e.g. 24h). Omit (or 0s) ⇒ no periodic refit; the drift, policy-epoch, and staleness triggers still fire. A negative value is refused.

evals.drift

Field	Type	Default	Meaning
detectors	string[]	[]	Registered drift-detector ids (e.g. robust-seasonal-v1, psi-window-v1), each non-empty and unique. Empty ⇒ drift is UNMEASURED, not “no drift” — the drift-clean precondition of a calibrated claim (DQ4) is then satisfied vacuously rather than checked. Configuring an empty list while calibration.quality.enabled is on emits a validation warning.

The dashboard.capture dependency

evals run and sidecar evaluators re-score stored request/response bodies. Capture defaults to hash-only, so a default deployment’s evals run finds no bodies and returns a counted refusal naming config.DashboardCapture.response — a documented default state, not an error. Turn on capture to make eval runs executable:

dashboard:
  enabled: true
  capture:
    request: true      # default false (hash-only)
    response: true     # default false (hash-only)
    ttl: 720h          # default 720h; 0 ⇒ until retention sweeps
    path: ~/.tokentriage/content   # default ~/.tokentriage/content

Field	Type	Default	Meaning
request	bool	false	Capture request bodies to the content store.
response	bool	false	Capture response bodies — the ones an eval run re-scores.
ttl	duration	720h	Retention of captured bodies. 0 ⇒ kept until retention sweeps.
path	string	~/.tokentriage/content	Content-store directory.

A worked example

evals:
  outcomes:
    api_enabled: true        # POST /tt/api/v1/outcomes
    otel_ingest: false       # POST /tt/api/v1/outcomes/otel (basis: reported)
  evaluators:
    - id: code-output-nonempty     # CODE-kind: local, free — only id + sample_rate
      sample_rate: 1.0
    - id: judge-groundedness       # JUDGE (external LLM): metered ⇒ budget mandatory
      judge:
        external: allow
        model: gemini-2.5-flash    # a DIFFERENT family than the gpt/claude tiers judged
        api_key_env: JUDGE_API_KEY
      sample_rate: 0.05
      max_usd_per_day: 5.0
    - id: nli-groundedness         # SIDECAR (operator-run scorer service)
      sidecar:
        external: allow
        endpoint_env: HHEM_ENDPOINT
      sample_rate: 0.10
  validation:
    require_iaa: false             # raise-only judge-validation gate
  calibration:
    quality:
      enabled: true
      confidence: 0.90             # REQUIRED when enabled; no default — it IS the claim
      min_n: 50                    # raise-only; 50 is the floor and the default
      refit_interval: 24h          # omit/0s ⇒ no periodic refit
  drift:
    detectors: [robust-seasonal-v1, psi-window-v1]   # empty ⇒ drift UNMEASURED

Six end-to-end configurations ship as examples/evals/journey-{1..6}-*.yaml, walked through in examples/evals/README.md.