Evals & calibration configuration

The evals: block (internal/config/evals.go, §2.11) configures calibration & evals: outcome ingestion, the declared evaluator instances, the judge-validation rules, quality calibration, the annotation queue, and the drift detectors. It is additive over every prior schema: an absent block compiles byte-identically to a pre-feature config.

This block fills no defaults. ApplyDefaults leaves an Evals value byte-identical to what was decoded. The Default column below names the value the consuming code applies to an absent leaf, and the constant that spells it; nothing is written back into the decoded config. The leaves are the parameters of a statistical claim (confidence is the claim, not an operational knob), so a leaf a claim depends on is required when its block is on, and a leaf whose absence has a true reading (min_n ⇒ the named floor; refit_interval ⇒ no scheduled refit) is left absent and read that way.

evals.outcomes

The two ingestion doors, and they do not behave alike. Both default to false, and both are refused at config load when dashboard.enabled is false, because the management API that serves them is mounted only when the dashboard is on. Only otel_ingest is then consulted again when a request arrives.

Field	Type	Default	Meaning
api_enabled	bool	false	Your declaration that typed batch ingestion on POST /tt/api/v1/outcomes is wanted. It is read by the config loader and by nothing else (internal/config/evals.go:899), where it catches one combination: the flag set against a disabled dashboard. The handler is registered unconditionally (internal/api/dashsrv/outcomes.go:118-122); what actually decides whether a call lands is the mounted dashboard, the management API’s own role and scope check, and a wired decision-log sink. Measured on this build: with the whole evals.outcomes block deleted, POST /tt/api/v1/outcomes answered {"accepted":1,"refused":0} at HTTP 200. Keep it set, because it is what a reviewer of your config reads; do not reach for it as an off switch.
otel_ingest	bool	false	Serves POST /tt/api/v1/outcomes/otel (OTLP/HTTP JSON gen_ai.evaluation.result). Consulted again at request time (internal/api/dashsrv/outcomes_otel.go:441) before a byte of the body is read, which is the difference between this row and the one above it: off, the route answers feature_absent and names the leaf. Outcomes arriving this way are basis: reported by definition, so enabling it widens what may be recorded, not what may be claimed.

evals.evaluators[]

The declared scorer instances, in declaration order. Each names a registered evaluator id (§2.3) and carries its own sampling and budget discipline. An instance is one scorer: declaring two transport blocks under one id is refused.

Field	Type	Required	Meaning
id	string	yes	Names a registered evaluator descriptor. Unique within the list.
sample_rate	float	yes	Fraction of eligible requests this evaluator scores, in [0,1]. Not defaulted: 0 (never) and 1 (every eligible request) are both plausible readings of an omission, so it is refused rather than guessed. 0 is legal and means “declared but not sampling”.
max_usd_per_day	float	when judge: present	Per-evaluator daily spend cap (USD). Mandatory whenever judge: is present; optional otherwise. Must be finite and >= 0 (0 stops the evaluator at its first priced call; .inf is refused, since an infinite cap is not a cap). Enforced post-hoc. On a panel it caps the whole fan-out, not one seat.
judge	object	—	Configures the LLM-judge kind. Its presence is what makes the instance metered. See judge.
sidecar	object	—	Configures the operator-run scorer-service client. See sidecar.
sidecar_judge	object	—	Configures a judge served over the §2.4 sidecar wire. Config-complete and composition-incomplete: see sidecar_judge.

evals.evaluators[].judge

An LLM judge phones out on every sampled request. Its outcomes are basis: reported until the B.7 validation protocol promotes them. The first three leaves are required on every judge block, including one that sets method: panel, where the top-level model and api_key_env are still refused when absent even though the seats carry their own.

Field	Type	Required	Meaning
external	string	yes	The air-gap gate (SP-16). Must be the literal allow or the whole evaluator entry is refused at config validation. Any other value, including a typo or true, leaves the gate closed. The comparison is != "allow", never == "".
model	string	yes	The judge model id sent to the provider. An empty value is refused; so is the literal oracle (the shadow block’s grading-path selector, and sending it as a model id is a recorded wiring bug). Use a different model family than the tiers being judged: B.7 refuses a same-family judge, and on a panel that refusal extends to every seat against every configured tier.
api_key_env	string	yes	Names the environment variable holding the judge provider’s API key (uppercase, e.g. ANTHROPIC_API_KEY). It must never contain the key: the running config is served verbatim to any principal holding config:read.
endpoint_env	string	—	Names the environment variable holding the URL the judge POSTs to. Absent ⇒ the shipped Anthropic Messages endpoint, so a config without this leaf composes the client it composed before the leaf existed. A literal URL is refused at validation. The api_key_env key travels to whatever this resolves to, as a request header. Refused beside method: panel, where the seats are the transports.
method	string	—	The §2.1 judging method. Absent reads as pointwise-bool (EvalsJudgeMethodDefault), the shipped grader. The closed vocabulary is pointwise-bool | pointwise-rubric | pairwise | reference | panel; anything else is refused by config, which does not repeat your value back. See Judge methods for which of the five compose here.
self_consistency_k	int	—	The self-consistency sample count. Absent reads as 1 (EvalsSelfConsistencyKDefault), a single sample with no vote-margin feature. Present ⇒ >= 1. A value above 1 is refused on this leaf by tokentriage validate and again at boot: no scorer in internal/evaluator wraps judgecore’s k-sample executor, so a k>1 tuple would grade with one sample while B.2 was told the row carries a vote-margin feature. shadow.duplicate.self_consistency_k is a different leaf on a different plane and does compose.
rubric	object	iff pointwise-rubric	{id: <rubric-id>}. Required when method: pointwise-rubric and refused on every other method, both directions. id must be non-empty; membership is not resolved on any path a config file reaches, because the only method that reads a rubric is itself refused over this transport.
panel	object	iff panel	The panel composite. Required when method: panel and refused on every other method, both directions. See panel.

Two tuples compose from a config file, and the rest are refused twice. pointwise-bool, which is the absent-reading default, and panel at self_consistency_k: 1. pointwise-rubric, pairwise and reference are refused over a judge: (provider API) transport at tokentriage validate and again at boot, by name; the message says that nothing in internal/evaluator drives them there, and that the composition site will not quietly grade pointwise-bool instead.

evals.evaluators[].judge.panel

A panel is one instrument, not several: its MethodVersion hashes the member tuple, so B.7 validates and promotes the committee rather than its seats. It votes with all its members or it does not vote, so a member that cannot be reached refuses the whole invocation.

Field	Type	Required	Meaning
members	array	yes	The panel’s transports. >= 3 (EvalsPanelMinMembers) from >= 2 distinct model families (EvalsPanelMinFamilies). There is no maximum. The floors are not restated at boot: judgecore.NewPanel is dry-built over name-only seats, so validate and boot cannot disagree about them.
rule	string	—	The aggregation rule. Absent reads as majority (EvalsPanelRuleMajority); present must equal it, because the vocabulary is closed at one value today.

evals.evaluators[].judge.panel.members[]

Each member is exactly one transport kind. A member naming both is two scorers in one seat; a member naming neither is an empty chair. Both are refused.

Field	Type	Required	Meaning
model	string	with api_key_env	The provider-API seat’s model id. Its family must differ from every other seat’s and from every configured tier’s, since every tier is a possible generator. A key with no model authenticates nothing, so the pair is required together.
api_key_env	string	with model	Names this seat’s key variable. One seat, one credential, one destination.
endpoint_env	string	—	This seat’s optional transport override. Absent ⇒ the shipped Anthropic Messages endpoint. Refused on a sidecar_endpoint_env member, which already names its endpoint.
sidecar_endpoint_env	string	the other kind	Names a §2.4 sidecar seat’s base URL. A literal URL is refused. This build refuses such a seat at construction: a seat votes through judgecore.Transport.GradeText, and the sidecar client declines GradeText, so a panel seated with one would turn away every row it judged. Use provider members for every seat.

evals.evaluators[].sidecar

An operator-run scorer service (e.g. an NLI/HHEM endpoint). It POSTs captured request and response bodies to the service, so it is egress and is gated like the judge. The block validates in full and reaches no composition site: tokentriage validate and boot both stop it at evaluator.NewScorer, on the ground that resolving the endpoint variable there would be a second configuration path skipping the external: allow gate. That is why the worked example below declares no sidecar: entry.

Field	Type	Required	Meaning
external	string	yes	The air-gap gate. Must be the literal allow or the entry is refused.
endpoint_env	string	yes	Names the environment variable holding the sidecar base URL (e.g. HHEM_ENDPOINT). A literal URL is refused at config validation, because an endpoint routinely carries its own authorization in host or path.

evals.evaluators[].sidecar_judge

A judge served over the §2.4 sidecar wire: a sibling of the NLI sidecar contract, inheriting its discipline. The whole block validates in full and is then refused, in its own words, as config-complete and composition-incomplete. tokentriage validate stops it and so does boot, naming evaluator.NewSidecarJudgeScorer as the constructor the composition site would have to call. Declaring it beside a judge: block on one entry is refused separately.

Field	Type	Required	Meaning
external	string	yes	The third air-gap gate, the same literal allow. A loopback address is still egress: this client POSTs captured request and response bodies to the endpoint.
endpoint_env	string	yes	Names the environment variable holding the sidecar judge’s base URL. A literal URL is refused.
method	string	yes	The method served over the wire, from the closed set pointwise-rubric | pairwise | reference. pointwise-bool never crosses this wire (there is no boolean item shape) and a panel is a composite over transports rather than a transport.
rubric	object	iff pointwise-rubric	{id: <rubric-id>}, under the same both-directions coherence rule as the judge’s.

evals.validation

The judge-validation rules (B.7, DQ16). Four leaves plus a scheduling block. Every one of them is restart-required, because a judge-tuple change is a B.7 version boundary.

Field	Type	Default	Meaning
require_iaa	bool	false	A raise-only knob. false admits a single-annotator gold set with its disclosure (iaa: null, unmeasured). true upgrades that disclosure to a refusal: such a gold set then cannot license judge promotion. Neither value weakens the κ floor that already applies wherever agreement is measured.
gold_sources	string[]	[outcomes]	Where gold labels may come from. Closed vocabulary outcomes | feedback, no duplicates, and an out-of-vocabulary entry is refused without your value being repeated back. feedback admits only metrics whose rubric is the binary satisfaction question, disclosed through the mapping id every gold-set revision carries.
min_kappa	float	0.5 (EvalsMinKappaFloor)	Raise-only over B.7’s frozen promotion floor, which this constant mirrors from evaluator.JudgeKappaFloor. Absent ⇒ the frozen floor applies unchanged. A value below the floor is refused, naming the floor and your value; a value above 1 is refused too, because Cohen’s κ never exceeds 1 and such a bar could promote nothing. Every validation record prints the floor its own run applied.
selective_risk	float	0.05 (EvalsSelectiveRiskDefault)	The selective-judging risk α: the accept threshold is chosen so the exact upper bound on error-among-accepted is <= α, holdout-verified. Must be strictly between 0 and 1; both endpoints are refused, since α = 0 accepts nothing and α = 1 certifies nothing.
auto_validate	object	the loop off	Schedules the runtime B.7 loop. Its zero value is the whole loop off. See auto_validate.

evals.validation.auto_validate

The loop is edge-triggered on a config change and on new admitted labels, with a periodic sweep underneath. With it off, judges stay reported unless an operator asks for a run with tokentriage evals validate.

Field	Type	Default	Meaning
enabled	bool	false	Gates the whole loop. Off, no validation run is ever scheduled. The API’s affirmative negative (auto_validate: false beside subjects_observed: N) is what makes the off state readable rather than an empty list.
interval	duration	24h (EvalsAutoValidateIntervalDefault)	The periodic sweep cadence. A negative value is refused. 0s and an omitted key are the same value (the type is not a pointer) and both read as the default sweep. Edge triggers are free and unaffected by this leaf.
new_labels_threshold	int	25 (EvalsAutoValidateLabelsDefault)	Re-run a subject when this many new admitted labels arrive for it. Present ⇒ >= 1.

evals.annotation_queue

Field	Type	Default	Meaning
enabled	bool	false	Serves the annotation queue, a curation front-end over the existing POST /outcomes write path rather than a second one. Off, POST /tt/api/v1/evals/annotation-queue/claim answers feature_absent and names the leaf. Turning it on is not sufficient on its own: a claim serves the row’s stored prompt and completion verbatim, so it also needs body capture and an admin-role credential.

evals.calibration.quality

Configures the quality-calibration fitter (§2.4) that turns raw evaluator scores into probabilities whose error is measured on a held-out slice. See How calibration works.

Field	Type	Required	Meaning
enabled	bool	—	Gates fitting. false ⇒ no artifact is written and no calibrated claim can be emitted at all.
confidence	float	when enabled	The nominal coverage level 1−α of the intervals the fitter produces (0.90 ⇒ α = 0.10). No default: it is the claim, and this package will not choose an operator’s α. Must be strictly between 0 and 1 whenever present.
min_n	int	—	The per-segment refuse-below-N floor. Raise-only: absent ⇒ the named floor 50 (calib.DefaultMinSegmentN) applies; a configured value below 50 is refused. It floors the calibration split, so a segment needs ≈ 4× this many fit-eligible rows (~200) before any claim is licensable.
refit_interval	duration	—	The scheduled refit cadence (e.g. 24h). Omit (or 0s) ⇒ no periodic refit; the drift, policy-epoch, and staleness triggers still fire. A negative value is refused.

evals.drift

Field	Type	Default	Meaning
detectors	string[]	[]	Registered drift-detector ids (e.g. robust-seasonal-v1, psi-window-v1), each non-empty and unique. Config checks shape only; membership is resolved on the read path, where an id the registry does not carry is dropped from the scan and the response publishes the detector list it actually ran. Empty ⇒ drift is UNMEASURED, not “no drift”: the drift-clean precondition of a calibrated claim (DQ4) is then satisfied vacuously rather than checked. Configuring an empty list while calibration.quality.enabled is on emits a validation warning.

The dashboard.capture dependency

evals run and sidecar evaluators re-score stored request/response bodies. Capture defaults to hash-only, so a default deployment’s evals run finds no bodies and returns a counted refusal naming config.DashboardCapture.response, which is a documented default state rather than an error. Turn on capture to make eval runs executable:

dashboard:
  enabled: true
  capture:
    request: true      # default false (hash-only)
    response: true     # default false (hash-only)
    ttl: 720h          # default 720h; 0 ⇒ until retention sweeps
    path: ~/.tokentriage/content   # default ~/.tokentriage/content

Field	Type	Default	Meaning
request	bool	false	Capture request bodies to the content store.
response	bool	false	Capture response bodies, the ones an eval run re-scores.
ttl	duration	720h	Retention of captured bodies. 0 ⇒ kept until retention sweeps.
path	string	~/.tokentriage/content	Content-store directory.

One neighbouring leaf is worth knowing about here: dashboard.state_dir is defaulted at decode to ~/.tokentriage/dash (internal/config/dashboard.go:22), so deleting it from a file does not leave the judge runtime without a state directory. The B.7 promotion registry opens under it.

A worked example

The single-judge shape, with the ingestion door, the fitter and the validation block:

evals:
  outcomes:
    api_enabled: true        # read at config load; the door is dashboard.enabled
    otel_ingest: false       # POST /tt/api/v1/outcomes/otel (basis: reported)
  annotation_queue:
    enabled: true            # needs capture + an admin-role credential besides
  evaluators:
    - id: code-output-nonempty     # CODE-kind: local, free — only id + sample_rate
      sample_rate: 1.0
    - id: judge-task-completion    # JUDGE (external LLM): metered ⇒ budget mandatory
      judge:
        external: allow
        model: claude-haiku-4-5    # a DIFFERENT family than the gpt tiers judged
        api_key_env: JUDGE_API_KEY
        # method: omitted ⇒ pointwise-bool; self_consistency_k: omitted ⇒ 1
      sample_rate: 0.05
      max_usd_per_day: 5.0
  validation:
    require_iaa: false             # raise-only judge-validation gate
    gold_sources: [outcomes]       # the default reading, written out
    min_kappa: 0.6                 # raise-only over the frozen 0.5
    selective_risk: 0.05           # the default reading, written out
    auto_validate:
      enabled: true
      interval: 24h
      new_labels_threshold: 25
  calibration:
    quality:
      enabled: true
      confidence: 0.90             # REQUIRED when enabled; no default — it IS the claim
      min_n: 50                    # raise-only; 50 is the floor and the default
      refit_interval: 24h          # omit/0s ⇒ no periodic refit
  drift:
    detectors: [robust-seasonal-v1, psi-window-v1]   # empty ⇒ drift UNMEASURED

And the panel shape. The top-level model and api_key_env are still required here, even though the three seats carry their own, and endpoint_env is refused at this level:

evals:
  evaluators:
    - id: judge-task-completion
      sample_rate: 0.05
      max_usd_per_day: 5.0         # ONE cap for the whole fan-out, not per seat
      judge:
        external: allow
        model: claude-haiku-4-5    # required even on method: panel
        api_key_env: JUDGE_API_KEY # required even on method: panel
        method: panel
        panel:
          rule: majority           # the one legal value; omit for the same reading
          members:                 # >= 3 seats from >= 2 families, no maximum
            - model: claude-haiku-4-5
              api_key_env: ANTHROPIC_API_KEY
            - model: gemini-2.5-flash
              api_key_env: GEMINI_API_KEY
            - model: mistral-large-2411
              api_key_env: MISTRAL_API_KEY

Both blocks were loaded with tokentriage validate before they were published here. Every seat’s family has to differ from every configured tier’s family too, so the models above are the ones that pass beside a gpt-family tier pair; swap in whatever differs from yours. Configure a judge walks the same keys as a task, with the text each wrong value prints. Six end-to-end configurations ship as examples/evals/journey-{1..6}-*.yaml, walked through in examples/evals/README.md.