Configuration in production

A configuration that is wrong in production fails in one of two places, and they are not equally pleasant. It either fails on your terminal, where you fix it, or it fails at boot, where the pod crash-loops and your traffic goes nowhere.

This page is about pushing every failure into the first category: what to run before you deploy, which checks only the running binary can make, and how to confirm from the outside that the thing which started is the thing you meant to start.

Prerequisites: the tokentriage binary somewhere you can run it: your laptop, a CI job, or the pod itself. Nothing here needs a cluster.

Check it before it ships

tokentriage validate loads a config exactly as boot does and prints what it resolved:

$ tokentriage validate -c tokentriage.yaml
mode:      ledger
listen:    127.0.0.1:8787   admin: 127.0.0.1:9090
upstreams: 1
  - anthropic    anthropic http://127.0.0.1:47811
tiers:     0 (cheapest -> most capable)
rules:     0
OK: configuration is valid

Read the summary as well as the last line. It is the config as the binary understood it, so a typo that lands in the wrong block shows up as a missing upstream or a zero tier count rather than as an error.

Decoding is strict: an unknown key is a failure, never a silent ignore.

$ tokentriage validate -c tokentriage.yaml
Error: configuration is invalid: config: yaml: unmarshal errors:
  line 32: field rotate_mb not found in type config.Config

That was rotate_mb written at the top level instead of under ledger.decision_log. A looser loader would have accepted the file and rotated at the default forever.

Mode-dependent requirements are checked too, which is what catches a half-finished promotion up the trust ladder:

$ tokentriage validate -c tokentriage.yaml
Error: configuration is invalid: config tokentriage.yaml: tiers: at least one tier is required in shadow mode
routing.default_tier: required in shadow mode

Both errors are printed, not only the first. The exit code is 1, so validate && deploy works in a pipeline.

Check the config a chart will actually mount

On Kubernetes the file you wrote is not the file that runs. The chart renders a ConfigMap from your values, injecting the keys it owns. Validate that instead:

$ helm template tt deploy/helm/tokentriage -f my-values.yaml \
  | python -c "import sys,yaml;[print(d['data']['tokentriage.yaml']) for d in yaml.safe_load_all(sys.stdin) if d and d.get('kind')=='ConfigMap']" \
  | tokentriage validate --config -
mode:      ledger
listen:    0.0.0.0:8787   admin: 0.0.0.0:9090
upstreams: 1
  - anthropic    anthropic http://mock-upstream:8080
tiers:     0 (cheapest -> most capable)
rules:     0
OK: configuration is valid

Note what the chart injected: server.listen and server.admin bind 0.0.0.0, because loopback is unreachable from outside a container. That upstream is a local mock, which is why it is a plain service name.

The chart owns a small set of keys: server, shared_state, instances, dashboard.auth and cache.store.l3. It refuses them inside config: with a message naming the typed value to use instead. Everything else in config: passes through verbatim.

What validate does not catch

Validate is necessary and not sufficient, and the gap is worth naming rather than discovering.

Checked by validate	Checked only when the engine starts
strict decoding, unknown keys	descriptor-aware provider checks
required fields per mode	the routing-policy compile matrix
bounds and enum vocabularies	governance and limit registry names
credential fields are env references	detector and evaluator registry names

A green validate means “this will not fail for a structural reason”. tokentriage doctor closes most of the remainder. It runs the same registry compile the boot path runs, so it catches a routing policy that decodes but cannot compile:

$ tokentriage doctor -c tokentriage.yaml
✓ binary — tokentriage dev (none), embedded pricing snapshot litellm@2026-07-14#179affb — clean
✓ config — tokentriage.yaml — strict decode clean: mode ledger, 1 upstream(s), 0 tier(s), 0 rule(s)
✓ compile — provider registry + routing policy compiled — 1 upstream(s) wired — clean
✓ ports — listen 127.0.0.1:8787 bindable, admin 127.0.0.1:9090 bindable — clean
✓ ledger — ./data/decisions.jsonl — not yet created; parent dir exists, created on first request — clean
✓ pricing — embedded table litellm@2026-07-14#179affb loaded, no overrides configured — clean
✓ keys — 1 upstream(s), no *_env credential references (inbound forwarded) — clean

ok — all checks passed (exit 0)

That capture is trimmed and comes from a source build, so a release prints its version in place of dev. Diagnose a running instance covers the full report and the checks that only make sense against something already running.

Read the boot banner

The banner is the cheapest verification surface you have: it is what the process decided, printed once, in the log you already collect.

tokentriage dev · mode: ledger
proxy: http://127.0.0.1:8787   (point your app's base_url here)
dashboard: http://127.0.0.1:9090/ui/
ledger: ./data/decisions.jsonl

Four facts, each of which you can be wrong about: the mode you are in, where traffic should point, whether the dashboard came up, and which file the ledger is being written to. That last line is the one to check after any change to ledger.decision_log.path.

The banner also reports partial failure. Start an instance whose admin port is already taken and it says so, without failing:

dashboard unavailable: listen tcp 127.0.0.1:9090: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. — proxy traffic unaffected (run: tokentriage doctor)
tokentriage dev · mode: ledger
proxy: http://127.0.0.1:8788   (point your app's base_url here)
ledger: ./data/decisions.jsonl

The dashboard: line is gone and the data plane still serves. A request to :8788 returned 200 in that state. That long bind message is Windows’s own wording, quoted as captured; Linux says address already in use in the same position.

Zero-config quickstarts turn things on that a file does not

The quickstart flags exist so a first run needs no file, and they are not equivalent to an empty config:

$ tokentriage run --anthropic
$ tokentriage run --simple ollama/llama3.1:8b --complex anthropic/claude-sonnet-4-6

The difference that matters in production is the dashboard. The binary’s own default is dashboard off; a quickstart turns it on (add --no-dashboard to opt out), and the Helm chart also ships it on and secured. A config file you wrote yourself has it off unless you wrote dashboard.enabled: true.

The two are not the same thing, though. A quickstart’s dashboard binds loopback with auth off, which is right for a laptop and wrong for anything else; the chart flips it to authMode: token and refuses to express an open one. Do not carry a quickstart into production and assume it arrived secured.

That is not only a UI question. The retention sweeper is started as part of the dashboard composition, so a headless deployment rotates the ledger but never prunes it. The decision ledger on disk covers the consequence.

Changing config on a running deployment

Where the config file is writable, on a host install or a Compose bind-mount, the admin plane exposes the same config as a contract: POST /config/validate on bytes you send, POST /config/diff against what is running, GET /config/running for the file verbatim with its generation, and POST /config/apply behind a compare-and-swap on that generation.

A change set containing anything restart-required is refused with 422 restart_required before any write, naming the exact paths. Nothing is written and traffic is untouched, so the decision to restart stays yours.

On Kubernetes the mount is read-only and every deployment-critical key is restart-required, so the answer is always the same: change the value and helm upgrade. Upgrade and roll back is that procedure.

Where the config lives on each deployment shape

Shape	Path	Writable	How you change it
host binary	wherever --config points	yes	edit the file, or the admin config API
Compose	/etc/tokentriage/tokentriage.yaml, a bind-mount	as mounted	edit the mounted file and restart
Kubernetes	/etc/tokentriage/tokentriage.yaml, a ConfigMap projection	no	edit values, helm upgrade

The container images bake a config at that path so docker run works with no mount; a real deployment mounts over it. The full key reference is the configuration reference.