Runbooks

You arrive here with a symptom, not a subject. This page is indexed by what you are seeing, and every entry ends in a command or a config key rather than an explanation.

Work through an entry in the same order each time: confirm the cause with a check, apply the fix, then confirm the symptom is gone. Where the correct action is to do nothing, that is stated as the fix, because several failures here are self-healing and a restart makes them worse.

Find your symptom

What you are seeing	Go to
the pod is crash-looping after a config change	a pod that will not start
the volume is filling up	the ledger disk is filling
totals look too low, or declog_dropped_total moved	the ledger is losing rows
the dashboard shows far fewer rows than the CLI	the dashboard lost history
requests are returning 429	requests are being turned away
responses carry x-tt-coordination: suspended	a coordinated limit degraded
the provider is failing and traffic is not moving	an upstream is down
the dashboard is unreachable but traffic is fine	the admin plane did not come up
an unfamiliar model is missing from spend	a model is unpriced
a replica vanished from the fleet count	a replica left the roster

What should page you

The Helm chart ships three alert rules, rendered when metrics.prometheusRule.enabled is set and the Prometheus Operator’s CRD is present. Each one watches a place where the product would otherwise be quietly wrong rather than loudly broken:

Alert	Expression	Severity	Runbook
TokenTriageDeclogDrops	rate(tokentriage_declog_dropped_total[5m]) > 0	warning	the ledger is losing rows
TokenTriagePricingMisses	sum(rate(tokentriage_pricing_miss_total[15m])) by (model) > 0	warning	a model is unpriced
TokenTriageEnforcementSuspended	max by (limit_id) (tokentriage_governance_limit_suspended) > 0	critical	a coordinated limit degraded

The drop rule fires on any drop, because the correct number of dropped rows is zero.

Alert on those symptoms rather than on causes underneath them. A Redis outage is not worth paging for on its own, because coordination is fail-open, traffic keeps flowing, and the store self-heals; the suspension gauge is what tells you a guarantee is not being held.

A pod that will not start

Confirm. The config is validated by the binary at boot, not by Helm at render, so a config that installs cleanly can still abort. Read the logs:

$ kubectl logs -n tokentriage tt-tokentriage-0 --previous

A configuration fault prints the validator’s own message. A partly-finished promotion up the trust ladder is the common one:

Error: configuration is invalid: config tokentriage.yaml: tiers: at least one tier is required in shadow mode
routing.default_tier: required in shadow mode

Fix. Roll back to the last good revision, then fix the values and validate before trying again:

$ helm rollback tt -n tokentriage
$ helm template tt deploy/helm/tokentriage -f my-values.yaml \
  | python -c "import sys,yaml;[print(d['data']['tokentriage.yaml']) for d in yaml.safe_load_all(sys.stdin) if d and d.get('kind')=='ConfigMap']" \
  | tokentriage validate --config -

Confirm the fix. kubectl rollout status completes and helm test tt -n tokentriage succeeds. Configuration in production covers what validate does and does not catch.

The ledger disk is filling

Confirm. Check the ledger’s size and what is beside it:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage doctor --config /etc/tokentriage/tokentriage.yaml
✓ ledger — /data/.tokentriage/decisions.jsonl — 902 rows, 0 torn lines, writable — clean

Fix. Three causes, in the order they are worth checking.

Rotation may be off. ledger.decision_log.rotate_mb at 0 or below lets one file grow without bound; the default is 256.

Retention may not be running. The sweeper starts as part of the dashboard composition, so a deployment with dashboard.enabled: false rotates but never prunes. Either enable the dashboard or accept unbounded history and size for it.

Retention may be configured to keep everything. Both retention.max_age and retention.max_total_mb default to 0, which means forever. Set at least one:

retention:
  max_age: 720h
  max_total_mb: 512

The sweep runs hourly and the first one happens an interval after start, so nothing is deleted the moment you apply this.

Confirm the fix. Companions older than the age limit disappear within the hour, and each deletion writes an audit row with its reason. The decision ledger on disk covers the growth arithmetic behind the numbers you pick.

The ledger is losing rows

Confirm. Compare the counter against what was served:

$ curl -s localhost:9090/metrics | grep declog_dropped
tokentriage_declog_dropped_total 1539

Fix. The writer drops when its bounded queue is full, which means the disk is not keeping up rather than that the queue is too small. Check storage first: a stalled volume, a filesystem that has filled, or a burst far beyond what the volume absorbs. Raising ledger.decision_log.queue above its default of 8192 buys a longer burst at the cost of memory and does not help a writer that is permanently slower than the traffic.

Confirm the fix. The counter stops moving. It does not reset, because counters live in the process, so use rate() rather than the raw total.

The dashboard lost history

Confirm. Ask both surfaces for the same total. They read the same ledger and should agree:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- tokentriage top --log /data/.tokentriage/
TOTAL             2303  2760000  805000  $6.785000

$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq '.provenance.rows'
484

Fix. The read-model was rebuilt from the ledger file, so it holds only the active file’s rows and none of the rotated companions. Rebuild it from the directory, then restart the pod so it reopens the rebuilt index:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage index rebuild --index /data/.tokentriage/index/decisions.sqlite \
                              --log /data/.tokentriage/
rebuilt index /data/.tokentriage/index/decisions.sqlite from /data/.tokentriage/
  rows indexed:    2303
$ kubectl delete pod -n tokentriage tt-tokentriage-0

Confirm the fix. The two totals agree. Nothing was lost: the index is derived, and the ledger held every row throughout.

Requests are being turned away

Confirm. A 429 carries a typed reason. Read it from the response body rather than inferring from the status:

$ curl -i -H "Authorization: Bearer $VK" http://localhost:8787/v1/messages -d @body.json

Fix. The reason names which limit stopped the request and at which scope. Why a request was refused is the page for that diagnosis, and it maps every error type to the config key that owns it.

Confirm the fix. The refusal stops, and the limit’s own panel shows headroom again.

A coordinated limit degraded

Confirm. The same fact appears on three surfaces at once: the response header x-tt-coordination: suspended, a suspended:[<limit_id>] entry on the decision row, and a widened bound on the governance health panel:

$ curl -s -H "Authorization: Bearer $TT_TOKEN" \
    localhost:9090/tt/api/v1/governance/health | jq '.redis, .suspensions'

Fix. Nothing destructive. Coordination is fail-open: the data plane kept serving the whole time, the bound widened rather than a number being asserted that could not be backed, and the store is re-adopted by a single half-open probe when it returns.

Confirm the fix. The header stops appearing. Its absence is the healthy default, and there is no positive “ok” value to look for. Governance across replicas covers what each store mode guarantees while degraded.

An upstream is down

Confirm. /health will keep answering 200, because it only reports that the process is up. Ask the surface that knows:

$ curl -s -H "Authorization: Bearer $TT_TOKEN" localhost:9090/tt/api/status | jq '.upstreams'

in_cooldown: true means the upstream has been taken out of rotation after repeated failures. auth_health.state of missing_env means the credential the config names is not present in the environment, which fails in a way that looks like a provider outage and is not.

Fix. For a real outage, cooldown handles rotation and recovery on its own. For a missing credential, supply the environment variable the config names. The config carries the name, not the value. For a provider whose failure should move traffic elsewhere, cascades is the mechanism.

Confirm the fix. in_cooldown returns to false and requests to that upstream succeed.

The admin plane did not come up

Confirm. A bind failure on the admin port does not stop the process, so it runs and the data plane serves. Read the first line of the boot log:

dashboard unavailable: listen tcp 127.0.0.1:9090: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. — proxy traffic unaffected (run: tokentriage doctor)

That wording is Windows’s; Linux says address already in use in the same position. The banner that follows has no dashboard: line.

Fix. Free the port or move the admin bind. On Kubernetes this is rare, because the pod owns its network namespace, and it points at a second container in the pod or a host-network setting.

Confirm the fix. The banner prints its dashboard: line and /metrics answers.

A model is unpriced

Confirm. tokentriage_pricing_miss_total carries the model label, and the rollup discloses the exclusion under the total it is missing from.

Fix. The rows are recorded and counted; only the dollar is absent, and it is left blank rather than filled in as $0.00. Add a pricing entry for the model through the overrides overlay. Where a cost figure comes from covers what the figure excludes and how the disclosure travels with it.

Confirm the fix. The miss counter stops moving for that model and the rollup’s excluded-row count drops.

A replica left the roster

Confirm. Fleet reads carry their coverage: replicas_answered below replicas_total with is_partial: true is a disclosed partial, not a smaller-looking complete total.

$ tokentriage fleet list -c tokentriage.yaml

Fix. A drained or removed replica stays in the coverage denominator until you say otherwise, which is the right choice: a number that silently shrank would be indistinguishable from a number that was always that size. Remove it explicitly:

$ tokentriage fleet decommission <name>

The removal is audited and drops the replica from N.

Confirm the fix. fleet list no longer shows it, and reads stop reporting a partial. Multi-replica & scale-out covers the coverage vocabulary.

Procedures that live elsewhere, and where they went

Several operational tasks are owned by the section that owns their subject, because the procedure is inseparable from the semantics:

Task	Page
back up, restore, rebuild the read-model	back up and restore
upgrade, roll back, change the image	upgrade and roll back
pick store: for a governance limit	enforcement classes
roll out enforcement shadow-first	governance rollout
rotate or revoke a virtual key	virtual keys
promote a route on shadow evidence	your first routing rollout