Runbooks

Step-by-step procedures for the things an operator actually does. Each assumes the Deployment artifacts; the fuller operator docs live in deploy/docs/ and docs/SCALEOUT.md. Where a procedure touches the coordinated fleet it keeps the honest framing: coordination is fail-open and disclosed — a degraded store never blocks traffic, it widens a published bound and stamps it on the wire.

First deploy (Helm)

1. helm install tt deploy/helm/tokentriage --namespace tokentriage --create-namespace -f my-values.yaml
2. Read the post-install NOTES — they resolve the port-forward and the credential one-liner for your release.
3. Port-forward the admin plane and confirm liveness and reachability separately:

   $ curl -s localhost:9090/health        # -> 200, process up
   $ curl -s localhost:9090/tt/api/status # -> engine + provider reachability (never fabricated)

4. Mint the first management token on the host: tokentriage token new --role admin --state-dir <state-dir> (prints a ttk_ bootstrap bearer once).

Upgrade

1. Bump the image and helm upgrade (the StatefulSet rolls one pod at a time; each keeps its own PVC).
2. Config changes go through config-as-contract (validate → diff → apply), so every change has a diff, a generation, and an audit row. See deploy/docs/UPGRADES.md.

Back up / rebuild the ledger

1. Back up ~/.tokentriage/decisions.jsonl (rotated files too) or snapshot the pod’s PVC.
2. To rebuild the derived read-model after a restore or if it’s ever corrupted:

   $ tokentriage index rebuild --log ~/.tokentriage/decisions.jsonl

   It drops and replays the log into a fresh index; the index can be deleted at any time. See deploy/docs/BACKUP_RESTORE.md.

Rotate keys and tokens

1. Rotate a virtual key with a grace window so in-flight callers don’t break: tokentriage keys rotate <KID> --grace 24h (mints a successor, prints it once).
2. Revoke is terminal: tokentriage keys revoke <KID>. Management tokens rotate via tokentriage token revoke <ID> + tokentriage token new. Every mutation is audited.

Revoke a key fleet-wide and prove propagation

Goal: revoke a virtual key on one host and confirm every other replica honours the revoke within the disclosed bound.

Prerequisites: the fleet runs with shared_state.postgres (external: allow) — the Postgres revocation mirror. Without it, keys revoke stays local-only and dials nothing (the air-gap default).

1. Revoke on any host, passing the config so the mutation dual-writes to the mirror:

   $ tokentriage keys revoke vk_abc123 --config /etc/tokentriage/tokentriage.yaml
   revoked vk_abc123

   The command revokes in the local governance.db (immediate, next snapshot swap) and writes the tombstone to the Postgres mirror — a NOTIFY doorbell plus a periodic full-snapshot backstop propagate it to the other replicas.
2. Read the health panel on another replica (port-forward its admin plane) to confirm the PG tier is live and see the propagation cadence it discloses verbatim:

   $ curl -s localhost:9090/tt/api/v1/governance/health | jq '.capability, .postgres'

   {
     "shared_state_enabled": true,
     "tiers": ["postgres"],
     "note": "shared-state tier enabled: postgres (cross-replica key revocation mirror; a revoke propagates within the disclosed bound). No Redis; rate limits + spend are single-node."
   }
   {
     "up": true,
     "replica_ready": true,
     "revocation_bound": "30s",
     "staleness_bound": "1m0s",
     "pending_propagation": 0
   }

   (Illustrative output.) revocation_bound: "30s" is the honest bound — pg.DefaultRevocationBound in internal/shared/pg/replica.go. Every replica re-reads the full snapshot at least once per bound, so the revoke is honored within it even if the NOTIFY doorbell is missed.
3. Present the revoked key to any replica and confirm it is refused (401). A require_key deployment refuses it fleet-wide within the bound.

Diagnose a suspended limit

Goal: a caller (or a decision row) shows x-tt-coordination: suspended — find out which guarantee degraded and confirm it is fail-open, not a silent widening.

The header is a presence flag with the sole value suspended (internal/trace, HeaderCoordination / CoordinationSuspended). Absence is the healthy default — there is no “ok” value, and a single-node / air-gapped instance never emits it.

1. Read the same fact from the three surfaces it discloses on (they agree by construction):
   • The response header — x-tt-coordination: suspended, stamped at every terminal chokepoint (admit and refusal, including the error path).
   • The decision row — suspended:[<limit_id>] names exactly which limit(s) degraded.
   • The /governance/health panel — the widened bound, the breaker state, the reason.
2. Pull the panel and look at suspensions[] and the store tiers:

   $ curl -s localhost:9090/tt/api/v1/governance/health | jq '.redis, .suspensions'

   {
     "reachable": false,
     "state": "open",
     "consecutive_failures": 5
   }
   {
     "limit_id": "org_acme_monthly_budget",
     "class": "spend",
     "reason": "heartbeat_stale",
     "fail_posture": "open",
     "bound": { "approximate": true, "degraded": true }
   }

   (Illustrative output.) Each reason is a typed cause: breaker_open / deadline_exceeded (a store: redis counter fell back to local), heartbeat_stale (the spend accountant is degraded), or replica_not_ready (the PG identity mirror hasn’t seeded).
3. Map the cause to the mechanism:
   • store: redis exact counter — the shared redisx circuit breaker tripped after DefaultFailureThreshold (5) consecutive transport/substrate failures and the counter fell back to per-replica local enforcement.
   • store: share cap — membership decayed to self (N froze), so the ÷N bound widened toward the full per-replica amount (membership_stale, ≤ N×limit).
   • __identity — the Postgres revocation mirror hasn’t seeded; identity resolution applies its declared fail posture (required ⇒ closed/503, optional/off ⇒ open).
4. Do nothing destructive. The breaker auto-recovers: while open it fails fast for DefaultProbeInterval (5 s), then admits exactly one half-open probe whose outcome closes or re-opens it. When the store returns, one probe re-adopts it within the liveness window and all three disclosures clear on their own.

Choose a coordination store per limit class

Goal: pick store: for a governance limit so it enforces correctly across the fleet — and avoid the one combination the validator refuses.

Prerequisites: any non-local store needs the coordination tier — shared_state.redis with external: allow and an addr. Without it, tokentriage validate refuses a store: redis limit (boot would abort with ErrExternalNotAllowed), and a store: share limit is accepted but no-op (N stays 1, so it enforces the full per-replica amount — a warning, never an error).

Limit type (class)	store: local	store: redis	store: share
rpm (admission-counted)	per-replica	exact fleet (server-time, EVALSHA)	global-approx ÷N
tpm (reserve/reconcile)	per-replica	REFUSED at validate	÷N (burst widens — disclosed)
concurrency (in-flight lease)	per-replica	REFUSED at validate	÷N cap ceil(C/N_live)
budget_usd, quota_tokens, quota_requests (ledger-settled)	(no store — converges via the scoped accountant)	error	error

1. rpm → redis for exact fleet-wide rate. The distributed GCRA script is acquire-only, which is exactly right for an admission counter — it’s exact over Redis.
2. tpm → share, never redis. tpm reserves then reconciles (a signed refund/debit at settlement). The distributed Redis Lua is acquire-only and has no reconcile step, so tpm + redis mis-enforces in both directions — the validator refuses it:

   $ tokentriage validate -c tokentriage.yaml
   govern: limit "team_ml_tpm": type "tpm" (reserve/reconcile) cannot bind store "redis" —
   the distributed rate store is acquire-only (the reserve→reconcile settlement is not
   implemented for it), so a per-request token reconcile would mis-enforce in both
   directions; use the default local store or store: share

   store: share drives the local GCRA limiter (which reconciles correctly) over a ÷N per-replica token window, so it binds soundly.
3. concurrency → share for a fleet cap, never redis. A concurrency limit is a live gauge with no rate counter, so store: redis would be silently ignored (per-replica while you believe it’s fleet-wide) — the validator refuses it and points you at store: share, which reads the same live membership and caps each replica at ceil(C/N_live).
4. Budgets and quotas take no store. budget_usd, quota_tokens, and quota_requests are ledger-settled and converge cross-replica in their own class meter via the scoped accountant — spend in the dollar meter, tokens in the token meter, never blurred. Setting any store: on them is a config error.

Scale out on CPU with the GA HPA

Goal: let Kubernetes add replicas under load while the coordinated fleet stays correct.

Prerequisites: the shared-state tier is enabled (so new pods actually coordinate), and metrics-server is installed in the cluster.

1. Turn on autoscaling in your values (default is off):

   autoscaling:
     enabled: true
     minReplicas: 2
     maxReplicas: 6
     targetCPUUtilizationPercentage: 80

2. helm upgrade and confirm the HPA exists and targets the StatefulSet:

   $ kubectl get hpa -n tokentriage
   NAME   REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS
   tt     StatefulSet/tt          31%/80%         2         6         2

3. Watch it scale under load. On a scale-up, each new pod joins the roster on its first heartbeat, and a store: share ÷N limit re-derives its per-replica bound from the new live membership — the fleet ceiling ≈ amount holds at any N; a token quota keeps converging over the larger fleet.

Trial the coordinated fleet locally (3 replicas)

Goal: exercise the shared-state coordination path on a laptop before you commit to it in production.

1. The backends are off by default, behind per-backend profiles. Bring them up alongside the proxy:

   $ docker compose -f docker-compose.yml -f deploy/compose/compose.prod.yaml \
       -f deploy/compose/compose.backends.yaml --profile redis --profile postgres up -d

   Compose provides the container; it never wires TokenTriage to it — that’s your typed consent.
2. In your config, opt in with the same external: "allow" string the binary demands and the canonical env names (supply the env values by name in your env_file/host env — never as literals in config):

   shared_state:
     redis:
       external: "allow"
       addr: "redis:6379"
       password_env: TT_SHAREDSTATE_REDIS_PASSWORD
     postgres:
       external: "allow"
       dsn_env: TT_SHAREDSTATE_POSTGRES_DSN

   For the passwordless dev containers, TT_SHAREDSTATE_REDIS_PASSWORD may be empty and the DSN is postgres://tokentriage@postgres:5432/tokentriage?sslmode=disable.
3. Confirm the tier is live from the health panel — tiers should list both:

   $ curl -s localhost:9090/tt/api/v1/governance/health | jq '.capability.tiers'
   ["redis","postgres"]

   Enabling this breaks the single-binary air-gap posture for the coordination tier — the binary re-emits that disclosure at boot, and the panel’s note says so plainly.

Enable enforcement safely

1. Add the limit with posture: { mode: shadow } — it records would_refuse and enforces nothing.
2. Dry-run it over your logs: tokentriage govern preview -c tokentriage.yaml.
3. When the shadow evidence looks right, promote to posture: { mode: "on", enforce: hard, fail: open }. Keep fail: open unless you specifically want a hard stop disclosed as a 503. See Governance.

Promote a route on evidence

1. Run in shadow; read tokentriage report for would-route savings.
2. Fit a threshold from your logs — this writes a candidate artifact (inert) and prints its id: tokentriage calibrate -c config.yaml --target-frontier-pct 0.4. Then promote it: tokentriage calibrate --activate <candidate-id>.
3. Confirm the honesty verdict with judged shadow duplication: tokentriage eval --quality shadow. Only promote to route once it reads BEATS (a move without shadow evidence wears the route without shadow evidence badge).

Read provenance in an audit

1. Every dollar carries its pricing_snapshot, estimated flag, and source; an unpriced row is usd: null and counted, never $0.00.
2. Use the “view as table” affordance on any chart to see the exact rows behind it, and the append-only audit trail (POST /tt/api/v1/audit, or the Settings surface) for every config mutation with its generation and content hash.