# Runbooks

Start from the symptom you have. Each entry names the cause, the check that confirms it, and the fix.

Source: https://tokentriage.com/docs/operate/runbooks

---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';

You arrive here with a symptom, not a subject. This page is indexed by what you are seeing, and
every entry ends in a command or a config key rather than an explanation.

Work through an entry in the same order each time: confirm the cause with a check, apply the
fix, then confirm the symptom is gone. Where the correct action is to do nothing, that is
stated as the fix, because several failures here are self-healing and a restart makes them
worse.

## Find your symptom

| What you are seeing | Go to |
|---|---|
| the pod is crash-looping after a config change | [a pod that will not start](#a-pod-that-will-not-start) |
| the volume is filling up | [the ledger disk is filling](#the-ledger-disk-is-filling) |
| totals look too low, or `declog_dropped_total` moved | [the ledger is losing rows](#the-ledger-is-losing-rows) |
| the dashboard shows far fewer rows than the CLI | [the dashboard lost history](#the-dashboard-lost-history) |
| requests are returning `429` | [requests are being turned away](#requests-are-being-turned-away) |
| responses carry `x-tt-coordination: suspended` | [a coordinated limit degraded](#a-coordinated-limit-degraded) |
| the provider is failing and traffic is not moving | [an upstream is down](#an-upstream-is-down) |
| the dashboard is unreachable but traffic is fine | [the admin plane did not come up](#the-admin-plane-did-not-come-up) |
| an unfamiliar model is missing from spend | [a model is unpriced](#a-model-is-unpriced) |
| a replica vanished from the fleet count | [a replica left the roster](#a-replica-left-the-roster) |

## What should page you

The Helm chart ships three alert rules, rendered when `metrics.prometheusRule.enabled` is set
and the Prometheus Operator's CRD is present. Each one watches a place where the product would
otherwise be quietly wrong rather than loudly broken:

| Alert | Expression | Severity | Runbook |
|---|---|---|---|
| `TokenTriageDeclogDrops` | `rate(tokentriage_declog_dropped_total[5m]) > 0` | warning | [the ledger is losing rows](#the-ledger-is-losing-rows) |
| `TokenTriagePricingMisses` | `sum(rate(tokentriage_pricing_miss_total[15m])) by (model) > 0` | warning | [a model is unpriced](#a-model-is-unpriced) |
| `TokenTriageEnforcementSuspended` | `max by (limit_id) (tokentriage_governance_limit_suspended) > 0` | critical | [a coordinated limit degraded](#a-coordinated-limit-degraded) |

The drop rule fires on **any** drop, because the correct number of dropped rows is zero.

<Callout type="key">
Nothing ships that alerts on the volume filling. That is the failure with the shortest fuse and
the least warning, so add a rule against your own storage metric —
`kubelet_volume_stats_available_bytes` on Kubernetes, through
`metrics.prometheusRule.additionalRules`, which rides alongside the shipped group verbatim.
</Callout>

Alert on those symptoms rather than on causes underneath them. A Redis outage is not worth
paging for on its own, because coordination is fail-open, traffic keeps flowing, and the store
self-heals; the suspension gauge is what tells you a guarantee is not being held.

## A pod that will not start

**Confirm.** The config is validated by the binary at boot, not by Helm at render, so a config
that installs cleanly can still abort. Read the logs:

```console
$ kubectl logs -n tokentriage tt-tokentriage-0 --previous
```

A configuration fault prints the validator's own message. A partly-finished promotion up the
trust ladder is the common one:

```console
Error: configuration is invalid: config tokentriage.yaml: tiers: at least one tier is required in shadow mode
routing.default_tier: required in shadow mode
```

**Fix.** Roll back to the last good revision, then fix the values and validate before trying
again:

```console
$ helm rollback tt -n tokentriage
$ helm template tt deploy/helm/tokentriage -f my-values.yaml \
  | python -c "import sys,yaml;[print(d['data']['tokentriage.yaml']) for d in yaml.safe_load_all(sys.stdin) if d and d.get('kind')=='ConfigMap']" \
  | tokentriage validate --config -
```

**Confirm the fix.** `kubectl rollout status` completes and `helm test tt -n tokentriage`
succeeds. [Configuration in production](/docs/operate/configuration) covers what validate does
and does not catch.

## The ledger disk is filling

**Confirm.** Check the ledger's size and what is beside it:

```console
$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage doctor --config /etc/tokentriage/tokentriage.yaml
✓ ledger — /data/.tokentriage/decisions.jsonl — 902 rows, 0 torn lines, writable — clean
```

**Fix.** Three causes, in the order they are worth checking.

Rotation may be off. `ledger.decision_log.rotate_mb` at `0` or below lets one file grow without
bound; the default is `256`.

Retention may not be running. The sweeper starts as part of the dashboard composition, so a
deployment with `dashboard.enabled: false` rotates but never prunes. Either enable the dashboard
or accept unbounded history and size for it.

Retention may be configured to keep everything. Both `retention.max_age` and
`retention.max_total_mb` default to `0`, which means forever. Set at least one:

```yaml
retention:
  max_age: 720h
  max_total_mb: 512
```

The sweep runs hourly and the first one happens an interval after start, so nothing is deleted
the moment you apply this.

**Confirm the fix.** Companions older than the age limit disappear within the hour, and each
deletion writes an audit row with its reason. [The decision ledger on
disk](/docs/operate/ledger-lifecycle) covers the growth arithmetic behind the numbers you pick.

## The ledger is losing rows

**Confirm.** Compare the counter against what was served:

```console
$ curl -s localhost:9090/metrics | grep declog_dropped
tokentriage_declog_dropped_total 1539
```

**Fix.** The writer drops when its bounded queue is full, which means the disk is not keeping
up rather than that the queue is too small. Check storage first: a stalled volume, a filesystem
that has filled, or a burst far beyond what the volume absorbs. Raising
`ledger.decision_log.queue` above its default of `8192` buys a longer burst at the cost of
memory and does not help a writer that is permanently slower than the traffic.

**Confirm the fix.** The counter stops moving. It does not reset, because counters live in the
process, so use `rate()` rather than the raw total.

<Callout type="warning">
The dropped rows are gone and no rebuild recovers them. Every figure computed over the ledger
for that window is short by a number of rows you know and a dollar amount you do not. Note the
window in anything you report from that period.
</Callout>

## The dashboard lost history

**Confirm.** Ask both surfaces for the same total. They read the same ledger and should agree:

```console
$ kubectl exec -n tokentriage tt-tokentriage-0 -- tokentriage top --log /data/.tokentriage/
TOTAL             2303  2760000  805000  $6.785000

$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq '.provenance.rows'
484
```

**Fix.** The read-model was rebuilt from the ledger *file*, so it holds only the active file's
rows and none of the rotated companions. Rebuild it from the **directory**, then restart the
pod so it reopens the rebuilt index:

```console
$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage index rebuild --index /data/.tokentriage/index/decisions.sqlite \
                              --log /data/.tokentriage/
rebuilt index /data/.tokentriage/index/decisions.sqlite from /data/.tokentriage/
  rows indexed:    2303
$ kubectl delete pod -n tokentriage tt-tokentriage-0
```

**Confirm the fix.** The two totals agree. Nothing was lost: the index is derived, and the
ledger held every row throughout.

## Requests are being turned away

**Confirm.** A `429` carries a typed reason. Read it from the response body rather than
inferring from the status:

```console
$ curl -i -H "Authorization: Bearer $VK" http://localhost:8787/v1/messages -d @body.json
```

**Fix.** The reason names which limit stopped the request and at which scope. [Why a request
was refused](/docs/build/governance-refusals) is the page for that diagnosis, and it maps every
error type to the config key that owns it.

**Confirm the fix.** The refusal stops, and the limit's own panel shows headroom again.

## A coordinated limit degraded

**Confirm.** The same fact appears on three surfaces at once: the response header
`x-tt-coordination: suspended`, a `suspended:[<limit_id>]` entry on the decision row, and a
widened bound on the governance health panel:

```console
$ curl -s -H "Authorization: Bearer $TT_TOKEN" \
    localhost:9090/tt/api/v1/governance/health | jq '.redis, .suspensions'
```

**Fix.** Nothing destructive. Coordination is fail-open: the data plane kept serving the whole
time, the bound widened rather than a number being asserted that could not be backed, and the
store is re-adopted by a single half-open probe when it returns.

<Callout type="warning">
Do not restart the fleet to clear a suspension. The restart throws away warm state, fixes
nothing, and the disclosure would have cleared on its own. A suspension is a disclosed change
of state, not an outage.
</Callout>

**Confirm the fix.** The header stops appearing. Its absence is the healthy default, and there
is no positive "ok" value to look for. [Governance across
replicas](/docs/build/governance-distributed) covers what each store mode guarantees while
degraded.

## An upstream is down

**Confirm.** `/health` will keep answering `200`, because it only reports that the process is
up. Ask the surface that knows:

```console
$ curl -s -H "Authorization: Bearer $TT_TOKEN" localhost:9090/tt/api/status | jq '.upstreams'
```

`in_cooldown: true` means the upstream has been taken out of rotation after repeated failures.
`auth_health.state` of `missing_env` means the credential the config *names* is not present in
the environment, which fails in a way that looks like a provider outage and is not.

**Fix.** For a real outage, cooldown handles rotation and recovery on its own. For a missing
credential, supply the environment variable the config names. The config carries the name, not
the value. For a provider whose failure should move traffic elsewhere,
[cascades](/docs/build/routing-cascades) is the mechanism.

**Confirm the fix.** `in_cooldown` returns to `false` and requests to that upstream succeed.

## The admin plane did not come up

**Confirm.** A bind failure on the admin port does not stop the process, so it runs and
the data plane serves. Read the first line of the boot log:

```console
dashboard unavailable: listen tcp 127.0.0.1:9090: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. — proxy traffic unaffected (run: tokentriage doctor)
```

That wording is Windows's; Linux says `address already in use` in the same position. The banner
that follows has no `dashboard:` line.

**Fix.** Free the port or move the admin bind. On Kubernetes this is rare, because the pod owns
its network namespace, and it points at a second container in the pod or a host-network setting.

**Confirm the fix.** The banner prints its `dashboard:` line and `/metrics` answers.

## A model is unpriced

**Confirm.** `tokentriage_pricing_miss_total` carries the model label, and the rollup discloses
the exclusion under the total it is missing from.

**Fix.** The rows are recorded and counted; only the dollar is absent, and it is left blank
rather than filled in as `$0.00`. Add a pricing entry for the model through the overrides
overlay. [Where a cost figure comes from](/docs/build/cost-figures) covers what the figure
excludes and how the disclosure travels with it.

**Confirm the fix.** The miss counter stops moving for that model and the rollup's excluded-row
count drops.

## A replica left the roster

**Confirm.** Fleet reads carry their coverage: `replicas_answered` below `replicas_total` with
`is_partial: true` is a disclosed partial, not a smaller-looking complete total.

```console
$ tokentriage fleet list -c tokentriage.yaml
```

**Fix.** A drained or removed replica stays in the coverage denominator until you say otherwise,
which is the right choice: a number that silently shrank would be indistinguishable from a
number that was always that size. Remove it explicitly:

```console
$ tokentriage fleet decommission <name>
```

The removal is audited and drops the replica from `N`.

**Confirm the fix.** `fleet list` no longer shows it, and reads stop reporting a partial.
[Multi-replica & scale-out](/docs/operate/scale-out) covers the coverage vocabulary.

<Details summary="Procedures that live elsewhere, and where they went">

Several operational tasks are owned by the section that owns their subject, because the
procedure is inseparable from the semantics:

| Task | Page |
|---|---|
| back up, restore, rebuild the read-model | [back up and restore](/docs/operate/backup-restore) |
| upgrade, roll back, change the image | [upgrade and roll back](/docs/operate/upgrades) |
| pick `store:` for a governance limit | [enforcement classes](/docs/build/governance-enforcement-classes) |
| roll out enforcement shadow-first | [governance rollout](/docs/build/governance-rollout) |
| rotate or revoke a virtual key | [virtual keys](/docs/build/governance-keys) |
| promote a route on shadow evidence | [your first routing rollout](/docs/get-started/first-routing-rollout) |

</Details>

<NextSteps items={[
  {title: "Diagnose a running instance", href: "/docs/operate/diagnostics", description: "The four read-only surfaces, and which one is entitled to answer your question."},
  {title: "The decision ledger on disk", href: "/docs/operate/ledger-lifecycle", description: "The growth, rotation and retention behind two of the runbooks above."},
  {title: "Back up and restore", href: "/docs/operate/backup-restore", description: "The procedure the worst symptoms end in, executed rather than transcribed."},
  {title: "Production hardening", href: "/docs/operate/hardening", description: "The checklist that prevents several of these before they happen."}
]} />