Diagnose a running instance

Something looks wrong and you need to find out what, without restarting anything and without guessing. TokenTriage gives you four read-only surfaces, and the first useful skill is knowing which of them can answer your question.

Prerequisites: the ability to reach the admin plane, by kubectl port-forward or a shell on the host. A credential for the parts of the admin API that need one. Nothing here writes, and nothing here restarts.

Which surface answers which question

Start with doctor

doctor is a read-only diagnostic. It does not boot the proxy, write anything, or print a secret. Point it at the config the instance is running:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage doctor --config /etc/tokentriage/tokentriage.yaml
✓ binary — tokentriage compose (none), embedded pricing snapshot litellm@2026-07-14#179affb — clean
✓ config — /etc/tokentriage/tokentriage.yaml — strict decode clean: mode ledger, 1 upstream(s), 0 tier(s), 0 rule(s)
✓ compile — provider registry + routing policy compiled — 1 upstream(s) wired — clean
! ports — listen 0.0.0.0:8787 in_use, admin 0.0.0.0:9090 in_use — a port is in use by another process
    listen 0.0.0.0:8787 is held by another process — stop it or set server.listen
    admin 0.0.0.0:9090 is held by another process — stop it or set server.admin
✓ ledger — /data/.tokentriage/decisions.jsonl — 902 rows, 0 torn lines, writable — clean
✓ index — /data/.tokentriage/index/decisions.sqlite — indexed_through 2026-08-10T13:02:45.875Z (offset 510,153), 1,231 bytes behind the ledger — clean
✓ pricing — embedded table litellm@2026-07-14#179affb loaded, no overrides configured — clean
✓ keys — 1 upstream(s), no *_env credential references (inbound forwarded) — clean
✓ dashboard — enabled — auth token, state /data/.tokentriage/dash, index /data/.tokentriage/index/decisions.sqlite, demo false — would compose (config valid; stores open at boot)
○ instance — not probed — pass --probe to check a running instance's admin plane over HTTP

ok — no hard failures, 1 soft warning(s) (exit 0)

The version string reads compose because that image was built from source with that version argument; a release prints its own version there.

Three lines in that report do more work than the rest.

ledger gives you the row count, whether any line is torn, and whether the path is writable. A crash mid-write leaves at most one truncated final line, which readers skip and count, so a non-zero torn count is worth noting rather than panicking about.

index publishes how far behind the read-model is, in bytes. A steadily growing lag means the indexer is not keeping up; a lag that stops changing on a busy instance means it has stopped.

ports reads as a problem and is not, when you run doctor inside a running pod. It checks bindability, and the process you are diagnosing is holding the ports. One soft warning of exactly that shape is the healthy answer.

Hard-tier failures on config, compile or the ledger exit 1 in both text and --json mode, so tokentriage doctor --json || remediate works in a script.

○ config — no config file (quickstart-less env) — checking the binary + defaults; pass --config to check one
    run: tokentriage doctor --config <path>   (to strictly check a specific config)
○ compile — skipped — no valid config to compile (see the config check)

Add --probe for the running process

--probe makes doctor talk to a live admin plane over HTTP rather than only reading files:

$ kubectl exec -n tokentriage tt-tokentriage-0 -- \
    tokentriage doctor --config /etc/tokentriage/tokentriage.yaml --probe
! instance — http://127.0.0.1:9090/health OK but /tt/api/status unavailable: HTTP 401

ok — no hard failures, 2 soft warning(s) (exit 0)

That warning is the secured-by-default behaviour working, and the transcript is trimmed to the two lines it added. /health answered and the admin API declined an unauthenticated probe, which is what the bundled helm test asserts. On a deployment with the dashboard off and a loopback admin bind you get the status payload instead.

Read the status surface

/tt/api/status is the one that knows about providers:

$ curl -s -H "Authorization: Bearer $TT_TOKEN" http://localhost:9090/tt/api/status

{
  "mode": "ledger",
  "listen": "127.0.0.1:8787",
  "upstreams": {
    "anthropic": {
      "base_url": "http://127.0.0.1:47811",
      "dialect": "anthropic",
      "in_cooldown": false,
      "auth_scheme": "header",
      "auth_health": { "configured": false, "fresh": false, "state": "missing_env" }
    }
  },
  "tiers": null,
  "pricing_snapshot": "litellm@2026-07-14#179affb",
  "proxy_base_url": "http://127.0.0.1:8787"
}

That upstream is a local mock with no credential configured, which is why auth_health reads missing_env; a live upstream with api_key_env set and the variable present reads configured: true. Three fields carry most of the diagnostic value. in_cooldown says whether an upstream has been taken out of rotation after failures. auth_health says whether the credential the config names exists in the environment. pricing_snapshot says which price table this process is costing against.

Provider health is reported, not guessed. A field that cannot be known is absent or typed unknown rather than filled in optimistically.

The auth middleware ships with the dashboard, so this surface needs a bearer token only where dashboard.enabled is true. On a headless deployment it answers unauthenticated to anything that can reach the admin plane, which is one reason that plane binds loopback by default.

Confirm the fleet you think you have

On a single instance, ask anyway. The answer is more useful than an empty list:

$ tokentriage fleet list -c tokentriage.yaml
no durable roster: shared_state.postgres is not enabled (external: allow). N falls back to the "heartbeat-window" basis (the live heartbeat hash within the purge window).

It names why there is no roster and what the coverage denominator falls back to, instead of printing zero rows and letting you conclude the fleet is empty. With the Postgres tier enabled it prints the active, non-decommissioned replicas. Multi-replica & scale-out covers what that denominator does to a fleet-wide number.

Reconcile two surfaces that should agree

The most informative check available is a comparison, because a single number cannot tell you it is wrong. Ask the CLI and the dashboard for the same total:

$ tokentriage top --log /data/.tokentriage/
TOTAL             2303  2760000  805000  $6.785000

$ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \
    -d '{}' http://localhost:9090/tt/api/v1/query | jq '.provenance.rows'
2303

Agreement means the read-model holds the whole ledger. Disagreement has one common cause: a read-model rebuilt from the ledger file rather than its directory, which omits every rotated companion. Back up and restore has the fix.

A second comparison worth running is the request counter against the row count. Where tokentriage_requests_total exceeds the rows in the ledger, the difference is on tokentriage_declog_dropped_total, and every total you compute from the file is short by that many rows.

What this does not cover

The metric catalogue, what each series means and what an empty scrape implies belong to the metrics catalogue. Diagnosing a refused request, a 401, 429 or 503 with a request id in hand, is why a request was refused. Acting on what you find is runbooks.