# Diagnose a running instance Four surfaces answer four different questions. Learn which one can call a deployment healthy, and which only says the process is up. Source: https://tokentriage.com/docs/operate/diagnostics --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; Something looks wrong and you need to find out what, without restarting anything and without guessing. TokenTriage gives you four read-only surfaces, and the first useful skill is knowing which of them can answer your question. **Prerequisites:** the ability to reach the admin plane, by `kubectl port-forward` or a shell on the host. A credential for the parts of the admin API that need one. Nothing here writes, and nothing here restarts. ## Which surface answers which question | Surface | Auth | Answers | Does not answer | |---|---|---|---| | `GET /health` | none | is the process up | anything about providers, storage, or config | | `GET /tt/api/status` | bearer, when the dashboard is on | mode, upstreams, cooldowns, credential presence, pricing snapshot | whether the ledger is being written | | `tokentriage doctor` | none, runs locally | config, compile, ports, ledger, index, pricing, credentials | live traffic | | `GET /metrics` | none | counters and gauges over time, including drops | why a counter moved | `GET /health` returns `200` with the body `OK` and asserts nothing else. It is what the kubelet's probes check, so **"the pod is Ready" means the process is running**. It does not mean a provider is reachable, the disk is writable, or the admin plane came up. Reachability lives on `/tt/api/status`, which is the surface to check when a deployment looks up but behaves as though it is not. ## Start with doctor `doctor` is a read-only diagnostic. It does not boot the proxy, write anything, or print a secret. Point it at the config the instance is running: ```console $ kubectl exec -n tokentriage tt-tokentriage-0 -- \ tokentriage doctor --config /etc/tokentriage/tokentriage.yaml ✓ binary — tokentriage compose (none), embedded pricing snapshot litellm@2026-07-14#179affb — clean ✓ config — /etc/tokentriage/tokentriage.yaml — strict decode clean: mode ledger, 1 upstream(s), 0 tier(s), 0 rule(s) ✓ compile — provider registry + routing policy compiled — 1 upstream(s) wired — clean ! ports — listen 0.0.0.0:8787 in_use, admin 0.0.0.0:9090 in_use — a port is in use by another process listen 0.0.0.0:8787 is held by another process — stop it or set server.listen admin 0.0.0.0:9090 is held by another process — stop it or set server.admin ✓ ledger — /data/.tokentriage/decisions.jsonl — 902 rows, 0 torn lines, writable — clean ✓ index — /data/.tokentriage/index/decisions.sqlite — indexed_through 2026-08-10T13:02:45.875Z (offset 510,153), 1,231 bytes behind the ledger — clean ✓ pricing — embedded table litellm@2026-07-14#179affb loaded, no overrides configured — clean ✓ keys — 1 upstream(s), no *_env credential references (inbound forwarded) — clean ✓ dashboard — enabled — auth token, state /data/.tokentriage/dash, index /data/.tokentriage/index/decisions.sqlite, demo false — would compose (config valid; stores open at boot) ○ instance — not probed — pass --probe to check a running instance's admin plane over HTTP ok — no hard failures, 1 soft warning(s) (exit 0) ``` The version string reads `compose` because that image was built from source with that version argument; a release prints its own version there. Three lines in that report do more work than the rest. **`ledger`** gives you the row count, whether any line is torn, and whether the path is writable. A crash mid-write leaves at most one truncated final line, which readers skip and count, so a non-zero torn count is worth noting rather than panicking about. **`index`** publishes how far behind the read-model is, in bytes. A steadily growing lag means the indexer is not keeping up; a lag that stops changing on a busy instance means it has stopped. **`ports`** reads as a problem and is not, when you run `doctor` inside a running pod. It checks bindability, and the process you are diagnosing is holding the ports. One soft warning of exactly that shape is the healthy answer. Hard-tier failures on config, compile or the ledger exit `1` in both text and `--json` mode, so `tokentriage doctor --json || remediate` works in a script.
`doctor` with no flag looks for `./tokentriage.yaml` in the working directory. Inside a container the config is mounted at `/etc/tokentriage/tokentriage.yaml` and the working directory has none, so two checks skip: ```console ○ config — no config file (quickstart-less env) — checking the binary + defaults; pass --config to check one run: tokentriage doctor --config (to strictly check a specific config) ○ compile — skipped — no valid config to compile (see the config check) ``` A skip is not a failure and the summary still exits `0`. Pass `--config` whenever you diagnose a container, or the two checks that matter most did not run.
## Add `--probe` for the running process `--probe` makes `doctor` talk to a live admin plane over HTTP rather than only reading files: ```console $ kubectl exec -n tokentriage tt-tokentriage-0 -- \ tokentriage doctor --config /etc/tokentriage/tokentriage.yaml --probe ! instance — http://127.0.0.1:9090/health OK but /tt/api/status unavailable: HTTP 401 ok — no hard failures, 2 soft warning(s) (exit 0) ``` That warning is the secured-by-default behaviour working, and the transcript is trimmed to the two lines it added. `/health` answered and the admin API declined an unauthenticated probe, which is what the bundled `helm test` asserts. On a deployment with the dashboard off and a loopback admin bind you get the status payload instead. ## Read the status surface `/tt/api/status` is the one that knows about providers: ```console $ curl -s -H "Authorization: Bearer $TT_TOKEN" http://localhost:9090/tt/api/status ``` ```json { "mode": "ledger", "listen": "127.0.0.1:8787", "upstreams": { "anthropic": { "base_url": "http://127.0.0.1:47811", "dialect": "anthropic", "in_cooldown": false, "auth_scheme": "header", "auth_health": { "configured": false, "fresh": false, "state": "missing_env" } } }, "tiers": null, "pricing_snapshot": "litellm@2026-07-14#179affb", "proxy_base_url": "http://127.0.0.1:8787" } ``` That upstream is a local mock with no credential configured, which is why `auth_health` reads `missing_env`; a live upstream with `api_key_env` set and the variable present reads `configured: true`. Three fields carry most of the diagnostic value. `in_cooldown` says whether an upstream has been taken out of rotation after failures. `auth_health` says whether the credential the config *names* exists in the environment. `pricing_snapshot` says which price table this process is costing against. Provider health is reported, not guessed. A field that cannot be known is absent or typed unknown rather than filled in optimistically. The auth middleware ships with the dashboard, so this surface needs a bearer token only where `dashboard.enabled` is true. On a headless deployment it answers unauthenticated to anything that can reach the admin plane, which is one reason that plane binds loopback by default. ## Confirm the fleet you think you have On a single instance, ask anyway. The answer is more useful than an empty list: ```console $ tokentriage fleet list -c tokentriage.yaml no durable roster: shared_state.postgres is not enabled (external: allow). N falls back to the "heartbeat-window" basis (the live heartbeat hash within the purge window). ``` It names why there is no roster and what the coverage denominator falls back to, instead of printing zero rows and letting you conclude the fleet is empty. With the Postgres tier enabled it prints the active, non-decommissioned replicas. [Multi-replica & scale-out](/docs/operate/scale-out) covers what that denominator does to a fleet-wide number. ## Reconcile two surfaces that should agree The most informative check available is a comparison, because a single number cannot tell you it is wrong. Ask the CLI and the dashboard for the same total: ```console $ tokentriage top --log /data/.tokentriage/ TOTAL 2303 2760000 805000 $6.785000 ``` ```console $ curl -s -X POST -H "Authorization: Bearer $TT_TOKEN" -H 'Content-Type: application/json' \ -d '{}' http://localhost:9090/tt/api/v1/query | jq '.provenance.rows' 2303 ``` Agreement means the read-model holds the whole ledger. Disagreement has one common cause: a read-model rebuilt from the ledger *file* rather than its directory, which omits every rotated companion. [Back up and restore](/docs/operate/backup-restore) has the fix. A second comparison worth running is the request counter against the row count. Where `tokentriage_requests_total` exceeds the rows in the ledger, the difference is on `tokentriage_declog_dropped_total`, and every total you compute from the file is short by that many rows. ## What this does not cover The metric catalogue, what each series means and what an empty scrape implies belong to [the metrics catalogue](/docs/build/observability-metrics-reference). Diagnosing a *refused* request, a 401, 429 or 503 with a request id in hand, is [why a request was refused](/docs/build/governance-refusals). Acting on what you find is [runbooks](/docs/operate/runbooks).