# Operating governance Read the health panel, a suspension, and the export of what was turned away, then reconcile spend without adopting a number. Source: https://tokentriage.com/docs/build/governance-operations --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; Caps are on, traffic is flowing, and most Tuesdays the job is thirty seconds on one panel before you close the tab. The other Tuesdays are the ones this page is for: something has gone quiet and requests are getting through that should not be, or a number in a report disagrees with a number in a ledger, or a config reload has left a counter behaving oddly. Each section below is one job, and the page assumes governance is already enforcing on your traffic. If it is not, [turn governance on](/docs/build/governance-rollout) is the climb and [governance](/docs/build/governance) is the map. Every endpoint here lives on the admin plane at the viewer role floor, so a read-only `ttk_` token is enough for all of it. That plane is served only where `dashboard.enabled` is `true`, which a config file leaves `false` until you set it; [the management API](/docs/build/governance-api) has the port and the credential. ## Check that governance is healthy The first question is not "is anything over its cap" but "is every cap still being checked". `GET /tt/api/v1/governance/health` answers that in one object. On a single-node instance it looks like this: ```json { "kind": "governance/distributed-health", "capability": { "shared_state_enabled": false, "tiers": [], "note": "air-gapped: no shared-state tier configured — this is a single-node instance; distributed limits, cross-replica convergence, and reconcile are not active" }, "suspensions": [], "convergence": [] } ``` Read `capability.note` first: it is the field that tells you what the empty lists below it mean. Nothing is broken there. With no `shared_state` block configured the endpoint returns this shape rather than a 404 or a blank panel, and the arrays are empty because there is no shared tier to report on. Switch a shared-state tier on and the same response grows. `shared_state_enabled` flips true, `tiers` fills with `redis`, `postgres`, or both, and a section appears for each: `redis` carries `reachable`, `state` (`closed`, `open`, or `half-open`), `consecutive_failures` and the last transport error; `postgres` carries `up`, `replica_ready`, `revocation_bound`, `staleness_bound` and `pending_propagation`. A `convergence` row per `budget_usd` scope arrives with the `redis` tier, since the component that folds cross-replica spend is built only alongside a Redis client. What those numbers bound is [governance across replicas](/docs/build/governance-distributed). `suspensions` needs one of those tiers as well, which is worth knowing before you lean on this panel during an incident. Nothing populates it on a single-node instance, so the array stays empty whatever happens to your limits, and the decision row is where that disclosure shows up instead. For per-limit live state, `GET /tt/api/v1/governance/status` returns each node that has a live limit instance. Every limit carries a four-valued `posture` string (`alert_only`, `enforcing`, `fail_closed`, `suspended`), a `fraction`, and its overrun bound; spend limits add `burn`, `limit` and a `forecast` block, count limits add `observed` and `ceiling`. One kind of limit is absent from it. A rate limit on `store: redis` declines to report a live count, because the count lives in Redis and reading it back would cost a round trip on a display path, so the endpoint leaves that limit out rather than showing a zero. The same decline means such a limit can drive none of the success-side rate headers on a served response either. The `forecast` block is worth one sentence, because it looks broken and is not: ```json "forecast":{"method":"none","already_exhausted":false,"insufficient":true} ``` Every spend limit reports `method: none` with `insufficient: true` in this build. The accountant computes no burn-rate projection, so the block says there is not enough to fit rather than drawing a line through two points; `already_exhausted` is the field that carries information, and it tracks whether the cap is used up. Do not wait for a projection. A `tpm` limit's `observed` needs similar care, because it counts tokens *reserved* at the gate rather than tokens settled. The reserve is the prompt byte count divided by four, plus `max_tokens` capped at 16,384, or the full 16,384 when the request sets no `max_tokens`. Settlement hands back the difference, so a window running ahead of your token usage is the estimate rather than a leak. The bound envelope declares an `estimate_skew_p95` for the rolling reserve-versus-settled delta, and nothing fills it, so that field does not appear on a live surface. [How a limit counts](/docs/build/governance-enforcement-classes) has the arithmetic. ## See what is being turned away Two endpoints hold the record of stopped requests. `GET /tt/api/v1/governance/refusals` is the live window, newest first, narrowable with `?node=` to one subtree, and paged at 100 rows by default with a ceiling of 1,000. `GET /tt/api/v1/governance/exports/refusals` is the archive twin: uncapped and complete by default, window-scoped, and served as `format=json` (the default) or `format=csv`. Both read the canonical decision log, so they agree with your rows and with what your callers saw. Item fields are covered on [why was this request refused](/docs/build/governance-refusals). The CSV is the one to hand to somebody else, because it carries its own provenance. The file opens with `#`-prefixed comment lines; strip the `# ` and the block is a JSON object, `{"tokentriage_export": {…}}`, holding the schema version, `generated_at`, the instance name, the requested window, the row count and a `demo` flag. Then a frozen column order: | column | what it holds | |---|---| | `ts`, `req_id` | the row's timestamp and request id, for joining back to the ledger | | `node`, `kid` | the tenancy node and the virtual key that held it | | `type` | the error family, one of the eight values in [the refusal wire reference](/docs/build/governance-refusal-reference#reason-to-status) | | `limit_id`, `limit_type` | which limit acted, blank on a 401 | | `posture` | the enforcement string that limit reported | | `approximate` | whether the figure behind the decision was an estimate | | `http_status` | 401, 403, 429, or 503 | A 401 names no limit on either surface, so a support export tells an unknown credential nothing about your limit set. Two mechanisms stack there. An `invalid_key` stop writes neither a `refused_by` nor an `enforcement` block to the row in the first place, so there is nothing to read back, and each surface then blanks those columns again as a second guard. A CSV scan that runs out of its query budget ends the file with a `# truncated: true` comment rather than passing a short file off as whole. Its companion, `GET /tt/api/v1/governance/exports/usage`, rolls consumption up per node and period bucket. `?period=` takes `day`, `week`, `month` or `total` and defaults to `month`. Each row carries `requests`, the three token counts, spend, `unpriced_rows` and `estimated_rows`. Stopped requests are left out, since they consumed nothing upstream. A group with no priced rows reports no dollar figure at all: the CSV column `spend_usd` renders the literal `UNPRICED` rather than `0.00`, and the JSON `spend` object carries `usd: null` with `unpriced: true`. Both exports answer 404 `feature_absent` with an enable hint on an instance whose decision log has not been written yet. This page covers the evidence that enforcement happened. Analysing what a tenant spent is a different job with a different owner: [cost analytics](/docs/build/cost-analytics). ## Read a suspension `suspended: ["team_rpm"]` on a decision row is the most important thing governance can tell you, and it slips past readers because the row around it usually looks ordinary. It names every limit whose guarantee was degraded while that one request was being decided. The default is availability: [fail open with disclosure](/docs/build/governance#fail-open-with-disclosure). Two different faults put one of your limit ids in that array, and they want opposite reactions. A spend or quota limit lands there when the accountant folding its counter has gone quiet. There was no figure to compare the request against, so the cap checked nothing and the request went through unmeasured. That is the case where a cap has stopped being a cap. A rate or concurrency limit lands there when its shared coordination state could not be consulted, either because the Redis store was unreachable or because a `store: share` cap's view of the fleet froze. This replica's own limiter answered instead, so the amount you configured still held locally. The loss is fleet-wide: every replica is now enforcing that whole amount alone, so the combined ceiling widens toward N times it. The id is stamped whether that fallback admitted the request or returned a 429, so seeing it does not tell you the request succeeded. Read the array against the limit type before you act on it. A suspended `budget_usd` is spending unwatched and the ledger rows are the only place the overage is recorded, so fix the accountant and then count what got through from the rows rather than from the counter. A suspended `rpm` is still capped per replica, so the question there is how many replicas you are running. One id in that array is reserved. `__identity` is not one of your limits: it means the key store itself was degraded when the request arrived, either because its snapshot rebuild failed or because a Postgres mirror replica has not seeded or has stopped refreshing. The gate kept serving the last good snapshot and marked the guarantee suspended. Some of that reaches the health panel, and less of it than the field list suggests. A `suspensions` entry carries `limit_id`, `limit_type`, `node`, `class`, a `reason`, a `fail_posture` of `open` or `closed`, the cap, and the widened bound. Only two kinds of entry are ever built. One is a `budget_usd` limit whose accountant heartbeat is stale, with `heartbeat_stale` as its reason and its cap beside the bound. The other is a single `__identity` entry with `replica_not_ready`, raised when a Postgres mirror is wired and has not seeded. The wire vocabulary also declares the reasons `breaker_open` and `deadline_exceeded`, and nothing in the product writes either one. So a rate or concurrency limit suspended by a store outage does not reach that list, whatever the outage. It reaches you on the decision row and through the `redis` section of the same response, and no count-class entry is built to badge `open` or `closed`. None of this is a reason to edit a limit. A suspension is an availability event about the thing doing the counting, so the fix sits upstream of governance. ## Get told before a cap is reached Alert fractions are a property of the limit, not of your alerting rules. Add them to the limit itself: ```yaml governance: limits: - id: org_monthly_budget attach: { level: org } type: budget_usd amount_usd: 1000 period: month posture: mode: "on" thresholds: [0.5, 0.8, 0.95] # default: none ``` Each fraction fires once per period per limit instance, at the moment the folded observed value reaches it, and the set resets when the period rolls. A second rule, `enforcement_suspended`, fires per enforcing instance when the accountant heartbeat goes stale and resolves when it recovers; it takes no configuration. Both are evaluated on a wakeup from the accountant plus a 30-second poll, and once at startup, so a limit already over a threshold alerts straight away. Three defaults decide whether anything reaches you, and the first one catches people out. Governance alerting is built as part of the dashboard, so `dashboard.enabled`, which is `false` in a config file until you set it, gates both rules: with the dashboard off, a threshold is recorded as crossed and nobody is told. Then `thresholds` itself defaults to none, so an unannotated limit is watched by nothing. And `alerts.channels` defaults to empty, so out of the box a crossing publishes to the dashboard alert stream, appends to `/alerts/governance_history.jsonl` (`~/.tokentriage/dash` by default), and is delivered nowhere else. Add a channel for a webhook, Slack message or email; governance alerts fan out to every configured channel and are independent of `alerts.rules`. One shape validates and does nothing: `thresholds` on an `rpm`, `tpm` or `concurrency` limit. The fractions are range-checked and then ignored, because both rules read the spend and quota accountant, which those three types do not use. ## Check the rollup against the ledger `govern reconcile` answers one question: does the derived per-scope spend still agree with the canonical ledger? It folds your decision log through the same arithmetic the proxy accounts with, compares each scope against a claimed rollup, and writes nothing back. ```console $ tokentriage govern reconcile -c tokentriage.yaml --log ~/.tokentriage/decisions.jsonl govern reconcile — reconcile/canonical-ledger-vs-rollup DISPLAY-AND-ALERT ONLY: reconcile recomputes from the canonical ledgers and NEVER adopts (CG7). ledger: .../decisions.jsonl rows folded: 8 rollup source: (none — every rollup treated as absent/0) DRIFT — canonical burn (recomputed from the ledger) − claimed rollup: [budget_usd] org_monthly_budget@org:acme canonical (ledger) $0.000037800 (37800 nanodollars) rollup (claimed) $0.000000000 (0 nanodollars) drift +$0.000037800 (37800 nanodollars) ⟵ DRIFT RECONCILE: 1 scope(s) DRIFTED. This is an ALERT, not a fix. POLICY: NO rollup was adopted, corrected, or written (adopted=false). The canonical ledgers remain the sole system of record for usage; resolve drift at its source, never by adoption. ``` That capture is from a sandbox pointed at a stand-in upstream, which is why the dollars are fractions of a cent; the shape is what yours prints. The ledger path is elided. That run found nothing. No `--rollup` was passed, so every claimed rollup was read as zero and each scope reported its whole burn as drift, which the closing line then counts. Read the `rollup source:` line before believing the alarm. To compare against a figure, pass `--rollup rollup.json`, a map of `limit_id@node` (with `|dim=val` selectors appended for a per-model scope) to nanodollars. Two more things before you schedule it. `--as-of` pins the comparison instant to the rollup's capture time, so a run near a calendar boundary does not manufacture full-value drift by folding into a fresh period. And on an instance with no decision log yet, the command errors on the fold instead of printing an empty report, which makes a fresh install look like a failure; `-c` is required too, even though the help text shows a default. `GET /tt/api/v1/governance/reconcile` is the read-only twin, and it is narrower than it looks. It runs the same recomputation against an absent rollup, so the reading above applies, but only where a shared-state tier is configured. Without one it answers `200` with no scopes at all and an `alert` string saying this is a single-node instance, which is the same wiring that empties `suspensions`. On a single node the CLI is the surface that does the work. ## What a reload keeps, and what it drops "Carry-over" in this product is about hot reloads, not about budgets. Unused budget does not roll into the next period, there is no config key for that, and a period boundary resets the counter to zero. What is carried is live counter state across a governance re-arm. Editing one limit rebuilds the whole compiled set, so an untouched limit would otherwise restart its window at zero and admit a burst. The rule: a limit whose **counter identity** is unchanged keeps its counter, and a limit whose identity changed starts fresh. Nine fields make up that identity: `id`, `type`, `algorithm`, `period`, `amount`, `burst`, `store`, the whole `attach` block, and the whole `match` block. Two things are excluded so that editing them keeps the counter: the `posture:` block, including `thresholds`, and the override ceiling. Promoting a limit from `shadow` to `on` therefore costs nothing. The trap is `burst` on a window algorithm. `fixed_window` and `sliding_window` ignore the value entirely, but the carry compares it literally, so a burst-only edit on such a limit drops the counter and buys a window of extra admissions while changing enforcement by nothing. [Rate algorithms](/docs/build/governance-rate-algorithms) has the full inert-knob picture. Which limits participate is narrower than it sounds. Local-store rate limits and concurrency gauges are carried. A `store: redis` limit keeps its state in Redis and a rebuild does not touch it. A `store: share` rate limit is skipped, so it does restart its window on every re-arm. Spend and quota limits ride the accountant instead, with one asymmetry: a budget *added* by a reload begins at zero for the current period, because only a process restart replays the ledger to backfill it. ## A five-minute runbook Symptom to first look, on the surfaces above. This is the governance slice; the general version is [troubleshooting](/docs/reference/troubleshooting). | symptom | look here first | likely cause | |---|---|---| | a caller got a 429 from a limit you thought was watching | the limit's `posture:` block in your config file | a limit written with no `posture:` enforces from the moment it exists, and `/governance/limits` reports `off`, `shadow` and soft alike as `alert_only` | | a spend cap acted after the money was gone | `enforcement.bound` and `note` on the 429 body | a `budget_usd` counter folds off the request path, so in-flight and streaming rows are missing from it | | traffic is flowing past an enforcing spend cap | `suspended` on the row first, then `suspensions` on `/governance/health` | the accountant heartbeat is stale, so the cap checked nothing and failed open | | rate limits admitted a burst right after a reload | the nine counter-identity fields against your diff | an edit changed the identity, so that limit's counter restarted at zero | | `keys list` shows a key the proxy answers 401 to | the CLI's `--state-dir` against `governance.store.path` | `keys` opens `/governance.db` and does not read the config's store path | | `govern reconcile` reports drifted scopes | the `rollup source:` line | no `--rollup` was passed, so every claimed rollup was treated as zero | | a budget crossed 80% and nobody heard | `dashboard.enabled`, then `posture.thresholds` on the limit, then `alerts.channels` | governance alerting rides the dashboard, thresholds default to none, and channels default to empty | ## What this page does not cover The mechanics behind the health panel, including what converges and the 30-second revocation and 60-second staleness bounds, are [governance across replicas](/docs/build/governance-distributed). Every endpoint named here, with its parameters and role floor, is [the management API](/docs/build/governance-api). Working back from one 429 to the limit that produced it is [why was this request refused](/docs/build/governance-refusals), and the field-level contract behind these exports is [the refusal wire reference](/docs/build/governance-refusal-reference). Spend analysis, attribution and reporting are [cost analytics](/docs/build/cost-analytics). The knobs that validate and do nothing are catalogued in [what governance does not do](/docs/build/governance-gaps).