Operating governance

Caps are on, traffic is flowing, and most Tuesdays the job is thirty seconds on one panel before you close the tab. The other Tuesdays are the ones this page is for: something has gone quiet and requests are getting through that should not be, or a number in a report disagrees with a number in a ledger, or a config reload has left a counter behaving oddly.

Each section below is one job, and the page assumes governance is already enforcing on your traffic. If it is not, turn governance on is the climb and governance is the map. Every endpoint here lives on the admin plane at the viewer role floor, so a read-only ttk_ token is enough for all of it. That plane is served only where dashboard.enabled is true, which a config file leaves false until you set it; the management API has the port and the credential.

Check that governance is healthy

The first question is not “is anything over its cap” but “is every cap still being checked”. GET /tt/api/v1/governance/health answers that in one object. On a single-node instance it looks like this:

{
  "kind": "governance/distributed-health",
  "capability": {
    "shared_state_enabled": false,
    "tiers": [],
    "note": "air-gapped: no shared-state tier configured — this is a single-node instance; distributed limits, cross-replica convergence, and reconcile are not active"
  },
  "suspensions": [],
  "convergence": []
}

Read capability.note first: it is the field that tells you what the empty lists below it mean. Nothing is broken there. With no shared_state block configured the endpoint returns this shape rather than a 404 or a blank panel, and the arrays are empty because there is no shared tier to report on.

Switch a shared-state tier on and the same response grows. shared_state_enabled flips true, tiers fills with redis, postgres, or both, and a section appears for each: redis carries reachable, state (closed, open, or half-open), consecutive_failures and the last transport error; postgres carries up, replica_ready, revocation_bound, staleness_bound and pending_propagation. A convergence row per budget_usd scope arrives with the redis tier, since the component that folds cross-replica spend is built only alongside a Redis client. What those numbers bound is governance across replicas.

suspensions needs one of those tiers as well, which is worth knowing before you lean on this panel during an incident. Nothing populates it on a single-node instance, so the array stays empty whatever happens to your limits, and the decision row is where that disclosure shows up instead.

For per-limit live state, GET /tt/api/v1/governance/status returns each node that has a live limit instance. Every limit carries a four-valued posture string (alert_only, enforcing, fail_closed, suspended), a fraction, and its overrun bound; spend limits add burn, limit and a forecast block, count limits add observed and ceiling. One kind of limit is absent from it. A rate limit on store: redis declines to report a live count, because the count lives in Redis and reading it back would cost a round trip on a display path, so the endpoint leaves that limit out rather than showing a zero. The same decline means such a limit can drive none of the success-side rate headers on a served response either.

The forecast block is worth one sentence, because it looks broken and is not:

"forecast":{"method":"none","already_exhausted":false,"insufficient":true}

Every spend limit reports method: none with insufficient: true in this build. The accountant computes no burn-rate projection, so the block says there is not enough to fit rather than drawing a line through two points; already_exhausted is the field that carries information, and it tracks whether the cap is used up. Do not wait for a projection.

A tpm limit’s observed needs similar care, because it counts tokens reserved at the gate rather than tokens settled. The reserve is the prompt byte count divided by four, plus max_tokens capped at 16,384, or the full 16,384 when the request sets no max_tokens. Settlement hands back the difference, so a window running ahead of your token usage is the estimate rather than a leak. The bound envelope declares an estimate_skew_p95 for the rolling reserve-versus-settled delta, and nothing fills it, so that field does not appear on a live surface. How a limit counts has the arithmetic.

See what is being turned away

Two endpoints hold the record of stopped requests. GET /tt/api/v1/governance/refusals is the live window, newest first, narrowable with ?node= to one subtree, and paged at 100 rows by default with a ceiling of 1,000. GET /tt/api/v1/governance/exports/refusals is the archive twin: uncapped and complete by default, window-scoped, and served as format=json (the default) or format=csv. Both read the canonical decision log, so they agree with your rows and with what your callers saw. Item fields are covered on why was this request refused.

The CSV is the one to hand to somebody else, because it carries its own provenance. The file opens with #-prefixed comment lines; strip the # and the block is a JSON object, {"tokentriage_export": {…}}, holding the schema version, generated_at, the instance name, the requested window, the row count and a demo flag. Then a frozen column order:

column	what it holds
ts, req_id	the row’s timestamp and request id, for joining back to the ledger
node, kid	the tenancy node and the virtual key that held it
type	the error family, one of the eight values in the refusal wire reference
limit_id, limit_type	which limit acted, blank on a 401
posture	the enforcement string that limit reported
approximate	whether the figure behind the decision was an estimate
http_status	401, 403, 429, or 503

A 401 names no limit on either surface, so a support export tells an unknown credential nothing about your limit set. Two mechanisms stack there. An invalid_key stop writes neither a refused_by nor an enforcement block to the row in the first place, so there is nothing to read back, and each surface then blanks those columns again as a second guard. A CSV scan that runs out of its query budget ends the file with a # truncated: true comment rather than passing a short file off as whole.

Its companion, GET /tt/api/v1/governance/exports/usage, rolls consumption up per node and period bucket. ?period= takes day, week, month or total and defaults to month. Each row carries requests, the three token counts, spend, unpriced_rows and estimated_rows. Stopped requests are left out, since they consumed nothing upstream. A group with no priced rows reports no dollar figure at all: the CSV column spend_usd renders the literal UNPRICED rather than 0.00, and the JSON spend object carries usd: null with unpriced: true. Both exports answer 404 feature_absent with an enable hint on an instance whose decision log has not been written yet.

This page covers the evidence that enforcement happened. Analysing what a tenant spent is a different job with a different owner: cost analytics.

Read a suspension

suspended: ["team_rpm"] on a decision row is the most important thing governance can tell you, and it slips past readers because the row around it usually looks ordinary. It names every limit whose guarantee was degraded while that one request was being decided. The default is availability: fail open with disclosure.

Two different faults put one of your limit ids in that array, and they want opposite reactions.

A spend or quota limit lands there when the accountant folding its counter has gone quiet. There was no figure to compare the request against, so the cap checked nothing and the request went through unmeasured. That is the case where a cap has stopped being a cap.

A rate or concurrency limit lands there when its shared coordination state could not be consulted, either because the Redis store was unreachable or because a store: share cap’s view of the fleet froze. This replica’s own limiter answered instead, so the amount you configured still held locally. The loss is fleet-wide: every replica is now enforcing that whole amount alone, so the combined ceiling widens toward N times it. The id is stamped whether that fallback admitted the request or returned a 429, so seeing it does not tell you the request succeeded.

One id in that array is reserved. __identity is not one of your limits: it means the key store itself was degraded when the request arrived, either because its snapshot rebuild failed or because a Postgres mirror replica has not seeded or has stopped refreshing. The gate kept serving the last good snapshot and marked the guarantee suspended.

Some of that reaches the health panel, and less of it than the field list suggests. A suspensions entry carries limit_id, limit_type, node, class, a reason, a fail_posture of open or closed, the cap, and the widened bound. Only two kinds of entry are ever built. One is a budget_usd limit whose accountant heartbeat is stale, with heartbeat_stale as its reason and its cap beside the bound. The other is a single __identity entry with replica_not_ready, raised when a Postgres mirror is wired and has not seeded. The wire vocabulary also declares the reasons breaker_open and deadline_exceeded, and nothing in the product writes either one.

So a rate or concurrency limit suspended by a store outage does not reach that list, whatever the outage. It reaches you on the decision row and through the redis section of the same response, and no count-class entry is built to badge open or closed.

None of this is a reason to edit a limit. A suspension is an availability event about the thing doing the counting, so the fix sits upstream of governance.

Get told before a cap is reached

Alert fractions are a property of the limit, not of your alerting rules. Add them to the limit itself:

governance:
  limits:
    - id: org_monthly_budget
      attach: { level: org }
      type: budget_usd
      amount_usd: 1000
      period: month
      posture:
        mode: "on"
        thresholds: [0.5, 0.8, 0.95] # default: none

Each fraction fires once per period per limit instance, at the moment the folded observed value reaches it, and the set resets when the period rolls. A second rule, enforcement_suspended, fires per enforcing instance when the accountant heartbeat goes stale and resolves when it recovers; it takes no configuration. Both are evaluated on a wakeup from the accountant plus a 30-second poll, and once at startup, so a limit already over a threshold alerts straight away.

Three defaults decide whether anything reaches you, and the first one catches people out. Governance alerting is built as part of the dashboard, so dashboard.enabled, which is false in a config file until you set it, gates both rules: with the dashboard off, a threshold is recorded as crossed and nobody is told. Then thresholds itself defaults to none, so an unannotated limit is watched by nothing. And alerts.channels defaults to empty, so out of the box a crossing publishes to the dashboard alert stream, appends to <dashboard.state_dir>/alerts/governance_history.jsonl (~/.tokentriage/dash by default), and is delivered nowhere else. Add a channel for a webhook, Slack message or email; governance alerts fan out to every configured channel and are independent of alerts.rules.

One shape validates and does nothing: thresholds on an rpm, tpm or concurrency limit. The fractions are range-checked and then ignored, because both rules read the spend and quota accountant, which those three types do not use.

Check the rollup against the ledger

govern reconcile answers one question: does the derived per-scope spend still agree with the canonical ledger? It folds your decision log through the same arithmetic the proxy accounts with, compares each scope against a claimed rollup, and writes nothing back.

$ tokentriage govern reconcile -c tokentriage.yaml --log ~/.tokentriage/decisions.jsonl
govern reconcile — reconcile/canonical-ledger-vs-rollup
  DISPLAY-AND-ALERT ONLY: reconcile recomputes from the canonical ledgers and NEVER adopts (CG7).
  ledger: .../decisions.jsonl
  rows folded: 8
  rollup source: (none — every rollup treated as absent/0)

DRIFT — canonical burn (recomputed from the ledger) − claimed rollup:
  [budget_usd] org_monthly_budget@org:acme
      canonical (ledger)  $0.000037800   (37800 nanodollars)
      rollup   (claimed)  $0.000000000   (0 nanodollars)
      drift               +$0.000037800   (37800 nanodollars)  ⟵ DRIFT

RECONCILE: 1 scope(s) DRIFTED. This is an ALERT, not a fix.
POLICY: NO rollup was adopted, corrected, or written (adopted=false). The canonical ledgers
remain the sole system of record for usage; resolve drift at its source, never by adoption.

That capture is from a sandbox pointed at a stand-in upstream, which is why the dollars are fractions of a cent; the shape is what yours prints. The ledger path is elided.

Two more things before you schedule it. --as-of pins the comparison instant to the rollup’s capture time, so a run near a calendar boundary does not manufacture full-value drift by folding into a fresh period. And on an instance with no decision log yet, the command errors on the fold instead of printing an empty report, which makes a fresh install look like a failure; -c is required too, even though the help text shows a default.

GET /tt/api/v1/governance/reconcile is the read-only twin, and it is narrower than it looks. It runs the same recomputation against an absent rollup, so the reading above applies, but only where a shared-state tier is configured. Without one it answers 200 with no scopes at all and an alert string saying this is a single-node instance, which is the same wiring that empties suspensions. On a single node the CLI is the surface that does the work.

What a reload keeps, and what it drops

“Carry-over” in this product is about hot reloads, not about budgets. Unused budget does not roll into the next period, there is no config key for that, and a period boundary resets the counter to zero.

What is carried is live counter state across a governance re-arm. Editing one limit rebuilds the whole compiled set, so an untouched limit would otherwise restart its window at zero and admit a burst. The rule: a limit whose counter identity is unchanged keeps its counter, and a limit whose identity changed starts fresh.

Nine fields make up that identity: id, type, algorithm, period, amount, burst, store, the whole attach block, and the whole match block. Two things are excluded so that editing them keeps the counter: the posture: block, including thresholds, and the override ceiling. Promoting a limit from shadow to on therefore costs nothing.

The trap is burst on a window algorithm. fixed_window and sliding_window ignore the value entirely, but the carry compares it literally, so a burst-only edit on such a limit drops the counter and buys a window of extra admissions while changing enforcement by nothing. Rate algorithms has the full inert-knob picture.

Which limits participate is narrower than it sounds. Local-store rate limits and concurrency gauges are carried. A store: redis limit keeps its state in Redis and a rebuild does not touch it. A store: share rate limit is skipped, so it does restart its window on every re-arm.

Spend and quota limits ride the accountant instead, with one asymmetry: a budget added by a reload begins at zero for the current period, because only a process restart replays the ledger to backfill it.

A five-minute runbook

Symptom to first look, on the surfaces above. This is the governance slice; the general version is troubleshooting.

symptom	look here first	likely cause
a caller got a 429 from a limit you thought was watching	the limit’s posture: block in your config file	a limit written with no posture: enforces from the moment it exists, and /governance/limits reports off, shadow and soft alike as alert_only
a spend cap acted after the money was gone	enforcement.bound and note on the 429 body	a budget_usd counter folds off the request path, so in-flight and streaming rows are missing from it
traffic is flowing past an enforcing spend cap	suspended on the row first, then suspensions on /governance/health	the accountant heartbeat is stale, so the cap checked nothing and failed open
rate limits admitted a burst right after a reload	the nine counter-identity fields against your diff	an edit changed the identity, so that limit’s counter restarted at zero
keys list shows a key the proxy answers 401 to	the CLI’s --state-dir against governance.store.path	keys opens <state-dir>/governance.db and does not read the config’s store path
govern reconcile reports drifted scopes	the rollup source: line	no --rollup was passed, so every claimed rollup was treated as zero
a budget crossed 80% and nobody heard	dashboard.enabled, then posture.thresholds on the limit, then alerts.channels	governance alerting rides the dashboard, thresholds default to none, and channels default to empty

What this page does not cover

The mechanics behind the health panel, including what converges and the 30-second revocation and 60-second staleness bounds, are governance across replicas. Every endpoint named here, with its parameters and role floor, is the management API. Working back from one 429 to the limit that produced it is why was this request refused, and the field-level contract behind these exports is the refusal wire reference.

Spend analysis, attribution and reporting are cost analytics. The knobs that validate and do nothing are catalogued in what governance does not do.