# Turn governance on Take one cap from nothing to enforcing on your own traffic, watching it decide before it stops anything. Source: https://tokentriage.com/docs/build/governance-rollout --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; Someone has handed you a sentence like *make sure the support team cannot spend more than $2,000 a month*, and an afternoon to do it in. Writing the number down is the quick part. The work is arming it without discovering afterwards that it turned away traffic you needed. You get there in six steps. Declare the team, mint it a key, attach the cap in a mode that watches and cannot stop anything, read what it recorded, ask what it would have done to the traffic already in your log, then promote it one rung at a time. Each rung is a config edit you can walk back. You need a proxy already serving traffic and writing a decision log (the default path, `~/.tokentriage/decisions.jsonl`, needs no setup), the `tokentriage` binary on your PATH, and write access to the config file the proxy boots from. The long part is not the typing: it is leaving the cap in shadow until enough traffic has gone through the key to be worth reading. For background, what each limit type counts is [the six limit types](/docs/build/governance-limits), and how a cap resolves through the tree is [the tenancy hierarchy](/docs/build/governance-hierarchy). ## Declare the team and arm governance A cap names a place in a tree, and the tree is something you write down. Start with the tree and the arming switch, and leave the cap itself for two steps' time. ```yaml governance: enabled: true # default false require_key: required # off (default) | optional | required tenants: - org: acme teams: [support, ml] ``` That block materialises `org:acme`, `org:acme/team:support` and `org:acme/team:ml`. Governance arms when any one of three things is true: `enabled: true`, at least one entry under `governance.limits`, or `require_key` set to `optional` or `required`. Declaring a limit is enough on its own, which is worth knowing before you paste one in to see what it looks like. Set `enabled: true` anyway, because the config validator returns early without it and every key inside the block then goes unread. `require_key: required` turns away anonymous traffic with a 401, so every request has to name a node. Under the `off` default an unkeyed request resolves to a chain of one entry, `__global`, where your team cap is not. That is the one edit on this page a caller feels straight away. From the restart on, any caller not yet sending a key gets a 401. If your callers are not holding keys yet, write `require_key: optional` for now, hand the keys out in the next step, and come back for `required` once they are all using them. Either value starts a warning from `tokentriage validate`, because a bearer key crosses cleartext HTTP unless TLS terminates in front of the data plane. Set `governance.keys.tls_terminated_upstream: true` (default `false`) once it does. Nothing probes it, so setting it on a plaintext deployment silences a warning that was right. Restart the proxy against the edited file (`tokentriage run --config tokentriage.yaml`). `governance.enabled` and `governance.require_key` are restart-required; limits and tenants are hot-reloadable, which starts to matter once you add the cap itself. ### Validate is not the check that matters here `tokentriage validate` does not compile governance limits. Its governance checks stop at the config surface: the id charset, `attach` naming one of node or level, the enums under `posture:`, and the amount, period and store each matching what the type is. Every guard that belongs to a limit type runs later, inside that type's own compile. So `OK: configuration is valid` tells you the shape of the block is acceptable, and nothing about whether the process will start. Both of these came from `tokentriage run` on configs that `validate` had already passed: ```text Error: governance: compile limits: govern: limit "team_rpm": rpm requires a rate-class window period (e.g. 1s, 1m), not a calendar period ``` ```text Error: governance: compile limits: govern: limit "team_tpm": tpm limit "team_tpm": burst 500 is below the reserve cap 16384 — an ordinary max-token reserve could never fit (a permanent 100% refusal); raise burst (default is amount 500) to at least the reserve cap ``` A governance block that stops the boot passes `tokentriage validate` clean. The per-type guards run at compile, and compile happens at boot, so a bad limit takes the process down on the deploy rather than on the check. Boot the file somewhere quiet first, or run `tokentriage govern preview -c `, which runs the same compile and prints the same message behind its own `govern preview:` prefix. [Which shapes each command catches](/docs/build/governance-limits#compiling-the-set-before-it-boots) is on the limit-type reference. ## Mint a key for the team The cap will hang off `org:acme/team:support`, and a request reaches that node by presenting a key that names it. ```console $ tokentriage keys mint --owner org:acme/team:ml/user:dev1 --label dev1-laptop --expiry 2160h kid: vk_FEKZdMs2UQRi expires: 2026-11-07T20:41:14Z This secret is shown ONCE and cannot be recovered — copy it now: ttv_EeNpxgdtQYWPmdVoBJQ0r3pi… ``` That capture is from a local sandbox whose store has been destroyed, the secret's tail is elided, and it mints against a different node; put `org:acme/team:support` in your own `--owner`. The flag is `--owner`. The product's own operator guide names `--node` in two places, and no such flag exists. The secret prints once. The store keeps a SHA-256 hash of it and a `stub`, the first four characters after the `ttv_` prefix, which is enough to match a key against the copy its holder kept. The secret itself is not stored, so losing it means rotating rather than looking it up. `--owner` is carried through with no validation. A typo mints cleanly and yields a key whose chain resolves against nodes that do not exist, leaving only `__global` limits attached to it. Confirm a new key with `tokentriage keys list` and one call to `GET /tt/v1/me` before you hand it over. [Virtual keys](/docs/build/governance-keys) covers scoping, rotation with a grace window, and revocation. The `keys` commands open `/governance.db` and do not read `governance.store.path` or `dashboard.state_dir`, even when you pass `--config`. If either config key names a non-default location, a mint lands in a database the proxy does not open, and the key comes back 401 with nothing to explain it. Pass the matching directory to `--state-dir` on every `keys` invocation, or leave both config keys at their defaults. ## Attach the cap in shadow Now write the number, in the one mode that watches and cannot stop a request. ```yaml governance: limits: - id: support_monthly_budget attach: { node: "org:acme/team:support" } type: budget_usd amount_usd: 2000 # dollars, on the nanodollar grid period: month # day | week | month posture: { mode: shadow } # evaluated live, records, stops nothing ``` Shadow is an evaluation against live traffic, not a simulation. The limit reads the same counter an enforcing one reads, and a breach appends the limit id to `would_refuse` on that request's decision row. The one thing it cannot do is stop the request. Shadow is something you type. A limit written with no `posture:` block is enforcing, hard and fail-open from the moment it exists, because the defaults filled in when it resolves turn an unset mode into `on` and an unset `enforce` into `hard`. There is no declared-but-idle state to land in by accident, so a limit pasted from an example with its `posture:` trimmed off starts turning traffic away on the first breach. Write `mode: shadow` on every new limit and take it off later. Adding this limit is where the reload rules bite. `governance.limits.*` is hot, but adding the first spend or quota limit when no accountant was armed at boot is turned down with `adding a ledger-settled limit requires a restart (no accountant was armed at boot)`. Restart, and you get the better outcome anyway: a restart backfills the current period from your ledger, while a hot add starts the counter at zero and accounts from the reload instant. ## Read what shadow recorded Leave it running for as long as your traffic needs to be representative, then read the same thing from two ends. The row is the per-request end. This is the governance block of a decision row for a fourth request against a two-per-minute `rpm` limit in shadow, served 200: ```json { "kid": "vk_FEKZdMs2UQRi", "node": "org:acme/team:ml/user:dev1", "verdict": "would_refuse", "would_refuse": ["team_rpm"], "user_asserted": false, "rotation_grace": false, "reserved_tokens": 16391, "serving_key": { "kid": "vk_FEKZdMs2UQRi", "node": "org:acme/team:ml/user:dev1" }, "gate_us": { "a": 529, "b": 0 } } ``` `verdict` carries one of two values, `allow` and `would_refuse`, and the sibling `would_refuse` array names every limit that was over its cap. The summary is the other end. `GET /tt/api/v1/governance/shadow` on the admin port, with a `ttk_` token of viewer role or better, counts them for you: ```json { "shadow": true, "note": "shadow would-refuse counts: what each mode: shadow limit WOULD have refused had it been enforcing (derived from decision.governance.would_refuse); NOT live refusals", "limits": [ { "limit_id": "team_rpm", "would_refuse_count": 2, "rows_observed": 4 } ] } ``` `would_refuse_count` against `rows_observed` is the decision you came for: two of four rows on that sandbox. Read your own pair and ask whether you would have been content to turn that many requests away. If the answer is no, the edit is the amount or the node, not the mode. ## Preview the cap against traffic you already have Shadow answers what happens from now on. The other question is what this cap would have done to the traffic already in your log, and that one has an answer today. ```console $ tokentriage govern preview -c proposed.yaml --log ~/.tokentriage/decisions.jsonl governance preview — governance_dry_run shadow mode (mode: shadow — real evaluation on live traffic, recorded as would_refuse) is the primary pre-enablement instrument; this is the labeled dry-run over ledger history + live counters (§7.5) ledger: .../decisions.jsonl rows replayed: 5 window: 2026-08-09T20:44:50Z .. 2026-08-09T20:44:50Z EXACT — LedgerSettled recomputation (hand-count-verifiable, exact): [budget_usd] org_monthly_budget @ org:acme burn $0.000013 / limit $1000.000000 (frac 0.0000, rows 1, unpriced_excluded 0) APPROXIMATE — rate-class (label: rate_windows_approximated): [rpm] team_rpm @ org:acme/team:ml approx_peak 1 / amount 300 per 1m0s (rate_windows_approximated) [concurrency] user_concurrency @ level:user CANNOT-ESTIMATE: concurrency requires overlapping-flight reconstruction; the ledger records no reliable per-request flight interval — cannot estimate from history WOULD BE REFUSED RIGHT NOW (exact over the recorded ledger; the LIVE gate may lag by its disclosed overrun bound — in-flight/streaming rows not yet folded): (none) ``` That ran over a five-row sandbox ledger, so read the labels rather than the figures. There are three, and they are not interchangeable. | Label | Covers | What the figure is | |---|---|---| | `EXACT` | `budget_usd`, `quota_tokens`, `quota_requests` | recomputed by folding your ledger through the same accounting the live proxy uses, so you can hand-count it | | `APPROXIMATE` | `rpm`, `tpm` | a reconstructed peak, tagged `rate_windows_approximated`; the ledger records arrivals, not the window state the limiter held | | `CANNOT-ESTIMATE` | `concurrency` | no figure at all; overlapping flight needs a per-request interval the ledger does not carry | The budget you came here to arm is in the first row, which is the one you can check by hand. That is the whole report, printed in full here and nowhere else; the other governance pages quote the line they need and link back to it. Two things about the command itself. `-c` shows a default of `tokentriage.yaml` in its help and is still marked required, so leaving it off fails with `required flag(s) "config" not set`. And on a fresh install with no traffic yet, it errors on the missing log rather than printing an empty report. ## Promote to soft, then to hard Two edits left, and both are hot. `governance.limits.*` reloads without a restart: on a Unix host a `SIGHUP` makes the proxy re-read the config file, and `POST /tt/api/v1/config/apply` on the admin plane does the same on any platform. First, `mode: on` with `enforce: soft`. A breach stops nothing and writes nothing to the decision row. The alert thresholds are what tells you, and they fire once per period as the counter crosses each fraction you list, so put `1` in the list if you want the breach itself to alert. ```yaml - id: support_monthly_budget # … attach/type/amount_usd/period unchanged posture: mode: "on" enforce: soft # a breach lets the request through thresholds: [0.5, 0.8, 0.95] # fractions in [0,1]; default none ``` This rung changes nothing a caller can see, and that is what it is for: your deploy pipeline, your templating and the hot reload get exercised while the outcome stays the same. It costs you one thing. Only `mode: shadow` writes `would_refuse`, so the row field and the `/governance/shadow` counts go quiet the moment you leave shadow. Keep the numbers you read in the previous step. Then `enforce: hard` with `fail: open`, which is what you get when you write no `fail:` at all. ```yaml - id: support_monthly_budget # … attach/type/amount_usd/period unchanged posture: mode: "on" enforce: hard # a breach returns a 429 fail: open # default; keep serving when the cap cannot be checked thresholds: [0.5, 0.8, 0.95] ``` Pairing `fail: closed` with `enforce: soft` is a validation error, since a soft limit has nothing to fail closed on. What `fail: open` chooses when the accountant behind a budget is stale or dead is on [the governance front door](/docs/build/governance#fail-open-with-disclosure). A caller over the cap now gets a 429. This is the response head from a sandbox budget of `amount_usd: 0.000001`, so read the header shape rather than the figures: ```console HTTP/1.1 429 Too Many Requests Content-Type: application/json Ratelimit: "org_monthly_budget";r=0;t=1912510 Retry-After: 1912510 X-Tt-Refusal-Type: budget_exceeded X-Tt-Refused-By: org_monthly_budget X-Tt-Request-Id: tt_i3f2naenug5aanctu6pcjrofcy ``` `Retry-After` on a calendar budget counts the seconds to the next period boundary, 22 days on that capture, and is not a back-off hint. ## Confirm it is doing the work Three surfaces, in the order you will reach for them. `x-tt-refused-by` on the caller's own response names the limit that stopped it, and `x-tt-refusal-type` names the family. A caller can quote both to you without access to anything of yours. `GET /tt/api/v1/governance/refusals` is the feed, viewer role, narrowable with `?node=`. Each item carries the timestamp, the request id, the node, the `kid`, the type, the firing `limit_id` and `limit_type`, and whether the figure behind it was approximate. `GET /tt/api/v1/governance/limits` reports what is armed right now. This definition comes from a sandbox where the budget was declared as an org-level template, so read the fields rather than the amounts; a limit pinned the way yours is carries `"attach":"pinned"` and a `node` in place of `"attach":"template"` and a `level`: ```json {"limit_id":"org_monthly_budget","limit_type":"budget_usd","class":"spend","unit":"usd","period":"month","posture":"enforcing","attach":"template","level":"org","amount":{"usd":1000,"estimated":false,"unpriced":false},"override_allowed":true,"override_max":{"usd":5000,"estimated":false,"unpriced":false}} ``` The `posture` field there is a wire vocabulary of its own, and `alert_only` on it covers every rung below hard: `mode: shadow`, `enforce: soft` and `mode: off` all report that one string, so a limit watching your traffic looks the same as one switched off. Your config file is the one place the difference between them is written down. ## What can go wrong **The key comes back 401 while `keys list` shows it active.** The mint landed in a different `governance.db` than the proxy opened. Compare the CLI's `--state-dir` against `governance.store.path` and `dashboard.state_dir`. **The shadow summary stays empty.** Either the cap was not crossed, or the key's owner node is not the node the limit attaches to. `GET /tt/v1/me` prints the caller's `chain` and its resolved limits; a cap missing from that list will not fire for that caller. **The budget stops a request later than you expected.** A spend counter is folded from ledger rows after each request finishes, so spend already in flight is missing from the number admission reads, and the 429 publishes the gap. [How a limit counts](/docs/build/governance-enforcement-classes) has the arithmetic. **A second budget added by reload starts at zero.** A hot add accounts from the reload instant; only a restart backfills the current period from ledger history. Working back from a 429 you did not expect, to the limit, the node, and the edit that fixes it, is [the walk from a 429 back to its cause](/docs/build/governance-refusals). ## What this page does not cover The path, not the machinery on either side of it. What each limit type counts is [the six limit types](/docs/build/governance-limits). When a counter is written relative to the request, and therefore whether a cap can be crossed, is [how a limit counts](/docs/build/governance-enforcement-classes). Templates, per-node overrides and evaluation order are [the tenancy hierarchy](/docs/build/governance-hierarchy). Running the same caps on more than one replica is [governance across replicas](/docs/build/governance-distributed), and who is permitted to make these edits is [who can administer governance](/docs/build/governance-access-control). [Cap spend per virtual key](/docs/recipes/cap-spend-per-virtual-key) is the single-tenant version of the same task. It pins one budget to one node rather than templating a tree, states the ledger prerequisites for a spend limit, and works through the `fail: open` against `fail: closed` choice in more detail. Reach for it when the answer is one budget on one credential, and for this page when you are arming a tenancy tree and want the dry run and the promotion rungs.