Turn governance on — TokenTriage Docs

Someone has handed you a sentence like make sure the support team cannot spend more than $2,000 a month, and an afternoon to do it in. Writing the number down is the quick part. The work is arming it without discovering afterwards that it turned away traffic you needed.

You get there in six steps. Declare the team, mint it a key, attach the cap in a mode that watches and cannot stop anything, read what it recorded, ask what it would have done to the traffic already in your log, then promote it one rung at a time. Each rung is a config edit you can walk back.

You need a proxy already serving traffic and writing a decision log (the default path, ~/.tokentriage/decisions.jsonl, needs no setup), the tokentriage binary on your PATH, and write access to the config file the proxy boots from. The long part is not the typing: it is leaving the cap in shadow until enough traffic has gone through the key to be worth reading. For background, what each limit type counts is the six limit types, and how a cap resolves through the tree is the tenancy hierarchy.

Declare the team and arm governance

A cap names a place in a tree, and the tree is something you write down. Start with the tree and the arming switch, and leave the cap itself for two steps’ time.

governance:
  enabled: true # default false
  require_key: required # off (default) | optional | required
  tenants:
    - org: acme
      teams: [support, ml]

That block materialises org:acme, org:acme/team:support and org:acme/team:ml.

Governance arms when any one of three things is true: enabled: true, at least one entry under governance.limits, or require_key set to optional or required. Declaring a limit is enough on its own, which is worth knowing before you paste one in to see what it looks like. Set enabled: true anyway, because the config validator returns early without it and every key inside the block then goes unread.

require_key: required turns away anonymous traffic with a 401, so every request has to name a node. Under the off default an unkeyed request resolves to a chain of one entry, __global, where your team cap is not.

That is the one edit on this page a caller feels straight away. From the restart on, any caller not yet sending a key gets a 401. If your callers are not holding keys yet, write require_key: optional for now, hand the keys out in the next step, and come back for required once they are all using them.

Either value starts a warning from tokentriage validate, because a bearer key crosses cleartext HTTP unless TLS terminates in front of the data plane. Set governance.keys.tls_terminated_upstream: true (default false) once it does. Nothing probes it, so setting it on a plaintext deployment silences a warning that was right.

Restart the proxy against the edited file (tokentriage run --config tokentriage.yaml). governance.enabled and governance.require_key are restart-required; limits and tenants are hot-reloadable, which starts to matter once you add the cap itself.

Validate is not the check that matters here

tokentriage validate does not compile governance limits. Its governance checks stop at the config surface: the id charset, attach naming one of node or level, the enums under posture:, and the amount, period and store each matching what the type is. Every guard that belongs to a limit type runs later, inside that type’s own compile. So OK: configuration is valid tells you the shape of the block is acceptable, and nothing about whether the process will start.

Both of these came from tokentriage run on configs that validate had already passed:

Error: governance: compile limits: govern: limit "team_rpm": rpm requires a rate-class window period (e.g. 1s, 1m), not a calendar period

Error: governance: compile limits: govern: limit "team_tpm": tpm limit "team_tpm": burst 500 is below the reserve cap 16384 — an ordinary max-token reserve could never fit (a permanent 100% refusal); raise burst (default is amount 500) to at least the reserve cap

Mint a key for the team

The cap will hang off org:acme/team:support, and a request reaches that node by presenting a key that names it.

$ tokentriage keys mint --owner org:acme/team:ml/user:dev1 --label dev1-laptop --expiry 2160h
kid:     vk_FEKZdMs2UQRi
expires: 2026-11-07T20:41:14Z
This secret is shown ONCE and cannot be recovered — copy it now:
ttv_EeNpxgdtQYWPmdVoBJQ0r3pi…

That capture is from a local sandbox whose store has been destroyed, the secret’s tail is elided, and it mints against a different node; put org:acme/team:support in your own --owner. The flag is --owner. The product’s own operator guide names --node in two places, and no such flag exists.

The secret prints once. The store keeps a SHA-256 hash of it and a stub, the first four characters after the ttv_ prefix, which is enough to match a key against the copy its holder kept. The secret itself is not stored, so losing it means rotating rather than looking it up.

--owner is carried through with no validation. A typo mints cleanly and yields a key whose chain resolves against nodes that do not exist, leaving only __global limits attached to it. Confirm a new key with tokentriage keys list and one call to GET /tt/v1/me before you hand it over. Virtual keys covers scoping, rotation with a grace window, and revocation.

Attach the cap in shadow

Now write the number, in the one mode that watches and cannot stop a request.

governance:
  limits:
    - id: support_monthly_budget
      attach: { node: "org:acme/team:support" }
      type: budget_usd
      amount_usd: 2000 # dollars, on the nanodollar grid
      period: month # day | week | month
      posture: { mode: shadow } # evaluated live, records, stops nothing

Shadow is an evaluation against live traffic, not a simulation. The limit reads the same counter an enforcing one reads, and a breach appends the limit id to would_refuse on that request’s decision row. The one thing it cannot do is stop the request.

Adding this limit is where the reload rules bite. governance.limits.* is hot, but adding the first spend or quota limit when no accountant was armed at boot is turned down with adding a ledger-settled limit requires a restart (no accountant was armed at boot). Restart, and you get the better outcome anyway: a restart backfills the current period from your ledger, while a hot add starts the counter at zero and accounts from the reload instant.

Read what shadow recorded

Leave it running for as long as your traffic needs to be representative, then read the same thing from two ends.

The row is the per-request end. This is the governance block of a decision row for a fourth request against a two-per-minute rpm limit in shadow, served 200:

{
  "kid": "vk_FEKZdMs2UQRi",
  "node": "org:acme/team:ml/user:dev1",
  "verdict": "would_refuse",
  "would_refuse": ["team_rpm"],
  "user_asserted": false,
  "rotation_grace": false,
  "reserved_tokens": 16391,
  "serving_key": { "kid": "vk_FEKZdMs2UQRi", "node": "org:acme/team:ml/user:dev1" },
  "gate_us": { "a": 529, "b": 0 }
}

verdict carries one of two values, allow and would_refuse, and the sibling would_refuse array names every limit that was over its cap.

The summary is the other end. GET /tt/api/v1/governance/shadow on the admin port, with a ttk_ token of viewer role or better, counts them for you:

{
  "shadow": true,
  "note": "shadow would-refuse counts: what each mode: shadow limit WOULD have refused had it been enforcing (derived from decision.governance.would_refuse); NOT live refusals",
  "limits": [ { "limit_id": "team_rpm", "would_refuse_count": 2, "rows_observed": 4 } ]
}

would_refuse_count against rows_observed is the decision you came for: two of four rows on that sandbox. Read your own pair and ask whether you would have been content to turn that many requests away. If the answer is no, the edit is the amount or the node, not the mode.

Preview the cap against traffic you already have

Shadow answers what happens from now on. The other question is what this cap would have done to the traffic already in your log, and that one has an answer today.

$ tokentriage govern preview -c proposed.yaml --log ~/.tokentriage/decisions.jsonl
governance preview — governance_dry_run
  shadow mode (mode: shadow — real evaluation on live traffic, recorded as would_refuse) is the primary pre-enablement instrument; this is the labeled dry-run over ledger history + live counters (§7.5)
  ledger: .../decisions.jsonl
  rows replayed: 5  window: 2026-08-09T20:44:50Z .. 2026-08-09T20:44:50Z

EXACT — LedgerSettled recomputation (hand-count-verifiable, exact):
  [budget_usd] org_monthly_budget @ org:acme  burn $0.000013 / limit $1000.000000  (frac 0.0000, rows 1, unpriced_excluded 0)

APPROXIMATE — rate-class (label: rate_windows_approximated):
  [rpm] team_rpm @ org:acme/team:ml  approx_peak 1 / amount 300 per 1m0s  (rate_windows_approximated)
  [concurrency] user_concurrency @ level:user  CANNOT-ESTIMATE: concurrency requires overlapping-flight reconstruction; the ledger records no reliable per-request flight interval — cannot estimate from history

WOULD BE REFUSED RIGHT NOW (exact over the recorded ledger; the LIVE gate may lag
by its disclosed overrun bound — in-flight/streaming rows not yet folded):
  (none)

That ran over a five-row sandbox ledger, so read the labels rather than the figures. There are three, and they are not interchangeable.

Label	Covers	What the figure is
`EXACT`	`budget_usd`, `quota_tokens`, `quota_requests`	recomputed by folding your ledger through the same accounting the live proxy uses, so you can hand-count it
`APPROXIMATE`	`rpm`, `tpm`	a reconstructed peak, tagged `rate_windows_approximated`; the ledger records arrivals, not the window state the limiter held
`CANNOT-ESTIMATE`	`concurrency`	no figure at all; overlapping flight needs a per-request interval the ledger does not carry

The budget you came here to arm is in the first row, which is the one you can check by hand. That is the whole report, printed in full here and nowhere else; the other governance pages quote the line they need and link back to it.

Two things about the command itself. -c shows a default of tokentriage.yaml in its help and is still marked required, so leaving it off fails with required flag(s) "config" not set. And on a fresh install with no traffic yet, it errors on the missing log rather than printing an empty report.

Promote to soft, then to hard

Two edits left, and both are hot. governance.limits.* reloads without a restart: on a Unix host a SIGHUP makes the proxy re-read the config file, and POST /tt/api/v1/config/apply on the admin plane does the same on any platform.

First, mode: on with enforce: soft. A breach stops nothing and writes nothing to the decision row. The alert thresholds are what tells you, and they fire once per period as the counter crosses each fraction you list, so put 1 in the list if you want the breach itself to alert.

    - id: support_monthly_budget
      # … attach/type/amount_usd/period unchanged
      posture:
        mode: "on"
        enforce: soft # a breach lets the request through
        thresholds: [0.5, 0.8, 0.95] # fractions in [0,1]; default none

This rung changes nothing a caller can see, and that is what it is for: your deploy pipeline, your templating and the hot reload get exercised while the outcome stays the same. It costs you one thing. Only mode: shadow writes would_refuse, so the row field and the /governance/shadow counts go quiet the moment you leave shadow. Keep the numbers you read in the previous step.

Then enforce: hard with fail: open, which is what you get when you write no fail: at all.

    - id: support_monthly_budget
      # … attach/type/amount_usd/period unchanged
      posture:
        mode: "on"
        enforce: hard # a breach returns a 429
        fail: open # default; keep serving when the cap cannot be checked
        thresholds: [0.5, 0.8, 0.95]

Pairing fail: closed with enforce: soft is a validation error, since a soft limit has nothing to fail closed on. What fail: open chooses when the accountant behind a budget is stale or dead is on the governance front door.

A caller over the cap now gets a 429. This is the response head from a sandbox budget of amount_usd: 0.000001, so read the header shape rather than the figures:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Ratelimit: "org_monthly_budget";r=0;t=1912510
Retry-After: 1912510
X-Tt-Refusal-Type: budget_exceeded
X-Tt-Refused-By: org_monthly_budget
X-Tt-Request-Id: tt_i3f2naenug5aanctu6pcjrofcy

Retry-After on a calendar budget counts the seconds to the next period boundary, 22 days on that capture, and is not a back-off hint.

Confirm it is doing the work

Three surfaces, in the order you will reach for them.

x-tt-refused-by on the caller’s own response names the limit that stopped it, and x-tt-refusal-type names the family. A caller can quote both to you without access to anything of yours.

GET /tt/api/v1/governance/refusals is the feed, viewer role, narrowable with ?node=. Each item carries the timestamp, the request id, the node, the kid, the type, the firing limit_id and limit_type, and whether the figure behind it was approximate.

GET /tt/api/v1/governance/limits reports what is armed right now. This definition comes from a sandbox where the budget was declared as an org-level template, so read the fields rather than the amounts; a limit pinned the way yours is carries "attach":"pinned" and a node in place of "attach":"template" and a level:

{"limit_id":"org_monthly_budget","limit_type":"budget_usd","class":"spend","unit":"usd","period":"month","posture":"enforcing","attach":"template","level":"org","amount":{"usd":1000,"estimated":false,"unpriced":false},"override_allowed":true,"override_max":{"usd":5000,"estimated":false,"unpriced":false}}

The posture field there is a wire vocabulary of its own, and alert_only on it covers every rung below hard: mode: shadow, enforce: soft and mode: off all report that one string, so a limit watching your traffic looks the same as one switched off. Your config file is the one place the difference between them is written down.

What can go wrong

The key comes back 401 while keys list shows it active. The mint landed in a different governance.db than the proxy opened. Compare the CLI’s --state-dir against governance.store.path and dashboard.state_dir.

The shadow summary stays empty. Either the cap was not crossed, or the key’s owner node is not the node the limit attaches to. GET /tt/v1/me prints the caller’s chain and its resolved limits; a cap missing from that list will not fire for that caller.

The budget stops a request later than you expected. A spend counter is folded from ledger rows after each request finishes, so spend already in flight is missing from the number admission reads, and the 429 publishes the gap. How a limit counts has the arithmetic.

A second budget added by reload starts at zero. A hot add accounts from the reload instant; only a restart backfills the current period from ledger history.

Working back from a 429 you did not expect, to the limit, the node, and the edit that fixes it, is the walk from a 429 back to its cause.

What this page does not cover

The path, not the machinery on either side of it. What each limit type counts is the six limit types. When a counter is written relative to the request, and therefore whether a cap can be crossed, is how a limit counts. Templates, per-node overrides and evaluation order are the tenancy hierarchy. Running the same caps on more than one replica is governance across replicas, and who is permitted to make these edits is who can administer governance.

Cap spend per virtual key is the single-tenant version of the same task. It pins one budget to one node rather than templating a tree, states the ledger prerequisites for a spend limit, and works through the fail: open against fail: closed choice in more detail. Reach for it when the answer is one budget on one credential, and for this page when you are arming a tenancy tree and want the dry run and the promotion rungs.