What governance does not do
Inert config keys, attachments that select nothing, validation that is not a boot guarantee, and fields that read better than they behave.
On this page
Documentation that only describes what works leaves you to discover the rest during an incident. This page is the other half of the governance section: the settings that load clean and change nothing, the attachments that select no node, the checks that pass before a config that stops the process, and the fields whose names promise more than they carry.
Nothing here is a bug report. Each item is a known boundary of what shipped, and each says what to do instead. Where another page explains one of these in its own context, it is linked from the item.
Config that validates and then does nothing
A setting that loads clean and does nothing is worse than one that is absent, because it
reads as configured. Three of those live under governance. The fourth row below fails
one step earlier: a named enforcement policy is not a config key at all, so a limit that
names one does not load.
| key | what it looks like | what it does |
|---|---|---|
governance.keys.default_expiry (default unset) |
a lifetime stamped on every key you mint | parsed and range-checked at load, then read by nothing. A mint with no --expiry gets no expiry, whatever the file says |
governance.keys.prefix (default ttv_) |
the prefix a minted secret carries | consumed by nothing. The prefix a mint writes is a constant in the binary, beside the fixed body length and checksum |
burst: on fixed_window or sliding_window (default amount) |
a shallower allowance than the window rate | not read by either limiter, local or Redis-backed. Only gcra and token_bucket carry a burst term |
a named bundle of posture: settings |
one mode, enforce and fail set referenced from several limits | not a config key. The schema rejects unknown fields, so a limit naming a policy fails to load. The registry holds a map and a registration helper (EnforcementPolicy, RegisterPolicy) that nothing calls |
Both shipped governance example configs set default_expiry: 90d, and a key minted under
either one does not lapse. Pass --expiry on the command line and confirm the EXPIRES
column in keys list.
Virtual keys
covers the two governance.keys settings where you meet them.
The inert burst carries a sting the other keys do not. Editing it is not free. burst is
one of the nine fields that make up a counter’s identity across a hot reload, and the carry
compares it literally. So a burst-only edit on a window limit using the default local store
fails that comparison, and the limit restarts its window at zero. You buy one window of extra
admissions and change what is enforced by nothing. Leave burst off a window algorithm, or
move the limit to gcra, where the value is read.
Rate algorithms
has the per-algorithm picture.
Shapes that select nothing
These are the expensive ones. The config looks right, govern preview prints the limit as
though it were armed, and no surface reports that it fanned out to zero nodes.
attach: { level: key }
A template instantiates at every node that exists at its level, and no key-level node exists.
Minting a key does not create one. The config plane emits org, team and user nodes from
tenants:, and the runtime nodes table has two writers, the node-create route and the demo
seeder, neither of which runs on a mint. The chain a request carries does hold a key leaf, so
selection has one side of the intersection and finds nothing on the other.
Be precise about what is possible. The node-create route does not check kind, so a
key-level node is not structurally forbidden; you would be hand-creating one node per key,
which gives up the reason to write a template at all. Pin the limit at the key’s owner node,
or attach at level: user. The commented governance block in
configs/tokentriage.example.yaml ships a tpm limit at attach: { level: key }, so the
shape has a published example behind it.
attach: { level: global }
global is in the level vocabulary config validation checks against, and nothing creates a
node at that level. An org’s parent is the synthetic root rather than a node sitting at a
global level, so the template fans out to nothing. The node-create route would take a
hand-made one here too, and you do not want that: the spelling that works needs no node at
all. attach: { node: __global } is permitted without tree membership.
match.model and match.provider on rpm or concurrency
Those two types are evaluated at the first gate, which resolves with the endpoint alone. The
model and provider dims are empty strings at that moment, because the body has not been
parsed yet, and a match against an empty string fails, so the limit is dropped before it is
instantiated. match.endpoint is fine on both. match.model on tpm is fine too, since a
tpm limit is evaluated at the second gate, after the body has been read.
A model glob that stops at a slash
Model patterns go through Go’s path.Match, where * matches inside one path segment and
does not cross a /. So llama* misses accounts/fireworks/models/llama-v3-8b, and you want
accounts/fireworks/models/llama* instead. The same function backs a key’s model scope, so
one mis-written glob can under-select a counter and over-restrict a key at once.
The tenancy hierarchy works
these through against a live resolution.
Validation that is not a boot guarantee
tokentriage validate does not compile governance limits. Its governance checks stop at the
config surface. They cover the id charset, attach naming one of node or level, the enums
under posture:, and the amount, period and store each matching what the type is. The guards
a limit type owns, and the registry’s store-class rule, run at compile instead, and compile
happens at boot. So a governance block can print OK: configuration is valid and take the
process down on deploy.
Both of these came from tokentriage run on files that validate had already passed:
Error: governance: compile limits: govern: limit "team_rpm": rpm requires a rate-class window period (e.g. 1s, 1m), not a calendar period
Error: governance: compile limits: govern: limit "team_tpm": tpm limit "team_tpm": burst 500 is below the reserve cap 16384 — an ordinary max-token reserve could never fit (a permanent 100% refusal); raise burst (default is amount 500) to at least the reserve cap
The first message is reachable from the shipped example. examples/governance/tokentriage.governance.yaml
declares its team_rpm as type: rpm with no period: key at all, so that file validates
clean and stops the boot. A tpm limit binding store: redis behaves the same way: it is
turned down at boot by the store-class guard, not at validate, even though the operator guide
and the distributed example both state validate catches it.
Fields that do not say what they look like they say
Six surfaces report something adjacent to the question you are asking them. None of them changes what is enforced. Each changes what you conclude, which is the more expensive error mid-incident.
| surface | what it looks like | what it holds |
|---|---|---|
decision.governance.verdict on a row |
the outcome of the gate | allow or would_refuse, and nothing else. A stopped request’s row still reads allow; the stop is recorded in the sibling decision.refused_by_limit block |
observed on a concurrency_exceeded body |
in-flight requests at the moment of the stop | a literal 0, whatever the true count. Read limit and treat observed as absent; the live gauge is on GET /tt/v1/me |
STATE in keys list |
the key’s lifecycle state | active for a key whose expiry passed months ago. The sweep that would restamp it has no caller in the binary, and expiry is applied at the gate on every request instead |
REVOKED in token list |
whether the token can be used | the stored revoked flag alone. A ttk_ whose --ttl lapsed prints false and reads as usable |
the output of keys rotate |
the successor’s terms | no expires: line, even when the successor inherited its predecessor’s expiry. keys list shows the truth |
N scope(s) DRIFTED from govern reconcile |
a mismatch worth paging on | with no --rollup, every claimed rollup is read as zero, so each scope reports its whole burn as drift. Read the rollup source: line before the closing line |
Anyone counting stopped requests out of the ledger reads decision.refused_by_limit, not
verdict, and a query filtering on verdict for a refusal value comes back empty.
Why was this request refused?
shows the row beside the response it belongs to;
operating governance
shows the reconcile run that produces the alarming line.
Capabilities that are narrower on the way in
Five capabilities exist in the engine and are reachable from less than you would look for: a hand-edited file, Go code, one auth mode, or the CLI but not the management API. Each item names the surface that does reach it.
- Role bindings. Delegated self-serve is admin-managed, and that concretely means writing
a line into
role-bindings.jsonunder the dashboard state directory. No API route and no CLI verb creates or removes one, and nothing lists the whole file: the self-serve read route shows a caller only their own bindings. The store re-reads the file on every check, so there is nothing to restart. - Delegation under auth mode
token. The binding check starts from a username, and a bearer principal carries a role without one, so a delegated call from attk_ortta_token gets a 403 whatever the file says. Delegation is ausers-mode feature. - The login limiter. Fifteen attempts per source per one-minute window over at most 8,192
tracked sources, and all three numbers are constants passed at startup.
dashboard.authcarriesmodeandsession_ttland nothing else, so the advice to raise the rate for a shared egress is a rebuild rather than an edit. - Per-key guardrail policy. A key record has a guardrail-policy column, a validator, and a
clamp applied when the key resolves. Four surfaces mint a key: the CLI, the dashboard route,
the delegated route, and the demo seeder. Each fills some mix of label, owner node, scopes
and expiry, none of them fills the policy, and
keys minthas no flag for one. A policied key is reachable from Go code and from no operator surface. - A grace window over the management API.
POST /tt/api/v1/governance/keys/{kid}/rotatealways rotates with grace0, so the predecessor goes dark in the same transaction.tokentriage keys rotate <kid> --grace 24his the way to get a window, and it is what you want when callers are holding the old secret.
Where the disclosure is thinner than the enforcement
A limit can enforce well and report itself badly. Two of those are worth knowing before you build a dashboard or an alert on top of governance, and a third is folded into the disclosure that closes this section.
A converging quota enforces and does not appear on the convergence panel. A
quota_tokens or quota_requests limit sums its remote term across replicas, stops requests
on the total, and renders no convergence row: that table filters to budget_usd. What reaches
you for the two quota types is the convergence_lag_rows term on the 429 body and the row that
stop writes.
The tuning behind those numbers is compile-time. Flush cadence at 1s, convergence
staleness at 30s and cell retention at 10 minutes are constants the converger takes as its
zero-value options at boot. shared_state.redis accepts external, addr, password_env
and tls, and none of them moves any of the three. The 30-second key-revocation bound is the
same story one tier over: a constant of the postgres mirror, with no knob under
shared_state.postgres.
Governance across replicas
lists the full set with the residuals they imply.
Which algorithm behaviours are pinned by a vector, and which stores each can bind
The rate algorithms are covered unevenly, and the gaps run in a direction worth knowing.
| algorithm | pinned by conformance vectors | store: local |
store: share |
store: redis |
|---|---|---|---|---|
gcra |
admit, remaining, retry-after, reset-after, resulting state | yes | yes | yes |
fixed_window |
admit and resulting counter only | yes | no | yes |
sliding_window |
admit and resulting counter only | yes | no | yes |
token_bucket |
no vectors at all | yes | no | no |
The window pair’s Remaining, RetryAfter and ResetAfter are the fields a caller backs off
on, and no vector asserts them. token_bucket is the same arithmetic as gcra over the same
single integer of state, and its correctness rides on GCRA’s vectors, which is why it takes
neither of the coordinated stores: it carries no per-step effective-rate form for share and
no Lua asset for redis.
One ceiling crosses a store boundary in a way that surprises people. A fixed_window or
sliding_window limit above 65,535 is turned down at compile because the local limiter packs
its counter into 16 bits, and that check runs whether or not the limit names store: redis,
where a Redis integer has no such bound. The fix is to change algorithm to gcra, not to
change store.
What governance does not attempt at all
Four questions arrive at this subsystem and belong elsewhere.
- Which model serves the request is routing, which runs between the two governance gates and cannot overrule a cap.
- What is inside the request or the response is guardrails, which inspect content rather than counting it.
- What was spent, by whom, attributed how is cost analytics. Governance reads a spend total to decide admission, and reports it back only against the caps it enforces.
- What a request costs is the pricing layer’s answer. Governance owns no price table; it compares a cap against a figure something else produced, and an unpriced row is excluded from a budget fold rather than counted as zero.
Per-tenant provider credentials are not a shipped configuration surface either. A tenancy node is a name with a parent, and it carries no credential field, so you cannot attach one team’s OpenAI key to that team’s subtree. The upstream credential is set per upstream, and what keeps attribution answerable is the serving-key block on every row a key served.
What this page does not cover
Gaps in the rest of the product live with their own subjects. Routing’s equivalent list is what routing does not do, what a guard cannot inspect is air gap and limits, and what a cost figure is entitled to claim is the honesty model.
Two governance topics that look like gaps are bounds rather than omissions, and they have their own treatment. Why a spend cap can be crossed before it acts is how a limit counts, and what a fleet’s counters lag by is governance across replicas.