Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

What governance does not do

Inert config keys, attachments that select nothing, validation that is not a boot guarantee, and fields that read better than they behave.

ExplanationLast updated

On this page

Documentation that only describes what works leaves you to discover the rest during an incident. This page is the other half of the governance section: the settings that load clean and change nothing, the attachments that select no node, the checks that pass before a config that stops the process, and the fields whose names promise more than they carry.

Nothing here is a bug report. Each item is a known boundary of what shipped, and each says what to do instead. Where another page explains one of these in its own context, it is linked from the item.

Config that validates and then does nothing

A setting that loads clean and does nothing is worse than one that is absent, because it reads as configured. Three of those live under governance. The fourth row below fails one step earlier: a named enforcement policy is not a config key at all, so a limit that names one does not load.

key what it looks like what it does
governance.keys.default_expiry (default unset) a lifetime stamped on every key you mint parsed and range-checked at load, then read by nothing. A mint with no --expiry gets no expiry, whatever the file says
governance.keys.prefix (default ttv_) the prefix a minted secret carries consumed by nothing. The prefix a mint writes is a constant in the binary, beside the fixed body length and checksum
burst: on fixed_window or sliding_window (default amount) a shallower allowance than the window rate not read by either limiter, local or Redis-backed. Only gcra and token_bucket carry a burst term
a named bundle of posture: settings one mode, enforce and fail set referenced from several limits not a config key. The schema rejects unknown fields, so a limit naming a policy fails to load. The registry holds a map and a registration helper (EnforcementPolicy, RegisterPolicy) that nothing calls

Both shipped governance example configs set default_expiry: 90d, and a key minted under either one does not lapse. Pass --expiry on the command line and confirm the EXPIRES column in keys list. Virtual keys covers the two governance.keys settings where you meet them.

The inert burst carries a sting the other keys do not. Editing it is not free. burst is one of the nine fields that make up a counter’s identity across a hot reload, and the carry compares it literally. So a burst-only edit on a window limit using the default local store fails that comparison, and the limit restarts its window at zero. You buy one window of extra admissions and change what is enforced by nothing. Leave burst off a window algorithm, or move the limit to gcra, where the value is read. Rate algorithms has the per-algorithm picture.

Shapes that select nothing

These are the expensive ones. The config looks right, govern preview prints the limit as though it were armed, and no surface reports that it fanned out to zero nodes.

attach: { level: key }

A template instantiates at every node that exists at its level, and no key-level node exists. Minting a key does not create one. The config plane emits org, team and user nodes from tenants:, and the runtime nodes table has two writers, the node-create route and the demo seeder, neither of which runs on a mint. The chain a request carries does hold a key leaf, so selection has one side of the intersection and finds nothing on the other.

Be precise about what is possible. The node-create route does not check kind, so a key-level node is not structurally forbidden; you would be hand-creating one node per key, which gives up the reason to write a template at all. Pin the limit at the key’s owner node, or attach at level: user. The commented governance block in configs/tokentriage.example.yaml ships a tpm limit at attach: { level: key }, so the shape has a published example behind it.

attach: { level: global }

global is in the level vocabulary config validation checks against, and nothing creates a node at that level. An org’s parent is the synthetic root rather than a node sitting at a global level, so the template fans out to nothing. The node-create route would take a hand-made one here too, and you do not want that: the spelling that works needs no node at all. attach: { node: __global } is permitted without tree membership.

match.model and match.provider on rpm or concurrency

Those two types are evaluated at the first gate, which resolves with the endpoint alone. The model and provider dims are empty strings at that moment, because the body has not been parsed yet, and a match against an empty string fails, so the limit is dropped before it is instantiated. match.endpoint is fine on both. match.model on tpm is fine too, since a tpm limit is evaluated at the second gate, after the body has been read.

A model glob that stops at a slash

Model patterns go through Go’s path.Match, where * matches inside one path segment and does not cross a /. So llama* misses accounts/fireworks/models/llama-v3-8b, and you want accounts/fireworks/models/llama* instead. The same function backs a key’s model scope, so one mis-written glob can under-select a counter and over-restrict a key at once. The tenancy hierarchy works these through against a live resolution.

Validation that is not a boot guarantee

tokentriage validate does not compile governance limits. Its governance checks stop at the config surface. They cover the id charset, attach naming one of node or level, the enums under posture:, and the amount, period and store each matching what the type is. The guards a limit type owns, and the registry’s store-class rule, run at compile instead, and compile happens at boot. So a governance block can print OK: configuration is valid and take the process down on deploy.

Both of these came from tokentriage run on files that validate had already passed:

Error: governance: compile limits: govern: limit "team_rpm": rpm requires a rate-class window period (e.g. 1s, 1m), not a calendar period
Error: governance: compile limits: govern: limit "team_tpm": tpm limit "team_tpm": burst 500 is below the reserve cap 16384 — an ordinary max-token reserve could never fit (a permanent 100% refusal); raise burst (default is amount 500) to at least the reserve cap

The first message is reachable from the shipped example. examples/governance/tokentriage.governance.yaml declares its team_rpm as type: rpm with no period: key at all, so that file validates clean and stops the boot. A tpm limit binding store: redis behaves the same way: it is turned down at boot by the store-class guard, not at validate, even though the operator guide and the distributed example both state validate catches it.

Fields that do not say what they look like they say

Six surfaces report something adjacent to the question you are asking them. None of them changes what is enforced. Each changes what you conclude, which is the more expensive error mid-incident.

surface what it looks like what it holds
decision.governance.verdict on a row the outcome of the gate allow or would_refuse, and nothing else. A stopped request’s row still reads allow; the stop is recorded in the sibling decision.refused_by_limit block
observed on a concurrency_exceeded body in-flight requests at the moment of the stop a literal 0, whatever the true count. Read limit and treat observed as absent; the live gauge is on GET /tt/v1/me
STATE in keys list the key’s lifecycle state active for a key whose expiry passed months ago. The sweep that would restamp it has no caller in the binary, and expiry is applied at the gate on every request instead
REVOKED in token list whether the token can be used the stored revoked flag alone. A ttk_ whose --ttl lapsed prints false and reads as usable
the output of keys rotate the successor’s terms no expires: line, even when the successor inherited its predecessor’s expiry. keys list shows the truth
N scope(s) DRIFTED from govern reconcile a mismatch worth paging on with no --rollup, every claimed rollup is read as zero, so each scope reports its whole burn as drift. Read the rollup source: line before the closing line

Anyone counting stopped requests out of the ledger reads decision.refused_by_limit, not verdict, and a query filtering on verdict for a refusal value comes back empty. Why was this request refused? shows the row beside the response it belongs to; operating governance shows the reconcile run that produces the alarming line.

Capabilities that are narrower on the way in

Five capabilities exist in the engine and are reachable from less than you would look for: a hand-edited file, Go code, one auth mode, or the CLI but not the management API. Each item names the surface that does reach it.

  • Role bindings. Delegated self-serve is admin-managed, and that concretely means writing a line into role-bindings.json under the dashboard state directory. No API route and no CLI verb creates or removes one, and nothing lists the whole file: the self-serve read route shows a caller only their own bindings. The store re-reads the file on every check, so there is nothing to restart.
  • Delegation under auth mode token. The binding check starts from a username, and a bearer principal carries a role without one, so a delegated call from a ttk_ or tta_ token gets a 403 whatever the file says. Delegation is a users-mode feature.
  • The login limiter. Fifteen attempts per source per one-minute window over at most 8,192 tracked sources, and all three numbers are constants passed at startup. dashboard.auth carries mode and session_ttl and nothing else, so the advice to raise the rate for a shared egress is a rebuild rather than an edit.
  • Per-key guardrail policy. A key record has a guardrail-policy column, a validator, and a clamp applied when the key resolves. Four surfaces mint a key: the CLI, the dashboard route, the delegated route, and the demo seeder. Each fills some mix of label, owner node, scopes and expiry, none of them fills the policy, and keys mint has no flag for one. A policied key is reachable from Go code and from no operator surface.
  • A grace window over the management API. POST /tt/api/v1/governance/keys/{kid}/rotate always rotates with grace 0, so the predecessor goes dark in the same transaction. tokentriage keys rotate <kid> --grace 24h is the way to get a window, and it is what you want when callers are holding the old secret.

Where the disclosure is thinner than the enforcement

A limit can enforce well and report itself badly. Two of those are worth knowing before you build a dashboard or an alert on top of governance, and a third is folded into the disclosure that closes this section.

A converging quota enforces and does not appear on the convergence panel. A quota_tokens or quota_requests limit sums its remote term across replicas, stops requests on the total, and renders no convergence row: that table filters to budget_usd. What reaches you for the two quota types is the convergence_lag_rows term on the 429 body and the row that stop writes.

The tuning behind those numbers is compile-time. Flush cadence at 1s, convergence staleness at 30s and cell retention at 10 minutes are constants the converger takes as its zero-value options at boot. shared_state.redis accepts external, addr, password_env and tls, and none of them moves any of the three. The 30-second key-revocation bound is the same story one tier over: a constant of the postgres mirror, with no knob under shared_state.postgres. Governance across replicas lists the full set with the residuals they imply.

Which algorithm behaviours are pinned by a vector, and which stores each can bind

The rate algorithms are covered unevenly, and the gaps run in a direction worth knowing.

algorithm pinned by conformance vectors store: local store: share store: redis
gcra admit, remaining, retry-after, reset-after, resulting state yes yes yes
fixed_window admit and resulting counter only yes no yes
sliding_window admit and resulting counter only yes no yes
token_bucket no vectors at all yes no no

The window pair’s Remaining, RetryAfter and ResetAfter are the fields a caller backs off on, and no vector asserts them. token_bucket is the same arithmetic as gcra over the same single integer of state, and its correctness rides on GCRA’s vectors, which is why it takes neither of the coordinated stores: it carries no per-step effective-rate form for share and no Lua asset for redis.

One ceiling crosses a store boundary in a way that surprises people. A fixed_window or sliding_window limit above 65,535 is turned down at compile because the local limiter packs its counter into 16 bits, and that check runs whether or not the limit names store: redis, where a Redis integer has no such bound. The fix is to change algorithm to gcra, not to change store.

What governance does not attempt at all

Four questions arrive at this subsystem and belong elsewhere.

  • Which model serves the request is routing, which runs between the two governance gates and cannot overrule a cap.
  • What is inside the request or the response is guardrails, which inspect content rather than counting it.
  • What was spent, by whom, attributed how is cost analytics. Governance reads a spend total to decide admission, and reports it back only against the caps it enforces.
  • What a request costs is the pricing layer’s answer. Governance owns no price table; it compares a cap against a figure something else produced, and an unpriced row is excluded from a budget fold rather than counted as zero.

Per-tenant provider credentials are not a shipped configuration surface either. A tenancy node is a name with a parent, and it carries no credential field, so you cannot attach one team’s OpenAI key to that team’s subtree. The upstream credential is set per upstream, and what keeps attribution answerable is the serving-key block on every row a key served.

What this page does not cover

Gaps in the rest of the product live with their own subjects. Routing’s equivalent list is what routing does not do, what a guard cannot inspect is air gap and limits, and what a cost figure is entitled to claim is the honesty model.

Two governance topics that look like gaps are bounds rather than omissions, and they have their own treatment. Why a spend cap can be crossed before it acts is how a limit counts, and what a fleet’s counters lag by is governance across replicas.