# Governance across replicas What converges across a fleet, what stays per replica, and the bound each store carries when it cannot be exact Source: https://tokentriage.com/docs/build/governance-distributed --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import Screenshot from '../../../components/Screenshot.astro'; import GovernanceConvergeDiagram from '../../../components/GovernanceConvergeDiagram.astro'; You run three pods behind a load balancer. A cap of 600 requests a minute, written once and deployed three times, admits 1,800, because each pod counts its own arrivals and cannot see the other two. The same arithmetic applies to a monthly budget, a token quota and an in-flight cap. Coordination closes some of that gap, and it costs you a network dependency for the part it closes. Nothing here is on out of the box, and no limit goes fleet-wide because you scaled up. You pick a coordination mode per limit, and each mode carries a different bound. This page covers which limits converge, what selects that mode, and which direction the remaining error runs. Running the fleet itself, the replica roster and the read-time ledger merge are in [multi-replica and scale-out](/docs/operate/scale-out). ## Nothing dials until you ask for it The cost of the distributed tier is an outbound connection, so the condition that creates one is worth knowing precisely. With no `shared_state` block, the binary opens no coordination connection at all. Every limit falls to `store: local`, its default. No convergence loop is attached, no fleet membership is read, and no coordination header is written. A pod in a fleet of ten behaves like a single node. Two consent gates open the tier, each a literal string. `shared_state.redis` needs `external: allow` plus an `addr`; `shared_state.postgres` needs `external: allow` plus a `dsn_env` naming the environment variable that holds the DSN. An address without the consent string is a validation error rather than a quiet skip. ```yaml shared_state: redis: external: allow # without this, nothing dials addr: "10.0.0.10:6379" password_env: TT_REDIS_PASSWORD # the env var name, not the secret tls: true governance: enabled: true limits: - id: org_rpm_fleet attach: { node: __global } type: rpm amount: 600 period: 1m # a rate limit with no period stops the boot algorithm: gcra store: redis # default is local posture: { mode: shadow } ``` Granting the consent produces a warning every time you validate, and the warning names what you gave up: ```console $ tokentriage validate --config fleet.yaml warning: shared_state.redis: a network redis at "10.0.0.10:6379" is enabled — the single-binary air-gap posture is broken for the coordination tier (disclosed, §2.8) warning: shared_state.redis: a network redis at "10.0.0.10:6379" is enabled — the single-binary air-gap posture is broken for the coordination tier (disclosed, §2.8) OK: configuration is valid ``` That capture is from a source build against the config above, in a file that also carries the usual `server:` and `upstreams:` blocks, with the summary lines trimmed. The warning prints twice because validation runs once when the config loads and again when the limit registries are built. Even after consent the client is lazy: constructing it dials nothing, and the first operation opens the socket. A single node says so on `GET /tt/api/v1/governance/health`, in its own words: ```json { "kind": "governance/distributed-health", "capability": { "shared_state_enabled": false, "tiers": [], "note": "air-gapped: no shared-state tier configured — this is a single-node instance; distributed limits, cross-replica convergence, and reconcile are not active" }, "suspensions": [], "convergence": [] } ``` ## What converges, and what each store does Coordination is not one feature. Spend and quota converge through the accountant, the off-path worker that folds finished ledger rows into counters; rate and in-flight limits go through a counter store you name on the limit. Reading the two as one thing is the fastest way to get the bound wrong. | type | how it goes fleet-wide | what selects it | which way the error runs | |---|---|---|---| | `budget_usd` | converges: each replica publishes its own burn cell and folds in the others | `shared_state.redis` plus any of these three armed; a `store:` key here is a config error | over-admits by whatever peers burned since their last publish | | `quota_tokens` | the same fold, metered in tokens | as above | as above | | `quota_requests` | the same fold, metered in forwarded rows | as above | as above | | `rpm` | one atomic script step against the shared counter | `store: redis` | tight while the store answers, up to N×limit while it does not | | `rpm`, `tpm` | each replica enforces `ceil(limit / N_live)` | `store: share` | over-admits by up to N−1 from rounding | | `concurrency` | each replica caps its gauge at `ceil(C / N_live)` | `store: share` | the same rounding term, on in-flight slots | | `tpm`, `concurrency` | no shared-counter mode | `store: redis` is turned down | `tpm` at the compile, `concurrency` at validate | Each store carries its own bound as a fixed string. Only one of the three reaches the wire: the limits API renders an `enforcement` block for `store: share` and omits it for `local` and `redis`, whose strings stay inside the binary. | store | the bound it carries | |---|---| | `local` | `per-replica only: fleet ≤ N×limit` | | `redis` | `exact; ≤ N×limit during store outage` | | `share` | `fleet ≤ N×ceil(limit/N_live) ≈ limit (over-admission ≤ N−1 from ÷N rounding, + joiners×share per liveness window on scale-up); under-service under LB skew` | | `share`, membership stale | `membership_stale: N decayed to self — fleet ≤ N×limit until the coordination store recovers` | | `share` on `concurrency` | `per-replica cap = ceil(C/N_live); fleet in-flight ≤ N×ceil(C/N_live) ≈ C (over-admission ≤ N−1 from ÷N rounding)` | The trailing clause on the share bound names load-balancer skew: if traffic does not split evenly, one replica can hit its share while the fleet as a whole is under the limit. The last row is the same arithmetic in the concurrency limit's own words, counted in in-flight slots. **Spend and quota converge, and you write nothing on the limit to get it.** The three types whose counter is settled from ledger rows reject a `store:` key outright; [how a limit counts](/docs/build/governance-enforcement-classes) draws that class. Convergence attaches when a Redis tier exists and at least one of the three is armed, and the mechanism is worth a picture. Read it left to right. Each replica meters its own burn from its own ledger, and once per flush it writes one field of a shared per-scope hash and reads the whole hash back. The write sets that replica's own field, keyed on its replica id, rather than incrementing a shared total, so a retried flush or a restart under the same id rewrites the same value instead of adding a second copy. The replica then takes `remote = combined − own`, and admission compares local plus remote against the cap. The cadence is one second, so a peer's last second of burn is not folded yet, and that unfolded burn is what a 429 publishes as `convergence_lag_rows` beside the cap it enforced. The quota classes ride the same fold in their own meter: tokens for `quota_tokens`, forwarded rows for `quota_requests`. The remote term lands in that meter and not in dollars, so a token count is not rendered as money anywhere. The validator's old per-replica quota warning was retired when that shipped, though the shipped distributed example config still carries the retired text in a comment. A limit is fleet-wide because its config says so, not because you added replicas. Where the config says nothing, every counter is per replica and the store's bound reads `per-replica only: fleet ≤ N×limit`. **`rpm` with `store: redis` is the one fleet-exact counter.** Each admission is a single atomic script step against the shared counter, timed by the Redis server clock, so your replicas do not have to agree on a clock among themselves. The key is a fixed prefix plus a hash of the scope key, so tenant identifiers stay out of the shared keyspace. One round trip is bounded at 5 ms, and a slower answer falls back per verdict rather than waiting. **`tpm` with `store: redis` is turned down, and later than you would expect.** The distributed script is acquire-only, with no signed settlement, so a token reserve could not be corrected against actual usage in either direction. That check lives in the limit compiler, which runs at boot and inside `govern preview`, not inside `tokentriage validate`. `type: tpm` with `store: redis` passes `tokentriage validate` with `OK: configuration is valid` and then stops the process at boot. Run `tokentriage govern preview -c --log ` before you deploy, which runs the same compile. Then use `store: share` for a fleet-wide `tpm` cap, or the default `local` to keep that cap per replica. ```console $ tokentriage govern preview -c tpm-redis.yaml --log ./decisions.jsonl Error: govern preview: govern: limit "org_tpm_fleet": type "tpm" (reserve/reconcile) cannot bind store "redis" — the distributed rate store is acquire-only (the reserve→reconcile settlement is not implemented for it), so a per-request token reconcile would mis-enforce in both directions; use the default local store or store: share Error: govern preview: govern: limit "org_tpm_fleet": type "tpm" (reserve/reconcile) cannot bind store "redis" — the distributed rate store is acquire-only (the reserve→reconcile settlement is not implemented for it), so a per-request token reconcile would mis-enforce in both directions; use the default local store or store: share ``` The message lands on stderr twice: the command prints its own error and the process prints it again on the way out. Boot prints the same text behind a `governance: compile limits:` prefix and exits. `concurrency` with `store: redis` is turned down earlier, at validate, and the message points at the share cap instead. It runs no counter store at all: the cap is a live gauge in memory, and `store: share` only divides that gauge. ## The share store divides the limit by the live fleet A third store name buys a fleet-wide ceiling without a round trip: `store: share`. It gives `rpm`, `tpm` and `concurrency` an approximate fleet mode with no coordination call on the request path. Each replica enforces `ceil(limit / N_live)`, where `N_live` is the live replica count read from heartbeat membership as one atomic load. Six hundred requests a minute across three live replicas becomes 200 per replica. For `concurrency` there is no counter to share, so the same membership drives a cap instead: each replica admits `ceil(C / N_live)` in flight. The rate types also require `algorithm: gcra`, the default; any other algorithm stops the compile, because the share store drives a per-step effective rate that only `gcra` exposes. Membership comes from the heartbeat plane, which stands up when `shared_state.redis` is configured. Without it, `store: share` still loads and enforces with `N` pinned at 1, so it behaves identically to `store: local`. That is a warning rather than an error, and it names the remedy: ```console $ tokentriage validate --config share.yaml warning: governance.limits[0].store: "share" has no membership source without shared_state.redis (external: allow) — N stays 1, so it enforces the full per-replica limit (identical to store: local); add shared_state.redis to derive the ÷N fleet share OK: configuration is valid ``` The summary lines between the warning and the verdict are left out of that capture. This warning prints once, not twice: it needs the limit registries, which only the second validation pass has. Two behaviours follow from dividing rather than sharing. A change in `N` does not rebuild the limiter: only the effective rate and burst are re-read per step, so window state carries across a membership change instead of resetting into a burst. And the number a caller is told moves with `N`, because `/tt/v1/me` and the rate headers report the effective share, so an `rpm` limit of 600 on a three-replica fleet advertises 200. When the membership snapshot goes stale, `N` decays to self. Every verdict is then stamped suspended with the reason `membership_stale`, and each replica widens to the full configured amount, which is the `fleet ≤ N×limit` row in the bound table. Traffic keeps flowing against that wider ceiling rather than stopping. The staleness rides the verdict rather than a store-wide flag, so the row and the response for that request agree. ## When the coordination store goes away A shared counter is a dependency you have to be able to lose, so losing it degrades the guarantee rather than the traffic. Two things engage the fallback. One operation that misses the 5 ms deadline degrades that call. Five consecutive failures trip the circuit breaker, which counts a fatal server reply on a live socket as a failure, so "reachable" does not pass for healthy. An open breaker fails fast instead of waiting. The fallback is per verdict: the embedded local limiter answers, and the verdict carries `suspended` plus one of four reasons, `redis deadline exceeded`, `redis breaker open`, `redis eval error`, or `redis reply malformed`. That local answer decides the request, the decision row lists the limit in `suspended`, and the response carries `x-tt-coordination: suspended` whether it was admitted or turned away. Recovery is positive rather than a timeout: after the probe interval the next call drives one half-open probe, and a good answer re-closes the breaker. The bound widens meanwhile, and the direction is over-admission. Every replica is enforcing the full limit alone, so the fleet can admit up to N times it until the store returns. That is the second half of the `redis` bound string, and the reason to size a Redis-backed cap for what an outage would cost. One thing the panel will not tell you: a `redis` limit renders no coordination-mode block there at all, and the store's own mode keeps reporting the exact strategy while the breaker is open. The widening rides the per-verdict suspension, the row it writes, and the outage note on the 429, not that field. ## Revocation crosses replicas on two numbers Revoking a virtual key on the replica that minted it takes effect at once. Making the other replicas agree is a propagation problem, and the health panel publishes two numbers for it rather than one. The mechanism is a Postgres doorbell with a backstop. Each replica subscribes for change notifications, refreshes on every doorbell, retries a missed one at an eighth of the bound, and re-reads the full snapshot at least once per bound regardless. So the **revocation bound is 30 seconds**: a revoke takes hold within it even if the notification is lost. The second number is detection. A replica is flagged stale only when its last successful refresh is older than twice the bound, so the **staleness bound is 60 seconds**. One bound of grace absorbs a single missed backstop instead of alarming on scheduling jitter. Both appear on the panel because a replica sitting between them is past the propagation bound and not yet flagged, and the smaller number alone would hide that case. A stale replica keeps serving its last-good key snapshot and discloses `suspended:["__identity"]` on its rows. A replica that has not seeded at all is the one case here that stops requests, and only under `require_key: required`, where it returns a 503 rather than guessing a 401. Both of the faults that stop a request rather than degrading are on [the governance front door](/docs/build/governance#fail-open-with-disclosure). ## Replica identity is the lever you own Convergence is keyed on the replica id, so the id is the part of this you have to get right. It resolves in order: `replica.name` in config, then the environment variable named by `replica.name_env`, then the hostname, then a per-process nonce. The first three survive a restart. Only the nonce does not, and it says so on the fleet surface. ```console $ TT_REPLICA_NAME=tt-gov-a tokentriage run -c replica.yaml {"time":"2026-08-09T15:35:17.0535792-07:00","level":"INFO","msg":"replica identity resolved","name":"tt-gov-a","source":"env","stable":true} ``` That is the first line of a boot whose config sets `replica.name_env: TT_REPLICA_NAME`, with the rest of the log left out; the timestamp carries the capture machine's offset. Read `source` and `stable`. A name has to look like a DNS label, lowercase and at most 63 characters, because it travels as a hash field and as a value in your ledger. An unstable id mints a fresh convergence cell on every restart, and the old one keeps counting until it ages out. A stable identity is a requirement rather than a suggestion once you converge calendar budgets across pod churn: set `replica.name_env` to the pod name through the downward API, or run a StatefulSet so a rescheduled pod returns under its prior name. ## The residuals, stated plainly Cell hygiene shipped, so a departed replica's cell is bounded rather than permanent. Each cell carries a server-clock beat; a peer cell whose beat is older than the ten-minute retention window drops out of the fold, and the next publish deletes it inside the same atomic script. Decommissioning a replica through a live replica's admin API purges its cells by name. So a replica that returns under a new id double-counts its own burn for at most one retention window, and a dead replica's previous-period burn drops out of the new period inside the same window instead of turning legitimate spend away indefinitely. Three residuals are left, and they run in different directions. **A partition that starts before the first success is silent.** A replica that has not reached Redis since boot has no successful flush to age out. Its remote term is zero and the bound it discloses on a refusal is byte-identical to a healthy single node's, while it enforces the full cap alone. The degraded and stale disclosures widen only once a replica has succeeded at least once, so this one over-admits without saying so. **A replica that died running a pre-beat binary leaves a cell nothing can age.** Its cell carries no server-clock beat, and the age test keeps a cell whose age is unknowable rather than dropping it, because dropping a live peer's burn would over-admit in the money-losing direction. That stale burn stays in every survivor's remote term until the replica is decommissioned by name. It arises only where a replica died before it was upgraded. **The convergence table on the health panel renders `budget_usd` rows only.** A `quota_tokens` or `quota_requests` limit converges, enforces and returns 429s on the remote term, and shows no convergence row. That panel's suspension list is spend-only for the same reason, so what reaches you for those two is the `convergence_lag_rows` term on the 429 and the row that refusal writes. None of the tuning parameters behind those numbers is a config key. | parameter | value | where it lives | |---|---|---| | flush cadence | 1s | compile-time constant | | convergence staleness | 30s | compile-time constant | | cell retention | 10min | compile-time constant | | revocation bound | 30s | compile-time constant | | breaker threshold, probe interval | 5 failures, 5s | not exposed on `shared_state.redis` | | per-operation Redis deadline | 5ms | not exposed on `shared_state.redis` | `shared_state.redis` accepts four keys, `external`, `addr`, `password_env` and `tls`, and none of them tunes the numbers above. The heartbeat cadence and liveness window that feed `N_live` are the exception: `fleet.heartbeat_ms` and `fleet.liveness_window_ms`, which default to 2000 and 15000 when you leave them out. ## What this page does not cover Standing up a fleet is [multi-replica and scale-out](/docs/operate/scale-out): the chart, the roster, and why the decision ledger stays per pod. Which store each limit type accepts is [the six limit types](/docs/build/governance-limits), and the algorithm constraint behind `store: share` is in [rate algorithms](/docs/build/governance-rate-algorithms). Clearing a suspension day to day is [operating governance](/docs/build/governance-operations). What was spent across the fleet, and by whom, is reporting rather than enforcement: [cost analytics](/docs/build/cost-analytics) owns it.