Governance across replicas

You run three pods behind a load balancer. A cap of 600 requests a minute, written once and deployed three times, admits 1,800, because each pod counts its own arrivals and cannot see the other two. The same arithmetic applies to a monthly budget, a token quota and an in-flight cap.

Coordination closes some of that gap, and it costs you a network dependency for the part it closes. Nothing here is on out of the box, and no limit goes fleet-wide because you scaled up. You pick a coordination mode per limit, and each mode carries a different bound.

This page covers which limits converge, what selects that mode, and which direction the remaining error runs. Running the fleet itself, the replica roster and the read-time ledger merge are in multi-replica and scale-out.

Nothing dials until you ask for it

The cost of the distributed tier is an outbound connection, so the condition that creates one is worth knowing precisely.

With no shared_state block, the binary opens no coordination connection at all. Every limit falls to store: local, its default. No convergence loop is attached, no fleet membership is read, and no coordination header is written. A pod in a fleet of ten behaves like a single node.

Two consent gates open the tier, each a literal string. shared_state.redis needs external: allow plus an addr; shared_state.postgres needs external: allow plus a dsn_env naming the environment variable that holds the DSN. An address without the consent string is a validation error rather than a quiet skip.

shared_state:
  redis:
    external: allow # without this, nothing dials
    addr: "10.0.0.10:6379"
    password_env: TT_REDIS_PASSWORD # the env var name, not the secret
    tls: true

governance:
  enabled: true
  limits:
    - id: org_rpm_fleet
      attach: { node: __global }
      type: rpm
      amount: 600
      period: 1m # a rate limit with no period stops the boot
      algorithm: gcra
      store: redis # default is local
      posture: { mode: shadow }

Granting the consent produces a warning every time you validate, and the warning names what you gave up:

$ tokentriage validate --config fleet.yaml
warning: shared_state.redis: a network redis at "10.0.0.10:6379" is enabled — the single-binary air-gap posture is broken for the coordination tier (disclosed, §2.8)
warning: shared_state.redis: a network redis at "10.0.0.10:6379" is enabled — the single-binary air-gap posture is broken for the coordination tier (disclosed, §2.8)
OK: configuration is valid

That capture is from a source build against the config above, in a file that also carries the usual server: and upstreams: blocks, with the summary lines trimmed. The warning prints twice because validation runs once when the config loads and again when the limit registries are built. Even after consent the client is lazy: constructing it dials nothing, and the first operation opens the socket.

A single node says so on GET /tt/api/v1/governance/health, in its own words:

{
  "kind": "governance/distributed-health",
  "capability": {
    "shared_state_enabled": false,
    "tiers": [],
    "note": "air-gapped: no shared-state tier configured — this is a single-node instance; distributed limits, cross-replica convergence, and reconcile are not active"
  },
  "suspensions": [],
  "convergence": []
}

What converges, and what each store does

Coordination is not one feature. Spend and quota converge through the accountant, the off-path worker that folds finished ledger rows into counters; rate and in-flight limits go through a counter store you name on the limit. Reading the two as one thing is the fastest way to get the bound wrong.

type	how it goes fleet-wide	what selects it	which way the error runs
budget_usd	converges: each replica publishes its own burn cell and folds in the others	shared_state.redis plus any of these three armed; a store: key here is a config error	over-admits by whatever peers burned since their last publish
quota_tokens	the same fold, metered in tokens	as above	as above
quota_requests	the same fold, metered in forwarded rows	as above	as above
rpm	one atomic script step against the shared counter	store: redis	tight while the store answers, up to N×limit while it does not
rpm, tpm	each replica enforces ceil(limit / N_live)	store: share	over-admits by up to N−1 from rounding
concurrency	each replica caps its gauge at ceil(C / N_live)	store: share	the same rounding term, on in-flight slots
tpm, concurrency	no shared-counter mode	store: redis is turned down	tpm at the compile, concurrency at validate

Each store carries its own bound as a fixed string. Only one of the three reaches the wire: the limits API renders an enforcement block for store: share and omits it for local and redis, whose strings stay inside the binary.

store	the bound it carries
local	per-replica only: fleet ≤ N×limit
redis	exact; ≤ N×limit during store outage
share	fleet ≤ N×ceil(limit/N_live) ≈ limit (over-admission ≤ N−1 from ÷N rounding, + joiners×share per liveness window on scale-up); under-service under LB skew
share, membership stale	membership_stale: N decayed to self — fleet ≤ N×limit until the coordination store recovers
share on concurrency	per-replica cap = ceil(C/N_live); fleet in-flight ≤ N×ceil(C/N_live) ≈ C (over-admission ≤ N−1 from ÷N rounding)

The trailing clause on the share bound names load-balancer skew: if traffic does not split evenly, one replica can hit its share while the fleet as a whole is under the limit. The last row is the same arithmetic in the concurrency limit’s own words, counted in in-flight slots.

Spend and quota converge, and you write nothing on the limit to get it. The three types whose counter is settled from ledger rows reject a store: key outright; how a limit counts draws that class. Convergence attaches when a Redis tier exists and at least one of the three is armed, and the mechanism is worth a picture.

Read it left to right. Each replica meters its own burn from its own ledger, and once per flush it writes one field of a shared per-scope hash and reads the whole hash back. The write sets that replica’s own field, keyed on its replica id, rather than incrementing a shared total, so a retried flush or a restart under the same id rewrites the same value instead of adding a second copy. The replica then takes remote = combined − own, and admission compares local plus remote against the cap. The cadence is one second, so a peer’s last second of burn is not folded yet, and that unfolded burn is what a 429 publishes as convergence_lag_rows beside the cap it enforced.

The quota classes ride the same fold in their own meter: tokens for quota_tokens, forwarded rows for quota_requests. The remote term lands in that meter and not in dollars, so a token count is not rendered as money anywhere. The validator’s old per-replica quota warning was retired when that shipped, though the shipped distributed example config still carries the retired text in a comment.

rpm with store: redis is the one fleet-exact counter. Each admission is a single atomic script step against the shared counter, timed by the Redis server clock, so your replicas do not have to agree on a clock among themselves. The key is a fixed prefix plus a hash of the scope key, so tenant identifiers stay out of the shared keyspace. One round trip is bounded at 5 ms, and a slower answer falls back per verdict rather than waiting.

tpm with store: redis is turned down, and later than you would expect. The distributed script is acquire-only, with no signed settlement, so a token reserve could not be corrected against actual usage in either direction. That check lives in the limit compiler, which runs at boot and inside govern preview, not inside tokentriage validate.

$ tokentriage govern preview -c tpm-redis.yaml --log ./decisions.jsonl
Error: govern preview: govern: limit "org_tpm_fleet": type "tpm" (reserve/reconcile) cannot bind store "redis" — the distributed rate store is acquire-only (the reserve→reconcile settlement is not implemented for it), so a per-request token reconcile would mis-enforce in both directions; use the default local store or store: share
Error: govern preview: govern: limit "org_tpm_fleet": type "tpm" (reserve/reconcile) cannot bind store "redis" — the distributed rate store is acquire-only (the reserve→reconcile settlement is not implemented for it), so a per-request token reconcile would mis-enforce in both directions; use the default local store or store: share

The message lands on stderr twice: the command prints its own error and the process prints it again on the way out. Boot prints the same text behind a governance: compile limits: prefix and exits.

concurrency with store: redis is turned down earlier, at validate, and the message points at the share cap instead. It runs no counter store at all: the cap is a live gauge in memory, and store: share only divides that gauge.

The share store divides the limit by the live fleet

A third store name buys a fleet-wide ceiling without a round trip: store: share. It gives rpm, tpm and concurrency an approximate fleet mode with no coordination call on the request path.

Each replica enforces ceil(limit / N_live), where N_live is the live replica count read from heartbeat membership as one atomic load. Six hundred requests a minute across three live replicas becomes 200 per replica. For concurrency there is no counter to share, so the same membership drives a cap instead: each replica admits ceil(C / N_live) in flight. The rate types also require algorithm: gcra, the default; any other algorithm stops the compile, because the share store drives a per-step effective rate that only gcra exposes.

Membership comes from the heartbeat plane, which stands up when shared_state.redis is configured. Without it, store: share still loads and enforces with N pinned at 1, so it behaves identically to store: local. That is a warning rather than an error, and it names the remedy:

$ tokentriage validate --config share.yaml
warning: governance.limits[0].store: "share" has no membership source without shared_state.redis (external: allow) — N stays 1, so it enforces the full per-replica limit (identical to store: local); add shared_state.redis to derive the ÷N fleet share
OK: configuration is valid

The summary lines between the warning and the verdict are left out of that capture. This warning prints once, not twice: it needs the limit registries, which only the second validation pass has.

Two behaviours follow from dividing rather than sharing. A change in N does not rebuild the limiter: only the effective rate and burst are re-read per step, so window state carries across a membership change instead of resetting into a burst. And the number a caller is told moves with N, because /tt/v1/me and the rate headers report the effective share, so an rpm limit of 600 on a three-replica fleet advertises 200.

When the membership snapshot goes stale, N decays to self. Every verdict is then stamped suspended with the reason membership_stale, and each replica widens to the full configured amount, which is the fleet ≤ N×limit row in the bound table. Traffic keeps flowing against that wider ceiling rather than stopping. The staleness rides the verdict rather than a store-wide flag, so the row and the response for that request agree.

When the coordination store goes away

A shared counter is a dependency you have to be able to lose, so losing it degrades the guarantee rather than the traffic.

Two things engage the fallback. One operation that misses the 5 ms deadline degrades that call. Five consecutive failures trip the circuit breaker, which counts a fatal server reply on a live socket as a failure, so “reachable” does not pass for healthy. An open breaker fails fast instead of waiting.

The fallback is per verdict: the embedded local limiter answers, and the verdict carries suspended plus one of four reasons, redis deadline exceeded, redis breaker open, redis eval error, or redis reply malformed. That local answer decides the request, the decision row lists the limit in suspended, and the response carries x-tt-coordination: suspended whether it was admitted or turned away. Recovery is positive rather than a timeout: after the probe interval the next call drives one half-open probe, and a good answer re-closes the breaker.

The bound widens meanwhile, and the direction is over-admission. Every replica is enforcing the full limit alone, so the fleet can admit up to N times it until the store returns. That is the second half of the redis bound string, and the reason to size a Redis-backed cap for what an outage would cost.

localhost:9090/ui/governance/healthDemo data

One thing the panel will not tell you: a redis limit renders no coordination-mode block there at all, and the store’s own mode keeps reporting the exact strategy while the breaker is open. The widening rides the per-verdict suspension, the row it writes, and the outage note on the 429, not that field.

Revocation crosses replicas on two numbers

Revoking a virtual key on the replica that minted it takes effect at once. Making the other replicas agree is a propagation problem, and the health panel publishes two numbers for it rather than one.

The mechanism is a Postgres doorbell with a backstop. Each replica subscribes for change notifications, refreshes on every doorbell, retries a missed one at an eighth of the bound, and re-reads the full snapshot at least once per bound regardless. So the revocation bound is 30 seconds: a revoke takes hold within it even if the notification is lost.

The second number is detection. A replica is flagged stale only when its last successful refresh is older than twice the bound, so the staleness bound is 60 seconds. One bound of grace absorbs a single missed backstop instead of alarming on scheduling jitter. Both appear on the panel because a replica sitting between them is past the propagation bound and not yet flagged, and the smaller number alone would hide that case.

A stale replica keeps serving its last-good key snapshot and discloses suspended:["__identity"] on its rows. A replica that has not seeded at all is the one case here that stops requests, and only under require_key: required, where it returns a 503 rather than guessing a 401. Both of the faults that stop a request rather than degrading are on the governance front door.

Replica identity is the lever you own

Convergence is keyed on the replica id, so the id is the part of this you have to get right. It resolves in order: replica.name in config, then the environment variable named by replica.name_env, then the hostname, then a per-process nonce. The first three survive a restart. Only the nonce does not, and it says so on the fleet surface.

$ TT_REPLICA_NAME=tt-gov-a tokentriage run -c replica.yaml
{"time":"2026-08-09T15:35:17.0535792-07:00","level":"INFO","msg":"replica identity resolved","name":"tt-gov-a","source":"env","stable":true}

That is the first line of a boot whose config sets replica.name_env: TT_REPLICA_NAME, with the rest of the log left out; the timestamp carries the capture machine’s offset. Read source and stable. A name has to look like a DNS label, lowercase and at most 63 characters, because it travels as a hash field and as a value in your ledger.

The residuals, stated plainly

Cell hygiene shipped, so a departed replica’s cell is bounded rather than permanent. Each cell carries a server-clock beat; a peer cell whose beat is older than the ten-minute retention window drops out of the fold, and the next publish deletes it inside the same atomic script. Decommissioning a replica through a live replica’s admin API purges its cells by name. So a replica that returns under a new id double-counts its own burn for at most one retention window, and a dead replica’s previous-period burn drops out of the new period inside the same window instead of turning legitimate spend away indefinitely.

Three residuals are left, and they run in different directions.

A partition that starts before the first success is silent. A replica that has not reached Redis since boot has no successful flush to age out. Its remote term is zero and the bound it discloses on a refusal is byte-identical to a healthy single node’s, while it enforces the full cap alone. The degraded and stale disclosures widen only once a replica has succeeded at least once, so this one over-admits without saying so.

A replica that died running a pre-beat binary leaves a cell nothing can age. Its cell carries no server-clock beat, and the age test keeps a cell whose age is unknowable rather than dropping it, because dropping a live peer’s burn would over-admit in the money-losing direction. That stale burn stays in every survivor’s remote term until the replica is decommissioned by name. It arises only where a replica died before it was upgraded.

The convergence table on the health panel renders budget_usd rows only. A quota_tokens or quota_requests limit converges, enforces and returns 429s on the remote term, and shows no convergence row. That panel’s suspension list is spend-only for the same reason, so what reaches you for those two is the convergence_lag_rows term on the 429 and the row that refusal writes.

None of the tuning parameters behind those numbers is a config key.

parameter	value	where it lives
flush cadence	1s	compile-time constant
convergence staleness	30s	compile-time constant
cell retention	10min	compile-time constant
revocation bound	30s	compile-time constant
breaker threshold, probe interval	5 failures, 5s	not exposed on shared_state.redis
per-operation Redis deadline	5ms	not exposed on shared_state.redis

shared_state.redis accepts four keys, external, addr, password_env and tls, and none of them tunes the numbers above. The heartbeat cadence and liveness window that feed N_live are the exception: fleet.heartbeat_ms and fleet.liveness_window_ms, which default to 2000 and 15000 when you leave them out.

What this page does not cover

Standing up a fleet is multi-replica and scale-out: the chart, the roster, and why the decision ledger stays per pod. Which store each limit type accepts is the six limit types, and the algorithm constraint behind store: share is in rate algorithms. Clearing a suspension day to day is operating governance. What was spent across the fleet, and by whom, is reporting rather than enforcement: cost analytics owns it.