Multi-replica & scale-out

TokenTriage is single-node by default and honest about what changes when you scale. The single most important distinction:

The single-node default

replicaCount: 1 is the GA default and coordinates no fleet. The shared-state backends (Redis + Postgres) are opt-in — each rides the config’s external: "allow" gate, so nothing dials out unless you type the consent string. Multi-replica is additionally preview-gated in the chart via experimental.multiReplica.enabled: false; replicaCount > 1 is refused unless that gate is fully open.

Disclosed convergence bounds

Governance limits carry a store: mode that trades exactness for round-trips:

• local — per-replica, no coordination.
• redis — fleet coordination via a server-time EVALSHA under a 5 ms deadline. rpm is exact (acquire-only atomic GCRA). Spend/quota (budget_usd / quota_*) instead converge — an idempotent per-replica cell plus a fold of the others (remote = combined − own, never an INCRBY) — carrying a disclosed convergence bound, not exact. A tpm limit cannot bind redis — see below.
• share — a global approximation via a ÷N split, with a disclosed bound: fleet ≤ N × ceil(amount / N_live) ≈ amount (over-admission ≤ N−1 from rounding).

Token and request quotas converge rather than ÷N-split (each replica folds its local burn into a per-scope cell and reads the sum of the others as a remote term). A tpm rate limit binds local (per-replica) or store: share, where it enforces a ÷N per-replica token window — the local GCRA runs the signed reserve→reconcile correctly. It is refused at validate on store: redis: the distributed rate script is acquire-only and has no signed reserve→reconcile, so rather than silently mis-enforce it fails closed. (rpm is acquire-only and binds redis for exact fleet-wide counting.) One honest caveat on share: a tpm limit’s token burst divides ÷N, so the fleet burst ceiling widens as replicas scale — the sustained rate stays ≈ the limit; only the burst dimension loosens, and it is disclosed. The declared bounds are published verbatim: a 15 s liveness window and a 30 s revocation bound.

Degradation is fail-open and disclosed

If the shared store is unreachable, coordination is suspended (traffic still flows) and the one fact is disclosed on three surfaces: a response header x-tt-coordination: suspended, the decision row (suspended:[<limit_id>]), and the governance-limits panel (the bound widens to membership_stale, ≤ N × limit). A healthy fleet carries no coordination header — the absence is the honest default.

Distributed identity gating — two numbers, not one

Cross-replica key revocation (via shared_state.postgres + external: allow) runs a Postgres LISTEN/NOTIFY fast-path with a periodic full-snapshot backstop that catches a dropped doorbell. Two thresholds govern it, and the health panel discloses both because they mean different things:

• The revocation bound is 30 s (pg.DefaultRevocationBound, internal/shared/pg/replica.go). Every replica re-reads the full snapshot at least once per bound, so a revoke is honored within 30 s even if the NOTIFY is missed. The panel prints revocation_bound: "30s".
• Identity-staleness detection fires at 2× that bound — 60 s. staleFactor = 2, so StalenessBound() = staleFactor × bound = 60s, disclosed as staleness_bound: "1m0s". A seeded replica whose last successful refresh has aged past 60 s (a genuine mid-life partition, past several missed backstops — one bound of grace absorbs a single transient miss) trips Stale(), which flips IdentityStale().

When staleness trips, identity gating is fail-open with disclosure — the same rule as every other governance fault (internal/govern/engine.go). A stale or never-seeded replica keeps serving its last-good key snapshot; only the health signal flips, and the data-plane rows disclose suspended:["__identity"] (discloseIdentityStale, appended on both admits and StageA identity refusals). A dead or degraded identity store never fabricates a 401 — internal/proxy/govern.go stamps that suspension onto the wire from the same one fact that writes the row (row == wire). The one exception is a require_key: required deployment on a never-seeded replica (a cold boot during an outage): it honestly returns 503 (“cannot verify”), lifted the instant the replica first seeds — it does not guess a 401.

The two sections below are deep operational caveats for the distributed tier under replica churn. A single-node or stable-fleet deployment never hits either — they’re collapsed so the page reads as a how-to, not a design doc. Expand them if you run a churning multi-replica fleet.

Deep dive: the one-time redis rate-counter relocation (bounded 2×)

Counter identity is stable across config edits — spelling out a default (e.g. algorithm: gcra on an rpm limit that omitted it) neither drops the live local gauge nor moves the redis key. Making that guarantee honest had one bounded, one-time side effect: for a store: redis rate limit whose config omitted algorithm:, the redis key now derives from the effective algorithm (gcra) rather than the empty string, so on the first upgrade to the build carrying this normalization the counter relocates once (the old key is abandoned and expires by its own TTL; a fresh key starts).

Deep dive: convergence residuals under replica churn

Spend and quota classes (budget_usd, quota_tokens, quota_requests) converge: each replica publishes an idempotent per-replica cell (an HSET keyed on the replica id, not an INCRBY), the combined burn is the sum of all cells, and remote = combined − own. A restart under the same replica id re-writes the same cell, so a burn is never double-counted. But the combined hash has no per-cell liveness — the publish PEXPIREs the whole hash to the staleness TTL, which any live replica refreshes every flush. So a departed replica’s cell (a crash, a scale-down, or a rolling replacement that returns under a new id) persists as a phantom publisher for as long as the fleet is alive.

On the opt-in distributed tier with replica churn — a single-node or stable-fleet deployment is unaffected — this has three disclosed consequences (docs/GOVERNANCE.md §5):

• Cross-period over-refuse (fail-closed). The convergence key embeds the period name (month), not the period instant, so a calendar budget reuses one hash across every month. Live replicas overwrite their own cell each flush, but a departed replica’s cell cannot self-reset — so at a period rollover its previous-period burn is summed into the new period’s remote, and legitimate spend can be over-refused until the ghost cells clear. Magnitude ≈ Σ(departed replicas’ last-period burn). The direction is over-refuse, never a wrong admit of unbudgeted spend.
• Same-period double-count (conditional). If a replica returns under a new id (a k8s pod reschedule, or the random-id fallback when the hostname is unavailable) and its local ledger survived (a persistent volume that CatchUp re-reads), it counts its own prior burn once locally and again as the orphan cell’s remote.
• From-boot partition silence (fail-open). A replica that has never reached Redis since boot publishes a bound byte-identical to a healthy single-node deployment while each blind replica fail-opens locally to the full cap. The degraded/stale disclosure only widens once a replica has succeeded at least once, so a from-boot partition is silent.

Mitigations

• Keep replica ids stable across restarts. A restart then reuses its cell rather than orphaning it (the shipped TestReplicaRestartNoDoubleCount covers this).
• Prefer a stable-identity deployment — a StatefulSet over a Deployment — when converging calendar budgets across heavy pod churn, so a rescheduled pod returns under its prior id.
• A full fleet stop clears the hash. With all publishers idle the whole-hash TTL expires, so a coordinated full-fleet restart resets the phantom cells.

The proper fix — period-instant-scoped keys plus per-cell liveness — is a disclosed post-round-2 distributed follow-up, tracked alongside the quota-class convergence work.

GA vs preview — the honest nuance

Cross-replica governance coordination is proven on a real cluster. But be precise about what’s still preview or pending:

• The Helm values key remains experimental.multiReplica, and the chart still prints preview banners referencing it.
• The Kubernetes cross-pod /query fan-out is pending (bounded on a per-pod advertise_url downward-API residual; proven on Compose with real Redis + Postgres).
• Cross-replica trace_linked retry is honestly absent.
• Federation (static cross-fleet instances) is a separate feature and remains PREVIEW (experimental.federation).

The Fleet surface

The /fleet dashboard shows the replica roster by heartbeat age. A single-node deployment is a fleet of one, reported honestly as {1 of 1} (not an empty list). Two bounds are published and are not the same kind of number: liveness_window_ms is declared (the configured window, default 15 000 ms over a 2 000 ms heartbeat), and max_abs_skew_ms is measured (the largest clock disagreement observed). A degraded: true roster is a frozen snapshot with an as_of timestamp — read the age, not the cards, as current.

See Governance for the limit semantics and Deployment for the chart.