Multi-replica & scale-out

Running several replicas raises one question that decides everything else: when a request lands on replica two, what does replica two know about what replicas one and three have already done?

The answer is split, and the split is the thing to learn.

Read the figure left to right. Each replica writes only to its own volume; the shared store on the upper right carries governance and nothing else; the fleet view on the lower right is computed when you ask, from whichever replicas answer. The dashed line is the one that does not become a shared file.

The single-node default

replicaCount: 1 is the default and coordinates nothing, which is the right choice for most deployments. Growing past it is gated in two places at once, and the gate is worth meeting on paper before you meet it in a terminal:

$ helm template tt deploy/helm/tokentriage -f my-values.yaml --set replicaCount=2
Error: values don't meet the specifications of the schema(s) in the following chart(s):
tokentriage:
- at '': 'allOf' failed
  - at '': validation failed
    - at '/experimental/multiReplica/enabled': value must be true
    - at '/externalState': validation failed
      - at '/externalState/postgres/enabled': value must be true
      - at '/externalState/redis/enabled': value must be true

The values schema refuses first and names all three missing pieces. A second, template-level guard behind it spells out why, and you see it if schema validation is skipped:

Error: execution error at (tokentriage/templates/statefulset.yaml:29:28): replicaCount > 1 requires experimental.multiReplica.enabled plus externalState.redis and externalState.postgres — the shared-state tier that COORDINATES governance across replicas (keys, budgets, rate/quota/concurrency limits; fail-open and disclosed). Each replica keeps its own canonical ledger by design; the fleet view is a read-time coverage-typed merge, and scaling without the shared-state tier would run N uncoordinated gateways.

Both backends ride the config’s consent gate, so nothing dials out until you type external: "allow" yourself. The chart cannot type it for you.

What coordination costs, per limit

A governance limit carries a store: that trades exactness against round trips. The word “global” is absent from the vocabulary, because a limit states how coordinated it is:

store:	Mode	Bound	Round trip per admission
local	per-replica	none; each replica enforces the full amount	no
redis	exact fleet	exact, server-time, under a 5 ms deadline	yes
share	global approximation	fleet ≤ N × ceil(amount / N_live), over-admitting at most N−1 from rounding	no

Spend and token quotas do not split. Each replica folds its own burn into an idempotent per-replica cell and reads the sum of the others as a remote term, so the classes converge rather than divide, and the lag is bounded by the flush cadence. A token meter converges in tokens and a dollar meter in dollars; the two are not blurred.

Not every combination is available, and the validator refuses the ones that would mis-enforce rather than accepting them quietly. Enforcement classes is the page that maps limit type to store, including the two refusals and why they exist.

Degradation is disclosed on three surfaces

When the coordination store goes away, traffic keeps flowing. That is the design, and the price of it is that a guarantee is not being held while you are not being told. So you are told, three times, from one fact:

1. The response header carries x-tt-coordination: suspended, stamped at every terminal point including refusals and the error path.
2. The decision row names the limits: suspended:[<limit_id>].
3. The limits panel widens the published bound rather than asserting one it cannot back.

A healthy fleet carries no coordination header at all. There is no positive “ok” value, and the absence is the default, which means you cannot mistake a missing header for a healthy one you forgot to check.

Recovery needs nothing from you. A single half-open probe re-adopts the store when it returns and all three disclosures clear. A coordinated limit degraded is the runbook, and its fix is “do nothing destructive”.

Reads confess their coverage

The fleet read surfaces answer from whichever replicas responded, and say so. Every response carries replicas_total, replicas_answered, is_partial, a basis, and a per-replica timestamp. A dead replica stays in the denominator as a typed unreachable card rather than being dropped.

That matters more than it sounds. A merge that quietly excluded a silent replica would return a smaller number that looks exactly like a complete one, and the reader would have no way to tell a quiet week from a missing pod.

A drained replica stays in the denominator until you remove it explicitly, which is the same principle:

$ tokentriage fleet decommission <name>

The removal is audited and drops the replica from N. Until then, reads honestly report a partial.

Two residuals under replica churn, and how to avoid both

Both apply only to the distributed tier with replicas coming and going. A stable fleet meets neither.

Convergence cells outlive departed replicas. Each replica publishes an idempotent cell keyed on its own id, and the combined hash has no per-cell liveness. A replica that leaves, whether by a crash, a scale-down, or a reschedule that returns under a new id, leaves a cell behind. At a calendar period rollover its previous-period burn is summed into the new period’s remote term, so legitimate spend can be over-refused until the ghost cells clear. The direction is over-refuse, never a wrong admit of unbudgeted spend, and the magnitude is the sum of the departed replicas’ last-period burn.

Keep replica identities stable across restarts and a restart reuses its cell instead of orphaning one. Prefer a StatefulSet over a Deployment when converging calendar budgets under heavy churn. A coordinated full-fleet stop expires the whole hash and clears the phantoms.

A rate counter relocates once, on one upgrade. For a store: redis rate limit whose config omitted algorithm:, the Redis key now derives from the effective algorithm rather than from an empty string, so on the first upgrade to the build carrying that normalisation the counter moves once: the old key is abandoned to its own expiry and a fresh one starts. During the rolling upgrade the fleet briefly runs two key namespaces and the limit may admit up to twice its amount for the rollout window. It is over-admit, transient and self-healing; spelling algorithm: gcra explicitly before you upgrade avoids it entirely, and the shipped examples already do.

What is not coordinated yet

Publishing this list is more useful than the headline, so here it is with reasons rather than adjectives.

The cross-pod query fan-out on Kubernetes is bounded on a residual. Each pod needs to advertise its own stable DNS name through the downward API, plus a per-peer token. The property, a disclosed partial when a replica does not answer, is proven against real Redis and Postgres on the Compose harness, where membership discovery and per-replica advertisement light it fully. The Kubernetes instantiation waits on that advertisement.

Trace-linked cross-replica retry is absent. There is no per-row client-asserted trace channel at the merge layer, so the aggregate does not report one. It is absent rather than reported as zero, which are different claims.

Federation is a different feature and remains preview. Static cross-fleet peering under experimental.federation gives a combined read view over instances that are not one coordinated fleet: a reachable peer shows its card, an unreachable one is a labelled degrade, and nothing is merged across them. It makes no claim of shared state.

What this does not cover

Limit semantics, enforcement modes and the store-selection rules are governance’s. Letting Kubernetes move the replica count for you is autoscaling. Each replica’s own ledger behaves exactly as a single node’s, so the growth and retention arithmetic on the decision ledger on disk applies per replica, and so does the backup, N times over.