# Multi-replica & scale-out
Governance coordinates across replicas. The ledger does not, and every fleet number tells you which of the two it is.
Source: https://tokentriage.com/docs/operate/scale-out
---
import Callout from '../../../components/Callout.astro';
import Details from '../../../components/Details.astro';
import NextSteps from '../../../components/NextSteps.astro';
import ScaleOutDiagram from '../../../components/ScaleOutDiagram.astro';
Running several replicas raises one question that decides everything else: when a request lands
on replica two, what does replica two know about what replicas one and three have already done?
The answer is split, and the split is the thing to learn.
Across N replicas with the shared-state tier, **governance state coordinates across the fleet.
The decision ledger does not.** Each replica keeps its own canonical ledger on its own volume,
and the fleet view is a read-time merge over those, typed by how much of the fleet it covered.
Redis and Postgres are the coordination tier, not the system of record for usage.
Read the figure left to right. Each replica writes only to its own volume; the shared store on
the upper right carries governance and nothing else; the fleet view on the lower right is
computed when you ask, from whichever replicas answer. The dashed line is the one that does not
become a shared file.
## The single-node default
`replicaCount: 1` is the default and coordinates nothing, which is the right choice for most
deployments. Growing past it is gated in two places at once, and the gate is worth meeting on
paper before you meet it in a terminal:
```console
$ helm template tt deploy/helm/tokentriage -f my-values.yaml --set replicaCount=2
Error: values don't meet the specifications of the schema(s) in the following chart(s):
tokentriage:
- at '': 'allOf' failed
- at '': validation failed
- at '/experimental/multiReplica/enabled': value must be true
- at '/externalState': validation failed
- at '/externalState/postgres/enabled': value must be true
- at '/externalState/redis/enabled': value must be true
```
The values schema refuses first and names all three missing pieces. A second, template-level
guard behind it spells out why, and you see it if schema validation is skipped:
```console
Error: execution error at (tokentriage/templates/statefulset.yaml:29:28): replicaCount > 1 requires experimental.multiReplica.enabled plus externalState.redis and externalState.postgres — the shared-state tier that COORDINATES governance across replicas (keys, budgets, rate/quota/concurrency limits; fail-open and disclosed). Each replica keeps its own canonical ledger by design; the fleet view is a read-time coverage-typed merge, and scaling without the shared-state tier would run N uncoordinated gateways.
```
Both backends ride the config's consent gate, so nothing dials out until you type
`external: "allow"` yourself. The chart cannot type it for you.
## What coordination costs, per limit
A governance limit carries a `store:` that trades exactness against round trips. The word
"global" is absent from the vocabulary, because a limit states how coordinated it is:
| `store:` | Mode | Bound | Round trip per admission |
|---|---|---|---|
| `local` | per-replica | none; each replica enforces the full amount | no |
| `redis` | exact fleet | exact, server-time, under a 5 ms deadline | yes |
| `share` | global approximation | `fleet ≤ N × ceil(amount / N_live)`, over-admitting at most `N−1` from rounding | no |
Spend and token quotas do not split. Each replica folds its own burn into an idempotent
per-replica cell and reads the sum of the others as a remote term, so the classes converge
rather than divide, and the lag is bounded by the flush cadence. A token meter converges in
tokens and a dollar meter in dollars; the two are not blurred.
Not every combination is available, and the validator refuses the ones that would mis-enforce
rather than accepting them quietly. [Enforcement
classes](/docs/build/governance-enforcement-classes) is the page that maps limit type to store,
including the two refusals and why they exist.
## Degradation is disclosed on three surfaces
When the coordination store goes away, traffic keeps flowing. That is the design, and the price
of it is that a guarantee is not being held while you are not being told. So you are told,
three times, from one fact:
1. **The response header** carries `x-tt-coordination: suspended`, stamped at every terminal
point including refusals and the error path.
2. **The decision row** names the limits: `suspended:[]`.
3. **The limits panel** widens the published bound rather than asserting one it cannot back.
A healthy fleet carries no coordination header at all. There is no positive "ok" value, and the
absence is the default, which means you cannot mistake a missing header for a healthy one you
forgot to check.
Recovery needs nothing from you. A single half-open probe re-adopts the store when it returns
and all three disclosures clear. [A coordinated limit
degraded](/docs/operate/runbooks#a-coordinated-limit-degraded) is the runbook, and its fix is
"do nothing destructive".
## Reads confess their coverage
The fleet read surfaces answer from whichever replicas responded, and say so. Every response
carries `replicas_total`, `replicas_answered`, `is_partial`, a basis, and a per-replica
timestamp. A dead replica stays in the denominator as a typed unreachable card rather than being
dropped.
That matters more than it sounds. A merge that quietly excluded a silent replica would return a
smaller number that looks exactly like a complete one, and the reader would have no way to tell
a quiet week from a missing pod.
A drained replica stays in the denominator until you remove it explicitly, which is the same
principle:
```console
$ tokentriage fleet decommission
```
The removal is audited and drops the replica from `N`. Until then, reads honestly report a
partial.
Both apply only to the distributed tier with replicas coming and going. A stable fleet meets
neither.
**Convergence cells outlive departed replicas.** Each replica publishes an idempotent cell keyed
on its own id, and the combined hash has no per-cell liveness. A replica that leaves, whether by
a crash, a scale-down, or a reschedule that returns under a *new* id, leaves a cell behind. At a calendar
period rollover its previous-period burn is summed into the new period's remote term, so
legitimate spend can be over-refused until the ghost cells clear. The direction is
over-**refuse**, never a wrong admit of unbudgeted spend, and the magnitude is the sum of the
departed replicas' last-period burn.
Keep replica identities stable across restarts and a restart reuses its cell instead of
orphaning one. Prefer a StatefulSet over a Deployment when converging calendar budgets under
heavy churn. A coordinated full-fleet stop expires the whole hash and clears the phantoms.
**A rate counter relocates once, on one upgrade.** For a `store: redis` rate limit whose config
omitted `algorithm:`, the Redis key now derives from the effective algorithm rather than from an
empty string, so on the first upgrade to the build carrying that normalisation the counter moves
once: the old key is abandoned to its own expiry and a fresh one starts. During the rolling
upgrade the fleet briefly runs two key namespaces and the limit may admit up to twice its amount
for the rollout window. It is over-admit, transient and self-healing; spelling `algorithm: gcra`
explicitly before you upgrade avoids it entirely, and the shipped examples already do.
## What is not coordinated yet
Publishing this list is more useful than the headline, so here it is with reasons rather than
adjectives.
**The cross-pod query fan-out on Kubernetes is bounded on a residual.** Each pod needs to
advertise its own stable DNS name through the downward API, plus a per-peer token. The
property, a disclosed partial when a replica does not answer, is proven against real Redis and
Postgres on the Compose harness, where membership discovery and per-replica advertisement light
it fully. The Kubernetes instantiation waits on that advertisement.
**Trace-linked cross-replica retry is absent.** There is no per-row client-asserted trace channel
at the merge layer, so the aggregate does not report one. It is absent rather than reported as
zero, which are different claims.
**Federation is a different feature and remains preview.** Static cross-fleet peering under
`experimental.federation` gives a combined read view over instances that are *not* one
coordinated fleet: a reachable peer shows its card, an unreachable one is a labelled degrade,
and nothing is merged across them. It makes no claim of shared state.
The Helm values key is still `experimental.multiReplica`, and the chart still prints preview
banners referencing it, even though cross-replica governance coordination is proven on a real
cluster. Read the banner as naming the values namespace rather than the maturity of the
coordination.
## What this does not cover
Limit semantics, enforcement modes and the store-selection rules are
[governance](/docs/build/governance)'s. Letting Kubernetes move the replica count for you is
[autoscaling](/docs/operate/autoscaling). Each replica's own ledger behaves exactly as a single
node's, so the growth and retention arithmetic on [the decision ledger on
disk](/docs/operate/ledger-lifecycle) applies per replica, and so does the backup, N times over.