GitHubBook a demoStart routing
Browse docs

Autoscaling

Turn on the HorizontalPodAutoscaler in the Helm chart — the autoscaling block, why minReplicas must be >= 2, the shared-state tier it requires, and the honest per-replica share re-derivation (and brief membership-join transient) on every scale event.

Last updated

On this page

The Helm chart ships an optional HorizontalPodAutoscaler that scales the TokenTriage StatefulSet on CPU (and, optionally, memory). It is off by default — autoscaling is a GA multi-replica knob, so turning it on means running a coordinated fleet, and the chart refuses to render an HPA that could grow past one replica without the shared-state tier underneath it.

This page shows the autoscaling block, the prerequisites the chart enforces, and the one honest consequence you should understand before enabling it: on a scale event, a store: share ÷N limit re-derives its per-replica bound from the new live membership.

What the HPA needs first

autoscaling.enabled: true grows the fleet between minReplicas and maxReplicas. Because any fleet larger than one replica must coordinate governance, the chart’s topology guard (G3) gates on the ceiling of the reachable fleet — the max of replicaCount, minReplicas, and maxReplicas — and refuses to render unless the full shared-state tier is present:

  • experimental.multiReplica.enabled: true — the preview gate for any fleet > 1.
  • externalState.redis — enabled, with the external: "allow" consent string.
  • externalState.postgres — enabled, with the external: "allow" consent string.

Miss the tier and helm install/helm template fails before a single manifest is emitted:

Error: replicaCount > 1 requires experimental.multiReplica.enabled plus
externalState.redis and externalState.postgres — the shared-state tier that
COORDINATES governance across replicas (keys, budgets, rate/quota/concurrency
limits; fail-open and disclosed). Each replica keeps its own canonical ledger by
design; the fleet view is a read-time coverage-typed merge, and scaling without
the shared-state tier would run N uncoordinated gateways.

Enable autoscaling

The chart maps externalState.redis / externalState.postgres onto the binary’s shared_state config (that key is chart-managed — set it through this seam, never through config.shared_state). A minimal enabling values file:

autoscaling:
  enabled: true
  minReplicas: 2                        # chart default; the fleet floor. Keep >= 2.
  maxReplicas: 6                        # chart default; the HPA ceiling.
  targetCPUUtilizationPercentage: 80    # chart default; 0 would omit the CPU metric.
  targetMemoryUtilizationPercentage: 0  # 0 => omit the memory metric (default off).

experimental:
  multiReplica:
    enabled: true                       # PREVIEW gate — required for any fleet > 1.

externalState:                          # → shared_state.* (config is chart-managed)
  redis:
    enabled: true
    external: "allow"                   # the binary's own consent string (required)
    addr: redis.data.svc:6379
    auth: { existingSecret: tt-redis, key: password }
  postgres:
    enabled: true
    external: "allow"
    dsn: { existingSecret: tt-postgres, key: dsn }
helm upgrade --install tt deploy/helm/tokentriage \
  --namespace tokentriage --create-namespace \
  -f autoscaling-values.yaml

Those five autoscaling values are the chart defaults verbatim — enabled: false, minReplicas: 2, maxReplicas: 6, targetCPUUtilizationPercentage: 80, targetMemoryUtilizationPercentage: 0. Two escape hatches exist for advanced setups: autoscaling.extraMetrics (raw autoscaling/v2 metric entries, appended verbatim) and autoscaling.behavior (an autoscaling/v2 scaleUp/scaleDown block, passed through).

What renders

An autoscaling/v2 HorizontalPodAutoscaler targeting the StatefulSet, with the CPU metric included and the memory metric omitted (because its target is 0):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tt-tokentriage            # derives from the release/chart fullname
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: tt-tokentriage
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

replicaCount is only the seed

When autoscaling.enabled is true the StatefulSet seeds its replica count at autoscaling.minReplicas, and the HPA owns the count from then on. replicaCount is not the source of truth in this mode.

No sessionAffinity is set on any Service (DS12): the data plane is stateless per request and each pod owns its own ledger, so sticky sessions would defeat load spreading without buying correctness — the fleet read merges per-pod ledgers.

The honest consequence of a scale event

TokenTriage discloses what changes when the fleet grows, and does not paper over the transient.

A governance limit with store: share enforces a global approximation via a ÷N split across the live fleet. On every scale event that split re-derives its per-replica bound from the new live membership, so the fleet ceiling holds at any N:

fleet ≤ N × ceil(amount / N_live) ≈ amount (over-admission ≤ N−1 from rounding).

The one honest caveat is a brief membership-join transient on scale-up. A newly-started pod is not in the roster until its first heartbeat, so for that short window the ÷N share is momentarily computed against the smaller live N — the fleet can briefly admit slightly more than the steady-state ceiling. This is bounded, fail-open, and disclosed, not hidden:

  • the response header x-tt-coordination (suspended on any verdict-level coordination suspension — a rate-class store fallback, a share membership_stale cap, or identity staleness);
  • the /governance/limits panel, which widens the shown bound rather than asserting a number it cannot back.

Token and request quotas converge over the larger fleet rather than ÷N-splitting, and if the shared store is unreachable coordination is suspended fail-open (traffic keeps flowing) with the degradation stamped on the response, the decision row, and the limits panel. See Scale-out for the full set of disclosed bounds and store modes.

A PDB is valid once minReplicas >= 2

A PodDisruptionBudget on a singleton is refused (it would block node drains for no benefit). The singleton guard keys on the autoscaling floor, so an autoscaled fleet with minReplicas >= 2 (even with replicaCount left at the default 1) is not a singleton and its pdb.enabled: true is accepted — the StatefulSet seeds at minReplicas.