Sizing and capacity

Capacity planning for a gateway usually means guessing at CPU. Here the dominant resource is storage, because TokenTriage’s job is to keep a durable record of every request, and that record is the only thing on the machine that grows without you doing anything.

This page separates what has been measured from what has not. The storage arithmetic is measured and you can reproduce it. The compute figures are not, and rather than manufacture them, this page says so and points at the chart’s starting point.

The one number to derive yourself

Ledger size is the product of three things you control: how many requests you take, how large a row is for your traffic, and how long you keep the history.

active file   ≈ rotate_mb                          (bounded, by construction)
history       ≈ requests × row_bytes ÷ 53          (compressed companions)
volume        ≈ rotate_mb + history + headroom

The 53 is a compression ratio measured on real rows, and the section after next explains where it comes from and when to distrust it.

Measure row_bytes on your own traffic rather than adopting a figure. Twenty single-message Anthropic completions through a local mock gave this:

$ wc -c -l data/decisions.jsonl
   20 34620 data/decisions.jsonl

1,731 bytes per row. Rows grow with the number of messages, tool definitions and business dimensions you attach, and grow again in shadow mode, which records the would-route decision alongside the actual one. A row costed against a model with cache tiers carries a few more fields. None of that changes the shape of the arithmetic, only the constant.

Worked: a hundred thousand requests a day

Take a deployment serving 100,000 requests a day with rows of the measured size, keeping ninety days of history.

Quantity	Value	Where it comes from
rows per day	100,000	your traffic
raw bytes per day	~165 MiB	100,000 × 1,731
compressed per day	~3.2 MiB	÷ 53
ninety days of history	~280 MiB	× 90
active file	≤ 256 MiB	rotate_mb default
volume	1 GiB is comfortable	sum, doubled for headroom

The striking part is that the bounded active file dominates a ninety-day history. That is the right instinct to carry away: rotation, not retention, is what keeps a ledger volume small, and history is nearly free.

Set the policy that produces the second row of that table:

ledger:
  decision_log:
    rotate_mb: 256 # default; bounds the active file
retention:
  max_age: 2160h # ninety days
  max_total_mb: 512 # a ceiling regardless of age

Both retention limits default to 0, which means keep forever, and the sweeper only runs when the dashboard is enabled. The decision ledger on disk covers both traps.

Where the compression ratio comes from

Driving 2,500 requests through an instance with rotate_mb: 1 produced four companions:

$ ls -la data/
-rw-r--r-- 1  19668 decisions-20260810T125338-1.jsonl.gz
-rw-r--r-- 1  19588 decisions-20260810T125338-2.jsonl.gz
-rw-r--r-- 1  19635 decisions-20260810T125338-3.jsonl.gz
-rw-r--r-- 1  19647 decisions-20260810T125338.jsonl.gz
-rw-r--r-- 1 159192 decisions.jsonl

Each holds a full mebibyte of rows in about 19.6 KB. Divide 1,048,576 by 19,647 and the ratio is 53.4.

Distrust that ratio when your traffic is less uniform than the run behind it. Those rows shared one model, one prompt shape and one price, which is close to the best case for gzip on JSON. A mixed workload across many models with varied dimensions compresses less. The number is worth using as a planning estimate and worth re-measuring from your own companions once you have a week of them:

$ ls -la /data/.tokentriage/*.jsonl.gz

What saturates first

Three things can become the constraint, and they fail in different ways.

Storage is the one that ends the deployment. A full volume stops the writer, and the writer drops rather than blocking, so traffic keeps flowing while the record of it stops. That failure is silent on every surface except a counter. Alert on free bytes, because nothing ships that does.

The writer queue is the early warning for storage. It holds 8,192 rows by default and drops past that, counting each one. A moving tokentriage_declog_dropped_total on an instance whose disk is not yet full means the volume is slow rather than small.

CPU is what the horizontal autoscaler watches, and the reason to add replicas rather than grow one. Adding replicas is not free: it requires the shared-state tier so governance stays coordinated, and each replica keeps its own ledger on its own volume, so your storage arithmetic multiplies by N. Autoscaling covers the gate and the transient.

What is not measured

Three figures an operator reasonably wants are not published here, and inventing them would be worse than the gap.

Steady-state memory. No figure is published. The chart’s suggested starting point — requests: {cpu: 250m, memory: 256Mi}, limits: {cpu: "1", memory: 512Mi} — is guidance for a single-node install rather than a measurement of one, and resources is empty by default so the chart imposes nothing you did not choose. Set requests from what your own deployment uses under load, and treat the suggestion as a place to start.

CPU per request. Not published as a per-request figure. What has been measured, on bare metal under a documented harness, is the gateway’s added latency: the overhead TokenTriage contributes on top of the provider call. That is a different quantity from CPU cost and it lives on the benchmarks page with its conditions.

File-descriptor behaviour. Not measured. The proxy holds connections to upstreams plus the ledger and index handles; no published figure bounds that against concurrency. Size your ulimit generously and watch it rather than trusting an estimate from us.

Artifact sizes, for image pull and mirror planning

The deployed artifact is one static binary in a FROM scratch image, with no base layer, no shell and no package manager. The layers, from a local build of the compose image:

Layer	Size
the binary	38.6 MB
vendored license texts	1.85 MB
CA certificate bundle	197 kB
baked config, user database, data dir	~37 kB

That binary is from a plain go build with debug information present; the release pipeline strips it, so a published image is smaller. Both figures are far below the point where image size drives a mirroring decision. Install air-gapped covers what crosses a gap in practice.