How a limit counts

You set two limits. One caps spend in dollars, the other caps requests per minute. Both are enforcing, and both are a number in the same config block. The rate limit stops the 301st request in a minute and not the 300th. The spend cap, in the capture below, had already recorded 12.6 times its own figure by the time it stopped anything.

That gap is not a tuning problem and it is not a defect. The two limits belong to different enforcement classes, and a class is one promise: when the counter is written, relative to the request being counted. Three of the four classes write at the admission gate. The fourth writes afterwards, off the request path, and the distance between those two moments is the overshoot.

Four classes cover all six limit types, and every type belongs to one of them. Once you know a limit’s class you can predict what a burst does to it, whether its cap can be passed, what it costs per request, what happens when you run more than one replica, and what its 429 tells the caller. The six limit types is the per-type reference; this page is the model underneath it.

The four refusals, side by side

The classes are not an internal taxonomy you have to take on trust. Every governance 429 carries an enforcement block, and that block is shaped by the class. Here is a budget_usd cap of amount_usd: 0.000001, which one request is enough to blow through.

$ curl -i -X POST http://127.0.0.1:8790/v1/chat/completions \
    -H 'content-type: application/json' \
    -H "Authorization: Bearer $TT_KEY" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is the capital of Germany?"}]}'
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Ratelimit: "org_monthly_budget";r=0;t=1912510
Retry-After: 1912510
X-Tt-Refusal-Type: budget_exceeded
X-Tt-Refused-By: org_monthly_budget
X-Tt-Request-Id: tt_i3f2naenug5aanctu6pcjrofcy

{"error":{"type":"budget_exceeded","message":"request refused by limit org_monthly_budget","refused_by":{"limit_id":"org_monthly_budget","limit_type":"budget_usd","unit":"usd","node":"org:acme","period":"month","resets_at":"2026-09-01T00:00:00Z","retry_after_s":1912510,"observed":0.0000126,"limit":0.000001},"enforcement":{"posture":"enforcing","approximate":true,"bound":{"tee_lag_rows":0,"dropped_rows":0},"note":"spend from in-flight and streaming requests admitted before this refusal may not yet be reflected"}}}

That capture comes from a source build against a local sandbox with a stub OpenAI-compatible upstream, so the port, the request id and the key are ours; yours differ. The dollar figures are arithmetic over the stub’s token counts rather than anyone’s bill. Read observed against limit: 0.0000126 against 0.000001, so the cap was 12.6 times behind when it stopped the request. retry_after_s: 1912510 is 22 days, because a calendar budget resets on the next month boundary and not on a rolling window.

Three other limits, driven against the same sandbox, produce three different enforcement blocks.

Two of them claim precision and two admit an approximation, and the two approximations are not the same shape. budget_usd carries a bound object and a sentence explaining what is missing from its number. tpm says approximate: true and stops there, because a reserve is an estimate but nothing is folding behind it. Four classes, four disclosures, and the wire tells you which is which without you reading any config.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Ratelimit: "team_rpm";r=0;t=20
Retry-After: 20
X-Tt-Refusal-Type: rate_limited
X-Tt-Refused-By: team_rpm
X-Tt-Request-Id: tt_e6gbpncee32dbu7y5jpsfr4ax4

{"error":{"type":"rate_limited","message":"request refused by limit team_rpm","refused_by":{"limit_id":"team_rpm","limit_type":"rpm","unit":"requests","node":"org:acme/team:ml","period":"1m0s","retry_after_s":20},"enforcement":{"posture":"enforcing","approximate":false}}}

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Ratelimit: "team_tpm";r=0;t=39
Retry-After: 39
X-Tt-Refusal-Type: rate_limited
X-Tt-Refused-By: team_tpm

{"error":{"type":"rate_limited","message":"request refused by limit team_tpm","refused_by":{"limit_id":"team_tpm","limit_type":"tpm","unit":"tokens","node":"org:acme/team:ml","period":"1m0s","retry_after_s":39},"enforcement":{"posture":"enforcing","approximate":true}}}

{"error":{"type":"concurrency_exceeded","message":"request refused by limit user_concurrency","refused_by":{"limit_id":"user_concurrency","limit_type":"concurrency","unit":"concurrent-requests","node":"org:acme/team:ml/user:dev1","observed":0,"limit":2},"enforcement":{"posture":"enforcing","approximate":false}}}

When the counter is written

Those four disclosures come from one difference, and it is a difference in timing rather than in arithmetic. The diagram puts all four classes on a single request’s timeline so you can see where each one touches its counter.

Read the diagram left to right along one request: the admission gate, the forwarded call, the response, settlement. The three upper rows all touch their counter at the gate. rpm counts the arrival and there is nothing to hand back. concurrency takes a lease and returns it at settlement, so its number tracks the request’s whole lifetime. tpm reserves an estimated token cost and corrects it to the actual at settlement. The bottom row is the different one: at the gate it only reads, and writes nothing there, because its counter lives in the boxed region on the right, folded from ledger rows after the fact.

The four classes

LedgerSettled — spend and quota

Counts money and volume: nanodollars for budget_usd, input plus output tokens for quota_tokens, forwarded rows for quota_requests, accumulated per scope over a calendar period. A worker off the request path, the accountant, folds each finished request’s ledger row into those counters once the response is done. The gate itself writes nothing; it reads a published snapshot of the counter, republished on every ingest and on a five-second heartbeat, and trusted for thirty seconds before it counts as stale.

So the number the gate reads is behind reality, and that lag is the overshoot. It covers every request in flight, every stream still producing tokens, and every row still queued on the bounded channel feeding the accountant or dropped when that 4,096-deep queue filled. Those last two are the tee_lag_rows and dropped_rows in the bound above. A burst is the worst case for this class: the whole burst is admitted against one snapshot. Per request the gate pays one atomic load and a map probe, with no I/O and no lock. burst: is accepted on these three types and read by nothing.

AdmissionCounted — rpm

Counts arrivals, one per request. The gate calls a probe of cost 1 against the algorithm cell and the write is the decision, so there is nothing to lag behind. Within one replica the cap holds: the cell is a compare-and-swap loop over a single integer.

The one-way part matters. There is no refund on this class, so an arrival that was admitted stays counted even when the request goes on to be turned away by something else. Burst behaviour comes from the algorithm: burst: defaults to amount, so a drained gcra key admits a full burst at once and then one arrival per emission interval. With the default store: local the cost is a compare-and-swap against an in-memory cell and no I/O at all. store: redis makes it one round trip instead, bounded by a 5ms per-operation deadline that falls back to the local limiter and discloses suspended on the row.

ReserveReconcile — tpm

Counts tokens, but it has none to count at admission, so it counts an estimate. The gate reserves input estimate + output component against a gcra cell. A tpm limit is pinned to that algorithm, and it is also the one rpm runs by default. At settlement the limit commits the signed difference between the estimate and the usage that was recorded.

Between those two writes the counter overstates by whatever was over-reserved, which is the safe direction: capacity is held that nobody used, and the correction gives it back. The unsafe direction is an under-estimate. A reconcile debit is committed unconditionally, with no admit-or-decline decision attached, so an under-estimated request pushes the window past its cap and the next caller meets a fuller window than the arithmetic suggested. Cost is the acquire at the gate plus one more write at settlement. store: redis is turned away when governance compiles at boot, because the distributed script is acquire-only and cannot carry the correction.

InflightLease — concurrency

Counts requests in flight right now: an integer gauge per scope key, spread over 256 shards. The gate takes a lease, settlement returns it, and the release is idempotent, so a double release cannot free two slots. The cap cannot be passed; the acquire loop declines at the cap.

Being a live gauge is a trade. The number cannot lag, because there is no history in it, and for the same reason there is nothing to reconstruct: it is the present instant, and it drops to zero on restart. A burst meets the cap immediately, since there is no window to fill. Cost is one compare-and-swap at the gate and one decrement at the release. The gauge is per-replica; store: share divides the cap by the live replica count instead of sharing a counter.

A tpm reserve is a worst case, not a measurement

Because a tpm limit has to decide before the tokens exist, its reserve is pessimistic by construction, and the arithmetic is worth knowing before you size one. The input half is the prompt bytes divided by four. The output half is min(max_tokens, 16384), or the full 16,384 when the caller sent no max_tokens at all.

The one-sentence request in the capture above carries no max_tokens, so it reserves 16,391 tokens: seven for its prompt estimate, 16,384 for an output nobody bounded. Against a 20,000-token window the success headers say so, and they ride every served response because governance.keys.disclose_limits defaults to on.

Ratelimit-Policy: "team_tpm";q=20000;w=60
X-Ratelimit-Limit-Tokens: 20000
X-Ratelimit-Remaining-Tokens: 3609

The reserve is given back at settlement, so an over-reserve costs nothing beyond the seconds it was held. What it does cost is headroom, and the product guards the pathological case at boot rather than letting you discover it in production. burst: on a tpm limit must be at least the 16,384 reserve cap, or the process declines to start:

Error: governance: compile limits: govern: limit "team_tpm": tpm limit "team_tpm": burst 500 is below the reserve cap 16384 — an ordinary max-token reserve could never fit (a permanent 100% refusal); raise burst (default is amount 500) to at least the reserve cap

burst: defaults to amount, so this is also the practical floor on a tpm limit’s amount. Note where that message comes from: tokentriage validate does not run any limit type’s compile step, so it reports the config clean and the guard fires at boot. Callers that set max_tokens reserve less and meet fewer 429s; the other half of the reserve moves with the size of the prompt they send.

Order decides which limit answers

With several limits selected for a request, the engine walks them within a gate deepest node first, then by limit id, then by node id, and stops at the first one that is over. It compares no amounts. So when a loose deep limit and a tight shallow one are both over, the deep one is named on the 429. “The tightest limit wins” holds only in the weak sense that any breach stops the request.

Class decides something coarser and earlier: which of the two gates a limit is evaluated at. AdmissionCounted and InflightLease run before the body is read, so they see the endpoint and nothing else. LedgerSettled and ReserveReconcile run after parsing, where the model, the provider and the token estimate exist.

The same split shows up in govern preview

govern preview replays your recorded ledger against a proposed config, and it sorts its own output into three labels rather than presenting one confident list. Those three lines are below, cut out of a longer report with # … marking what was dropped; turn governance on prints one in full, closing stanza included.

# …
EXACT — LedgerSettled recomputation (hand-count-verifiable, exact):
# …
APPROXIMATE — rate-class (label: rate_windows_approximated):
# …
  [concurrency] user_concurrency @ level:user  CANNOT-ESTIMATE: concurrency requires overlapping-flight reconstruction; the ledger records no reliable per-request flight interval — cannot estimate from history
# …

Three labels over four classes: LedgerSettled is EXACT, the two rate classes share APPROXIMATE, and InflightLease is CANNOT-ESTIMATE. The closing stanza asks the same question of the live counters, and it discloses its own lag while answering.

Notice that this is the inverse of the live gate. At admission a spend cap is the approximate one and rpm is the precise one. Over history a spend recomputation is checkable by hand, because the ledger holds the rows it folds from, while window state cannot be rebuilt from row timestamps and per-request flight intervals were not recorded at all. Same four classes, same question about where the state lives, opposite answers.

What this page does not cover

Per-type config keys, defaults and validation errors are in the six limit types, and the choice between gcra, fixed_window, sliding_window and token_bucket is in rate algorithms. Convergence bounds and what each store discloses belong to governance across replicas. Every field a 429 can carry is in the refusal wire reference. Reading what was spent, and by whom, is reporting rather than enforcement: cost analytics owns that.