# Why was this request refused? Start from a 429 and work back to the limit, the node, and the change that fixes it Source: https://tokentriage.com/docs/build/governance-refusals --- import Callout from '../../../components/Callout.astro'; import NextSteps from '../../../components/NextSteps.astro'; import GovernanceGateDiagram from '../../../components/GovernanceGateDiagram.astro'; Someone pastes a 429 into your issue tracker. It has a status line, a handful of headers and a JSON body, and what they want to know is which cap they hit, whose cap it was, and whether you can raise it. All three answers are already in front of you. This page works outward from that response. First the headers that name the reason and the limit. Then the body field that names the tenant the cap belongs to. Then the ledger row, which carries the same block under the same request id. After that come the traps, and a table from what you were handed to the config key that changes it. It assumes governance is armed on your gateway and that you know roughly what a limit is; [Governance](/docs/build/governance) is the front door, and [the six limit types](/docs/build/governance-limits) covers what each cap counts. The complete field contract is [every field a governance stop can carry](/docs/build/governance-refusal-reference); this page is the diagnosis rather than the catalogue. ## Read the headers first The headers are the cheapest answer available, and one of them rides every governance stop whether or not a limit was involved. ```console HTTP/1.1 429 Too Many Requests Content-Type: application/json Ratelimit: "team_rpm";r=0;t=20 Retry-After: 20 X-Tt-Refusal-Type: rate_limited X-Tt-Refused-By: team_rpm X-Tt-Request-Id: tt_e6gbpncee32dbu7y5jpsfr4ax4 {"error":{"type":"rate_limited","message":"request refused by limit team_rpm","refused_by":{"limit_id":"team_rpm","limit_type":"rpm","unit":"requests","node":"org:acme/team:ml","period":"1m0s","retry_after_s":20},"enforcement":{"posture":"enforcing","approximate":false}}} ``` That capture is a source build driving a local sandbox with a stand-in upstream, so the request id and the limit ids are that machine's. The header names and the body shape are what your own callers get. | header | what it tells you | when it is present | |---|---|---| | `x-tt-refusal-type` | the reason, from a closed set of eight | every governance stop, without exception | | `x-tt-refused-by` | the id of the limit that fired | only where one named limit fired, so absent on a 401, a 403, and a 503 that names none | | `x-tt-request-id` | the id of this request's ledger row | every response, stopped or served | | `retry-after` | whole seconds until a retry could succeed | a 429 whose limit knows a time; a 503 carries none | | `ratelimit` | the IETF structured field, `"";r=0;t=` | a 429 naming a limit | Read them in that order. `x-tt-refusal-type` says what kind of thing stopped the request, `x-tt-refused-by` says which one, and `x-tt-request-id` gets you everything else. Header names are case-insensitive and the server canonicalises them on the way out, which is why the capture shows `X-Tt-Refusal-Type` where the table shows `x-tt-refusal-type`. Match them case-insensitively. ## Then the body, which names the tenant The header names the limit. The body names whose limit it is, and that is usually the question behind the question. `error.refused_by` is one object, and `node` is the field most people are looking for without knowing it. On a breached cap it carries the tenancy node that cap is attached to, not the caller's own node. In the preceding capture the sandbox's key sits at `org:acme/team:ml/user:dev1` and the cap that stopped it sits at `org:acme/team:ml`, so this is the team's rate cap. Raising that user's own limit would change nothing. | field | what it answers | |---|---| | `limit_id` | which definition fired, matching `x-tt-refused-by` | | `limit_type` | which of the six types it is, so you know what it counts | | `unit` | `usd`, `tokens`, `requests`, or `concurrent-requests` | | `node` | the tenancy node the cap is attached to | | `period` | a calendar period like `month`, or a window like `1m0s` | | `resets_at` | the calendar boundary, on a calendar period only | | `retry_after_s` | seconds, matching the `Retry-After` header | | `observed` and `limit` | the breach numbers, present together or not at all | Two of those rows are conditional, and the conditions trip people up. `resets_at` appears on a calendar period and is absent on a rolling window. `observed` and `limit` are emitted as a pair, and a rate stop omits both, `rpm` and `tpm` alike, because their counter store cannot report the pair cheaply. Their absence is not a defect. The sibling `enforcement` block is the class disclosure, and [how a limit counts](/docs/build/governance-enforcement-classes) reads it out. The whole `refused_by` object is dropped on a 401, and on a 503 that names no limit, so `node` is guaranteed only where `refused_by` itself is. On a 403 it holds the caller's own node, since no cap was involved. ## The row carries the same block Everything so far is also on disk, under the same request id, and it got there from the same value in memory. One function stamps the response headers, the response body, and the decision row's `refused_by_limit` block before any byte goes out. A row and the response the caller saw cannot disagree, and nothing is stopped on the wire without a row being written. Take the `x-tt-request-id` off the paste and search your decision log for it. Here is the row a 401 wrote when a presented [virtual key](/docs/build/governance-keys) did not resolve: ```json { "http": { "path": "/v1/chat/completions", "dialect": "openai", "status": 401, "endpoint": "chat" }, "decision": { "governance": { "kid": "", "node": "", "verdict": "allow", "gate_us": { "a": 0, "b": 0 } }, "refused_by_limit": { "type": "invalid_key" } } } ``` Three fields mark a request that did not go through. Two of them are on that excerpt: `http.status`, and the presence of `decision.refused_by_limit`. The third is `outcome.finish_reason`, which reads `limit_refused` on every governance stop. ## The verdict field does not say refused Look at the row again. The caller got a 401, and `decision.governance.verdict` reads `allow`. That is not a bug in the capture, and it will cost you an afternoon if you meet it in a query instead of here. `verdict` holds two values and only two, `allow` and `would_refuse`. There is no third value for a stopped request. The field reports what **shadow** evaluation found: `would_refuse` names limits in `mode: shadow` that were over their cap and were recorded rather than acted on. A shadow breach stops nothing, so a row carrying `would_refuse` was admitted by governance; where such a row is also a stop, a different enforcing limit did the stopping. [Governance](/docs/build/governance#shadow-first) shows a shadow row in full. A query that counts stopped requests by filtering `decision.governance.verdict` returns zero, whatever your traffic did. Filter on the presence of `decision.refused_by_limit`, or on `outcome.finish_reason` equal to `limit_refused`, and rebuild any dashboard panel or alert written the other way. ## Symptom, cause, and what to change Eight reasons ship, and the status follows from the reason rather than the other way round. Find your `x-tt-refusal-type` in the first column. | `error.type` | status | what happened | what to change | |---|---|---|---| | `invalid_key` | 401 | the presented `ttv_` did not resolve, or none was sent while `require_key` is `required` | find the `kid` with `tokentriage keys list` and read `STATE`; a lapsed key still lists as `active`, so read `EXPIRES` too | | `scope_violation` | 403 | the endpoint, or the served model or provider, is outside the key's scope | re-mint with the value added to `--models`, `--providers` or `--endpoints`; scopes cannot be edited in place | | `rate_limited` | 429 | an `rpm` or `tpm` window is full, and `limit_type` says which | raise `amount:` on the named limit, raise `burst:` for spiky traffic (only `gcra` and `token_bucket` read it), or wait `retry_after_s` | | `quota_exhausted` | 429 | a `quota_tokens` or `quota_requests` cap is spent for the period | raise `amount:`, or wait for `resets_at`; nothing is carried forward across a period boundary | | `budget_exceeded` | 429 | a `budget_usd` cap is spent for the period | raise `amount_usd:`, or store a per-node override where the template declares `override: { allow: true, max_amount_usd: … }`, which enforces at the next governance reload | | `concurrency_exceeded` | 429 | the node is at its in-flight `amount` | raise `amount:`; a lease frees when a request finishes, so no clock helps | | `quota_cost_unfittable` | 429 | one request's token reserve is larger than the whole `tpm` burst, so no wait admits it | raise `burst:` to at least the reserve the caller takes, or have the caller send a smaller `max_tokens` | | `enforcement_unavailable` | 503 | the enforcement question could not be answered, and something was set to stop rather than serve | clear the underlying fault, or set `posture: { fail: open }` on the limit named in `x-tt-refused-by` | Two rows have narrower causes than they look. `quota_cost_unfittable` can only come from a `tpm` limit: the one other rate type, `rpm`, probes at a cost of one unit, and one unit always fits a burst floored at one. And `scope_violation` names the violated dimension in `error.message`, with a `refused_by` carrying only the holder's own node, so the body will not list what the key is allowed to reach. ## Which gate stopped it, and in what order A limit's cap is not the only thing that decides the answer. Where the limit is evaluated decides what the caller can be told at all, and which of several over-cap limits gets named. Read the left column downward as one request and the right column as what the caller can be told at that point. StageA runs before a byte of the body is parsed, so it holds the credential and the endpoint but not the model: it is the only moment that can answer 401, and it is where `rpm` and `concurrency` caps are evaluated. StageB runs after parsing and routing, where spend and quota are probed and the `tpm` reserve is taken. ServedCheck re-asserts model and provider scope against the model that will serve. Settlement stops nothing, because the outcome is already decided. Inside one gate, the engine orders the selected instances by node depth descending, then limit id ascending, then node id ascending, and stops at the first one over its cap. It compares no amounts. The sandbox behind the captures on this page resolved four caps for one key, which is enough to see both consequences: 1. `user_concurrency` at `org:acme/team:ml/user:dev1`, depth 3, evaluated at StageA 2. `team_rpm` at `org:acme/team:ml`, depth 2, evaluated at StageA 3. `team_tpm` at `org:acme/team:ml`, depth 2, evaluated at StageB 4. `org_monthly_budget` at `org:acme`, depth 1, evaluated at StageB The first consequence is that the **deepest** limit that is over gets named, not the tightest. `team_tpm` and `org_monthly_budget` are both evaluated at StageB, so when both are over, the 429 says `team_tpm`, because it sits a level deeper. The org budget can be the harder wall and still go unmentioned. All four of those caps produced a 429 of their own against this sandbox as their counters moved, each naming itself. The second consequence is quieter. `team_rpm` counts the arrival at StageA and there is no refund on that class, so a request that goes on to be stopped by `org_monthly_budget` at StageB has already spent its arrival. Leases and reserves are handed back at settlement; a counted arrival is not. Your rate window therefore drains slightly faster than your served traffic, which is expected rather than drift. ## Two answers that are not a breached cap **A 401 tells the caller almost nothing, and it withholds on both surfaces.** There is no limit id, no node, and no `refused_by` object, on the wire or on the row. One uniform message covers an unknown key, a revoked key and an expired one alike. So a caller cannot use a 401 to discover whether a tenant or a credential exists, and you cannot diagnose one from the response body either. Go to `tokentriage keys list` and the key's `STATE` and `EXPIRES` columns instead. **A 503 `enforcement_unavailable` is not a breached cap.** It means the gate could not answer the enforcement question and was configured to stop rather than serve. Two paths reach it: a limit that could not be checked while carrying `fail: closed` with `enforce: hard`, and `require_key: required` on a replica with no loadable key snapshot, which a cold replica can serve for the moments before its key store seeds. The first names its limit in `x-tt-refused-by`; the second names nothing, because no limit was involved. Neither carries a `Retry-After`, since there is no time at which the retry is known to work. Every other internal fault degrades to an allow with a disclosure, which [Governance](/docs/build/governance#fail-open-with-disclosure) covers. ## When the numbers look wrong A `budget_usd` stop can report an `observed` larger than its own `limit`, and that is the design rather than a fault: ```json {"error":{"type":"budget_exceeded","message":"request refused by limit org_monthly_budget","refused_by":{"limit_id":"org_monthly_budget","limit_type":"budget_usd","unit":"usd","node":"org:acme","period":"month","resets_at":"2026-09-01T00:00:00Z","retry_after_s":1912510,"observed":0.0000126,"limit":0.000001},"enforcement":{"posture":"enforcing","approximate":true,"bound":{"tee_lag_rows":0,"dropped_rows":0},"note":"spend from in-flight and streaming requests admitted before this refusal may not yet be reflected"}}} ``` `observed` is 0.0000126 against a `limit` of 0.000001, so the counter was 12.6 times past its cap when it acted. The dollar figures are arithmetic over a stub upstream's token counts and are nobody's bill. The body says why in the same breath: `approximate: true`, plus a `note` naming what is missing from the number. A spend counter is folded from ledger rows off the request path, so admission reads a snapshot that does not yet contain anything still in flight. [How a limit counts](/docs/build/governance-enforcement-classes) has the mechanism and the size of the gap. `observed` on a `concurrency_exceeded` body is hard-coded to `0` in this build, so it reads `0` of `2` at the instant two requests were in flight. Read `limit` and treat `observed` as absent; the live gauge is on `GET /tt/v1/me`. This is catalogued in [what governance does not do](/docs/build/governance-gaps). ## Reading refusals in bulk One 429 answers one question. A pattern needs the feed, and two admin-plane endpoints serve it. Both take a `node` query parameter that scopes the answer to a subtree, and both sit at the viewer role floor. `GET /tt/api/v1/governance/refusals` is the live view: newest first, `limit` defaulting to 100 and clamped at 1000, cursor-paginated, with the older `offset` parameter deprecated because an offset into a growing feed re-reads and skips rows. It is bounded at 1000 rows and `page.total` saturates there, so treat it as the recent window rather than the archive. Each item carries `ts`, `req_id`, `node`, `kid`, `type`, `limit_id`, `limit_type`, `posture`, `approximate` and `http_status`, and a 401 item carries no `limit_id` for the same reason the wire does not. `GET /tt/api/v1/governance/exports/refusals` is the archive twin: complete by default, window-scoped, and available as JSON or CSV through `format=`. The CSV columns are frozen in the order of that field list, and the file opens with `#`-prefixed comment lines that are the export's provenance stamp; strip the `# ` and the block is the JSON header object, carrying the schema version, `generated_at`, the instance, the window, the row count and a `demo` flag. A partial scan appends a `# truncated: true` line rather than presenting a short file as whole. [The management API](/docs/build/governance-api) has both operations in full. ## What this page does not cover The field-by-field contract, including every header and every conditional field, is [the refusal wire reference](/docs/build/governance-refusal-reference). Why a counter can be passed before it acts is [how a limit counts](/docs/build/governance-enforcement-classes). Which caps a given key resolves to in the first place, and how an override raises one, is [the tenancy hierarchy](/docs/build/governance-hierarchy). Turning a limit on without stopping traffic while you learn its shape is [turn governance on](/docs/build/governance-rollout). What a tenant spent, as opposed to what stopped them spending it, is reporting rather than enforcement: [cost analytics](/docs/build/cost-analytics) owns that.