# Multi-tenancy patterns Four tenancy shapes you can paste and adapt, with what each one guarantees, what it costs, and how it fails. Source: https://tokentriage.com/docs/build/governance-tenancy-patterns --- import Callout from '../../../components/Callout.astro'; import Details from '../../../components/Details.astro'; import NextSteps from '../../../components/NextSteps.astro'; You have an org chart, a customer list, or a set of background jobs that behave nothing like either. Before you can cap anything you have to decide what a tenant is, and that decision is expensive to change afterwards: it names every counter, every key, and every row you will later query. Four shapes cover most of what people build. Each is a `governance:` block the shipped binary loads and compiles, with the tree, the limits, what it guarantees, what it costs, and where it lets you down. Read [the tenancy hierarchy](/docs/build/governance-hierarchy) first; it owns the walk from a key to a set of caps, and everything here assumes it. Every limit below carries `posture: { mode: shadow }`, because a limit written without that block starts turning traffic away the moment it arms. [Turn governance on](/docs/build/governance-rollout) is the ladder from there. ## The vocabulary you get Three levels are declarable, and the four patterns are four ways of deciding what those three words mean. ```yaml governance: tenants: - org: northwind # whatever you decide an org is teams: [billing] # whatever you decide a team is users: { billing: [svc_invoice] } # and whatever you decide a user is ``` `org`, `team` and `user` are node kinds, not job titles. Nothing checks that a team is a team, and a level you have no use for can be dropped from the bottom: an org with teams and no users is a complete tree. Dropping from the middle is what config stops, because a `users:` group has to name a team you declared in `teams:`. A fourth rung exists on the principal's chain, the key leaf `org:northwind/team:billing/user:svc_invoice/key:vk_…`, and it is not a tree node, so no limit attaches there. That leaves three levels to spend, and not always on the thing your HR system models. ## One tenant per customer The SaaS shape: an org per paying customer, one spend template that every customer picks up, and a rate cap you can vary per customer. ```yaml governance: enabled: true require_key: required tenants: - org: northwind teams: [ingest, assistant] - org: contoso teams: [assistant] limits: - id: customer_monthly_budget # the template every customer lands on attach: { level: org } type: budget_usd amount_usd: 500 period: month posture: { mode: shadow } override: { allow: true, max_amount_usd: 5000 } - id: northwind_rpm # one pinned rate cap per customer attach: { node: "org:northwind" } type: rpm amount: 600 period: 1m posture: { mode: shadow } - id: contoso_rpm # contoso bought a bigger one attach: { node: "org:contoso" } type: rpm amount: 3000 period: 1m posture: { mode: shadow } ``` **What it guarantees.** Each customer org carries its own monthly dollar counter and its own rate cell, so a customer running hot cannot consume another's remaining budget. Adding a customer to `tenants:` gives them the $500 template with no edit to any limit. **How a request resolves.** A key owned by `org:northwind/team:ingest` has the chain `…/team:ingest/key:vk_…`, `org:northwind/team:ingest`, `org:northwind`, `__global`. Two instances are selected: `customer_monthly_budget @ org:northwind`, from the org-level template, and `northwind_rpm @ org:northwind`, pinned. `contoso_rpm` sits on a node outside this chain and contributes nothing. The rate cap settles at the first gate, before the body is read; the budget at the second, after routing has chosen a model. Mint one key per customer service rather than one per customer, with `tokentriage keys mint --owner org:northwind/team:ingest --label northwind-ingest`, so a leaked ingest credential can be rotated without touching the customer's assistant. **What it costs.** One spend counter per customer, folded off the request path, and one rate cell per customer in memory. The budget scales with no config change; the rate cap costs one limit block per customer, so the file grows with your customer list. **How it fails.** A customer who needs a different rate cap gets a pinned limit at their own node, because `rpm` does not accept a per-node override. What does not work is keeping an org-level `rpm` template and pinning a larger one over the top: both instances bind the same request, nothing compares their amounts, and the smaller one stops it. Give each customer's rate cap one home. The monthly budget can be crossed before it acts, because `budget_usd` reads a counter folded from settled rows and spend already in flight is missing from it. The 429 publishes that gap as `observed` beside `limit`. Raising one customer to $5,000 is a per-node override against the ceiling this template declares, in force at the next governance reload. ## One tenant per internal team The common first deployment: one org for the company, a team per group, and users below for attribution rather than enforcement. ```yaml governance: enabled: true require_key: required tenants: - org: acme teams: [support, ml, search] users: support: [alice, bob] ml: [dev1] limits: - id: team_monthly_budget # every team gets the same $2,000 attach: { level: team } type: budget_usd amount_usd: 2000 period: month posture: { mode: shadow, thresholds: [0.5, 0.8, 0.95] } override: { allow: true, max_amount_usd: 10000 } - id: team_rpm attach: { level: team } type: rpm amount: 600 period: 1m burst: 60 posture: { mode: shadow } - id: org_monthly_budget # the backstop the teams cannot sum past attach: { node: "org:acme" } type: budget_usd amount_usd: 8000 period: month posture: { mode: shadow } ``` **What it guarantees.** "The support team must not spend more than $2,000 a month" is the template on its own: every team gets $2,000, and support is left on it. A team that needs more is raised by a per-node override, bounded by the $10,000 ceiling declared here. `org_monthly_budget` sits in every team's chain, so however high the team caps are raised, the company still stops at $8,000. **How a request resolves.** A key at `org:acme/team:support/user:alice` selects `team_rpm @ org:acme/team:support` at the first gate, then `team_monthly_budget @ org:acme/team:support` and `org_monthly_budget @ org:acme` at the second, in that order: deeper nodes first, then limit id ascending. The user node selects nothing, and that is what it is for. It exists so the row's `node` and `serving_key` name a person rather than a group. **What it costs.** Two counters per team plus one for the org. The user level adds nodes to the tree and no enforcement work. Thresholds default to none; the three written here fire at 50%, 80% and 95% of the team cap and stop nothing on their own. **How it fails.** The $2,000 is a soft edge, for the same reason as the customer budget: a settled counter lags the requests in flight. The second failure bites during rollout. Adding this limit set through a hot reload starts every new counter at zero, so a cap added on the 12th governs the rest of the month rather than all of it; only a process restart backfills the current period from ledger history. Editing an amount behaves the other way, carrying the counter forward, so tightening $2,000 to $1,200 applies against the burn already accumulated. ## One tenant per feature or agent Your org chart is often not the thing that overspends. Nothing requires the tree to model people, so name the nodes after workloads instead. ```yaml governance: enabled: true require_key: required tenants: - org: platform teams: [interactive, batch, nightly_agent] limits: - id: interactive_rpm attach: { node: "org:platform/team:interactive" } type: rpm amount: 1200 period: 1m posture: { mode: shadow } - id: batch_daily_budget attach: { node: "org:platform/team:batch" } type: budget_usd amount_usd: 300 period: day posture: { mode: shadow } - id: nightly_agent_daily_budget attach: { node: "org:platform/team:nightly_agent" } type: budget_usd amount_usd: 50 period: day posture: { mode: shadow } - id: nightly_agent_concurrency attach: { node: "org:platform/team:nightly_agent" } type: concurrency amount: 4 posture: { mode: shadow } - id: embeddings_rpm # the orthogonal axis: a dim selector attach: { node: "org:platform" } type: rpm amount: 3000 period: 1m match: { endpoint: embeddings } posture: { mode: shadow } - id: platform_month # the fleet-wide spelling attach: { node: "__global" } type: budget_usd amount_usd: 4000 period: month posture: { mode: shadow } ``` **What it guarantees.** The nightly agent has $50 a day and four in-flight slots. When it loops, it drains its own counters and starts collecting 429s while the interactive endpoint keeps serving, because the two resolve into different cells. `platform_month` pinned at `__global` is the total nobody passes, and it is the one attachment that also reaches traffic carrying no key, which starts mattering the moment `require_key` is anything softer than `required`. **How a request resolves.** The agent's key, owned by `org:platform/team:nightly_agent`, selects `nightly_agent_concurrency` and `nightly_agent_daily_budget` at its own node plus `platform_month @ __global`. `embeddings_rpm @ org:platform` joins the set only on an embeddings call, since `match:` is tested per limit before any node fan-out. `interactive_rpm` is outside this chain. `match:` is the axis orthogonal to the tree, with a closed vocabulary of `model`, `provider` and `endpoint`. It narrows what a limit counts, and it splits the counter: one cell per distinct value of every dim the limit constrains. `match.model` and `match.provider` cannot fire on `rpm` or `concurrency`. Both types are evaluated before the request body is parsed, so those two dims are empty strings at that moment and the limit is dropped from the set. It loads, validates, compiles, and counts nothing. Narrow those two types with `match.endpoint`, which is known at that gate, or move the cap onto `tpm` or `budget_usd`, which run after the body is read. **What it costs.** One counter per node per limit. The dim-scoped one keeps a separate cell per distinct value it sees, which for `embeddings_rpm` is one. The concurrency gauge is held for the life of the process rather than reclaimed. **How it fails.** The concurrency gauge lives in one replica's memory, so a restart drops it to zero and a rolling restart briefly admits more than four. Its 429 body also reports `"observed": 0` beside a live `limit`, so read the gauge from `GET /tt/v1/me` rather than from the response the agent saw. ## A reseller, two levels down When your customer has their own customers, all three levels are in play at once. ```yaml governance: enabled: true require_key: required tenants: - org: northwind # your customer teams: [billing, support] # their departments users: billing: [svc_invoice, svc_dunning] # their service accounts support: [svc_triage] - org: globex teams: [research] users: research: [svc_lit] limits: - id: customer_month # what the customer bought attach: { level: org } type: budget_usd amount_usd: 2500 period: month posture: { mode: shadow } override: { allow: true, max_amount_usd: 25000 } - id: department_month # how they split it attach: { level: team } type: budget_usd amount_usd: 500 period: month posture: { mode: shadow } override: { allow: true, max_amount_usd: 2500 } - id: service_rpm # one pace cap per service account attach: { level: user } type: rpm amount: 120 period: 1m posture: { mode: shadow } ``` **What it guarantees.** Two nested spend ceilings and a pace cap at the leaf. A department can be raised to $2,500 without your customer's $2,500 moving, and the customer cap binds every department under it whatever the departments were individually allowed. **How a request resolves.** A key at `org:northwind/team:billing/user:svc_invoice` selects three instances, deepest first: `service_rpm @ …/user:svc_invoice` at depth three, `department_month @ …/team:billing` at depth two, and `customer_month @ org:northwind` at depth one. The rate cap settles at the first gate and both budgets at the second. **What it costs.** The user-level template makes one rate cell per service account, so counter count tracks your customers' service accounts rather than your customer count. This is also the pattern where `tenants:` becomes an inventory. Nodes can instead be created through the management API into the runtime plane, which keeps the YAML small; a runtime-created node has no effect on the live gate until the next governance reload. **How it fails.** Depth runs out. Four rungs exist and only three are nodes, so a reseller whose customers have customers has nowhere to put the fourth, and no config adds a level. Two ways out. Flatten, giving each end customer its own org and moving the reseller-wide roll-up to [cost analytics](/docs/build/cost-analytics) rather than enforcement. Or encode one rung down: put the end customer at `team:` and their department at `user:`, and take per-credential attribution from the key, since every row a key served carries `serving_key.kid` whether or not a node exists for it. `keys mint --owner` accepts any string. A typo mints a key whose chain resolves against nodes that do not exist, so only `__global` limits attach and the tenant runs uncapped, with no warning at mint, at boot, or on the row. Confirm every new key with one call to `GET /tt/v1/me` and read the `chain` it prints before you hand the secret over; the repair is to revoke and mint again. ## Before you boot any of them All four blocks on this page were loaded by the shipped binary. `tokentriage validate` returns `OK: configuration is valid` for each, and `tokentriage govern preview` compiles each against the same limit-type registries the proxy arms with. Run both, because `validate` does not compile limits. A clean `validate` is not evidence that the block boots: the per-type guards live in the compiler, and a config that trips one stops the process at startup. `govern preview` runs that compile before it reads your ledger, so a compile error surfaces even against an empty log. Two lines carry that, and the run below is cut back to them with `# …` marking what was dropped; [turn governance on](/docs/build/governance-rollout#preview-the-cap-against-traffic-you-already-have) prints a report in full. ```console $ tokentriage govern preview -c per-feature.yaml --log ./empty.jsonl # … rows replayed: 0 # … [budget_usd] batch_daily_budget @ org:platform/team:batch burn $0.000000 / limit $300.000000 (frac 0.0000, rows 0, unpriced_excluded 0) # … ``` That ran against the per-feature block on an empty decision log created for the check, so every figure is zero and the node names are the point. Each line is one scope the recomputation folded, and a pinned limit gets its line either way. A template is the weaker case: it resolves to per-node lines only once the log carries rows underneath it, and until then a rate template prints `@ level:team` while a spend template is absent from the `EXACT` block. Every limit in this set is pinned, which is why each line names a node. ## Choices that apply to all four **Where to put the budget.** Higher is a firmer guarantee: an org cap sits in every descendant's chain, so it binds whatever the teams below were individually allowed. Lower is a better blast radius: a team that exhausts its own cap stops only itself. Most deployments want both, which is what the per-team pattern does. The price is that a caller can be stopped by a cap it was not watching, and `GET /tt/v1/me` is where it can see the shallow ones. **Template or pinned.** A template fans out to every node at its level and picks up new nodes as you declare them. A pin is one instance at one node. Use a template for the cap that is the same for everyone and a pin for the exception, remembering that both bind when both are selected. `override:` belongs to templates. Moving it onto the pinned `northwind_rpm` stops `validate`. The same line lands twice, because the error a command returns is printed once by the CLI framework and once by the binary's top level: ```console $ tokentriage validate --config per-customer.yaml Error: configuration is invalid: config per-customer.yaml: governance.limits[1].override: override is meaningful only on a template (level) limit Error: configuration is invalid: config per-customer.yaml: governance.limits[1].override: override is meaningful only on a template (level) limit ``` `allow` defaults to `false`, so a template with no `override:` block accepts no per-node override. The ceiling is the other half of that key, and omitting it is the mistake worth naming. `override: { allow: true }` with no `max_amount` or `max_amount_usd` declares no ceiling, and any positive amount is then accepted for any node. An override also reaches only `budget_usd`, `quota_tokens` and `quota_requests`; on `rpm`, `tpm` and `concurrency` it is turned down with a reason, which is why every rate cap here is pinned per node. **Turning `require_key` on.** It defaults to `off`. Set to `required`, it answers anonymous traffic with a 401 from the moment governance arms, and traffic that predates your keys carries no `ttv_`, so all of it stops at once. Roll it in two steps: set `optional`, which arms governance and still admits anonymous requests, mint and distribute keys, then flip to `required` once the anonymous share of your rows reaches zero. Both edits want a restart, unlike `governance.limits` and `governance.tenants`, which reload hot.
One warning stands while `governance.keys.tls_terminated_upstream` is unset, because bearer keys would otherwise cross cleartext HTTP. That field is an attestation you make, not a probe the binary runs, so set it once TLS terminates in front of the data plane. The other fires when `required` is paired with an empty `tenants:`, and says you are about to answer all traffic with a 401. Each is printed twice, because the config is validated once as it loads and again against the live registries, and both warning lists reach the terminal.
**What you cannot express.** Per-tenant provider credentials are not a shipped config surface. A tenancy node carries no credential field, so every tenant's traffic leaves on your credentials, read per upstream from the environment variable named in `api_key_env`. What answers "whose request was that" is `decision.governance.serving_key` on the row, carrying the `kid`, the node, and `rotated_from` for a key that came out of a rotation. [Virtual keys](/docs/build/governance-keys) has the rest. ## What this page does not cover The shapes, not the machinery. How a chain becomes a set of instances, and what an override can and cannot raise, is [the tenancy hierarchy](/docs/build/governance-hierarchy). What each type counts and where it can overshoot is [the six limit types](/docs/build/governance-limits). Who may write an override inside a subtree is [delegated self-serve](/docs/build/governance-self-serve). Turning these rows into spend per tenant is [cost analytics](/docs/build/cost-analytics).