Multi-tenancy patterns — TokenTriage Docs

You have an org chart, a customer list, or a set of background jobs that behave nothing like either. Before you can cap anything you have to decide what a tenant is, and that decision is expensive to change afterwards: it names every counter, every key, and every row you will later query.

Four shapes cover most of what people build. Each is a governance: block the shipped binary loads and compiles, with the tree, the limits, what it guarantees, what it costs, and where it lets you down.

Read the tenancy hierarchy first; it owns the walk from a key to a set of caps, and everything here assumes it. Every limit below carries posture: { mode: shadow }, because a limit written without that block starts turning traffic away the moment it arms. Turn governance on is the ladder from there.

The vocabulary you get

Three levels are declarable, and the four patterns are four ways of deciding what those three words mean.

governance:
  tenants:
    - org: northwind # whatever you decide an org is
      teams: [billing] # whatever you decide a team is
      users: { billing: [svc_invoice] } # and whatever you decide a user is

org, team and user are node kinds, not job titles. Nothing checks that a team is a team, and a level you have no use for can be dropped from the bottom: an org with teams and no users is a complete tree. Dropping from the middle is what config stops, because a users: group has to name a team you declared in teams:. A fourth rung exists on the principal’s chain, the key leaf org:northwind/team:billing/user:svc_invoice/key:vk_…, and it is not a tree node, so no limit attaches there.

That leaves three levels to spend, and not always on the thing your HR system models.

One tenant per customer

The SaaS shape: an org per paying customer, one spend template that every customer picks up, and a rate cap you can vary per customer.

governance:
  enabled: true
  require_key: required
  tenants:
    - org: northwind
      teams: [ingest, assistant]
    - org: contoso
      teams: [assistant]
  limits:
    - id: customer_monthly_budget # the template every customer lands on
      attach: { level: org }
      type: budget_usd
      amount_usd: 500
      period: month
      posture: { mode: shadow }
      override: { allow: true, max_amount_usd: 5000 }
    - id: northwind_rpm # one pinned rate cap per customer
      attach: { node: "org:northwind" }
      type: rpm
      amount: 600
      period: 1m
      posture: { mode: shadow }
    - id: contoso_rpm # contoso bought a bigger one
      attach: { node: "org:contoso" }
      type: rpm
      amount: 3000
      period: 1m
      posture: { mode: shadow }

What it guarantees. Each customer org carries its own monthly dollar counter and its own rate cell, so a customer running hot cannot consume another’s remaining budget. Adding a customer to tenants: gives them the $500 template with no edit to any limit.

How a request resolves. A key owned by org:northwind/team:ingest has the chain …/team:ingest/key:vk_…, org:northwind/team:ingest, org:northwind, __global. Two instances are selected: customer_monthly_budget @ org:northwind, from the org-level template, and northwind_rpm @ org:northwind, pinned. contoso_rpm sits on a node outside this chain and contributes nothing. The rate cap settles at the first gate, before the body is read; the budget at the second, after routing has chosen a model.

Mint one key per customer service rather than one per customer, with tokentriage keys mint --owner org:northwind/team:ingest --label northwind-ingest, so a leaked ingest credential can be rotated without touching the customer’s assistant.

What it costs. One spend counter per customer, folded off the request path, and one rate cell per customer in memory. The budget scales with no config change; the rate cap costs one limit block per customer, so the file grows with your customer list.

How it fails. A customer who needs a different rate cap gets a pinned limit at their own node, because rpm does not accept a per-node override. What does not work is keeping an org-level rpm template and pinning a larger one over the top: both instances bind the same request, nothing compares their amounts, and the smaller one stops it. Give each customer’s rate cap one home.

The monthly budget can be crossed before it acts, because budget_usd reads a counter folded from settled rows and spend already in flight is missing from it. The 429 publishes that gap as observed beside limit. Raising one customer to $5,000 is a per-node override against the ceiling this template declares, in force at the next governance reload.

One tenant per internal team

The common first deployment: one org for the company, a team per group, and users below for attribution rather than enforcement.

governance:
  enabled: true
  require_key: required
  tenants:
    - org: acme
      teams: [support, ml, search]
      users:
        support: [alice, bob]
        ml: [dev1]
  limits:
    - id: team_monthly_budget # every team gets the same $2,000
      attach: { level: team }
      type: budget_usd
      amount_usd: 2000
      period: month
      posture: { mode: shadow, thresholds: [0.5, 0.8, 0.95] }
      override: { allow: true, max_amount_usd: 10000 }
    - id: team_rpm
      attach: { level: team }
      type: rpm
      amount: 600
      period: 1m
      burst: 60
      posture: { mode: shadow }
    - id: org_monthly_budget # the backstop the teams cannot sum past
      attach: { node: "org:acme" }
      type: budget_usd
      amount_usd: 8000
      period: month
      posture: { mode: shadow }

What it guarantees. “The support team must not spend more than $2,000 a month” is the template on its own: every team gets $2,000, and support is left on it. A team that needs more is raised by a per-node override, bounded by the $10,000 ceiling declared here. org_monthly_budget sits in every team’s chain, so however high the team caps are raised, the company still stops at $8,000.

How a request resolves. A key at org:acme/team:support/user:alice selects team_rpm @ org:acme/team:support at the first gate, then team_monthly_budget @ org:acme/team:support and org_monthly_budget @ org:acme at the second, in that order: deeper nodes first, then limit id ascending. The user node selects nothing, and that is what it is for. It exists so the row’s node and serving_key name a person rather than a group.

What it costs. Two counters per team plus one for the org. The user level adds nodes to the tree and no enforcement work. Thresholds default to none; the three written here fire at 50%, 80% and 95% of the team cap and stop nothing on their own.

How it fails. The $2,000 is a soft edge, for the same reason as the customer budget: a settled counter lags the requests in flight.

The second failure bites during rollout. Adding this limit set through a hot reload starts every new counter at zero, so a cap added on the 12th governs the rest of the month rather than all of it; only a process restart backfills the current period from ledger history. Editing an amount behaves the other way, carrying the counter forward, so tightening $2,000 to $1,200 applies against the burn already accumulated.

One tenant per feature or agent

Your org chart is often not the thing that overspends. Nothing requires the tree to model people, so name the nodes after workloads instead.

governance:
  enabled: true
  require_key: required
  tenants:
    - org: platform
      teams: [interactive, batch, nightly_agent]
  limits:
    - id: interactive_rpm
      attach: { node: "org:platform/team:interactive" }
      type: rpm
      amount: 1200
      period: 1m
      posture: { mode: shadow }
    - id: batch_daily_budget
      attach: { node: "org:platform/team:batch" }
      type: budget_usd
      amount_usd: 300
      period: day
      posture: { mode: shadow }
    - id: nightly_agent_daily_budget
      attach: { node: "org:platform/team:nightly_agent" }
      type: budget_usd
      amount_usd: 50
      period: day
      posture: { mode: shadow }
    - id: nightly_agent_concurrency
      attach: { node: "org:platform/team:nightly_agent" }
      type: concurrency
      amount: 4
      posture: { mode: shadow }
    - id: embeddings_rpm # the orthogonal axis: a dim selector
      attach: { node: "org:platform" }
      type: rpm
      amount: 3000
      period: 1m
      match: { endpoint: embeddings }
      posture: { mode: shadow }
    - id: platform_month # the fleet-wide spelling
      attach: { node: "__global" }
      type: budget_usd
      amount_usd: 4000
      period: month
      posture: { mode: shadow }

What it guarantees. The nightly agent has $50 a day and four in-flight slots. When it loops, it drains its own counters and starts collecting 429s while the interactive endpoint keeps serving, because the two resolve into different cells. platform_month pinned at __global is the total nobody passes, and it is the one attachment that also reaches traffic carrying no key, which starts mattering the moment require_key is anything softer than required.

How a request resolves. The agent’s key, owned by org:platform/team:nightly_agent, selects nightly_agent_concurrency and nightly_agent_daily_budget at its own node plus platform_month @ __global. embeddings_rpm @ org:platform joins the set only on an embeddings call, since match: is tested per limit before any node fan-out. interactive_rpm is outside this chain.

match: is the axis orthogonal to the tree, with a closed vocabulary of model, provider and endpoint. It narrows what a limit counts, and it splits the counter: one cell per distinct value of every dim the limit constrains.

What it costs. One counter per node per limit. The dim-scoped one keeps a separate cell per distinct value it sees, which for embeddings_rpm is one. The concurrency gauge is held for the life of the process rather than reclaimed.

How it fails. The concurrency gauge lives in one replica’s memory, so a restart drops it to zero and a rolling restart briefly admits more than four. Its 429 body also reports "observed": 0 beside a live limit, so read the gauge from GET /tt/v1/me rather than from the response the agent saw.

A reseller, two levels down

When your customer has their own customers, all three levels are in play at once.

governance:
  enabled: true
  require_key: required
  tenants:
    - org: northwind # your customer
      teams: [billing, support] # their departments
      users:
        billing: [svc_invoice, svc_dunning] # their service accounts
        support: [svc_triage]
    - org: globex
      teams: [research]
      users:
        research: [svc_lit]
  limits:
    - id: customer_month # what the customer bought
      attach: { level: org }
      type: budget_usd
      amount_usd: 2500
      period: month
      posture: { mode: shadow }
      override: { allow: true, max_amount_usd: 25000 }
    - id: department_month # how they split it
      attach: { level: team }
      type: budget_usd
      amount_usd: 500
      period: month
      posture: { mode: shadow }
      override: { allow: true, max_amount_usd: 2500 }
    - id: service_rpm # one pace cap per service account
      attach: { level: user }
      type: rpm
      amount: 120
      period: 1m
      posture: { mode: shadow }

What it guarantees. Two nested spend ceilings and a pace cap at the leaf. A department can be raised to $2,500 without your customer’s $2,500 moving, and the customer cap binds every department under it whatever the departments were individually allowed.

How a request resolves. A key at org:northwind/team:billing/user:svc_invoice selects three instances, deepest first: service_rpm @ …/user:svc_invoice at depth three, department_month @ …/team:billing at depth two, and customer_month @ org:northwind at depth one. The rate cap settles at the first gate and both budgets at the second.

What it costs. The user-level template makes one rate cell per service account, so counter count tracks your customers’ service accounts rather than your customer count. This is also the pattern where tenants: becomes an inventory. Nodes can instead be created through the management API into the runtime plane, which keeps the YAML small; a runtime-created node has no effect on the live gate until the next governance reload.

How it fails. Depth runs out. Four rungs exist and only three are nodes, so a reseller whose customers have customers has nowhere to put the fourth, and no config adds a level. Two ways out. Flatten, giving each end customer its own org and moving the reseller-wide roll-up to cost analytics rather than enforcement. Or encode one rung down: put the end customer at team: and their department at user:, and take per-credential attribution from the key, since every row a key served carries serving_key.kid whether or not a node exists for it.

Before you boot any of them

All four blocks on this page were loaded by the shipped binary. tokentriage validate returns OK: configuration is valid for each, and tokentriage govern preview compiles each against the same limit-type registries the proxy arms with.

Run both, because validate does not compile limits. A clean validate is not evidence that the block boots: the per-type guards live in the compiler, and a config that trips one stops the process at startup. govern preview runs that compile before it reads your ledger, so a compile error surfaces even against an empty log. Two lines carry that, and the run below is cut back to them with # … marking what was dropped; turn governance on prints a report in full.

$ tokentriage govern preview -c per-feature.yaml --log ./empty.jsonl
# …
  rows replayed: 0
# …
  [budget_usd] batch_daily_budget @ org:platform/team:batch  burn $0.000000 / limit $300.000000  (frac 0.0000, rows 0, unpriced_excluded 0)
# …

That ran against the per-feature block on an empty decision log created for the check, so every figure is zero and the node names are the point. Each line is one scope the recomputation folded, and a pinned limit gets its line either way. A template is the weaker case: it resolves to per-node lines only once the log carries rows underneath it, and until then a rate template prints @ level:team while a spend template is absent from the EXACT block. Every limit in this set is pinned, which is why each line names a node.

Choices that apply to all four

Where to put the budget. Higher is a firmer guarantee: an org cap sits in every descendant’s chain, so it binds whatever the teams below were individually allowed. Lower is a better blast radius: a team that exhausts its own cap stops only itself. Most deployments want both, which is what the per-team pattern does. The price is that a caller can be stopped by a cap it was not watching, and GET /tt/v1/me is where it can see the shallow ones.

Template or pinned. A template fans out to every node at its level and picks up new nodes as you declare them. A pin is one instance at one node. Use a template for the cap that is the same for everyone and a pin for the exception, remembering that both bind when both are selected.

override: belongs to templates. Moving it onto the pinned northwind_rpm stops validate. The same line lands twice, because the error a command returns is printed once by the CLI framework and once by the binary’s top level:

$ tokentriage validate --config per-customer.yaml
Error: configuration is invalid: config per-customer.yaml: governance.limits[1].override: override is meaningful only on a template (level) limit
Error: configuration is invalid: config per-customer.yaml: governance.limits[1].override: override is meaningful only on a template (level) limit

allow defaults to false, so a template with no override: block accepts no per-node override. The ceiling is the other half of that key, and omitting it is the mistake worth naming. override: { allow: true } with no max_amount or max_amount_usd declares no ceiling, and any positive amount is then accepted for any node. An override also reaches only budget_usd, quota_tokens and quota_requests; on rpm, tpm and concurrency it is turned down with a reason, which is why every rate cap here is pinned per node.

Turning require_key on. It defaults to off. Set to required, it answers anonymous traffic with a 401 from the moment governance arms, and traffic that predates your keys carries no ttv_, so all of it stops at once. Roll it in two steps: set optional, which arms governance and still admits anonymous requests, mint and distribute keys, then flip to required once the anonymous share of your rows reaches zero. Both edits want a restart, unlike governance.limits and governance.tenants, which reload hot.

The two warnings validate prints when you flip require_key

One warning stands while governance.keys.tls_terminated_upstream is unset, because bearer keys would otherwise cross cleartext HTTP. That field is an attestation you make, not a probe the binary runs, so set it once TLS terminates in front of the data plane. The other fires when required is paired with an empty tenants:, and says you are about to answer all traffic with a 401. Each is printed twice, because the config is validated once as it loads and again against the live registries, and both warning lists reach the terminal.

What you cannot express. Per-tenant provider credentials are not a shipped config surface. A tenancy node carries no credential field, so every tenant’s traffic leaves on your credentials, read per upstream from the environment variable named in api_key_env. What answers “whose request was that” is decision.governance.serving_key on the row, carrying the kid, the node, and rotated_from for a key that came out of a rotation. Virtual keys has the rest.

What this page does not cover

The shapes, not the machinery. How a chain becomes a set of instances, and what an override can and cannot raise, is the tenancy hierarchy. What each type counts and where it can overshoot is the six limit types. Who may write an override inside a subtree is delegated self-serve. Turning these rows into spend per tenant is cost analytics.