Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Reversible de-identification

The tokenize and re-identify round trip — what the vault is, which verbs are reversible, and the failure modes that matter.

How-to guideLast updated

On this page

redact and mask throw the value away: the model answers about [REDACTED:EMAIL] and so does your client. Reversible de-identification keeps the answer usable. The input gate substitutes a placeholder, the model reasons about that, and the output gate puts the real value back before your client sees the response.

The tokenize and re-identify round trip, and where the real value existsThis diagram shows successful tokenization on supported input and restoration on buffered output. A request-local vault pairs minted tokens with original values. When the provider returns a token minted by this same request, the buffered reverse leg can restore it before delivery. Streaming restoration is unavailable. The vault mapping is not persisted or shared across requests; separate capture and other sinks retain their own policies. Failed inspection follows the configured mode and failure posture; the diagram is not a universal egress proof.placeholder onlyrequest-local mapping boundarysuccessful input + buffered outputclientyour appinput gatetokenizemints placeholder+ vault entryoutput gatereidentifyengine-internal,never configuredthis request's vault[TTsbjk:EMAIL_1] → [email protected]upstreamsees the placeholderand nothing elserealwritereadbuffered reverse leg — request-owned tokens onlyThe vault is minted on first need and dies with the request.Vault mappings are not persisted; capture and other sinks have separate policies.

The round trip

Configure the forward verb only. Measured on a live proxy with action: tokenize bound to pii.email on the input phase:

# what the client sent
"my email is [email protected]"

# what the upstream actually received
"my email is [TTsbjk:EMAIL_1]"

# what the client got back
"Sure — I have noted my email is [email protected] and will follow up there."

Both legs are on the decision row — the forward verb with its detector and finding, the reverse leg without one, because no detector runs on it:

{"binding":"pii_tokenize_in","phase":"input","detector":"pii.email","action":"tokenize",
 "findings":[{"entity":"EMAIL","spans":[{"field":"messages[0].content","off":12,"len":15}],
 "raw_score":0.85,"confidence":{"basis":"reported","point":0.85}}],"mode":"enforce"}
{"binding":"pii_tokenize_in","phase":"output","detector":"","action":"reidentify","mode":"enforce"}

tokenize or pseudonymize

Both are Mutates: true, Reversible: true and both round-trip identically. They differ only in what the model sees, and the same request under each verb shows why you would pick one:

Verb What the upstream receives Pick it when
tokenize [TTsbjk:EMAIL_1] the model must not mistake the surrogate for real data
pseudonymize [email protected] a realistic-looking value reads better to the model

Within one request the same original always maps to the same placeholder — that is entity consistency, and it matters: a prompt mentioning one person twice must not read as two people. Measured:

"ping [email protected] and [email protected] again"
   ⇒ "ping [TTj5xz:EMAIL_1] and [TTj5xz:EMAIL_1] again"

Which verbs are reversible, and why that bit is trustworthy

Reversible is not authored per fire. It is derived from one canonical action table, which is also what validates a verb in the first place — so the table cannot drift from the disclosure:

Verb Reversible
tokenize · pseudonymize · reidentify yes
redact · mask · rewrite · block · warn · observe · reask no

This table describes the canonical verb taxonomy, not gateway availability: stock activation rejects rewrite and reask. The lower-level Go re-ask callback exists for custom embeddings.

redact and mask are irreversible and no configuration changes that. If you need the answer to come back intact, the choice is made at the forward verb.

guardrails explain prints the resolved ladder with both bits, so you can check before arming:

  LADDER — the verb each severity resolves to (fall-down included)
    SEVERITY  ACTION    REVERSIBLE  THIS BUILD CAN DO IT
    low       tokenize  yes         yes

The vault is per-request

The mapping lives on one request’s state and dies with it. It is minted on first reversible need, never before, and there is no package-level vault, no registry of vaults, and no constructor that takes one from anywhere but a request. The anti-pattern this avoids by construction is a process-wide vault shared across clients — a cross-tenant re-identification pool.

Each phase mints its own. The output phase used to be seeded with the input phase’s vault, which made the reverse leg undo the output binding’s own work on the same request.

Like every other credential in the product, the config names the variable and never carries the value — the same discipline described in Security & secrets.

guardrails:
  vault:
    key_env: TT_VAULT_KEY      # the NAME of the variable, never the key

Two acknowledgement keys, and what they actually do

  • raw_egress_ack is retained compatibility syntax. It does not authorize supported raw egress in this build: stock activation rejects every external NeedsRaw binding even with the ack. Destination-specific external consent remains a separate requirement for supported external calls.
  • fuzzy_reid is accepted, compiled onto the binding, and read by nothing.

The failure modes

The reverse leg mutates, so it discloses. What matters is what it does when it cannot complete — and in every case the answer is a safe failure rather than a fabricated clean round trip.

A placeholder the model mangled is not restored. If the model drops a bracket or splits the placeholder across a word boundary, the exact match fails and the mangled marker survives to the client verbatim. That is the safe failure: an unreadable marker, never someone else’s data. The round-trip corpus measures how often it happens rather than papering over it.

A cached response is never re-identified. A cache hit’s bytes are some earlier request’s answer; nothing in them was tokenized by this request, so there is nothing of ours to restore — and restoring anyway is not a no-op, it is a cross-request leak. This was measured, not theorized: under an early process-wide nonce, a stored body reading the answer about [TT0000:DRILL_TOKEN_1] is 42 was served to a later client as the answer about that client’s own secret is 42, with the row disclosing a clean round trip — fabricated provenance on top of another request’s cached answer.

On the streaming lane the reverse leg is not implemented. The action table licenses reidentify at the stream phase, but the restoring code is the buffered output gate’s. A cross-frame placeholder needs the hold-back window, and that is a separate seam — see Phases & streaming.