Search

↑↓ to move↵ to openesc to closetop 6 documentation results

Book a demoTry TokenTriage

In development · Explore the product preview. Join the launch list or book a walkthrough.

Browse docs

Toxicity, topical, code & media detectors

The toxicity wordlist, topical scaffold, code tripwire, and media presence gate — what each catches and what it does not.

ReferenceLast updated

On this page

Four embedded registrations, one per category. Each is scoped narrowly and says so; three of the four are measured against a corpus their own author wrote, which is exactly why none of them mints a fixture basis. The secrets and injection detectors are on the previous page; the inventory has the summary table.

toxicity.wordlist — a cheap pre-filter, weak on purpose

Covers. Literal, lower-cased, word-boundary matches of a short embedded English list: 17 single tokens and 16 phrases across TOXICITY_THREAT, TOXICITY_SELF_HARM, TOXICITY_INSULT, TOXICITY_PROFANITY. Threat and self-harm are phrases, not single tokens.

Not covered.

  • Identity-directed hate speech — racial, ethnic, religious, disability and orientation slurs. The list contains none, deliberately: a slur table committed to a public repository is a permanent, liftable harm surface, and such tables age asymmetrically. This is the single most important thing a moderation layer is asked to catch, and this detector does not catch it. The registered sidecar.safety classifier is not an available stock remedy: its activation is rejected pending runtime/model qualification.
  • A wordlist is not moderation. Everything implicit, sarcastic, coded, non-English, or in another register — polite, bureaucratic and legalistic abuse — is invisible to it.
  • It false-positives on exactly the text a reasonable person would want allowed. Measured FP rate 0.1818 (4 of 22 negatives) — a quotable denominator, and the one number here worth quoting because it is the bad one. The four: a quoted insult, a suicide-prevention page, self-directed speech, and documentation about toxicity detection. It has no notion of speaker, target, quotation or negation.
  • Trivial obfuscation defeats it on the first attempt. k1ll yourself, spaced letters, a zero-width character, a homoglyph, base64 — all pass. It runs none of the pii.* de-obfuscation passes.
  • Its numbers are self-certifying. The frozen corpus has no toxicity cases, so the only measurement is a corpus the shipping card wrote itself. That is why the basis stays reported.
  • The score is a per-class constant (threat/self-harm 0.90, insult 0.60, profanity 0.30) — an ordering over classes, not a probability. threshold: 0.8 selects threat and self-harm; 0.5 adds insult; 0.2 adds profanity. That is the whole of the tuning available.
  • Input and output only. It implements no stream matcher and does not declare the phase.

topical.keywords — a scaffold, not topic control

Covers. Four professional-advice subjects by request-or-assertion phrasing rather than bare subject noun: MEDICAL_ADVICE, LEGAL_ADVICE, FINANCIAL_ADVICE, ELECTORAL_POLITICS — about 40 patterns, with spans, so a binding may redact rather than only block.

Not covered. The shipped lexicon is a scaffold, not a coverage claim — “off-topic” is an operator’s definition and this build offers no config surface for a deployment’s own topic list. Its recall and precision are measured against a 34-case fixture the shipping card wrote (the frozen corpus contains no topical class; what it can measure is a false-positive rate, asserted at zero across all 259 cases). Any subject outside the four, any phrasing the patterns do not contain, paraphrase, indirect requests, other languages, multi-turn subjects. Topic allow-listing (“answer only about banking”) is out of reach entirely — this is a deny matcher, and an allow-list guardrail needs a classifier. Split-token and base64 evasion are not resisted, because topical is not an adversarial category and paying 3× per call to resist an out-of-model threat has no buyer. Every topic carries the same raw score (0.60), so severity_bands cannot separate them.

code.patterns — a tripwire, not a code analyzer

Covers. Literal matches of 12 dangerous-construct patterns in model output, across 7 classes: remote-fetch-piped-to-shell, recursive force-delete rooted at / or an unguarded variable, dynamic evaluation, shell process spawning, unsafe deserialization, unescaped DOM sinks, and chmod 777. One RE2 pass per field — linear, no backtracking, so a large document cannot be turned into a budget attack.

Also not covered. Semantic equivalence and composition — subprocess.Popen from a list, an importlib round-trip, or three individually innocuous lines that together download, chmod and execute. Obfuscation defeats it on the first attempt: base64, string concatenation, hex escapes, a variable holding the command. It runs no de-obfuscation passes at all. Constructs split across lines (os.system(\n cmd\n)) — every pattern requires call syntax on one line, because a bare keyword fires on every discussion of evaluation. Safe spellings are deliberately exempt (ast.literal_eval(, json.loads(, any dotted x.eval(), so a genuinely dangerous method named .eval( is also a miss. And it has no notion of negation, quotation or fencing: a refusal that says never write curl … | sh matches exactly as the advice to run it does.

Its measured figures: recall 13 of 13 and precision 13 of 13 findings — both denominators below the floor, so both are tallies. Its false-positive rate is 0.0000 (0 of 22 negatives), whose denominator does clear the bar — and it is the least informative number on the page, because the hard negatives are the safe spellings the same author already knew the patterns exempt. It says the exemptions work. It says nothing about what real model output would produce.

Output phase only. A stream binding naming it is refused at boot, not degraded.

media.inventory — a presence gate

Covers. The presence of non-text parts a dialect’s media seam recognizes, one finding per part under a closed entity vocabulary (MEDIA_IMAGE, MEDIA_AUDIO, MEDIA_VIDEO, MEDIA_FILE, MEDIA_UNFETCHED_REFERENCE, MEDIA_INLINE_OVERSIZE, MEDIA_CENSUS_PARTIAL), plus each part’s declared kind and mime, its source kind (inline vs. unfetched reference), and a size gate over inline parts defaulting to 5,242,880 bytes. Basis exact, point 1.0: that a part of a declared kind exists at a known path is a deterministic read of the request bytes.

Also not covered.

  • The mime type is client-declared and may be a lie, at zero cost to the liar. Nothing is sniffed from the bytes, so any mime-shaped policy is advisory — it gates what the client said.

  • URL parts are never fetched, so their size and true type are unknown (reported as size 0 and an empty mime, meaning unknown, never empty). The size gate cannot see a remote image at all. Gate on MEDIA_UNFETCHED_REFERENCE instead.

  • An empty inventory is not a statement that there is no media — it means “no part I recognize”. An unreadable body, an unrecognized content shape and a genuinely text-only request are indistinguishable to this detector alone. The outer content extractor separately records unsupported or partial coverage and applies failure posture.

  • On a dialect without the required media seam, such as the current bedrock and gemini paths, uninspectable_body discloses the gap. Monitor/open can forward; acknowledged enforcing/closed refuses the affected request before provider dispatch.

  • The score is always 1.0, so thresholds and severity bands cannot express varying certainty. Findings are document-level. Stock activation rejects span-dependent mutation bindings for this detector; supported actions are observe, warn and block.

  • The byte count is derived from base64 length, not measured — exact for well-formed base64, over-counting by ~1.4% at 76-column wrapping and by up to 3× for a percent-encoded data URI.

  • More than 256 parts in one request stops the census, disclosed as MEDIA_CENSUS_PARTIAL. Parts outside a message’s content array — a system prompt’s blocks, tool definitions, top-level provider extensions — are not walked. Anthropic tool_result content is walked one level deep.

  • Any efficacy measurement at all. The frozen corpus has no multimodal cases and this detector has no corpus of its own. exact here is a statement about determinism, not measured precision.

  • Input phase only. output or stream is refused at boot.