llm-security-commons/README.md

llm-security-commons

Runtime-neutral core for LLM and agent security detection: detector data, normative contracts and a conformance corpus that several runtimes can share.

Detection logic gets reimplemented every time it crosses a language boundary, and the copies drift: the Node scanner flags a zero-width carrier the Python guard misses, and nobody notices until an incident. This repository holds the part that should never have been copied — the pattern tables, the code-point carriers, the calibration thresholds, the finding contract, and a fixture corpus with expected verdicts — so that two independent implementations can be held to the same answer on the same input.

It is for anyone building or maintaining a detector for prompt injection, secret egress, unicode-carrier smuggling or active content in untrusted text, on any runtime.

It holds no runnable code. Data, specifications and fixtures only.

Install

Nothing to install — this repository is vendored into consumers, not installed.

As a git subtree (recommended: history is preserved and upgrades are a single command):

git subtree add --prefix vendor/commons \
  https://git.fromaitochitta.com/open/llm-security-commons.git v0.4.1 --squash

# later, to move to a newer tag
git subtree pull --prefix vendor/commons \
  https://git.fromaitochitta.com/open/llm-security-commons.git <newer-tag> --squash

Or pin a tag and copy — fork-and-own is an explicitly supported path:

git clone --depth 1 --branch v0.4.1 \
  https://git.fromaitochitta.com/open/llm-security-commons.git

Always vendor a tag, never main. The tag is what a conformance result can be attributed to.

Requirements

A JSON parser and the ability to read a text file. That is the entire dependency surface, and keeping it that small is the point.

What it does

Path	Contents
lexicon/injection-lexicon.json	Prompt-injection pattern lexicon: 83 patterns in four severity families (critical, high, medium, hybrid), each with a stable id and per-runtime aliases. The thematic class (override:, evasion:, hitl-trap:, …) is the id prefix, not the family.
codepoints/carriers.json	Invisible and deceptive carriers: zero-width characters, BIDI controls, Unicode Tag block ranges, and the homoglyph map.
signatures/secret-egress.json	Credential and token shapes that must never leave a machine, in a portable regex dialect.
signatures/malware-signatures.json	Known-bad identity for the malicious-code class (SIG): seven tight signatures over four families — PHP webshells, reverse shells, cryptominers, offensive tooling. Seven signatures are not malware coverage, and the file says so.
signatures/active-content.json	Active content that renders or fetches on its own — Markdown images, links, reference definitions and autolinks, data: URIs, active HTML. The EchoLeak class.
calibration/calibration.json	The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself.
mapping/owasp-map.json	Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP).
schema/finding.schema.json	Normative. The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as not applicable, with the reason.
schema/conformance-declaration.schema.json	Normative. The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs.
spec/conformance-corpus.md	Normative. How to read the corpus: what a case is, why input.txt is bytes rather than text, what exact-within-scope requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as not-applicable rather than as a failure.
conformance/	90 cases. One directory per case: input.txt in, expected.json out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Six cover active content and are measured against the one runtime that implements that table — not-applicable for the other, not failing. See conformance/manifest.json.
spec/decode-pipeline.md	Planned, still not shipped as of v0.4.1. The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one.
docs/extraction-plan.md	Informative: where each file was seeded from, and what v0.1.0 promised.
docs/lexicon-port-divergence.md	Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions. Most of it is still open, and the two rows that closed in v0.4.0 closed because the runtime that owns the value decided, not because this document found them wrong.

Every JSON file carries a top-level version. Every normative specification carries a Status: normative marker. Rows marked Planned are named here because the layout is part of the contract, but the file does not exist yet — they are not links, and nothing in v0.4.1 depends on them.

Each data file records its own provenance and, in verified, how strongly it is backed. calibration/calibration.json is currently the one file that says false: it was transcribed from a prose summary rather than diffed against a running implementation.

How a consumer proves it conforms

Run every conformance/<case>/input.txt through your detector and compare the finding ids to expected.json — exactly, but only within the data files the case names in scope. spec/conformance-corpus.md is the normative reading; the short version is that a runtime must raise every listed finding and no other finding from the same table, and that what it does with tables outside the case's scope is not compared.

Disagreement means your runtime is wrong, or the fixture is — and the fixture only changes in its own commit, with the reason written down.

There is no CI in this organisation and nothing runs that comparison automatically. It runs in each consumer's own test suite, against a pinned tag.

The corpus covers two tables, and they do not carry equal weight — treating them as one number would misreport both:

• lexicon/injection-lexicon.json — 84 cases over 83 patterns. Both seeding runtimes implement it and both ratified its id space. One pattern carries a second, variant case; see case_id_derivation.variant_suffix in the manifest.
• signatures/active-content.json — 6 cases. One runtime implements it. For a runtime that does not, these cases are not-applicable, a third verdict beside pass and fail: a runtime declares which commons data files it implements, and a case scoped outside that set was never addressed to it. See §1.1 — and note that not-applicable says the corpus did not ask, never that the runtime is blind.

Four cases remain unshipped, for the carrier and secret-egress tables, and neither is blocked on effort. Carriers has no adoptable id space: one runtime labels the same carrier differently depending on pipeline stage, which would make a case's verdict depend on the entry point it was measured through. Secret egress is not an id question at all — the two runtimes carry different tables, 19 entries against 25, cut at different granularities. conformance/manifest.json records both blockers under scope_planned.blockers, measured, so the gap is visible rather than inferred.

Non-goals

• Not a scanner. There is no engine here, and there will not be one. If you are looking for something to run, you want a consumer — llm-security for Claude Code.
• Not a framework or a library. No package manifest, no dependencies, no build.
• Not a general-purpose Unicode or regex toolkit. The tables cover what the detection classes need, not the standard.
• Not a vulnerability feed. No CVEs, no advisories, nothing time-sensitive. Everything here is offline and deterministic.
• Not a policy engine. calibration.json publishes the thresholds; deciding what to do when one is crossed belongs to the consumer.
• Not the place to fix a consumer's behaviour. Data extracted from an implementation is kept behaviour-identical on purpose. A disagreement is reported to that implementation and decided there, where it is tested.

Known limitations

• Coverage is the union of what the seed implementations detected, not of what exists. A class absent from the tables above has not been shown to work anywhere.
• The corpus is narrower than the data. conformance/ constrains two of the seven data files. The other five are published, provenance-checked and unfixtured: a runtime can pass every case and still read calibration.json wrongly. Passing the corpus is evidence about the injection lexicon and about active content, and about nothing else.
• A pass count is unreadable without the declared table set. A runtime implementing one table and a runtime implementing four can print the same number. not-applicable cases must be reported, not dropped from the denominator — 76/83 and 76 passed, 6 not-applicable describe different runtimes.
• The six active-content cases prove less than the 83. Their payloads come from the only runtime that implements the table, so no second implementation's agreement could be measured. They pin one runtime's behaviour as a contract a future implementer can be held to; they are not cross-runtime agreement.
• Regex portability is a real risk. Pattern data is written for a common subset, but engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose engine rejects a pattern must report it rather than silently skip it — a skipped pattern is an invisible false negative.
• Fixtures prove agreement, not correctness. Two runtimes passing the same corpus agree with each other and with the fixture author. A wrong expected.json makes both wrong identically.
• The homoglyph map is finite. Confusable coverage is a long tail; absence from the map is not evidence a character is safe.

Reporting a wrong entry

A wrong code point or a mis-escaped regex here is a silent false negative in every runtime that reads it, so it is a security report even though nothing runs. Send it privately — see SECURITY.md, which also explains why a confirmed defect in extracted data is decided in the runtime it came from before it is changed here.

Changelog

See CHANGELOG.md.

License

MIT — see LICENSE. Fork-and-own is an intended use, not a tolerated one.