llm-security-commons/README.md

# llm-security-commons

Runtime-neutral core for LLM and agent security detection: detector data, normative contracts and a conformance corpus that several runtimes can share.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Detection logic gets reimplemented every time it crosses a language boundary, and the
copies drift: the Node scanner flags a zero-width carrier the Python guard misses, and
nobody notices until an incident. This repository holds the part that should never have
been copied — the pattern tables, the code-point carriers, the calibration thresholds, the
finding contract, and a fixture corpus with expected verdicts — so that two independent
implementations can be held to the same answer on the same input.

It is for anyone building or maintaining a detector for prompt injection, secret egress,
unicode-carrier smuggling or active content in untrusted text, on any runtime.

**It holds no runnable code.** Data, specifications and fixtures only.

## Table of Contents

- [Install](#install)
- [Requirements](#requirements)
- [What it does](#what-it-does)
- [Non-goals](#non-goals)
- [Known limitations](#known-limitations)
- [Contributing](#contributing)
- [Reporting a wrong entry](#reporting-a-wrong-entry)
- [Changelog](#changelog)
- [License](#license)

## Install

Nothing to install — this repository is **vendored into consumers**, not installed.

As a `git subtree` (recommended: history is preserved and upgrades are a single command):

```bash
git subtree add --prefix vendor/commons \
  https://git.fromaitochitta.com/open/llm-security-commons.git v0.6.0 --squash

# later, to move to a newer tag
git subtree pull --prefix vendor/commons \
  https://git.fromaitochitta.com/open/llm-security-commons.git <newer-tag> --squash
```

Or pin a tag and copy — `fork-and-own` is an explicitly supported path:

```bash
git clone --depth 1 --branch v0.6.0 \
  https://git.fromaitochitta.com/open/llm-security-commons.git
```

Always vendor **a tag**, never `main`. The tag is what a conformance result can be
attributed to.

## Requirements

A JSON parser and the ability to read a text file. That is the entire dependency surface,
and keeping it that small is the point.

## What it does

| Path | Contents |
| --- | --- |
| [`lexicon/injection-lexicon.json`](lexicon/injection-lexicon.json) | Prompt-injection pattern lexicon: 83 patterns in four **severity** families (`critical`, `high`, `medium`, `hybrid`), each with a stable `id` and per-runtime aliases. The thematic class (`override:`, `evasion:`, `hitl-trap:`, …) is the id prefix, not the family. |
| [`codepoints/carriers.json`](codepoints/carriers.json) | Invisible and deceptive carriers: zero-width characters, BIDI controls, Unicode Tag block ranges, and the homoglyph map. Carries three commons-owned ids (`carrier:zero-width`, `carrier:bidi-override`, `carrier:unicode-tag`) for the carriers observable on an input surface — the only id space here that was *named* rather than adopted verbatim from a runtime, and the file records why. |
| [`signatures/secret-egress.json`](signatures/secret-egress.json) | Credential and token shapes that must never leave a machine, in a portable regex dialect. |
| [`signatures/malware-signatures.json`](signatures/malware-signatures.json) | Known-bad **identity** for the malicious-code class (`SIG`): seven tight signatures over four families — PHP webshells, reverse shells, cryptominers, offensive tooling. Seven signatures are not malware coverage, and the file says so. |
| [`signatures/active-content.json`](signatures/active-content.json) | Active content that renders or fetches on its own — Markdown images, links, reference definitions and autolinks, `data:` URIs, active HTML. The EchoLeak class. Raw HTML carries **two** classes: `active:raw-html` for what a renderer acts on unattended, `active:raw-html-link` for anchors, which need a human. One pattern, one scan, two buckets — the file spells that out, because giving the second class its own pass would double-count. |
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself. |
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
| [`schema/conformance-declaration.schema.json`](schema/conformance-declaration.schema.json) | **Normative.** The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs. |
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, what `exact-within-scope` requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as `not-applicable` rather than as a failure. |
| [`conformance/`](conformance/) | 94 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Seven cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. Three cover the input-side carriers, added in v0.5.0. See [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.6.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions. Most of it is still open, and the two rows that closed in v0.4.0 closed because the runtime that owns the value decided, not because this document found them wrong. |

Every JSON file carries a top-level `version`. Every normative specification carries a
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
part of the contract, but the file does not exist yet — they are not links, and nothing in
v0.6.0 depends on them.

Each data file records its own provenance and, in `verified`, how strongly it is backed.
`calibration/calibration.json` is currently the one file that says `false`: it was
transcribed from a prose summary rather than diffed against a running implementation.

### How a consumer proves it conforms

Run every `conformance/<case>/input.txt` through your detector and compare the finding ids
to `expected.json` — exactly, but only within the data files the case names in `scope`.
[`spec/conformance-corpus.md`](spec/conformance-corpus.md) is the normative reading;
the short version is that a runtime must raise every listed finding and no other finding
*from the same table*, and that what it does with tables outside the case's scope is not
compared.

Disagreement means your runtime is wrong, or the fixture is — and the fixture only changes
in its own commit, with the reason written down.

There is **no CI in this organisation** and nothing runs that comparison automatically. It
runs in each consumer's own test suite, against a pinned tag.

The corpus covers three tables, and they do not carry equal weight — treating them as one
number would misreport all three:

- `lexicon/injection-lexicon.json` — 84 cases over 83 patterns. Both seeding runtimes
  implement it and both ratified its id space. One pattern carries a second, variant case:
  the rule for when that is legal is normative in [§6](spec/conformance-corpus.md), and
  `case_id_derivation.variant_suffix` in the manifest carries the measurement behind it.
- `signatures/active-content.json` — 7 cases, one per published id, the seventh added in
  v0.6.0 when the seed runtime split raw HTML into two carrier classes. One runtime
  implements it. For a runtime that
  does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
  runtime declares which commons data files it implements, and a case scoped outside that
  set was never addressed to it. See [§1.1](spec/conformance-corpus.md) — and note that
  `not-applicable` says the corpus did not ask, never that the runtime is blind.
- `codepoints/carriers.json` — 3 cases, added in v0.5.0. **Both** runtimes implement these
  tables; only one has published a label for the finding. So these three are `not-applicable`
  for the other *today*, and this is the one place in the corpus where that verdict records a
  missing **name** rather than a missing capability. It lapses the moment that runtime names
  its label and the alias is added.

One case remains unshipped, for the secret-egress table, and it is not blocked on effort. The
reasons are three, they were measured, and they are independent — none of them dissolves under
anything this repository can run alone. **(1) There is no id space on the commons side.** The
seed this table was ported from carries a name and a pattern per entry and nothing else, so its
entries are keyed by human-readable name while the other runtime emits `egress:<id>` labels —
and a fixture names labels. This is the hard blocker, and the only one of the three that an
answer can resolve; the answer belongs to the runtimes that own the seeds, not to a name coined
here. **(2) Match semantics disagree**, and an id space would not close it: this table declares
first-match-wins with `ordering.normative: true`, the other runtime reports every match, and one
witness — an `Authorization` header holding a three-part JWT — produces one label here and two
there. That difference is exactly what an `expected.json` encodes. **(3) Membership diverges in
both directions, and the divergence is inherited rather than introduced.** The two sides hold 19
entries and 25, but they are ports of two *different* source tables in one source repository, so
re-measuring either port cannot close it. `conformance/manifest.json` records all three under
`scope_planned.blockers`, and the method behind every number is in
[the divergence measurement](docs/secret-egress-divergence.md).

The carrier blocker closed in v0.5.0 and is kept, with its retired text, under
`scope_planned.blockers_resolved` — including the correction one runtime volunteered against
a general rule this repository had written down and should not have.

## Non-goals

- **Not a scanner.** There is no engine here, and there will not be one. If you are looking
  for something to run, you want a consumer — `llm-security` for Claude Code.
- **Not a framework or a library.** No package manifest, no dependencies, no build.
- **Not a general-purpose Unicode or regex toolkit.** The tables cover what the detection
  classes need, not the standard.
- **Not a vulnerability feed.** No CVEs, no advisories, nothing time-sensitive. Everything
  here is offline and deterministic.
- **Not a policy engine.** `calibration.json` publishes the thresholds; deciding what to do
  when one is crossed belongs to the consumer.
- **Not the place to fix a consumer's behaviour.** Data extracted from an implementation is
  kept behaviour-identical on purpose. A disagreement is reported to that implementation
  and decided there, where it is tested.

## Known limitations

- **Coverage is the union of what the seed implementations detected**, not of what exists.
  A class absent from the tables above has not been shown to work anywhere.
- **The corpus is narrower than the data.** `conformance/` constrains two of the seven data
  files. The other five are published, provenance-checked and unfixtured: a runtime can
  pass every case and still read `calibration.json` wrongly. Passing the corpus is evidence
  about the injection lexicon and about active content, and about nothing else.
- **A pass count is unreadable without the declared table set.** A runtime implementing one
  table and a runtime implementing four can print the same number. `not-applicable` cases
  must be reported, not dropped from the denominator — `76/83` and `76 passed, 6
  not-applicable` describe different runtimes.
- **The seven active-content cases prove less than the 83.** Their payloads come from the only
  runtime that implements the table, so no second implementation's agreement could be
  measured. They pin one runtime's behaviour as a contract a future implementer can be held
  to; they are not cross-runtime agreement.
- **Regex portability is a real risk.** Pattern data is written for a common subset, but
  engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose
  engine rejects a pattern must report it rather than silently skip it — a skipped pattern
  is an invisible false negative.
- **Fixtures prove agreement, not correctness.** Two runtimes passing the same corpus agree
  with each other and with the fixture author. A wrong `expected.json` makes both wrong
  identically.
- **The homoglyph map is finite.** Confusable coverage is a long tail; absence from the map
  is not evidence a character is safe.

## Contributing

[CONVENTIONS.md](CONVENTIONS.md) is the whole rule set a change here is held to: the charter
(nothing runs, and why that is load-bearing rather than fussy), the file conventions, when a
detection value is allowed to move, how the two version numbers work, and the four offline
checks that stand in for the CI this organisation does not have.

It also answers the question the forge surface raises on its own: **pull requests are switched
off, deliberately.** This repository is vendored into independent runtimes that pin a tag, so a
change to detection data changes what they *find* — that has to be coordinated with each
consumer before it exists, which a merge button cannot do. Fork-and-own is the supported path;
a wrong entry is reported privately.

## Reporting a wrong entry

A wrong code point or a mis-escaped regex here is a silent false negative in every runtime
that reads it, so it is a security report even though nothing runs. Send it privately — see
[SECURITY.md](SECURITY.md), which also explains why a confirmed defect in extracted data is
decided in the runtime it came from before it is changed here.

## Changelog

See [CHANGELOG.md](CHANGELOG.md).

## License

MIT — see [LICENSE](LICENSE). Fork-and-own is an intended use, not a tolerated one.