open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

Squashed 'scanners/commons/' changes from 0ffee85..532d70d

Browse source 
532d70d release(0.2.0): the contract gained a MUST, so the tag has to move
946f51d fix(active-content,conformance): cite line numbers per commit — they do not resolve at the pin
bdcb1f1 feat(conformance): ship the six active-content cases; the id space already existed
807c0d4 feat(spec): add not-applicable, so a single-runtime table stops reading as 7 defects
a1578e6 fix(conformance): record the guard's internal-surface position on _LEX_PAYLOADS
4d351d2 fix(mapping): state that three of four OWASP maps have no production consumer
f082a91 fix(lexicon,docs): retract the claim that the guard's port cites severity.mjs

git-subtree-dir: scanners/commons
git-subtree-split: 532d70d5ed2f9b23a8efad760ef490356cf52ada
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:02:33 +02:00
commit 9b95fc607e
20 changed files with 533 additions and 37 deletions
Download patch file Download diff file Expand all files Collapse all files

50
README.md
View file

@ -24,17 +24,17 @@ As a `git subtree` (recommended: history is preserved and upgrades are a single
```bash
git subtree add --prefix vendor/commons \
https://git.fromaitochitta.com/open/llm-security-commons.git v0.1.0 --squash
https://git.fromaitochitta.com/open/llm-security-commons.git v0.2.0 --squash
# later, to move to a newer tag
git subtree pull --prefix vendor/commons \
https://git.fromaitochitta.com/open/llm-security-commons.git v0.2.0 --squash
https://git.fromaitochitta.com/open/llm-security-commons.git <newer-tag> --squash
```
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
```bash
git clone --depth 1 --branch v0.1.0 \
git clone --depth 1 --branch v0.2.0 \
https://git.fromaitochitta.com/open/llm-security-commons.git
```
@ -59,15 +59,15 @@ and keeping it that small is the point.
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
| [`conformance/`](conformance/) | 83 cases, one per injection-lexicon pattern. One directory per case: `input.txt` in, `expected.json` out. Ground truth. Both seeding runtimes were measured producing the same verdict on all 83 — see [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, not in v0.1.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`conformance/`](conformance/) | 89 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 83 cover the injection lexicon, one per pattern, and both seeding runtimes were measured producing the same verdict on all 83. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.2.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
Every JSON file carries a top-level `version`. Every normative specification carries a
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
part of the contract, but the file does not exist yet — they are not links, and nothing in
v0.1.0 depends on them.
v0.2.0 depends on them.
Each data file records its own provenance and, in `verified`, how strongly it is backed.
`calibration/calibration.json` is currently the one file that says `false`: it was
@ -88,12 +88,24 @@ in its own commit, with the reason written down.
There is **no CI in this organisation** and nothing runs that comparison automatically. It
runs in each consumer's own test suite, against a pinned tag.
The v0.1.0 corpus covers `lexicon/injection-lexicon.json`, the one table whose finding ids
are ratified by both seeding runtimes. Eleven further cases exist in a seed suite for the
carrier, active-content and secret-egress tables and are **not** shipped: naming a finding
in those tables would mean minting a cross-runtime id space no runtime has agreed to.
`conformance/manifest.json` names them under `scope_planned` so the gap is visible rather
than inferred.
The corpus covers two tables, and they do not carry equal weight — treating them as one
number would misreport both:
- `lexicon/injection-lexicon.json` — 83 cases. Both seeding runtimes implement it and both
ratified its id space.
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
runtime declares which commons data files it implements, and a case scoped outside that
set was never addressed to it. See [§1.1](spec/conformance-corpus.md) — and note that
`not-applicable` says the corpus did not ask, never that the runtime is blind.
Four cases remain unshipped, for the carrier and secret-egress tables, and neither is
blocked on effort. Carriers has no adoptable id space: one runtime labels the same carrier
differently depending on pipeline stage, which would make a case's verdict depend on the
entry point it was measured through. Secret egress is not an id question at all — the two
runtimes carry *different tables*, 18 entries against 25, cut at different granularities.
`conformance/manifest.json` records both blockers under `scope_planned.blockers`, measured,
so the gap is visible rather than inferred.
## Non-goals
@ -114,10 +126,18 @@ than inferred.
- **Coverage is the union of what the seed implementations detected**, not of what exists.
A class absent from the tables above has not been shown to work anywhere.
- **The corpus is narrower than the data.** `conformance/` constrains one of the seven data
files. The other six are published, provenance-checked and unfixtured: a runtime can
- **The corpus is narrower than the data.** `conformance/` constrains two of the seven data
files. The other five are published, provenance-checked and unfixtured: a runtime can
pass every case and still read `calibration.json` wrongly. Passing the corpus is evidence
about the injection lexicon and about nothing else.
about the injection lexicon and about active content, and about nothing else.
- **A pass count is unreadable without the declared table set.** A runtime implementing one
table and a runtime implementing four can print the same number. `not-applicable` cases
must be reported, not dropped from the denominator — `76/83` and `76 passed, 6
not-applicable` describe different runtimes.
- **The six active-content cases prove less than the 83.** Their payloads come from the only
runtime that implements the table, so no second implementation's agreement could be
measured. They pin one runtime's behaviour as a contract a future implementer can be held
to; they are not cross-runtime agreement.
- **Regex portability is a real risk.** Pattern data is written for a common subset, but
engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose
engine rejects a pattern must report it rather than silently skip it — a skipped pattern