open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

release(0.3.0): a detection pattern changed value — that is new here

Browse source 
v0.2.0 changed what a runtime must DECLARE. This one changes what a conforming
runtime FINDS. A consumer that vendors 0.3.0 and re-runs its suite will see a
finding on inputs that produced none under 0.2.0. In 0.x that is a minor bump by
the rules; the CHANGELOG says so at the top rather than leaving it to be found in
a diff.

The lexicon 0.6.0 entry that sat under [Unreleased] is folded into this release —
committed but never tagged, and 0.7.0 supersedes its central claim (that commons
does not follow the upstream change).

Version sync: README's subtree-add and clone examples move v0.2.0 -> v0.3.0, the
two "as of v0.2.0" markers move with them, the corpus row goes 89 -> 90 cases and
83 -> 84 lexicon cases, and the scope breakdown now says 84 cases over 83 patterns
rather than implying one case per pattern. schema/conformance-declaration.schema.json
gains its own row; the corpus-spec row now mentions the declared table set, since
§1.1 is the reason a reader would go looking.

Verification run before tagging: all JSON well-formed; every non-conformance JSON
carries a top-level version; every spec carries Status: normative; charter guard
clean (no executable code); manifest count == cases[] length == directories on disk
== 90, no duplicate ids; every case's recorded bytes and sha256 recomputed from
disk, zero mismatches, no trailing newlines; measured count_by_scope equals the
manifest's; declaration schema valid draft 2020-12 with its example validating.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HuudQLGMyMenuyeZk8fKs5
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:39:51 +02:00
commit 4641a7b518
2 changed files with 137 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

18
README.md
View file

@ -24,7 +24,7 @@ As a `git subtree` (recommended: history is preserved and upgrades are a single
```bash
git subtree add --prefix vendor/commons \
https://git.fromaitochitta.com/open/llm-security-commons.git v0.2.0 --squash
https://git.fromaitochitta.com/open/llm-security-commons.git v0.3.0 --squash
# later, to move to a newer tag
git subtree pull --prefix vendor/commons \
@ -34,7 +34,7 @@ git subtree pull --prefix vendor/commons \
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
```bash
git clone --depth 1 --branch v0.2.0 \
git clone --depth 1 --branch v0.3.0 \
https://git.fromaitochitta.com/open/llm-security-commons.git
```
@ -58,16 +58,17 @@ and keeping it that small is the point.
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself. |
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
| [`conformance/`](conformance/) | 89 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 83 cover the injection lexicon, one per pattern, and both seeding runtimes were measured producing the same verdict on all 83. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.2.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`schema/conformance-declaration.schema.json`](schema/conformance-declaration.schema.json) | **Normative.** The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs. |
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, what `exact-within-scope` requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as `not-applicable` rather than as a failure. |
| [`conformance/`](conformance/) | 90 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.3.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
Every JSON file carries a top-level `version`. Every normative specification carries a
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
part of the contract, but the file does not exist yet — they are not links, and nothing in
v0.2.0 depends on them.
v0.3.0 depends on them.
Each data file records its own provenance and, in `verified`, how strongly it is backed.
`calibration/calibration.json` is currently the one file that says `false`: it was
@ -91,8 +92,9 @@ runs in each consumer's own test suite, against a pinned tag.
The corpus covers two tables, and they do not carry equal weight — treating them as one
number would misreport both:
- `lexicon/injection-lexicon.json` — 83 cases. Both seeding runtimes implement it and both
ratified its id space.
- `lexicon/injection-lexicon.json` — 84 cases over 83 patterns. Both seeding runtimes
implement it and both ratified its id space. One pattern carries a second, variant case;
see `case_id_derivation.variant_suffix` in the manifest.
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
runtime declares which commons data files it implements, and a case scoped outside that