open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

docs: measure the lexicon port divergence, correct the case count

Browse source 
The guard's coverage.py turned out to be readable from a session here — the
read boundary covers llm-security only. Counting it instead of estimating
corrected two published numbers and surfaced the real conformance blocker.

Counted, not estimated: CORE_CASES holds 134 cases (128 caught, 6 gap), not
the "126 classes + 4 gaps-must-hold" the extraction plan claimed. 105 are
convertible to static input/expected; 29 assert a runtime API surface this
repository does not own, and are named as out of scope rather than faked.

Measured while counting, and the reason the corpus is still blocked: the
commons lexicon and the guard's are two ports of one source file, and they
disagree. 83/83 patterns correspond, 64 byte-identical, 6 differ only by
escaping and are proven equivalent by string identity, and 13 behave
differently — with a witness input for each and misses in both directions.
Cause is two different ReDoS mitigations of the same table: the Node side
bounds the run ({0,256}), the guard excludes the anchor character ([^><]).
Each has a recall cost the other does not.

No data file changed. Behaviour preservation is a v0.1.0 invariant, so the
divergence is reported to the owning repositories, not fixed here. The 8
hybrid severities the guard's port assigns are deliberately NOT copied in —
a second-hand port is not the producing module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 21:40:26 +02:00
commit 22d3a78591
4 changed files with 193 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

43
docs/extraction-plan.md
View file

@ -55,7 +55,7 @@ data, never from recollection or inference.
| `calibration/calibration.json` | `scanners/lib/severity.mjs` — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. *(Corrected 2026-08-09: this row said "thresholds + scanner caps" and the README said "entropy floors, scan caps, disposition ranks". No scan cap, entropy floor or disposition rank was delivered — searched across the whole dump: 0 occurrences each. Both rows now describe what arrived.)* |
| `mapping/owasp-map.json` | `scanners/lib/severity.mjs` — `OWASP_MAP` (+ 3 sibling maps in the same file) |
| `schema/finding.schema.json` | modelled on `scanners/lib/sarif-formatter.mjs`'s SARIF shape |
| `conformance/` | union of the guard repo's `coverage.py` matrix (126 classes + 4 gaps-must-hold) and `llm-security/examples/` |
| `conformance/` | union of the guard repo's `coverage.py` matrix and `llm-security/examples/`. *(Corrected 2026-08-09: this row said "126 classes + 4 gaps-must-hold". Counted from the imported `CORE_CASES` manifest, the matrix holds **134** cases — 128 `caught` and **6** `gap`. See the conformance-source breakdown below.)* |
## Constraints
@ -257,6 +257,40 @@ repository's central rule: for every other file, a consumer that disagrees is wr
one, until the module is supplied in executable form, a disagreement is not automatically the
consumer's bug.
## Conformance sources — counted 2026-08-09
The guard's `coverage.py` is readable from a session here (the read boundary covers
`llm-security` only, not the guard repository), so the matrix was imported and counted rather
than estimated. `CORE_CASES` holds **134** cases: 128 `caught`, 6 `gap`.
Only part of it can become a static `input.txt` / `expected.json` pair, because only part of
it is text-in / findings-out:
| Probe kind | Count | Convertible? |
| --- | --- | --- |
| `_scan_case` — scan a payload, assert a label | 103 | yes |
| `_scan_case_okf` — scan an OKF document body | 2 | yes, if the fixture input may be a structured document rather than plain text |
| `_predicate_case` — assert an engine-internal predicate (disposition compounds, documented gaps) | 13 | no |
| `_raise_case` — assert the guard raises a typed error (contract asserters, OKF rejecters) | 16 | no |
The 29 non-convertible cases are not a coverage hole in this repository. They exercise a
runtime's **API surface** — that a Python call raises `OKFPathError`, that a disposition
engine composes two findings a particular way. This repository publishes data and fixtures and
explicitly does not own an API, so those rows belong to the guard's own suite, the same way
the `Planned` rows in the README are named rather than faked.
Of the 103 `_scan_case` rows, 83 are one-per-lexicon-pattern. Those are also the rows most
affected by the id question below.
**Blocking issue found while counting.** The guard and the Node table are two ports of one
source file and they do not agree: 64 of 83 patterns byte-identical, 6 differing only by
escaping, and **13 behaviourally divergent with concrete witness inputs** — in both
directions. They also disagree about pattern identity itself (`override:ignore-previous` vs
`override: ignore previous instructions`). Full measurement, method and limits:
[`lexicon-port-divergence.md`](lexicon-port-divergence.md). Nothing was changed in any data
file as a result — behaviour preservation holds, and the divergence is reported to the two
owning repositories instead.
## Definition of done for v0.1.0
1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md`
@ -269,9 +303,10 @@ consumer's bug.
`signatures/malware-signatures.json` (seed data not delivered),
`spec/decode-pipeline.md` (needs the decode implementation; a normative spec inferred
from a data dump would carry a `Status: normative` marker and pass every convention
check while asserting something nobody verified) and `conformance/` (corpus form is an
open decision — the guard's coverage matrix is probe-based, not a static
input/expected table).
check while asserting something nobody verified) and `conformance/` (blocked — see
*Conformance sources* above: 105 of the guard's 134 cases are convertible, but a fixture
cannot name a finding until commons owns a pattern id both runtimes map to, and 13
patterns behave differently between the two ports).
They are absent rather than stubbed on purpose: an empty `conformance/` and a
contentless normative spec would both *pass* the mechanical checks in this document