open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

docs: measure the lexicon port divergence, correct the case count

Browse source 
The guard's coverage.py turned out to be readable from a session here — the
read boundary covers llm-security only. Counting it instead of estimating
corrected two published numbers and surfaced the real conformance blocker.

Counted, not estimated: CORE_CASES holds 134 cases (128 caught, 6 gap), not
the "126 classes + 4 gaps-must-hold" the extraction plan claimed. 105 are
convertible to static input/expected; 29 assert a runtime API surface this
repository does not own, and are named as out of scope rather than faked.

Measured while counting, and the reason the corpus is still blocked: the
commons lexicon and the guard's are two ports of one source file, and they
disagree. 83/83 patterns correspond, 64 byte-identical, 6 differ only by
escaping and are proven equivalent by string identity, and 13 behave
differently — with a witness input for each and misses in both directions.
Cause is two different ReDoS mitigations of the same table: the Node side
bounds the run ({0,256}), the guard excludes the anchor character ([^><]).
Each has a recall cost the other does not.

No data file changed. Behaviour preservation is a v0.1.0 invariant, so the
divergence is reported to the owning repositories, not fixed here. The 8
hybrid severities the guard's port assigns are deliberately NOT copied in —
a second-hand port is not the producing module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 21:40:26 +02:00
commit 22d3a78591
4 changed files with 193 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

16
CHANGELOG.md
View file

@ -43,13 +43,25 @@ file records its own result and its own limits.
summary rather than as code, so no differential check was possible, and the file names the
checks that were not run instead of attaching a caveat to a pass.
- `docs/lexicon-port-divergence.md` — informative. A differential comparison of the two
ports of `injection-patterns.mjs` (this repository's and the Python guard's): 83/83
patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven
equivalent, and **13 behave differently**, with a witness input for each and misses on
both sides. The cause is two different ReDoS mitigations of one table. **No data file was
changed** — behaviour preservation holds and the finding is reported to the owning
repositories.
### Not included
- `signatures/malware-signatures.json` — seed data not yet delivered.
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
from a data dump would be worse than an absent one.
- `conformance/` — the corpus form is an open decision: the Python guard's coverage matrix
is probe-based rather than a static input/expected table.
- `conformance/` — blocked, and the blocker is now identified. 105 of the guard's 134
coverage cases are convertible to static `input.txt`/`expected.json`; the other 29 assert
a runtime's API surface, which this repository does not own. But no fixture can be written
yet at all: an `expected.json` names findings, and the two runtimes have no shared name for
the same pattern (`override:ignore-previous` vs `override: ignore previous instructions`).
A commons-owned pattern id is a prerequisite for the corpus, not a parallel task.
These are named in the README as planned rather than linked, so nothing in the repository
points at a file that does not exist.