docs: measure the lexicon port divergence, correct the case count

The guard's coverage.py turned out to be readable from a session here — the
read boundary covers llm-security only. Counting it instead of estimating
corrected two published numbers and surfaced the real conformance blocker.

Counted, not estimated: CORE_CASES holds 134 cases (128 caught, 6 gap), not
the "126 classes + 4 gaps-must-hold" the extraction plan claimed. 105 are
convertible to static input/expected; 29 assert a runtime API surface this
repository does not own, and are named as out of scope rather than faked.

Measured while counting, and the reason the corpus is still blocked: the
commons lexicon and the guard's are two ports of one source file, and they
disagree. 83/83 patterns correspond, 64 byte-identical, 6 differ only by
escaping and are proven equivalent by string identity, and 13 behave
differently — with a witness input for each and misses in both directions.
Cause is two different ReDoS mitigations of the same table: the Node side
bounds the run ({0,256}), the guard excludes the anchor character ([^><]).
Each has a recall cost the other does not.

No data file changed. Behaviour preservation is a v0.1.0 invariant, so the
divergence is reported to the owning repositories, not fixed here. The 8
hybrid severities the guard's port assigns are deliberately NOT copied in —
a second-hand port is not the producing module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3

CHANGELOG.md

@@ -43,13 +43,25 @@ file records its own result and its own limits.
 summary rather than as code, so no differential check was possible, and the file names the
 checks that were not run instead of attaching a caveat to a pass.
 
+- `docs/lexicon-port-divergence.md` — informative. A differential comparison of the two
+  ports of `injection-patterns.mjs` (this repository's and the Python guard's): 83/83
+  patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven
+  equivalent, and **13 behave differently**, with a witness input for each and misses on
+  both sides. The cause is two different ReDoS mitigations of one table. **No data file was
+  changed** — behaviour preservation holds and the finding is reported to the owning
+  repositories.
+
 ### Not included
 
 - `signatures/malware-signatures.json` — seed data not yet delivered.
 - `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
   from a data dump would be worse than an absent one.
-- `conformance/` — the corpus form is an open decision: the Python guard's coverage matrix
+- `conformance/` — blocked, and the blocker is now identified. 105 of the guard's 134
-  is probe-based rather than a static input/expected table.
+  coverage cases are convertible to static `input.txt`/`expected.json`; the other 29 assert
+  a runtime's API surface, which this repository does not own. But no fixture can be written
+  yet at all: an `expected.json` names findings, and the two runtimes have no shared name for
+  the same pattern (`override:ignore-previous` vs `override: ignore previous instructions`).
+  A commons-owned pattern id is a prerequisite for the corpus, not a parallel task.
 
 These are named in the README as planned rather than linked, so nothing in the repository
 points at a file that does not exist.

README.md

@@ -61,6 +61,7 @@ and keeping it that small is the point.
 | `spec/decode-pipeline.md` | **Planned, not in v0.1.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
 | `conformance/` | **Planned, not in v0.1.0.** One directory per case: `input.txt` in, `expected.json` out. Ground truth. |
 | [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
+| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
 
 Every JSON file carries a top-level `version`. Every normative specification carries a
 `Status: normative` marker. Rows marked **Planned** are named here because the layout is

docs/extraction-plan.md

@@ -55,7 +55,7 @@ data, never from recollection or inference.
 | `calibration/calibration.json` | `scanners/lib/severity.mjs` — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. *(Corrected 2026-08-09: this row said "thresholds + scanner caps" and the README said "entropy floors, scan caps, disposition ranks". No scan cap, entropy floor or disposition rank was delivered — searched across the whole dump: 0 occurrences each. Both rows now describe what arrived.)* |
 | `mapping/owasp-map.json` | `scanners/lib/severity.mjs` — `OWASP_MAP` (+ 3 sibling maps in the same file) |
 | `schema/finding.schema.json` | modelled on `scanners/lib/sarif-formatter.mjs`'s SARIF shape |
-| `conformance/` | union of the guard repo's `coverage.py` matrix (126 classes + 4 gaps-must-hold) and `llm-security/examples/` |
+| `conformance/` | union of the guard repo's `coverage.py` matrix and `llm-security/examples/`. *(Corrected 2026-08-09: this row said "126 classes + 4 gaps-must-hold". Counted from the imported `CORE_CASES` manifest, the matrix holds **134** cases — 128 `caught` and **6** `gap`. See the conformance-source breakdown below.)* |
 
 ## Constraints
 
@@ -257,6 +257,40 @@ repository's central rule: for every other file, a consumer that disagrees is wr
 one, until the module is supplied in executable form, a disagreement is not automatically the
 consumer's bug.
 
+## Conformance sources — counted 2026-08-09
+
+The guard's `coverage.py` is readable from a session here (the read boundary covers
+`llm-security` only, not the guard repository), so the matrix was imported and counted rather
+than estimated. `CORE_CASES` holds **134** cases: 128 `caught`, 6 `gap`.
+
+Only part of it can become a static `input.txt` / `expected.json` pair, because only part of
+it is text-in / findings-out:
+
+| Probe kind | Count | Convertible? |
+| --- | --- | --- |
+| `_scan_case` — scan a payload, assert a label | 103 | yes |
+| `_scan_case_okf` — scan an OKF document body | 2 | yes, if the fixture input may be a structured document rather than plain text |
+| `_predicate_case` — assert an engine-internal predicate (disposition compounds, documented gaps) | 13 | no |
+| `_raise_case` — assert the guard raises a typed error (contract asserters, OKF rejecters) | 16 | no |
+
+The 29 non-convertible cases are not a coverage hole in this repository. They exercise a
+runtime's **API surface** — that a Python call raises `OKFPathError`, that a disposition
+engine composes two findings a particular way. This repository publishes data and fixtures and
+explicitly does not own an API, so those rows belong to the guard's own suite, the same way
+the `Planned` rows in the README are named rather than faked.
+
+Of the 103 `_scan_case` rows, 83 are one-per-lexicon-pattern. Those are also the rows most
+affected by the id question below.
+
+**Blocking issue found while counting.** The guard and the Node table are two ports of one
+source file and they do not agree: 64 of 83 patterns byte-identical, 6 differing only by
+escaping, and **13 behaviourally divergent with concrete witness inputs** — in both
+directions. They also disagree about pattern identity itself (`override:ignore-previous` vs
+`override: ignore previous instructions`). Full measurement, method and limits:
+[`lexicon-port-divergence.md`](lexicon-port-divergence.md). Nothing was changed in any data
+file as a result — behaviour preservation holds, and the divergence is reported to the two
+owning repositories instead.
+
 ## Definition of done for v0.1.0
 
 1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md`
@@ -269,9 +303,10 @@ consumer's bug.
    `signatures/malware-signatures.json` (seed data not delivered),
    `spec/decode-pipeline.md` (needs the decode implementation; a normative spec inferred
    from a data dump would carry a `Status: normative` marker and pass every convention
-   check while asserting something nobody verified) and `conformance/` (corpus form is an
+   check while asserting something nobody verified) and `conformance/` (blocked — see
-   open decision — the guard's coverage matrix is probe-based, not a static
+   *Conformance sources* above: 105 of the guard's 134 cases are convertible, but a fixture
-   input/expected table).
+   cannot name a finding until commons owns a pattern id both runtimes map to, and 13
+   patterns behave differently between the two ports).
 
    They are absent rather than stubbed on purpose: an empty `conformance/` and a
    contentless normative spec would both *pass* the mechanical checks in this document

docs/lexicon-port-divergence.md

@@ -0,0 +1,139 @@
+# Lexicon port divergence — commons vs the Python guard
+
+**Status: informative.** Nothing here is normative and nothing here changes a data file. It
+records a measured disagreement between two ports of one source table, so the decision can be
+taken where each table is tested. Under this repository's behaviour-preservation invariant,
+a divergence found here is **reported, not fixed**.
+
+Produced 2026-08-09. Every number below came from a command; the scripts live in the session
+scratchpad rather than in this repository, because executable code here would breach the
+charter. They are reproducible from the method column.
+
+## What was compared
+
+| Side | Artefact | Version |
+| --- | --- | --- |
+| commons | [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json) | file `version` as committed |
+| guard | `llm-ingestion-pipeline-security` `src/llm_ingestion_guard/injection_lexicon.json` | lexicon `version` 1.0, repo v0.3.4, commit `0bf0729` |
+
+Both are **ports of the same file**: `llm-security/scanners/lib/injection-patterns.mjs`. The
+guard's JSON says so in its own `note` field — *"Injection lexicon ported from llm-security
+injection-patterns.mjs. Single source of truth."* Commons extracted the same table from an
+operator dump of that module.
+
+That is what makes the comparison worth running. These are not two different detectors that
+happen to overlap; they are two transcriptions of one table, and where they disagree, they
+disagree about what the same source says.
+
+## Result
+
+| Measure | Method | Result |
+| --- | --- | --- |
+| Pattern count, both sides | count entries | 83 and 83 |
+| Label correspondence | match commons `label` to guard `desc`, em-dash normalised to hyphen | **83/83** |
+| Regex source byte-identical | string compare | **64/83** |
+| Differing regex text | string compare | 19 |
+| — of those, provably equivalent | unescape commons' JS-isms (`\/` → `/`, `\uXXXX` → literal) and compare for string identity | **6/6 identical** |
+| — of those, behaviourally divergent | differential match-set comparison (offsets + matched text), targeted corpus per pattern family | **13**, each with a concrete witness input |
+| Total input comparisons | count | 401 |
+| Flags | compare declared flags | **0 differences** |
+| Severity / family | commons family vs guard `severity` | **8 differences** (all `hybrid`) |
+
+64 identical + 6 escaping-only + 13 divergent = 83.
+
+The 6 escaping-only pairs are equivalent by **string identity after unescaping**, not merely
+by absence of a counter-example: `<\/system>` vs `</system>`, `text\/html` vs `text/html`,
+`(?:https?:\/\/…)` vs `(?:https?://…)`, `(?:~\/)?` vs `(?:~/)?`, and two `\uXXXX`-vs-literal
+character classes. Commons writes code points as escapes by convention; the guard's `note`
+records the same choice for non-ASCII.
+
+## The 13, and which side misses what
+
+Divergence is **not one-directional**. Each side has inputs the other's pattern does not match.
+
+| Pattern | Direction | Witness input |
+| --- | --- | --- |
+| `html-obfuscation: display:none element with content` | commons misses; span differs | attribute run padded past 256 chars |
+| `html-obfuscation: visibility:hidden element` | span only | `<div data-x="<" style="visibility:hidden">x</div>` |
+| `html-obfuscation: off-screen positioned element` | span only | same shape, `position:absolute` |
+| `html-obfuscation: zero font-size element` | span only | same shape, `font-size:0` |
+| `html-obfuscation: zero opacity element` | span only | same shape, `opacity:0` |
+| `html-obfuscation: zero-size overflow-hidden element` | span only | same shape, `height:0;overflow:hidden` |
+| `html-obfuscation: injection in aria-label attribute` | **commons misses** | `<a aria-label="` + 300 × `a` + `ignore">x</a>` |
+| `markdown: suspicious instruction in link-reference comment` | **guard misses** | `[//]: # (a (b) ignore)` |
+| `markdown: injection payload in link anchor text` | **guard misses** | `[click ignore this](http://e.com/a(b))` |
+| `sub-agent: spawn instruction with dangerous capability keywords` | **guard misses** | `create an agent that` + 20 filler words + `execute` |
+| `sub-agent: delegation with safety bypass instruction` | **guard misses** | `delegate to a new agent` + 200 chars + `bypass` |
+| `hybrid-xss: <script> tag in content (agent context XSS)` | **commons misses**; span differs | `<script>alert(1)` (unclosed), `<script src=x.js>` |
+| `hybrid-xss: iframe with executable src (agent context XSS)` | **guard misses** | `<iframe data-x="<" src="javascript:alert(1)">` |
+
+"Span only" means both sides produce a match on the same input but over different extents —
+the guard's match starts at an interior `<`. Whether that matters depends on whether a
+consumer reports offsets or evidence text; it does not change whether a finding is raised.
+
+The commons-side misses were confirmed in a real JS engine (Node v25.8.2, `RegExp` built from
+the committed JSON), not only in the Python harness used for the differential.
+
+## Why they diverge: two different ReDoS mitigations of one table
+
+This is not drift, and framing it as a bug in either repository would be wrong.
+
+Both ports have been hardened against catastrophic backtracking, by **different strategies**:
+
+- **The Node side bounds the run.** `[^"]{0,256}`, `[^>]{1,256}`. Cost: an attacker who pads
+  the attribute past 256 characters falls out of the pattern.
+- **The guard excludes the anchor character.** `[^><]`, `[^\]\[]`, `[^)(]`. Cost: content that
+  legitimately contains that character stops matching.
+
+The guard's campaign is documented in its own CHANGELOG (v0.3.2 and v0.3.4: *"exclude the
+character that opens the pattern's own anchor"*, a sweep over *"150 patterns across 11
+tables"*), including its measured recall cost — it names exactly the two exceptions this
+comparison rediscovered: *"URLs containing a literal `(` inside a markdown link target and
+comment bodies containing a literal `(` before the keyword."* The bounded `{0,12}` / `{0,120}`
+sub-agent quantifiers produce misses that the guard's exception list does not mention.
+
+Neither strategy is free, and neither is obviously right. That is the decision the two owning
+repositories have to take, and it is not commons' to take for them.
+
+## Severity: the 8 hybrid patterns
+
+Commons records the `hybrid` family with `severity: null` and a note that the seed dump did
+not supply it, so a consumer **MUST NOT** assume one. The guard's port assigns all eight
+`high`.
+
+**This has deliberately not been copied into commons.** The guard's port is a second-hand
+transcription, not the producing module; adopting its value would convert a documented gap
+into an unverified claim, which is the defect class this repository's changelog already
+records twice. It is reported instead, as a strong hint that the Node source assigns `high`,
+for `llm-security` to confirm from the module.
+
+## What this does not show
+
+- **Not that either runtime misses an attack.** All witness payloads were re-tested against
+  every pattern table commons holds (111 compiled rules: 83 lexicon, 28 from
+  `active-content.json` and `secret-egress.json`). All of them still produce a finding, via
+  `active-content: constructs.raw-html`. The divergence is in the **finding set** — which
+  labels are raised — not in whether anything is raised at all. That still matters, because a
+  finding set is exactly what a `conformance/expected.json` would encode.
+- **Not that `llm-security` behaves as described here.** Only its extracted pattern table was
+  available. Whether the Node engine runs an active-content table that compensates cannot be
+  determined from this repository.
+- **Not dump-to-module fidelity.** Every check above proves the two *ports* agree or disagree.
+  That either matches `injection-patterns.mjs` remains `llm-security`'s assertion,
+  reproducible only in a session with read access to it.
+- **Not exhaustive.** The corpus is targeted per pattern family, 401 comparisons. Absence of a
+  witness proves nothing except for the 6 escaping-only pairs, which are settled by string
+  identity rather than by the corpus.
+
+## Consequence for `conformance/`
+
+An `expected.json` names findings. Naming a finding needs a stable id, and the two runtimes do
+not have one: the same pattern is `override:ignore-previous` in the guard and
+`override: ignore previous instructions` in the Node table. The guard's JSON happens to carry
+both — `id` and `desc` — which is evidence that a commons-owned id is achievable rather than
+speculative.
+
+So the id question is a **prerequisite** for the corpus, not a parallel task: until commons
+owns a pattern id both ports map to, no fixture can be written, including for the 64 patterns
+that are byte-identical. And for the 13 divergent patterns a fixture cannot be authored at all
+without first deciding whose recall cost is the contract.