open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

Merge commit 'a640f43d73' as 'scanners/commons'

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 20:40:16 +02:00
commit 3b919f39b4
183 changed files with 6245 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

432
scanners/commons/docs/extraction-plan.md Normal file
View file

@ -0,0 +1,432 @@
# Extraction plan — v0.1.0
**Status: informative.** This is the plan of record for how this repository came to exist,
copied verbatim (structure preserved, lightly reformatted) from the operator brief that
opened it. It is **not** normative: nothing here constrains a consumer. When it disagrees
with `spec/` or `schema/`, those win.
Origin: **Phase 4 of the `llm-security` v8 plan**, which lives in the sibling repository
`llm-security`. That repository is context only — no session in this repository reads from
or writes to it.
## Charter
No engine code. Only: JSON data, normative specs, and a conformance corpus that several
runtimes (Node in `llm-security`, Python in a guard repo, a wiki) can run against and get
an identical verdict from. The pattern is copied from the sibling repository
`portfolio-optimiser-commons` (hard charter: "nothing here may import/depend on a
framework").
## Layout
```
llm-security-commons/
README.md # charter: data+contract+fixtures only, no engine code
lexicon/injection-lexicon.json
codepoints/carriers.json # zero-width, BIDI, Unicode-Tag ranges, homoglyph map
signatures/secret-egress.json
signatures/malware-signatures.json
signatures/active-content.json # EchoLeak: MD image/link/refdef/autolink, data:, active HTML
calibration/calibration.json # risk-score tiers, verdict/band cutoffs, grade thresholds
mapping/owasp-map.json # prefix -> LLM/ASI/AST/MCP
schema/finding.schema.json # + SARIF & JSONL profiles. Status: normative
spec/decode-pipeline.md # normative RFC-2119 decode order
conformance/ # {case}/input.txt + {case}/expected.json
STATE.md # LOCAL-ONLY / gitignored (mirror commons convention)
```
Every JSON file carries a top-level `"version"` field. Every spec carries a
`Status: normative` marker.
## v0.1.0 seed sources
`llm-security` is the canonical and richest source. Security-critical tables (homoglyph map,
secret patterns, malware signatures) MUST come from real source data, never from recollection
or inference.
**The read boundary changed on 2026-08-09 (operator decision), and it matters to how the
rows below should be read.** This section previously stated that sessions here have no read
access and that content arrives only as an operator-supplied dump. Sessions may now **read
`llm-security` read-only from its public remote at a pinned commit** —
`ssh://git@git.fromaitochitta.com/open/llm-security.git` @ `b0de0ca`, verified as
`origin/main` — by cloning into a scratchpad. The sibling working copy on this machine
remains **absolutely off limits**, as does any form of writing.
The consequence is a method change, not just a convenience: files seeded from a dump are
**differentially verified against the module** rather than trusted as transcriptions. That
change has already caught four attribution defects — a re-escaping story that never happened
in the lexicon, an inverted escape in one pattern, a severity citation pointing at a file with
no severities in it, and two family names misremembered in a working note. Every one was a
transcription artefact. Where a row below still says a dump was the delivery, the
`provenance` block in the file itself records whether that dump has since been superseded by
a direct read.
| Target | Seed source in `llm-security` (unless noted) |
|---|---|
| `lexicon/injection-lexicon.json` | `scanners/lib/injection-patterns.mjs` |
| `codepoints/carriers.json` | `scanners/unicode-scanner.mjs` + `scanners/lib/string-utils.mjs` (incl. `HOMOGLYPH_MAP`) |
| `signatures/secret-egress.json` | `hooks/scripts/pre-edit-secrets.mjs` — `SECRET_PATTERNS`, the **18-entry hook table**, NOT the PCRE-flavored agent-consumed variant in `knowledge/secrets-patterns.md`. *(Corrected 2026-08-09: this row originally named `knowledge/secrets-patterns.md` as the source file. The delivered dump named `pre-edit-secrets.mjs` and stated explicitly that the two are different files. The row now names the file that was actually delivered.)* |
| `signatures/malware-signatures.json` | `knowledge/signatures.json` (the SIG scanner) |
| `signatures/active-content.json` | currently only in a guard repo's `active_content.py`. If unavailable: stub with a version field and a TODO naming the source |
| `calibration/calibration.json` | `scanners/lib/severity.mjs` — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. *(Corrected 2026-08-09: this row said "thresholds + scanner caps" and the README said "entropy floors, scan caps, disposition ranks". No scan cap, entropy floor or disposition rank was delivered — searched across the whole dump: 0 occurrences each. Both rows now describe what arrived.)* |
| `mapping/owasp-map.json` | `scanners/lib/severity.mjs` — `OWASP_MAP` (+ 3 sibling maps in the same file) |
| `schema/finding.schema.json` | modelled on `scanners/lib/sarif-formatter.mjs`'s SARIF shape |
| `conformance/` | union of the guard repo's `coverage.py` matrix and `llm-security/examples/`. *(Corrected 2026-08-09: this row said "126 classes + 4 gaps-must-hold". Counted from the imported `CORE_CASES` manifest, the matrix holds **134** cases — 128 `caught` and **6** `gap`. See the conformance-source breakdown below.)* |
## Constraints
- Offline / deterministic only — no network, no model calls inside the data itself.
- Forgejo `open/` — never GitHub.
- MIT license, fork-and-own.
- `STATE.md` is LOCAL-ONLY (gitignored) — same convention as the rest of the polyrepo.
- **Behaviour preservation is the point:** this must not change a single finding in
`llm-security` when it is later consumed from here. That consumption happens in
`llm-security`'s own Phase 5 steps 3–4 — **not here.**
## Verification log
Every claim of fidelity below was produced by a command, not by reading. The check scripts
themselves deliberately do **not** live in this repository — executable code here would
breach the charter. They are reproducible from the description given.
### `signatures/active-content.json` — extracted 2026-08-09
Source: `llm-ingestion-pipeline-security` v0.3.4, commit `0bf0729` (2026-08-03),
`src/llm_ingestion_guard/active_content.py` + `calibration.py`. Read-only; nothing in that
repository was modified.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed | `python3 -m json.tool` | pass |
| Patterns compile as Python `re` | translate `(?<` → `(?P<`, compile all 17 with declared flags | 17/17, 0 failures |
| Patterns compile as ECMAScript | `new RegExp(pattern, flags)` on all 17 | 17/17, 0 failures |
| Pattern text matches source | compare against the live `re.Pattern.pattern` of each source object, inline flags stripped | 12/17 byte-identical; 5 differ only by the documented `redundant-quote-escape` normalisation |
| The 5 normalised patterns behave identically | differential match-set comparison (offsets + captured text) against the source objects over a 30-input adversarial corpus: bare quotes, escaped quotes, markdown titles containing quotes, quoted/unquoted HTML attributes, quote runs of length 1–5 | 150 comparisons, 0 differences |
| The normalisation is necessary | `new RegExp('\\"', 'u')` and `'v'` in Node | both throw `Invalid escape`; the bare form compiles under `""`, `"u"` and `"v"` |
| Severities, ordinary severity, opacity floors, active-tag set, pass order | compare against `calibration.ACTIVE_CONTENT_SEVERITY`, `ACTIVE_CONTENT_ORDINARY_SEVERITY`, `URL_OPAQUE_*`, `active_content._ACTIVE_TAGS`, and the scan-call order in `scan_active_content` | all identical (23/23 tags, 6/6 severities, 4/4 floors) |
Not verified, and not claimed: that the Node consumer's active-content behaviour matches
this table. The source module states the Node port shares its severities; that is the
module's claim, and confirming it needs the Node file.
### `schema/finding.schema.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/sarif-formatter.mjs`, supplied as an operator dump. No
commit hash accompanied it, so provenance is recorded as `unknown` rather than guessed.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed | `python3 -m json.tool` | pass |
| Valid JSON Schema | `jsonschema` `check_schema` against draft 2020-12 | pass |
| Accepts/rejects findings correctly | 2 valid + 3 invalid findings (missing `scanner`, unknown severity, `line: 0`) | 5/5 as intended |
| SARIF profile reproduces the source | re-implemented the mapping **from the commons JSON alone** and diffed `JSON.stringify` against the real `toSARIF` over 10 envelope shapes: empty, missing `scanners`, empty `scanners`, scanner with no findings, all five severities plus an unknown and an `undefined` one, five slug edge cases (double space, tab, newline, leading/trailing space, mixed case), a rule-id collision, all seven optional-field combinations, two scanners, and an explicit `version` argument | 10/10 identical, 0 differences |
| The three `known_lossiness` claims are true | executed each against the real formatter | all three confirmed, **and one earlier claim corrected**: punctuation does *not* collapse — the slug lowercases and collapses whitespace only, so `Zero-width carrier` and `Zero-width carrier!` remain distinct ids. The wrong claim was published in the first draft of this file and fixed before commit. |
Not verified, and recorded in the file as open: the finding **producer** was not supplied, so
the property list is a lower bound; `scanner` and `severity` are required by design rather
than by evidence; and the JSONL profile is left explicitly `unspecified` rather than
invented, because "one finding per line" is inference.
### `lexicon/injection-lexicon.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/injection-patterns.mjs`, supplied as operator dump 2/2
through the local coord mailbox. No commit hash accompanied it, so provenance is recorded
as `unknown` rather than guessed.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present, LF, trailing newline, no raw invisible code points | `python3 -m json.tool` + a byte scan for U+200B/200C/200D/FEFF/00AD and the Tag block | pass, 0 raw invisible code points |
| Pattern text and flags reproduce the source | rebuilt all four arrays **from the commons JSON alone** (`new RegExp(p.pattern, p.flags ?? '')`) and diffed label, `.source` and `.flags` against the imported dump module | 83/83 compared, 0 differences; 81/83 byte-identical, 2 declared-normalised |
| Flags were read mechanically, not by eye | extracted from each literal via `.flags` | critical 15×`i` / 3×`m` / 3 none, high 32×`i`, medium 20×`i` / 2 none, hybrid 8×`i` |
| Every pattern compiles in both runtimes | `new RegExp(src, flags)` and again with `u` in Node; `re.compile` with the equivalent `re.I`/`re.M` in Python | 83/83 in all three modes, 0 failures |
| The 2 normalised patterns behave identically | differential match-set comparison (offsets + matched text) against the source objects, bare and under `u`, over a 208-input adversarial corpus: every class member, the near-misses excluded from each class (U+00AD, U+2060, U+180E, Cyrillic х, the uppercase set, Greek look-alikes), run boundaries, repeats, empty input | 832 comparisons, 0 differences |
| Class membership was counted, not assumed | enumerated the code points inside each character class directly from the dump bytes | zero-width class = 4 (U+200B, U+200C, U+200D, U+FEFF — **not** U+00AD); Cyrillic class = 7 (U+0430, U+0435, U+043E, U+0440, U+0441, U+0456, U+0443) |
| `\/` is portable, not a defect | 9 patterns carry the redundant escape a JS regex literal requires; compiled in Node bare, Node `u`, and Python `re` | accepted by all three — kept byte-identical, recorded as a translation note for engines that reject unknown escapes |
Not verified, and not claimed: that the dump matches the module it was transcribed from.
Every check above proves this JSON agrees with **the dump**; dump-to-module fidelity is
`llm-security`'s assertion, reproducible only in a session with read access to that
repository. The severity the engine assigns to `HYBRID_PATTERNS` was not supplied and is
left `null` rather than inferred from its three sibling arrays.
### `codepoints/carriers.json` — extracted 2026-08-09
Source: `llm-security/scanners/unicode-scanner.mjs` (charset constants) and
`llm-security/scanners/lib/string-utils.mjs` (`HOMOGLYPH_MAP`), supplied as operator dump
2/2 through the local coord mailbox. No commit hash accompanied it.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present, no raw invisible code points | `python3 -m json.tool` + byte scan for zero-width, BIDI and Tag-block characters | pass, 0 raw invisible code points |
| Five of the six tables reproduce the source constants | rebuilt each **from the commons JSON alone** (`parseInt(codepoint.slice(2), 16)`) and diffed against the imported dump module | `ZERO_WIDTH_CHARS` 5/5, `BIDI_CHARS` 9/9, `CYRILLIC_CONFUSABLES` 13/13, tag start/end — 0 differences |
| The homoglyph map reproduces the source, including order | rebuilt the object from the entries array and compared keys, values and the whole object | 28/28 keys, values and insertion order identical |
| The map folds identically | applied NFKC + lookup with both the rebuilt and the source table over 12 inputs (Cyrillic and Greek injection spellings, Norwegian and German orthography, empty) | 0 differences |
| The exclusion rationale in the source comment is true | checked whether any of `åøæÅØÆäöüßéèêñç` is a key | 0 touched — ordinary Norwegian and German orthography is not folded |
| Convenience `char` fields agree with their own `codepoint` field | `String.fromCodePoint` round-trip on every entry | 41/41, 0 mismatches |
| Character names are not from recollection | resolved every name through Python `unicodedata` against the Unicode character database | all resolved |
| Table sizes were counted, not quoted | counted from the imported constants | homoglyph map holds **28** entries, not the "~25" the dump's own comment estimates; the counted number is the one recorded |
Not verified, and marked `verified: false` **in the file itself**: the two Supplementary
Private Use Area ranges. They arrived as a source comment with no constant behind them, so
unlike the other five tables there was nothing to import and diff. That asymmetry is
recorded per-table rather than averaged into a single file-level verdict.
Recorded and deliberately **not** reconciled, in `cross_table_notes`: the repository now
holds three overlapping Cyrillic sets and two overlapping zero-width sets, and none agree
exactly. The zero-width carrier table includes U+00AD while the lexicon's pattern class does
not; the lexicon's class contains U+0456 which `CYRILLIC_CONFUSABLES` lacks, and
`CYRILLIC_CONFUSABLES` contains U+0445 which the class lacks.
**Two claims that stood here until 2026-08-09 were wrong, and both are corrected rather
than quietly dropped.**
The first was a count: this paragraph said *six of the confusables have no entry in the fold
map*. It is **zero**. All 13 members of `CYRILLIC_CONFUSABLES` are keys in the 28-entry fold
map, and `codepoints/carriers.json` has said so correctly since it was written —
`cross_table_notes.cyrillic.in_confusables_but_not_foldable` is an empty array. The prose,
not the data, carried the error. The number six most likely came from the comparison run the
other way: the fold map holds **five** Cyrillic keys that are not confusables (U+0423,
U+0455, U+0456, U+0458, U+04CF), which `llm-security` confirms independently. A count
asserted in prose beside a data file that contradicts it is the cheapest kind of defect to
catch and the easiest to propagate, so the check is now: read the array, do not restate it.
The second was a characterisation. This paragraph said the sets are *deliberately distinct*.
`llm-security` states plainly that the U+0456 / U+0445 divergence **is drift, not design**.
They are not harmonising it now — that would be a behaviour change inside a
behaviour-preservation release — and have logged it as a v8.x task on their side. So the
divergence stays in the data exactly as it is, but it is recorded here as a known defect
awaiting a fix upstream, not as an intentional distinction. Those two readings imply opposite
things about whether a future consumer should expect the sets to converge.
### `signatures/secret-egress.json` — extracted 2026-08-09
Source: `llm-security/hooks/scripts/pre-edit-secrets.mjs` (`SECRET_PATTERNS`), supplied as
operator dump 2/2 through the local coord mailbox. No commit hash accompanied it.
**The seed-source row above was wrong and has been corrected.** It named
`knowledge/secrets-patterns.md`; the dump named `hooks/scripts/pre-edit-secrets.mjs` and
stated that the two are different tables — the second is PCRE-flavoured and agent-consumed
and stays where it is. Recording a source file that was never delivered is the same defect
class as the lossiness claim corrected in `finding.schema.json`, so it is corrected here in
the same commit as the file it describes.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present, LF, trailing newline | `python3 -m json.tool` + byte scan | pass |
| Pattern text and flags reproduce the source | rebuilt the table **from the commons JSON alone**, sorted by the declared `order`, and diffed name, `.source` and `.flags` against the imported dump module | 18/18, 0 differences, **18/18 byte-identical** — no normalisation needed |
| Every pattern compiles in both runtimes | `new RegExp` bare and under `u` in Node; `re.compile` with `re.I` where declared in Python | 18/18 in all three modes, 0 failures |
| The ordering contract holds, and is not decorative | reproduced first-match labelling from the commons order for a Bearer header containing a JWT and for a bare JWT, against the source table | both labels identical to source: header case → `Authorization header with token`, bare case → `JWT (three-part token)` |
| Reordering is detectable, not silent | ran the same Bearer input through a reversed table | label changes to `JWT (three-part token)` — order is load-bearing, which is why every entry carries an explicit `order` field |
| `order` is contiguous | compared to `range(18)` | 0–17, no gaps |
Not verified, and not claimed: that the dump matches the module. Not supplied, and therefore
not invented: any severity or per-entry disposition — the source table carries a name and a
pattern and nothing else. Out of scope by the dump's own statement: the runtime
policy-injected custom patterns (entries 19+). A consumer matching only this table matches
**less** than the seed hook does when a policy is loaded.
### `mapping/owasp-map.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/severity.mjs` (`OWASP_MAP`, `OWASP_AGENTIC_MAP`,
`OWASP_SKILLS_MAP`, `OWASP_MCP_MAP`), supplied as operator dump 2/2 through the local coord
mailbox. No commit hash accompanied it.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present | `python3 -m json.tool` | pass |
| All four maps reproduce the source | rebuilt each exported object **from the commons JSON alone**, iterating the declared prefix list, and compared `JSON.stringify` against the imported dump module | 4/4 identical — keys, order, values and empty arrays |
| Key sets are identical across taxonomies, and counted | compared key order across all four in both the commons file and the source | one shared order, **16** prefixes, in all eight objects |
| Empty arrays survive as arrays | type- and length-checked every prefix the source maps to nothing | `agentic` TRG/AST, `skills` WFL/SIG, `mcp` WFL/TRG/SIG/AST — all still `[]`, none dropped or nulled |
| Code prefixes are homogeneous per taxonomy | extracted the alphabetic prefix of every code with a regex and asserted one per map | LLM, ASI, AST, MCP — no mixed map |
Recorded as an **open question in the file**, because it is the one thing a consumer can get
wrong while matching this map exactly: the dump does not state which *edition* of each
taxonomy the codes belong to. That matters and is not pedantry — OWASP's Top 10 for LLM
Applications was renumbered between editions, and in the 2025 edition `LLM06` is Excessive
Agency, with earlier standalone entries consolidated into other numbers and System Prompt
Leakage and Vector and Embedding Weaknesses added as `LLM07` and `LLM08`
([OWASP project page](https://owasp.org/www-project-top-10-for-large-language-model-applications/),
[2025 edition summary](https://www.gravitee.io/blog/owasp-top-10-for-llm-applications-2025-a-practical-guide)).
A bare `LLM06` therefore does not identify a risk. Two runtimes can reproduce this map
perfectly and still publish reports that disagree about what a finding means. `taxonomy_name`
is left `null` rather than guessed, and the question is reported to `llm-security`.
Not supplied, and therefore not invented: what each scanner prefix means. `UNI`, `ENT`, `PRM`
and the rest are reproduced as opaque keys. Deliberately not inherited: the dump notes that a
fallback map inside one of `llm-security`'s own agent definitions covers only part of the
prefix set — that is drift in the consumer, and the complete 16-prefix set is what moves here.
### `calibration/calibration.json` — transcribed 2026-08-09
Source: `llm-security/scanners/lib/severity.mjs`, delivered as a **prose summary** inside
operator dump 2/2 — not as source code.
**This file is the exception in this repository, and it is marked as such in its own
`verification` block.** Every other data file here was rebuilt from its commons JSON and
diffed against an imported module. There was nothing to import here: the constants arrived
as human-written prose describing the module. The differential check was therefore not run
— not "passed with caveats", not run — and the file records `verified: false` with the
specific checks that were skipped.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present | `python3 -m json.tool` | pass |
| Risk bands are contiguous and non-overlapping | compared each band's `max + 1` to the next band's `min` across 0–100 | contiguous, no gap, no shared value |
| Band boundaries agree with the verdict thresholds | compared the BLOCK and WARNING score triggers to the band lower bounds | BLOCK 65 = Critical band min; WARNING 15 = Medium band min |
| `reachable_minimum` is arithmetic, not a new claim | recomputed `base + min(cap, log2(2) * mult)` independently for all four tiers | 80 / 48 / 20 / 4 — exact, because `log2(2)` is exactly 1 |
| Rebuild-from-commons and diff against source | **not run** — no importable source | — |
| Differential scoring over a corpus | **not run** — the formulas are engine and were not supplied in runnable form | — |
Recorded in the file as `not_supplied`, so the absence is visible rather than inferred: the
README, this plan's seed-source table and this plan's layout block all described *this file*
as holding **entropy floors, scan caps and disposition ranks**. None of the three arrived in
any dump — searched across the entire dump message: `entropy` 0 occurrences, `disposition` 0,
`rank` 0, `floor` 0. All three descriptions have been corrected to name what arrived rather
than what was expected. (`signatures/active-content.json` does carry a Shannon-entropy floor,
extracted from the guard repository; that is a different file with a different source, and is
not what the calibration rows promised.)
The consequence is stated in the file and is worth repeating here, because it inverts this
repository's central rule: for every other file, a consumer that disagrees is wrong. For this
one, until the module is supplied in executable form, a disagreement is not automatically the
consumer's bug.
## Conformance sources — counted 2026-08-09
The guard's `coverage.py` is readable from a session here (the read boundary covers
`llm-security` only, not the guard repository), so the matrix was imported and counted rather
than estimated. `CORE_CASES` holds **134** cases: 128 `caught`, 6 `gap`.
Only part of it can become a static `input.txt` / `expected.json` pair, because only part of
it is text-in / findings-out:
| Probe kind | Count | Convertible? |
| --- | --- | --- |
| `_scan_case` — scan a payload, assert a label | 103 | yes |
| `_scan_case_okf` — scan an OKF document body | 2 | yes, if the fixture input may be a structured document rather than plain text |
| `_predicate_case` — assert an engine-internal predicate (disposition compounds, documented gaps) | 13 | no |
| `_raise_case` — assert the guard raises a typed error (contract asserters, OKF rejecters) | 16 | no |
The 29 non-convertible cases are not a coverage hole in this repository. They exercise a
runtime's **API surface** — that a Python call raises `OKFPathError`, that a disposition
engine composes two findings a particular way. This repository publishes data and fixtures and
explicitly does not own an API, so those rows belong to the guard's own suite, the same way
the `Planned` rows in the README are named rather than faked.
**Convertibility has a second condition, and it is the one easy to miss.** A case is a
candidate only if the probe is text-in / findings-out **and** the label it asserts maps to data
this repository actually publishes. A fixture asserting a label commons has no table for would
quietly extend the contract to cover engine behaviour the charter excludes — the corpus would
then be testing something this repository does not define.
| Group | Cases | Asserted label maps to commons data? |
| --- | --- | --- |
| `lexicon` | 83 | Yes — `lexicon/injection-lexicon.json`, now via the `id` field |
| `active` | 7 | Yes — `signatures/active-content.json` |
| `carrier` | 8 | Partly. `output:*-present` and `lexicon:unicode-tags-present` rest on the code-point tables in `codepoints/carriers.json`. The five `sanitize:*` rows assert **stripper** behaviour, which is engine. |
| `egress` | 1 | Yes — `signatures/secret-egress.json` |
| `entropy` | 2 | **No.** Shannon-entropy detection is the guard's `entropy.py`; commons publishes no entropy table. (`active-content.json` carries an opacity floor — a different thing, for a different pass.) |
| `decoded` | 2 | **No.** These assert a finding *after* a decode pass. The decode order is exactly what `spec/decode-pipeline.md` would fix, and it is Planned because it is engine. |
| `okf` (`_scan_case_okf`) | 2 | No — the OKF document model is not published here. |
So the buildable set is smaller than 105: roughly 83 + 7 + 3 + 1 ≈ 94. The rest are named here
so a later session does not rediscover them as a gap. *(Revised 2026-08-10: this sentence
ended "minus the 13 lexicon patterns whose expected behaviour is still disputed". Measurement
showed the dispute does not reach these inputs, so the 13 are in. What kept the built corpus
at 83 rather than 94 turned out to be a different constraint entirely — the 11 non-lexicon
cases have no ratified cross-runtime finding id. See* Conformance corpus *below.)*
**Blocking issue found while counting.** The guard and the Node table are two ports of one
source file and they do not agree: 64 of 83 patterns byte-identical, 6 differing only by
escaping, and **13 behaviourally divergent with concrete witness inputs** — in both
directions. They also disagree about pattern identity itself (`override:ignore-previous` vs
`override: ignore previous instructions`). Full measurement, method and limits:
[`lexicon-port-divergence.md`](lexicon-port-divergence.md). Nothing was changed in any data
file as a result — behaviour preservation holds, and the divergence is reported to the two
owning repositories instead.
**Half of it is now cleared (operator decision, 2026-08-09).**
`lexicon/injection-lexicon.json` 0.2.0 publishes a commons-owned `id` per pattern with an
`aliases` object per runtime, adopted verbatim from the guard's port rather than invented.
Findings can therefore be named. ~~What still blocks the corpus is the other half: the 13
divergent patterns have no agreed expected behaviour, and a fixture for them would have to
pick a winner.~~ **The other half cleared on 2026-08-10 without anyone picking a winner** —
the divergence is not reachable from any input the corpus contains. See *Conformance corpus*
below.
## Definition of done for v0.1.0
1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md`
gitignored.
2. Every file in the layout above present and populated from verified seed data — or
explicitly and visibly stubbed where the source was unavailable.
**Status 2026-08-09, second revision: still unmet, and still why no tag has been cut —
but the remaining gap is two items, not three.** All **eight** JSON artefacts in the
layout now exist. `signatures/malware-signatures.json` was the last, added once
`knowledge/signatures.json` became readable at `b0de0ca` and reproduced from the file
rather than from a dump.
Seven of the eight are differentially verified against their source; three of those
(`lexicon`, `mapping/owasp-map`, `signatures/malware-signatures`) are verified against the
source **module** at a pinned commit rather than against a transcription, and
`schema/finding.schema.json` is closed against its producer. The eighth,
`calibration/calibration.json`, still carries `verified: false` — its source arrived as a
prose summary with no code behind it, so no differential check was possible, and the file
names the checks that were not run rather than attaching a caveat to a pass.
Two items remain absent, and only one of them still blocks:
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
from a data dump would carry a `Status: normative` marker and pass every convention
check in this document while asserting something nobody verified.
- ~~`conformance/`~~ — **built 2026-08-10, 83 cases.** See *Conformance corpus* below.
`spec/decode-pipeline.md` is absent rather than stubbed on purpose: a contentless
normative spec would *pass* every mechanical check in this document while making the
repository look more finished than it is. It is named as **Planned** in the README
instead, and is not linked, so nothing points at a file that does not exist.
### Conformance corpus — built 2026-08-10
The remaining blocker was the 13 divergent patterns, and it **dissolved under measurement
rather than under a decision**. The divergence was measured on witness inputs — padding past
256 characters, an interior `<`, an unclosed `<script>` — and the corpus payloads contain
none of those shapes. Run through both runtimes' public entry points, all 83 produce
identical lexicon finding sets, 13 of 13 divergent patterns included.
| Check | Method | Result |
| --- | --- | --- |
| Payload id set equals the commons id set | compared the 83 `_LEX_PAYLOADS` keys to the 83 lexicon ids | identical, both directions empty |
| Asserted id fires in both runtimes | `scanForInjection()` at `b0de0ca` and `scan_output(source=OUTPUT)` at `0bf0729`, labels mapped to ids through the lexicon's own `aliases` | **83/83** |
| Lexicon finding **sets** identical between runtimes | set equality per case, not membership | **83/83** |
| The 13 known-divergent patterns, on their own case input | same comparison, restricted to those 13 | **13/13 agree** |
| Severity agrees with what commons publishes | commons family `severity` vs the severity the guard emitted per finding | **83/83** |
| The corpus verifies from the corpus alone | a second harness that reads only the case directories, re-runs both runtimes on the bytes it finds, and checks every `expected.json` field including the digests | **83 cases, 0 failures** |
| The fixtures discriminate | deleted the middle third of each input and re-scanned; a fixture that still passes is not testing anything | 76/83 broke. The 7 survivors are the shortest payloads, where the mutation leaves the trigger intact — a weak mutation, not a weak fixture |
| Case ids are collision-free after the filesystem transform | `:` → `__` across all 83 | 83 unique; `__` occurs nowhere in the id space, so the transform is one-to-one |
Measured at the **public entry point of each runtime**, never at a rebuilt regex table. That
is the same discipline the retraction in [`lexicon-port-divergence.md`](lexicon-port-divergence.md)
was forced to adopt: a measurement taken at the wrong layer produces a number that describes
neither runtime.
**Scope, and why it is 83 and not 94.** The guard's matrix holds 105 convertible cases, of
which ~94 assert a label that maps to data this repository publishes. Only 83 shipped. The
lexicon is the sole table whose finding id space is **ratified by both runtimes**; the
carrier, active-content and secret-egress tables have no agreed cross-runtime finding names,
so writing fixtures for their 11 cases would mint a contract in the same stroke as the tag,
unilaterally. They are named in `conformance/manifest.json` under `scope_planned`.
The inputs were authored by one runtime — the guard's `coverage.py` — and are reproduced
verbatim. What makes them usable as a *cross-runtime* corpus is not their origin but the
measurement above, which ran them through the other runtime too. The asymmetry is recorded
in the manifest rather than averaged away.
Normative semantics live in [`spec/conformance-corpus.md`](../spec/conformance-corpus.md),
not here.
3. All JSON well-formed, every data file carrying `"version"`, every spec carrying
`Status: normative`.
4. Tagged `v0.1.0` and pushed.
5. A `coord` message sent to `llm-security` announcing that the repository and `v0.1.0`
exist, so Phase 5 step 3 (vendoring) can start from there.

271
scanners/commons/docs/lexicon-port-divergence.md Normal file
View file

@ -0,0 +1,271 @@
# Lexicon port divergence — commons vs the Python guard
**Status: informative.** Nothing here is normative and nothing here changes a data file. It
records a measured disagreement between two ports of one source table, so the decision can be
taken where each table is tested. Under this repository's behaviour-preservation invariant,
a divergence found here is **reported, not fixed**.
Produced 2026-08-09. Every number below came from a command; the scripts live in the session
scratchpad rather than in this repository, because executable code here would breach the
charter. They are reproducible from the method column.
**Revised the same day, after `llm-security`'s source became readable and both runtimes
replied.** Four things changed, and three of them are corrections to this file rather than
new results:
1. The claim that **neither runtime misses an attack** is **retracted**. It does. See
*What this does not show* — the measurement behind that claim unioned pattern tables
belonging to two different runtimes and read the result as a statement about each.
2. One of the 13 divergences does not reach report level, so **12** is the number that
changes what a report says. The 13 still blocks `conformance/`.
3. The `hybrid` **severity is resolved** to `high` — the reported hint was correct, and the
citation behind it was not.
4. The **pattern id space is ratified** by both runtimes.
Corrections are marked in place rather than edited away, because a reader who saw the first
version needs to know which sentence moved.
## What was compared
| Side | Artefact | Version |
| --- | --- | --- |
| commons | [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json) | file `version` as committed |
| guard | `llm-ingestion-pipeline-security` `src/llm_ingestion_guard/injection_lexicon.json` | lexicon `version` 1.0, repo v0.3.4, commit `0bf0729` |
Both are **ports of the same file**: `llm-security/scanners/lib/injection-patterns.mjs`. The
guard's JSON says so in its own `note` field — *"Injection lexicon ported from llm-security
injection-patterns.mjs. Single source of truth."* Commons extracted the same table from an
operator dump of that module.
That is what makes the comparison worth running. These are not two different detectors that
happen to overlap; they are two transcriptions of one table, and where they disagree, they
disagree about what the same source says.
## Result
| Measure | Method | Result |
| --- | --- | --- |
| Pattern count, both sides | count entries | 83 and 83 |
| Label correspondence | match commons `label` to guard `desc`, em-dash normalised to hyphen | **83/83** |
| Regex source byte-identical | string compare | **64/83** |
| Differing regex text | string compare | 19 |
| — of those, provably equivalent | unescape commons' JS-isms (`\/` → `/`, `\uXXXX` → literal) and compare for string identity | **6/6 identical** |
| — of those, behaviourally divergent | differential match-set comparison (offsets + matched text), targeted corpus per pattern family | **13**, each with a concrete witness input |
| — of those, divergent at REPORT level | re-check whether a sibling pattern raises an equivalent finding on the same witness | **12** — one of the 13 is a label-set difference only |
| Total input comparisons | count | 401 |
| Flags | compare declared flags | **0 differences** |
| Severity / family | commons family vs guard `severity` | **8 differences** (all `hybrid`) — resolved, see *Severity* below |
64 identical + 6 escaping-only + 13 divergent = 83.
**Read the 13 and the 12 as answering different questions.** Thirteen patterns produce
different label sets. Twelve of those change what a report would say. The gap is the
`hybrid-xss: iframe with executable src` row: the guard's version of that one pattern misses
the witness, but its `hybrid-xss: javascript: URI scheme` pattern fires on the same input at
the same severity and the same OWASP anchor, so a reader of the guard's report still sees a
`high` / `LLM01` finding on that payload. Measured, not reasoned: the guard's table matched
`hybrid-xss:javascript-uri` (high, LLM01) and nothing else; the Node engine matched both
`hybrid-xss: javascript: URI scheme` and `hybrid-xss: iframe with executable src`. A
`conformance/expected.json` would have to encode the difference, because a fixture names
labels — which is why the 13 stood as the number that blocked the corpus. *(Superseded
2026-08-10: it blocks a fixture written over a **witness** input, and the corpus contains
none. See* Consequence for `conformance/` *below.)*
The 6 escaping-only pairs are equivalent by **string identity after unescaping**, not merely
by absence of a counter-example: `<\/system>` vs `</system>`, `text\/html` vs `text/html`,
`(?:https?:\/\/…)` vs `(?:https?://…)`, `(?:~\/)?` vs `(?:~/)?`, and two `\uXXXX`-vs-literal
character classes. Commons writes code points as escapes by convention; the guard's `note`
records the same choice for non-ASCII.
## The 13, and which side misses what
Divergence is **not one-directional**. Each side has inputs the other's pattern does not match.
| Pattern | Direction | Witness input |
| --- | --- | --- |
| `html-obfuscation: display:none element with content` | commons misses; span differs | attribute run padded past 256 chars |
| `html-obfuscation: visibility:hidden element` | span only | `<div data-x="<" style="visibility:hidden">x</div>` |
| `html-obfuscation: off-screen positioned element` | span only | same shape, `position:absolute` |
| `html-obfuscation: zero font-size element` | span only | same shape, `font-size:0` |
| `html-obfuscation: zero opacity element` | span only | same shape, `opacity:0` |
| `html-obfuscation: zero-size overflow-hidden element` | span only | same shape, `height:0;overflow:hidden` |
| `html-obfuscation: injection in aria-label attribute` | **commons misses** | `<a aria-label="` + 300 × `a` + `ignore">x</a>` |
| `markdown: suspicious instruction in link-reference comment` | **guard misses** | `[//]: # (a (b) ignore)` |
| `markdown: injection payload in link anchor text` | **guard misses** | `[click ignore this](http://e.com/a(b))` |
| `sub-agent: spawn instruction with dangerous capability keywords` | **guard misses** | `create an agent that` + 20 filler words + `execute` |
| `sub-agent: delegation with safety bypass instruction` | **guard misses** | `delegate to a new agent` + 200 chars + `bypass` |
| `hybrid-xss: <script> tag in content (agent context XSS)` | **commons misses**; span differs | `<script>alert(1)` (unclosed), `<script src=x.js>` |
| `hybrid-xss: iframe with executable src (agent context XSS)` | **guard misses the label, not the attack** | `<iframe data-x="<" src="javascript:alert(1)">` |
The last row is the one of the 13 that does not reach report level. The guard's
`hybrid-xss:javascript-uri` (`javascript\s*:`, high, LLM01) matches that witness, so the
payload is still flagged at the same severity and anchor; only the label set differs — one
finding instead of two. The remaining 12 rows change what a report says.
"Span only" means both sides produce a match on the same input but over different extents —
the guard's match starts at an interior `<`. Whether that matters depends on whether a
consumer reports offsets or evidence text; it does not change whether a finding is raised.
The commons-side misses were confirmed in a real JS engine (Node v25.8.2, `RegExp` built from
the committed JSON), not only in the Python harness used for the differential.
## Why they diverge: two different ReDoS mitigations of one table
This is not drift, and framing it as a bug in either repository would be wrong.
Both ports have been hardened against catastrophic backtracking, by **different strategies**:
- **The Node side bounds the run.** `[^"]{0,256}`, `[^>]{1,256}`. Cost: an attacker who pads
the attribute past 256 characters falls out of the pattern.
- **The guard excludes the anchor character.** `[^><]`, `[^\]\[]`, `[^)(]`. Cost: content that
legitimately contains that character stops matching.
**Every divergence on the guard's side is documented at source, and traceable to the commit
that introduced it.** An earlier draft of this file claimed the two sub-agent bounds were not;
that was wrong, and it was wrong because the search behind it never looked outside the
CHANGELOG. Both mechanisms are named in `lexicon.py`'s own module docstring:
> **Bounded token gaps** — the two sub-agent patterns whose seed form nested `.*?` are ported
> with `(?:\S+\s+){0,N}?`.
>
> **Anchor exclusion** — […] Measured across all 83 patterns arm by arm, two markdown patterns
> had this defect; both now exclude the anchor character from the run.
`git log -S` separates the two: the eight `[^><]` patterns (six html-obfuscation, two
hybrid-xss) arrived with `cff0437`, *"fix(output): 19 quadratic regex runs on the output path"*
— the v0.3.2 campaign, whose CHANGELOG describes exactly this remedy (*"exclude the character
that opens the pattern's own anchor (`[` for markdown, `<` for tags)"*) across a sweep of
*"150 patterns across 11 tables"*. The `{0,12}` / `{0,120}` sub-agent bounds are older still:
they arrived with the original port commit `f397cd9`, so they were never a divergence
introduced later — they are how that table was transcribed in the first place.
The v0.3.4 entry also states the measured recall cost, naming precisely the two exceptions this
comparison rediscovered: *"URLs containing a literal `(` inside a markdown link target and
comment bodies containing a literal `(` before the keyword."*
Worth recording, because it anticipates the criticism the Node side invites: the guard
considered bounding those runs and **rejected it**, on the grounds that *"the content is
attacker-controlled, so padding past a bound would be a one-line bypass."* That is the same
objection the `{0,256}` witness above demonstrates against the Node table. The two projects
reached opposite conclusions from the same reasoning, which is the substance of the
disagreement — not an oversight on either side.
Neither strategy is free, and neither is obviously right. That is the decision the two owning
repositories have to take, and it is not commons' to take for them.
## Severity: the 8 hybrid patterns
**Resolved 2026-08-09. The two sides never disagreed; only the evidence did.**
Commons recorded the `hybrid` family with `severity: null` and a note that the seed dump did
not supply it, so a consumer **MUST NOT** assume one. The guard's port assigned all eight
`high`. Copying the guard's value would have converted a documented gap into an unverified
claim, so it was reported instead — and the report was right: the value **is** `high`,
confirmed at the module, and `lexicon/injection-lexicon.json` 0.5.0 now carries it. The eight
differences in the table above are closed.
The part worth keeping is where the value lives. It is not a field. The engine assigns it by
pushing `HYBRID_PATTERNS` matches straight into the `high` bucket at
`injection-patterns.mjs:274-281`. The guard's port cites `severity.mjs` — a file that
contains **no injection-family severity at all**. So the guard held the right value behind a
citation that leads nowhere, and a reviewer following that citation to check the number would
have found nothing and drawn no conclusion. Refusing to copy it was the right call for a
reason better than the one given at the time: not merely that a port is second-hand, but that
this particular port could not have read what it claimed to.
## What this does not show
- ~~**Not that either runtime misses an attack.**~~ **Retracted 2026-08-09. It does.** This
bullet claimed that every witness payload still produced a finding via
`active-content: constructs.raw-html`, so no attack went unflagged. The measurement behind
it was wrong in method, not in arithmetic: the payloads were run against the **union of
every pattern table this repository holds** — 111 rules across the lexicon,
`active-content.json` and `secret-egress.json` — and the rescuing hit came from
`active-content.json`. That table is the **Python guard's**. `llm-security` has no
active-content table at all. A union of commons tables is not any single runtime's
coverage, and treating it as one turned two runtimes' combined reach into a claim about
each of them.
Measured properly, through `llm-security`'s own entry point `scanForInjection()` — the
whole engine, with normalisation, homoglyph folding, the rot13 variant and all four pattern
arrays, at `b0de0ca`:
| Witness | `scanForInjection()` result |
| --- | --- |
| `<script>alert(1)` (unclosed) | `found: false` — no finding at all |
| `<script src=x.js>` | `found: false` — no finding at all |
| `<a aria-label="` + 300 × `a` + `ignore">` | `found: false` — no finding at all |
Controls in the same run behave as expected: `<script>alert(1)</script>` returns `high`
(hybrid-xss), and the short aria-label variant returns `critical`. So the `{0,256}` window
is a real evasion window and the `<script>` pattern really does require a closing tag.
`llm-security` reached the same three results independently and attributes the cause to
their own v7.8.3 #24 ReDoS hardening, which traded recall for boundedness without seeing
the window. Three confirmed recall holes, logged there as a v8.x task.
What survives from the original bullet is only this: the divergence is *also* in the
finding set, which is what a `conformance/expected.json` encodes.
- **Not that the guard misses an attack.** The guard's side of the 13 was re-checked the same
way, and the one row that looked like a miss (`hybrid-xss: iframe with executable src`) is
covered by a sibling pattern at the same severity and anchor. Its remaining divergences are
label-set and span differences.
- **Not dump-to-module fidelity.** *Superseded 2026-08-09.* Every check above proves the two
*ports* agree or disagree. Commons' side is now settled separately: the lexicon is verified
byte-identical to `injection-patterns.mjs` at `b0de0ca`, 83/83, which is recorded in the
data file rather than here. The guard's fidelity to the module remains its own to
establish.
- **Not exhaustive.** The corpus is targeted per pattern family, 401 comparisons. Absence of a
witness proves nothing except for the 6 escaping-only pairs, which are settled by string
identity rather than by the corpus.
## Consequence for `conformance/`
An `expected.json` names findings. Naming a finding needs a stable id, and the two runtimes do
not have one: the same pattern is `override:ignore-previous` in the guard and
`override: ignore previous instructions` in the Node table. The guard's JSON happens to carry
both — `id` and `desc` — which is evidence that a commons-owned id is achievable rather than
speculative.
So the id question is a **prerequisite** for the corpus, not a parallel task: until commons
owns a pattern id both ports map to, no fixture can be written, including for the 64 patterns
that are byte-identical. And for the 13 divergent patterns a fixture cannot be authored at all
without first deciding whose recall cost is the contract.
**Resolved for the first half, 2026-08-09 (operator decision).** `lexicon/injection-lexicon.json`
0.2.0 now carries a commons-owned `id` per pattern, plus an `aliases` object naming what each
seeding runtime calls it. The id was **adopted verbatim from the guard's port**, not invented
here — that port already carried both names, so the mapping came from source data. The
detection data is provably unmoved: stripping `id`, `aliases` and the new `pattern_id_space`
block reproduces the previous committed file **byte for byte** (23 566 bytes, identical).
**Ratified by both runtimes, 2026-08-09.** `llm-security` accepted the id space as-is,
including the 0.2.0 proposal, and treats an id change as breaking on the same terms; the
guard confirmed the space its own port supplied. `lexicon/injection-lexicon.json` 0.5.0
records both. The id is a cross-runtime contract now, not a proposal.
~~**The second half of the blocker stands, and it did not get smaller.**~~ **Dissolved
2026-08-10 by measurement, not by a decision.** This paragraph said the 13 divergent
patterns had no agreed expected behaviour, so their fixtures could not be authored, and that
someone would have to pick whose recall cost was the contract.
Nobody had to. The question was never asked of the right inputs. Every divergence in the
table above was found on a **witness** input — an attribute run padded past 256 characters,
an interior `<`, an unclosed `<script>`. The corpus is built from the seed suite's payloads,
which are short, unpadded and contain none of those shapes. Run through both runtimes'
public entry points, all 83 patterns produce **identical lexicon finding sets**, and that
includes 13 of 13 of the divergent ones on their own case input. Method, commits and counts:
[`conformance/manifest.json`](../conformance/manifest.json).
So the 13 carry no marker in the corpus and no caveat. Marking them would assert a doubt the
measurement disproves for these inputs, which is a different defect from the one it would
appear to prevent.
**What still stands is everything above this heading.** The divergence is real, it is
unresolved, and it will reappear the moment a fixture is written over a witness input.
`llm-security` has decided **not** to adopt the guard's regex strategy at this point: v0.1.0
is a behaviour-preservation release on their side too, and swapping strategies mid-vendoring
would void their own golden gate. Both behaviours stay registered as known divergence per
pattern. Their three confirmed recall holes are logged as a v8.x task; when it lands they
will say so, and those rows can close then. Until then the correct description of each is
*"known divergence, `llm-security` side has an open recall hole, measured 2026-08-09; not
reachable from any input in the v0.1.0 corpus"* — not *"undecided"*, and not *"resolved"*.