llm-security-commons/docs/extraction-plan.md
Kjell Tore Guttormsen 93f7bc024f docs: close three honesty gaps the validation pass missed
1. The layout block still described calibration.json as holding "entropy
   floors, scan caps, disposition ranks" — the same never-delivered content
   the seed-source row and the README row were already corrected for. Third
   copy, same defect. Grepped the repo to confirm no fourth: the remaining
   hits either describe the absence or belong to active-content.json's own
   Shannon-entropy floor, which is a real thing from a different source.
   Tightened the calibration log entry so it cannot be read as denying that.

2. Definition of Done item 2 claimed every file is present or "explicitly and
   visibly stubbed". Three are neither — they are absent and marked Planned.
   Item 2 now records that it is unmet, which three files are missing, why
   absent beats stubbed (an empty conformance/ and a contentless normative
   spec would both PASS the mechanical checks in this document while making
   the repo look more finished than it is), and that the tag decision is the
   operator's. The DoD is what a future session reads to decide whether
   v0.1.0 is done; it must not assert a bar the repo does not clear.

3. The lexicon's "83/83 compile in both runtimes" claim had been produced
   against the raw dump, not against the committed JSON — this repo's stated
   standard is that a fidelity claim comes from a command over the artifact.
   Re-run against lexicon/injection-lexicon.json: 83/83 in Python re, 83/83
   in Node bare and under `u`. Also confirmed the \uXXXX escapes survive the
   JSON round-trip with the right semantics — the zero-width class matches
   U+200B and rejects U+00AD, the Cyrillic class matches U+0430 and rejects
   U+0445. The claim stands as written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3
2026-08-09 21:20:06 +02:00

287 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Extraction plan — v0.1.0
**Status: informative.** This is the plan of record for how this repository came to exist,
copied verbatim (structure preserved, lightly reformatted) from the operator brief that
opened it. It is **not** normative: nothing here constrains a consumer. When it disagrees
with `spec/` or `schema/`, those win.
Origin: **Phase 4 of the `llm-security` v8 plan**, which lives in the sibling repository
`llm-security`. That repository is context only — no session in this repository reads from
or writes to it.
## Charter
No engine code. Only: JSON data, normative specs, and a conformance corpus that several
runtimes (Node in `llm-security`, Python in a guard repo, a wiki) can run against and get
an identical verdict from. The pattern is copied from the sibling repository
`portfolio-optimiser-commons` (hard charter: "nothing here may import/depend on a
framework").
## Layout
```
llm-security-commons/
README.md # charter: data+contract+fixtures only, no engine code
lexicon/injection-lexicon.json
codepoints/carriers.json # zero-width, BIDI, Unicode-Tag ranges, homoglyph map
signatures/secret-egress.json
signatures/malware-signatures.json
signatures/active-content.json # EchoLeak: MD image/link/refdef/autolink, data:, active HTML
calibration/calibration.json # risk-score tiers, verdict/band cutoffs, grade thresholds
mapping/owasp-map.json # prefix -> LLM/ASI/AST/MCP
schema/finding.schema.json # + SARIF & JSONL profiles. Status: normative
spec/decode-pipeline.md # normative RFC-2119 decode order
conformance/ # {case}/input.txt + {case}/expected.json
STATE.md # LOCAL-ONLY / gitignored (mirror commons convention)
```
Every JSON file carries a top-level `"version"` field. Every spec carries a
`Status: normative` marker.
## v0.1.0 seed sources
`llm-security` is the canonical and richest source. **This repository's sessions have no
read access to it** — content arrives only as an operator-supplied dump. Security-critical
tables (homoglyph map, secret patterns, malware signatures) MUST come from real source
data, never from recollection or inference.
| Target | Seed source in `llm-security` (unless noted) |
|---|---|
| `lexicon/injection-lexicon.json` | `scanners/lib/injection-patterns.mjs` |
| `codepoints/carriers.json` | `scanners/unicode-scanner.mjs` + `scanners/lib/string-utils.mjs` (incl. `HOMOGLYPH_MAP`) |
| `signatures/secret-egress.json` | `hooks/scripts/pre-edit-secrets.mjs``SECRET_PATTERNS`, the **18-entry hook table**, NOT the PCRE-flavored agent-consumed variant in `knowledge/secrets-patterns.md`. *(Corrected 2026-08-09: this row originally named `knowledge/secrets-patterns.md` as the source file. The delivered dump named `pre-edit-secrets.mjs` and stated explicitly that the two are different files. The row now names the file that was actually delivered.)* |
| `signatures/malware-signatures.json` | `knowledge/signatures.json` (the SIG scanner) |
| `signatures/active-content.json` | currently only in a guard repo's `active_content.py`. If unavailable: stub with a version field and a TODO naming the source |
| `calibration/calibration.json` | `scanners/lib/severity.mjs` — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. *(Corrected 2026-08-09: this row said "thresholds + scanner caps" and the README said "entropy floors, scan caps, disposition ranks". No scan cap, entropy floor or disposition rank was delivered — searched across the whole dump: 0 occurrences each. Both rows now describe what arrived.)* |
| `mapping/owasp-map.json` | `scanners/lib/severity.mjs``OWASP_MAP` (+ 3 sibling maps in the same file) |
| `schema/finding.schema.json` | modelled on `scanners/lib/sarif-formatter.mjs`'s SARIF shape |
| `conformance/` | union of the guard repo's `coverage.py` matrix (126 classes + 4 gaps-must-hold) and `llm-security/examples/` |
## Constraints
- Offline / deterministic only — no network, no model calls inside the data itself.
- Forgejo `open/` — never GitHub.
- MIT license, fork-and-own.
- `STATE.md` is LOCAL-ONLY (gitignored) — same convention as the rest of the polyrepo.
- **Behaviour preservation is the point:** this must not change a single finding in
`llm-security` when it is later consumed from here. That consumption happens in
`llm-security`'s own Phase 5 steps 34 — **not here.**
## Verification log
Every claim of fidelity below was produced by a command, not by reading. The check scripts
themselves deliberately do **not** live in this repository — executable code here would
breach the charter. They are reproducible from the description given.
### `signatures/active-content.json` — extracted 2026-08-09
Source: `llm-ingestion-pipeline-security` v0.3.4, commit `0bf0729` (2026-08-03),
`src/llm_ingestion_guard/active_content.py` + `calibration.py`. Read-only; nothing in that
repository was modified.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed | `python3 -m json.tool` | pass |
| Patterns compile as Python `re` | translate `(?<``(?P<`, compile all 17 with declared flags | 17/17, 0 failures |
| Patterns compile as ECMAScript | `new RegExp(pattern, flags)` on all 17 | 17/17, 0 failures |
| Pattern text matches source | compare against the live `re.Pattern.pattern` of each source object, inline flags stripped | 12/17 byte-identical; 5 differ only by the documented `redundant-quote-escape` normalisation |
| The 5 normalised patterns behave identically | differential match-set comparison (offsets + captured text) against the source objects over a 30-input adversarial corpus: bare quotes, escaped quotes, markdown titles containing quotes, quoted/unquoted HTML attributes, quote runs of length 15 | 150 comparisons, 0 differences |
| The normalisation is necessary | `new RegExp('\\"', 'u')` and `'v'` in Node | both throw `Invalid escape`; the bare form compiles under `""`, `"u"` and `"v"` |
| Severities, ordinary severity, opacity floors, active-tag set, pass order | compare against `calibration.ACTIVE_CONTENT_SEVERITY`, `ACTIVE_CONTENT_ORDINARY_SEVERITY`, `URL_OPAQUE_*`, `active_content._ACTIVE_TAGS`, and the scan-call order in `scan_active_content` | all identical (23/23 tags, 6/6 severities, 4/4 floors) |
Not verified, and not claimed: that the Node consumer's active-content behaviour matches
this table. The source module states the Node port shares its severities; that is the
module's claim, and confirming it needs the Node file.
### `schema/finding.schema.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/sarif-formatter.mjs`, supplied as an operator dump. No
commit hash accompanied it, so provenance is recorded as `unknown` rather than guessed.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed | `python3 -m json.tool` | pass |
| Valid JSON Schema | `jsonschema` `check_schema` against draft 2020-12 | pass |
| Accepts/rejects findings correctly | 2 valid + 3 invalid findings (missing `scanner`, unknown severity, `line: 0`) | 5/5 as intended |
| SARIF profile reproduces the source | re-implemented the mapping **from the commons JSON alone** and diffed `JSON.stringify` against the real `toSARIF` over 10 envelope shapes: empty, missing `scanners`, empty `scanners`, scanner with no findings, all five severities plus an unknown and an `undefined` one, five slug edge cases (double space, tab, newline, leading/trailing space, mixed case), a rule-id collision, all seven optional-field combinations, two scanners, and an explicit `version` argument | 10/10 identical, 0 differences |
| The three `known_lossiness` claims are true | executed each against the real formatter | all three confirmed, **and one earlier claim corrected**: punctuation does *not* collapse — the slug lowercases and collapses whitespace only, so `Zero-width carrier` and `Zero-width carrier!` remain distinct ids. The wrong claim was published in the first draft of this file and fixed before commit. |
Not verified, and recorded in the file as open: the finding **producer** was not supplied, so
the property list is a lower bound; `scanner` and `severity` are required by design rather
than by evidence; and the JSONL profile is left explicitly `unspecified` rather than
invented, because "one finding per line" is inference.
### `lexicon/injection-lexicon.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/injection-patterns.mjs`, supplied as operator dump 2/2
through the local coord mailbox. No commit hash accompanied it, so provenance is recorded
as `unknown` rather than guessed.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present, LF, trailing newline, no raw invisible code points | `python3 -m json.tool` + a byte scan for U+200B/200C/200D/FEFF/00AD and the Tag block | pass, 0 raw invisible code points |
| Pattern text and flags reproduce the source | rebuilt all four arrays **from the commons JSON alone** (`new RegExp(p.pattern, p.flags ?? '')`) and diffed label, `.source` and `.flags` against the imported dump module | 83/83 compared, 0 differences; 81/83 byte-identical, 2 declared-normalised |
| Flags were read mechanically, not by eye | extracted from each literal via `.flags` | critical 15×`i` / 3×`m` / 3 none, high 32×`i`, medium 20×`i` / 2 none, hybrid 8×`i` |
| Every pattern compiles in both runtimes | `new RegExp(src, flags)` and again with `u` in Node; `re.compile` with the equivalent `re.I`/`re.M` in Python | 83/83 in all three modes, 0 failures |
| The 2 normalised patterns behave identically | differential match-set comparison (offsets + matched text) against the source objects, bare and under `u`, over a 208-input adversarial corpus: every class member, the near-misses excluded from each class (U+00AD, U+2060, U+180E, Cyrillic х, the uppercase set, Greek look-alikes), run boundaries, repeats, empty input | 832 comparisons, 0 differences |
| Class membership was counted, not assumed | enumerated the code points inside each character class directly from the dump bytes | zero-width class = 4 (U+200B, U+200C, U+200D, U+FEFF — **not** U+00AD); Cyrillic class = 7 (U+0430, U+0435, U+043E, U+0440, U+0441, U+0456, U+0443) |
| `\/` is portable, not a defect | 9 patterns carry the redundant escape a JS regex literal requires; compiled in Node bare, Node `u`, and Python `re` | accepted by all three — kept byte-identical, recorded as a translation note for engines that reject unknown escapes |
Not verified, and not claimed: that the dump matches the module it was transcribed from.
Every check above proves this JSON agrees with **the dump**; dump-to-module fidelity is
`llm-security`'s assertion, reproducible only in a session with read access to that
repository. The severity the engine assigns to `HYBRID_PATTERNS` was not supplied and is
left `null` rather than inferred from its three sibling arrays.
### `codepoints/carriers.json` — extracted 2026-08-09
Source: `llm-security/scanners/unicode-scanner.mjs` (charset constants) and
`llm-security/scanners/lib/string-utils.mjs` (`HOMOGLYPH_MAP`), supplied as operator dump
2/2 through the local coord mailbox. No commit hash accompanied it.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present, no raw invisible code points | `python3 -m json.tool` + byte scan for zero-width, BIDI and Tag-block characters | pass, 0 raw invisible code points |
| Five of the six tables reproduce the source constants | rebuilt each **from the commons JSON alone** (`parseInt(codepoint.slice(2), 16)`) and diffed against the imported dump module | `ZERO_WIDTH_CHARS` 5/5, `BIDI_CHARS` 9/9, `CYRILLIC_CONFUSABLES` 13/13, tag start/end — 0 differences |
| The homoglyph map reproduces the source, including order | rebuilt the object from the entries array and compared keys, values and the whole object | 28/28 keys, values and insertion order identical |
| The map folds identically | applied NFKC + lookup with both the rebuilt and the source table over 12 inputs (Cyrillic and Greek injection spellings, Norwegian and German orthography, empty) | 0 differences |
| The exclusion rationale in the source comment is true | checked whether any of `帿ŨÆäöüßéèêñç` is a key | 0 touched — ordinary Norwegian and German orthography is not folded |
| Convenience `char` fields agree with their own `codepoint` field | `String.fromCodePoint` round-trip on every entry | 41/41, 0 mismatches |
| Character names are not from recollection | resolved every name through Python `unicodedata` against the Unicode character database | all resolved |
| Table sizes were counted, not quoted | counted from the imported constants | homoglyph map holds **28** entries, not the "~25" the dump's own comment estimates; the counted number is the one recorded |
Not verified, and marked `verified: false` **in the file itself**: the two Supplementary
Private Use Area ranges. They arrived as a source comment with no constant behind them, so
unlike the other five tables there was nothing to import and diff. That asymmetry is
recorded per-table rather than averaged into a single file-level verdict.
Recorded and deliberately **not** reconciled, in `cross_table_notes`: the repository now
holds three overlapping Cyrillic sets and two overlapping zero-width sets, and none agree
exactly. The zero-width carrier table includes U+00AD while the lexicon's pattern class does
not; the lexicon's class contains U+0456 which `CYRILLIC_CONFUSABLES` lacks, and
`CYRILLIC_CONFUSABLES` contains U+0445 which the class lacks; and six of the confusables
have no entry in the fold map. The dump states the presence set and the fold map are
deliberately distinct. The U+0456 / U+0445 divergence is reported to `llm-security` rather
than fixed here.
### `signatures/secret-egress.json` — extracted 2026-08-09
Source: `llm-security/hooks/scripts/pre-edit-secrets.mjs` (`SECRET_PATTERNS`), supplied as
operator dump 2/2 through the local coord mailbox. No commit hash accompanied it.
**The seed-source row above was wrong and has been corrected.** It named
`knowledge/secrets-patterns.md`; the dump named `hooks/scripts/pre-edit-secrets.mjs` and
stated that the two are different tables — the second is PCRE-flavoured and agent-consumed
and stays where it is. Recording a source file that was never delivered is the same defect
class as the lossiness claim corrected in `finding.schema.json`, so it is corrected here in
the same commit as the file it describes.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present, LF, trailing newline | `python3 -m json.tool` + byte scan | pass |
| Pattern text and flags reproduce the source | rebuilt the table **from the commons JSON alone**, sorted by the declared `order`, and diffed name, `.source` and `.flags` against the imported dump module | 18/18, 0 differences, **18/18 byte-identical** — no normalisation needed |
| Every pattern compiles in both runtimes | `new RegExp` bare and under `u` in Node; `re.compile` with `re.I` where declared in Python | 18/18 in all three modes, 0 failures |
| The ordering contract holds, and is not decorative | reproduced first-match labelling from the commons order for a Bearer header containing a JWT and for a bare JWT, against the source table | both labels identical to source: header case → `Authorization header with token`, bare case → `JWT (three-part token)` |
| Reordering is detectable, not silent | ran the same Bearer input through a reversed table | label changes to `JWT (three-part token)` — order is load-bearing, which is why every entry carries an explicit `order` field |
| `order` is contiguous | compared to `range(18)` | 017, no gaps |
Not verified, and not claimed: that the dump matches the module. Not supplied, and therefore
not invented: any severity or per-entry disposition — the source table carries a name and a
pattern and nothing else. Out of scope by the dump's own statement: the runtime
policy-injected custom patterns (entries 19+). A consumer matching only this table matches
**less** than the seed hook does when a policy is loaded.
### `mapping/owasp-map.json` — extracted 2026-08-09
Source: `llm-security/scanners/lib/severity.mjs` (`OWASP_MAP`, `OWASP_AGENTIC_MAP`,
`OWASP_SKILLS_MAP`, `OWASP_MCP_MAP`), supplied as operator dump 2/2 through the local coord
mailbox. No commit hash accompanied it.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present | `python3 -m json.tool` | pass |
| All four maps reproduce the source | rebuilt each exported object **from the commons JSON alone**, iterating the declared prefix list, and compared `JSON.stringify` against the imported dump module | 4/4 identical — keys, order, values and empty arrays |
| Key sets are identical across taxonomies, and counted | compared key order across all four in both the commons file and the source | one shared order, **16** prefixes, in all eight objects |
| Empty arrays survive as arrays | type- and length-checked every prefix the source maps to nothing | `agentic` TRG/AST, `skills` WFL/SIG, `mcp` WFL/TRG/SIG/AST — all still `[]`, none dropped or nulled |
| Code prefixes are homogeneous per taxonomy | extracted the alphabetic prefix of every code with a regex and asserted one per map | LLM, ASI, AST, MCP — no mixed map |
Recorded as an **open question in the file**, because it is the one thing a consumer can get
wrong while matching this map exactly: the dump does not state which *edition* of each
taxonomy the codes belong to. That matters and is not pedantry — OWASP's Top 10 for LLM
Applications was renumbered between editions, and in the 2025 edition `LLM06` is Excessive
Agency, with earlier standalone entries consolidated into other numbers and System Prompt
Leakage and Vector and Embedding Weaknesses added as `LLM07` and `LLM08`
([OWASP project page](https://owasp.org/www-project-top-10-for-large-language-model-applications/),
[2025 edition summary](https://www.gravitee.io/blog/owasp-top-10-for-llm-applications-2025-a-practical-guide)).
A bare `LLM06` therefore does not identify a risk. Two runtimes can reproduce this map
perfectly and still publish reports that disagree about what a finding means. `taxonomy_name`
is left `null` rather than guessed, and the question is reported to `llm-security`.
Not supplied, and therefore not invented: what each scanner prefix means. `UNI`, `ENT`, `PRM`
and the rest are reproduced as opaque keys. Deliberately not inherited: the dump notes that a
fallback map inside one of `llm-security`'s own agent definitions covers only part of the
prefix set — that is drift in the consumer, and the complete 16-prefix set is what moves here.
### `calibration/calibration.json` — transcribed 2026-08-09
Source: `llm-security/scanners/lib/severity.mjs`, delivered as a **prose summary** inside
operator dump 2/2 — not as source code.
**This file is the exception in this repository, and it is marked as such in its own
`verification` block.** Every other data file here was rebuilt from its commons JSON and
diffed against an imported module. There was nothing to import here: the constants arrived
as human-written prose describing the module. The differential check was therefore not run
— not "passed with caveats", not run — and the file records `verified: false` with the
specific checks that were skipped.
| Check | Method | Result |
| --- | --- | --- |
| JSON well-formed, `version` present | `python3 -m json.tool` | pass |
| Risk bands are contiguous and non-overlapping | compared each band's `max + 1` to the next band's `min` across 0100 | contiguous, no gap, no shared value |
| Band boundaries agree with the verdict thresholds | compared the BLOCK and WARNING score triggers to the band lower bounds | BLOCK 65 = Critical band min; WARNING 15 = Medium band min |
| `reachable_minimum` is arithmetic, not a new claim | recomputed `base + min(cap, log2(2) * mult)` independently for all four tiers | 80 / 48 / 20 / 4 — exact, because `log2(2)` is exactly 1 |
| Rebuild-from-commons and diff against source | **not run** — no importable source | — |
| Differential scoring over a corpus | **not run** — the formulas are engine and were not supplied in runnable form | — |
Recorded in the file as `not_supplied`, so the absence is visible rather than inferred: the
README, this plan's seed-source table and this plan's layout block all described *this file*
as holding **entropy floors, scan caps and disposition ranks**. None of the three arrived in
any dump — searched across the entire dump message: `entropy` 0 occurrences, `disposition` 0,
`rank` 0, `floor` 0. All three descriptions have been corrected to name what arrived rather
than what was expected. (`signatures/active-content.json` does carry a Shannon-entropy floor,
extracted from the guard repository; that is a different file with a different source, and is
not what the calibration rows promised.)
The consequence is stated in the file and is worth repeating here, because it inverts this
repository's central rule: for every other file, a consumer that disagrees is wrong. For this
one, until the module is supplied in executable form, a disagreement is not automatically the
consumer's bug.
## Definition of done for v0.1.0
1. Repository initialized, Forgejo remote `open/llm-security-commons`, MIT, `STATE.md`
gitignored.
2. Every file in the layout above present and populated from verified seed data — or
explicitly and visibly stubbed where the source was unavailable.
**Status 2026-08-09: unmet, deliberately, and this is why no tag has been cut.** Six of
the nine data files are present and verified. Three are neither present nor stubbed:
`signatures/malware-signatures.json` (seed data not delivered),
`spec/decode-pipeline.md` (needs the decode implementation; a normative spec inferred
from a data dump would carry a `Status: normative` marker and pass every convention
check while asserting something nobody verified) and `conformance/` (corpus form is an
open decision — the guard's coverage matrix is probe-based, not a static
input/expected table).
They are absent rather than stubbed on purpose: an empty `conformance/` and a
contentless normative spec would both *pass* the mechanical checks in this document
while making the repository look more finished than it is. They are named as **Planned**
in the README instead, and are not linked, so nothing points at a file that does not
exist. Whether v0.1.0 ships without a conformance corpus is an operator decision, not a
session one: the README's own pitch is that consumers can be held to the same answer on
the same input, and a tag with zero cases cannot deliver that.
3. All JSON well-formed, every data file carrying `"version"`, every spec carrying
`Status: normative`.
4. Tagged `v0.1.0` and pushed.
5. A `coord` message sent to `llm-security` announcing that the repository and `v0.1.0`
exist, so Phase 5 step 3 (vendoring) can start from there.