llm-security-commons/docs/extraction-plan.md

Extraction plan — v0.1.0

Status: informative. This is the plan of record for how this repository came to exist, copied verbatim (structure preserved, lightly reformatted) from the operator brief that opened it. It is not normative: nothing here constrains a consumer. When it disagrees with spec/ or schema/, those win.

Origin: Phase 4 of the llm-security v8 plan, which lives in the sibling repository llm-security. That repository is context only — no session in this repository reads from or writes to it.

Charter

No engine code. Only: JSON data, normative specs, and a conformance corpus that several runtimes (Node in llm-security, Python in a guard repo, a wiki) can run against and get an identical verdict from. The pattern is copied from the sibling repository portfolio-optimiser-commons (hard charter: "nothing here may import/depend on a framework").

Layout

llm-security-commons/
  README.md                       # charter: data+contract+fixtures only, no engine code
  lexicon/injection-lexicon.json
  codepoints/carriers.json        # zero-width, BIDI, Unicode-Tag ranges, homoglyph map
  signatures/secret-egress.json
  signatures/malware-signatures.json
  signatures/active-content.json  # EchoLeak: MD image/link/refdef/autolink, data:, active HTML
  calibration/calibration.json    # entropy floors, scan caps, disposition ranks
  mapping/owasp-map.json          # prefix -> LLM/ASI/AST/MCP
  schema/finding.schema.json      # + SARIF & JSONL profiles. Status: normative
  spec/decode-pipeline.md         # normative RFC-2119 decode order
  conformance/                    # {case}/input.txt + {case}/expected.json
  STATE.md                        # LOCAL-ONLY / gitignored (mirror commons convention)

Every JSON file carries a top-level "version" field. Every spec carries a Status: normative marker.

v0.1.0 seed sources

llm-security is the canonical and richest source. This repository's sessions have no read access to it — content arrives only as an operator-supplied dump. Security-critical tables (homoglyph map, secret patterns, malware signatures) MUST come from real source data, never from recollection or inference.

Target	Seed source in llm-security (unless noted)
lexicon/injection-lexicon.json	scanners/lib/injection-patterns.mjs
codepoints/carriers.json	scanners/unicode-scanner.mjs + scanners/lib/string-utils.mjs (incl. HOMOGLYPH_MAP)
signatures/secret-egress.json	knowledge/secrets-patterns.md — the 18-entry hook table, NOT the PCRE-flavored agent-consumed variant
signatures/malware-signatures.json	knowledge/signatures.json (the SIG scanner)
signatures/active-content.json	currently only in a guard repo's active_content.py. If unavailable: stub with a version field and a TODO naming the source
calibration/calibration.json	scanners/lib/severity.mjs — thresholds + scanner caps
mapping/owasp-map.json	scanners/lib/severity.mjs — OWASP_MAP (+ 3 sibling maps in the same file)
schema/finding.schema.json	modelled on scanners/lib/sarif-formatter.mjs's SARIF shape
conformance/	union of the guard repo's coverage.py matrix (126 classes + 4 gaps-must-hold) and llm-security/examples/

Constraints

• Offline / deterministic only — no network, no model calls inside the data itself.
• Forgejo open/ — never GitHub.
• MIT license, fork-and-own.
• STATE.md is LOCAL-ONLY (gitignored) — same convention as the rest of the polyrepo.
• Behaviour preservation is the point: this must not change a single finding in llm-security when it is later consumed from here. That consumption happens in llm-security's own Phase 5 steps 3–4 — not here.

Verification log

Every claim of fidelity below was produced by a command, not by reading. The check scripts themselves deliberately do not live in this repository — executable code here would breach the charter. They are reproducible from the description given.

signatures/active-content.json — extracted 2026-08-09

Source: llm-ingestion-pipeline-security v0.3.4, commit 0bf0729 (2026-08-03), src/llm_ingestion_guard/active_content.py + calibration.py. Read-only; nothing in that repository was modified.

Check	Method	Result
JSON well-formed	python3 -m json.tool	pass
Patterns compile as Python re	translate (?< → (?P<, compile all 17 with declared flags	17/17, 0 failures
Patterns compile as ECMAScript	new RegExp(pattern, flags) on all 17	17/17, 0 failures
Pattern text matches source	compare against the live re.Pattern.pattern of each source object, inline flags stripped	12/17 byte-identical; 5 differ only by the documented redundant-quote-escape normalisation
The 5 normalised patterns behave identically	differential match-set comparison (offsets + captured text) against the source objects over a 30-input adversarial corpus: bare quotes, escaped quotes, markdown titles containing quotes, quoted/unquoted HTML attributes, quote runs of length 1–5	150 comparisons, 0 differences
The normalisation is necessary	new RegExp('\\"', 'u') and 'v' in Node	both throw Invalid escape; the bare form compiles under "", "u" and "v"
Severities, ordinary severity, opacity floors, active-tag set, pass order	compare against calibration.ACTIVE_CONTENT_SEVERITY, ACTIVE_CONTENT_ORDINARY_SEVERITY, URL_OPAQUE_*, active_content._ACTIVE_TAGS, and the scan-call order in scan_active_content	all identical (23/23 tags, 6/6 severities, 4/4 floors)

Not verified, and not claimed: that the Node consumer's active-content behaviour matches this table. The source module states the Node port shares its severities; that is the module's claim, and confirming it needs the Node file.

schema/finding.schema.json — extracted 2026-08-09

Source: llm-security/scanners/lib/sarif-formatter.mjs, supplied as an operator dump. No commit hash accompanied it, so provenance is recorded as unknown rather than guessed.

Check	Method	Result
JSON well-formed	python3 -m json.tool	pass
Valid JSON Schema	jsonschema check_schema against draft 2020-12	pass
Accepts/rejects findings correctly	2 valid + 3 invalid findings (missing scanner, unknown severity, line: 0)	5/5 as intended
SARIF profile reproduces the source	re-implemented the mapping from the commons JSON alone and diffed JSON.stringify against the real toSARIF over 10 envelope shapes: empty, missing scanners, empty scanners, scanner with no findings, all five severities plus an unknown and an undefined one, five slug edge cases (double space, tab, newline, leading/trailing space, mixed case), a rule-id collision, all seven optional-field combinations, two scanners, and an explicit version argument	10/10 identical, 0 differences
The three known_lossiness claims are true	executed each against the real formatter	all three confirmed, and one earlier claim corrected: punctuation does not collapse — the slug lowercases and collapses whitespace only, so Zero-width carrier and Zero-width carrier! remain distinct ids. The wrong claim was published in the first draft of this file and fixed before commit.

Not verified, and recorded in the file as open: the finding producer was not supplied, so the property list is a lower bound; scanner and severity are required by design rather than by evidence; and the JSONL profile is left explicitly unspecified rather than invented, because "one finding per line" is inference.

lexicon/injection-lexicon.json — extracted 2026-08-09

Source: llm-security/scanners/lib/injection-patterns.mjs, supplied as operator dump 2/2 through the local coord mailbox. No commit hash accompanied it, so provenance is recorded as unknown rather than guessed.

Check	Method	Result
JSON well-formed, version present, LF, trailing newline, no raw invisible code points	python3 -m json.tool + a byte scan for U+200B/200C/200D/FEFF/00AD and the Tag block	pass, 0 raw invisible code points
Pattern text and flags reproduce the source	rebuilt all four arrays from the commons JSON alone (new RegExp(p.pattern, p.flags ?? '')) and diffed label, .source and .flags against the imported dump module	83/83 compared, 0 differences; 81/83 byte-identical, 2 declared-normalised
Flags were read mechanically, not by eye	extracted from each literal via .flags	critical 15×i / 3×m / 3 none, high 32×i, medium 20×i / 2 none, hybrid 8×i
Every pattern compiles in both runtimes	new RegExp(src, flags) and again with u in Node; re.compile with the equivalent re.I/re.M in Python	83/83 in all three modes, 0 failures
The 2 normalised patterns behave identically	differential match-set comparison (offsets + matched text) against the source objects, bare and under u, over a 208-input adversarial corpus: every class member, the near-misses excluded from each class (U+00AD, U+2060, U+180E, Cyrillic х, the uppercase set, Greek look-alikes), run boundaries, repeats, empty input	832 comparisons, 0 differences
Class membership was counted, not assumed	enumerated the code points inside each character class directly from the dump bytes	zero-width class = 4 (U+200B, U+200C, U+200D, U+FEFF — not U+00AD); Cyrillic class = 7 (U+0430, U+0435, U+043E, U+0440, U+0441, U+0456, U+0443)
\/ is portable, not a defect	9 patterns carry the redundant escape a JS regex literal requires; compiled in Node bare, Node u, and Python re	accepted by all three — kept byte-identical, recorded as a translation note for engines that reject unknown escapes

Not verified, and not claimed: that the dump matches the module it was transcribed from. Every check above proves this JSON agrees with the dump; dump-to-module fidelity is llm-security's assertion, reproducible only in a session with read access to that repository. The severity the engine assigns to HYBRID_PATTERNS was not supplied and is left null rather than inferred from its three sibling arrays.

Definition of done for v0.1.0

1. Repository initialized, Forgejo remote open/llm-security-commons, MIT, STATE.md gitignored.
2. Every file in the layout above present and populated from verified seed data — or explicitly and visibly stubbed where the source was unavailable.
3. All JSON well-formed, every data file carrying "version", every spec carrying Status: normative.
4. Tagged v0.1.0 and pushed.
5. A coord message sent to llm-security announcing that the repository and v0.1.0 exist, so Phase 5 step 3 (vendoring) can start from there.