llm-security-commons/docs/extraction-plan.md

Extraction plan — v0.1.0

Status: informative. This is the plan of record for how this repository came to exist, copied verbatim (structure preserved, lightly reformatted) from the operator brief that opened it. It is not normative: nothing here constrains a consumer. When it disagrees with spec/ or schema/, those win.

Origin: Phase 4 of the llm-security v8 plan, which lives in the sibling repository llm-security. That repository is context only — no session in this repository reads from or writes to it.

Charter

No engine code. Only: JSON data, normative specs, and a conformance corpus that several runtimes (Node in llm-security, Python in a guard repo, a wiki) can run against and get an identical verdict from. The pattern is copied from the sibling repository portfolio-optimiser-commons (hard charter: "nothing here may import/depend on a framework").

Layout

llm-security-commons/
  README.md                       # charter: data+contract+fixtures only, no engine code
  lexicon/injection-lexicon.json
  codepoints/carriers.json        # zero-width, BIDI, Unicode-Tag ranges, homoglyph map
  signatures/secret-egress.json
  signatures/malware-signatures.json
  signatures/active-content.json  # EchoLeak: MD image/link/refdef/autolink, data:, active HTML
  calibration/calibration.json    # risk-score tiers, verdict/band cutoffs, grade thresholds
  mapping/owasp-map.json          # prefix -> LLM/ASI/AST/MCP
  schema/finding.schema.json      # + SARIF & JSONL profiles. Status: normative
  spec/decode-pipeline.md         # normative RFC-2119 decode order
  conformance/                    # {case}/input.txt + {case}/expected.json
  STATE.md                        # LOCAL-ONLY / gitignored (mirror commons convention)

Every JSON file carries a top-level "version" field. Every spec carries a Status: normative marker.

v0.1.0 seed sources

llm-security is the canonical and richest source. This repository's sessions have no read access to it — content arrives only as an operator-supplied dump. Security-critical tables (homoglyph map, secret patterns, malware signatures) MUST come from real source data, never from recollection or inference.

Target	Seed source in llm-security (unless noted)
lexicon/injection-lexicon.json	scanners/lib/injection-patterns.mjs
codepoints/carriers.json	scanners/unicode-scanner.mjs + scanners/lib/string-utils.mjs (incl. HOMOGLYPH_MAP)
signatures/secret-egress.json	hooks/scripts/pre-edit-secrets.mjs — SECRET_PATTERNS, the 18-entry hook table, NOT the PCRE-flavored agent-consumed variant in knowledge/secrets-patterns.md. (Corrected 2026-08-09: this row originally named knowledge/secrets-patterns.md as the source file. The delivered dump named pre-edit-secrets.mjs and stated explicitly that the two are different files. The row now names the file that was actually delivered.)
signatures/malware-signatures.json	knowledge/signatures.json (the SIG scanner)
signatures/active-content.json	currently only in a guard repo's active_content.py. If unavailable: stub with a version field and a TODO naming the source
calibration/calibration.json	scanners/lib/severity.mjs — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. (Corrected 2026-08-09: this row said "thresholds + scanner caps" and the README said "entropy floors, scan caps, disposition ranks". No scan cap, entropy floor or disposition rank was delivered — searched across the whole dump: 0 occurrences each. Both rows now describe what arrived.)
mapping/owasp-map.json	scanners/lib/severity.mjs — OWASP_MAP (+ 3 sibling maps in the same file)
schema/finding.schema.json	modelled on scanners/lib/sarif-formatter.mjs's SARIF shape
conformance/	union of the guard repo's coverage.py matrix and llm-security/examples/. (Corrected 2026-08-09: this row said "126 classes + 4 gaps-must-hold". Counted from the imported CORE_CASES manifest, the matrix holds 134 cases — 128 caught and 6 gap. See the conformance-source breakdown below.)

Constraints

• Offline / deterministic only — no network, no model calls inside the data itself.
• Forgejo open/ — never GitHub.
• MIT license, fork-and-own.
• STATE.md is LOCAL-ONLY (gitignored) — same convention as the rest of the polyrepo.
• Behaviour preservation is the point: this must not change a single finding in llm-security when it is later consumed from here. That consumption happens in llm-security's own Phase 5 steps 3–4 — not here.

Verification log

Every claim of fidelity below was produced by a command, not by reading. The check scripts themselves deliberately do not live in this repository — executable code here would breach the charter. They are reproducible from the description given.

signatures/active-content.json — extracted 2026-08-09

Source: llm-ingestion-pipeline-security v0.3.4, commit 0bf0729 (2026-08-03), src/llm_ingestion_guard/active_content.py + calibration.py. Read-only; nothing in that repository was modified.

Check	Method	Result
JSON well-formed	python3 -m json.tool	pass
Patterns compile as Python re	translate (?< → (?P<, compile all 17 with declared flags	17/17, 0 failures
Patterns compile as ECMAScript	new RegExp(pattern, flags) on all 17	17/17, 0 failures
Pattern text matches source	compare against the live re.Pattern.pattern of each source object, inline flags stripped	12/17 byte-identical; 5 differ only by the documented redundant-quote-escape normalisation
The 5 normalised patterns behave identically	differential match-set comparison (offsets + captured text) against the source objects over a 30-input adversarial corpus: bare quotes, escaped quotes, markdown titles containing quotes, quoted/unquoted HTML attributes, quote runs of length 1–5	150 comparisons, 0 differences
The normalisation is necessary	new RegExp('\\"', 'u') and 'v' in Node	both throw Invalid escape; the bare form compiles under "", "u" and "v"
Severities, ordinary severity, opacity floors, active-tag set, pass order	compare against calibration.ACTIVE_CONTENT_SEVERITY, ACTIVE_CONTENT_ORDINARY_SEVERITY, URL_OPAQUE_*, active_content._ACTIVE_TAGS, and the scan-call order in scan_active_content	all identical (23/23 tags, 6/6 severities, 4/4 floors)

Not verified, and not claimed: that the Node consumer's active-content behaviour matches this table. The source module states the Node port shares its severities; that is the module's claim, and confirming it needs the Node file.

schema/finding.schema.json — extracted 2026-08-09

Source: llm-security/scanners/lib/sarif-formatter.mjs, supplied as an operator dump. No commit hash accompanied it, so provenance is recorded as unknown rather than guessed.

Check	Method	Result
JSON well-formed	python3 -m json.tool	pass
Valid JSON Schema	jsonschema check_schema against draft 2020-12	pass
Accepts/rejects findings correctly	2 valid + 3 invalid findings (missing scanner, unknown severity, line: 0)	5/5 as intended
SARIF profile reproduces the source	re-implemented the mapping from the commons JSON alone and diffed JSON.stringify against the real toSARIF over 10 envelope shapes: empty, missing scanners, empty scanners, scanner with no findings, all five severities plus an unknown and an undefined one, five slug edge cases (double space, tab, newline, leading/trailing space, mixed case), a rule-id collision, all seven optional-field combinations, two scanners, and an explicit version argument	10/10 identical, 0 differences
The three known_lossiness claims are true	executed each against the real formatter	all three confirmed, and one earlier claim corrected: punctuation does not collapse — the slug lowercases and collapses whitespace only, so Zero-width carrier and Zero-width carrier! remain distinct ids. The wrong claim was published in the first draft of this file and fixed before commit.

Not verified, and recorded in the file as open: the finding producer was not supplied, so the property list is a lower bound; scanner and severity are required by design rather than by evidence; and the JSONL profile is left explicitly unspecified rather than invented, because "one finding per line" is inference.

lexicon/injection-lexicon.json — extracted 2026-08-09

Source: llm-security/scanners/lib/injection-patterns.mjs, supplied as operator dump 2/2 through the local coord mailbox. No commit hash accompanied it, so provenance is recorded as unknown rather than guessed.

Check	Method	Result
JSON well-formed, version present, LF, trailing newline, no raw invisible code points	python3 -m json.tool + a byte scan for U+200B/200C/200D/FEFF/00AD and the Tag block	pass, 0 raw invisible code points
Pattern text and flags reproduce the source	rebuilt all four arrays from the commons JSON alone (new RegExp(p.pattern, p.flags ?? '')) and diffed label, .source and .flags against the imported dump module	83/83 compared, 0 differences; 81/83 byte-identical, 2 declared-normalised
Flags were read mechanically, not by eye	extracted from each literal via .flags	critical 15×i / 3×m / 3 none, high 32×i, medium 20×i / 2 none, hybrid 8×i
Every pattern compiles in both runtimes	new RegExp(src, flags) and again with u in Node; re.compile with the equivalent re.I/re.M in Python	83/83 in all three modes, 0 failures
The 2 normalised patterns behave identically	differential match-set comparison (offsets + matched text) against the source objects, bare and under u, over a 208-input adversarial corpus: every class member, the near-misses excluded from each class (U+00AD, U+2060, U+180E, Cyrillic х, the uppercase set, Greek look-alikes), run boundaries, repeats, empty input	832 comparisons, 0 differences
Class membership was counted, not assumed	enumerated the code points inside each character class directly from the dump bytes	zero-width class = 4 (U+200B, U+200C, U+200D, U+FEFF — not U+00AD); Cyrillic class = 7 (U+0430, U+0435, U+043E, U+0440, U+0441, U+0456, U+0443)
\/ is portable, not a defect	9 patterns carry the redundant escape a JS regex literal requires; compiled in Node bare, Node u, and Python re	accepted by all three — kept byte-identical, recorded as a translation note for engines that reject unknown escapes

Not verified, and not claimed: that the dump matches the module it was transcribed from. Every check above proves this JSON agrees with the dump; dump-to-module fidelity is llm-security's assertion, reproducible only in a session with read access to that repository. The severity the engine assigns to HYBRID_PATTERNS was not supplied and is left null rather than inferred from its three sibling arrays.

codepoints/carriers.json — extracted 2026-08-09

Source: llm-security/scanners/unicode-scanner.mjs (charset constants) and llm-security/scanners/lib/string-utils.mjs (HOMOGLYPH_MAP), supplied as operator dump 2/2 through the local coord mailbox. No commit hash accompanied it.

Check	Method	Result
JSON well-formed, version present, no raw invisible code points	python3 -m json.tool + byte scan for zero-width, BIDI and Tag-block characters	pass, 0 raw invisible code points
Five of the six tables reproduce the source constants	rebuilt each from the commons JSON alone (parseInt(codepoint.slice(2), 16)) and diffed against the imported dump module	ZERO_WIDTH_CHARS 5/5, BIDI_CHARS 9/9, CYRILLIC_CONFUSABLES 13/13, tag start/end — 0 differences
The homoglyph map reproduces the source, including order	rebuilt the object from the entries array and compared keys, values and the whole object	28/28 keys, values and insertion order identical
The map folds identically	applied NFKC + lookup with both the rebuilt and the source table over 12 inputs (Cyrillic and Greek injection spellings, Norwegian and German orthography, empty)	0 differences
The exclusion rationale in the source comment is true	checked whether any of åøæÅØÆäöüßéèêñç is a key	0 touched — ordinary Norwegian and German orthography is not folded
Convenience char fields agree with their own codepoint field	String.fromCodePoint round-trip on every entry	41/41, 0 mismatches
Character names are not from recollection	resolved every name through Python unicodedata against the Unicode character database	all resolved
Table sizes were counted, not quoted	counted from the imported constants	homoglyph map holds 28 entries, not the "~25" the dump's own comment estimates; the counted number is the one recorded

Not verified, and marked verified: false in the file itself: the two Supplementary Private Use Area ranges. They arrived as a source comment with no constant behind them, so unlike the other five tables there was nothing to import and diff. That asymmetry is recorded per-table rather than averaged into a single file-level verdict.

Recorded and deliberately not reconciled, in cross_table_notes: the repository now holds three overlapping Cyrillic sets and two overlapping zero-width sets, and none agree exactly. The zero-width carrier table includes U+00AD while the lexicon's pattern class does not; the lexicon's class contains U+0456 which CYRILLIC_CONFUSABLES lacks, and CYRILLIC_CONFUSABLES contains U+0445 which the class lacks; and six of the confusables have no entry in the fold map. The dump states the presence set and the fold map are deliberately distinct. The U+0456 / U+0445 divergence is reported to llm-security rather than fixed here.

signatures/secret-egress.json — extracted 2026-08-09

Source: llm-security/hooks/scripts/pre-edit-secrets.mjs (SECRET_PATTERNS), supplied as operator dump 2/2 through the local coord mailbox. No commit hash accompanied it.

The seed-source row above was wrong and has been corrected. It named knowledge/secrets-patterns.md; the dump named hooks/scripts/pre-edit-secrets.mjs and stated that the two are different tables — the second is PCRE-flavoured and agent-consumed and stays where it is. Recording a source file that was never delivered is the same defect class as the lossiness claim corrected in finding.schema.json, so it is corrected here in the same commit as the file it describes.

Check	Method	Result
JSON well-formed, version present, LF, trailing newline	python3 -m json.tool + byte scan	pass
Pattern text and flags reproduce the source	rebuilt the table from the commons JSON alone, sorted by the declared order, and diffed name, .source and .flags against the imported dump module	18/18, 0 differences, 18/18 byte-identical — no normalisation needed
Every pattern compiles in both runtimes	new RegExp bare and under u in Node; re.compile with re.I where declared in Python	18/18 in all three modes, 0 failures
The ordering contract holds, and is not decorative	reproduced first-match labelling from the commons order for a Bearer header containing a JWT and for a bare JWT, against the source table	both labels identical to source: header case → Authorization header with token, bare case → JWT (three-part token)
Reordering is detectable, not silent	ran the same Bearer input through a reversed table	label changes to JWT (three-part token) — order is load-bearing, which is why every entry carries an explicit order field
order is contiguous	compared to range(18)	0–17, no gaps

Not verified, and not claimed: that the dump matches the module. Not supplied, and therefore not invented: any severity or per-entry disposition — the source table carries a name and a pattern and nothing else. Out of scope by the dump's own statement: the runtime policy-injected custom patterns (entries 19+). A consumer matching only this table matches less than the seed hook does when a policy is loaded.

mapping/owasp-map.json — extracted 2026-08-09

Source: llm-security/scanners/lib/severity.mjs (OWASP_MAP, OWASP_AGENTIC_MAP, OWASP_SKILLS_MAP, OWASP_MCP_MAP), supplied as operator dump 2/2 through the local coord mailbox. No commit hash accompanied it.

Check	Method	Result
JSON well-formed, version present	python3 -m json.tool	pass
All four maps reproduce the source	rebuilt each exported object from the commons JSON alone, iterating the declared prefix list, and compared JSON.stringify against the imported dump module	4/4 identical — keys, order, values and empty arrays
Key sets are identical across taxonomies, and counted	compared key order across all four in both the commons file and the source	one shared order, 16 prefixes, in all eight objects
Empty arrays survive as arrays	type- and length-checked every prefix the source maps to nothing	agentic TRG/AST, skills WFL/SIG, mcp WFL/TRG/SIG/AST — all still [], none dropped or nulled
Code prefixes are homogeneous per taxonomy	extracted the alphabetic prefix of every code with a regex and asserted one per map	LLM, ASI, AST, MCP — no mixed map

Recorded as an open question in the file, because it is the one thing a consumer can get wrong while matching this map exactly: the dump does not state which edition of each taxonomy the codes belong to. That matters and is not pedantry — OWASP's Top 10 for LLM Applications was renumbered between editions, and in the 2025 edition LLM06 is Excessive Agency, with earlier standalone entries consolidated into other numbers and System Prompt Leakage and Vector and Embedding Weaknesses added as LLM07 and LLM08 (OWASP project page, 2025 edition summary). A bare LLM06 therefore does not identify a risk. Two runtimes can reproduce this map perfectly and still publish reports that disagree about what a finding means. taxonomy_name is left null rather than guessed, and the question is reported to llm-security.

Not supplied, and therefore not invented: what each scanner prefix means. UNI, ENT, PRM and the rest are reproduced as opaque keys. Deliberately not inherited: the dump notes that a fallback map inside one of llm-security's own agent definitions covers only part of the prefix set — that is drift in the consumer, and the complete 16-prefix set is what moves here.

calibration/calibration.json — transcribed 2026-08-09

Source: llm-security/scanners/lib/severity.mjs, delivered as a prose summary inside operator dump 2/2 — not as source code.

This file is the exception in this repository, and it is marked as such in its own verification block. Every other data file here was rebuilt from its commons JSON and diffed against an imported module. There was nothing to import here: the constants arrived as human-written prose describing the module. The differential check was therefore not run — not "passed with caveats", not run — and the file records verified: false with the specific checks that were skipped.

Check	Method	Result
JSON well-formed, version present	python3 -m json.tool	pass
Risk bands are contiguous and non-overlapping	compared each band's max + 1 to the next band's min across 0–100	contiguous, no gap, no shared value
Band boundaries agree with the verdict thresholds	compared the BLOCK and WARNING score triggers to the band lower bounds	BLOCK 65 = Critical band min; WARNING 15 = Medium band min
reachable_minimum is arithmetic, not a new claim	recomputed base + min(cap, log2(2) * mult) independently for all four tiers	80 / 48 / 20 / 4 — exact, because log2(2) is exactly 1
Rebuild-from-commons and diff against source	not run — no importable source	—
Differential scoring over a corpus	not run — the formulas are engine and were not supplied in runnable form	—

Recorded in the file as not_supplied, so the absence is visible rather than inferred: the README, this plan's seed-source table and this plan's layout block all described this file as holding entropy floors, scan caps and disposition ranks. None of the three arrived in any dump — searched across the entire dump message: entropy 0 occurrences, disposition 0, rank 0, floor 0. All three descriptions have been corrected to name what arrived rather than what was expected. (signatures/active-content.json does carry a Shannon-entropy floor, extracted from the guard repository; that is a different file with a different source, and is not what the calibration rows promised.)

The consequence is stated in the file and is worth repeating here, because it inverts this repository's central rule: for every other file, a consumer that disagrees is wrong. For this one, until the module is supplied in executable form, a disagreement is not automatically the consumer's bug.

Conformance sources — counted 2026-08-09

The guard's coverage.py is readable from a session here (the read boundary covers llm-security only, not the guard repository), so the matrix was imported and counted rather than estimated. CORE_CASES holds 134 cases: 128 caught, 6 gap.

Only part of it can become a static input.txt / expected.json pair, because only part of it is text-in / findings-out:

Probe kind	Count	Convertible?
_scan_case — scan a payload, assert a label	103	yes
_scan_case_okf — scan an OKF document body	2	yes, if the fixture input may be a structured document rather than plain text
_predicate_case — assert an engine-internal predicate (disposition compounds, documented gaps)	13	no
_raise_case — assert the guard raises a typed error (contract asserters, OKF rejecters)	16	no

The 29 non-convertible cases are not a coverage hole in this repository. They exercise a runtime's API surface — that a Python call raises OKFPathError, that a disposition engine composes two findings a particular way. This repository publishes data and fixtures and explicitly does not own an API, so those rows belong to the guard's own suite, the same way the Planned rows in the README are named rather than faked.

Of the 103 _scan_case rows, 83 are one-per-lexicon-pattern. Those are also the rows most affected by the id question below.

Blocking issue found while counting. The guard and the Node table are two ports of one source file and they do not agree: 64 of 83 patterns byte-identical, 6 differing only by escaping, and 13 behaviourally divergent with concrete witness inputs — in both directions. They also disagree about pattern identity itself (override:ignore-previous vs override: ignore previous instructions). Full measurement, method and limits: lexicon-port-divergence.md. Nothing was changed in any data file as a result — behaviour preservation holds, and the divergence is reported to the two owning repositories instead.

Half of it is now cleared (operator decision, 2026-08-09). lexicon/injection-lexicon.json 0.2.0 publishes a commons-owned id per pattern with an aliases object per runtime, adopted verbatim from the guard's port rather than invented. Findings can therefore be named. What still blocks the corpus is the other half: the 13 divergent patterns have no agreed expected behaviour, and a fixture for them would have to pick a winner.

Definition of done for v0.1.0

1. Repository initialized, Forgejo remote open/llm-security-commons, MIT, STATE.md gitignored.

2. Every file in the layout above present and populated from verified seed data — or explicitly and visibly stubbed where the source was unavailable.

   Status 2026-08-09: unmet, deliberately, and this is why no tag has been cut. Six of the nine data files are present and verified. Three are neither present nor stubbed: signatures/malware-signatures.json (seed data not delivered), spec/decode-pipeline.md (needs the decode implementation; a normative spec inferred from a data dump would carry a Status: normative marker and pass every convention check while asserting something nobody verified) and conformance/ (blocked — see Conformance sources above: 105 of the guard's 134 cases are convertible, but a fixture cannot name a finding until commons owns a pattern id both runtimes map to, and 13 patterns behave differently between the two ports).

   They are absent rather than stubbed on purpose: an empty conformance/ and a contentless normative spec would both pass the mechanical checks in this document while making the repository look more finished than it is. They are named as Planned in the README instead, and are not linked, so nothing points at a file that does not exist. Whether v0.1.0 ships without a conformance corpus is an operator decision, not a session one: the README's own pitch is that consumers can be held to the same answer on the same input, and a tag with zero cases cannot deliver that.

3. All JSON well-formed, every data file carrying "version", every spec carrying Status: normative.

4. Tagged v0.1.0 and pushed.

5. A coord message sent to llm-security announcing that the repository and v0.1.0 exist, so Phase 5 step 3 (vendoring) can start from there.