llm-security/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Versioning note: the repository tag versions the contract (file set, key names, case ids, disposition semantics). Each JSON file additionally carries its own "version" field, bumped when that file changes.

[Unreleased]

[0.2.0] — 2026-08-11

The contract gained a normative MUST, which is why this is a release rather than a metadata commit: a runtime that conformed to v0.1.0 does not conform to this one until it declares the set of commons data files it implements. In 0.x that is a minor bump by the rules, but it is breaking in substance, and a consumer reading only the version number should learn that here rather than from a failing suite.

Everything below this heading was previously listed as unreleased.

Retracted

• The claim that the Python guard's port cites severity.mjs for hybrid severity. It is false. It was carried in three places — lexicon/injection-lexicon.json (families[hybrid].severity_provenance.not_from), docs/lexicon-port-divergence.md (§ Severity: the 8 hybrid patterns), and the [0.1.0] entry below — and it was never measured here. It restated an assertion received from llm-security (coord message 20260809T201048Z) as a commons finding.

  Measured against the guard's own tree, which llm-ingestion-pipeline-security asked for twice before this was checked: severity.mjs has never appeared in src/llm_ingestion_guard/injection_lexicon.json at any point in that file's history (git log -S returns no commits), and at 0bf0729 — the commit conformance/manifest.json pins — the only tree-wide occurrence is docs/PLAN.md:114, correctly attributing the report module to output.mjs + severity.mjs. The guard's only source statement for the lexicon is the note at injection_lexicon.json:3, and it names injection-patterns.mjs.

  No detection data moves. families[hybrid].severity is still high, still sourced to injection-patterns.mjs:274-281, re-verified at b0de0ca; severity.mjs still contains zero occurrences of CRITICAL_PATTERNS, HIGH_PATTERNS, MEDIUM_PATTERNS and HYBRID_PATTERNS, re-measured the same day. Only the sentence about the other repository falls.

  The retraction is marked in place rather than edited away, and it is worth naming why this one survived review: the claim arrived bundled with a correct measurement of the same question, from a repository that had done its half properly. The correct half carried the incorrect half past the check — which is precisely the defect severity_provenance.not_from was written to warn about, one level up.

Added

• not-applicable, a third conformance verdict (spec/conformance-corpus.md §1.1). A runtime now declares the set of commons data files it implements; a case whose scope names a file outside that set is not-applicable rather than failed. §1 alone would have reported an architectural difference as a defect — one seeding runtime has no active-content table and never will, and 7 permanent failures say nothing a reader can use. The verdict is fenced so it cannot become an exit: it attaches to a table, never to a case (per-case opt-out is the silent skip §1 forbids), a declared set MUST NOT be narrowed to convert failures into not-applicable, and such cases MUST still be enumerated rather than dropped from the denominator. §8 now states the consequence: a pass count is unreadable without the declared set beside it.

• Six active-content conformance cases — active__markdown-image, active__markdown-link, active__reference-link, active__autolink, active__raw-html, active__data-uri. The corpus goes 83 → 89, and scope_covered gains signatures/active-content.json.

  Generated from measurement, not written: payload strings were extracted from the seed runtime's coverage.py by AST — evaluating each _scan_case argument in that module's own namespace rather than retyping detection data — then run through its public output gate. The fixtures were then re-read from disk by a separate checker that re-computed every digest, re-scanned the bytes and applied exact-within-scope independently of the generator, because a generator agreeing with itself proves nothing: 6 cases, 0 failed checks.

  These six prove less than the 83, and the manifest says so. Their payloads come from the only runtime implementing the table, so no second implementation's agreement could be measured. They pin one runtime's behaviour as a contract a future implementer can be held to — which is less than cross-runtime agreement and more than nothing.

• signatures/active-content.json 0.1.0 → 0.2.0 — a pattern_id_space block. Unlike the lexicon's, nothing was constructed: label_format and the constructs keys were already extracted verbatim, and their concatenation is what the seed runtime emits. The block states an id space the file already had implicitly, and records that it is ratified by one runtime rather than two.

Changed

• lexicon/injection-lexicon.json 0.5.0 → 0.5.1 — provenance metadata only; no pattern, id, alias, family or severity value changes.

• conformance/manifest.json 0.1.1 → 0.2.0 — the six cases, scope_covered, count_by_scope, separate provenance and measurement blocks for the active-content half (a different source structure at a different commit; one pin must not stand for two measurements), and scope_planned.blockers.

• spec/conformance-corpus.md §4 no longer claims scoping "asks a question both can answer." That held only while every case was scoped to the one table both runtimes implement, and stopped being true the moment a case was scoped to a single-runtime table. Scope narrows what is compared; it does not make every runtime a valid addressee. The superseded sentence is named in place rather than edited away.

Fixed

• §4 now states that "belongs to a data file" means published there, never "shares its prefix." The distinction has a live witness: the seed runtime emits active:oversize-input, a self-safety flag about its own scan cap, which carries the active: prefix but is no construct in signatures/active-content.json. A prefix-matching runtime would pull it into the comparison and fail a case over a finding the corpus makes no claim about. Recorded in that file under pattern_id_space.not_every_active_label as well.

• §6 now states the derivation's cost. case_id derives from pattern_id alone, so a single-finding scope holds at most one case per pattern id — there is nowhere in the name for a second. The seed runtime's matrix drives two payloads at active:markdown-image; measured, their in-scope finding sets are identical, and the second's only distinguishing signal (entropy:base64-blob) falls outside every table this repository publishes. It was dropped rather than given a discriminated id, which would have broken the reverse transform, and it is named in conformance/manifest.json under omitted_payloads so that 6 built from 7 offered reads as a decision rather than a miscount.

Measured, not shipped

• The remaining four cases are blocked on two distinct unresolved questions, now recorded under scope_planned.blockers instead of the earlier blanket "no runtime has agreed to an id space". That framing was wrong for both:

  • Carriers (3). No adoptable id space, and a second problem underneath. The guard emits two stage-coupled labels for one carrier — sanitize:zero-width on input, output:zero-width-present on output, same split for bidi and unicode-tag — while llm-security emits prose titles. A commons id must be invented stage-neutral, which no other id space here required. And since exact-within-scope compares a finding set, an id aliasing both labels makes the verdict depend on which entry point the runtime was measured through — an entry-point dependence the lexicon cases do not have.
  • Secret egress (1). Not an id-naming question at all. The two runtimes carry different tables: 18 entries here against the guard's 25, cut at different granularities (this file's single GitHub Token is four ids there, Private Key PEM Block three, Database connection string four), with membership diverging both ways. aws-access-key-id is the one clean 1:1 — which is why exactly one egress case was ever offered. That number was a symptom, not modesty. A shared id space presupposes a table reconciliation nobody has done.

[0.1.0] — 2026-08-10

Initial extraction. Runtime-neutral detection data, the finding contract, and a conformance corpus, extracted from the llm-security Node implementation and a Python guard without behaviour change — that invariant is the release, not a caveat on it.

What the tag is worth resting on: seven of the eight JSON artefacts were rebuilt from the commons file alone and diffed against their source implementation, three of them against the source module at a pinned commit. The eighth says verified: false about itself. The corpus holds 83 cases on which both seeding runtimes were measured agreeing exactly.

What it is not: spec/decode-pipeline.md does not exist, and the corpus constrains one of the seven data files. Both absences are named in Not included rather than papered over.

Added

• conformance/ — 83 cases, one per injection-lexicon pattern, plus manifest.json. Each case is a directory holding input.txt (the exact bytes, no trailing newline) and expected.json (the findings, named by commons pattern id).

  Both seeding runtimes were measured producing the same lexicon finding set on all 83, through their public entry points — scanForInjection() at b0de0ca and scan_output(source=OUTPUT) at 0bf0729 — with labels mapped to commons ids through the lexicon's own aliases block. Not through rebuilt regex tables: a table-level comparison yields a number that describes neither runtime, which is the mistake the divergence document had to retract.

  The 13 divergent patterns are in, unmarked, and that is the substantive result. Their divergence was measured on witness inputs — an attribute run padded past 256 characters, an interior <, an unclosed <script> — and none of those shapes occurs in a corpus payload. All 13 agree on their own case input. Nobody had to pick whose recall cost becomes the contract, because the question was never reachable from these inputs. A per-case caveat would have asserted a doubt the measurement disproves.

  Inputs are the guard's coverage.py payloads, reproduced verbatim. One runtime authored them; what makes them a cross-runtime corpus is the measurement through the other, and the manifest records the asymmetry rather than averaging it away.

• spec/conformance-corpus.md — normative. How a case is read: input.txt is bytes and is not to be trimmed or re-encoded, expected.json names findings by pattern_id only (severity and OWASP anchor are looked up in the lexicon, never restated), and exact-within-scope requires equality restricted to the data files the case names.

  The field is pattern_id, not id, because this repository already publishes an unrelated finding id: schema/finding.schema.json defines it as DS-<scanner>-<counter> from a process-global counter — stable across neither runs nor processes. Two normative documents using one word for a stable rule identity and a volatile per-emission sequence number would have produced runtimes failing every case for reasons unrelated to detection. §3.1 states the distinction and publishes the bridge a runtime actually needs: its own label maps to a pattern_id through the lexicon's aliases object, and a runtime absent from that object has no published way to be compared at all.

  Scoping is what makes exactness safe — the two runtimes do not implement the same set of tables, so a whole-report comparison would fail for reasons unrelated to the pattern under test. Exactness is what makes the corpus worth running — a contains-only corpus is passed by a runtime that flags everything. observed_out_of_scope is evidence, never expectation, and an absent runtime key means unmeasured, not measured-empty.

  The document also states the one place this repository's "every JSON file carries a top-level version" convention does not apply: fixtures are versioned as a corpus, in conformance/manifest.json. Stated rather than left to be discovered.

• schema/finding.schema.json — the finding contract plus the SARIF output profile. Normative. Closed against the producer in 0.2.0; the JSONL profile is not applicable.

• signatures/active-content.json — the EchoLeak class (CVE-2025-32711): 17 patterns, severities, opacity floors and pass order, from the Python guard.

• lexicon/injection-lexicon.json — 83 prompt-injection patterns in four families (21 critical, 32 high, 22 medium, 8 hybrid).

• codepoints/carriers.json — six carrier tables: zero-width characters, the Unicode Tags block, the Supplementary Private Use Areas, BIDI controls, the Cyrillic presence set and the 28-entry fold-to-Latin homoglyph map.

• signatures/secret-egress.json — the 18 fixed credential and token shapes. Array order is normative.

• mapping/owasp-map.json — four taxonomy maps (LLM, ASI, AST, MCP) over one shared 16-prefix key set.

• calibration/calibration.json — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds.

• signatures/malware-signatures.json — the known-bad-identity table for the SIG class: seven signatures over four families (webshell, reverse_shell, cryptominer, hacktool), reproduced verbatim from knowledge/signatures.json at b0de0ca, key order included, with the source file's byte length and SHA-256 pinned in provenance.

  The rules were the easy half. The file's substance is the line between the table and the engine, drawn in engine_behaviour_not_data: no rule carries a flags field, because the engine compiles every pattern with i unconditionally — so a consumer that compiles these case-sensitively silently under-matches all seven. Each pattern is also run against five decode variants, not just raw bytes; rules are filtered by an enabled-families policy; a rule fires once per file; operator rules are merged at scan time; and the loader defaults four missing fields rather than rejecting a rule. None of that travels with the data, and all of it changes what a consumer sees.

  Two honesty notes are in evidence_limits rather than in prose. Seven signatures are not malware coverage — a clean SIG result is not "no malware", and the seed runtime's own header calls the table "deliberately tight". And three of the seven match on names (xmrig, mimikatz, meterpreter), so a document discussing those tools matches; the seed runtime papers over this by excluding knowledge/, tests/, docs/ and node_modules/ from the scan, which is engine behaviour and does not come with the table.

  Verified: 7/7 rule objects field-identical to source including key order, no non-ASCII bytes, and all seven compile in Node bare, i and iu (21/21) and in Python re (7/7). Note the exact family spellings — reverse_shell, not reverse-shell, and cryptominer, not miner; they are policy keys, and the working note that seeded this file had both wrong.

Verification

Every file above except calibration.json was proven rather than transcribed: the data was rebuilt from the commons JSON alone and diffed against the source implementation. Each file records its own result and its own limits.

calibration/calibration.json carries verified: false. Its source arrived as a prose summary rather than as code, so no differential check was possible, and the file names the checks that were not run instead of attaching a caveat to a pass.

The corpus was verified the same way the data was — by a harness that does not share the generator's knowledge. It reads only the case directories, re-runs both runtimes on the bytes it finds there, and checks every field of every expected.json, digests included: 83 cases, 0 failures. Two further checks, because a corpus that cannot fail is not evidence: commons' family severity matches the severity the guard emits per finding, 83/83; and deleting the middle third of each input breaks 76 of 83 expectations. The 7 survivors are the shortest payloads, where the mutation leaves the trigger intact — that is a weak mutation, not a weak fixture, and it is recorded as such rather than rounded up.

• docs/lexicon-port-divergence.md — informative. A differential comparison of the two ports of injection-patterns.mjs (this repository's and the Python guard's): 83/83 patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven equivalent, and 13 behave differently, with a witness input for each and misses on both sides. The cause is two different ReDoS mitigations of one table. No data file was changed — behaviour preservation holds and the finding is reported to the owning repositories.

  Revised 2026-08-09 with one retraction. The document claimed that neither runtime misses an attack, on the grounds that every witness payload still produced a finding. It does miss. That measurement ran the payloads against the union of every pattern table this repository holds, and the rescuing hit came from active-content.json — the Python guard's table. llm-security has no active-content table at all, so a union of commons tables was read as a statement about each runtime separately. Re-measured through llm-security's own scanForInjection() at b0de0ca, all three witness payloads return found: false — no finding whatsoever — while controls in the same run behave normally. Three confirmed recall holes, which llm-security attributes to its v7.8.3 #24 ReDoS hardening and has logged as a v8.x task.

  Also corrected: one of the 13 divergences does not reach report level, because the guard's hybrid-xss:javascript-uri fires on the same witness at the same severity and anchor. The report-level number is 12. And the hybrid severity question that the document reported rather than resolved is now closed — the reported hint was right, the citation behind it was not.

  Revised again 2026-08-10. The document said 13 was the number blocking conformance/, since a fixture names labels. It blocks a fixture written over a witness input, and the corpus contains none — all 13 agree on their own case input. The divergence itself stands unresolved and unchanged; what was wrong was the claim about what it blocked.

  Corrections are marked in place rather than edited away.

Changed

• schema/finding.schema.json 0.1.0 → 0.2.0 — the schema is closed. It was seeded from sarif-formatter.mjs, which consumes findings, so its property list could only ever be a lower bound and additionalProperties had to stay open. The producer is now known — finding() in scanners/lib/output.mjs, line 32 — and it returns an object literal with exactly ten keys and no spread: id, scanner, severity, title, description, file, line, evidence, owasp, recommendation. additionalProperties is false, and the two keys the old schema never knew about (id, evidence) are added.

  id gets its own definition: DS-<prefix>-<counter>, pattern ^DS-[A-Za-z]+-[0-9]{3,}$. The {3,} is deliberate — padStart(3, '0') is a minimum, so a run emitting more than 999 findings produces four digits. The id comes from a process-global counter, so it is stable neither across runs nor across processes, and the definition says so before someone keys on it.

  Nullability is now evidence rather than convention. Five keys are emitted as null rather than omitted (opts.x || null), so a serialised finding always carries all ten. The exception is the four assigned straight from opts: omit description and the key is undefined and vanishes from the JSON. Verified by calling the real producer — ten keys in memory, nine after serialisation.

  owasp is a string, not an array, and not one code. Multiple codes are joined with , . Measured across the seed runtime: 31 distinct values over 157 emission sites, 13 of them multi-code, and four mix taxonomies inside a single value (LLM06, ASI02 and friends) with no discriminator saying which is which. That sharpens the edition problem mapping/owasp-map.json already records, and it has a consequence nobody had written down: sarif-formatter.mjs builds tags: [f.owasp], so a finding anchored to two taxonomies produces one SARIF tag with a comma in it. Nothing filtering on LLM06 will match. Reproduced end to end through the real finding() and toSARIF(), and logged as known_lossiness.owasp-tag-not-split — consumer behaviour in llm-security, not data, so it is reported rather than fixed here.

  The JSONL profile is not applicable, not unspecified — the distinction is the point. unspecified would claim a profile exists and merely has not been written down. No finding-JSONL exists: findings are emitted only inside a single JSON envelope (output.mjs:140). The one module that does write JSONL, audit-trail.mjs, writes audit events under a different schema — where owasp is an array. Same field name, different type, same repository. A consumer reading both through one code path will be wrong about one of them, so the profile records the trap instead of leaving a TODO.

  Verified: the schema is valid Draft 2020-12, every finding built by the real producer validates against it, and four negative controls (extra property, missing id, malformed id, unknown severity) are all rejected.

  One new open question, unpatched by design: the producer's JSDoc lists seventeen scanner prefixes including IDE, while all four maps in mapping/owasp-map.json are keyed on sixteen without it. An IDE finding has no taxonomy mapping in any map. Adding the key would be inventing detection data.

• lexicon/injection-lexicon.json 0.4.0 → 0.5.0 — the last null in the file is filled and the id space is ratified. Two blockers close, no detection data moves.

  families[hybrid].severity was null, deliberately, because the seed dump did not supply it. It is high — and the interesting part is where that is written. The hybrid family has no severity field anywhere; the engine assigns one by pushing HYBRID_PATTERNS matches straight into the high bucket at injection-patterns.mjs:274-281. Both this repository and the Python guard had first looked in severity.mjs, which contains no injection-family severity at all. The guard's port holds the right value behind that wrong citation, so severity_provenance.not_from records the miss explicitly: a wrong citation to a right value is the harder defect to catch later.

  Correction 2026-08-10 (see Unreleased): the two sentences about the guard's citation are false and were never measured here. The guard's port cites injection-patterns.mjs, the right file. Everything above about severity.mjs containing no injection-family severity, and about where the value actually lives, stands and has been re-measured.

  pattern_id_space.not_yet_confirmed is replaced by ratification. Both seeding runtimes agreed on 2026-08-09 — llm-security ratified the 0.2.0 proposal as-is and treats an id change as breaking on the same terms, and the guard confirmed the space its own port supplied. id is now a cross-runtime contract, which is what conformance/ was waiting on to be able to name a finding.

  alias_evidence.llm_security is sharpened rather than upgraded. All 83 alias strings were confirmed equal to the module's label field, in order — so the alias is certainly the pattern's name in the table. It is still not established that a finding carries it: the producer is output.mjs:finding(), which emits title and has no label key at all. Verified at table level, one level short of where it would matter. Match on id.

• lexicon/injection-lexicon.json 0.3.0 → 0.4.0 — verified against the source module instead of against the dump it was transcribed from, and two false provenance claims retracted. The source is now pinned: b0de0ca on the public remote, imported in Node and compared entry by entry on source, flags and label.

  The result is 83/83 byte-identical to source, which is not what the file previously claimed. It said two patterns had been rewritten from raw code points into \uXXXX escapes; the module already writes them escaped, so nothing had been rewritten. The stored pattern text was right the whole time — only the account of where it came from was wrong. The dump had rendered the module's escapes as the characters they denote, and this repository re-escaped them, arriving at the correct bytes by way of an incorrect story.

  The same inversion ran the other way in multi-lang:french, which carried the class spelled with a raw accented Latin e where the module writes it as the escape \u00e9 inside the same character class. That was the one pattern of 83 not byte-identical to source, and it is corrected. The two spellings are the same regular expression — verified in Node bare and under u, and in Python re, over accented, unaccented, uppercase and non-matching French input, with identical match offsets — so no behaviour moved. No pattern in the file contains a non-ASCII byte now, matching the module, whose regex literals are pure ASCII throughout.

  Structurally: normalisations is now [] with a normalisations_note, matching the convention already used in signatures/secret-egress.json, and a new source_fidelity block carries the counts, the method, the verified class membership, and both retractions in full. Retracted claims are recorded rather than deleted — the earlier equivalence evidence (692 Node comparisons, 236 Python) remains true, it is simply no longer load-bearing.

• lexicon/injection-lexicon.json 0.2.0 → 0.3.0 — the two aliases are no longer presented as equally backed. pattern_id_space.alias_evidence now records each one separately: llm_ingestion_guard is verified (the guard's coverage matrix asserts on that exact string, so it is demonstrably what a guard finding carries), while llm_security is not — it is the pattern table's own name, and the finding producer was never supplied, with the known Node finding shape using title rather than label. Averaging the two into one file-level claim would have repeated the defect this repository corrects per-table elsewhere.

  Also: normalisations[].affects now keys on id with the prose names kept beside it as affects_labels. An internal cross-reference on label was a second identity space inside the file the id was added to unify.

• lexicon/injection-lexicon.json 0.1.0 → 0.2.0 — every pattern gains a commons-owned id and an aliases object naming what each seeding runtime calls it, plus a top-level pattern_id_space block explaining the field. This exists because a conformance/ fixture has to name a finding and the two runtimes do not name the same pattern the same way.

  The id was adopted verbatim from the guard's port, which already carried both names, rather than invented here. Matching was by label ↔ desc with em-dash normalised to hyphen: 83/83, one-to-one, ids unique.

  No detection data moved. Labels, patterns and flags are byte-identical in sequence, no flags key was invented (78 before, 78 after), and stripping the three new fields reproduces the previous committed file byte for byte — 23 566 bytes, identical. All 83 patterns still compile in Node bare and under u (166/166) and in Python re (83/83).

  Neither llm-security nor the guard has ratified this id space yet; both were asked by coord on 2026-08-09, and the file says so rather than implying agreement.

Not included

• spec/decode-pipeline.md — needs the decode implementation. A normative spec inferred from a data dump would be worse than an absent one.
• Conformance for the other four tables. The corpus covers the injection lexicon only. The carrier, active-content and secret-egress tables have 11 convertible cases waiting in the guard's matrix, and no ratified cross-runtime finding id between them — writing those fixtures would mint a contract unilaterally, in the same stroke as the tag. Named in conformance/manifest.json under scope_planned.
• The 29 non-convertible cases of the guard's 134 assert a runtime's API surface — that a Python call raises OKFPathError, that a disposition engine composes two findings a particular way. This repository does not own an API, so those belong to the guard's suite.

spec/decode-pipeline.md is named in the README as planned rather than linked, so nothing in the repository points at a file that does not exist.