Kjell Tore Guttormsen
bbd03f9b52
4641a7b release(0.3.0): a detection pattern changed value — that is new here 1482c0b feat(schema,spec): give the §1.1 MUST a shape, since v0.2.0 shipped it without one 25a2cf9 feat(conformance): the witness case, and the derivation rule that had no room for it c1b2385 fix(lexicon): converge script-tag on its source — re-extraction, not revision 6f79a6e fix(lexicon,docs): the script-tag row reversed — commons is now the sole diverger git-subtree-dir: scanners/commons git-subtree-split: 4641a7b5184047460e3f10038b615a61e7a4ac21
581 lines
37 KiB
Markdown
581 lines
37 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to this project will be documented in this file.
|
|
|
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
|
|
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
|
Versioning note: the repository tag versions **the contract** (file set, key names,
|
|
case ids, disposition semantics). Each JSON file additionally carries its own
|
|
`"version"` field, bumped when that file changes.
|
|
|
|
## [Unreleased]
|
|
|
|
Nothing yet.
|
|
|
|
## [0.3.0] — 2026-08-11
|
|
|
|
**A detection pattern changed value. That has not happened in this repository before, and it
|
|
is the reason this is a release.** `v0.2.0` changed what a runtime must *declare*; this one
|
|
changes what a conforming runtime *finds*. A consumer that vendors `0.3.0` and re-runs its
|
|
suite will see a finding on inputs that produced none under `0.2.0`. In 0.x that is a minor
|
|
bump by the rules; read the first entry below before upgrading, not the version number.
|
|
|
|
The lexicon `0.6.0` entry previously sitting under *Unreleased* is folded in here — it was
|
|
committed but never tagged, and `0.7.0` supersedes its central claim.
|
|
|
|
### Changed
|
|
|
|
- **`lexicon/injection-lexicon.json` (`0.5.1` → `0.7.0`) — `hybrid-xss:script-tag` converged
|
|
on `llm-security`'s current form.**
|
|
|
|
```
|
|
0.6.0 and earlier <script\b[^>]*>[\s\S]*?<\/script> closing tag REQUIRED
|
|
0.7.0 <script\b[^>]*> opening tag only
|
|
```
|
|
|
|
Byte-identical to `llm-security`'s `RegExp.prototype.source` at `b1ba1fb`
|
|
(`scanners/lib/injection-patterns.mjs:170`), verified by compiling both. They dropped the
|
|
closing-tag requirement in `90f576f` (2026-08-10) because it was a recall hole:
|
|
`<script>alert(1)` and `<script src=x.js>` both returned no finding.
|
|
|
|
**This is re-extraction, not revision, and the distinction is the whole justification.**
|
|
This repository's behaviour-preservation invariant forbids commons from *correcting* seed
|
|
data it believes is wrong — that rule stands and was not weakened. It does not forbid
|
|
re-reading the source after the source itself moved and its owner asked. The lexicon's
|
|
declared provenance is `llm-security`'s injection table, and being loadable verbatim by
|
|
that table's owner is the one thing it exists to do; the standing alternative was a
|
|
per-pattern override in `llm-security`'s own loader, i.e. a published core its source
|
|
repository could not load as published.
|
|
|
|
**Direction matters for what it cannot break:** the new form matches a strict superset of
|
|
the old one, so relative to `0.6.0` it can add matches and cannot introduce a false
|
|
negative. The reverse change would not have been adoptable on the same reasoning.
|
|
|
|
Operator decision, 2026-08-11, on `llm-security`'s blocking request. Explicitly **not**
|
|
decided by the 2-of-3 majority across the three ports: a count of implementations is not a
|
|
mandate over detection data, and the provenance argument would hold with the guard on
|
|
either side.
|
|
|
|
Measured collateral: **none.** The full corpus was run under both patterns — 84/84
|
|
lexicon-scoped cases pass under `0.7.0`, and exactly one case's finding set differs between
|
|
the two forms (the new one below). The widening added no finding to any other case's input.
|
|
|
|
`source_fidelity` restructured to keep its numbers coordinate-bearing:
|
|
`patterns_byte_identical_to_source` keeps its key and its value (83) and gains the field it
|
|
was missing, `byte_identical_against_commit: b1ba1fb`. Against the original extraction
|
|
commit `b0de0ca` this file is now 82/83, recorded as `count_against_extraction_commit`.
|
|
`post_extraction_drift` — added in the folded-in `0.6.0` to record the then-open divergence
|
|
— is now marked `status: resolved in 0.7.0 by re-extraction` and carries the before/after
|
|
pattern text, so a consumer diffing against either commit has a coordinate for what it
|
|
finds.
|
|
|
|
- **`conformance/manifest.json` (`0.2.0` → `0.3.0`) — `case_id_derivation` extended with an
|
|
optional variant suffix.**
|
|
|
|
```
|
|
before case_id = <pattern_id, ':' → '__'>
|
|
after case_id = <pattern_id, ':' → '__'> [ '--' <variant-slug> ]
|
|
reverse truncate at first '--', then '__' → ':'
|
|
```
|
|
|
|
No existing case id moved, so this is additive. `--` was measured absent from all 83
|
|
ratified pattern ids and all 89 pre-existing case ids, which keeps the reverse transform
|
|
purely lexical — no lookup against the id list — the property the original one-to-one rule
|
|
was protecting.
|
|
|
|
The `one_case_per_pattern_id` key is **removed**, superseded by
|
|
`case_id_derivation.variant_suffix.supersedes`, which quotes its text. It was documentation
|
|
of the constraint, not data a consumer matches on, but it is called out here because a
|
|
removed key is normally a breaking change in this repository.
|
|
|
|
`omitted_payloads` gains `derivation_ground_withdrawn_in_0_3_0`: the guard's seventh
|
|
active-content payload was omitted on two grounds, and this change retires one of them. The
|
|
other stands, so the payload stays omitted — on one ground instead of two. **It was not
|
|
added back**; that is a separate decision, not a consequence of this one.
|
|
|
|
### Added
|
|
|
|
- **`conformance/hybrid-xss__script-tag--src-no-close/` (89 → 90 cases)** — input
|
|
`<script src=x.js>`, 17 bytes, expecting `hybrid-xss:script-tag`. The regression gate for
|
|
the change above, and the reason the corpus could not previously see it: the existing
|
|
`hybrid-xss__script-tag` input `<script>steal()</script>` matches the pattern under *both*
|
|
forms, so it passes either way. Reverting the pattern to its `0.6.0` form fails this case
|
|
and only this case — mutation-verified in both directions across all 90.
|
|
|
|
**The first case input authored in this repository** rather than reproduced verbatim from a
|
|
runtime's payload set, recorded in the new `authored_payloads` block rather than folded into
|
|
`payload_provenance`, whose value is precisely the claim that its inputs are verbatim
|
|
upstream. That claim stays exactly as strong as it was: 83 of 83. Both witnesses for this
|
|
axis were named by `llm-security` on 2026-08-10; this is the first of the two. Findings
|
|
measured through the guard's public API at `0dce50f` / `0.5.0`, with the existing case's
|
|
committed bytes and digest reproduced by the same harness in the same run as a control.
|
|
|
|
- **`schema/conformance-declaration.schema.json` (`0.1.0`)** — the shape a runtime publishes
|
|
alongside a conformance result, satisfying the §1.1 MUST that `v0.2.0` created and left
|
|
without a form. Requested by `llm-security` in those terms (runtime, commit measured,
|
|
implemented file paths) with the stated reason that two runtimes publishing free-form
|
|
declarations makes `83/83 + 6 not-applicable` unparseable by anyone but its author.
|
|
|
|
Carries the two arithmetic invariants §1.1 implies but cannot state unambiguously in prose:
|
|
the four verdict counts MUST sum to the total, and the total MUST equal the corpus case
|
|
count at the commit measured. Requires the enumeration arrays whenever their counts are
|
|
non-zero, which turns §1.1's "MUST still be enumerated" from prose into a schema failure.
|
|
Keeps `error` and `not_applicable` structurally distinct, per §1.1. Records
|
|
`declaration_source` — whether the declared set is derived from the runner's own constant or
|
|
hand-maintained beside it — because only the derived form makes the anti-narrowing fence
|
|
structural. **Deliberately not a gate:** nothing in this repository runs, and no validation
|
|
was asked for. Mutation-tested: the example validates, and five distinct defect classes are
|
|
rejected.
|
|
|
|
- **`spec/conformance-corpus.md` §1.1** — normative pointer to that schema, plus a SHOULD that
|
|
a runtime derive its declared set from the constant its runner uses to accept or reject a
|
|
`scope`, and record which it did.
|
|
|
|
### Fixed
|
|
|
|
- `docs/lexicon-port-divergence.md` — the `hybrid-xss:script-tag` row is closed on the
|
|
closing-tag axis, having reversed twice in three days (guard-diverges → commons-diverges →
|
|
converged). What remains open is the one-byte span difference: the guard excludes `<` from
|
|
its negated class and the other two do not, so `<script <x>` matches commons and
|
|
`llm-security` and not the guard. Measured by compiling all three forms, not reasoned from
|
|
the character classes; neither side has claimed it.
|
|
|
|
## [0.2.0] — 2026-08-11
|
|
|
|
The contract gained a normative MUST, which is why this is a release rather than a
|
|
metadata commit: **a runtime that conformed to `v0.1.0` does not conform to this one until
|
|
it declares the set of commons data files it implements.** In 0.x that is a minor bump by
|
|
the rules, but it is breaking in substance, and a consumer reading only the version number
|
|
should learn that here rather than from a failing suite.
|
|
|
|
Everything below this heading was previously listed as unreleased.
|
|
|
|
### Retracted
|
|
|
|
- **The claim that the Python guard's port cites `severity.mjs` for hybrid severity.** It is
|
|
false. It was carried in three places — `lexicon/injection-lexicon.json`
|
|
(`families[hybrid].severity_provenance.not_from`), `docs/lexicon-port-divergence.md`
|
|
(§ *Severity: the 8 hybrid patterns*), and the `[0.1.0]` entry below — and it was never
|
|
measured here. It restated an assertion received from `llm-security` (coord message
|
|
`20260809T201048Z`) as a commons finding.
|
|
|
|
Measured against the guard's own tree, which `llm-ingestion-pipeline-security` asked for
|
|
twice before this was checked: `severity.mjs` has **never** appeared in
|
|
`src/llm_ingestion_guard/injection_lexicon.json` at any point in that file's history
|
|
(`git log -S` returns no commits), and at `0bf0729` — the commit
|
|
`conformance/manifest.json` pins — the only tree-wide occurrence is `docs/PLAN.md:114`,
|
|
correctly attributing the *report* module to `output.mjs` + `severity.mjs`. The guard's
|
|
only source statement for the lexicon is the `note` at `injection_lexicon.json:3`, and it
|
|
names `injection-patterns.mjs`.
|
|
|
|
**No detection data moves.** `families[hybrid].severity` is still `high`, still sourced to
|
|
`injection-patterns.mjs:274-281`, re-verified at `b0de0ca`; `severity.mjs` still contains
|
|
zero occurrences of `CRITICAL_PATTERNS`, `HIGH_PATTERNS`, `MEDIUM_PATTERNS` and
|
|
`HYBRID_PATTERNS`, re-measured the same day. Only the sentence about the *other* repository
|
|
falls.
|
|
|
|
The retraction is marked in place rather than edited away, and it is worth naming why this
|
|
one survived review: the claim arrived bundled with a correct measurement of the same
|
|
question, from a repository that had done its half properly. The correct half carried the
|
|
incorrect half past the check — which is precisely the defect
|
|
`severity_provenance.not_from` was written to warn about, one level up.
|
|
|
|
### Added
|
|
|
|
- **`not-applicable`, a third conformance verdict** (`spec/conformance-corpus.md` §1.1). A
|
|
runtime now declares the set of commons data files it implements; a case whose `scope`
|
|
names a file outside that set is `not-applicable` rather than failed. §1 alone would have
|
|
reported an architectural difference as a defect — one seeding runtime has no
|
|
active-content table and never will, and 7 permanent failures say nothing a reader can use.
|
|
The verdict is fenced so it cannot become an exit: it attaches to a **table**, never to a
|
|
case (per-case opt-out is the silent skip §1 forbids), a declared set MUST NOT be narrowed
|
|
to convert failures into `not-applicable`, and such cases MUST still be enumerated rather
|
|
than dropped from the denominator. §8 now states the consequence: a pass count is
|
|
unreadable without the declared set beside it.
|
|
|
|
- **Six active-content conformance cases** — `active__markdown-image`, `active__markdown-link`,
|
|
`active__reference-link`, `active__autolink`, `active__raw-html`, `active__data-uri`. The
|
|
corpus goes 83 → 89, and `scope_covered` gains `signatures/active-content.json`.
|
|
|
|
Generated from measurement, not written: payload strings were extracted from the seed
|
|
runtime's `coverage.py` by AST — evaluating each `_scan_case` argument in that module's own
|
|
namespace rather than retyping detection data — then run through its public output gate.
|
|
The fixtures were then re-read from disk by a separate checker that re-computed every
|
|
digest, re-scanned the bytes and applied `exact-within-scope` independently of the
|
|
generator, because a generator agreeing with itself proves nothing: 6 cases, 0 failed
|
|
checks.
|
|
|
|
**These six prove less than the 83, and the manifest says so.** Their payloads come from
|
|
the only runtime implementing the table, so no second implementation's agreement could be
|
|
measured. They pin one runtime's behaviour as a contract a future implementer can be held
|
|
to — which is less than cross-runtime agreement and more than nothing.
|
|
|
|
- `signatures/active-content.json` **0.1.0 → 0.2.0** — a `pattern_id_space` block. Unlike the
|
|
lexicon's, nothing was constructed: `label_format` and the `constructs` keys were already
|
|
extracted verbatim, and their concatenation *is* what the seed runtime emits. The block
|
|
states an id space the file already had implicitly, and records that it is ratified by
|
|
**one** runtime rather than two.
|
|
|
|
### Changed
|
|
|
|
- `lexicon/injection-lexicon.json` **0.5.0 → 0.5.1** — provenance metadata only; no pattern,
|
|
id, alias, family or severity value changes.
|
|
|
|
- `conformance/manifest.json` **0.1.1 → 0.2.0** — the six cases, `scope_covered`,
|
|
`count_by_scope`, separate provenance and measurement blocks for the active-content half
|
|
(a different source structure at a different commit; one pin must not stand for two
|
|
measurements), and `scope_planned.blockers`.
|
|
|
|
- **`spec/conformance-corpus.md` §4 no longer claims scoping "asks a question both can
|
|
answer."** That held only while every case was scoped to the one table both runtimes
|
|
implement, and stopped being true the moment a case was scoped to a single-runtime table.
|
|
Scope narrows *what* is compared; it does not make every runtime a valid addressee. The
|
|
superseded sentence is named in place rather than edited away.
|
|
|
|
### Fixed
|
|
|
|
- **§4 now states that "belongs to a data file" means published there, never "shares its
|
|
prefix."** The distinction has a live witness: the seed runtime emits `active:oversize-input`,
|
|
a self-safety flag about its own scan cap, which carries the `active:` prefix but is no
|
|
construct in `signatures/active-content.json`. A prefix-matching runtime would pull it into
|
|
the comparison and fail a case over a finding the corpus makes no claim about. Recorded in
|
|
that file under `pattern_id_space.not_every_active_label` as well.
|
|
|
|
- **§6 now states the derivation's cost.** `case_id` derives from `pattern_id` alone, so a
|
|
single-finding scope holds at most one case per pattern id — there is nowhere in the name
|
|
for a second. The seed runtime's matrix drives *two* payloads at `active:markdown-image`;
|
|
measured, their in-scope finding sets are identical, and the second's only distinguishing
|
|
signal (`entropy:base64-blob`) falls outside every table this repository publishes. It was
|
|
dropped rather than given a discriminated id, which would have broken the reverse
|
|
transform, and it is named in `conformance/manifest.json` under `omitted_payloads` so that
|
|
6 built from 7 offered reads as a decision rather than a miscount.
|
|
|
|
### Measured, not shipped
|
|
|
|
- **The remaining four cases are blocked on two distinct unresolved questions**, now recorded
|
|
under `scope_planned.blockers` instead of the earlier blanket "no runtime has agreed to an
|
|
id space". That framing was wrong for both:
|
|
|
|
- **Carriers (3).** No adoptable id space, and a second problem underneath. The guard emits
|
|
two *stage-coupled* labels for one carrier — `sanitize:zero-width` on input,
|
|
`output:zero-width-present` on output, same split for bidi and unicode-tag — while
|
|
llm-security emits prose titles. A commons id must be invented stage-neutral, which no
|
|
other id space here required. And since `exact-within-scope` compares a finding *set*, an
|
|
id aliasing both labels makes the verdict depend on which entry point the runtime was
|
|
measured through — an entry-point dependence the lexicon cases do not have.
|
|
- **Secret egress (1).** Not an id-naming question at all. The two runtimes carry
|
|
**different tables**: 18 entries here against the guard's 25, cut at different
|
|
granularities (this file's single `GitHub Token` is four ids there, `Private Key PEM
|
|
Block` three, `Database connection string` four), with membership diverging both ways.
|
|
`aws-access-key-id` is the one clean 1:1 — which is why exactly one egress case was ever
|
|
offered. That number was a symptom, not modesty. A shared id space presupposes a table
|
|
reconciliation nobody has done.
|
|
|
|
## [0.1.0] — 2026-08-10
|
|
|
|
Initial extraction. Runtime-neutral detection data, the finding contract, and a conformance
|
|
corpus, extracted from the `llm-security` Node implementation and a Python guard **without
|
|
behaviour change** — that invariant is the release, not a caveat on it.
|
|
|
|
What the tag is worth resting on: seven of the eight JSON artefacts were rebuilt from the
|
|
commons file alone and diffed against their source implementation, three of them against the
|
|
source module at a pinned commit. The eighth says `verified: false` about itself. The corpus
|
|
holds 83 cases on which both seeding runtimes were measured agreeing exactly.
|
|
|
|
What it is not: `spec/decode-pipeline.md` does not exist, and the corpus constrains one of
|
|
the seven data files. Both absences are named in *Not included* rather than papered over.
|
|
|
|
### Added
|
|
|
|
- `conformance/` — **83 cases, one per injection-lexicon pattern**, plus `manifest.json`.
|
|
Each case is a directory holding `input.txt` (the exact bytes, no trailing newline) and
|
|
`expected.json` (the findings, named by commons pattern `id`).
|
|
|
|
Both seeding runtimes were measured producing the **same lexicon finding set on all 83**,
|
|
through their public entry points — `scanForInjection()` at `b0de0ca` and
|
|
`scan_output(source=OUTPUT)` at `0bf0729` — with labels mapped to commons ids through the
|
|
lexicon's own `aliases` block. Not through rebuilt regex tables: a table-level comparison
|
|
yields a number that describes neither runtime, which is the mistake the divergence
|
|
document had to retract.
|
|
|
|
**The 13 divergent patterns are in, unmarked, and that is the substantive result.** Their
|
|
divergence was measured on witness inputs — an attribute run padded past 256 characters,
|
|
an interior `<`, an unclosed `<script>` — and none of those shapes occurs in a corpus
|
|
payload. All 13 agree on their own case input. Nobody had to pick whose recall cost
|
|
becomes the contract, because the question was never reachable from these inputs. A
|
|
per-case caveat would have asserted a doubt the measurement disproves.
|
|
|
|
Inputs are the guard's `coverage.py` payloads, reproduced verbatim. One runtime authored
|
|
them; what makes them a cross-runtime corpus is the measurement through the other, and the
|
|
manifest records the asymmetry rather than averaging it away.
|
|
|
|
- `spec/conformance-corpus.md` — **normative.** How a case is read: `input.txt` is bytes and
|
|
is not to be trimmed or re-encoded, `expected.json` names findings by `pattern_id` only
|
|
(severity and OWASP anchor are looked up in the lexicon, never restated), and
|
|
`exact-within-scope` requires equality **restricted to the data files the case names**.
|
|
|
|
The field is `pattern_id`, not `id`, because this repository already publishes an unrelated
|
|
finding `id`: `schema/finding.schema.json` defines it as `DS-<scanner>-<counter>` from a
|
|
process-global counter — stable across neither runs nor processes. Two normative documents
|
|
using one word for a stable rule identity and a volatile per-emission sequence number would
|
|
have produced runtimes failing every case for reasons unrelated to detection. §3.1 states
|
|
the distinction and publishes the bridge a runtime actually needs: its own label maps to a
|
|
`pattern_id` through the lexicon's `aliases` object, and a runtime absent from that object
|
|
has no published way to be compared at all.
|
|
|
|
Scoping is what makes exactness safe — the two runtimes do not implement the same set of
|
|
tables, so a whole-report comparison would fail for reasons unrelated to the pattern under
|
|
test. Exactness is what makes the corpus worth running — a contains-only corpus is passed
|
|
by a runtime that flags everything. `observed_out_of_scope` is evidence, never expectation,
|
|
and an absent runtime key means **unmeasured**, not measured-empty.
|
|
|
|
The document also states the one place this repository's "every JSON file carries a
|
|
top-level `version`" convention does not apply: fixtures are versioned as a corpus, in
|
|
`conformance/manifest.json`. Stated rather than left to be discovered.
|
|
|
|
- `schema/finding.schema.json` — the finding contract plus the SARIF output profile.
|
|
Normative. Closed against the producer in 0.2.0; the JSONL profile is `not applicable`.
|
|
- `signatures/active-content.json` — the EchoLeak class (CVE-2025-32711): 17 patterns,
|
|
severities, opacity floors and pass order, from the Python guard.
|
|
- `lexicon/injection-lexicon.json` — 83 prompt-injection patterns in four families
|
|
(21 critical, 32 high, 22 medium, 8 hybrid).
|
|
- `codepoints/carriers.json` — six carrier tables: zero-width characters, the Unicode Tags
|
|
block, the Supplementary Private Use Areas, BIDI controls, the Cyrillic presence set and
|
|
the 28-entry fold-to-Latin homoglyph map.
|
|
- `signatures/secret-egress.json` — the 18 fixed credential and token shapes. Array order
|
|
is normative.
|
|
- `mapping/owasp-map.json` — four taxonomy maps (LLM, ASI, AST, MCP) over one shared
|
|
16-prefix key set.
|
|
- `calibration/calibration.json` — risk-score tier constants, verdict thresholds, risk-band
|
|
cutoffs, posture grade thresholds.
|
|
- `signatures/malware-signatures.json` — the known-bad-identity table for the `SIG` class:
|
|
seven signatures over four families (`webshell`, `reverse_shell`, `cryptominer`,
|
|
`hacktool`), reproduced verbatim from `knowledge/signatures.json` at `b0de0ca`, key order
|
|
included, with the source file's byte length and SHA-256 pinned in `provenance`.
|
|
|
|
The rules were the easy half. The file's substance is the line between the table and the
|
|
engine, drawn in `engine_behaviour_not_data`: **no rule carries a `flags` field**, because
|
|
the engine compiles every pattern with `i` unconditionally — so a consumer that compiles
|
|
these case-sensitively silently under-matches all seven. Each pattern is also run against
|
|
five decode variants, not just raw bytes; rules are filtered by an enabled-families policy;
|
|
a rule fires once per file; operator rules are merged at scan time; and the loader defaults
|
|
four missing fields rather than rejecting a rule. None of that travels with the data, and
|
|
all of it changes what a consumer sees.
|
|
|
|
Two honesty notes are in `evidence_limits` rather than in prose. Seven signatures are not
|
|
malware coverage — a clean `SIG` result is not "no malware", and the seed runtime's own
|
|
header calls the table "deliberately tight". And three of the seven match on **names**
|
|
(`xmrig`, `mimikatz`, `meterpreter`), so a document *discussing* those tools matches; the
|
|
seed runtime papers over this by excluding `knowledge/`, `tests/`, `docs/` and
|
|
`node_modules/` from the scan, which is engine behaviour and does not come with the table.
|
|
|
|
Verified: 7/7 rule objects field-identical to source including key order, no non-ASCII
|
|
bytes, and all seven compile in Node bare, `i` and `iu` (21/21) and in Python `re` (7/7).
|
|
Note the exact family spellings — `reverse_shell`, not `reverse-shell`, and `cryptominer`,
|
|
not `miner`; they are policy keys, and the working note that seeded this file had both wrong.
|
|
|
|
### Verification
|
|
|
|
Every file above except `calibration.json` was proven rather than transcribed: the data was
|
|
rebuilt **from the commons JSON alone** and diffed against the source implementation. Each
|
|
file records its own result and its own limits.
|
|
|
|
`calibration/calibration.json` carries `verified: false`. Its source arrived as a prose
|
|
summary rather than as code, so no differential check was possible, and the file names the
|
|
checks that were not run instead of attaching a caveat to a pass.
|
|
|
|
The corpus was verified the same way the data was — by a harness that does **not** share the
|
|
generator's knowledge. It reads only the case directories, re-runs both runtimes on the bytes
|
|
it finds there, and checks every field of every `expected.json`, digests included: **83
|
|
cases, 0 failures**. Two further checks, because a corpus that cannot fail is not evidence:
|
|
commons' family severity matches the severity the guard emits per finding, **83/83**; and
|
|
deleting the middle third of each input breaks **76 of 83** expectations. The 7 survivors are
|
|
the shortest payloads, where the mutation leaves the trigger intact — that is a weak
|
|
mutation, not a weak fixture, and it is recorded as such rather than rounded up.
|
|
|
|
- `docs/lexicon-port-divergence.md` — informative. A differential comparison of the two
|
|
ports of `injection-patterns.mjs` (this repository's and the Python guard's): 83/83
|
|
patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven
|
|
equivalent, and **13 behave differently**, with a witness input for each and misses on
|
|
both sides. The cause is two different ReDoS mitigations of one table. **No data file was
|
|
changed** — behaviour preservation holds and the finding is reported to the owning
|
|
repositories.
|
|
|
|
**Revised 2026-08-09 with one retraction.** The document claimed that *neither runtime
|
|
misses an attack*, on the grounds that every witness payload still produced a finding. It
|
|
does miss. That measurement ran the payloads against the **union of every pattern table
|
|
this repository holds**, and the rescuing hit came from `active-content.json` — the Python
|
|
guard's table. `llm-security` has no active-content table at all, so a union of commons
|
|
tables was read as a statement about each runtime separately. Re-measured through
|
|
`llm-security`'s own `scanForInjection()` at `b0de0ca`, all three witness payloads return
|
|
**`found: false`** — no finding whatsoever — while controls in the same run behave
|
|
normally. Three confirmed recall holes, which `llm-security` attributes to its v7.8.3 #24
|
|
ReDoS hardening and has logged as a v8.x task.
|
|
|
|
Also corrected: one of the 13 divergences does not reach report level, because the guard's
|
|
`hybrid-xss:javascript-uri` fires on the same witness at the same severity and anchor. The
|
|
report-level number is **12**. And the `hybrid` severity question that the document reported
|
|
rather than resolved is now closed — the reported hint was right, the citation behind it was
|
|
not.
|
|
|
|
**Revised again 2026-08-10.** The document said 13 was the number blocking `conformance/`,
|
|
since a fixture names labels. It blocks a fixture written over a **witness** input, and the
|
|
corpus contains none — all 13 agree on their own case input. The divergence itself stands
|
|
unresolved and unchanged; what was wrong was the claim about what it blocked.
|
|
|
|
Corrections are marked in place rather than edited away.
|
|
|
|
### Changed
|
|
|
|
- `schema/finding.schema.json` **0.1.0 → 0.2.0** — the schema is **closed**. It was seeded
|
|
from `sarif-formatter.mjs`, which *consumes* findings, so its property list could only ever
|
|
be a lower bound and `additionalProperties` had to stay open. The producer is now known —
|
|
`finding()` in `scanners/lib/output.mjs`, line 32 — and it returns an object literal with
|
|
**exactly ten keys and no spread**: `id`, `scanner`, `severity`, `title`, `description`,
|
|
`file`, `line`, `evidence`, `owasp`, `recommendation`. `additionalProperties` is `false`,
|
|
and the two keys the old schema never knew about (`id`, `evidence`) are added.
|
|
|
|
`id` gets its own definition: `DS-<prefix>-<counter>`, pattern `^DS-[A-Za-z]+-[0-9]{3,}$`.
|
|
The `{3,}` is deliberate — `padStart(3, '0')` is a minimum, so a run emitting more than 999
|
|
findings produces four digits. The id comes from a process-global counter, so it is stable
|
|
neither across runs nor across processes, and the definition says so before someone keys on it.
|
|
|
|
Nullability is now evidence rather than convention. Five keys are emitted as `null` rather
|
|
than omitted (`opts.x || null`), so a serialised finding always carries all ten. The
|
|
exception is the four assigned straight from `opts`: omit `description` and the key is
|
|
`undefined` and vanishes from the JSON. Verified by calling the real producer — ten keys in
|
|
memory, nine after serialisation.
|
|
|
|
**`owasp` is a string, not an array, and not one code.** Multiple codes are joined with
|
|
`, `. Measured across the seed runtime: 31 distinct values over 157 emission sites, 13 of
|
|
them multi-code, and **four mix taxonomies inside a single value** (`LLM06, ASI02` and
|
|
friends) with no discriminator saying which is which. That sharpens the edition problem
|
|
`mapping/owasp-map.json` already records, and it has a consequence nobody had written down:
|
|
`sarif-formatter.mjs` builds `tags: [f.owasp]`, so a finding anchored to two taxonomies
|
|
produces **one** SARIF tag with a comma in it. Nothing filtering on `LLM06` will match.
|
|
Reproduced end to end through the real `finding()` and `toSARIF()`, and logged as
|
|
`known_lossiness.owasp-tag-not-split` — consumer behaviour in `llm-security`, not data, so
|
|
it is reported rather than fixed here.
|
|
|
|
The **JSONL profile is `not applicable`, not `unspecified`** — the distinction is the point.
|
|
`unspecified` would claim a profile exists and merely has not been written down. No
|
|
finding-JSONL exists: findings are emitted only inside a single JSON envelope
|
|
(`output.mjs:140`). The one module that does write JSONL, `audit-trail.mjs`, writes *audit
|
|
events* under a different schema — where `owasp` is an **array**. Same field name, different
|
|
type, same repository. A consumer reading both through one code path will be wrong about one
|
|
of them, so the profile records the trap instead of leaving a TODO.
|
|
|
|
Verified: the schema is valid Draft 2020-12, every finding built by the real producer
|
|
validates against it, and four negative controls (extra property, missing `id`, malformed
|
|
`id`, unknown severity) are all rejected.
|
|
|
|
One new open question, unpatched by design: the producer's JSDoc lists **seventeen** scanner
|
|
prefixes including `IDE`, while all four maps in `mapping/owasp-map.json` are keyed on
|
|
**sixteen** without it. An `IDE` finding has no taxonomy mapping in any map. Adding the key
|
|
would be inventing detection data.
|
|
|
|
- `lexicon/injection-lexicon.json` **0.4.0 → 0.5.0** — the last null in the file is filled and
|
|
the id space is ratified. Two blockers close, no detection data moves.
|
|
|
|
`families[hybrid].severity` was `null`, deliberately, because the seed dump did not supply
|
|
it. It is **`high`** — and the interesting part is where that is written. The hybrid family
|
|
has no severity field anywhere; the engine assigns one by pushing `HYBRID_PATTERNS` matches
|
|
straight into the `high` bucket at `injection-patterns.mjs:274-281`. Both this repository
|
|
and the Python guard had first looked in `severity.mjs`, which contains no injection-family
|
|
severity at all. The guard's port holds the right value behind that wrong citation, so
|
|
`severity_provenance.not_from` records the miss explicitly: a wrong citation to a right
|
|
value is the harder defect to catch later.
|
|
|
|
> **Correction 2026-08-10 (see Unreleased):** the two sentences about *the guard's* citation
|
|
> are false and were never measured here. The guard's port cites `injection-patterns.mjs`,
|
|
> the right file. Everything above about `severity.mjs` containing no injection-family
|
|
> severity, and about where the value actually lives, stands and has been re-measured.
|
|
|
|
`pattern_id_space.not_yet_confirmed` is replaced by `ratification`. Both seeding runtimes
|
|
agreed on 2026-08-09 — `llm-security` ratified the 0.2.0 proposal as-is and treats an id
|
|
change as breaking on the same terms, and the guard confirmed the space its own port
|
|
supplied. `id` is now a cross-runtime contract, which is what `conformance/` was waiting
|
|
on to be able to name a finding.
|
|
|
|
`alias_evidence.llm_security` is sharpened rather than upgraded. All 83 alias strings were
|
|
confirmed equal to the module's `label` field, in order — so the alias is certainly the
|
|
pattern's name **in the table**. It is still not established that a finding carries it: the
|
|
producer is `output.mjs:finding()`, which emits `title` and has no `label` key at all.
|
|
Verified at table level, one level short of where it would matter. Match on `id`.
|
|
|
|
- `lexicon/injection-lexicon.json` **0.3.0 → 0.4.0** — verified against the source module
|
|
instead of against the dump it was transcribed from, and **two false provenance claims
|
|
retracted**. The source is now pinned: `b0de0ca` on the public remote, imported in Node
|
|
and compared entry by entry on `source`, `flags` and `label`.
|
|
|
|
The result is **83/83 byte-identical to source**, which is not what the file previously
|
|
claimed. It said two patterns had been rewritten from raw code points into `\uXXXX`
|
|
escapes; the module already writes them escaped, so nothing had been rewritten. The stored
|
|
pattern text was right the whole time — only the account of where it came from was wrong.
|
|
The dump had rendered the module's escapes as the characters they denote, and this
|
|
repository re-escaped them, arriving at the correct bytes by way of an incorrect story.
|
|
|
|
The same inversion ran the other way in `multi-lang:french`, which carried the class
|
|
spelled with a raw accented Latin `e` where the module writes it as the escape
|
|
`\u00e9` inside the same character class.
|
|
That was the one pattern of 83 not byte-identical to source,
|
|
and it is corrected. The two spellings are the same regular expression — verified in Node
|
|
bare and under `u`, and in Python `re`, over accented, unaccented, uppercase and
|
|
non-matching French input, with identical match offsets — so **no behaviour moved**. No
|
|
pattern in the file contains a non-ASCII byte now, matching the module, whose regex
|
|
literals are pure ASCII throughout.
|
|
|
|
Structurally: `normalisations` is now `[]` with a `normalisations_note`, matching the
|
|
convention already used in `signatures/secret-egress.json`, and a new `source_fidelity`
|
|
block carries the counts, the method, the verified class membership, and both retractions
|
|
in full. Retracted claims are recorded rather than deleted — the earlier equivalence
|
|
evidence (692 Node comparisons, 236 Python) remains true, it is simply no longer
|
|
load-bearing.
|
|
|
|
- `lexicon/injection-lexicon.json` **0.2.0 → 0.3.0** — the two aliases are no longer presented
|
|
as equally backed. `pattern_id_space.alias_evidence` now records each one separately:
|
|
`llm_ingestion_guard` is **verified** (the guard's coverage matrix asserts on that exact
|
|
string, so it is demonstrably what a guard finding carries), while `llm_security` is
|
|
**not** — it is the pattern table's own name, and the finding producer was never supplied,
|
|
with the known Node finding shape using `title` rather than `label`. Averaging the two into
|
|
one file-level claim would have repeated the defect this repository corrects per-table
|
|
elsewhere.
|
|
|
|
Also: `normalisations[].affects` now keys on `id` with the prose names kept beside it as
|
|
`affects_labels`. An internal cross-reference on label was a second identity space inside
|
|
the file the id was added to unify.
|
|
|
|
- `lexicon/injection-lexicon.json` **0.1.0 → 0.2.0** — every pattern gains a commons-owned
|
|
`id` and an `aliases` object naming what each seeding runtime calls it, plus a top-level
|
|
`pattern_id_space` block explaining the field. This exists because a `conformance/`
|
|
fixture has to name a finding and the two runtimes do not name the same pattern the same
|
|
way.
|
|
|
|
The id was **adopted verbatim from the guard's port**, which already carried both names,
|
|
rather than invented here. Matching was by `label` ↔ `desc` with em-dash normalised to
|
|
hyphen: 83/83, one-to-one, ids unique.
|
|
|
|
**No detection data moved.** Labels, patterns and flags are byte-identical in sequence,
|
|
no `flags` key was invented (78 before, 78 after), and stripping the three new fields
|
|
reproduces the previous committed file byte for byte — 23 566 bytes, identical. All 83
|
|
patterns still compile in Node bare and under `u` (166/166) and in Python `re` (83/83).
|
|
|
|
Neither `llm-security` nor the guard has ratified this id space yet; both were asked by
|
|
coord on 2026-08-09, and the file says so rather than implying agreement.
|
|
|
|
### Not included
|
|
|
|
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
|
|
from a data dump would be worse than an absent one.
|
|
- **Conformance for the other four tables.** The corpus covers the injection lexicon only.
|
|
The carrier, active-content and secret-egress tables have 11 convertible cases waiting in
|
|
the guard's matrix, and no ratified cross-runtime finding id between them — writing those
|
|
fixtures would mint a contract unilaterally, in the same stroke as the tag. Named in
|
|
`conformance/manifest.json` under `scope_planned`.
|
|
- The 29 non-convertible cases of the guard's 134 assert a runtime's **API surface** — that
|
|
a Python call raises `OKFPathError`, that a disposition engine composes two findings a
|
|
particular way. This repository does not own an API, so those belong to the guard's suite.
|
|
|
|
`spec/decode-pipeline.md` is named in the README as planned rather than linked, so nothing
|
|
in the repository points at a file that does not exist.
|