Squashed 'scanners/commons/' changes from 532d70d..4641a7b
4641a7b release(0.3.0): a detection pattern changed value — that is new here 1482c0b feat(schema,spec): give the §1.1 MUST a shape, since v0.2.0 shipped it without one 25a2cf9 feat(conformance): the witness case, and the derivation rule that had no room for it c1b2385 fix(lexicon): converge script-tag on its source — re-extraction, not revision 6f79a6e fix(lexicon,docs): the script-tag row reversed — commons is now the sole diverger git-subtree-dir: scanners/commons git-subtree-split: 4641a7b5184047460e3f10038b615a61e7a4ac21
This commit is contained in:
parent
9b95fc607e
commit
bbd03f9b52
9 changed files with 585 additions and 25 deletions
130
CHANGELOG.md
130
CHANGELOG.md
|
|
@ -11,6 +11,136 @@ case ids, disposition semantics). Each JSON file additionally carries its own
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
Nothing yet.
|
||||||
|
|
||||||
|
## [0.3.0] — 2026-08-11
|
||||||
|
|
||||||
|
**A detection pattern changed value. That has not happened in this repository before, and it
|
||||||
|
is the reason this is a release.** `v0.2.0` changed what a runtime must *declare*; this one
|
||||||
|
changes what a conforming runtime *finds*. A consumer that vendors `0.3.0` and re-runs its
|
||||||
|
suite will see a finding on inputs that produced none under `0.2.0`. In 0.x that is a minor
|
||||||
|
bump by the rules; read the first entry below before upgrading, not the version number.
|
||||||
|
|
||||||
|
The lexicon `0.6.0` entry previously sitting under *Unreleased* is folded in here — it was
|
||||||
|
committed but never tagged, and `0.7.0` supersedes its central claim.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`lexicon/injection-lexicon.json` (`0.5.1` → `0.7.0`) — `hybrid-xss:script-tag` converged
|
||||||
|
on `llm-security`'s current form.**
|
||||||
|
|
||||||
|
```
|
||||||
|
0.6.0 and earlier <script\b[^>]*>[\s\S]*?<\/script> closing tag REQUIRED
|
||||||
|
0.7.0 <script\b[^>]*> opening tag only
|
||||||
|
```
|
||||||
|
|
||||||
|
Byte-identical to `llm-security`'s `RegExp.prototype.source` at `b1ba1fb`
|
||||||
|
(`scanners/lib/injection-patterns.mjs:170`), verified by compiling both. They dropped the
|
||||||
|
closing-tag requirement in `90f576f` (2026-08-10) because it was a recall hole:
|
||||||
|
`<script>alert(1)` and `<script src=x.js>` both returned no finding.
|
||||||
|
|
||||||
|
**This is re-extraction, not revision, and the distinction is the whole justification.**
|
||||||
|
This repository's behaviour-preservation invariant forbids commons from *correcting* seed
|
||||||
|
data it believes is wrong — that rule stands and was not weakened. It does not forbid
|
||||||
|
re-reading the source after the source itself moved and its owner asked. The lexicon's
|
||||||
|
declared provenance is `llm-security`'s injection table, and being loadable verbatim by
|
||||||
|
that table's owner is the one thing it exists to do; the standing alternative was a
|
||||||
|
per-pattern override in `llm-security`'s own loader, i.e. a published core its source
|
||||||
|
repository could not load as published.
|
||||||
|
|
||||||
|
**Direction matters for what it cannot break:** the new form matches a strict superset of
|
||||||
|
the old one, so relative to `0.6.0` it can add matches and cannot introduce a false
|
||||||
|
negative. The reverse change would not have been adoptable on the same reasoning.
|
||||||
|
|
||||||
|
Operator decision, 2026-08-11, on `llm-security`'s blocking request. Explicitly **not**
|
||||||
|
decided by the 2-of-3 majority across the three ports: a count of implementations is not a
|
||||||
|
mandate over detection data, and the provenance argument would hold with the guard on
|
||||||
|
either side.
|
||||||
|
|
||||||
|
Measured collateral: **none.** The full corpus was run under both patterns — 84/84
|
||||||
|
lexicon-scoped cases pass under `0.7.0`, and exactly one case's finding set differs between
|
||||||
|
the two forms (the new one below). The widening added no finding to any other case's input.
|
||||||
|
|
||||||
|
`source_fidelity` restructured to keep its numbers coordinate-bearing:
|
||||||
|
`patterns_byte_identical_to_source` keeps its key and its value (83) and gains the field it
|
||||||
|
was missing, `byte_identical_against_commit: b1ba1fb`. Against the original extraction
|
||||||
|
commit `b0de0ca` this file is now 82/83, recorded as `count_against_extraction_commit`.
|
||||||
|
`post_extraction_drift` — added in the folded-in `0.6.0` to record the then-open divergence
|
||||||
|
— is now marked `status: resolved in 0.7.0 by re-extraction` and carries the before/after
|
||||||
|
pattern text, so a consumer diffing against either commit has a coordinate for what it
|
||||||
|
finds.
|
||||||
|
|
||||||
|
- **`conformance/manifest.json` (`0.2.0` → `0.3.0`) — `case_id_derivation` extended with an
|
||||||
|
optional variant suffix.**
|
||||||
|
|
||||||
|
```
|
||||||
|
before case_id = <pattern_id, ':' → '__'>
|
||||||
|
after case_id = <pattern_id, ':' → '__'> [ '--' <variant-slug> ]
|
||||||
|
reverse truncate at first '--', then '__' → ':'
|
||||||
|
```
|
||||||
|
|
||||||
|
No existing case id moved, so this is additive. `--` was measured absent from all 83
|
||||||
|
ratified pattern ids and all 89 pre-existing case ids, which keeps the reverse transform
|
||||||
|
purely lexical — no lookup against the id list — the property the original one-to-one rule
|
||||||
|
was protecting.
|
||||||
|
|
||||||
|
The `one_case_per_pattern_id` key is **removed**, superseded by
|
||||||
|
`case_id_derivation.variant_suffix.supersedes`, which quotes its text. It was documentation
|
||||||
|
of the constraint, not data a consumer matches on, but it is called out here because a
|
||||||
|
removed key is normally a breaking change in this repository.
|
||||||
|
|
||||||
|
`omitted_payloads` gains `derivation_ground_withdrawn_in_0_3_0`: the guard's seventh
|
||||||
|
active-content payload was omitted on two grounds, and this change retires one of them. The
|
||||||
|
other stands, so the payload stays omitted — on one ground instead of two. **It was not
|
||||||
|
added back**; that is a separate decision, not a consequence of this one.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **`conformance/hybrid-xss__script-tag--src-no-close/` (89 → 90 cases)** — input
|
||||||
|
`<script src=x.js>`, 17 bytes, expecting `hybrid-xss:script-tag`. The regression gate for
|
||||||
|
the change above, and the reason the corpus could not previously see it: the existing
|
||||||
|
`hybrid-xss__script-tag` input `<script>steal()</script>` matches the pattern under *both*
|
||||||
|
forms, so it passes either way. Reverting the pattern to its `0.6.0` form fails this case
|
||||||
|
and only this case — mutation-verified in both directions across all 90.
|
||||||
|
|
||||||
|
**The first case input authored in this repository** rather than reproduced verbatim from a
|
||||||
|
runtime's payload set, recorded in the new `authored_payloads` block rather than folded into
|
||||||
|
`payload_provenance`, whose value is precisely the claim that its inputs are verbatim
|
||||||
|
upstream. That claim stays exactly as strong as it was: 83 of 83. Both witnesses for this
|
||||||
|
axis were named by `llm-security` on 2026-08-10; this is the first of the two. Findings
|
||||||
|
measured through the guard's public API at `0dce50f` / `0.5.0`, with the existing case's
|
||||||
|
committed bytes and digest reproduced by the same harness in the same run as a control.
|
||||||
|
|
||||||
|
- **`schema/conformance-declaration.schema.json` (`0.1.0`)** — the shape a runtime publishes
|
||||||
|
alongside a conformance result, satisfying the §1.1 MUST that `v0.2.0` created and left
|
||||||
|
without a form. Requested by `llm-security` in those terms (runtime, commit measured,
|
||||||
|
implemented file paths) with the stated reason that two runtimes publishing free-form
|
||||||
|
declarations makes `83/83 + 6 not-applicable` unparseable by anyone but its author.
|
||||||
|
|
||||||
|
Carries the two arithmetic invariants §1.1 implies but cannot state unambiguously in prose:
|
||||||
|
the four verdict counts MUST sum to the total, and the total MUST equal the corpus case
|
||||||
|
count at the commit measured. Requires the enumeration arrays whenever their counts are
|
||||||
|
non-zero, which turns §1.1's "MUST still be enumerated" from prose into a schema failure.
|
||||||
|
Keeps `error` and `not_applicable` structurally distinct, per §1.1. Records
|
||||||
|
`declaration_source` — whether the declared set is derived from the runner's own constant or
|
||||||
|
hand-maintained beside it — because only the derived form makes the anti-narrowing fence
|
||||||
|
structural. **Deliberately not a gate:** nothing in this repository runs, and no validation
|
||||||
|
was asked for. Mutation-tested: the example validates, and five distinct defect classes are
|
||||||
|
rejected.
|
||||||
|
|
||||||
|
- **`spec/conformance-corpus.md` §1.1** — normative pointer to that schema, plus a SHOULD that
|
||||||
|
a runtime derive its declared set from the constant its runner uses to accept or reject a
|
||||||
|
`scope`, and record which it did.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- `docs/lexicon-port-divergence.md` — the `hybrid-xss:script-tag` row is closed on the
|
||||||
|
closing-tag axis, having reversed twice in three days (guard-diverges → commons-diverges →
|
||||||
|
converged). What remains open is the one-byte span difference: the guard excludes `<` from
|
||||||
|
its negated class and the other two do not, so `<script <x>` matches commons and
|
||||||
|
`llm-security` and not the guard. Measured by compiling all three forms, not reasoned from
|
||||||
|
the character classes; neither side has claimed it.
|
||||||
|
|
||||||
## [0.2.0] — 2026-08-11
|
## [0.2.0] — 2026-08-11
|
||||||
|
|
||||||
The contract gained a normative MUST, which is why this is a release rather than a
|
The contract gained a normative MUST, which is why this is a release rather than a
|
||||||
|
|
|
||||||
18
README.md
18
README.md
|
|
@ -24,7 +24,7 @@ As a `git subtree` (recommended: history is preserved and upgrades are a single
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git subtree add --prefix vendor/commons \
|
git subtree add --prefix vendor/commons \
|
||||||
https://git.fromaitochitta.com/open/llm-security-commons.git v0.2.0 --squash
|
https://git.fromaitochitta.com/open/llm-security-commons.git v0.3.0 --squash
|
||||||
|
|
||||||
# later, to move to a newer tag
|
# later, to move to a newer tag
|
||||||
git subtree pull --prefix vendor/commons \
|
git subtree pull --prefix vendor/commons \
|
||||||
|
|
@ -34,7 +34,7 @@ git subtree pull --prefix vendor/commons \
|
||||||
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
|
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone --depth 1 --branch v0.2.0 \
|
git clone --depth 1 --branch v0.3.0 \
|
||||||
https://git.fromaitochitta.com/open/llm-security-commons.git
|
https://git.fromaitochitta.com/open/llm-security-commons.git
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -58,16 +58,17 @@ and keeping it that small is the point.
|
||||||
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself. |
|
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself. |
|
||||||
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
|
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
|
||||||
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
|
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
|
||||||
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
|
| [`schema/conformance-declaration.schema.json`](schema/conformance-declaration.schema.json) | **Normative.** The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs. |
|
||||||
| [`conformance/`](conformance/) | 89 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 83 cover the injection lexicon, one per pattern, and both seeding runtimes were measured producing the same verdict on all 83. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
|
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, what `exact-within-scope` requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as `not-applicable` rather than as a failure. |
|
||||||
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.2.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
| [`conformance/`](conformance/) | 90 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
|
||||||
|
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.3.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
||||||
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
||||||
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
|
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
|
||||||
|
|
||||||
Every JSON file carries a top-level `version`. Every normative specification carries a
|
Every JSON file carries a top-level `version`. Every normative specification carries a
|
||||||
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
|
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
|
||||||
part of the contract, but the file does not exist yet — they are not links, and nothing in
|
part of the contract, but the file does not exist yet — they are not links, and nothing in
|
||||||
v0.2.0 depends on them.
|
v0.3.0 depends on them.
|
||||||
|
|
||||||
Each data file records its own provenance and, in `verified`, how strongly it is backed.
|
Each data file records its own provenance and, in `verified`, how strongly it is backed.
|
||||||
`calibration/calibration.json` is currently the one file that says `false`: it was
|
`calibration/calibration.json` is currently the one file that says `false`: it was
|
||||||
|
|
@ -91,8 +92,9 @@ runs in each consumer's own test suite, against a pinned tag.
|
||||||
The corpus covers two tables, and they do not carry equal weight — treating them as one
|
The corpus covers two tables, and they do not carry equal weight — treating them as one
|
||||||
number would misreport both:
|
number would misreport both:
|
||||||
|
|
||||||
- `lexicon/injection-lexicon.json` — 83 cases. Both seeding runtimes implement it and both
|
- `lexicon/injection-lexicon.json` — 84 cases over 83 patterns. Both seeding runtimes
|
||||||
ratified its id space.
|
implement it and both ratified its id space. One pattern carries a second, variant case;
|
||||||
|
see `case_id_derivation.variant_suffix` in the manifest.
|
||||||
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
|
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
|
||||||
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
|
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
|
||||||
runtime declares which commons data files it implements, and a case scoped outside that
|
runtime declares which commons data files it implements, and a case scoped outside that
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,23 @@
|
||||||
|
{
|
||||||
|
"$comment": "Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json. TWO things about this case are unlike the 83 lexicon cases, both recorded in manifest.json rather than only here. (1) The input was NOT taken from the guard's _LEX_PAYLOADS - this is the first case authored in this repository; see manifest.json `authored_payloads`. (2) It is a variant case: its id carries the `--src-no-close` suffix permitted by manifest.json `case_id_derivation.variant_suffix`, because the pattern id it belongs to already has a case. WHY IT EXISTS, so that a reader does not delete it as a duplicate of hybrid-xss__script-tag: that case's input `<script>steal()</script>` matches both the pre-0.7.0 form `<script\\b[^>]*>[\\s\\S]*?<\\/script>` and the post-0.7.0 form `<script\\b[^>]*>`, so it cannot see the difference between them. This input matches only the latter. A runtime - or a future edit to lexicon/injection-lexicon.json - that reinstates the closing-tag requirement fails this case and passes the other one. Before this case existed that reinstatement was invisible to the corpus, which is exactly how the drift recorded in the lexicon's post_extraction_drift went two days unseen by any gate on any side.",
|
||||||
|
"case_id": "hybrid-xss__script-tag--src-no-close",
|
||||||
|
"input": {
|
||||||
|
"file": "input.txt",
|
||||||
|
"bytes": 17,
|
||||||
|
"sha256": "722b7ed714d37bf3bb606b37e21897afb992737dcf629c9f91f0fb0cba6a272d"
|
||||||
|
},
|
||||||
|
"scope": [
|
||||||
|
"lexicon/injection-lexicon.json"
|
||||||
|
],
|
||||||
|
"match": "exact-within-scope",
|
||||||
|
"findings": [
|
||||||
|
{
|
||||||
|
"pattern_id": "hybrid-xss:script-tag"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"observed_out_of_scope": {
|
||||||
|
"llm_ingestion_guard": [
|
||||||
|
"active:raw-html"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
<script src=x.js>
|
||||||
|
|
@ -1,14 +1,21 @@
|
||||||
{
|
{
|
||||||
"version": "0.2.0",
|
"version": "0.3.0",
|
||||||
"id": "llm-security-commons/conformance",
|
"id": "llm-security-commons/conformance",
|
||||||
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
|
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
|
||||||
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
|
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
|
||||||
"case_id_derivation": {
|
"case_id_derivation": {
|
||||||
"rule": "case_id = pattern_id with ':' replaced by '__'",
|
"rule": "case_id = pattern_id with ':' replaced by '__', optionally followed by '--' and a variant slug",
|
||||||
"reverse": "pattern_id = case_id with '__' replaced by ':'",
|
"reverse": "pattern_id = case_id truncated at the first '--' if present, then '__' replaced by ':'",
|
||||||
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 89.",
|
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 90.",
|
||||||
"stability": "A case id is a stable identifier. Changing one is a BREAKING change.",
|
"stability": "A case id is a stable identifier. Changing one is a BREAKING change.",
|
||||||
"one_case_per_pattern_id": "The derivation takes a pattern id and nothing else, so a single-finding scope holds at most one case per pattern id — there is nowhere in the name to put a second. See spec/conformance-corpus.md section 6, and `omitted_payloads` below for the one payload this actually cost."
|
"variant_suffix": {
|
||||||
|
"added_in": "0.3.0",
|
||||||
|
"syntax": "'--' + a slug of [a-z0-9-]",
|
||||||
|
"purpose": "To hold a SECOND case for one pattern id when the two inputs answer different questions about the same rule. The first such case is hybrid-xss__script-tag--src-no-close: the original case's input matches the pattern under both its pre-0.7.0 and post-0.7.0 forms, so it cannot gate the difference between them, and the variant input can.",
|
||||||
|
"separator_is_unambiguous": "Measured, not assumed: '--' occurs in none of the 83 ratified pattern ids and in none of the 89 case ids that predate this version. Ids use single hyphens throughout. So the reverse transform stays purely LEXICAL — split at the first '--', no lookup against the id list required — which is the property the original one-to-one rule was protecting.",
|
||||||
|
"constraint": "A variant case MUST be scoped and matched exactly like the base case, and MUST expect the same pattern id. The suffix distinguishes INPUTS, never findings. It is not a licence to record a second, different verdict for one rule.",
|
||||||
|
"supersedes": "This replaces the `one_case_per_pattern_id` note carried through 0.2.0, which read: 'The derivation takes a pattern id and nothing else, so a single-finding scope holds at most one case per pattern id — there is nowhere in the name to put a second.' That was an accurate description of the rule and it cost a real case: see `omitted_payloads`, where the guard's seventh active-content payload was dropped for exactly this reason. The rule is now extended rather than worked around. NOTE that the omitted payload has NOT been added back — extending the derivation makes it expressible, but it was omitted on a second ground as well (its in-scope finding set is identical to the case built from coverage.py:476), and that ground still stands unexamined against the new rule. Adding it is a separate decision, not a consequence of this one."
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"match_semantics": "exact-within-scope",
|
"match_semantics": "exact-within-scope",
|
||||||
"scope_covered": [
|
"scope_covered": [
|
||||||
|
|
@ -30,12 +37,38 @@
|
||||||
"source": "llm-ingestion-pipeline-security src/llm_ingestion_guard/coverage.py:484, at commit de09711 (line numbers are commit-relative; the structure is the fifth `_scan_case` of `_build_cases()`'s `active` group)",
|
"source": "llm-ingestion-pipeline-security src/llm_ingestion_guard/coverage.py:484, at commit de09711 (line numbers are commit-relative; the structure is the fifth `_scan_case` of `_build_cases()`'s `active` group)",
|
||||||
"described_as": "opaque (base64) path segment",
|
"described_as": "opaque (base64) path segment",
|
||||||
"expected_label": "active:markdown-image",
|
"expected_label": "active:markdown-image",
|
||||||
"reason": "Its in-scope finding set is `[active:markdown-image]` — identical, measured, to the case built from coverage.py:476. The only thing that distinguishes it is `entropy:base64-blob`, and this repository publishes no entropy table, so the difference falls outside every declared scope. A second case could not have failed in any way the first does not, and the case-id derivation has no room for it (see `case_id_derivation.one_case_per_pattern_id`).",
|
"reason": "Its in-scope finding set is `[active:markdown-image]` — identical, measured, to the case built from coverage.py:476. The only thing that distinguishes it is `entropy:base64-blob`, and this repository publishes no entropy table, so the difference falls outside every declared scope. A second case could not have failed in any way the first does not, and the case-id derivation had no room for it.",
|
||||||
"$comment": "Recorded so that 6 built from 7 offered reads as a decision rather than as a miscount."
|
"$comment": "Recorded so that 6 built from 7 offered reads as a decision rather than as a miscount.",
|
||||||
|
"derivation_ground_withdrawn_in_0_3_0": "The second half of `reason` — that the derivation has no room for a second case per pattern id — stopped being true in 0.3.0, when `case_id_derivation.variant_suffix` was added. It is left in the text above with this note rather than silently deleted, because the omission decision was taken on TWO grounds and only this one lapsed. The first ground stands: the payload's in-scope finding set is identical to the case already built, so it still could not fail in any way that case does not. The payload therefore remains omitted, on one ground instead of two. Re-examining it is a separate decision."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"authored_payloads": [
|
||||||
|
{
|
||||||
|
"case_id": "hybrid-xss__script-tag--src-no-close",
|
||||||
|
"added_in": "0.3.0",
|
||||||
|
"added_date": "2026-08-11",
|
||||||
|
"input": "<script src=x.js>",
|
||||||
|
"authored_by": "llm-security-commons",
|
||||||
|
"$comment": "The FIRST case input authored in this repository rather than reproduced from a runtime's payload set, and it is flagged rather than folded into payload_provenance because that block's value is the claim that its inputs are verbatim upstream. That claim stays exactly as strong as it was — 83 of 83 — and this case sits outside it.",
|
||||||
|
"why_authored": "No runtime had a payload for it. The input has to distinguish `<script\\b[^>]*>` from `<script\\b[^>]*>[\\s\\S]*?<\\/script>`, and the guard's _LEX_PAYLOADS carries one payload per pattern id chosen to MATCH the pattern, not to discriminate between two candidate forms of it. Both witnesses for this axis were named by llm-security in their coord message of 2026-08-10 (`<script src=x.js>` and unclosed `<script>alert(1)`); this is the first of the two, and they offered to supply them.",
|
||||||
|
"why_not_authored_earlier": "It was declined on 2026-08-11 in a coord reply, on the ground that a fixture encoding a DISAGREEMENT is worse than an absent one: at that point the input matched both implementations and failed against commons' own published lexicon, so its expectation would have contradicted the data file it is scoped to. Lexicon 0.7.0 removed that contradiction. The stated order was 'settle the row, then the case is trivial to add', and this is that step.",
|
||||||
|
"measurement": {
|
||||||
|
"date": "2026-08-11",
|
||||||
|
"guard_commit": "0dce50f",
|
||||||
|
"guard_version": "0.5.0",
|
||||||
|
"method": "Measured through the guard's public API — scan_lexicon() and scan_active_content() imported from llm_ingestion_guard — not read off the regex. The same harness was run against the existing hybrid-xss__script-tag case in the same execution, and it reproduced that case's committed bytes (24) and sha256 exactly, which is what licenses trusting its output for the new one. Digest independently recomputed with shasum -a 256 over the file on disk: agrees.",
|
||||||
|
"guard_in_scope_result": [
|
||||||
|
"hybrid-xss:script-tag"
|
||||||
|
],
|
||||||
|
"guard_out_of_scope_result": [
|
||||||
|
"active:raw-html"
|
||||||
|
],
|
||||||
|
"llm_security_not_measured": "The llm-security side was verified at the PATTERN level only: its regex source at b1ba1fb is byte-identical to the lexicon's 0.7.0 pattern, and that pattern was compiled and tested against this input. Their runtime was NOT run over this fixture from here — this repository does not execute consumer code, and their own harness will report the verdict. Stated so that `observed_out_of_scope` naming only the guard reads as a measurement boundary rather than an oversight."
|
||||||
|
}
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"payload_provenance": {
|
"payload_provenance": {
|
||||||
"$scope": "The 83 lexicon cases. The 6 active-content cases have their own provenance in `active_content_provenance` below — they come from a different structure in the same file, at a different commit, and folding them in here would let one pin stand for two measurements.",
|
"$scope": "The 83 lexicon cases derived from _LEX_PAYLOADS. NOT all 84 lexicon-scoped cases as of 0.3.0 — the variant case hybrid-xss__script-tag--src-no-close was authored in this repository and is covered by `authored_payloads`, not by this block. The 6 active-content cases have their own provenance in `active_content_provenance` below — they come from a different structure in the same file, at a different commit, and folding them in here would let one pin stand for two measurements.",
|
||||||
"source_repo": "llm-ingestion-pipeline-security",
|
"source_repo": "llm-ingestion-pipeline-security",
|
||||||
"source_file": "src/llm_ingestion_guard/coverage.py",
|
"source_file": "src/llm_ingestion_guard/coverage.py",
|
||||||
"source_export": "_LEX_PAYLOADS",
|
"source_export": "_LEX_PAYLOADS",
|
||||||
|
|
@ -46,7 +79,7 @@
|
||||||
"stability_note": "llm-ingestion-pipeline-security states (coord message 2026-08-10T12:42:31Z) that _LEX_PAYLOADS and the 83 pattern ids are an internal surface on their side, with no README/CHANGELOG/docs statement promising id or payload stability - their own test suite enforces id coverage as their gate, not as a promise to this repository. Their stated position: if a payload changes upstream, this manifest's pin diverges and should be re-pinned; divergence is a re-pin signal, not a breach of a contract they never granted."
|
"stability_note": "llm-ingestion-pipeline-security states (coord message 2026-08-10T12:42:31Z) that _LEX_PAYLOADS and the 83 pattern ids are an internal surface on their side, with no README/CHANGELOG/docs statement promising id or payload stability - their own test suite enforces id coverage as their gate, not as a promise to this repository. Their stated position: if a payload changes upstream, this manifest's pin diverges and should be re-pinned; divergence is a re-pin signal, not a breach of a contract they never granted."
|
||||||
},
|
},
|
||||||
"measurement": {
|
"measurement": {
|
||||||
"$scope": "The 83 lexicon cases. The 6 active-content cases are measured in `active_content_measurement`.",
|
"$scope": "The 83 lexicon cases derived from _LEX_PAYLOADS, measured 2026-08-10. The variant case added in 0.3.0 is measured separately in `authored_payloads` — it did not exist on this date, and stretching this header to cover it would misdate its measurement. The 6 active-content cases are measured in `active_content_measurement`.",
|
||||||
"date": "2026-08-10",
|
"date": "2026-08-10",
|
||||||
"method": "Each payload was run through both runtimes' PUBLIC entry point — not through a rebuilt regex table — and the resulting finding labels were mapped to commons pattern ids through the lexicon's own aliases block. Comparing at the entry point is deliberate: a table-level comparison produces a number that describes neither runtime.",
|
"method": "Each payload was run through both runtimes' PUBLIC entry point — not through a rebuilt regex table — and the resulting finding labels were mapped to commons pattern ids through the lexicon's own aliases block. Comparing at the entry point is deliberate: a table-level comparison produces a number that describes neither runtime.",
|
||||||
"runtimes": [
|
"runtimes": [
|
||||||
|
|
@ -127,11 +160,12 @@
|
||||||
},
|
},
|
||||||
"guard_drift_note": "The guard's HEAD moved twice during this measurement (3c56d50 -> de09711 -> 398eb74). Measurement ran at de09711. Checked rather than hoped: 398eb74 touches only docs/PLAN-v1.md and tests/test_disposition.py, so no measured file changed under the measurement. active_content.py DID change between 0bf0729 — the commit signatures/active-content.json pins — and de09711; the change adds a scan-cap self-safety finding and touches no construct. See that file's `pattern_id_space.verified.at_commit_note`."
|
"guard_drift_note": "The guard's HEAD moved twice during this measurement (3c56d50 -> de09711 -> 398eb74). Measurement ran at de09711. Checked rather than hoped: 398eb74 touches only docs/PLAN-v1.md and tests/test_disposition.py, so no measured file changed under the measurement. active_content.py DID change between 0bf0729 — the commit signatures/active-content.json pins — and de09711; the change adds a scan-cap self-safety finding and touches no construct. See that file's `pattern_id_space.verified.at_commit_note`."
|
||||||
},
|
},
|
||||||
"count": 89,
|
"count": 90,
|
||||||
"count_by_scope": {
|
"count_by_scope": {
|
||||||
"lexicon/injection-lexicon.json": 83,
|
"lexicon/injection-lexicon.json": 84,
|
||||||
"signatures/active-content.json": 6
|
"signatures/active-content.json": 6
|
||||||
},
|
},
|
||||||
|
"count_by_scope_$comment": "84 lexicon-scoped cases, of which 83 come from the guard's _LEX_PAYLOADS one-per-pattern-id (see payload_provenance) and one was authored here (see authored_payloads). The lexicon count therefore no longer equals the pattern count, and that is deliberate as of 0.3.0 — a pattern id may carry a variant case. See case_id_derivation.variant_suffix.",
|
||||||
"cases": [
|
"cases": [
|
||||||
"override__ignore-previous",
|
"override__ignore-previous",
|
||||||
"override__forget-instructions",
|
"override__forget-instructions",
|
||||||
|
|
@ -213,6 +247,7 @@
|
||||||
"hybrid-recursive__inject-output",
|
"hybrid-recursive__inject-output",
|
||||||
"hybrid-recursive__conditional-response",
|
"hybrid-recursive__conditional-response",
|
||||||
"hybrid-xss__script-tag",
|
"hybrid-xss__script-tag",
|
||||||
|
"hybrid-xss__script-tag--src-no-close",
|
||||||
"hybrid-xss__javascript-uri",
|
"hybrid-xss__javascript-uri",
|
||||||
"hybrid-xss__event-handler",
|
"hybrid-xss__event-handler",
|
||||||
"hybrid-xss__iframe-src",
|
"hybrid-xss__iframe-src",
|
||||||
|
|
|
||||||
|
|
@ -94,7 +94,7 @@ Divergence is **not one-directional**. Each side has inputs the other's pattern
|
||||||
| `markdown: injection payload in link anchor text` | **guard misses** | `[click ignore this](http://e.com/a(b))` |
|
| `markdown: injection payload in link anchor text` | **guard misses** | `[click ignore this](http://e.com/a(b))` |
|
||||||
| `sub-agent: spawn instruction with dangerous capability keywords` | **guard misses** | `create an agent that` + 20 filler words + `execute` |
|
| `sub-agent: spawn instruction with dangerous capability keywords` | **guard misses** | `create an agent that` + 20 filler words + `execute` |
|
||||||
| `sub-agent: delegation with safety bypass instruction` | **guard misses** | `delegate to a new agent` + 200 chars + `bypass` |
|
| `sub-agent: delegation with safety bypass instruction` | **guard misses** | `delegate to a new agent` + 200 chars + `bypass` |
|
||||||
| `hybrid-xss: <script> tag in content (agent context XSS)` | **commons misses**; span differs | `<script>alert(1)` (unclosed), `<script src=x.js>` |
|
| `hybrid-xss: <script> tag in content (agent context XSS)` | **reversed 2026-08-11, see below** | `<script>alert(1)` (unclosed), `<script src=x.js>` |
|
||||||
| `hybrid-xss: iframe with executable src (agent context XSS)` | **guard misses the label, not the attack** | `<iframe data-x="<" src="javascript:alert(1)">` |
|
| `hybrid-xss: iframe with executable src (agent context XSS)` | **guard misses the label, not the attack** | `<iframe data-x="<" src="javascript:alert(1)">` |
|
||||||
|
|
||||||
The last row is the one of the 13 that does not reach report level. The guard's
|
The last row is the one of the 13 that does not reach report level. The guard's
|
||||||
|
|
@ -109,6 +109,76 @@ consumer reports offsets or evidence text; it does not change whether a finding
|
||||||
The commons-side misses were confirmed in a real JS engine (Node v25.8.2, `RegExp` built from
|
The commons-side misses were confirmed in a real JS engine (Node v25.8.2, `RegExp` built from
|
||||||
the committed JSON), not only in the Python harness used for the differential.
|
the committed JSON), not only in the Python harness used for the differential.
|
||||||
|
|
||||||
|
### `hybrid-xss:script-tag`: the row reversed, 2026-08-11
|
||||||
|
|
||||||
|
At the time this file was written, `<script>alert(1)` (unclosed) and `<script src=x.js>` were
|
||||||
|
commons misses only in the sense that commons was a byte-identical extraction of
|
||||||
|
`llm-security`, so "commons misses" meant "commons and `llm-security` both miss, the guard
|
||||||
|
does not." That symmetry broke on `llm-security`'s side.
|
||||||
|
|
||||||
|
`llm-security` commit `90f576f` (`fix(llm-security): v8.x-A - close <script> recall hole, add
|
||||||
|
whole-table ReDoS gate`) replaced the source pattern
|
||||||
|
|
||||||
|
```
|
||||||
|
<script\b[^>]*>[\s\S]*?<\/script>
|
||||||
|
```
|
||||||
|
|
||||||
|
with
|
||||||
|
|
||||||
|
```
|
||||||
|
<script\b[^>]*>
|
||||||
|
```
|
||||||
|
|
||||||
|
dropping the closing-tag requirement that produced the recall hole this file first measured.
|
||||||
|
Both witnesses now match on `llm-security`'s side. Verified directly against the module at
|
||||||
|
commit `c67bad3` (2026-08-10, first commit carrying the change) and re-confirmed unmoved at
|
||||||
|
`b1ba1fb` (2026-08-11, current public-remote HEAD as fetched this session) — `git diff
|
||||||
|
c67bad3..b1ba1fb -- scanners/lib/injection-patterns.mjs` is empty, so no later commit touched
|
||||||
|
the pattern a second time.
|
||||||
|
|
||||||
|
The guard's pattern was never the same construct: `<script\b[^><]*>`, negated-class excluding
|
||||||
|
both `>` and `<` (`src/llm_ingestion_guard/injection_lexicon.json:463`, unchanged since
|
||||||
|
`de09711`, re-confirmed unmoved through current guard HEAD `0dce50f`, 2026-08-11). The two
|
||||||
|
patterns now agree on **shape** — both drop the closing-tag requirement, both are a single
|
||||||
|
negated-class run immediately after `<script\b` — but they are still not byte-identical: the
|
||||||
|
guard's excluded-character class also excludes `<`, `llm-security`'s does not. That one-byte
|
||||||
|
difference is unresolved on both sides (see *Open questions* in the coordination thread); it
|
||||||
|
does not reopen the closing-tag question, which is settled.
|
||||||
|
|
||||||
|
**Commons converged in lexicon `0.7.0` (2026-08-11), and this row is closed on the closing-tag
|
||||||
|
axis.** Through `0.6.0` commons was the sole diverger of the three: both `llm-security` and the
|
||||||
|
guard matched `<script src=x.js>` and unclosed `<script>alert(1)`, and commons' extracted pattern
|
||||||
|
still required `</script>`. Commons now carries `<script\b[^>]*>` — byte-identical to
|
||||||
|
`llm-security`'s `RegExp.prototype.source` at `b1ba1fb`, verified by compiling both.
|
||||||
|
|
||||||
|
The decision was **re-extraction, not revision**, and the distinction is what made it available
|
||||||
|
at all under the behaviour-preservation invariant. That invariant forbids commons from correcting
|
||||||
|
seed data it believes is wrong; it does not forbid re-reading the source after the source itself
|
||||||
|
moved and its owner asked. The lexicon's declared provenance is `llm-security`'s injection table,
|
||||||
|
and being loadable verbatim by that table's owner is the one thing it exists to do — the standing
|
||||||
|
alternative was a per-pattern override in `llm-security`'s own loader, i.e. a published core that
|
||||||
|
its source repository could not load as published. Operator decision, 2026-08-11, on
|
||||||
|
`llm-security`'s blocking coord request. Explicitly **not** decided by the 2-of-3 majority: a
|
||||||
|
count of implementations is not a mandate over detection data, and the provenance argument would
|
||||||
|
have held with the guard on either side.
|
||||||
|
|
||||||
|
The direction matters for what it cannot break: the new form matches a strict superset of the old
|
||||||
|
one, so relative to `0.6.0` it can add matches and cannot add a false negative. The reverse change
|
||||||
|
would not have been adoptable on the same reasoning.
|
||||||
|
|
||||||
|
**What is still open is the one-byte span difference**, and convergence did not touch it. The
|
||||||
|
guard excludes `<` from its negated class and the other two do not, so:
|
||||||
|
|
||||||
|
| Witness | commons `0.7.0` / `llm-security` | guard |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `<script>steal()</script>` | match | match |
|
||||||
|
| `<script src=x.js>` | match | match |
|
||||||
|
| `<script>alert(1)` | match | match |
|
||||||
|
| `<script <x>` | **match** | **no match** |
|
||||||
|
|
||||||
|
Measured by compiling all three forms, not reasoned from the character classes. The last row is
|
||||||
|
the whole residual divergence and neither side has claimed it.
|
||||||
|
|
||||||
## Why they diverge: two different ReDoS mitigations of one table
|
## Why they diverge: two different ReDoS mitigations of one table
|
||||||
|
|
||||||
This is not drift, and framing it as a bug in either repository would be wrong.
|
This is not drift, and framing it as a bug in either repository would be wrong.
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
{
|
{
|
||||||
"version": "0.5.1",
|
"version": "0.7.0",
|
||||||
"id": "injection-lexicon",
|
"id": "injection-lexicon",
|
||||||
"description": "Prompt-injection pattern lexicon: the four pattern families a detector matches against untrusted text, and the severity family each belongs to. Data only. The variant building that feeds these patterns (normalisation, homoglyph folding, rot13, unicode-tag escalation) and the buried-payload heuristic are engine behaviour and are deliberately NOT described here.",
|
"description": "Prompt-injection pattern lexicon: the four pattern families a detector matches against untrusted text, and the severity family each belongs to. Data only. The variant building that feeds these patterns (normalisation, homoglyph folding, rot13, unicode-tag escalation) and the buried-payload heuristic are engine behaviour and are deliberately NOT described here.",
|
||||||
"owasp": "LLM01",
|
"owasp": "LLM01",
|
||||||
|
|
@ -76,10 +76,13 @@
|
||||||
"normalisations": [],
|
"normalisations": [],
|
||||||
"normalisations_note": "Empty by result, not by omission: all 83 patterns are byte-identical to the source module, so there is no transformation to declare. Through version 0.3.0 this array held one entry; it was retracted as false in 0.4.0. See source_fidelity.",
|
"normalisations_note": "Empty by result, not by omission: all 83 patterns are byte-identical to the source module, so there is no transformation to declare. Through version 0.3.0 this array held one entry; it was retracted as false in 0.4.0. See source_fidelity.",
|
||||||
"source_fidelity": {
|
"source_fidelity": {
|
||||||
"$comment": "This file applies NO transformation to the pattern text. All 83 patterns are byte-identical to the RegExp literal source in llm-security/scanners/lib/injection-patterns.mjs at commit b0de0ca. The `normalisations` array is empty because there is nothing to declare, not because the question was skipped.",
|
"$comment": "This file applies NO transformation to the pattern text. All 83 patterns are byte-identical to the RegExp literal source in llm-security/scanners/lib/injection-patterns.mjs - but READ THE COORDINATE BELOW, because it moved in version 0.7.0. Through 0.6.0 the coordinate was the original extraction commit b0de0ca. In 0.7.0 one pattern (hybrid-xss:script-tag) was RE-EXTRACTED at b1ba1fb, so the coordinate for a 83/83 claim is now b1ba1fb and the count against b0de0ca is 82/83. The `normalisations` array is empty because there is nothing to declare, not because the question was skipped.",
|
||||||
"patterns_total": 83,
|
"patterns_total": 83,
|
||||||
"patterns_byte_identical_to_source": 83,
|
"patterns_byte_identical_to_source": 83,
|
||||||
"method": "The module was imported in Node from a read-only clone of the public remote at b0de0ca, and every entry compared in array order on three axes: RegExp.prototype.source, RegExp.prototype.flags, and label. Family membership, array order and per-family counts were compared at the same time. Nothing was read by eye.",
|
"byte_identical_against_commit": "b1ba1fb",
|
||||||
|
"byte_identical_against_commit_date": "2026-08-11",
|
||||||
|
"byte_identical_against_commit_$comment": "The missing coordinate on `patterns_byte_identical_to_source`, added in 0.7.0 when that count stopped referring to a single commit for the whole file's life. The key and its value are unchanged - 83 patterns, byte-identical - and this field states WHICH source commit they are identical TO. Against the original extraction commit b0de0ca the count is 82/83; the one difference is the re-extracted pattern recorded in post_extraction_drift.",
|
||||||
|
"method": "The module was imported in Node from a read-only clone of the public remote at b0de0ca, and every entry compared in array order on three axes: RegExp.prototype.source, RegExp.prototype.flags, and label. Family membership, array order and per-family counts were compared at the same time. Nothing was read by eye. In 0.7.0 the single re-extracted pattern was verified the same way against b1ba1fb (injection-patterns.mjs:170); the other 82 were not re-imported, so their identity rests on the b0de0ca run plus the measured fact that `git diff b0de0ca..b1ba1fb -- scanners/lib/injection-patterns.mjs` changes exactly one `{ pattern, label }` line - the script-tag one - and otherwise only adds five comment lines. Exactly one commit in that range touches the file at all (90f576f).",
|
||||||
"class_membership_verified": {
|
"class_membership_verified": {
|
||||||
"zero_width_class": [
|
"zero_width_class": [
|
||||||
"U+200B",
|
"U+200B",
|
||||||
|
|
@ -112,7 +115,39 @@
|
||||||
"why_retracted": "The same transcription inversion in the opposite direction. The module writes pr[e\\u00e9]c[e\\u00e9]dentes? with escapes; this was the only one of the 83 patterns not byte-identical to source. Corrected in 0.4.0 to the module's spelling, which makes the count 83/83.",
|
"why_retracted": "The same transcription inversion in the opposite direction. The module writes pr[e\\u00e9]c[e\\u00e9]dentes? with escapes; this was the only one of the 83 patterns not byte-identical to source. Corrected in 0.4.0 to the module's spelling, which makes the count 83/83.",
|
||||||
"behaviour_impact": "None. The two spellings are the same regular expression. Verified by compiling both forms and comparing match presence and offsets in Node (bare and `u` mode) and in Python `re` over accented, unaccented, uppercase and non-matching French inputs - identical results throughout."
|
"behaviour_impact": "None. The two spellings are the same regular expression. Verified by compiling both forms and comparing match presence and offsets in Node (bare and `u` mode) and in Python `re` over accented, unaccented, uppercase and non-matching French inputs - identical results throughout."
|
||||||
}
|
}
|
||||||
]
|
],
|
||||||
|
"post_extraction_drift": {
|
||||||
|
"$comment": "The record of source drift AFTER the original extraction, and of how it was closed. Through 0.6.0 this block described an OPEN divergence: llm-security had moved one pattern and commons had not followed. In 0.7.0 that one pattern was re-extracted and the divergence is closed. The block is kept rather than deleted because a consumer diffing this file against b0de0ca still needs the coordinate that explains the one mismatch it will find.",
|
||||||
|
"status": "resolved in 0.7.0 by re-extraction",
|
||||||
|
"extracted_from_commit": "b0de0ca",
|
||||||
|
"extracted_from_commit_date": "2026-08-09",
|
||||||
|
"count_at_extraction": 83,
|
||||||
|
"drift": [
|
||||||
|
{
|
||||||
|
"pattern_id": "hybrid-xss:script-tag",
|
||||||
|
"source_commit": "90f576f",
|
||||||
|
"source_commit_date": "2026-08-10",
|
||||||
|
"change": "llm-security dropped the closing-tag requirement: <script\\b[^>]*>[\\s\\S]*?<\\/script> became <script\\b[^>]*>, closing a recall hole where `<script>alert(1)` (unclosed) and `<script src=x.js>` produced no finding.",
|
||||||
|
"commons_pattern_unchanged": false,
|
||||||
|
"commons_pattern_before_0_7_0": "<script\\b[^>]*>[\\s\\S]*?<\\/script>",
|
||||||
|
"commons_pattern_from_0_7_0": "<script\\b[^>]*>",
|
||||||
|
"resolved_in_version": "0.7.0",
|
||||||
|
"resolved_date": "2026-08-11",
|
||||||
|
"resolved_at_source_commit": "b1ba1fb",
|
||||||
|
"resolved_how": "RE-EXTRACTION, not revision, and the distinction is the whole justification. This file's declared provenance is llm-security's injection table; being loadable verbatim by that table's owner is the single thing it exists to do. Behaviour-preservation forbids commons from CORRECTING seed data it believes is wrong - it does not forbid re-reading the source after the source itself moved and its owner asked. The alternative on offer was a per-pattern override in llm-security's own loader, which would have meant the published core was not verbatim-loadable by the repository it was extracted from.",
|
||||||
|
"direction_note": "Widening, not narrowing: the new form matches a strict superset of the old one. It cannot introduce a false negative relative to 0.6.0, only additional matches. That is why this direction was safe to adopt while the reverse would not have been.",
|
||||||
|
"decided_by": "operator decision, 2026-08-11, on a llm-security coord request (their message 2026-08-11T10:46:28Z, blocking their step-4 table swap). NOT decided by majority: the guard also drops the closing tag, but a 2-of-3 count is not a mandate over detection data. The reason is provenance, and it would hold with the guard on either side.",
|
||||||
|
"reconfirmed_no_further_drift_at": "b1ba1fb",
|
||||||
|
"reconfirmed_date": "2026-08-11",
|
||||||
|
"reconfirmed_method": "git diff 90f576f..b1ba1fb -- scanners/lib/injection-patterns.mjs on a fresh fetch of the public remote is empty for this pattern; no later commit touched it a second time."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"count_against_llm_security_head_as_of_2026_08_11": 83,
|
||||||
|
"count_against_llm_security_head_note": "b1ba1fb was llm-security's public-remote HEAD when this was measured, 2026-08-11, and after the 0.7.0 re-extraction the count against it is 83/83. Was 82/83 in versions 0.6.0 and earlier. Not re-checked automatically; re-verify before citing against a later HEAD.",
|
||||||
|
"count_against_extraction_commit": 82,
|
||||||
|
"count_against_extraction_commit_note": "From 0.7.0 this file is 82/83 byte-identical to b0de0ca, the commit it was originally extracted from. That number will not improve: the re-extracted pattern is deliberately the newer form.",
|
||||||
|
"residual_divergence_vs_guard": "Closing the llm-security divergence does NOT make this pattern universal. The guard's port is <script\\b[^><]*> - its negated class also excludes `<` - so commons and the guard now agree on SHAPE (opening tag only) and still differ by one byte in the excluded set. Consequence, measured on construction rather than assumed: an input like `<script <x>` matches the commons/llm-security form and not the guard's. That row stays open in docs/lexicon-port-divergence.md as `span differs`, and neither side has claimed it."
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"families": [
|
"families": [
|
||||||
{
|
{
|
||||||
|
|
@ -942,7 +977,7 @@
|
||||||
{
|
{
|
||||||
"id": "hybrid-xss:script-tag",
|
"id": "hybrid-xss:script-tag",
|
||||||
"label": "hybrid-xss: <script> tag in content (agent context XSS)",
|
"label": "hybrid-xss: <script> tag in content (agent context XSS)",
|
||||||
"pattern": "<script\\b[^>]*>[\\s\\S]*?<\\/script>",
|
"pattern": "<script\\b[^>]*>",
|
||||||
"flags": "i",
|
"flags": "i",
|
||||||
"aliases": {
|
"aliases": {
|
||||||
"llm_security": "hybrid-xss: <script> tag in content (agent context XSS)",
|
"llm_security": "hybrid-xss: <script> tag in content (agent context XSS)",
|
||||||
|
|
|
||||||
248
schema/conformance-declaration.schema.json
Normal file
248
schema/conformance-declaration.schema.json
Normal file
|
|
@ -0,0 +1,248 @@
|
||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://git.fromaitochitta.com/open/llm-security-commons/raw/branch/main/schema/conformance-declaration.schema.json",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"status": "normative",
|
||||||
|
"title": "Conformance declaration",
|
||||||
|
"description": "The shape a runtime publishes alongside a conformance result, satisfying the MUST in spec/conformance-corpus.md section 1.1: declare the set of commons data files you implement, and publish that set with your result. Normative for the field names, types and the two arithmetic invariants stated in `invariants`. A result published without a declaration in this shape is not checkable by anyone but its author, which is most of the reason section 1.1 exists.",
|
||||||
|
"$comment": "Authored in this repository rather than extracted from a runtime, because no runtime had one to extract - section 1.1 shipped in commons v0.2.0 and created the obligation. Requested by llm-security (coord message 2026-08-11T10:46:28Z) in those terms: 'runtime name, the commit measured, and the list of commons file paths implemented', with the stated reason that two runtimes publishing free-form declarations makes `83/83 + 6 not-applicable` unparseable by anyone but its author. Deliberately NOT a gate: commons runs nothing (see CLAUDE.md charter), and llm-security explicitly did not ask for validation. Publishing the shape is the whole contribution.",
|
||||||
|
"requested_by": {
|
||||||
|
"runtime": "llm-security",
|
||||||
|
"coord_message_date": "2026-08-11T10:46:28Z",
|
||||||
|
"asked_for": "runtime name, the commit measured, the list of implemented commons file paths",
|
||||||
|
"beyond_the_ask": "`declaration_source`, `error`, and the enumeration arrays are not in their request. Each exists to make a section 1.1 MUST checkable rather than merely stated - see `invariants` and the per-property notes. Nothing here is required in order to satisfy their stated need."
|
||||||
|
},
|
||||||
|
"invariants": {
|
||||||
|
"$comment": "Two constraints that JSON Schema cannot express, stated normatively here because they are the point of the format rather than decoration on it. A validator that checks the schema and not these has checked the shape of a claim without checking the claim.",
|
||||||
|
"denominator_intact": "`result.total` MUST equal `passed + failed + not_applicable + error`. This is section 1.1's fence in arithmetic form: `76/83 passed` and `76 passed, 7 not-applicable` describe different runtimes, and only the second one can be checked. A declaration whose parts do not sum to its total has dropped cases somewhere between the run and the publication.",
|
||||||
|
"total_matches_corpus": "`result.total` MUST equal the number of cases the corpus contains at `commons_commit` - conformance/manifest.json's case count at that commit, not at HEAD. A runtime that ran fewer cases than the corpus holds has skipped some, which section 1 forbids outright."
|
||||||
|
},
|
||||||
|
"type": "object",
|
||||||
|
"$ref": "#/$defs/declaration",
|
||||||
|
"$defs": {
|
||||||
|
"declaration": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"runtime",
|
||||||
|
"commons_commit",
|
||||||
|
"declared_tables",
|
||||||
|
"result"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"runtime": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1,
|
||||||
|
"description": "The runtime publishing this result, named the same way it is named in the lexicon's `aliases` keys where it appears there (for example `llm_security`, `llm_ingestion_guard`). A runtime not registered in any commons alias block names itself freely; the constraint exists so a reader can join a declaration to an alias, not to restrict who may declare."
|
||||||
|
},
|
||||||
|
"runtime_commit": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 7,
|
||||||
|
"description": "The runtime's own commit that produced this result. SHOULD be present: without it the result cannot be reproduced, only believed."
|
||||||
|
},
|
||||||
|
"commons_commit": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 7,
|
||||||
|
"description": "The commons commit whose corpus was measured. REQUIRED, and it is the coordinate the whole declaration hangs on - case counts, case ids and table paths all move with it. A declaration citing a tag instead SHOULD give the commit the tag resolved to, because a tag can be moved and a commit cannot."
|
||||||
|
},
|
||||||
|
"commons_version": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "The commons release tag measured, if the measurement was taken at a tag (for example `v0.3.0`). Informative convenience; `commons_commit` is the authority."
|
||||||
|
},
|
||||||
|
"declared_tables": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"uniqueItems": true,
|
||||||
|
"items": {
|
||||||
|
"type": "string",
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9._/-]*\\.json$"
|
||||||
|
},
|
||||||
|
"description": "The set of commons data files this runtime implements, as repository-relative paths exactly as they appear in the commons tree at `commons_commit` (for example `lexicon/injection-lexicon.json`). This is the declaration section 1.1 requires. A case whose `scope` names a file outside this set is `not-applicable`; a case whose scope names a file inside it MUST be run, including the ones it fails."
|
||||||
|
},
|
||||||
|
"declaration_source": {
|
||||||
|
"enum": [
|
||||||
|
"derived-from-runner",
|
||||||
|
"hand-maintained"
|
||||||
|
],
|
||||||
|
"description": "Whether `declared_tables` is READ FROM the same constant the runner uses to accept or reject a scope (`derived-from-runner`), or maintained as a separate list beside it (`hand-maintained`). SHOULD be `derived-from-runner`. The distinction is not cosmetic: section 1.1 forbids narrowing a declared set in order to convert failures into `not-applicable` results, and only a derived declaration makes that fence structural - narrowing it is then a code change with tests behind it rather than an edit to a file nobody reads. The field exists so a reader can tell which kind of claim they are reading. Formulation owed to llm-security, who stated the constraint for their own side before commons had a field for it: 'a hand-maintained declaration is a claim; a derived one is a measurement.'"
|
||||||
|
},
|
||||||
|
"measured_date": {
|
||||||
|
"type": "string",
|
||||||
|
"format": "date",
|
||||||
|
"description": "The date the result was produced, ISO 8601. SHOULD be present: a declaration with no date cannot be ordered against a later one."
|
||||||
|
},
|
||||||
|
"result": {
|
||||||
|
"$ref": "#/$defs/result"
|
||||||
|
},
|
||||||
|
"not_applicable_cases": {
|
||||||
|
"$ref": "#/$defs/caseIdList",
|
||||||
|
"description": "Every case reported `not-applicable`, by case id. Section 1.1 requires that such cases still be enumerated in the result; this array is that enumeration, and it is REQUIRED whenever the count is non-zero. A bare count would satisfy the arithmetic and defeat the purpose - the reader needs to see WHICH questions were never addressed to this runtime in order to judge whether the declared set explains them."
|
||||||
|
},
|
||||||
|
"failed_cases": {
|
||||||
|
"$ref": "#/$defs/caseIdList",
|
||||||
|
"description": "Every case reported `fail`, by case id. REQUIRED whenever the count is non-zero. A failure this runtime is not naming is a failure nobody can reproduce."
|
||||||
|
},
|
||||||
|
"error_cases": {
|
||||||
|
"$ref": "#/$defs/caseIdList",
|
||||||
|
"description": "Every case reported as an error under section 1 - the runtime tried to run it and could not - by case id. REQUIRED whenever the count is non-zero. Kept distinct from `not_applicable_cases` because section 1.1 turns on exactly that distinction: an error means *this runtime tried and could not*, `not-applicable` means *this question was never addressed to this runtime*. Collapsing the two reports a capability gap as an architectural choice, or the reverse."
|
||||||
|
},
|
||||||
|
"notes": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Free text for anything the fields above cannot carry. MUST NOT be used to qualify a count: a number that needs prose to be read correctly is the wrong number."
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"allOf": [
|
||||||
|
{
|
||||||
|
"if": {
|
||||||
|
"properties": {
|
||||||
|
"result": {
|
||||||
|
"properties": {
|
||||||
|
"not_applicable": {
|
||||||
|
"minimum": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"not_applicable"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"result"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"then": {
|
||||||
|
"required": [
|
||||||
|
"not_applicable_cases"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"if": {
|
||||||
|
"properties": {
|
||||||
|
"result": {
|
||||||
|
"properties": {
|
||||||
|
"failed": {
|
||||||
|
"minimum": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"failed"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"result"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"then": {
|
||||||
|
"required": [
|
||||||
|
"failed_cases"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"if": {
|
||||||
|
"properties": {
|
||||||
|
"result": {
|
||||||
|
"properties": {
|
||||||
|
"error": {
|
||||||
|
"minimum": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"error"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"result"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"then": {
|
||||||
|
"required": [
|
||||||
|
"error_cases"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"result": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"total",
|
||||||
|
"passed",
|
||||||
|
"failed",
|
||||||
|
"not_applicable",
|
||||||
|
"error"
|
||||||
|
],
|
||||||
|
"$comment": "All five counts are REQUIRED, including the ones that are zero. An absent count is indistinguishable from a count the runtime did not track, and the `denominator_intact` invariant cannot be checked against a declaration with holes in it. `0` is information; a missing key is not.",
|
||||||
|
"properties": {
|
||||||
|
"total": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0,
|
||||||
|
"description": "Cases enumerated. MUST equal the corpus case count at `commons_commit`, and MUST equal the sum of the four verdict counts. See `invariants`."
|
||||||
|
},
|
||||||
|
"passed": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0
|
||||||
|
},
|
||||||
|
"failed": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0
|
||||||
|
},
|
||||||
|
"not_applicable": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0,
|
||||||
|
"description": "Cases whose `scope` names a data file outside `declared_tables`. MUST NOT be counted as passes and MUST NOT be dropped from `total`."
|
||||||
|
},
|
||||||
|
"error": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0,
|
||||||
|
"description": "Cases the runtime tried to run and could not, per section 1. Distinct from `not_applicable`."
|
||||||
|
},
|
||||||
|
"match": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "The match semantics the run used, if the corpus offers more than one (today: `exact-within-scope`). Informative; the per-case `match` field in each expected.json is the authority."
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"caseIdList": {
|
||||||
|
"type": "array",
|
||||||
|
"uniqueItems": true,
|
||||||
|
"items": {
|
||||||
|
"type": "string",
|
||||||
|
"pattern": "^[a-z0-9][a-z0-9-]*__[a-z0-9-]+$"
|
||||||
|
},
|
||||||
|
"$comment": "Case ids as they appear on disk under conformance/, which per conformance/manifest.json are derived from a pattern id by replacing ':' with '__'. The pattern here enforces that shape rather than the derivation itself, since the derivation is the manifest's contract and not this schema's."
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"example_$comment": "Illustrative only, not a record of any measurement. The counts are the shape llm-security stated it will publish once it pulls a commons carrying section 1.1 - see requested_by - and are NOT a result this repository has been shown. This note sits OUTSIDE `example` on purpose: `declaration` closes additionalProperties, so an example carrying its own $comment would not validate against the schema it illustrates.",
|
||||||
|
"example": {
|
||||||
|
"runtime": "llm_security",
|
||||||
|
"runtime_commit": "b1ba1fb",
|
||||||
|
"commons_commit": "0000000",
|
||||||
|
"commons_version": "v0.3.0",
|
||||||
|
"declared_tables": [
|
||||||
|
"lexicon/injection-lexicon.json"
|
||||||
|
],
|
||||||
|
"declaration_source": "derived-from-runner",
|
||||||
|
"measured_date": "2026-08-11",
|
||||||
|
"result": {
|
||||||
|
"total": 89,
|
||||||
|
"passed": 83,
|
||||||
|
"failed": 0,
|
||||||
|
"not_applicable": 6,
|
||||||
|
"error": 0,
|
||||||
|
"match": "exact-within-scope"
|
||||||
|
},
|
||||||
|
"not_applicable_cases": [
|
||||||
|
"active__autolink",
|
||||||
|
"active__data-uri",
|
||||||
|
"active__markdown-image",
|
||||||
|
"active__markdown-link",
|
||||||
|
"active__raw-html",
|
||||||
|
"active__reference-link"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -48,6 +48,22 @@ Two constraints keep the verdict from becoming an exit:
|
||||||
MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly
|
MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly
|
||||||
the silent skip §1 forbids.
|
the silent skip §1 forbids.
|
||||||
|
|
||||||
|
**The shape of the declaration.** A runtime publishing a declaration MUST use the shape in
|
||||||
|
[`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json):
|
||||||
|
runtime, the commons commit measured, the declared table set, and the four verdict counts with
|
||||||
|
their total. The requirement is the shape, not a validation step — nothing in this repository
|
||||||
|
runs, so no runtime is gated on it. It exists because a free-form declaration makes
|
||||||
|
`83 passed, 6 not-applicable` unparseable by anyone but its author, and a result only its
|
||||||
|
author can read is not a cross-runtime measurement. That schema also carries the two arithmetic
|
||||||
|
invariants this section implies but cannot state in prose without ambiguity: the counts MUST sum
|
||||||
|
to the total, and the total MUST equal the corpus case count at the commit measured.
|
||||||
|
|
||||||
|
A runtime SHOULD derive its declared set from the same constant its runner uses to accept or
|
||||||
|
reject a `scope`, rather than maintain a second list beside it, and SHOULD record which of the
|
||||||
|
two it did. The anti-narrowing constraint above is then structural rather than promised:
|
||||||
|
narrowing the declaration is narrowing what the runtime accepts, which is a change to code with
|
||||||
|
tests behind it instead of an edit to a published file nobody re-reads.
|
||||||
|
|
||||||
**What the verdict does not say.** `not-applicable` records that a runtime does not implement
|
**What the verdict does not say.** `not-applicable` records that a runtime does not implement
|
||||||
a named commons table. It is not a statement that the runtime is blind to the phenomenon
|
a named commons table. It is not a statement that the runtime is blind to the phenomenon
|
||||||
behind it — a runtime may well detect the same construct through some other mechanism that
|
behind it — a runtime may well detect the same construct through some other mechanism that
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue