feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement

The corpus was blocked on a decision nobody had to take. The 13 divergent
lexicon patterns were measured on witness inputs -- an attribute run padded
past 256 characters, an interior '<', an unclosed <script> -- and the corpus
payloads contain none of those shapes. Run through both runtimes' public
entry points, all 83 patterns produce identical lexicon finding sets, the 13
included. No winner picked, because the question was never reachable from
these inputs.

Measured at each runtime's entry point (scanForInjection() at b0de0ca,
scan_output(source=OUTPUT) at 0bf0729), never at a rebuilt regex table --
the layer mistake the divergence document already had to retract once.

- conformance/<case>/{input.txt,expected.json} x83, plus manifest.json
- spec/conformance-corpus.md, normative: bytes not text, id-only findings,
  exact-within-scope, and observed_out_of_scope as evidence not expectation

Verified by a harness that does not share the generator's knowledge: reads
only the case directories, re-runs both runtimes, checks every field
including digests -- 83 cases, 0 failures. Severity agrees with what commons
publishes 83/83. Deleting the middle third of each input breaks 76 of 83
expectations; the 7 survivors are the shortest payloads, a weak mutation
rather than a weak fixture, and are recorded as such.

Scope is 83 and not 94 for a different reason than expected: the 11
non-lexicon convertible cases have no ratified cross-runtime finding id, and
writing them would mint a contract unilaterally in the same stroke as the
tag. Named in manifest.json under scope_planned.

Docs corrected in place rather than edited away: extraction-plan and
lexicon-port-divergence both claimed the 13 blocked the corpus.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 04:40:56 +02:00
commit 49e1e79807
172 changed files with 2187 additions and 54 deletions

View file

@ -9,14 +9,59 @@ Versioning note: the repository tag versions **the contract** (file set, key nam
case ids, disposition semantics). Each JSON file additionally carries its own
`"version"` field, bumped when that file changes.
## [Unreleased]
## [0.1.0] — 2026-08-10
Initial extraction, in progress. Runtime-neutral detection data and the finding contract,
extracted from the `llm-security` Node implementation and a Python guard without behaviour
change. **Not yet tagged** — see *Not included* below.
Initial extraction. Runtime-neutral detection data, the finding contract, and a conformance
corpus, extracted from the `llm-security` Node implementation and a Python guard **without
behaviour change** — that invariant is the release, not a caveat on it.
What the tag is worth resting on: seven of the eight JSON artefacts were rebuilt from the
commons file alone and diffed against their source implementation, three of them against the
source module at a pinned commit. The eighth says `verified: false` about itself. The corpus
holds 83 cases on which both seeding runtimes were measured agreeing exactly.
What it is not: `spec/decode-pipeline.md` does not exist, and the corpus constrains one of
the seven data files. Both absences are named in *Not included* rather than papered over.
### Added
- `conformance/`**83 cases, one per injection-lexicon pattern**, plus `manifest.json`.
Each case is a directory holding `input.txt` (the exact bytes, no trailing newline) and
`expected.json` (the findings, named by commons pattern `id`).
Both seeding runtimes were measured producing the **same lexicon finding set on all 83**,
through their public entry points — `scanForInjection()` at `b0de0ca` and
`scan_output(source=OUTPUT)` at `0bf0729` — with labels mapped to commons ids through the
lexicon's own `aliases` block. Not through rebuilt regex tables: a table-level comparison
yields a number that describes neither runtime, which is the mistake the divergence
document had to retract.
**The 13 divergent patterns are in, unmarked, and that is the substantive result.** Their
divergence was measured on witness inputs — an attribute run padded past 256 characters,
an interior `<`, an unclosed `<script>` — and none of those shapes occurs in a corpus
payload. All 13 agree on their own case input. Nobody had to pick whose recall cost
becomes the contract, because the question was never reachable from these inputs. A
per-case caveat would have asserted a doubt the measurement disproves.
Inputs are the guard's `coverage.py` payloads, reproduced verbatim. One runtime authored
them; what makes them a cross-runtime corpus is the measurement through the other, and the
manifest records the asymmetry rather than averaging it away.
- `spec/conformance-corpus.md`**normative.** How a case is read: `input.txt` is bytes and
is not to be trimmed or re-encoded, `expected.json` names findings by `id` only (severity
and OWASP anchor are looked up in the lexicon, never restated), and `exact-within-scope`
requires equality **restricted to the data files the case names**.
Scoping is what makes exactness safe — the two runtimes do not implement the same set of
tables, so a whole-report comparison would fail for reasons unrelated to the pattern under
test. Exactness is what makes the corpus worth running — a contains-only corpus is passed
by a runtime that flags everything. `observed_out_of_scope` is evidence, never expectation,
and an absent runtime key means **unmeasured**, not measured-empty.
The document also states the one place this repository's "every JSON file carries a
top-level `version`" convention does not apply: fixtures are versioned as a corpus, in
`conformance/manifest.json`. Stated rather than left to be discovered.
- `schema/finding.schema.json` — the finding contract plus the SARIF output profile.
Normative. Closed against the producer in 0.2.0; the JSONL profile is `not applicable`.
- `signatures/active-content.json` — the EchoLeak class (CVE-2025-32711): 17 patterns,
@ -68,6 +113,15 @@ file records its own result and its own limits.
summary rather than as code, so no differential check was possible, and the file names the
checks that were not run instead of attaching a caveat to a pass.
The corpus was verified the same way the data was — by a harness that does **not** share the
generator's knowledge. It reads only the case directories, re-runs both runtimes on the bytes
it finds there, and checks every field of every `expected.json`, digests included: **83
cases, 0 failures**. Two further checks, because a corpus that cannot fail is not evidence:
commons' family severity matches the severity the guard emits per finding, **83/83**; and
deleting the middle third of each input breaks **76 of 83** expectations. The 7 survivors are
the shortest payloads, where the mutation leaves the trigger intact — that is a weak
mutation, not a weak fixture, and it is recorded as such rather than rounded up.
- `docs/lexicon-port-divergence.md` — informative. A differential comparison of the two
ports of `injection-patterns.mjs` (this repository's and the Python guard's): 83/83
patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven
@ -89,9 +143,14 @@ checks that were not run instead of attaching a caveat to a pass.
Also corrected: one of the 13 divergences does not reach report level, because the guard's
`hybrid-xss:javascript-uri` fires on the same witness at the same severity and anchor. The
report-level number is **12**; the 13 stands as the number blocking `conformance/`, since a
fixture names labels. And the `hybrid` severity question that the document reported rather
than resolved is now closed — the reported hint was right, the citation behind it was not.
report-level number is **12**. And the `hybrid` severity question that the document reported
rather than resolved is now closed — the reported hint was right, the citation behind it was
not.
**Revised again 2026-08-10.** The document said 13 was the number blocking `conformance/`,
since a fixture names labels. It blocks a fixture written over a **witness** input, and the
corpus contains none — all 13 agree on their own case input. The divergence itself stands
unresolved and unchanged; what was wrong was the claim about what it blocked.
Corrections are marked in place rather than edited away.
@ -232,12 +291,14 @@ checks that were not run instead of attaching a caveat to a pass.
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
from a data dump would be worse than an absent one.
- `conformance/` — still absent, with half the blocker cleared. 105 of the guard's 134
coverage cases are convertible to static `input.txt`/`expected.json`; the other 29 assert
a runtime's API surface, which this repository does not own. Findings can now be **named**
(see `pattern_id_space` above), but 13 patterns still have no agreed **expected
behaviour** — the two ports genuinely differ on them — so those fixtures cannot be
authored until the owning repositories answer.
- **Conformance for the other four tables.** The corpus covers the injection lexicon only.
The carrier, active-content and secret-egress tables have 11 convertible cases waiting in
the guard's matrix, and no ratified cross-runtime finding id between them — writing those
fixtures would mint a contract unilaterally, in the same stroke as the tag. Named in
`conformance/manifest.json` under `scope_planned`.
- The 29 non-convertible cases of the guard's 134 assert a runtime's **API surface** — that
a Python call raises `OKFPathError`, that a disposition engine composes two findings a
particular way. This repository does not own an API, so those belong to the guard's suite.
These are named in the README as planned rather than linked, so nothing in the repository
points at a file that does not exist.
`spec/decode-pipeline.md` is named in the README as planned rather than linked, so nothing
in the repository points at a file that does not exist.