feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement
The corpus was blocked on a decision nobody had to take. The 13 divergent
lexicon patterns were measured on witness inputs -- an attribute run padded
past 256 characters, an interior '<', an unclosed <script> -- and the corpus
payloads contain none of those shapes. Run through both runtimes' public
entry points, all 83 patterns produce identical lexicon finding sets, the 13
included. No winner picked, because the question was never reachable from
these inputs.
Measured at each runtime's entry point (scanForInjection() at b0de0ca,
scan_output(source=OUTPUT) at 0bf0729), never at a rebuilt regex table --
the layer mistake the divergence document already had to retract once.
- conformance/<case>/{input.txt,expected.json} x83, plus manifest.json
- spec/conformance-corpus.md, normative: bytes not text, id-only findings,
exact-within-scope, and observed_out_of_scope as evidence not expectation
Verified by a harness that does not share the generator's knowledge: reads
only the case directories, re-runs both runtimes, checks every field
including digests -- 83 cases, 0 failures. Severity agrees with what commons
publishes 83/83. Deleting the middle third of each input breaks 76 of 83
expectations; the 7 survivors are the shortest payloads, a weak mutation
rather than a weak fixture, and are recorded as such.
Scope is 83 and not 94 for a different reason than expected: the 11
non-lexicon convertible cases have no ratified cross-runtime finding id, and
writing them would mint a contract unilaterally in the same stroke as the
tag. Named in manifest.json under scope_planned.
Docs corrected in place rather than edited away: extraction-plan and
lexicon-port-divergence both claimed the 13 blocked the corpus.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
This commit is contained in:
parent
ff18768b71
commit
49e1e79807
172 changed files with 2187 additions and 54 deletions
29
README.md
29
README.md
|
|
@ -58,8 +58,9 @@ and keeping it that small is the point.
|
|||
| [`calibration/calibration.json`](calibration/calibration.json) | The numbers a detector must not invent: risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds. Transcribed from a prose summary, not differentially verified — the file says so itself. |
|
||||
| [`mapping/owasp-map.json`](mapping/owasp-map.json) | Finding-id prefix → OWASP taxonomy entry (LLM / ASI / AST / MCP). |
|
||||
| [`schema/finding.schema.json`](schema/finding.schema.json) | **Normative.** The finding contract — closed against its producer, ten properties — plus the SARIF output profile. The JSONL profile is recorded as `not applicable`, with the reason. |
|
||||
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, and what `exact-within-scope` requires of a runtime. |
|
||||
| [`conformance/`](conformance/) | 83 cases, one per injection-lexicon pattern. One directory per case: `input.txt` in, `expected.json` out. Ground truth. Both seeding runtimes were measured producing the same verdict on all 83 — see [`conformance/manifest.json`](conformance/manifest.json). |
|
||||
| `spec/decode-pipeline.md` | **Planned, not in v0.1.0.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
||||
| `conformance/` | **Planned, not in v0.1.0.** One directory per case: `input.txt` in, `expected.json` out. Ground truth. |
|
||||
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
||||
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions, and why no data file was changed because of it. |
|
||||
|
||||
|
|
@ -74,14 +75,26 @@ transcribed from a prose summary rather than diffed against a running implementa
|
|||
|
||||
### How a consumer proves it conforms
|
||||
|
||||
Run every `conformance/<case>/input.txt` through your detector, serialize the result per
|
||||
[`schema/finding.schema.json`](schema/finding.schema.json), and compare to
|
||||
`expected.json`. Disagreement means your runtime is wrong, or the fixture is — and the
|
||||
fixture only changes in its own commit, with the reason written down.
|
||||
Run every `conformance/<case>/input.txt` through your detector and compare the finding ids
|
||||
to `expected.json` — exactly, but only within the data files the case names in `scope`.
|
||||
[`spec/conformance-corpus.md`](spec/conformance-corpus.md) is the normative reading;
|
||||
the short version is that a runtime must raise every listed finding and no other finding
|
||||
*from the same table*, and that what it does with tables outside the case's scope is not
|
||||
compared.
|
||||
|
||||
Disagreement means your runtime is wrong, or the fixture is — and the fixture only changes
|
||||
in its own commit, with the reason written down.
|
||||
|
||||
There is **no CI in this organisation** and nothing runs that comparison automatically. It
|
||||
runs in each consumer's own test suite, against a pinned tag.
|
||||
|
||||
The v0.1.0 corpus covers `lexicon/injection-lexicon.json`, the one table whose finding ids
|
||||
are ratified by both seeding runtimes. Eleven further cases exist in a seed suite for the
|
||||
carrier, active-content and secret-egress tables and are **not** shipped: naming a finding
|
||||
in those tables would mean minting a cross-runtime id space no runtime has agreed to.
|
||||
`conformance/manifest.json` names them under `scope_planned` so the gap is visible rather
|
||||
than inferred.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Not a scanner.** There is no engine here, and there will not be one. If you are looking
|
||||
|
|
@ -100,7 +113,11 @@ runs in each consumer's own test suite, against a pinned tag.
|
|||
## Known limitations
|
||||
|
||||
- **Coverage is the union of what the seed implementations detected**, not of what exists.
|
||||
A class absent from `conformance/` has not been shown to work anywhere.
|
||||
A class absent from the tables above has not been shown to work anywhere.
|
||||
- **The corpus is narrower than the data.** `conformance/` constrains one of the seven data
|
||||
files. The other six are published, provenance-checked and unfixtured: a runtime can
|
||||
pass every case and still read `calibration.json` wrongly. Passing the corpus is evidence
|
||||
about the injection lexicon and about nothing else.
|
||||
- **Regex portability is a real risk.** Pattern data is written for a common subset, but
|
||||
engines differ (lookbehind, named groups, Unicode property escapes). A consumer whose
|
||||
engine rejects a pattern must report it rather than silently skip it — a skipped pattern
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue