feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement
The corpus was blocked on a decision nobody had to take. The 13 divergent
lexicon patterns were measured on witness inputs -- an attribute run padded
past 256 characters, an interior '<', an unclosed <script> -- and the corpus
payloads contain none of those shapes. Run through both runtimes' public
entry points, all 83 patterns produce identical lexicon finding sets, the 13
included. No winner picked, because the question was never reachable from
these inputs.
Measured at each runtime's entry point (scanForInjection() at b0de0ca,
scan_output(source=OUTPUT) at 0bf0729), never at a rebuilt regex table --
the layer mistake the divergence document already had to retract once.
- conformance/<case>/{input.txt,expected.json} x83, plus manifest.json
- spec/conformance-corpus.md, normative: bytes not text, id-only findings,
exact-within-scope, and observed_out_of_scope as evidence not expectation
Verified by a harness that does not share the generator's knowledge: reads
only the case directories, re-runs both runtimes, checks every field
including digests -- 83 cases, 0 failures. Severity agrees with what commons
publishes 83/83. Deleting the middle third of each input breaks 76 of 83
expectations; the 7 survivors are the shortest payloads, a weak mutation
rather than a weak fixture, and are recorded as such.
Scope is 83 and not 94 for a different reason than expected: the 11
non-lexicon convertible cases have no ratified cross-runtime finding id, and
writing them would mint a contract unilaterally in the same stroke as the
tag. Named in manifest.json under scope_planned.
Docs corrected in place rather than edited away: extraction-plan and
lexicon-port-divergence both claimed the 13 blocked the corpus.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
This commit is contained in:
parent
ff18768b71
commit
49e1e79807
172 changed files with 2187 additions and 54 deletions
|
|
@ -66,8 +66,10 @@ the same severity and the same OWASP anchor, so a reader of the guard's report s
|
|||
`high` / `LLM01` finding on that payload. Measured, not reasoned: the guard's table matched
|
||||
`hybrid-xss:javascript-uri` (high, LLM01) and nothing else; the Node engine matched both
|
||||
`hybrid-xss: javascript: URI scheme` and `hybrid-xss: iframe with executable src`. A
|
||||
`conformance/expected.json` still has to encode the difference, because a fixture names
|
||||
labels — which is why the 13 stands as the number that blocks the corpus.
|
||||
`conformance/expected.json` would have to encode the difference, because a fixture names
|
||||
labels — which is why the 13 stood as the number that blocked the corpus. *(Superseded
|
||||
2026-08-10: it blocks a fixture written over a **witness** input, and the corpus contains
|
||||
none. See* Consequence for `conformance/` *below.)*
|
||||
|
||||
The 6 escaping-only pairs are equivalent by **string identity after unescaping**, not merely
|
||||
by absence of a counter-example: `<\/system>` vs `</system>`, `text\/html` vs `text/html`,
|
||||
|
|
@ -241,12 +243,29 @@ including the 0.2.0 proposal, and treats an id change as breaking on the same te
|
|||
guard confirmed the space its own port supplied. `lexicon/injection-lexicon.json` 0.5.0
|
||||
records both. The id is a cross-runtime contract now, not a proposal.
|
||||
|
||||
**The second half of the blocker stands, and it did not get smaller.** The 13 divergent
|
||||
patterns still have no agreed expected behaviour, so their fixtures cannot be authored even
|
||||
though they can now be named. `llm-security` has decided **not** to adopt the guard's regex
|
||||
strategy at this point: v0.1.0 is a behaviour-preservation release on their side too, and
|
||||
swapping strategies mid-vendoring would void their own golden gate. Both behaviours are
|
||||
therefore registered as known divergence per pattern. Their three confirmed recall holes are
|
||||
logged as a v8.x task; when it lands they will say so, and those rows can close then. Until
|
||||
then the correct description of each is *"known divergence, `llm-security` side has an open
|
||||
recall hole, measured 2026-08-09"* — not *"undecided"*.
|
||||
~~**The second half of the blocker stands, and it did not get smaller.**~~ **Dissolved
|
||||
2026-08-10 by measurement, not by a decision.** This paragraph said the 13 divergent
|
||||
patterns had no agreed expected behaviour, so their fixtures could not be authored, and that
|
||||
someone would have to pick whose recall cost was the contract.
|
||||
|
||||
Nobody had to. The question was never asked of the right inputs. Every divergence in the
|
||||
table above was found on a **witness** input — an attribute run padded past 256 characters,
|
||||
an interior `<`, an unclosed `<script>`. The corpus is built from the seed suite's payloads,
|
||||
which are short, unpadded and contain none of those shapes. Run through both runtimes'
|
||||
public entry points, all 83 patterns produce **identical lexicon finding sets**, and that
|
||||
includes 13 of 13 of the divergent ones on their own case input. Method, commits and counts:
|
||||
[`conformance/manifest.json`](../conformance/manifest.json).
|
||||
|
||||
So the 13 carry no marker in the corpus and no caveat. Marking them would assert a doubt the
|
||||
measurement disproves for these inputs, which is a different defect from the one it would
|
||||
appear to prevent.
|
||||
|
||||
**What still stands is everything above this heading.** The divergence is real, it is
|
||||
unresolved, and it will reappear the moment a fixture is written over a witness input.
|
||||
`llm-security` has decided **not** to adopt the guard's regex strategy at this point: v0.1.0
|
||||
is a behaviour-preservation release on their side too, and swapping strategies mid-vendoring
|
||||
would void their own golden gate. Both behaviours stay registered as known divergence per
|
||||
pattern. Their three confirmed recall holes are logged as a v8.x task; when it lands they
|
||||
will say so, and those rows can close then. Until then the correct description of each is
|
||||
*"known divergence, `llm-security` side has an open recall hole, measured 2026-08-09; not
|
||||
reachable from any input in the v0.1.0 corpus"* — not *"undecided"*, and not *"resolved"*.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue