open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

feat(conformance): build the 83-case corpus; the 13-pattern blocker dissolves under measurement

Browse source 
The corpus was blocked on a decision nobody had to take. The 13 divergent
lexicon patterns were measured on witness inputs -- an attribute run padded
past 256 characters, an interior '<', an unclosed <script> -- and the corpus
payloads contain none of those shapes. Run through both runtimes' public
entry points, all 83 patterns produce identical lexicon finding sets, the 13
included. No winner picked, because the question was never reachable from
these inputs.

Measured at each runtime's entry point (scanForInjection() at b0de0ca,
scan_output(source=OUTPUT) at 0bf0729), never at a rebuilt regex table --
the layer mistake the divergence document already had to retract once.

- conformance/<case>/{input.txt,expected.json} x83, plus manifest.json
- spec/conformance-corpus.md, normative: bytes not text, id-only findings,
  exact-within-scope, and observed_out_of_scope as evidence not expectation

Verified by a harness that does not share the generator's knowledge: reads
only the case directories, re-runs both runtimes, checks every field
including digests -- 83 cases, 0 failures. Severity agrees with what commons
publishes 83/83. Deleting the middle third of each input breaks 76 of 83
expectations; the 7 survivors are the shortest payloads, a weak mutation
rather than a weak fixture, and are recorded as such.

Scope is 83 and not 94 for a different reason than expected: the 11
non-lexicon convertible cases have no ratified cross-runtime finding id, and
writing them would mint a contract unilaterally in the same stroke as the
tag. Named in manifest.json under scope_planned.

Docs corrected in place rather than edited away: extraction-plan and
lexicon-port-divergence both claimed the 13 blocked the corpus.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 04:40:56 +02:00
commit 49e1e79807
172 changed files with 2187 additions and 54 deletions
Download patch file Download diff file Expand all files Collapse all files

74
docs/extraction-plan.md
View file

@ -329,9 +329,12 @@ then be testing something this repository does not define.
| `decoded` | 2 | **No.** These assert a finding *after* a decode pass. The decode order is exactly what `spec/decode-pipeline.md` would fix, and it is Planned because it is engine. |
| `okf` (`_scan_case_okf`) | 2 | No — the OKF document model is not published here. |
So the buildable set is smaller than 105: roughly 83 + 7 + 3 + 1 ≈ 94, minus the 13 lexicon
patterns whose expected behaviour is still disputed. The rest are named here so a later session
does not rediscover them as a gap.
So the buildable set is smaller than 105: roughly 83 + 7 + 3 + 1 ≈ 94. The rest are named here
so a later session does not rediscover them as a gap. *(Revised 2026-08-10: this sentence
ended "minus the 13 lexicon patterns whose expected behaviour is still disputed". Measurement
showed the dispute does not reach these inputs, so the 13 are in. What kept the built corpus
at 83 rather than 94 turned out to be a different constraint entirely — the 11 non-lexicon
cases have no ratified cross-runtime finding id. See* Conformance corpus *below.)*
**Blocking issue found while counting.** The guard and the Node table are two ports of one
source file and they do not agree: 64 of 83 patterns byte-identical, 6 differing only by
@ -345,9 +348,11 @@ owning repositories instead.
**Half of it is now cleared (operator decision, 2026-08-09).**
`lexicon/injection-lexicon.json` 0.2.0 publishes a commons-owned `id` per pattern with an
`aliases` object per runtime, adopted verbatim from the guard's port rather than invented.
Findings can therefore be named. What still blocks the corpus is the other half: the 13
Findings can therefore be named. ~~What still blocks the corpus is the other half: the 13
divergent patterns have no agreed expected behaviour, and a fixture for them would have to
pick a winner.
pick a winner.~~ **The other half cleared on 2026-08-10 without anyone picking a winner** —
the divergence is not reachable from any input the corpus contains. See *Conformance corpus*
below.
## Definition of done for v0.1.0
@ -375,22 +380,51 @@ pick a winner.
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
from a data dump would carry a `Status: normative` marker and pass every convention
check in this document while asserting something nobody verified.
- `conformance/` — **half-unblocked.** The naming blocker is gone: the pattern id space
is ratified by both runtimes as of 2026-08-09, so a fixture can name a finding. What
remains is expected behaviour for the 13 patterns whose two ports genuinely differ.
`llm-security` has decided not to change strategy inside a behaviour-preservation
release, so those 13 stay divergent for now and are registered as known divergence per
pattern rather than resolved. Of the guard's 134 coverage cases, 105 are convertible to
static `input.txt` / `expected.json`; the other 29 assert a runtime's API surface, which
this repository does not own.
- ~~`conformance/`~~ — **built 2026-08-10, 83 cases.** See *Conformance corpus* below.
They are absent rather than stubbed on purpose: an empty `conformance/` and a
contentless normative spec would both *pass* the mechanical checks in this document
while making the repository look more finished than it is. They are named as **Planned**
in the README instead, and are not linked, so nothing points at a file that does not
exist. Whether v0.1.0 ships without a conformance corpus is an operator decision, not a
session one: the README's own pitch is that consumers can be held to the same answer on
the same input, and a tag with zero cases cannot deliver that.
`spec/decode-pipeline.md` is absent rather than stubbed on purpose: a contentless
normative spec would *pass* every mechanical check in this document while making the
repository look more finished than it is. It is named as **Planned** in the README
instead, and is not linked, so nothing points at a file that does not exist.
### Conformance corpus — built 2026-08-10
The remaining blocker was the 13 divergent patterns, and it **dissolved under measurement
rather than under a decision**. The divergence was measured on witness inputs — padding past
256 characters, an interior `<`, an unclosed `<script>` — and the corpus payloads contain
none of those shapes. Run through both runtimes' public entry points, all 83 produce
identical lexicon finding sets, 13 of 13 divergent patterns included.
| Check | Method | Result |
| --- | --- | --- |
| Payload id set equals the commons id set | compared the 83 `_LEX_PAYLOADS` keys to the 83 lexicon ids | identical, both directions empty |
| Asserted id fires in both runtimes | `scanForInjection()` at `b0de0ca` and `scan_output(source=OUTPUT)` at `0bf0729`, labels mapped to ids through the lexicon's own `aliases` | **83/83** |
| Lexicon finding **sets** identical between runtimes | set equality per case, not membership | **83/83** |
| The 13 known-divergent patterns, on their own case input | same comparison, restricted to those 13 | **13/13 agree** |
| Severity agrees with what commons publishes | commons family `severity` vs the severity the guard emitted per finding | **83/83** |
| The corpus verifies from the corpus alone | a second harness that reads only the case directories, re-runs both runtimes on the bytes it finds, and checks every `expected.json` field including the digests | **83 cases, 0 failures** |
| The fixtures discriminate | deleted the middle third of each input and re-scanned; a fixture that still passes is not testing anything | 76/83 broke. The 7 survivors are the shortest payloads, where the mutation leaves the trigger intact — a weak mutation, not a weak fixture |
| Case ids are collision-free after the filesystem transform | `:` → `__` across all 83 | 83 unique; `__` occurs nowhere in the id space, so the transform is one-to-one |
Measured at the **public entry point of each runtime**, never at a rebuilt regex table. That
is the same discipline the retraction in [`lexicon-port-divergence.md`](lexicon-port-divergence.md)
was forced to adopt: a measurement taken at the wrong layer produces a number that describes
neither runtime.
**Scope, and why it is 83 and not 94.** The guard's matrix holds 105 convertible cases, of
which ~94 assert a label that maps to data this repository publishes. Only 83 shipped. The
lexicon is the sole table whose finding id space is **ratified by both runtimes**; the
carrier, active-content and secret-egress tables have no agreed cross-runtime finding names,
so writing fixtures for their 11 cases would mint a contract in the same stroke as the tag,
unilaterally. They are named in `conformance/manifest.json` under `scope_planned`.
The inputs were authored by one runtime — the guard's `coverage.py` — and are reproduced
verbatim. What makes them usable as a *cross-runtime* corpus is not their origin but the
measurement above, which ran them through the other runtime too. The asymmetry is recorded
in the manifest rather than averaged away.
Normative semantics live in [`spec/conformance-corpus.md`](../spec/conformance-corpus.md),
not here.
3. All JSON well-formed, every data file carrying `"version"`, every spec carrying
`Status: normative`.
4. Tagged `v0.1.0` and pushed.