open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

fix(conformance): name the fixture field pattern_id, not id — the schema already owns that word

Browse source 
schema/finding.schema.json defines a finding `id` as DS-<scanner>-<counter>,
built from a process-global counter: stable across neither runs nor
processes, and the schema says so itself. The corpus keys its comparison on
the lexicon's stable rule identity. Two normative documents in one
repository using one word for both would produce runtimes failing every case
for a reason unrelated to detection.

Also adds spec section 3.1, which publishes the bridge a consumer actually
needs and which neither normative document named: a runtime's own label
reaches a pattern_id through the lexicon's `aliases` object, and a runtime
absent from that object has no published way to be compared -- a mapping
kept privately in a consumer is the drift this repository exists to prevent.

Regenerated all 83 fixtures; re-verified from the corpus alone against both
runtimes, 83 cases, 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 04:44:59 +02:00
commit 0ffee85a4b
85 changed files with 146 additions and 98 deletions
Download patch file Download diff file Expand all files Collapse all files

15
CHANGELOG.md
View file

@ -48,9 +48,18 @@ the seven data files. Both absences are named in *Not included* rather than pape
manifest records the asymmetry rather than averaging it away.
- `spec/conformance-corpus.md` — **normative.** How a case is read: `input.txt` is bytes and
is not to be trimmed or re-encoded, `expected.json` names findings by `id` only (severity
and OWASP anchor are looked up in the lexicon, never restated), and `exact-within-scope`
requires equality **restricted to the data files the case names**.
is not to be trimmed or re-encoded, `expected.json` names findings by `pattern_id` only
(severity and OWASP anchor are looked up in the lexicon, never restated), and
`exact-within-scope` requires equality **restricted to the data files the case names**.
The field is `pattern_id`, not `id`, because this repository already publishes an unrelated
finding `id`: `schema/finding.schema.json` defines it as `DS-<scanner>-<counter>` from a
process-global counter — stable across neither runs nor processes. Two normative documents
using one word for a stable rule identity and a volatile per-emission sequence number would
have produced runtimes failing every case for reasons unrelated to detection. §3.1 states
the distinction and publishes the bridge a runtime actually needs: its own label maps to a
`pattern_id` through the lexicon's `aliases` object, and a runtime absent from that object
has no published way to be compared at all.
Scoping is what makes exactness safe — the two runtimes do not implement the same set of
tables, so a whole-report comparison would fail for reasons unrelated to the pattern under