open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

fix(conformance): name the fixture field pattern_id, not id — the schema already owns that word

Browse source 
schema/finding.schema.json defines a finding `id` as DS-<scanner>-<counter>,
built from a process-global counter: stable across neither runs nor
processes, and the schema says so itself. The corpus keys its comparison on
the lexicon's stable rule identity. Two normative documents in one
repository using one word for both would produce runtimes failing every case
for a reason unrelated to detection.

Also adds spec section 3.1, which publishes the bridge a consumer actually
needs and which neither normative document named: a runtime's own label
reaches a pattern_id through the lexicon's `aliases` object, and a runtime
absent from that object has no published way to be compared -- a mapping
kept privately in a consumer is the drift this repository exists to prevent.

Regenerated all 83 fixtures; re-verified from the corpus alone against both
runtimes, 83 cases, 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 04:44:59 +02:00
commit 0ffee85a4b
85 changed files with 146 additions and 98 deletions
Download patch file Download diff file Expand all files Collapse all files

49
spec/conformance-corpus.md
View file

@ -40,25 +40,64 @@ means nothing.
"input": { "file": "input.txt", "bytes": 24, "sha256": "…" },
"scope": ["lexicon/injection-lexicon.json"],
"match": "exact-within-scope",
"findings": [{ "id": "hybrid-xss:script-tag" }],
"findings": [{ "pattern_id": "hybrid-xss:script-tag" }],
"observed_out_of_scope": { "llm_ingestion_guard": ["active:raw-html"] }
}
```
| Field | Normative? | Meaning |
| --- | --- | --- |
| `$comment` | no | Provenance prose. Carries no contract. |
| `case_id` | yes | Equals the directory name. |
| `input` | yes | Integrity of the input file. See §2. |
| `scope` | yes | Which commons data files this case constrains. See §4. |
| `match` | yes | How `findings` is to be compared. See §4. |
| `findings` | yes | The expected findings, named by commons pattern `id`. |
| `findings` | yes | The expected findings, named by commons `pattern_id`. See §3.1. |
| `observed_out_of_scope` | **no — informative** | What a named runtime was measured emitting outside `scope`. Evidence, never expectation. See §5. |
A finding carries an `id` and nothing else. Severity, OWASP anchor and human label are
properties of the pattern, published once in [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json)
and looked up there by `id`. They MUST NOT be restated in a fixture: a value written in
A finding carries a `pattern_id` and nothing else. Severity, OWASP anchor and human label
are properties of the pattern, published once in [`lexicon/injection-lexicon.json`](../lexicon/injection-lexicon.json)
and looked up there by that id. They MUST NOT be restated in a fixture: a value written in
two places is a value that will eventually disagree with itself.
### 3.1 `pattern_id` is not the finding `id`
The field is called `pattern_id` and not `id` because this repository already publishes a
different `id`, and confusing the two produces a runtime that fails every case for a reason
unrelated to detection.
| | Identifies | Stable? |
| --- | --- | --- |
| `pattern_id` — here; `families[].patterns[].id` in the lexicon | the **detection rule** | Yes. A stable cross-runtime contract, ratified by both seeding runtimes. |
| `id` — in [`schema/finding.schema.json`](../schema/finding.schema.json) | one **emitted finding** in one run | No. `DS-<scanner>-<counter>`, from a process-global counter: not stable across runs, not unique across processes. The schema says so itself. |
A runtime therefore MUST NOT compare a serialized finding's `id` to a fixture's
`pattern_id`. They answer different questions.
**Getting from a runtime's own label to a `pattern_id`.** Nothing requires a runtime to name
its patterns the way this repository does, and the two seeding runtimes do not: the same rule
is `override: ignore previous instructions` in one and `override:ignore-previous` in the
other. The mapping is published, per pattern, in the lexicon's `aliases` object:
```json
{
"id": "override:ignore-previous",
"aliases": {
"llm_security": "override: ignore previous instructions",
"llm_ingestion_guard": "override:ignore-previous"
}
}
```
(The lexicon's own field is spelled `id`, because that name is ratified and renaming it
would break both consumers. `pattern_id` here refers to exactly that value.)
A runtime SHOULD carry that id directly on its own pattern table, which makes the
comparison a lookup rather than a translation. A runtime that instead maps through `aliases`
MUST register its own alias there — a runtime absent from that object has no published way to
be compared, and a mapping maintained privately in a consumer is the drift this repository
exists to prevent.
Fixture files carry **no individual `version` field**, which is the one place this
repository's "every JSON file has a top-level `version`" convention does not apply. The
corpus is versioned as a whole in [`conformance/manifest.json`](../conformance/manifest.json).