open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

fix(spec): section 6 forbade a case the corpus ships, and the predicate under it was wrong too

Browse source 
The normative spec read "Such a payload MUST NOT be given a discriminated case
id" while manifest.json defined case_id_derivation.variant_suffix and
conformance/hybrid-xss__script-tag--src-no-close/ sat on disk under it. The
manifest was the correct party: the derivation was extended in corpus 0.3.0 and
the spec was never updated. No data moves here, only the text describing it.

The derivation block now carries the optional '--' suffix and the truncating
reverse, with the '--'-absence measurement stated as the reason the reverse
stays LEXICAL - re-measured at this commit rather than copied from 0.3.0's
numbers, and scoped to id spaces because '--' does occur inside pattern values.

The part that would have passed review while still being wrong: fixing only the
permission. Section 6 also reasoned that equal in-scope finding sets mean the
second case "cannot fail in any way the first does not" - and the shipped
variant falsifies exactly that. It expects the same single finding, same scope,
same match, and still gates what the base cannot, because the base input matches
the pattern under both its published and its superseded stricter form. The
discriminating signal is INSIDE the scope, in the form of the scoped rule. So
the MUST NOT is replaced by a predicate about failure surface rather than
finding sets, checked in both directions: it admits the shipped variant and
still excludes the omitted markdown-image payload, whose only distinguisher
lives in a table this repository does not publish.

manifest.json is untouched and stays at 0.5.0; no case directory moved. The
spec has no version of its own - "Through version 0.1.1" in section 4 is the
CORPUS version, verified against CHANGELOG [0.2.0] before acting, because the
session brief said otherwise. Section 6's stable-id and BREAKING sentences were
read, not edited.

Verified in scratchpad, never in the repo: the amended derivation transcribed
into a checker that reads all 94 cases back from disk, derives each pattern id,
round-trips it forward, and asserts the case expects it. All 94 reproduce.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M4xrxV3EXbSALqvB23kpeY
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 22:00:24 +02:00
commit cb784fea6f
3 changed files with 157 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

65
spec/conformance-corpus.md
View file

@ -211,8 +211,10 @@ Reading that absence as "this runtime emits nothing else" would be a claim nobod
## 6. Case ids
```
case_id = pattern_id with ":" replaced by "__"
pattern_id = case_id with "__" replaced by ":"
case_id = pattern_id with ":" replaced by "__",
optionally followed by "--" and a variant slug of [a-z0-9-]
pattern_id = case_id truncated at the first "--" if present,
then "__" replaced by ":"
```
`:` is not a legal filename character on Windows, and fork-and-own is a supported use of
@ -220,18 +222,53 @@ this repository, so the id space cannot reach the filesystem unchanged. `__` doe
anywhere in the ratified id space, so the transform is one-to-one — verified collision-free
across all cases rather than assumed.
The transform carries a consequence that is easy to miss: **a case id is derived from a
pattern id alone, so the corpus holds at most one case per `pattern_id` in single-finding
scopes.** There is nowhere in the name to put a second one. That is a real constraint, not a
formality — a source runtime's own test matrix may well drive two payloads at the same
pattern, as one of the seeding runtimes does for `active:markdown-image`. When it does, the
two payloads MUST be compared *within the case's scope* before a second case is minted: if
their in-scope finding sets are equal, the second case cannot fail in any way the first does
not, and its distinguishing signal lies outside the scope where this corpus makes no claim.
Such a payload MUST NOT be given a discriminated case id; the derivation rule is the
contract, and a suffix would break the reverse transform. It SHOULD instead be recorded as
a named omission in `conformance/manifest.json`, so the drop is visible rather than
inferred from a count.
`--` does not occur there either: the ratified ids use single hyphens throughout, measured
across every id space this repository publishes and every case id in the corpus. That
measurement is what keeps the reverse transform **lexical**. A runtime recovers a
`pattern_id` by splitting the string, and MUST NOT need a lookup against the published id
list to find where the id ends and the variant begins — a reverse transform that has to ask
which of two readings is real is a different rule from the one written above, and it fails
on the first id space that is vendored without its lookup table.
**A `pattern_id` may carry more than one case.** Through corpus version 0.2.0 this section
said the opposite: that a case id derives from a pattern id alone, so a single-finding scope
holds at most one case per pattern id, with nowhere in the name to put a second. Corpus
version 0.3.0 extended the derivation with the optional suffix above, and the corpus has
shipped a case under it since (`hybrid-xss__script-tag--src-no-close`, whose own `$comment`
carries the rationale for that one). The superseded sentence is named here rather than edited
away, because it was the stated ground on which a real payload was dropped —
`omitted_payloads` in [`conformance/manifest.json`](../conformance/manifest.json) records
that ground as withdrawn and a second, independent ground as still standing.
The bar for minting a second case is neither that the two inputs differ, nor that their
in-scope finding sets differ:
> A variant case MAY be minted when the second input can fail, **within the case's scope**,
> under a change to a scoped data file that the first input would pass. Where no edit to a
> published table separates the two inputs, the second case cannot fail in any way the first
> does not, and it MUST NOT be minted.
Equal in-scope finding sets do not settle that question, and reading them as if they did is
the error this paragraph replaces. The corpus's first variant case expects exactly the
finding set its base case expects — one `pattern_id`, one scope, one `match` — and still
gates something the base cannot: the base input matches the scoped pattern both in its
published form and in the stricter form that preceded it, so reinstating the stricter form
leaves it passing, while the variant input matches only the published form and fails. The
distinguishing signal is *inside* the scope, in the form of the scoped rule itself, which is
exactly what a finding-set comparison cannot see.
The payload that stays out is the mirror image. A source runtime's own test matrix may drive
two payloads at the same pattern, as one of the seeding runtimes does for
`active:markdown-image`, whose only difference is a signal from a table this repository does
not publish. No edit to any scoped file separates them, so the second case could not fail
where the first passes. Such a payload SHOULD be recorded as a named omission in
`conformance/manifest.json`, so the drop is visible rather than inferred from a count.
A variant case MUST be scoped and matched exactly like its base case, and MUST expect the
same `pattern_id`. **The suffix distinguishes inputs, never findings.** It is not a licence
to record a second, different verdict for one rule: two cases at one pattern id expecting
different findings within the same scope are not a variant pair, they are the corpus
contradicting itself.
**A case id is a stable identifier. Changing one is a BREAKING change** and requires a major
bump of the corpus version, exactly like changing a pattern id. Consumers name cases in