open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

feat(spec): add not-applicable, so a single-runtime table stops reading as 7 defects

Browse source 
Section 1 required every case to be run and every unrunnable one reported as an
error. Scoping a case to signatures/active-content.json would therefore have made
llm-security permanently fail seven cases for having no such table - reporting an
architectural difference as a defect, and telling a reader nothing.

Section 1.1: a runtime declares which commons data files it implements, and a case
scoped outside that set is `not-applicable` - a third verdict, distinct from section
1's error. Section 1's error means the runtime tried and could not; this means the
question was never addressed to it.

Fenced so it cannot become an exit. It attaches to a TABLE, never to a case, since
per-case opt-out is exactly the silent skip section 1 forbids. A declared set may not
be narrowed to convert failures into not-applicable ones. Such cases stay in the
denominator: `76/83` and `76 passed, 7 not-applicable` describe different runtimes,
and only the second can be checked.

Three consequences, written where they are read:

- Section 4 no longer claims scoping "asks a question both can answer". That held
  only while every case was scoped to the one table both runtimes implement. Scope
  narrows what is compared; it does not make every runtime a valid addressee. The
  superseded sentence is named in place rather than edited away.
- Section 4 now states that "belongs to a data file" means published there, never
  "shares its prefix". Live witness: the guard emits `active:oversize-input`, a flag
  about its own scan cap, which carries the prefix but is no construct in the table.
  A prefix-matching runtime would fail a case over a finding the corpus never claimed.
- Section 6 states the derivation's cost: case_id derives from pattern_id alone, so a
  single-finding scope holds at most one case per pattern id. When a source runtime
  drives two payloads at one pattern, they must be compared within scope before a
  second case is minted - and a discriminated case id is forbidden, since it would
  break the reverse transform.

Section 8: a pass count is unreadable without the declared set beside it, and a
not-applicable verdict proves nothing about detection in either direction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVouC9nsfrfV5jRSejxbvQ
This commit is contained in:
Kjell Tore Guttormsen 2026-08-10 21:14:58 +02:00
commit 807c0d45d8
1 changed files with 72 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

73
spec/conformance-corpus.md
View file

@ -19,6 +19,41 @@ engine rejects the pattern behind it — a silently skipped case is an invisible
negative, which is the failure mode this corpus exists to catch. A case that cannot be run
MUST be reported as an error, not as a pass.
### 1.1 Declared tables and `not-applicable`
The runtimes that share this core do not implement the same set of data files, and some
never will: a table can be meaningful in an output gate and meaningless in the surface next
to it. §1 as stated would make every such case a permanent failure, which reports an
architectural difference as a defect and tells a reader nothing.
A runtime claiming conformance therefore MUST declare **the set of commons data files it
implements**, and MUST publish that set alongside its conformance result. A case whose
`scope` (§4) names a data file outside the declared set MUST be reported as
**`not-applicable`** — a third verdict beside pass and fail, and a different thing from the
error of §1. §1's error means *this runtime tried and could not*; `not-applicable` means
*this question was never addressed to this runtime*.
A `not-applicable` case MUST still be enumerated in the result. It MUST NOT be counted as a
pass, and MUST NOT be silently dropped from the denominator: `76/83 passed` and
`76 passed, 7 not-applicable` describe different runtimes, and only the second one can be
checked.
Two constraints keep the verdict from becoming an exit:
- A runtime MUST NOT withdraw a table from its declared set in order to convert failing
cases into `not-applicable` ones. The declaration describes what the runtime implements,
not what it would prefer to be asked. Narrowing it is a visible change to a published
claim, and it MUST be reported as such rather than made quietly between runs.
- `not-applicable` attaches to a **table**, never to a case. A runtime that declares a table
MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly
the silent skip §1 forbids.
**What the verdict does not say.** `not-applicable` records that a runtime does not implement
a named commons table. It is not a statement that the runtime is blind to the phenomenon
behind it — a runtime may well detect the same construct through some other mechanism that
this corpus does not constrain, and reading the verdict as a coverage gap would assert
something nobody measured. See §8.
## 2. `input.txt` is bytes, not text
`input.txt` holds the input **verbatim**, UTF-8 encoded. A runtime MUST read it as bytes
@ -115,10 +150,27 @@ So a runtime MUST NOT raise a lexicon finding this case does not list, and MUST
one it does. Whether the same runtime also flags the input as active content, as a carrier,
or as anything else is **not** this case's business.
**"Belongs to" means published in that file, never "shares its prefix."** A runtime MUST
decide membership by looking the id up in the scoped data file, and MUST NOT infer it from
the id's leading segment. The two are not the same set, and the difference is not
hypothetical: one seeding runtime emits `active:oversize-input`, a self-safety flag about its
own scan cap, which carries the `active:` prefix but is no construct in
[`signatures/active-content.json`](../signatures/active-content.json) and is therefore out of
scope for a case scoped there. A prefix-matching implementation would pull it into the
comparison and fail a case over a finding the corpus makes no claim about.
The scoping is what makes the exactness safe. The two seeding runtimes do not implement the
same set of tables — one has no active-content table at all — so a whole-report comparison
would fail for reasons that have nothing to do with the pattern under test. Scoping the
comparison to one published table asks a question both can answer.
comparison to one published table asks a question about that table alone.
Through version 0.1.1 this paragraph ended "…asks a question both can answer", which held
only while every case was scoped to the one table both runtimes implement. It stopped being
true the moment a case was scoped to `signatures/active-content.json`, which only one of them
has. Scope narrows *what* is compared; it does not guarantee that every runtime is a valid
addressee of the comparison. That second question is answered by the declared table set in
§1.1, and a case scoped outside a runtime's declared set is `not-applicable` there rather
than failed.
The exactness is what makes the corpus worth running. A corpus that only checks that the
expected finding is *present* is passed by a runtime that flags everything.
@ -152,6 +204,19 @@ this repository, so the id space cannot reach the filesystem unchanged. `__` doe
anywhere in the ratified id space, so the transform is one-to-one — verified collision-free
across all cases rather than assumed.
The transform carries a consequence that is easy to miss: **a case id is derived from a
pattern id alone, so the corpus holds at most one case per `pattern_id` in single-finding
scopes.** There is nowhere in the name to put a second one. That is a real constraint, not a
formality — a source runtime's own test matrix may well drive two payloads at the same
pattern, as one of the seeding runtimes does for `active:markdown-image`. When it does, the
two payloads MUST be compared *within the case's scope* before a second case is minted: if
their in-scope finding sets are equal, the second case cannot fail in any way the first does
not, and its distinguishing signal lies outside the scope where this corpus makes no claim.
Such a payload MUST NOT be given a discriminated case id; the derivation rule is the
contract, and a suffix would break the reverse transform. It SHOULD instead be recorded as
a named omission in `conformance/manifest.json`, so the drop is visible rather than
inferred from a count.
**A case id is a stable identifier. Changing one is a BREAKING change** and requires a major
bump of the corpus version, exactly like changing a pattern id. Consumers name cases in
their own test suites; a renamed case is a silently disappeared test.
@ -176,3 +241,9 @@ one — it is why each case records how it was produced.
Nor is the corpus a coverage claim. It covers the pattern tables named in
`conformance/manifest.json` under `scope_covered`, and the tables under `scope_planned` are
named there precisely so that their absence is visible rather than inferred.
And a result is only readable next to the declared table set of §1.1. A pass count on its
own does not say which questions were asked: a runtime that implements one table and one
that implements four can print the same number. The declared set is what turns the number
back into a claim — which is also why a `not-applicable` verdict proves nothing in either
direction about the runtime's actual detection. It says the corpus did not ask.