Section 1 required every case to be run and every unrunnable one reported as an
error. Scoping a case to signatures/active-content.json would therefore have made
llm-security permanently fail seven cases for having no such table - reporting an
architectural difference as a defect, and telling a reader nothing.
Section 1.1: a runtime declares which commons data files it implements, and a case
scoped outside that set is `not-applicable` - a third verdict, distinct from section
1's error. Section 1's error means the runtime tried and could not; this means the
question was never addressed to it.
Fenced so it cannot become an exit. It attaches to a TABLE, never to a case, since
per-case opt-out is exactly the silent skip section 1 forbids. A declared set may not
be narrowed to convert failures into not-applicable ones. Such cases stay in the
denominator: `76/83` and `76 passed, 7 not-applicable` describe different runtimes,
and only the second can be checked.
Three consequences, written where they are read:
- Section 4 no longer claims scoping "asks a question both can answer". That held
only while every case was scoped to the one table both runtimes implement. Scope
narrows what is compared; it does not make every runtime a valid addressee. The
superseded sentence is named in place rather than edited away.
- Section 4 now states that "belongs to a data file" means published there, never
"shares its prefix". Live witness: the guard emits `active:oversize-input`, a flag
about its own scan cap, which carries the prefix but is no construct in the table.
A prefix-matching runtime would fail a case over a finding the corpus never claimed.
- Section 6 states the derivation's cost: case_id derives from pattern_id alone, so a
single-finding scope holds at most one case per pattern id. When a source runtime
drives two payloads at one pattern, they must be compared within scope before a
second case is minted - and a discriminated case id is forbidden, since it would
break the reverse transform.
Section 8: a pass count is unreadable without the declared set beside it, and a
not-applicable verdict proves nothing about detection in either direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVouC9nsfrfV5jRSejxbvQ
Kjell Tore Guttormsen
807c0d45d8