open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

fix(spec): section 7 stated its own premise and then applied itself beyond it

Browse source 
Section 7 justified the fixture-is-ground-truth ordering with "Two
implementations that return different verdicts" and then stated the rule
with no scope. For signatures/active-content.json there is no second
implementation, and the seed runtime has stated the classification behind
it is calibration it does not freeze. As written, section 7 turned a change
they reserved into a bug on their side.

New section 7.1 keys the scope on a structural property, never on a table
name: a scope only one runtime implements, whose payload that runtime
authored. It creates no fourth verdict - the declaration schema closes
result with additionalProperties:false over four counts, so a fifth would
break every consumer's parser. The case still fails and is still named in
failed_cases; what changes is what the failure licenses concluding.

Two limits are stated rather than left to inference: it does not reach a
third-party implementer of the same table, and it is not a licence for a
runtime to self-declare its own divergence as calibration.

manifest.json 0.6.1 -> 0.6.2 retires the open-question sentence, quoted
rather than dropped. The retirement is partial: "section 7 is NOT amended
by this block" stays true, because the spec was amended by its own release.

Neighbours measured over the whole repository, widened past "ground truth"
to the second paragraph's own wording. CONVENTIONS.md and CLAUDE.md carried
the premise and are changed; SECURITY.md gets a cross-reference only, since
its claim is about a fixture expecting too little and 7.1 does not narrow
that direction; README.md and docs/extraction-plan.md are named as
deliberately untouched.

Breaking in category, minor in number - 0.x, per the reading [0.3.0]
recorded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012pZ2FLQ6xkWvj2VcwgwnQv
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 23:31:28 +02:00
commit f0a511369d
6 changed files with 157 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

41
spec/conformance-corpus.md
View file

@ -284,6 +284,47 @@ This ordering is the whole point of the repository. Two implementations that ret
different verdicts on the same input are not holding different opinions; one of them has a
bug.
### 7.1 Where the second paragraph does not hold
**Through corpus version 0.8.1 this section stated the rule above with no scope at all**, and
the scope was load-bearing: the justification names *two* implementations. Where a case's
scope is a table only one runtime implements, and that runtime authored the payload the case
was extracted from, there is no second implementation whose disagreement the paragraph could
adjudicate. Which cases those are is recorded in the corpus, not asserted per run — see
`active_content_provenance.asymmetry` in
[`conformance/manifest.json`](../conformance/manifest.json).
For such a case, a disagreement by the **seed runtime itself** is a third thing, and it is
neither of the two the paragraph offers:
- The fixture is not proven wrong. It recorded that runtime's behaviour correctly at the
commit and version its own measurement block pins, and a later classification does not
reach back and falsify an earlier measurement.
- The runtime does not necessarily have a bug. Where the seed runtime has stated that the
classification behind such a table is calibration it does not freeze, a release that
classifies the payload differently is a change it reserved, not a defect.
So: the fixture MUST NOT be rewritten on the strength of the divergence alone; the divergence
SHOULD be recorded against the version pinned; and re-pinning the case to a later version of
the seed runtime is a separate decision, taken deliberately and released on its own. This is
the disposition §5 already applies to a stale `observed_out_of_scope` entry, extended to the
one place where it can reach a verdict — and a divergence recorded here is the signal that
the re-pinning decision is due, not a reason to leave it open.
Three things this does **not** do.
- **It creates no fourth verdict.** The counts of §1.1 and
[`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json)
are unchanged: a case whose expected findings are not produced still **fails**, and is still
named in `failed_cases`. What changes is what the failure licenses concluding, not what is
reported.
- **It does not reach a third-party implementer** of the same table. Against them the fixture
is the contract, exactly as §7 states — which is what these cases were minted to provide,
and the only thing they can prove while one runtime is all there is.
- **It is not a licence to self-declare.** The exemption is carried by the corpus's own
provenance record for the scope. A runtime MUST NOT claim it for a case by asserting that
its own divergence is calibration.
## 8. What conformance does and does not prove
Passing this corpus proves that a runtime agrees with the other runtimes that pass it, on