open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

fix(spec): section 7 stated its own premise and then applied itself beyond it

Browse source 
Section 7 justified the fixture-is-ground-truth ordering with "Two
implementations that return different verdicts" and then stated the rule
with no scope. For signatures/active-content.json there is no second
implementation, and the seed runtime has stated the classification behind
it is calibration it does not freeze. As written, section 7 turned a change
they reserved into a bug on their side.

New section 7.1 keys the scope on a structural property, never on a table
name: a scope only one runtime implements, whose payload that runtime
authored. It creates no fourth verdict - the declaration schema closes
result with additionalProperties:false over four counts, so a fifth would
break every consumer's parser. The case still fails and is still named in
failed_cases; what changes is what the failure licenses concluding.

Two limits are stated rather than left to inference: it does not reach a
third-party implementer of the same table, and it is not a licence for a
runtime to self-declare its own divergence as calibration.

manifest.json 0.6.1 -> 0.6.2 retires the open-question sentence, quoted
rather than dropped. The retirement is partial: "section 7 is NOT amended
by this block" stays true, because the spec was amended by its own release.

Neighbours measured over the whole repository, widened past "ground truth"
to the second paragraph's own wording. CONVENTIONS.md and CLAUDE.md carried
the premise and are changed; SECURITY.md gets a cross-reference only, since
its claim is about a fixture expecting too little and 7.1 does not narrow
that direction; README.md and docs/extraction-plan.md are named as
deliberately untouched.

Breaking in category, minor in number - 0.x, per the reading [0.3.0]
recorded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012pZ2FLQ6xkWvj2VcwgwnQv
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 23:31:28 +02:00
commit f0a511369d
6 changed files with 157 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

8
CONVENTIONS.md
View file

@ -107,10 +107,16 @@ this document:
- `<case-id>` is stable and descriptive. **Changing a case id is a breaking change** — a
published conformance result names it.
- `expected.json` is **ground truth**. If a runtime disagrees with it, the runtime is wrong.
- The one way that reverses: the fixture is proven wrong. Then the fixture changes **in its own
- One way that reverses: the fixture is proven wrong. Then the fixture changes **in its own
commit, with the reason written down** — never folded into a change that does something else,
because a fixture edit is the one edit that can make every conforming runtime wrong
identically.
- The other, added in `v0.9.0`: where a case's scope is a table only one runtime implements and
that runtime authored the payload, a divergence by **that** runtime is neither a proven-wrong
fixture nor necessarily its bug. The fixture is not rewritten on the divergence alone — it is
recorded against the version pinned, and re-pinning is a separate release. Through `v0.8.1`
this list carried only the first way. See
[`spec/conformance-corpus.md` §7.1](spec/conformance-corpus.md).
- A case declares the data files it is `scope`d to. A runtime that does not implement a scoped
table reports the case `not-applicable` — a third verdict beside pass and fail, and one that
must be reported rather than dropped from the denominator. See