docs(manifest): the active-content fixtures pin a version, and nothing said so

The seed runtime's v1.0.0 freezes its exported Python surface and explicitly
not its detection behaviour. The manifest pinned commit and version per
measurement block but never recorded that the thing pinned is a version rather
than a frozen classification.

manifest 0.5.2 -> 0.6.0, one new field next to active_content_provenance.
asymmetry, bounded by what the fixtures actually assert: all seven carry
pattern_id only, so three of the four calibration dimensions cannot move them.
Both pins named, not one. Their statement is attributed, not restated as ours.

Also closes the omission 0.7.3 named: the same misquote in
docs/secret-egress-divergence.md:75-76. Not the fix the note implied - those
lines are one single-backtick span across a line break, so the outer delimiter
is promoted to double backticks instead.

spec section 7 deliberately untouched and named in CHANGELOG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yZobrgUiRtpLSWx8i7u2Z
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 23:09:33 +02:00
commit 181bdcad16
3 changed files with 59 additions and 4 deletions

View file

@ -9,6 +9,60 @@ Versioning note: the repository tag versions **the contract** (file set, key nam
case ids, disposition semantics). Each JSON file additionally carries its own case ids, disposition semantics). Each JSON file additionally carries its own
`"version"` field, bumped when that file changes. `"version"` field, bumped when that file changes.
## [0.8.0] — 2026-08-13
**The seven active-content fixtures pin a VERSION of the seed runtime, and nothing said so.**
That runtime tagged `v1.0.0` on 2026-08-13 and stated that the freeze covers its exported Python
surface only, excluding detection behaviour: severities, thresholds, lexicon entries and
dispositions are calibration there and move in minor and patch releases. The manifest already
pinned commit and version per measurement block, but nowhere recorded that the thing pinned is a
version rather than a frozen classification. No case is minted, no data file is touched, no id is
proposed.
### Added
- **`conformance/manifest.json` 0.5.2 → 0.6.0 —
`active_content_provenance.pins_a_version_not_a_frozen_classification`.** One field, scoping the
neighbouring `asymmetry` rather than replacing it, and deliberately narrower than the runtime's
own statement. The exposure is bounded by what the fixtures assert, which was read from all seven
rather than assumed: every finding carries `pattern_id` and nothing else — no severity, no
threshold, no disposition — so three of the four dimensions that runtime names as calibration
cannot move one of these cases at all. The fourth can, because which `active:` ids a payload
yields *is* the classification. The field names both pins (six at 0.4.0 / `de09711`, the seventh
at 0.7.0 / `be9759b`) rather than one, since a single version would flatten two measurements into
one header — the defect `superseded_for_one_case` exists to prevent. Their v1.0.0 statement is
**attributed** to their coord message of 2026-08-13T20:40:31Z, not restated as a fact measured
from this side.
- The field also names the disposition of a future divergence, so it is not left to be inferred: a
later 1.x that classifies one of these payloads differently is not a breach by them and does not
make the fixture wrong. The fixture stays ground truth at its pinned version, the divergence is
measured and recorded, and re-pinning is a separate decision — the same disposition this corpus
already applies to a stale `observed_out_of_scope` entry.
### Fixed
- **`docs/secret-egress-divergence.md:75-76` carried the same misquote `conformance/manifest.json`
had corrected in 0.7.3**, named there as a deliberate omission and closed here. The field's value
ends ``ascending `order` `` — the backticks are the field's own. The fix is *not* the one the
omission note implied: those two lines are a single code span delimited by **single** backticks
across a line break, so inserting the field's backticks inside it would have terminated the span
at the first one and rendered the quote broken. The outer delimiter is promoted to double
backticks instead, which is what lets the inner singles survive. The manifest's correction ported
as a literal string because JSON has no backtick semantics; markdown does. Verified by extracting
the span from the file on disk, unfolding the line break, and comparing to the decoded value in
`signatures/secret-egress.json` — equal — and by confirming no backtick run of length ≥ 2 sits
inside the span.
### Not done, and named rather than left silent
- **`spec/conformance-corpus.md` section 7 is untouched.** It states the disagreement rule without
scope: *"The fixture is ground truth. A runtime that disagrees is wrong."* Read against the
active-content scope, whose only implementing runtime has now said in writing that its
classification may legitimately move, that rule would call a calibration change there a bug. The
manifest field records the interaction and explicitly does not amend the spec. Whether the
normative rule needs a scope is a decision for its own release.
## [0.7.3] — 2026-08-13 ## [0.7.3] — 2026-08-13
**The README still argued the premise 0.7.2 retired, and the two files sat on a public remote **The README still argued the premise 0.7.2 retired, and the two files sat on a public remote

View file

@ -1,5 +1,5 @@
{ {
"version": "0.5.2", "version": "0.6.0",
"id": "llm-security-commons/conformance", "id": "llm-security-commons/conformance",
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.", "description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.", "$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
@ -137,7 +137,8 @@
"source_version": "0.4.0", "source_version": "0.4.0",
"extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.", "extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.",
"$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.", "$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.",
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing." "asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing.",
"pins_a_version_not_a_frozen_classification": "Scopes `asymmetry` above rather than replacing it. The seed runtime tagged `v1.0.0` on 2026-08-13 (commit `98ebc07`) and stated in the release notice (coord message 2026-08-13T20:40:31Z) that the freeze covers its EXPORTED PYTHON SURFACE only and deliberately excludes detection behaviour: severities, thresholds, lexicon entries and the dispositions they produce are calibration there, and calibration moves in minor and patch releases. That is their statement about their own repository, attributed here rather than restated as a fact measured from this side. What it means for these cases is bounded by what the fixtures actually assert, which was read rather than assumed: an active-content `expected.json` carries `pattern_id` membership and nothing else — no severity, no threshold, no disposition — so three of the four dimensions they name as calibration cannot move one of these cases at all. The fourth can: which `active:` ids a payload yields IS the classification, and they say explicitly it is not frozen data a third implementor can be held to. So each of these cases pins the guard VERSION recorded in its own measurement block — six at 0.4.0 / `de09711`, the seventh at 0.7.0 / `be9759b`; naming one version here would flatten two pins into one, the defect `superseded_for_one_case` exists to prevent — and does not pin a permanent contract on that runtime. A later 1.x that classifies one of these payloads differently is therefore not a breach by them and does not make the fixture wrong: the fixture stays ground truth at its pinned version, the divergence is measured and recorded, and re-pinning is a separate decision — the disposition this corpus already applies to a stale `observed_out_of_scope` entry. spec/conformance-corpus.md section 7 states the general disagreement rule without that scope and is NOT amended by this block; whether a normative rule needs one for a table whose only implementing runtime declines to freeze its classification is an open question, recorded here and not settled here."
}, },
"active_content_measurement": { "active_content_measurement": {
"date": "2026-08-10", "date": "2026-08-10",

View file

@ -72,8 +72,8 @@ rather than by measurement.
This is the finding a membership table would hide, and it is the one a consumer implementing This is the finding a membership table would hide, and it is the one a consumer implementing
from commons will get wrong first. from commons will get wrong first.
`signatures/secret-egress.json` declares `match_semantics: "first match wins; patterns are `signatures/secret-egress.json` declares ``match_semantics: "first match wins; patterns are
evaluated in ascending order"`, marks `ordering.normative: true`, and names evaluated in ascending `order`"``, marks `ordering.normative: true`, and names
`last_entry_is_load_bearing: "JWT (three-part token)"` — the JWT entry is placed last `last_entry_is_load_bearing: "JWT (three-part token)"` — the JWT entry is placed last
precisely so a token inside an `Authorization` header is reported as the header, not as a precisely so a token inside an `Authorization` header is reported as the header, not as a
bare JWT. bare JWT.