open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

docs(manifest): the active-content fixtures pin a version, and nothing said so

Browse source 
The seed runtime's v1.0.0 freezes its exported Python surface and explicitly
not its detection behaviour. The manifest pinned commit and version per
measurement block but never recorded that the thing pinned is a version rather
than a frozen classification.

manifest 0.5.2 -> 0.6.0, one new field next to active_content_provenance.
asymmetry, bounded by what the fixtures actually assert: all seven carry
pattern_id only, so three of the four calibration dimensions cannot move them.
Both pins named, not one. Their statement is attributed, not restated as ours.

Also closes the omission 0.7.3 named: the same misquote in
docs/secret-egress-divergence.md:75-76. Not the fix the note implied - those
lines are one single-backtick span across a line break, so the outer delimiter
is promoted to double backticks instead.

spec section 7 deliberately untouched and named in CHANGELOG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yZobrgUiRtpLSWx8i7u2Z
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 23:09:33 +02:00
commit c23aea9062
3 changed files with 59 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

5
conformance/manifest.json
View file

@ -1,5 +1,5 @@
{
"version": "0.5.2",
"version": "0.6.0",
"id": "llm-security-commons/conformance",
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
@ -137,7 +137,8 @@
"source_version": "0.4.0",
"extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.",
"$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.",
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing."
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing.",
"pins_a_version_not_a_frozen_classification": "Scopes `asymmetry` above rather than replacing it. The seed runtime tagged `v1.0.0` on 2026-08-13 (commit `98ebc07`) and stated in the release notice (coord message 2026-08-13T20:40:31Z) that the freeze covers its EXPORTED PYTHON SURFACE only and deliberately excludes detection behaviour: severities, thresholds, lexicon entries and the dispositions they produce are calibration there, and calibration moves in minor and patch releases. That is their statement about their own repository, attributed here rather than restated as a fact measured from this side. What it means for these cases is bounded by what the fixtures actually assert, which was read rather than assumed: an active-content `expected.json` carries `pattern_id` membership and nothing else — no severity, no threshold, no disposition — so three of the four dimensions they name as calibration cannot move one of these cases at all. The fourth can: which `active:` ids a payload yields IS the classification, and they say explicitly it is not frozen data a third implementor can be held to. So each of these cases pins the guard VERSION recorded in its own measurement block — six at 0.4.0 / `de09711`, the seventh at 0.7.0 / `be9759b`; naming one version here would flatten two pins into one, the defect `superseded_for_one_case` exists to prevent — and does not pin a permanent contract on that runtime. A later 1.x that classifies one of these payloads differently is therefore not a breach by them and does not make the fixture wrong: the fixture stays ground truth at its pinned version, the divergence is measured and recorded, and re-pinning is a separate decision — the disposition this corpus already applies to a stale `observed_out_of_scope` entry. spec/conformance-corpus.md section 7 states the general disagreement rule without that scope and is NOT amended by this block; whether a normative rule needs one for a table whose only implementing runtime declines to freeze its classification is an open question, recorded here and not settled here."
},
"active_content_measurement": {
"date": "2026-08-10",