open/llm-security-commons
1
0
Fork 0
Code Issues Releases 1 Activity

docs(manifest): the precision field stated its exposure as a hand-derived count

Browse source 
0.8.0 added a field whose whole purpose is precision, and bounded the exposure
with "three of the four dimensions ... cannot move one of these cases at all".
The total was derived by hand over a taxonomy the field had itself
recategorized. Lexicon entries ARE one of the seed runtime's four calibration
dimensions, and they are not absent from these fixtures: two of the seven carry
a lexicon id in observed_out_of_scope. And "the fourth" substituted the
classification for lexicon entries as the fourth item of their sentence.

manifest 0.6.0 -> 0.6.1: the field enumerates three named things and totals
none of them. A lexicon change ages the residue as evidence without moving a
verdict (spec section 5); the classification is the one that can move one.
Retired sentence quoted in an AMENDED IN clause, not dropped.

No measurement changed, no verdict moved, no case or data file touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yZobrgUiRtpLSWx8i7u2Z
This commit is contained in:
Kjell Tore Guttormsen 2026-08-13 23:15:10 +02:00
commit c75c546614
2 changed files with 34 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
conformance/manifest.json
View file

@ -1,5 +1,5 @@
{
"version": "0.6.0",
"version": "0.6.1",
"id": "llm-security-commons/conformance",
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
@ -138,7 +138,7 @@
"extraction_method": "The payload strings were located by parsing coverage.py's AST for `_scan_case` calls whose first argument is `\"active\"`, and each payload expression was evaluated against that module's own namespace (which supplies the base64 constant one of them embeds). Retyping them here would have been the from-memory failure this repository forbids for detection data.",
"$comment": "A different structure and a different commit from `payload_provenance` above, which covers the 83 lexicon cases. Six of the seven extracted payloads became cases; the seventh is in `omitted_payloads` with its measured reason.",
"asymmetry": "These payloads were authored by the only runtime that implements the table they test, so unlike the lexicon corpus there is no second runtime whose agreement could be measured. That is a real limit on what these six cases prove, and it is stated rather than averaged away: they pin one runtime's behaviour as a contract a future implementer can be held to, which is less than the lexicon cases prove and more than nothing.",
"pins_a_version_not_a_frozen_classification": "Scopes `asymmetry` above rather than replacing it. The seed runtime tagged `v1.0.0` on 2026-08-13 (commit `98ebc07`) and stated in the release notice (coord message 2026-08-13T20:40:31Z) that the freeze covers its EXPORTED PYTHON SURFACE only and deliberately excludes detection behaviour: severities, thresholds, lexicon entries and the dispositions they produce are calibration there, and calibration moves in minor and patch releases. That is their statement about their own repository, attributed here rather than restated as a fact measured from this side. What it means for these cases is bounded by what the fixtures actually assert, which was read rather than assumed: an active-content `expected.json` carries `pattern_id` membership and nothing else — no severity, no threshold, no disposition — so three of the four dimensions they name as calibration cannot move one of these cases at all. The fourth can: which `active:` ids a payload yields IS the classification, and they say explicitly it is not frozen data a third implementor can be held to. So each of these cases pins the guard VERSION recorded in its own measurement block — six at 0.4.0 / `de09711`, the seventh at 0.7.0 / `be9759b`; naming one version here would flatten two pins into one, the defect `superseded_for_one_case` exists to prevent — and does not pin a permanent contract on that runtime. A later 1.x that classifies one of these payloads differently is therefore not a breach by them and does not make the fixture wrong: the fixture stays ground truth at its pinned version, the divergence is measured and recorded, and re-pinning is a separate decision — the disposition this corpus already applies to a stale `observed_out_of_scope` entry. spec/conformance-corpus.md section 7 states the general disagreement rule without that scope and is NOT amended by this block; whether a normative rule needs one for a table whose only implementing runtime declines to freeze its classification is an open question, recorded here and not settled here."
"pins_a_version_not_a_frozen_classification": "Scopes `asymmetry` above rather than replacing it. The seed runtime tagged `v1.0.0` on 2026-08-13 (commit `98ebc07`) and stated in the release notice (coord message 2026-08-13T20:40:31Z) that the freeze covers its EXPORTED PYTHON SURFACE only and deliberately excludes detection behaviour: severities, thresholds, lexicon entries and the dispositions they produce are calibration there, and calibration moves in minor and patch releases. That is their statement about their own repository, attributed here rather than restated as a fact measured from this side. What it means for these cases is bounded by what the fixtures actually assert, which was read from all seven rather than assumed, and is ENUMERATED rather than totalled. (i) Severities, thresholds and dispositions are absent from every `expected.json` in this scope — each finding carries `pattern_id` and nothing else — so calibration in those three dimensions changes no verdict here. (ii) Lexicon entries are absent from `findings` in this scope but PRESENT in `observed_out_of_scope` on two of the seven: `active__data-uri` holds `data-uri:executable` and `active__raw-html` holds `hybrid-xss:script-tag`, both verified as members of `lexicon/injection-lexicon.json` and non-members of `signatures/active-content.json`. So a lexicon calibration change ages those entries as evidence WITHOUT moving a verdict — under spec/conformance-corpus.md section 5 a runtime MUST NOT be failed for disagreeing with them — and under this corpus's own rule a stale residue entry is pinned, not rewritten. This is the exposure `active_content_measurement_0_7_0.movement_sweep.residue_is_the_field_no_test_protects` names as a class. (iii) The active-content classification itself is the one that CAN move a verdict, and it is what they actually addressed: they say explicitly it is not frozen data a third implementor can be held to. It is not one of the four items in the calibration sentence quoted above. So each of these cases pins the guard VERSION recorded in its own measurement block — six at 0.4.0 / `de09711`, the seventh at 0.7.0 / `be9759b`; naming one version here would flatten two pins into one, the defect `superseded_for_one_case` exists to prevent — and does not pin a permanent contract on that runtime. A later 1.x that classifies one of these payloads differently is therefore not a breach by them and does not make the fixture wrong: the fixture stays ground truth at its pinned version, the divergence is measured and recorded, and re-pinning is a separate decision — the disposition this corpus already applies to a stale `observed_out_of_scope` entry. spec/conformance-corpus.md section 7 states the general disagreement rule without that scope and is NOT amended by this block; whether a normative rule needs one for a table whose only implementing runtime declines to freeze its classification is an open question, recorded here and not settled here. AMENDED IN 0.6.1: as published in 0.6.0 this field read \"three of the four dimensions they name as calibration cannot move one of these cases at all. The fourth can: which `active:` ids a payload yields IS the classification\". Two defects in one sentence, and the enumeration above replaces the total rather than restating it. The count was hand-derived over a taxonomy this field had itself recategorized: `lexicon entries` IS one of their four and is NOT absent from these fixtures, so \"cannot move one of these cases at all\" was too strong for the evidence field even though it holds for every verdict. And \"the fourth\" silently substituted the classification for `lexicon entries` as the fourth item of their sentence, which it is not. No measurement changed and no verdict moved; only the description of the exposure did."
},
"active_content_measurement": {
"date": "2026-08-10",