schema/conformance-declaration.schema.json 0.1.0 — what a runtime publishes
alongside a conformance result: runtime, the commons commit measured, the declared
table set, and the four verdict counts.
Requested by llm-security (coord 2026-08-11T10:46:28Z) in those terms, with the
reason that matters: two runtimes publishing free-form declarations makes
"83/83 + 6 not-applicable" unparseable by anyone but its author, which is most of
the value of the MUST gone.
The schema does more than name fields, and each addition beyond their ask exists to
turn a §1.1 MUST into a failure rather than a sentence:
- The two arithmetic invariants §1.1 implies but cannot state unambiguously in
prose — counts MUST sum to total, total MUST equal the corpus case count at the
commit measured — are stated normatively in `invariants`. JSON Schema cannot
express either.
- The enumeration arrays are REQUIRED whenever their counts are non-zero, which is
§1.1's "MUST still be enumerated" made checkable. A bare count would satisfy the
arithmetic and defeat the purpose.
- `error` and `not_applicable` stay structurally distinct. §1.1 turns on that
distinction: the first means this runtime tried and could not, the second means
the question was never addressed to it.
- All five counts are required including zeros. An absent count is
indistinguishable from one the runtime never tracked.
- `declaration_source` records whether the declared set is DERIVED from the
constant the runner uses to accept or reject a scope, or hand-maintained beside
it. Only the derived form makes the anti-narrowing fence structural: narrowing it
is then a code change with tests behind it. Formulation owed to llm-security, who
stated the constraint for their own side before commons had a field for it.
Deliberately not a gate. Nothing in this repository runs (charter), and no
validation was asked for. Publishing the shape is the whole contribution.
Mutation-tested rather than eyeballed: the example validates as written, and five
distinct defect classes are rejected — missing enumeration when n/a > 0, missing
enumeration when failed > 0, a dropped zero-count, a non-conforming table path, an
unknown property. The example's `$comment` sits outside the example object because
`declaration` closes additionalProperties, so an example carrying its own $comment
would not validate against the schema it illustrates.
Its six example case ids were checked against the directories on disk — the first
guess had one wrong name. The caseIdList pattern was checked against all 89
existing ids, not just the new shape.
spec §1.1 gains the normative pointer plus a SHOULD that a runtime derive its
declared set from the runner's own constant and record which it did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HuudQLGMyMenuyeZk8fKs5
Kjell Tore Guttormsen
1482c0bc48