feat(schema,spec): give the §1.1 MUST a shape, since v0.2.0 shipped it without one

schema/conformance-declaration.schema.json 0.1.0 — what a runtime publishes
alongside a conformance result: runtime, the commons commit measured, the declared
table set, and the four verdict counts.

Requested by llm-security (coord 2026-08-11T10:46:28Z) in those terms, with the
reason that matters: two runtimes publishing free-form declarations makes
"83/83 + 6 not-applicable" unparseable by anyone but its author, which is most of
the value of the MUST gone.

The schema does more than name fields, and each addition beyond their ask exists to
turn a §1.1 MUST into a failure rather than a sentence:

- The two arithmetic invariants §1.1 implies but cannot state unambiguously in
  prose — counts MUST sum to total, total MUST equal the corpus case count at the
  commit measured — are stated normatively in `invariants`. JSON Schema cannot
  express either.
- The enumeration arrays are REQUIRED whenever their counts are non-zero, which is
  §1.1's "MUST still be enumerated" made checkable. A bare count would satisfy the
  arithmetic and defeat the purpose.
- `error` and `not_applicable` stay structurally distinct. §1.1 turns on that
  distinction: the first means this runtime tried and could not, the second means
  the question was never addressed to it.
- All five counts are required including zeros. An absent count is
  indistinguishable from one the runtime never tracked.
- `declaration_source` records whether the declared set is DERIVED from the
  constant the runner uses to accept or reject a scope, or hand-maintained beside
  it. Only the derived form makes the anti-narrowing fence structural: narrowing it
  is then a code change with tests behind it. Formulation owed to llm-security, who
  stated the constraint for their own side before commons had a field for it.

Deliberately not a gate. Nothing in this repository runs (charter), and no
validation was asked for. Publishing the shape is the whole contribution.

Mutation-tested rather than eyeballed: the example validates as written, and five
distinct defect classes are rejected — missing enumeration when n/a > 0, missing
enumeration when failed > 0, a dropped zero-count, a non-conforming table path, an
unknown property. The example's `$comment` sits outside the example object because
`declaration` closes additionalProperties, so an example carrying its own $comment
would not validate against the schema it illustrates.

Its six example case ids were checked against the directories on disk — the first
guess had one wrong name. The caseIdList pattern was checked against all 89
existing ids, not just the new shape.

spec §1.1 gains the normative pointer plus a SHOULD that a runtime derive its
declared set from the runner's own constant and record which it did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HuudQLGMyMenuyeZk8fKs5
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:39:37 +02:00
commit 1482c0bc48
2 changed files with 264 additions and 0 deletions

View file

@ -48,6 +48,22 @@ Two constraints keep the verdict from becoming an exit:
MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly
the silent skip §1 forbids.
**The shape of the declaration.** A runtime publishing a declaration MUST use the shape in
[`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json):
runtime, the commons commit measured, the declared table set, and the four verdict counts with
their total. The requirement is the shape, not a validation step — nothing in this repository
runs, so no runtime is gated on it. It exists because a free-form declaration makes
`83 passed, 6 not-applicable` unparseable by anyone but its author, and a result only its
author can read is not a cross-runtime measurement. That schema also carries the two arithmetic
invariants this section implies but cannot state in prose without ambiguity: the counts MUST sum
to the total, and the total MUST equal the corpus case count at the commit measured.
A runtime SHOULD derive its declared set from the same constant its runner uses to accept or
reject a `scope`, rather than maintain a second list beside it, and SHOULD record which of the
two it did. The anti-narrowing constraint above is then structural rather than promised:
narrowing the declaration is narrowing what the runtime accepts, which is a change to code with
tests behind it instead of an edit to a published file nobody re-reads.
**What the verdict does not say.** `not-applicable` records that a runtime does not implement
a named commons table. It is not a statement that the runtime is blind to the phenomenon
behind it — a runtime may well detect the same construct through some other mechanism that