feat(schema,spec): give the §1.1 MUST a shape, since v0.2.0 shipped it without one
schema/conformance-declaration.schema.json 0.1.0 — what a runtime publishes alongside a conformance result: runtime, the commons commit measured, the declared table set, and the four verdict counts. Requested by llm-security (coord 2026-08-11T10:46:28Z) in those terms, with the reason that matters: two runtimes publishing free-form declarations makes "83/83 + 6 not-applicable" unparseable by anyone but its author, which is most of the value of the MUST gone. The schema does more than name fields, and each addition beyond their ask exists to turn a §1.1 MUST into a failure rather than a sentence: - The two arithmetic invariants §1.1 implies but cannot state unambiguously in prose — counts MUST sum to total, total MUST equal the corpus case count at the commit measured — are stated normatively in `invariants`. JSON Schema cannot express either. - The enumeration arrays are REQUIRED whenever their counts are non-zero, which is §1.1's "MUST still be enumerated" made checkable. A bare count would satisfy the arithmetic and defeat the purpose. - `error` and `not_applicable` stay structurally distinct. §1.1 turns on that distinction: the first means this runtime tried and could not, the second means the question was never addressed to it. - All five counts are required including zeros. An absent count is indistinguishable from one the runtime never tracked. - `declaration_source` records whether the declared set is DERIVED from the constant the runner uses to accept or reject a scope, or hand-maintained beside it. Only the derived form makes the anti-narrowing fence structural: narrowing it is then a code change with tests behind it. Formulation owed to llm-security, who stated the constraint for their own side before commons had a field for it. Deliberately not a gate. Nothing in this repository runs (charter), and no validation was asked for. Publishing the shape is the whole contribution. Mutation-tested rather than eyeballed: the example validates as written, and five distinct defect classes are rejected — missing enumeration when n/a > 0, missing enumeration when failed > 0, a dropped zero-count, a non-conforming table path, an unknown property. The example's `$comment` sits outside the example object because `declaration` closes additionalProperties, so an example carrying its own $comment would not validate against the schema it illustrates. Its six example case ids were checked against the directories on disk — the first guess had one wrong name. The caseIdList pattern was checked against all 89 existing ids, not just the new shape. spec §1.1 gains the normative pointer plus a SHOULD that a runtime derive its declared set from the runner's own constant and record which it did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HuudQLGMyMenuyeZk8fKs5
This commit is contained in:
parent
25a2cf9643
commit
1482c0bc48
2 changed files with 264 additions and 0 deletions
|
|
@ -48,6 +48,22 @@ Two constraints keep the verdict from becoming an exit:
|
|||
MUST run every case scoped to it, including the ones it fails. Per-case opt-out is exactly
|
||||
the silent skip §1 forbids.
|
||||
|
||||
**The shape of the declaration.** A runtime publishing a declaration MUST use the shape in
|
||||
[`schema/conformance-declaration.schema.json`](../schema/conformance-declaration.schema.json):
|
||||
runtime, the commons commit measured, the declared table set, and the four verdict counts with
|
||||
their total. The requirement is the shape, not a validation step — nothing in this repository
|
||||
runs, so no runtime is gated on it. It exists because a free-form declaration makes
|
||||
`83 passed, 6 not-applicable` unparseable by anyone but its author, and a result only its
|
||||
author can read is not a cross-runtime measurement. That schema also carries the two arithmetic
|
||||
invariants this section implies but cannot state in prose without ambiguity: the counts MUST sum
|
||||
to the total, and the total MUST equal the corpus case count at the commit measured.
|
||||
|
||||
A runtime SHOULD derive its declared set from the same constant its runner uses to accept or
|
||||
reject a `scope`, rather than maintain a second list beside it, and SHOULD record which of the
|
||||
two it did. The anti-narrowing constraint above is then structural rather than promised:
|
||||
narrowing the declaration is narrowing what the runtime accepts, which is a change to code with
|
||||
tests behind it instead of an edit to a published file nobody re-reads.
|
||||
|
||||
**What the verdict does not say.** `not-applicable` records that a runtime does not implement
|
||||
a named commons table. It is not a statement that the runtime is blind to the phenomenon
|
||||
behind it — a runtime may well detect the same construct through some other mechanism that
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue