schema/conformance-declaration.schema.json 0.1.0 — what a runtime publishes
alongside a conformance result: runtime, the commons commit measured, the declared
table set, and the four verdict counts.
Requested by llm-security (coord 2026-08-11T10:46:28Z) in those terms, with the
reason that matters: two runtimes publishing free-form declarations makes
"83/83 + 6 not-applicable" unparseable by anyone but its author, which is most of
the value of the MUST gone.
The schema does more than name fields, and each addition beyond their ask exists to
turn a §1.1 MUST into a failure rather than a sentence:
- The two arithmetic invariants §1.1 implies but cannot state unambiguously in
prose — counts MUST sum to total, total MUST equal the corpus case count at the
commit measured — are stated normatively in `invariants`. JSON Schema cannot
express either.
- The enumeration arrays are REQUIRED whenever their counts are non-zero, which is
§1.1's "MUST still be enumerated" made checkable. A bare count would satisfy the
arithmetic and defeat the purpose.
- `error` and `not_applicable` stay structurally distinct. §1.1 turns on that
distinction: the first means this runtime tried and could not, the second means
the question was never addressed to it.
- All five counts are required including zeros. An absent count is
indistinguishable from one the runtime never tracked.
- `declaration_source` records whether the declared set is DERIVED from the
constant the runner uses to accept or reject a scope, or hand-maintained beside
it. Only the derived form makes the anti-narrowing fence structural: narrowing it
is then a code change with tests behind it. Formulation owed to llm-security, who
stated the constraint for their own side before commons had a field for it.
Deliberately not a gate. Nothing in this repository runs (charter), and no
validation was asked for. Publishing the shape is the whole contribution.
Mutation-tested rather than eyeballed: the example validates as written, and five
distinct defect classes are rejected — missing enumeration when n/a > 0, missing
enumeration when failed > 0, a dropped zero-count, a non-conforming table path, an
unknown property. The example's `$comment` sits outside the example object because
`declaration` closes additionalProperties, so an example carrying its own $comment
would not validate against the schema it illustrates.
Its six example case ids were checked against the directories on disk — the first
guess had one wrong name. The caseIdList pattern was checked against all 89
existing ids, not just the new shape.
spec §1.1 gains the normative pointer plus a SHOULD that a runtime derive its
declared set from the runner's own constant and record which it did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HuudQLGMyMenuyeZk8fKs5
The schema was seeded from sarif-formatter.mjs, which consumes findings.
That could only ever establish a lower bound on the property set, so
additionalProperties had to stay open. The producer is now readable:
finding() in scanners/lib/output.mjs line 32, at b0de0ca. It returns an
object literal with exactly ten keys and no spread, so the set is complete
and the schema closes.
Added: id and evidence, the two keys a consumer-side reading could not see.
id gets its own definition, DS-<prefix>-<counter>, with pattern
^DS-[A-Za-z]+-[0-9]{3,} - the {3,} because padStart(3) is a minimum, so a
run past 999 findings produces four digits. It comes from a process-global
counter and is stable neither across runs nor across processes; the
definition says so before someone keys on it.
Nullability is evidence now, not convention. Five keys are emitted as null
rather than omitted, so a serialised finding always carries all ten. The
four assigned straight from opts are the exception: omit description and
the key is undefined and disappears from the JSON. Reproduced against the
real producer - ten keys in memory, nine serialised.
owasp is a string, not an array, and not one code. Multiple codes are
joined with ", ". Measured across the seed runtime: 31 distinct values over
157 sites, 13 multi-code, and four that mix taxonomies inside a single
value with no discriminator. That has a consequence nobody had written
down: sarif-formatter builds tags: [f.owasp], so "LLM06, ASI02" becomes ONE
tag with a comma in it and nothing filtering on LLM06 matches. Reproduced
end to end through the real finding() and toSARIF(), logged as
known_lossiness.owasp-tag-not-split. It is consumer behaviour, not data, so
it is reported rather than fixed here.
The JSONL profile is set to "not applicable" rather than "unspecified".
The distinction carries weight: unspecified would assert a profile exists
and has merely not been written down. There is no finding-JSONL - findings
are emitted only inside one JSON envelope. The single module that does
write JSONL, audit-trail.mjs, writes audit events under a different schema
where owasp is an ARRAY. Same field name, different type, same repository.
The profile records that trap instead of leaving a TODO.
Verified: valid Draft 2020-12, every finding built by the real producer
validates, and four negative controls are rejected.
One new open question left unpatched: the producer's JSDoc lists seventeen
scanner prefixes including IDE, while all four maps in owasp-map.json are
keyed on sixteen without it. An IDE finding has no taxonomy mapping
anywhere. Adding the key would be inventing detection data.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SNMcqrfNyoLRQ7qXUFZnb9
The finding contract plus the SARIF 2.1.0 output profile, modelled on
llm-security's sarif-formatter.mjs (operator dump). Draft 2020-12, valid
against the metaschema.
The SARIF profile is proven, not transcribed: the mapping was re-implemented
from the commons JSON alone and diffed against the real toSARIF over 10
envelope shapes (all severities incl. unknown/undefined, slug edge cases,
rule-id collision, every optional-field combination, multi-scanner, explicit
version) — 0 differences.
Three things left honest rather than closed:
- additionalProperties stays open. The formatter CONSUMES findings, so fields
it ignores are invisible in this evidence; the property list is a lower
bound until the producer is supplied.
- The JSONL profile is status=unspecified with a TODO. "One finding per line"
is inference, and a guessed normative contract is worse than a missing one.
- Node and the Python guard share only severity+owasp; the rest of the field
names diverge (scanner/title vs detector/label). Recorded as an open
question for v0.2.0, not settled silently here.
One published claim was wrong and is corrected: rule-id collision does NOT
cover punctuation, only case and whitespace runs. Caught by executing the
claim instead of asserting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0191AKc2qW6tmXDFSx1xn53q
