schema/conformance-declaration.schema.json 0.1.0 — what a runtime publishes
alongside a conformance result: runtime, the commons commit measured, the declared
table set, and the four verdict counts.
Requested by llm-security (coord 2026-08-11T10:46:28Z) in those terms, with the
reason that matters: two runtimes publishing free-form declarations makes
"83/83 + 6 not-applicable" unparseable by anyone but its author, which is most of
the value of the MUST gone.
The schema does more than name fields, and each addition beyond their ask exists to
turn a §1.1 MUST into a failure rather than a sentence:
- The two arithmetic invariants §1.1 implies but cannot state unambiguously in
prose — counts MUST sum to total, total MUST equal the corpus case count at the
commit measured — are stated normatively in `invariants`. JSON Schema cannot
express either.
- The enumeration arrays are REQUIRED whenever their counts are non-zero, which is
§1.1's "MUST still be enumerated" made checkable. A bare count would satisfy the
arithmetic and defeat the purpose.
- `error` and `not_applicable` stay structurally distinct. §1.1 turns on that
distinction: the first means this runtime tried and could not, the second means
the question was never addressed to it.
- All five counts are required including zeros. An absent count is
indistinguishable from one the runtime never tracked.
- `declaration_source` records whether the declared set is DERIVED from the
constant the runner uses to accept or reject a scope, or hand-maintained beside
it. Only the derived form makes the anti-narrowing fence structural: narrowing it
is then a code change with tests behind it. Formulation owed to llm-security, who
stated the constraint for their own side before commons had a field for it.
Deliberately not a gate. Nothing in this repository runs (charter), and no
validation was asked for. Publishing the shape is the whole contribution.
Mutation-tested rather than eyeballed: the example validates as written, and five
distinct defect classes are rejected — missing enumeration when n/a > 0, missing
enumeration when failed > 0, a dropped zero-count, a non-conforming table path, an
unknown property. The example's `$comment` sits outside the example object because
`declaration` closes additionalProperties, so an example carrying its own $comment
would not validate against the schema it illustrates.
Its six example case ids were checked against the directories on disk — the first
guess had one wrong name. The caseIdList pattern was checked against all 89
existing ids, not just the new shape.
spec §1.1 gains the normative pointer plus a SHOULD that a runtime derive its
declared set from the runner's own constant and record which it did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HuudQLGMyMenuyeZk8fKs5
Section 1 required every case to be run and every unrunnable one reported as an
error. Scoping a case to signatures/active-content.json would therefore have made
llm-security permanently fail seven cases for having no such table - reporting an
architectural difference as a defect, and telling a reader nothing.
Section 1.1: a runtime declares which commons data files it implements, and a case
scoped outside that set is `not-applicable` - a third verdict, distinct from section
1's error. Section 1's error means the runtime tried and could not; this means the
question was never addressed to it.
Fenced so it cannot become an exit. It attaches to a TABLE, never to a case, since
per-case opt-out is exactly the silent skip section 1 forbids. A declared set may not
be narrowed to convert failures into not-applicable ones. Such cases stay in the
denominator: `76/83` and `76 passed, 7 not-applicable` describe different runtimes,
and only the second can be checked.
Three consequences, written where they are read:
- Section 4 no longer claims scoping "asks a question both can answer". That held
only while every case was scoped to the one table both runtimes implement. Scope
narrows what is compared; it does not make every runtime a valid addressee. The
superseded sentence is named in place rather than edited away.
- Section 4 now states that "belongs to a data file" means published there, never
"shares its prefix". Live witness: the guard emits `active:oversize-input`, a flag
about its own scan cap, which carries the prefix but is no construct in the table.
A prefix-matching runtime would fail a case over a finding the corpus never claimed.
- Section 6 states the derivation's cost: case_id derives from pattern_id alone, so a
single-finding scope holds at most one case per pattern id. When a source runtime
drives two payloads at one pattern, they must be compared within scope before a
second case is minted - and a discriminated case id is forbidden, since it would
break the reverse transform.
Section 8: a pass count is unreadable without the declared set beside it, and a
not-applicable verdict proves nothing about detection in either direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVouC9nsfrfV5jRSejxbvQ
schema/finding.schema.json defines a finding `id` as DS-<scanner>-<counter>,
built from a process-global counter: stable across neither runs nor
processes, and the schema says so itself. The corpus keys its comparison on
the lexicon's stable rule identity. Two normative documents in one
repository using one word for both would produce runtimes failing every case
for a reason unrelated to detection.
Also adds spec section 3.1, which publishes the bridge a consumer actually
needs and which neither normative document named: a runtime's own label
reaches a pattern_id through the lexicon's `aliases` object, and a runtime
absent from that object has no published way to be compared -- a mapping
kept privately in a consumer is the drift this repository exists to prevent.
Regenerated all 83 fixtures; re-verified from the corpus alone against both
runtimes, 83 cases, 0 failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
The corpus was blocked on a decision nobody had to take. The 13 divergent
lexicon patterns were measured on witness inputs -- an attribute run padded
past 256 characters, an interior '<', an unclosed <script> -- and the corpus
payloads contain none of those shapes. Run through both runtimes' public
entry points, all 83 patterns produce identical lexicon finding sets, the 13
included. No winner picked, because the question was never reachable from
these inputs.
Measured at each runtime's entry point (scanForInjection() at b0de0ca,
scan_output(source=OUTPUT) at 0bf0729), never at a rebuilt regex table --
the layer mistake the divergence document already had to retract once.
- conformance/<case>/{input.txt,expected.json} x83, plus manifest.json
- spec/conformance-corpus.md, normative: bytes not text, id-only findings,
exact-within-scope, and observed_out_of_scope as evidence not expectation
Verified by a harness that does not share the generator's knowledge: reads
only the case directories, re-runs both runtimes, checks every field
including digests -- 83 cases, 0 failures. Severity agrees with what commons
publishes 83/83. Deleting the middle third of each input breaks 76 of 83
expectations; the 7 survivors are the shortest payloads, a weak mutation
rather than a weak fixture, and are recorded as such.
Scope is 83 and not 94 for a different reason than expected: the 11
non-lexicon convertible cases have no ratified cross-runtime finding id, and
writing them would mint a contract unilaterally in the same stroke as the
tag. Named in manifest.json under scope_planned.
Docs corrected in place rather than edited away: extraction-plan and
lexicon-port-divergence both claimed the 13 blocked the corpus.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhXDL82FRrQWedEmUg12Pj
