Closes the gap STATE has been carrying since e1511f9. Section 1.1's runtime
behaviour has been correct since then -- the third verdict is real, the
declared set is one constant with two uses -- but what we PUBLISHED was a
console.log summary: the right facts in a format only its author could
parse, which is most of what section 1.1 exists to prevent. Commons
shipped a shape for it in v0.3.0
(schema/conformance-declaration.schema.json 0.1.0), so the artifact can
now exist.
Every field is counted from the run rather than restated. The cases record
their own verdict as they execute, keeping `failed` (ran and disagreed)
apart from `error` (could not run) on exactly the distinction section 1
turns on. `commons_commit` is read out of the subtree-pull subject in our
own history rather than transcribed into a constant that would drift at
the next pull, and it refuses to publish a coordinate it cannot determine
-- a fabricated commit is worse than no declaration. The artifact is
gitignored: a committed declaration keeps asserting what was true once,
and nothing makes it wrong out loud when it stops being.
Validated once against the vendored schema with a real 2020-12
implementation: VALID, and the validator proven discriminating by six
negative controls it rejected (dropped zero-count, unknown key,
out-of-enum source, non-integer count, missing enumeration, malformed
case id). Continuous validation would mean a Python dependency in a suite
that has none, so what stays is the cheap half that actually drifts -- the
two key sets, asserted exactly.
TWO DEFECTS FOUND BY MUTATING THIS GATE, both in its own first draft:
1. It lived in `after()`. Measured on Node 25.8.2: an assertion that fails
in an after hook prints under "failing tests" and marks the suite red,
but leaves `fail 0` and exit code ZERO. `npm test` and CI would have
read a falsified declaration as green. The gate against "reports
success without running" was itself reporting success without running.
It is now a test, declared last, and the verdict-count assertion is
what guards the ordering that makes "last" meaningful.
2. Nothing tied the PUBLISHED `declared_tables` to the runner's constant.
Substituting a literal list left every other assertion green, because
they all read the constant rather than what was published -- so
`declaration_source: derived-from-runner` could be a lie with no code
change to point at. Now asserted identical.
Seven mutations, all exiting non-zero: dropped zero-count, falsified
not_applicable, unpublished field, hand-maintained tables, lied-about
source, and a hardcoded `passed` combined with a genuine case failure.
Published this run: 90 total, 84 passed, 0 failed, 6 not-applicable,
0 error, at commons 4641a7b (v0.3.0). Full suite 2192 pass / 0 fail /
6 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XDdiKC9ZXmcSUQ2m84s6y
The subtree pull carries exactly one detection-data change: commons
converged `hybrid-xss:script-tag` on our open-tag-only form
(`<script\b[^>]*>`), dropping the `[\s\S]*?<\/script>` tail that was a
recall hole. Measured before the pull, not taken on their word: a
throwaway differential compared all 83 positions of the v0.3.0 lexicon
against the live source tables on source, flags, label and alias --
0 divergences, in order. The four vendored files were then re-hashed
against `git show v0.3.0:<file>` upstream; all four byte-identical.
Everything else in v0.2.0..v0.3.0 is additive: the CHANGELOG, the
divergence doc, spec text, the new §1.1 declaration schema, and one new
conformance case.
That new case is why the corpus tripwire moves. `manifest.count` is now
90 and `count_by_scope['lexicon/injection-lexicon.json']` is 84, because
`hybrid-xss__script-tag--src-no-close` gives the script pattern a SECOND
case. The tripwire fired on its own (actual 90, expected 89) rather than
being adjusted pre-emptively, so it is proven live this session.
`aliasMap.size` deliberately stays 83: the case-to-pattern relation is
now many-to-one, and only the alias map is a bijection. The header
comment says so explicitly, so the next reader does not "fix" the 83
into an 84.
Corpus: 84/84 passed, 6 not-applicable. Golden gate untouched (8/8) --
no table is built from the lexicon yet, which is the next commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XDdiKC9ZXmcSUQ2m84s6y
Commons v0.2.0 (532d70d) grew a MUST: a runtime claiming conformance must
declare the set of commons data files it implements and publish that set with
its result. The corpus went 83 -> 89; the six new cases are scoped to
signatures/active-content.json, which this runtime does not implement. Under
1 they would be six permanent failures; under 1.1 they are not-applicable, a
third verdict distinct from 1's error.
Published shape: 83/83 passed, 6 not-applicable. The six are enumerated by
name, reported as skipped rather than passed, and stay in the denominator.
The declaration is DERIVED from the same constant the runner uses to accept a
scope (DECLARED_TABLES), so it cannot drift from what the suite actually runs
- this is what was promised to commons in reply 20260811T104628Z.
Anti-narrowing was NOT enforced by construction, contrary to the claim in the
reply. Measured: setting DECLARED_TABLES to the empty set turns all 89 cases
not-applicable and leaves the suite GREEN with zero cases run - exactly the
exit 1.1 forbids. "Visible as a code change" describes a reviewer, not a gate.
Closed with a derived floor rather than a second hand-maintained table list
(which would be the parallel declaration we promised not to keep): a commons
table whose aliases name llm_security has registered this runtime as a
consumer per 3.1, and a registered consumer that stops declaring the table is
withdrawing a published claim. The universe of tables comes from the manifest;
membership comes from each table's own aliases. Measured: only the injection
lexicon names us, so the floor is one table and the other three carry no
obligation.
Mutation-proven, all six firing:
declaration -> [] green, 0 cases -> 1 fail (the defect above)
declaration -> wrong table 7 fail
over-declare an unimplemented one 6 fail
count_by_scope 83 -> 82 1 fail
drop a case from manifest.cases 2 fail
strip our alias registration 85 fail
Suite 2173 -> 2181 (+2 gates, +6 not-applicable), 2174 pass, 0 real failures.
The one red under parallel load was pre-compact-scan size-cap, a known timing
flake: 358 ms alone against a 1000 ms cap, 1886 ms under load. Budget untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uj6iB8uUUUFHLgAya1hZt7
Vendored commons v0.1.0 carries a cross-runtime conformance corpus. Until now
it was measured by a throwaway script, which makes 83/83 a claim rather than a
gate - and step 4 swaps the very tables it constrains, so the measurement has
to survive into that step or it protects nothing.
Comparison is exact-within-scope per spec section 4: every listed finding must
be raised and no other lexicon finding may be. Findings are named by commons
pattern_id, which scanForInjection() does not carry - it returns our labels.
Section 3.1 permits a runtime registered in the lexicon's aliases object to
compare through it, and we are registered. Measured first, not assumed: the
map is a total bijection, 83 labels to 83 ids, no duplicates, family membership
agreeing throughout. Nothing here restates a pattern's id, severity or label,
so nothing here can drift from the lexicon.
Deliberately NOT done: adding an id field to our 83 table entries. It would
change the source file the golden gate pins by sha256, forcing a re-bless in
the middle of a behaviour-preservation measurement, and duplicate what step 4
does anyway when the table itself starts loading from commons JSON.
Case discovery is driven by manifest.cases and cross-checked against the
directories on disk, because section 1 requires every case to run and a
deleted case dir would otherwise shrink the gate silently. An unimplemented
match or scope throws rather than skips (section 4). Input bytes and sha256 are
both verified before scanning - two fixtures carry characters invisible on
screen.
Proven red-capable in both directions by mutating the subject, not the harness:
neutering one pattern failed exactly override__disregard; widening one to
[aeiou] failed 82 cases on extra findings. Source restored byte-identical after
each.
Suite 2158, 85 new. The one red in the parallel run is the known
pre-compact-scan size-cap flake (366 ms alone, 1060 ms under load).
