Kjell Tore Guttormsen 8d990e06d3 test(llm-security): v8 Phase 5 step 1 - golden baseline before any table swap

Records the reference artifacts Phase 5 swaps will be measured against, and
the gate that reads them. No extraction yet: this is the "before" picture,
and it had to land first or every later comparison would be confounded.
Typed test/ rather than feat/ deliberately - the only production change is
one widened export; the rest is gate, artifacts and generator.

Written failing-first (7 red on missing artifacts), then generated.

Three layers, because the plan's "assert .source/.flags of every regex" is
necessary but not sufficient:

  1. regex records - .source/.flags off the COMPILED object. After a swap a
     pattern is new RegExp(jsonString, flags), so the plan's named hazard
     (JSON backslash-doubling on 83+18 regexes) is visible here and nowhere
     else. Source-text comparison cannot see it.
  2. table records - key/value digests. Most of what Phase 4 moves is not a
     regex at all: HOMOGLYPH_MAP (x3, AS-IS), the typosquat tokens and the
     four OWASP maps are char->char and string->string data. A regex-only
     dump is blind to a broken homoglyph swap, i.e. to the bulk of the
     payload. Operator decision: widen the dump.
  3. file records - sha256 of the five moving-set sources. This dissolves
     STATE's open question (how to enumerate every regex): it is complete by
     construction, covering inline regexes in function bodies that no export
     walk reaches, with no JS parser in a zero-dep repo. A lexical count
     would have pinned a lie - severity.mjs scores 4 "regexes" that way and
     exports none.

Both layers were proven to fire, not assumed to: mutating one HOMOGLYPH_MAP
entry reddens the table layer, and widening an inline regex inside
decodeHexEscapes (unreachable by any export walk) reddens the file layer.

HOMOGLYPH_MAP is now exported from string-utils.mjs. That export is a source
change the plan already flags as a surface hazard ("private tables become
loaded"), so it is pre-paid here rather than confounding the before/after.

Reference run: the 61 showcase payloads through the real hook entry points,
sequentially - array order is semantic and the plan forbids key-sorting, so
concurrency is removed rather than sorted away. 61/61 match expectation,
which also settles the plan's open assumption that payloads.json expectations
match current behaviour. Coverage is recorded, not assumed: 47/83 patterns,
with the other 36 listed by key so the gate never implies coverage it lacks.

Suite counts are per-file, each file run alone - a total is unattributable,
and the three known timing-sensitive files flake only under concurrency.
2045 pass / 0 fail across 91 files, matching the pre-existing count exactly.
The gate's own file is excluded (it reads the artifact the run produces) and
that exclusion is named in the artifact.

Suite: 2053/2053, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez

2026-08-09 12:57:51 +02:00

2.9 KiB

Raw Blame History

Golden baseline — v8 Phase 5

Reference artifacts recorded before any commons extraction, so that each table-by-table swap in Phase 5 can be proven behaviour-preserving (or rolled back). Generated and checked by one code path:

node scripts/golden-baseline.mjs            # check only, exits 1 on drift
node scripts/golden-baseline.mjs --write    # (re-)bless
node scripts/golden-baseline.mjs --write --suite   # also refresh suite counts (slow)

The gate is tests/lib/golden-baseline.test.mjs.

The artifacts

File	What it pins	Why that layer exists
`patterns.json`	`.source` + `.flags` of every RegExp reachable from the walked modules' exports	After a swap a pattern is `new RegExp(jsonString, flags)`. The plan's named hazard — JSON backslash-doubling — is visible only on the compiled object.
↳ table records	key/value digest of `HOMOGLYPH_MAP`, `TYPOSQUAT_SUSPICIOUS_TOKENS`, `SEVERITY`, the four OWASP maps	Most of what Phase 4 moves is not a regex. A regex-only dump is blind to a broken homoglyph or OWASP-map swap, i.e. blind to the bulk of the payload.
↳ file records	sha256 of the five source files in the moving set	The completeness layer, complete by construction: it covers regexes inlined in function bodies (`NAMED` at `string-utils.mjs:291`, the BIDI/tag/PUA ranges at 357–404) that no export walk can reach.
`reference-run.json`	the 61 showcase payloads through the real hook entry points, plus coverage	Byte identity over a corpus that trips a handful of patterns would prove almost nothing, so coverage is recorded and uncovered patterns are listed by name.
`suite-counts.json`	per-file pass/fail, each file run alone	A total is unattributable, and `npm test` runs files concurrently where three timing-sensitive files flake. Running each alone is the only reproducible recording.

Why there is no regex enumerator

The obvious design — parse the sources and enumerate every regex literal — was rejected. It needs a JS parser this zero-dependency repo does not have, and a lexical approximation is contaminated by comments and division (severity.mjs scores 4 "regexes" that way and exports none). The two-layer split — export walk for what the scanners actually use, file digest for everything else — answers the same question without a parser.

During a swap

A diff here means the swap changed observable behaviour. Roll the swap back. --write is for deliberately re-blessing a change you have already decided is correct, not for making the gate quiet.

Known coverage gap

The reference run exercises 47 of 83 injection patterns. The other 36 are listed by key under coverage.uncoveredPatterns — a swap that breaks one of those is not caught by the reference run, only by the pattern dump. That is an honest limit of the corpus, not a bug in the gate; closing it means growing the conformance corpus (Phase 5 step 5).

2.9 KiB Raw Blame History Unescape Escape