test(llm-security): v8 Phase 5 step 1 - golden baseline before any table swap

Records the reference artifacts Phase 5 swaps will be measured against, and
the gate that reads them. No extraction yet: this is the "before" picture,
and it had to land first or every later comparison would be confounded.
Typed test/ rather than feat/ deliberately - the only production change is
one widened export; the rest is gate, artifacts and generator.

Written failing-first (7 red on missing artifacts), then generated.

Three layers, because the plan's "assert .source/.flags of every regex" is
necessary but not sufficient:

  1. regex records - .source/.flags off the COMPILED object. After a swap a
     pattern is new RegExp(jsonString, flags), so the plan's named hazard
     (JSON backslash-doubling on 83+18 regexes) is visible here and nowhere
     else. Source-text comparison cannot see it.
  2. table records - key/value digests. Most of what Phase 4 moves is not a
     regex at all: HOMOGLYPH_MAP (x3, AS-IS), the typosquat tokens and the
     four OWASP maps are char->char and string->string data. A regex-only
     dump is blind to a broken homoglyph swap, i.e. to the bulk of the
     payload. Operator decision: widen the dump.
  3. file records - sha256 of the five moving-set sources. This dissolves
     STATE's open question (how to enumerate every regex): it is complete by
     construction, covering inline regexes in function bodies that no export
     walk reaches, with no JS parser in a zero-dep repo. A lexical count
     would have pinned a lie - severity.mjs scores 4 "regexes" that way and
     exports none.

Both layers were proven to fire, not assumed to: mutating one HOMOGLYPH_MAP
entry reddens the table layer, and widening an inline regex inside
decodeHexEscapes (unreachable by any export walk) reddens the file layer.

HOMOGLYPH_MAP is now exported from string-utils.mjs. That export is a source
change the plan already flags as a surface hazard ("private tables become
loaded"), so it is pre-paid here rather than confounding the before/after.

Reference run: the 61 showcase payloads through the real hook entry points,
sequentially - array order is semantic and the plan forbids key-sorting, so
concurrency is removed rather than sorted away. 61/61 match expectation,
which also settles the plan's open assumption that payloads.json expectations
match current behaviour. Coverage is recorded, not assumed: 47/83 patterns,
with the other 36 listed by key so the gate never implies coverage it lacks.

Suite counts are per-file, each file run alone - a total is unattributable,
and the three known timing-sensitive files flake only under concurrency.
2045 pass / 0 fail across 91 files, matching the pre-existing count exactly.
The gate's own file is excluded (it reads the artifact the run produces) and
that exclusion is named in the artifact.

Suite: 2053/2053, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 12:57:51 +02:00
commit 8d990e06d3
8 changed files with 2524 additions and 1 deletions

46
tests/golden/README.md Normal file
View file

@ -0,0 +1,46 @@
# Golden baseline — v8 Phase 5
Reference artifacts recorded **before** any commons extraction, so that each
table-by-table swap in Phase 5 can be proven behaviour-preserving (or rolled
back). Generated and checked by one code path:
```bash
node scripts/golden-baseline.mjs # check only, exits 1 on drift
node scripts/golden-baseline.mjs --write # (re-)bless
node scripts/golden-baseline.mjs --write --suite # also refresh suite counts (slow)
```
The gate is `tests/lib/golden-baseline.test.mjs`.
## The artifacts
| File | What it pins | Why that layer exists |
|------|--------------|-----------------------|
| `patterns.json` | `.source` + `.flags` of every RegExp reachable from the walked modules' exports | After a swap a pattern is `new RegExp(jsonString, flags)`. The plan's named hazard — JSON backslash-doubling — is visible **only** on the compiled object. |
| ↳ table records | key/value digest of `HOMOGLYPH_MAP`, `TYPOSQUAT_SUSPICIOUS_TOKENS`, `SEVERITY`, the four OWASP maps | Most of what Phase 4 moves is not a regex. A regex-only dump is blind to a broken homoglyph or OWASP-map swap, i.e. blind to the bulk of the payload. |
| ↳ file records | sha256 of the five source files in the moving set | The completeness layer, complete **by construction**: it covers regexes inlined in function bodies (`NAMED` at `string-utils.mjs:291`, the BIDI/tag/PUA ranges at 357404) that no export walk can reach. |
| `reference-run.json` | the 61 showcase payloads through the real hook entry points, plus coverage | Byte identity over a corpus that trips a handful of patterns would prove almost nothing, so coverage is recorded and uncovered patterns are listed by name. |
| `suite-counts.json` | per-file pass/fail, each file run alone | A total is unattributable, and `npm test` runs files concurrently where three timing-sensitive files flake. Running each alone is the only reproducible recording. |
## Why there is no regex enumerator
The obvious design — parse the sources and enumerate every regex literal — was
rejected. It needs a JS parser this zero-dependency repo does not have, and a
lexical approximation is contaminated by comments and division (`severity.mjs`
scores 4 "regexes" that way and exports none). The two-layer split — export
walk for what the scanners actually use, file digest for everything else —
answers the same question without a parser.
## During a swap
A diff here means the swap changed observable behaviour. **Roll the swap back.**
`--write` is for deliberately re-blessing a change you have already decided is
correct, not for making the gate quiet.
## Known coverage gap
The reference run exercises 47 of 83 injection patterns. The other 36 are
listed by key under `coverage.uncoveredPatterns` — a swap that breaks one of
those is not caught by the reference run, only by the pattern dump. That is an
honest limit of the corpus, not a bug in the gate; closing it means growing
the conformance corpus (Phase 5 step 5).