llm-security/tests/lib
Kjell Tore Guttormsen c67bad3752 test(llm-security): v8 Phase 5 step 3 - run the 83-case commons corpus
Vendored commons v0.1.0 carries a cross-runtime conformance corpus. Until now
it was measured by a throwaway script, which makes 83/83 a claim rather than a
gate - and step 4 swaps the very tables it constrains, so the measurement has
to survive into that step or it protects nothing.

Comparison is exact-within-scope per spec section 4: every listed finding must
be raised and no other lexicon finding may be. Findings are named by commons
pattern_id, which scanForInjection() does not carry - it returns our labels.
Section 3.1 permits a runtime registered in the lexicon's aliases object to
compare through it, and we are registered. Measured first, not assumed: the
map is a total bijection, 83 labels to 83 ids, no duplicates, family membership
agreeing throughout. Nothing here restates a pattern's id, severity or label,
so nothing here can drift from the lexicon.

Deliberately NOT done: adding an id field to our 83 table entries. It would
change the source file the golden gate pins by sha256, forcing a re-bless in
the middle of a behaviour-preservation measurement, and duplicate what step 4
does anyway when the table itself starts loading from commons JSON.

Case discovery is driven by manifest.cases and cross-checked against the
directories on disk, because section 1 requires every case to run and a
deleted case dir would otherwise shrink the gate silently. An unimplemented
match or scope throws rather than skips (section 4). Input bytes and sha256 are
both verified before scanning - two fixtures carry characters invisible on
screen.

Proven red-capable in both directions by mutating the subject, not the harness:
neutering one pattern failed exactly override__disregard; widening one to
[aeiou] failed 82 cases on extra findings. Source restored byte-identical after
each.

Suite 2158, 85 new. The one red in the parallel run is the known
pre-compact-scan size-cap flake (366 ms alone, 1060 ms under load).
2026-08-10 21:00:59 +02:00
..
audit-trail.test.mjs feat(llm-security)!: v8 Phase 3 step 1 - remove the deprecated mode env-vars 2026-08-09 10:17:47 +02:00
bash-normalize.test.mjs feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command 2026-04-08 08:58:35 +02:00
build-zip.mjs feat(llm-security): /security ide-scan <url> — Marketplace/OpenVSX/direct VSIX (v6.4.0) 2026-04-17 17:16:26 +02:00
commons-loader.test.mjs test(llm-security): v8 Phase 5 step 3 - invert the unvendored-commons test 2026-08-10 20:52:04 +02:00
conformance-corpus.test.mjs test(llm-security): v8 Phase 5 step 3 - run the 83-case commons corpus 2026-08-10 21:00:59 +02:00
diff-engine-exact-pass.test.mjs fix(llm-security): misc scanner correctness — reflog FP, diff exact-pass, hex dedupe, SARIF version (#20,#22,#50,#54,#56) 2026-07-18 10:36:51 +02:00
distribution-stats.test.mjs feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command 2026-04-08 08:58:35 +02:00
doc-consistency.test.mjs docs(llm-security): trim README badge row 8 -> 4 (repo-standard BADGE-COUNT) 2026-08-04 11:50:12 +02:00
file-discovery.test.mjs fix(llm-security): normalization/discovery evasion + SIG embedded-base64 & custom rules (#21,#23,#30,#36,#42,#52,#55) 2026-07-18 10:35:56 +02:00
git-clone-gitattributes.test.mjs feat(git-clone): E12 — .gitattributes filter-driver post-clone advisory 2026-04-30 15:29:13 +02:00
git-clone-sandbox.test.mjs feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command 2026-04-08 08:58:35 +02:00
golden-baseline.test.mjs fix(llm-security): golden gate - coverage block measured something else 2026-08-09 13:06:45 +02:00
injection-patterns.test.mjs fix(llm-security): v8.x-A - close <script> recall hole, add whole-table ReDoS gate 2026-08-10 14:53:29 +02:00
mcp-description-cache.test.mjs fix(llm-security): scanner robustness — ReDoS, MCP-stdout DoS, redirect loop, atomic writes (#24,#53,#31,#25,#51) 2026-07-18 10:15:11 +02:00
output.test.mjs feat(llm-security)!: v7.0.0 commit 6 — tests, docs, version bump 2026-04-19 22:26:35 +02:00
policy-loader.test.mjs feat(llm-security)!: v8 Phase 3 step 1 - remove the deprecated mode env-vars 2026-08-09 10:17:47 +02:00
severity.test.mjs docs(severity): B3 — document info as scoring-inert (v7.2.0 prep) 2026-04-29 13:56:11 +02:00
skill-registry-atomic.test.mjs fix(llm-security): scanner robustness — ReDoS, MCP-stdout DoS, redirect loop, atomic writes (#24,#53,#31,#25,#51) 2026-07-18 10:15:11 +02:00
string-utils-hidden-unicode.test.mjs feat(unicode): E1 — extend hidden-Unicode detection to PUA-A and PUA-B 2026-04-29 14:18:49 +02:00
string-utils-homoglyph.test.mjs feat(injection): E16 — homoglyph NFKC fold before every pattern match 2026-04-29 14:22:05 +02:00
string-utils-tokens.test.mjs fix(dep): B7 — token-overlap typosquat heuristic alongside Levenshtein 2026-04-29 14:10:53 +02:00
string-utils.test.mjs fix(llm-security): normalization/discovery evasion + SIG embedded-base64 & custom rules (#21,#23,#30,#36,#42,#52,#55) 2026-07-18 10:35:56 +02:00
v8-env-removal.test.mjs feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs 2026-08-09 10:25:03 +02:00
workflow-yaml-state.test.mjs fix(llm-security): YAML/workflow parser divergence — block scalars + bare if: (#32,#33,#43) 2026-07-18 10:35:56 +02:00
yaml-frontmatter.test.mjs fix(llm-security): YAML/workflow parser divergence — block scalars + bare if: (#32,#33,#43) 2026-07-18 10:35:56 +02:00