llm-security/tests
Kjell Tore Guttormsen c67bad3752 test(llm-security): v8 Phase 5 step 3 - run the 83-case commons corpus
Vendored commons v0.1.0 carries a cross-runtime conformance corpus. Until now
it was measured by a throwaway script, which makes 83/83 a claim rather than a
gate - and step 4 swaps the very tables it constrains, so the measurement has
to survive into that step or it protects nothing.

Comparison is exact-within-scope per spec section 4: every listed finding must
be raised and no other lexicon finding may be. Findings are named by commons
pattern_id, which scanForInjection() does not carry - it returns our labels.
Section 3.1 permits a runtime registered in the lexicon's aliases object to
compare through it, and we are registered. Measured first, not assumed: the
map is a total bijection, 83 labels to 83 ids, no duplicates, family membership
agreeing throughout. Nothing here restates a pattern's id, severity or label,
so nothing here can drift from the lexicon.

Deliberately NOT done: adding an id field to our 83 table entries. It would
change the source file the golden gate pins by sha256, forcing a re-bless in
the middle of a behaviour-preservation measurement, and duplicate what step 4
does anyway when the table itself starts loading from commons JSON.

Case discovery is driven by manifest.cases and cross-checked against the
directories on disk, because section 1 requires every case to run and a
deleted case dir would otherwise shrink the gate silently. An unimplemented
match or scope throws rather than skips (section 4). Input bytes and sha256 are
both verified before scanning - two fixtures carry characters invisible on
screen.

Proven red-capable in both directions by mutating the subject, not the harness:
neutering one pattern failed exactly override__disregard; widening one to
[aeiou] failed 82 cases on extra findings. Source restored byte-identical after
each.

Suite 2158, 85 new. The one red in the parallel run is the known
pre-compact-scan size-cap flake (366 ms alone, 1060 ms under load).
2026-08-10 21:00:59 +02:00
..
e2e test(llm-security): cover TRG/SIG/AST in e2e pipeline + SIG miner/hacktool/rot13 families (#58,#59) 2026-07-18 10:46:19 +02:00
fixtures feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs 2026-08-09 10:25:03 +02:00
golden fix(llm-security): v8.x-A - close <script> recall hole, add whole-table ReDoS gate 2026-08-10 14:53:29 +02:00
helpers test(llm-security): add JetBrains fixture tree + build helper 2026-04-18 10:49:49 +02:00
hooks feat(llm-security)!: v8 Phase 3 step 1 - remove the deprecated mode env-vars 2026-08-09 10:17:47 +02:00
lib test(llm-security): v8 Phase 5 step 3 - run the 83-case commons corpus 2026-08-10 21:00:59 +02:00
scanners feat(llm-security)!: v8 Phase 3 complete - riskScoreV1, posture heuristic, docs 2026-08-09 10:25:03 +02:00