llm-ingestion-pipeline-secu.../tests
Kjell Tore Guttormsen f397cd94e1 feat(lexicon): JSON injection lexicon + variant-set scan (TDD) [skip-docs]
Build order step 4 — the load-bearing port from the llm-security seed
(injection-patterns.mjs + string-utils.mjs), stdlib-only.

- injection_lexicon.json: 83 patterns (CRITICAL/HIGH/HYBRID/MEDIUM) as the
  single source of truth (regex + id + severity + owasp + desc), compiled once
  by a thin loader. Decoupled from the engine for a future TS port.
- scan_lexicon(text, source, max_scan_chars) -> Report: matches every pattern
  against a deduped variant set (raw / normalized / homoglyph-folded / rot13),
  plus unicode-tag presence signal and the cognitive-load trap.
- normalize_for_scan chain ported: unicode-tags -> bidi -> HTML-entities ->
  unicode/hex/URL escapes -> whole-string base64 (reuses entropy.try_decode_base64)
  -> collapse letter-spacing; plus fold_homoglyphs / rot13.
- Self-safety (OWASP LLM10): input-size cap (scan prefix + flag oversize) and
  ReDoS-safe port — the two nested-.*? sub-agent patterns bounded to
  (?:\S+\s+){0,N}?; verified true positives still fire.
- Non-Latin data (homoglyph map, BIDI block) built from explicit code points;
  JSON non-ASCII kept as \uXXXX escapes.

24 tests; 55 green total.

[skip-docs]: README positioning + honest-limitations is a deliberate build-order
step-11 deliverable (steps 1-3 likewise left README frozen). README status line
("pre-implementation") is stale and flagged for the step-11 refresh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 17:20:21 +02:00
..
test_entropy.py feat(entropy): high-entropy/encoded-blob detection with decode-and-rescan (TDD) 2026-07-04 09:43:24 +02:00
test_lexicon.py feat(lexicon): JSON injection lexicon + variant-set scan (TDD) [skip-docs] 2026-07-04 17:20:21 +02:00
test_report.py feat: scaffold package + report and sanitize modules (TDD) 2026-07-04 09:24:20 +02:00
test_sanitize.py feat: scaffold package + report and sanitize modules (TDD) 2026-07-04 09:24:20 +02:00