1
0
Fork 0

test(coverage): runnable threat-coverage matrix (real-case validation gate)

Add a single declarative manifest proving, in one place, every vulnerability
class the guard stops — and the documented gaps it does not. This is the
real-case validation gate ahead of any v1.0 freeze (v1.0 stays parked until
verified on real cases).

- src/llm_ingestion_guard/coverage.py: stdlib-only manifest (CORE_CASES) +
  narrated runner. `python -m llm_ingestion_guard.coverage` prints
  class -> OWASP -> expected -> observed -> verdict; exit 0 iff every caught
  class is caught and every documented gap holds. 126 caught classes + 4 gaps.
  Lexicon cases are generated from load_lexicon() via a payload dict, so a
  pattern with no payload fails loudly at import (self-verifying).
- tests/test_coverage_matrix.py: asserts total recall, that every documented
  gap still holds, and completeness (every lexicon id + every OWASP anchor has
  a case). Adds the full 25-pattern LLM02 secret-egress set (fixtures assembled
  from split tokens so no secret shape sits in source) and the container-layer
  front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink).
- README + CHANGELOG: point to the runnable matrix.

+165 tests (357 -> 522). No core dependency added.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-15 11:20:22 +02:00
commit f5eae9a16e
4 changed files with 853 additions and 0 deletions

View file

@ -73,6 +73,25 @@ machinery, and the contract asserters `assert_tool_less` /
`assert_credential_allowlist` / `scoped_env`. See
[the end-to-end showcase](tests/test_showcase.py) for a full worked pipeline.
## Verify what it stops (coverage matrix)
Before wiring the guard into your pipeline, see — in one place — every
vulnerability class it stops, and the ones it deliberately does not:
```bash
python -m llm_ingestion_guard.coverage # prints the matrix; exit 0 = all as documented
```
Each row drives the real guard with a live payload and prints
`class -> OWASP anchor -> expected -> observed -> verdict`. The manifest
([`coverage.py`](src/llm_ingestion_guard/coverage.py)) is the single source of
truth for [`tests/test_coverage_matrix.py`](tests/test_coverage_matrix.py), which
asserts **total recall** across every defended class, that **every documented gap
still holds** (a closed gap fails the test, forcing a doc update), and — the
honesty guard — that **every lexicon pattern has a case** so the matrix cannot
fall behind the lexicon. The full LLM02 secret-egress set and the container-layer
front-end (CSV formula-injection, zip-slip/bomb, symlink) are asserted there too.
## The reusable contract (adopt-this checklist)
The actual product is this checklist, encoded as code you wire in order: