test(coverage): runnable threat-coverage matrix (real-case validation gate)
Add a single declarative manifest proving, in one place, every vulnerability class the guard stops — and the documented gaps it does not. This is the real-case validation gate ahead of any v1.0 freeze (v1.0 stays parked until verified on real cases). - src/llm_ingestion_guard/coverage.py: stdlib-only manifest (CORE_CASES) + narrated runner. `python -m llm_ingestion_guard.coverage` prints class -> OWASP -> expected -> observed -> verdict; exit 0 iff every caught class is caught and every documented gap holds. 126 caught classes + 4 gaps. Lexicon cases are generated from load_lexicon() via a payload dict, so a pattern with no payload fails loudly at import (self-verifying). - tests/test_coverage_matrix.py: asserts total recall, that every documented gap still holds, and completeness (every lexicon id + every OWASP anchor has a case). Adds the full 25-pattern LLM02 secret-egress set (fixtures assembled from split tokens so no secret shape sits in source) and the container-layer front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink). - README + CHANGELOG: point to the runnable matrix. +165 tests (357 -> 522). No core dependency added.
This commit is contained in:
parent
66f3cbf4f5
commit
f5eae9a16e
4 changed files with 853 additions and 0 deletions
19
README.md
19
README.md
|
|
@ -73,6 +73,25 @@ machinery, and the contract asserters `assert_tool_less` /
|
|||
`assert_credential_allowlist` / `scoped_env`. See
|
||||
[the end-to-end showcase](tests/test_showcase.py) for a full worked pipeline.
|
||||
|
||||
## Verify what it stops (coverage matrix)
|
||||
|
||||
Before wiring the guard into your pipeline, see — in one place — every
|
||||
vulnerability class it stops, and the ones it deliberately does not:
|
||||
|
||||
```bash
|
||||
python -m llm_ingestion_guard.coverage # prints the matrix; exit 0 = all as documented
|
||||
```
|
||||
|
||||
Each row drives the real guard with a live payload and prints
|
||||
`class -> OWASP anchor -> expected -> observed -> verdict`. The manifest
|
||||
([`coverage.py`](src/llm_ingestion_guard/coverage.py)) is the single source of
|
||||
truth for [`tests/test_coverage_matrix.py`](tests/test_coverage_matrix.py), which
|
||||
asserts **total recall** across every defended class, that **every documented gap
|
||||
still holds** (a closed gap fails the test, forcing a doc update), and — the
|
||||
honesty guard — that **every lexicon pattern has a case** so the matrix cannot
|
||||
fall behind the lexicon. The full LLM02 secret-egress set and the container-layer
|
||||
front-end (CSV formula-injection, zip-slip/bomb, symlink) are asserted there too.
|
||||
|
||||
## The reusable contract (adopt-this checklist)
|
||||
|
||||
The actual product is this checklist, encoded as code you wire in order:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue