1
0
Fork 0

test(coverage): runnable threat-coverage matrix (real-case validation gate)

Add a single declarative manifest proving, in one place, every vulnerability
class the guard stops — and the documented gaps it does not. This is the
real-case validation gate ahead of any v1.0 freeze (v1.0 stays parked until
verified on real cases).

- src/llm_ingestion_guard/coverage.py: stdlib-only manifest (CORE_CASES) +
  narrated runner. `python -m llm_ingestion_guard.coverage` prints
  class -> OWASP -> expected -> observed -> verdict; exit 0 iff every caught
  class is caught and every documented gap holds. 126 caught classes + 4 gaps.
  Lexicon cases are generated from load_lexicon() via a payload dict, so a
  pattern with no payload fails loudly at import (self-verifying).
- tests/test_coverage_matrix.py: asserts total recall, that every documented
  gap still holds, and completeness (every lexicon id + every OWASP anchor has
  a case). Adds the full 25-pattern LLM02 secret-egress set (fixtures assembled
  from split tokens so no secret shape sits in source) and the container-layer
  front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink).
- README + CHANGELOG: point to the runnable matrix.

+165 tests (357 -> 522). No core dependency added.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-15 11:20:22 +02:00
commit f5eae9a16e
4 changed files with 853 additions and 0 deletions

View file

@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added — runnable threat-coverage matrix
A single declarative manifest (`llm_ingestion_guard.coverage`) that proves, in one
place, every vulnerability class the guard stops — and the documented gaps it does
not. Two consumers of the same source of truth:
- `python -m llm_ingestion_guard.coverage` — a narrated matrix
(`class -> OWASP -> expected -> observed -> verdict`); exit 0 iff every caught
class is caught and every documented gap holds. Stdlib-only, CI-usable.
- `tests/test_coverage_matrix.py` — asserts total recall over the core matrix
(carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the
contract asserters, the disposition engine, OKF T1T7), asserts every documented
gap still holds, and guards completeness (every lexicon pattern id, and every
OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress
set and the container-layer front-end classes (CSV formula-injection, zip-slip,
zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.
This is the real-case validation gate ahead of a v1.0 freeze.
## [0.2.0] — 2026-07-06
### Added — OKF adapter (stream 1)