1
0
Fork 0

fix(owasp): qualify every catalog number with the 2026 edition

OWASP's Top 10 for LLM Applications 2026 renumbered the list; only LLM01
and LLM02 kept their meaning. Every finding's `owasp` was written against
2025 with no year, so EchoLeak findings said LLM05, which in 2026 means
Data and Model Poisoning. Row 2 of the 2026 done-criterion counted 95
such sites at 6702d7a (82 old meaning, 13 wrong); it is now 0.

- Producers name the risk, never the number: one table (`_owasp.py`)
  owns `LLMnn:2026`. The vendored lexicon stays byte-equal with commons
  and gets its year where the finding is made; a commons number whose
  meaning moved between editions raises instead of being guessed.
- Mechanical map from the 2025 meaning (verified against the publisher
  in session 1): LLM05 -> LLM10:2026, LLM06 -> LLM03:2026,
  LLM10 -> LLM06:2026, LLM08 -> LLM09:2026, LLM04 -> LLM05:2026.

The 13 wrong-meaning sites, judged:
- Factual poisoning (PoisonedRAG gap, and the test that follows it):
  LLM05:2026. The 2026 text names "RAG Knowledge Base Poisoning" as its
  example 6. It said LLM09, which was Misinformation.
- Fail-secure (README row, matrix row): no number. It is the guard's own
  failure semantics; no edition describes it. Was LLM10.
- Parser-surface refusals (nested mapping, block scalar, quoted flow
  element, off-allowlist key): no number. Refusing a shape the parser
  would have to interpret is not resource consumption. Was LLM10. The
  anchor and merge-key rows stay LLM06:2026 (alias expansion is DoS).
- `resource` URL and cross-link scheme validation (4): no number. They
  validate received bundle metadata; LLM10:2026 is about model output
  and says input validation belongs to LLM01. Was LLM05.
- `_raise_case` default (OKF path traversal, absolute path, reserved
  name, non-.md, bundle escape): no number. None is prompt injection.
  Was LLM01.
A doubtful number would be the misreading the year exists to prevent,
so `Case.owasp` is now Optional and the matrix prints "-".

Self-safety (input caps, ReDoS) keeps LLM06:2026: the 2026 text, read at
the publisher, covers "input flood ... exploiting processing
inefficiencies" and application-side rechunking. The README's LLM09
anchor (Misinformation) is dropped; it was the factual-poisoning row.

The permanent test (tests/test_catalog_numbers.py) fails on an old or
wrong number, on a number without the year outside the vendored
lexicon, and on any finding the coverage matrix drives whose `owasp`
lacks the newest year; "newest" is read from the catalog register. The
census holds its wrong-rules to their counts at the frozen revision
only, so the frozen listing (da1aa0fa...6f) is unchanged.
This commit is contained in:
Kjell Tore Guttormsen 2026-10-31 14:20:35 +01:00
commit 1c6816cdfe
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
32 changed files with 405 additions and 145 deletions

View file

@ -8,7 +8,7 @@ Three layers, one honest picture:
test flips and the docs must change). The **completeness guard** asserts every
lexicon pattern id has a case, so the matrix cannot silently fall behind the
lexicon.
2. **The full LLM02 secret-egress set** — all secret patterns, one case each.
2. **The full LLM02:2026 secret-egress set** — all secret patterns, one case each.
Every fixture is assembled at call time from tokens split across ``""`` joins,
so the *source* carries no recognizable secret shape (prefix, scheme, PEM dash
run) — the pre-write secret hook and gitleaks stay green — while the runtime
@ -74,11 +74,12 @@ def test_every_lexicon_pattern_has_a_coverage_case():
def test_matrix_covers_every_owasp_llm_class_it_claims():
# Every OWASP-LLM anchor the guard advertises should appear in the matrix.
owasp = {case.owasp for case in CORE_CASES}
for expected in ("LLM01", "LLM02", "LLM05", "LLM06", "LLM09", "LLM10"):
for expected in ("LLM01:2026", "LLM02:2026", "LLM03:2026", "LLM05:2026", "LLM06:2026",
"LLM10:2026"):
assert expected in owasp, f"{expected} claimed but not represented in the matrix"
# --- layer 2: the full LLM02 secret-egress set ------------------------------
# --- layer 2: the full LLM02:2026 secret-egress set ------------------------------
# Every secret token is split across `_mk` argument boundaries so the source
# contains no matchable secret shape (prefix / scheme / PEM dash run); `_mk`
# rejoins the parts into a well-formed synthetic secret at call time.
@ -117,7 +118,7 @@ _EGRESS_PAYLOADS = {
def test_egress_fixture_covers_every_secret_pattern():
# Completeness guard for LLM02: every shipped secret pattern has a case.
# Completeness guard for LLM02:2026: every shipped secret pattern has a case.
assert set(_EGRESS_PAYLOADS) == {pattern.id for pattern in _SECRET_PATTERNS}