Kjell Tore Guttormsen
1c6816cdfe
OWASP's Top 10 for LLM Applications 2026 renumbered the list; only LLM01
and LLM02 kept their meaning. Every finding's `owasp` was written against
2025 with no year, so EchoLeak findings said LLM05, which in 2026 means
Data and Model Poisoning. Row 2 of the 2026 done-criterion counted 95
such sites at 6702d7a (82 old meaning, 13 wrong); it is now 0.
- Producers name the risk, never the number: one table (`_owasp.py`)
owns `LLMnn:2026`. The vendored lexicon stays byte-equal with commons
and gets its year where the finding is made; a commons number whose
meaning moved between editions raises instead of being guessed.
- Mechanical map from the 2025 meaning (verified against the publisher
in session 1): LLM05 -> LLM10:2026, LLM06 -> LLM03:2026,
LLM10 -> LLM06:2026, LLM08 -> LLM09:2026, LLM04 -> LLM05:2026.
The 13 wrong-meaning sites, judged:
- Factual poisoning (PoisonedRAG gap, and the test that follows it):
LLM05:2026. The 2026 text names "RAG Knowledge Base Poisoning" as its
example 6. It said LLM09, which was Misinformation.
- Fail-secure (README row, matrix row): no number. It is the guard's own
failure semantics; no edition describes it. Was LLM10.
- Parser-surface refusals (nested mapping, block scalar, quoted flow
element, off-allowlist key): no number. Refusing a shape the parser
would have to interpret is not resource consumption. Was LLM10. The
anchor and merge-key rows stay LLM06:2026 (alias expansion is DoS).
- `resource` URL and cross-link scheme validation (4): no number. They
validate received bundle metadata; LLM10:2026 is about model output
and says input validation belongs to LLM01. Was LLM05.
- `_raise_case` default (OKF path traversal, absolute path, reserved
name, non-.md, bundle escape): no number. None is prompt injection.
Was LLM01.
A doubtful number would be the misreading the year exists to prevent,
so `Case.owasp` is now Optional and the matrix prints "-".
Self-safety (input caps, ReDoS) keeps LLM06:2026: the 2026 text, read at
the publisher, covers "input flood ... exploiting processing
inefficiencies" and application-side rechunking. The README's LLM09
anchor (Misinformation) is dropped; it was the factual-poisoning row.
The permanent test (tests/test_catalog_numbers.py) fails on an old or
wrong number, on a number without the year outside the vendored
lexicon, and on any finding the coverage matrix drives whose `owasp`
lacks the newest year; "newest" is read from the catalog register. The
census holds its wrong-rules to their counts at the frozen revision
only, so the frozen listing (da1aa0fa...6f) is unchanged.
108 lines
4.5 KiB
Python
108 lines
4.5 KiB
Python
"""Tests for the input-size cap on the TRANSFORM surfaces (OWASP LLM06:2026).
|
|
|
|
The scan surfaces (``scan_lexicon`` / ``scan_output``) cap by *truncating*: they
|
|
read the prefix, emit an ``oversize-input`` finding, and return findings only.
|
|
That trade does not transfer here. ``sanitize`` / ``fence`` / ``neutralize``
|
|
return **content**, so truncating would either drop user data silently or hand
|
|
back a tail that never went through the transform — a one-line bypass, since an
|
|
attacker controls where in the document the payload sits.
|
|
|
|
So these three fail *secure* instead. The invariant under test:
|
|
|
|
returned text is ALWAYS fully transformed, or not returned at all.
|
|
"""
|
|
import inspect
|
|
|
|
import pytest
|
|
|
|
from llm_ingestion_guard import prepare_input
|
|
from llm_ingestion_guard.calibration import MAX_INPUT_CHARS
|
|
from llm_ingestion_guard.contract import ContractViolation, OversizeInputError
|
|
from llm_ingestion_guard.fence import fence
|
|
from llm_ingestion_guard.neutralize import neutralize
|
|
from llm_ingestion_guard.sanitize import sanitize
|
|
|
|
# Every transform surface, called through a uniform (text, max) shim so the
|
|
# invariant is asserted once per surface rather than restated three times.
|
|
TRANSFORMS = (
|
|
("sanitize", lambda text, cap: sanitize(text, max_input_chars=cap)),
|
|
("fence", lambda text, cap: fence(text, max_input_chars=cap)),
|
|
("neutralize", lambda text, cap: neutralize(text, max_input_chars=cap)),
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
|
def test_oversize_input_is_rejected(name, call):
|
|
with pytest.raises(OversizeInputError):
|
|
call("a" * 101, 100)
|
|
|
|
|
|
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
|
def test_input_exactly_at_the_cap_is_accepted(name, call):
|
|
# The cap is the largest accepted size, not the smallest rejected one.
|
|
result = call("a" * 100, 100)
|
|
assert result.text is not None
|
|
|
|
|
|
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
|
def test_rejection_is_a_contract_violation(name, call):
|
|
# Subclass, so a pipeline with a broad `except ContractViolation` around its
|
|
# quarantined stage keeps failing closed instead of meeting a new exception
|
|
# type it has never heard of.
|
|
with pytest.raises(ContractViolation) as exc:
|
|
call("a" * 101, 100)
|
|
assert exc.value.code == "oversize-input"
|
|
|
|
|
|
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
|
def test_error_carries_sizes_never_content(name, call):
|
|
# Same alert-safety property ContractViolation already promises: the raised
|
|
# object must be routable to an alert channel without leaking the payload.
|
|
secret = "CANARYVALUE"
|
|
with pytest.raises(OversizeInputError) as exc:
|
|
call(secret * 100, 100)
|
|
assert secret not in str(exc.value)
|
|
assert not any(secret in d for d in exc.value.details)
|
|
# Substring, not tuple membership: `details` is ("sanitize",), so `"101" not
|
|
# in details` would pass trivially and keep passing if a size were ever
|
|
# folded into the string.
|
|
assert not any("101" in d for d in exc.value.details)
|
|
assert name in exc.value.details
|
|
|
|
|
|
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
|
def test_error_reports_both_the_size_and_the_cap(name, call):
|
|
with pytest.raises(OversizeInputError) as exc:
|
|
call("a" * 101, 100)
|
|
message = str(exc.value)
|
|
assert "101" in message and "100" in message
|
|
|
|
|
|
@pytest.mark.parametrize("name,call", TRANSFORMS, ids=[n for n, _ in TRANSFORMS])
|
|
def test_under_cap_is_unchanged_by_the_guard(name, call):
|
|
# The cap must not perturb the ordinary path: clean prose still round-trips
|
|
# the way each surface's own tests already pin.
|
|
result = call("ordinary prose", 100)
|
|
assert "ordinary prose" in result.text
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"func", [sanitize, fence, neutralize], ids=["sanitize", "fence", "neutralize"]
|
|
)
|
|
def test_default_cap_comes_from_calibration(func):
|
|
# Calibration numbers live in ONE module; a literal re-typed at a call site
|
|
# is the drift this pins against.
|
|
default = inspect.signature(func).parameters["max_input_chars"].default
|
|
assert default == MAX_INPUT_CHARS
|
|
|
|
|
|
def test_prepare_input_inherits_the_cap():
|
|
# The input path is sanitize -> fence, and step 1 is where the document
|
|
# arrives, so the whole path is bounded by sanitize's guard alone.
|
|
with pytest.raises(OversizeInputError):
|
|
prepare_input("a" * (MAX_INPUT_CHARS + 1))
|
|
|
|
|
|
def test_prepare_input_accepts_a_document_at_the_cap():
|
|
result = prepare_input("a" * MAX_INPUT_CHARS)
|
|
assert result.fenced
|