test(redos): one CPU clock for every bound, and a second row measured dead
`5667063` moved test_output.py's ReDoS bounds off the wall clock, because a loaded machine steals wall seconds without adding any cycles and two rows failed at 2.24s / 3.66s against a 2.0s bound while census had the CPU. The remaining ten bounds in five other files still ran on `time.monotonic()` and carried the same defect. They now share ONE clock. The clock is IMPORTED, not copied: `tests/redos_clock.py`. Five private copies would leave four of them unpinned -- the instrument test (test_the_redos_clock_ignores_time_this_process_did_not_spend) can only pin the implementation it calls, and the suite already holds that rule for the code it measures. Every ported row was verified the only way a time bound can be: the vulnerable form patched back in, red demanded, `git checkout --` after. Measured against the 2.0s bound (3.0s for the url arm): active_content long-attr `{0,63}` -> `*` RED neutralize long-attr same patch RED output gate long-attr same patch 12.41s okf link graph `[^\]\[]` -> `[^\]]` 6.91s sanitize comment str.find -> `<!--.*?-->` 17.56s lexicon md-link-anchor-text 319.14s lexicon md-link-anchor-url 8.55s lexicon md-link-ref-comment 37.82s Two rows did not go red, for two different reasons. test_sanitize.py::test_legitimate_comment_heavy_document is the legitimate SIDE of a separation, not a second pin on the defect: closed comments never withhold the required literal, so the lazy form runs it in 0.016s. Recorded in place. test_lexicon.py::test_redos_pathological_subagent_input_returns_fast is DEAD -- the same zero-signal shape the `<a ` carrier had, found by the same method. The seed form is `(?:.*?\s+)?` (llm-security 7.8.0, injection-patterns.mjs:84) and this repo has never carried it: the bounded `{0,12}?` port is in the pattern table's first commit. Patched in by hand at the row's own size: shipped 0.135s vs seed 0.113s, separation 1.2x. Not the keyword gate either -- a variant that reaches the inner branch stays linear over four doublings (exponent ~1.0), because the nesting is one lazy run inside an OPTIONAL group, never a repeated one. Left standing with the measurement written into it; picking a new carrier is an operator call, like the wall-clock row above it. The dead sibling row named in STATE is fixed: test_active_content.py's long-attribute row swaps carrier `<a ` -> `<script `, for the reason `5667063` established on its composed-gate twin -- 0.7.0's own no-URL narrowing put `<a>` in `_URL_AFFORDANCE_TAGS`, so the tag returns inert BEFORE its body reaches the arm the row guards. Re-measured here, not inherited: `<a ` 0.041s and NO findings against the vulnerable form; `<script ` 19.349s against 0.052s shipped, 373x apart. `test_pathological_input_returns_within_a_bound` deliberately keeps its wall clock (operator decision): it claims to catch a hang, and only a wall clock catches one. 792 tests, 129/129, 6/6.
This commit is contained in:
parent
566706360a
commit
c48a2923ac
7 changed files with 112 additions and 54 deletions
35
tests/redos_clock.py
Normal file
35
tests/redos_clock.py
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
"""The one clock every ReDoS bound in this suite is measured against.
|
||||
|
||||
Process CPU time, not wall clock: a ReDoS blowup is spent cycles, and a loaded
|
||||
machine steals wall clock without adding any. In 0.7.0 these bounds ran on
|
||||
``time.monotonic()`` and two of them failed at 2.24s / 3.66s against a 2.0s
|
||||
bound while two census processes had the CPU; the same rows passed 3/3 on an
|
||||
idle machine. The scans had not slowed down — they were descheduled.
|
||||
|
||||
This lives in its own module, imported by all six test files, rather than being
|
||||
copied into each. The suite already holds that rule for the code it measures
|
||||
("never re-implement a predicate you measure — import it"), and it binds harder
|
||||
here: ``test_output.py::test_the_redos_clock_ignores_time_this_process_did_not_spend``
|
||||
pins ONE implementation. Five copies would leave four of them unpinned and free
|
||||
to drift back to a wall clock without a single test going red.
|
||||
|
||||
What this clock gives up: a scan that BLOCKS forever burns no CPU, so it would
|
||||
hang the suite instead of failing it. Acceptable for every caller here — these
|
||||
scanners are pure regex over an in-memory string, with no I/O and no locks, so
|
||||
the only way they can be slow is by spending cycles. It is also why
|
||||
``test_output.py::test_pathological_input_returns_within_a_bound`` deliberately
|
||||
keeps a wall clock: that row claims to catch "a hang or a blowup", and only a
|
||||
wall clock catches the first.
|
||||
"""
|
||||
import time
|
||||
|
||||
|
||||
def scan_seconds(scanner, payload) -> float:
|
||||
"""CPU seconds ``scanner(payload)`` cost.
|
||||
|
||||
Pinned by ``test_the_redos_clock_ignores_time_this_process_did_not_spend``
|
||||
in ``test_output.py``, which carries the measurements behind the choice.
|
||||
"""
|
||||
start = time.process_time()
|
||||
scanner(payload)
|
||||
return time.process_time() - start
|
||||
|
|
@ -18,8 +18,6 @@ from __future__ import annotations
|
|||
|
||||
import pytest
|
||||
|
||||
import time
|
||||
|
||||
from llm_ingestion_guard import (
|
||||
scan_active_content,
|
||||
scan_output,
|
||||
|
|
@ -29,6 +27,7 @@ from llm_ingestion_guard import (
|
|||
)
|
||||
from llm_ingestion_guard.okf import import_bundle, Origin, Channel
|
||||
from llm_ingestion_guard.report import Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
# The zero-click EchoLeak primitive: an auto-fetched markdown image URL.
|
||||
_ECHOLEAK = ""
|
||||
|
|
@ -447,10 +446,25 @@ _ATTR_REDOS_N = 100_000
|
|||
|
||||
|
||||
def test_crafted_long_attribute_tag_stays_bounded():
|
||||
payload = "<a " + "A" * _ATTR_REDOS_N + ">"
|
||||
start = time.monotonic()
|
||||
scan_active_content(payload)
|
||||
assert time.monotonic() - start < 2.0
|
||||
# The carrier is `<script `, not the `<a ` this row shipped with through
|
||||
# 0.7.0, because 0.7.0's own no-URL narrowing killed the row: `<a>` is in
|
||||
# `_URL_AFFORDANCE_TAGS`, so a bare `<a ...>` carrying no URL attribute is
|
||||
# inert and returns BEFORE its body reaches `URL_IN_TEXT_RE` — the arm this
|
||||
# row exists to guard. Re-measured here with the pre-fix uncapped scheme run
|
||||
# patched back in, at _ATTR_REDOS_N through `scan_active_content`:
|
||||
#
|
||||
# <a ...> 0.041s and NO findings <- dead: never reaches the arm
|
||||
# <script ...> 19.349s and one finding <- the arm, still quadratic
|
||||
#
|
||||
# So the `<a ` row was green against the vulnerable form — separation 1.2x,
|
||||
# zero signal. With `<script ` it is 0.052s shipped vs 19.349s vulnerable,
|
||||
# 373x apart, with the bound 38x above the shipped side. `<script>` is the
|
||||
# durable carrier: active by NAME with no attributes at all, so no future
|
||||
# URL-shaped narrowing can make it inert the way it just did to `<a >`.
|
||||
# Same fix, same reason, as test_output.py::test_gate_is_bounded_on_the_
|
||||
# long_attribute_arm — the composed-gate twin of this row.
|
||||
payload = "<script " + "A" * _ATTR_REDOS_N + ">"
|
||||
assert scan_seconds(scan_active_content, payload) < 2.0
|
||||
|
||||
|
||||
def test_url_defanging_survives_the_redos_fix():
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@ Detection is ``text -> findings`` (design principle 3): pure, no I/O, no
|
|||
mutation. Disposition (WARN / QUARANTINE / FAIL_SECURE) is the caller's.
|
||||
"""
|
||||
import base64
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
|
@ -24,6 +23,7 @@ from llm_ingestion_guard.lexicon import (
|
|||
scan_lexicon,
|
||||
)
|
||||
from llm_ingestion_guard.report import Report, Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
# --- loader ------------------------------------------------------------------
|
||||
|
|
@ -230,12 +230,34 @@ def test_oversize_input_is_capped_and_flagged():
|
|||
def test_redos_pathological_subagent_input_returns_fast():
|
||||
# A crafted string that would force catastrophic backtracking on the
|
||||
# ORIGINAL nested-`.*?` sub-agent pattern. The bounded port stays linear.
|
||||
#
|
||||
# MEASURED DEAD, and left standing pending an operator decision — the same
|
||||
# zero-signal shape the `<a ` carrier had in test_active_content.py, found by
|
||||
# the same method (patch the vulnerable form back in and demand red). The
|
||||
# seed's actual form is `(?:.*?\s+)?` (llm-security 7.8.0,
|
||||
# scanners/lib/injection-patterns.mjs:84); this repo has never carried it —
|
||||
# the bounded `{0,12}?` port is in the pattern table's FIRST commit (f397cd9),
|
||||
# so there is no in-repo form to revert to. Patched in by hand, at the row's
|
||||
# own 8000-word size:
|
||||
#
|
||||
# shipped 0.135s seed form 0.113s <- separation 1.2x, no signal
|
||||
#
|
||||
# Not a payload-size problem and not the keyword gate either: the payload
|
||||
# never supplies the trailing keyword the outer alternation requires, and a
|
||||
# variant that DOES reach the inner branch (`...that reads ` + the same
|
||||
# padding) stays linear too — 0.026 / 0.029 / 0.060 / 0.129s over four
|
||||
# doublings, exponent ~1.0. The nesting the comment names is one lazy run
|
||||
# inside an OPTIONAL group, never inside a repeated one, so there is no
|
||||
# per-start rescan for the payload to pay for.
|
||||
#
|
||||
# Reviving it needs a payload shape that makes the seed form actually blow
|
||||
# up; two shapes were tried and neither did. Until then this row proves the
|
||||
# scanner runs, not that the port is bounded. Deliberately NOT redesigned
|
||||
# here: choosing a new carrier is the same call the operator reserved for the
|
||||
# `test_pathological_input_returns_within_a_bound` row.
|
||||
evil = "spawn an agent that " + ("word " * 8000)
|
||||
start = time.monotonic()
|
||||
r = scan_lexicon(evil)
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 2.0
|
||||
assert isinstance(r, Report)
|
||||
assert scan_seconds(scan_lexicon, evil) < 2.0
|
||||
assert isinstance(scan_lexicon(evil), Report)
|
||||
|
||||
|
||||
# --- crafted ReDoS payloads against the JSON pattern table (OWASP LLM10) -----
|
||||
|
|
@ -293,6 +315,4 @@ _LEXICON_REDOS_ROWS = [
|
|||
)
|
||||
def test_crafted_redos_payload_stays_bounded_in_the_lexicon(unit, n, bound):
|
||||
payload = (unit * (n // len(unit) + 1))[:n]
|
||||
start = time.monotonic()
|
||||
scan_lexicon(payload)
|
||||
assert time.monotonic() - start < bound
|
||||
assert scan_seconds(scan_lexicon, payload) < bound
|
||||
|
|
|
|||
|
|
@ -11,12 +11,11 @@ empty report; only active-content constructs are ever rewritten. Mutation lives
|
|||
here, kept separate from the report-only output gate (design principles 3 & 4).
|
||||
The transform is pure ``text -> (defanged_text, report)`` — no I/O, no globals.
|
||||
"""
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_guard.neutralize import neutralize
|
||||
from llm_ingestion_guard.report import Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
def test_clean_output_is_byte_identical():
|
||||
|
|
@ -181,10 +180,12 @@ _ATTR_REDOS_N = 100_000
|
|||
|
||||
|
||||
def test_crafted_long_attribute_tag_stays_bounded():
|
||||
# Carrier stays `<a `, unlike the scanner-side twin in test_active_content.py:
|
||||
# the mutator keeps the whole tag set via `is_defangable_tag`, so 0.7.0's
|
||||
# no-URL narrowing did not make `<a >` inert here. Verified by measurement,
|
||||
# not by symmetry — see the comment on that row for what killed it there.
|
||||
payload = "<a " + "A" * _ATTR_REDOS_N + ">"
|
||||
start = time.monotonic()
|
||||
neutralize(payload)
|
||||
assert time.monotonic() - start < 2.0
|
||||
assert scan_seconds(neutralize, payload) < 2.0
|
||||
|
||||
|
||||
def test_url_defanging_inside_a_tag_survives_the_redos_fix():
|
||||
|
|
|
|||
|
|
@ -17,8 +17,6 @@ OKF spec facts used here (verified against okf/SPEC.md, 2026-07-06):
|
|||
"""
|
||||
import pytest
|
||||
|
||||
import time
|
||||
|
||||
from llm_ingestion_guard.okf import (
|
||||
parse_frontmatter,
|
||||
scan_concept,
|
||||
|
|
@ -41,6 +39,7 @@ from llm_ingestion_guard.okf import (
|
|||
from llm_ingestion_guard.report import Report
|
||||
from llm_ingestion_guard.disposition import Trust, Disposition, PRESET_USER_UPLOAD
|
||||
from llm_ingestion_guard import screen_output
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
# --- happy path: split + parse the minimal flat subset -----------------------
|
||||
|
|
@ -676,9 +675,7 @@ _LINK_REDOS_N = 100_000
|
|||
|
||||
|
||||
def test_crafted_link_payload_stays_bounded():
|
||||
start = time.monotonic()
|
||||
extract_link_targets("[" * _LINK_REDOS_N)
|
||||
assert time.monotonic() - start < 2.0
|
||||
assert scan_seconds(extract_link_targets, "[" * _LINK_REDOS_N) < 2.0
|
||||
|
||||
|
||||
# The destination run behind the label gets no row: `[^)\s]+` needs only one
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ from llm_ingestion_guard.active_content import scan_active_content
|
|||
from llm_ingestion_guard.lexicon import scan_lexicon
|
||||
from llm_ingestion_guard.output import scan_output, scan_secret_egress
|
||||
from llm_ingestion_guard.report import Report, Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
# --- fixtures assembled at runtime (never contiguous in source) --------------
|
||||
|
|
@ -352,22 +353,11 @@ def test_pathological_input_returns_within_a_bound():
|
|||
|
||||
# --- crafted ReDoS payloads against OUR OWN patterns (OWASP LLM10) -----------
|
||||
#
|
||||
# Every bound below goes through `_scan_seconds`, so the rows share ONE clock
|
||||
# and one derivation. The neighbouring test above keeps its own wall clock on
|
||||
# purpose -- see the instrument test for why the two must not be merged.
|
||||
|
||||
|
||||
def _scan_seconds(scanner, payload) -> float:
|
||||
"""CPU seconds a scan cost -- the clock the ReDoS bounds are derived against.
|
||||
|
||||
Process CPU time, not wall clock: a ReDoS blowup is spent cycles, and a
|
||||
loaded machine steals wall clock without adding any. Pinned by
|
||||
``test_the_redos_clock_ignores_time_this_process_did_not_spend``, which
|
||||
carries the measurements and what this clock gives up.
|
||||
"""
|
||||
start = time.process_time()
|
||||
scanner(payload)
|
||||
return time.process_time() - start
|
||||
# Every bound below goes through `scan_seconds`, so the rows share ONE clock and
|
||||
# one derivation -- and since `redos_clock` is imported, not copied, that "one"
|
||||
# now spans every ReDoS bound in the suite, not just this file's. The
|
||||
# neighbouring test above keeps its own wall clock on purpose -- see the
|
||||
# instrument test for why the two must not be merged.
|
||||
|
||||
|
||||
# The gap the test above explicitly does NOT cover. Every pattern here has the
|
||||
|
|
@ -441,7 +431,7 @@ _REDOS_PAYLOADS = [
|
|||
)
|
||||
def test_crafted_redos_payload_stays_bounded(scanner, unit, n):
|
||||
payload = (unit * (n // len(unit) + 1))[:n]
|
||||
assert _scan_seconds(scanner, payload) < 2.0
|
||||
assert scan_seconds(scanner, payload) < 2.0
|
||||
|
||||
|
||||
def test_the_redos_clock_ignores_time_this_process_did_not_spend():
|
||||
|
|
@ -475,7 +465,7 @@ def test_the_redos_clock_ignores_time_this_process_did_not_spend():
|
|||
#
|
||||
# A sleep is the defect class at its purest: wall-clock seconds this process
|
||||
# did not spend. 0.4s is 4x the assertion, so this cannot pass by timing luck.
|
||||
assert _scan_seconds(lambda _: time.sleep(0.4), "") < 0.1
|
||||
assert scan_seconds(lambda _: time.sleep(0.4), "") < 0.1
|
||||
|
||||
|
||||
def test_crafted_redos_payload_bounded_through_the_public_gate():
|
||||
|
|
@ -484,7 +474,7 @@ def test_crafted_redos_payload_bounded_through_the_public_gate():
|
|||
# invokes is bounded too -- with the worst measured payload (`<a:`, 660x the
|
||||
# slowest legitimate content of the same size).
|
||||
payload = ("<a:" * (_REDOS_N // 3 + 1))[:_REDOS_N]
|
||||
assert _scan_seconds(scan_output, payload) < 2.0
|
||||
assert scan_seconds(scan_output, payload) < 2.0
|
||||
|
||||
|
||||
def test_gate_is_bounded_on_the_payload_the_first_sweep_missed():
|
||||
|
|
@ -496,7 +486,7 @@ def test_gate_is_bounded_on_the_payload_the_first_sweep_missed():
|
|||
# test_lexicon.py::test_crafted_redos_payload_stays_bounded_in_the_lexicon;
|
||||
# this row exists so the composed gate a caller actually invokes is covered.
|
||||
payload = "[" * _REDOS_N
|
||||
assert _scan_seconds(scan_output, payload) < 2.0
|
||||
assert scan_seconds(scan_output, payload) < 2.0
|
||||
|
||||
|
||||
def test_gate_is_bounded_on_the_long_attribute_arm():
|
||||
|
|
@ -525,7 +515,7 @@ def test_gate_is_bounded_on_the_long_attribute_arm():
|
|||
# attributes at all, so no future URL-shaped narrowing can make it inert the
|
||||
# way it just did to `<a >`.
|
||||
payload = "<script " + "A" * _REDOS_N + ">"
|
||||
assert _scan_seconds(scan_output, payload) < 2.0
|
||||
assert scan_seconds(scan_output, payload) < 2.0
|
||||
|
||||
|
||||
# --- ZWJ inside emoji sequences on the output gate ---------------------------
|
||||
|
|
|
|||
|
|
@ -4,10 +4,9 @@ Core invariants (BRIEF §9): clean input returns byte-identical with an all-zero
|
|||
report; the sanitizer only ever *removes* — its output is always a subsequence
|
||||
of the input.
|
||||
"""
|
||||
import time
|
||||
|
||||
from llm_ingestion_guard.sanitize import sanitize
|
||||
from llm_ingestion_guard.report import Severity, Source
|
||||
from redos_clock import scan_seconds
|
||||
|
||||
|
||||
def _is_subsequence(sub: str, full: str) -> bool:
|
||||
|
|
@ -92,19 +91,21 @@ _REDOS_N = 100_000
|
|||
|
||||
def test_crafted_comment_payload_stays_bounded():
|
||||
payload = ("<!--" * (_REDOS_N // 4 + 1))[:_REDOS_N]
|
||||
start = time.monotonic()
|
||||
sanitize(payload)
|
||||
assert time.monotonic() - start < 2.0
|
||||
assert scan_seconds(sanitize, payload) < 2.0
|
||||
|
||||
|
||||
def test_legitimate_comment_heavy_document_is_far_under_the_bound():
|
||||
# The bound above only has signal if ordinary comment-dense content is
|
||||
# nowhere near it: this is the same size, 100% closed comments.
|
||||
#
|
||||
# This row is the legitimate SIDE of that separation, not a second pin on the
|
||||
# defect: patching the lazy `<!--.*?-->` form back in leaves it green (0.016s),
|
||||
# because closed comments never make the required literal go missing. It is
|
||||
# the tighter of the two bounds and so the more load-sensitive, which is why
|
||||
# it moves to the CPU clock along with its neighbour.
|
||||
unit = "<!-- a note -->"
|
||||
payload = (unit * (_REDOS_N // len(unit) + 1))[:_REDOS_N]
|
||||
start = time.monotonic()
|
||||
sanitize(payload)
|
||||
assert time.monotonic() - start < 0.5
|
||||
assert scan_seconds(sanitize, payload) < 0.5
|
||||
|
||||
|
||||
def test_comment_stripping_survives_the_redos_fix():
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue