fix(sanitize,okf,active_content): three quadratic patterns, two on the input path
The generalised sweep found what 0.3.2's hand-written rows missed. All three are
the documented class -- a run in front of a required literal that never arrives,
so every start position rescans the tail -- and all three are worse than the
0.3.3 findings, because `sanitize`, `neutralize`, `scan_active_content` and the
okf link graph apply NO input cap. `scan_lexicon`/`scan_output` are the only
entry points that do, so there is no ceiling to extrapolate to.
sanitize._HTML_COMMENT_RE `<!--`*100_000 20.1s, exponent 1.96-2.14
active_content.URL_IN_TEXT_RE `<a `+`A`*100_000 12.99s / 14.9s, exponent ~2.0
okf._MD_LINK_RE `[`*100_000 7.1s, exponent 1.99-2.05
Each fix is the one the pattern's own shape allows, not a copied choice:
- The comment stripper drops the regex for `str.find`. Excluding `<` would lose
every comment containing markup; bounding the run would be a carrier bypass
of the exact construct the stripper exists to remove.
- `URL_IN_TEXT_RE` bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
Bounding is safe *here* only because it is a defanger inside a tag already
flagged `active:raw-html`. A lookbehind was measured too and rejected: it
drops `-http://evil.com`, a one-character evasion. Bounded: 0.185s at 1M.
- `_MD_LINK_RE` excludes `[`, matching `active_content.MD_LINK_RE` exactly,
including the nested-label trade already documented there.
`sanitize` claimed "no catastrophic backtracking" in a comment; that claim was
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
676 tests (+10), coverage 128/128 + 6/6 gaps, sweep clean across 150 patterns.
The okf destination run gets no row: `[^)\s]+` cannot fail, so a row for it
could never go red.
This commit is contained in:
parent
abbfe5f0fd
commit
73fa1b99ae
9 changed files with 223 additions and 7 deletions
|
|
@ -22,8 +22,24 @@ _ZERO_WIDTH = frozenset({0x200B, 0x200C, 0x200D, 0xFEFF, 0x00AD})
|
|||
_BIDI = frozenset({0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069})
|
||||
_TAG_LO, _TAG_HI = 0xE0000, 0xE007F # Unicode Tags block (U+E0000–U+E007F)
|
||||
|
||||
# Span carriers. Lazy `.*?` + explicit terminator — no catastrophic backtracking.
|
||||
_HTML_COMMENT_RE = re.compile(r"<!--.*?-->", re.DOTALL)
|
||||
# Span carriers.
|
||||
#
|
||||
# ReDoS note (OWASP LLM10). The comment stripper used to be `<!--.*?-->` with a
|
||||
# comment that absence of nesting meant no catastrophic backtracking. That claim
|
||||
# was wrong in the same way `output`'s was: a lazy run in front of a REQUIRED
|
||||
# literal costs a full tail rescan at *every* start position when the literal
|
||||
# never arrives, so `<!--` repeated to 100_000 chars measured 20.1s (exponent
|
||||
# 1.96–2.14 over four doublings) — and this module, unlike `scan_lexicon` /
|
||||
# `scan_output`, applies no input cap, so nothing bounds that above.
|
||||
#
|
||||
# Neither of the two fixes used elsewhere fits here. Excluding the opener (`<`)
|
||||
# from the run would drop every comment containing markup — `<!-- <b>x</b> -->`
|
||||
# is the ordinary case, not an edge one. Bounding the run would be a one-line
|
||||
# carrier bypass: a comment padded past the bound is exactly what this stripper
|
||||
# exists to remove. So the scan is done with `str.find`, which is linear and
|
||||
# semantically identical to the lazy regex — leftmost opener, nearest following
|
||||
# terminator, unterminated trailer left in place.
|
||||
_COMMENT_OPEN, _COMMENT_CLOSE = "<!--", "-->"
|
||||
# `data:` not preceded by a letter (so "metadata:" / "userdata:" do not match),
|
||||
# consuming up to the next whitespace / quote / angle bracket / closing paren.
|
||||
_DATA_URI_RE = re.compile(r"(?<![A-Za-z])data:[^\s'\"<>)]+", re.IGNORECASE)
|
||||
|
|
@ -43,6 +59,32 @@ def _redact(s: str, show_start: int = 12, show_end: int = 4) -> str:
|
|||
return f"{s[:show_start]}...{s[-show_end:]}"
|
||||
|
||||
|
||||
def _strip_html_comments(text: str) -> tuple[str, int]:
|
||||
"""Remove ``<!-- ... -->`` spans; return the cleaned text and the count.
|
||||
|
||||
Linear replacement for the quantifier form (see the ReDoS note above). An
|
||||
unterminated ``<!--`` is left verbatim, matching the regex it replaces.
|
||||
"""
|
||||
if _COMMENT_OPEN not in text:
|
||||
return text, 0
|
||||
out: list[str] = []
|
||||
pos = count = 0
|
||||
while True:
|
||||
start = text.find(_COMMENT_OPEN, pos)
|
||||
if start == -1:
|
||||
break
|
||||
end = text.find(_COMMENT_CLOSE, start + len(_COMMENT_OPEN))
|
||||
if end == -1: # unterminated — not a comment, keep the rest verbatim
|
||||
break
|
||||
out.append(text[pos:start])
|
||||
pos = end + len(_COMMENT_CLOSE)
|
||||
count += 1
|
||||
if not count:
|
||||
return text, 0
|
||||
out.append(text[pos:])
|
||||
return "".join(out), count
|
||||
|
||||
|
||||
def _decode_tags(codepoints: list[int]) -> str:
|
||||
"""Decode Unicode-tag codepoints to their hidden ASCII (cp - 0xE0000)."""
|
||||
out = []
|
||||
|
|
@ -75,7 +117,7 @@ def sanitize(text: str, source: Source = Source.INPUT) -> SanitizeResult:
|
|||
cleaned = "".join(kept) if (zero_width or bidi or tag_cps) else text
|
||||
|
||||
# Span carriers.
|
||||
cleaned, n_comments = _HTML_COMMENT_RE.subn("", cleaned)
|
||||
cleaned, n_comments = _strip_html_comments(cleaned)
|
||||
cleaned, n_data = _DATA_URI_RE.subn("", cleaned)
|
||||
|
||||
if zero_width:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue