1
0
Fork 0

fix(output): 19 quadratic regex runs on the output path, worst ~5.7h at the cap

The output gate claimed LLM10 self-safety on the grounds that its patterns have
no nested quantifiers. True, and irrelevant: nesting is not what makes these
blow up. A run in front of a REQUIRED literal, reachable from a short anchor, is
enough -- crafted input repeats the anchor and never supplies the literal, so
every start position rescans the tail. Quadratic, not exponential, and the
max_scan_chars cap does not help: it bounds the input, and quadratic work on a
bounded input is still hours.

Measured, not argued. `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone; the
composed gate on that payload took 458.7s, extrapolating to ~5.7 hours at the
1_000_000-char input the gate itself accepts. Size-matched ordinary prose runs
0.31s, so the separation is 18x-660x -- unlike the blob in the neighbouring
test, which is the *faster* side of prose and never exercised backtracking.

Two fixes, chosen per pattern rather than uniformly:

- active_content + lexicon JSON (15 runs): exclude the character that opens the
  pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot reach
  past the next start position and the per-start costs telescope. Verified to
  cost no recall: long URLs, long alt text, and `<` inside a quoted attribute
  all still match. Bounding instead would have been linear too but wrong here --
  the content is attacker-controlled, so padding past a bound would be a
  one-line bypass of the EchoLeak class this table exists to catch.
- connstr egress (4 runs): bound the password at MAX_CONNSTR_VALUE. The
  exclusion fix is unavailable -- the anchor character is `/` and passwords
  containing `/` are the common case (measured: they match today). The residual
  miss is a credential over 256 chars; a token that long is still caught by
  egress:jwt-token.

hybrid-xss:script-tag had neither option: its run is the script BODY, which may
legitimately contain `<`. It now matches the opening tag and drops the
`</script>` requirement. That also closes a fail-open -- `<script>alert(1)`
unclosed was silently missed -- at the cost of flagging prose that merely
mentions `<script>`, now documented.

Found by the composed-gate test staying red after every individual scanner was
already linear: the lexicon's six html-obfuscation patterns were the remaining
813x. A per-scanner test alone would have shipped that.

662 passed (was 642), and faster than before the fix.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-31 18:31:58 +02:00
commit cff043787d
10 changed files with 220 additions and 30 deletions

View file

@ -104,31 +104,52 @@ def redact(s: str, show_start: int = 16, show_end: int = 6) -> str:
# --- active-content constructs (the shared pattern table) ---------------------
#
# ReDoS note (OWASP LLM10) — every run below excludes the character that OPENS
# this pattern's own anchor: `[` for the markdown forms, `<` for the autolink and
# the raw tag. That exclusion is what keeps the table linear, and it is not
# cosmetic. Each of these is a run followed by a REQUIRED literal (`]`, `)`,
# `>`); if the run may cross the next anchor, then crafted input that repeats the
# anchor and never supplies the literal makes every start position rescan the
# whole tail — quadratic time, no nested quantifier needed. Measured before the
# exclusions: `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone and ~5.7 HOURS
# extrapolated to the 1_000_000-char cap the gate accepts. With the exclusion a
# run cannot reach past the next anchor, so the per-start costs telescope.
# Bounding the runs instead ({0,256}) would also be linear but is the WRONG fix
# here: the content is attacker-controlled, so padding past the bound would be a
# one-line detection bypass of the very EchoLeak class this table exists to
# catch. See tests/test_output.py::test_crafted_redos_payload_stays_bounded.
#
# Markdown image / inline link: `[text](url "title")`. `url` stops at the first
# `)` or whitespace (balanced-paren URLs matched conservatively — see the
# neutralize scope note).
# neutralize scope note). `[` is excluded per the ReDoS note above; a URL that
# needs a literal `[` (an IPv6 host literal) must percent-encode it anyway.
MD_IMAGE_RE = re.compile(
r"!\[(?P<alt>[^\]]*)\]\(\s*(?P<url>[^)\s]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
r"!\[(?P<alt>[^\]\[]*)\]\(\s*(?P<url>[^)\s\[]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
)
MD_LINK_RE = re.compile(
r"(?<!!)\[(?P<text>[^\]]*)\]\(\s*(?P<url>[^)\s]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
r"(?<!!)\[(?P<text>[^\]\[]*)\]\(\s*(?P<url>[^)\s\[]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
)
# Reference-style link definition: `[label]: destination`. Only fires when the
# destination is absolute (has a scheme or is protocol-relative) — a footnote
# `[1]: some plain text` is not a link target and is left alone.
MD_REFDEF_RE = re.compile(
r"(?m)^(?P<pre>[ ]{0,3}\[[^\]]+\]:\s*)(?P<url>[A-Za-z][\w+.\-]*:\S+|//\S+)"
r"(?m)^(?P<pre>[ ]{0,3}\[[^\]\[]+\]:\s*)(?P<url>[A-Za-z][\w+.\-]*:\S+|//\S+)"
)
# Angle-bracket autolink: `<scheme:...>`.
AUTOLINK_RE = re.compile(r"<(?P<url>[A-Za-z][A-Za-z0-9+.\-]*:[^>\s]+)>")
# Angle-bracket autolink: `<scheme:...>`. A URL inside `<...>` cannot contain a
# raw `<`, so excluding it costs no recall (verified) and bounds the run.
AUTOLINK_RE = re.compile(r"<(?P<url>[A-Za-z][A-Za-z0-9+.\-]*:[^>\s<]+)>")
# Standalone `data:` URI in prose (not preceded by a letter/digit -> "metadata:"
# is not a match), consuming to the next whitespace / quote / bracket.
DATA_URI_RE = re.compile(r"(?<![A-Za-z0-9])data:[^\s'\"<>)]+", re.IGNORECASE)
# Raw HTML tag. Attribute values may hold `>` inside quotes, so quoted runs are
# consumed atomically. A tag is *active* if it is an inherently-executing element,
# carries an event handler, or carries a URL-bearing attribute.
HTML_TAG_RE = re.compile(r"<(?P<slash>/?)(?P<name>[A-Za-z][A-Za-z0-9:-]*)(?P<attrs>(?:[^>\"']|\"[^\"]*\"|'[^']*')*)>")
# carries an event handler, or carries a URL-bearing attribute. The unquoted-char
# branch excludes `<` per the ReDoS note above — a raw `<` cannot appear in an
# unquoted attribute region anyway, and a `<` inside a QUOTED value is still
# consumed by the quoted branches, so this costs no recall (verified).
HTML_TAG_RE = re.compile(r"<(?P<slash>/?)(?P<name>[A-Za-z][A-Za-z0-9:-]*)(?P<attrs>(?:[^>\"'<]|\"[^\"]*\"|'[^']*')*)>")
_EVENT_ATTR_RE = re.compile(r"\bon[a-z]+\s*=", re.IGNORECASE)
_URL_ATTR_RE = re.compile(
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)\s*=",