1
0
Fork 0

fix(output): 19 quadratic regex runs on the output path, worst ~5.7h at the cap

The output gate claimed LLM10 self-safety on the grounds that its patterns have
no nested quantifiers. True, and irrelevant: nesting is not what makes these
blow up. A run in front of a REQUIRED literal, reachable from a short anchor, is
enough -- crafted input repeats the anchor and never supplies the literal, so
every start position rescans the tail. Quadratic, not exponential, and the
max_scan_chars cap does not help: it bounds the input, and quadratic work on a
bounded input is still hours.

Measured, not argued. `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone; the
composed gate on that payload took 458.7s, extrapolating to ~5.7 hours at the
1_000_000-char input the gate itself accepts. Size-matched ordinary prose runs
0.31s, so the separation is 18x-660x -- unlike the blob in the neighbouring
test, which is the *faster* side of prose and never exercised backtracking.

Two fixes, chosen per pattern rather than uniformly:

- active_content + lexicon JSON (15 runs): exclude the character that opens the
  pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot reach
  past the next start position and the per-start costs telescope. Verified to
  cost no recall: long URLs, long alt text, and `<` inside a quoted attribute
  all still match. Bounding instead would have been linear too but wrong here --
  the content is attacker-controlled, so padding past a bound would be a
  one-line bypass of the EchoLeak class this table exists to catch.
- connstr egress (4 runs): bound the password at MAX_CONNSTR_VALUE. The
  exclusion fix is unavailable -- the anchor character is `/` and passwords
  containing `/` are the common case (measured: they match today). The residual
  miss is a credential over 256 chars; a token that long is still caught by
  egress:jwt-token.

hybrid-xss:script-tag had neither option: its run is the script BODY, which may
legitimately contain `<`. It now matches the opening tag and drops the
`</script>` requirement. That also closes a fail-open -- `<script>alert(1)`
unclosed was silently missed -- at the cost of flagging prose that merely
mentions `<script>`, now documented.

Found by the composed-gate test staying red after every individual scanner was
already linear: the lexicon's six html-obfuscation patterns were the remaining
813x. A per-scanner test alone would have shipped that.

662 passed (was 642), and faster than before the fix.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-31 18:31:58 +02:00
commit cff043787d
10 changed files with 220 additions and 30 deletions

View file

@ -104,31 +104,52 @@ def redact(s: str, show_start: int = 16, show_end: int = 6) -> str:
# --- active-content constructs (the shared pattern table) ---------------------
#
# ReDoS note (OWASP LLM10) — every run below excludes the character that OPENS
# this pattern's own anchor: `[` for the markdown forms, `<` for the autolink and
# the raw tag. That exclusion is what keeps the table linear, and it is not
# cosmetic. Each of these is a run followed by a REQUIRED literal (`]`, `)`,
# `>`); if the run may cross the next anchor, then crafted input that repeats the
# anchor and never supplies the literal makes every start position rescan the
# whole tail — quadratic time, no nested quantifier needed. Measured before the
# exclusions: `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone and ~5.7 HOURS
# extrapolated to the 1_000_000-char cap the gate accepts. With the exclusion a
# run cannot reach past the next anchor, so the per-start costs telescope.
# Bounding the runs instead ({0,256}) would also be linear but is the WRONG fix
# here: the content is attacker-controlled, so padding past the bound would be a
# one-line detection bypass of the very EchoLeak class this table exists to
# catch. See tests/test_output.py::test_crafted_redos_payload_stays_bounded.
#
# Markdown image / inline link: `[text](url "title")`. `url` stops at the first
# `)` or whitespace (balanced-paren URLs matched conservatively — see the
# neutralize scope note).
# neutralize scope note). `[` is excluded per the ReDoS note above; a URL that
# needs a literal `[` (an IPv6 host literal) must percent-encode it anyway.
MD_IMAGE_RE = re.compile(
r"!\[(?P<alt>[^\]]*)\]\(\s*(?P<url>[^)\s]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
r"!\[(?P<alt>[^\]\[]*)\]\(\s*(?P<url>[^)\s\[]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
)
MD_LINK_RE = re.compile(
r"(?<!!)\[(?P<text>[^\]]*)\]\(\s*(?P<url>[^)\s]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
r"(?<!!)\[(?P<text>[^\]\[]*)\]\(\s*(?P<url>[^)\s\[]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
)
# Reference-style link definition: `[label]: destination`. Only fires when the
# destination is absolute (has a scheme or is protocol-relative) — a footnote
# `[1]: some plain text` is not a link target and is left alone.
MD_REFDEF_RE = re.compile(
r"(?m)^(?P<pre>[ ]{0,3}\[[^\]]+\]:\s*)(?P<url>[A-Za-z][\w+.\-]*:\S+|//\S+)"
r"(?m)^(?P<pre>[ ]{0,3}\[[^\]\[]+\]:\s*)(?P<url>[A-Za-z][\w+.\-]*:\S+|//\S+)"
)
# Angle-bracket autolink: `<scheme:...>`.
AUTOLINK_RE = re.compile(r"<(?P<url>[A-Za-z][A-Za-z0-9+.\-]*:[^>\s]+)>")
# Angle-bracket autolink: `<scheme:...>`. A URL inside `<...>` cannot contain a
# raw `<`, so excluding it costs no recall (verified) and bounds the run.
AUTOLINK_RE = re.compile(r"<(?P<url>[A-Za-z][A-Za-z0-9+.\-]*:[^>\s<]+)>")
# Standalone `data:` URI in prose (not preceded by a letter/digit -> "metadata:"
# is not a match), consuming to the next whitespace / quote / bracket.
DATA_URI_RE = re.compile(r"(?<![A-Za-z0-9])data:[^\s'\"<>)]+", re.IGNORECASE)
# Raw HTML tag. Attribute values may hold `>` inside quotes, so quoted runs are
# consumed atomically. A tag is *active* if it is an inherently-executing element,
# carries an event handler, or carries a URL-bearing attribute.
HTML_TAG_RE = re.compile(r"<(?P<slash>/?)(?P<name>[A-Za-z][A-Za-z0-9:-]*)(?P<attrs>(?:[^>\"']|\"[^\"]*\"|'[^']*')*)>")
# carries an event handler, or carries a URL-bearing attribute. The unquoted-char
# branch excludes `<` per the ReDoS note above — a raw `<` cannot appear in an
# unquoted attribute region anyway, and a `<` inside a QUOTED value is still
# consumed by the quoted branches, so this costs no recall (verified).
HTML_TAG_RE = re.compile(r"<(?P<slash>/?)(?P<name>[A-Za-z][A-Za-z0-9:-]*)(?P<attrs>(?:[^>\"'<]|\"[^\"]*\"|'[^']*')*)>")
_EVENT_ATTR_RE = re.compile(r"\bon[a-z]+\s*=", re.IGNORECASE)
_URL_ATTR_RE = re.compile(
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)\s*=",

View file

@ -40,10 +40,25 @@ ENTROPY_HEX_FLOOR_LEN = 64
# --- lexicon: self-safety + variant thresholds ------------------------------
# Input-size cap (OWASP LLM10): large enough for a real ingested document;
# beyond it the scanner reads the prefix and flags, so runtime stays bounded
# even on a decompression-bomb-sized input.
# beyond it the scanner reads the prefix and flags, so every sub-scanner sees a
# bounded input. Note what this cap does NOT buy: bounded input is only bounded
# runtime if the patterns are linear in it. A quadratic pattern turns this cap
# into hours of work, which is what crafted input against the output path was
# measured to do before the ReDoS fix (see active_content's pattern-table note).
MAX_SCAN_CHARS = 1_000_000
# --- output: secret-egress self-safety --------------------------------------
# Longest password a connection-string pattern will match. A bound is required
# (not merely nice) because the run sits in front of a mandatory `@`: unbounded,
# crafted input repeating `redis://:` and never supplying the `@` makes every
# start position rescan the tail — quadratic. Excluding the anchor character the
# way the active-content table does is not available here, since that character
# is `/` and passwords containing `/` are the common case.
# 256 is generous for a password and cheap to scan; the residual miss is a
# credential longer than this, which for the realistic case (a token used as a
# DB password) is still caught by the jwt-token / high-specificity patterns.
MAX_CONNSTR_VALUE = 256
# Minimum length before the rot13 variant is scanned — shorter strings hit
# rot13-look-alike false positives.
ROT13_MIN_LEN = 40

View file

@ -236,7 +236,7 @@
},
{
"id": "html-obfuscation:display-none",
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*display\\s*:\\s*none[^\"]*\"[^>]*>",
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*display\\s*:\\s*none[^\"]*\"[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -244,7 +244,7 @@
},
{
"id": "html-obfuscation:visibility-hidden",
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*visibility\\s*:\\s*hidden[^\"]*\"[^>]*>",
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*visibility\\s*:\\s*hidden[^\"]*\"[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -252,7 +252,7 @@
},
{
"id": "html-obfuscation:offscreen",
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*position\\s*:\\s*absolute[^\"]*-\\d{3,}px[^\"]*\"[^>]*>",
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*position\\s*:\\s*absolute[^\"]*-\\d{3,}px[^\"]*\"[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -260,7 +260,7 @@
},
{
"id": "html-obfuscation:zero-font",
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*font-size\\s*:\\s*0[^\"]*\"[^>]*>",
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*font-size\\s*:\\s*0[^\"]*\"[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -268,7 +268,7 @@
},
{
"id": "html-obfuscation:zero-opacity",
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*opacity\\s*:\\s*0[^\"]*\"[^>]*>",
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*opacity\\s*:\\s*0[^\"]*\"[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -276,7 +276,7 @@
},
{
"id": "html-obfuscation:zero-size-overflow",
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*(?:height|width)\\s*:\\s*0[^\"]*overflow\\s*:\\s*hidden[^\"]*\"[^>]*>",
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*(?:height|width)\\s*:\\s*0[^\"]*overflow\\s*:\\s*hidden[^\"]*\"[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -460,7 +460,7 @@
},
{
"id": "hybrid-xss:script-tag",
"regex": "<script\\b[^>]*>[\\s\\S]*?</script>",
"regex": "<script\\b[^><]*>",
"flags": "i",
"severity": "high",
"owasp": "LLM01",
@ -484,7 +484,7 @@
},
{
"id": "hybrid-xss:iframe-src",
"regex": "<iframe\\b[^>]*src\\s*=\\s*[\"\\'][^\"\\']*(?:javascript:|data:text/html)",
"regex": "<iframe\\b[^><]*src\\s*=\\s*[\"\\'][^\"\\']*(?:javascript:|data:text/html)",
"flags": "i",
"severity": "high",
"owasp": "LLM01",

View file

@ -46,8 +46,19 @@ length. The report is meant to be logged; it must not become the leak.
**Self-safety (OWASP LLM10).** The output is capped once to ``max_scan_chars``
and a single ``output:oversize-input`` finding is emitted if it was truncated;
every sub-scanner then sees bounded input. The egress patterns are linear
(anchored prefixes / negated character classes — no nested quantifiers).
every sub-scanner then sees bounded input.
Bounded input is not by itself bounded runtime, and this module used to claim it
was. The egress patterns have no nested quantifiers — that part was true — but
absence of nesting does not imply linearity. A run in front of a *required*
literal (here: the password run before ``@``) makes every start position rescan
the tail when the literal never arrives, which is quadratic in the scanned
length. Crafted input repeating ``redis://:`` measured 8.2s at 100_000 chars and
extrapolated to hours at the 1_000_000-char cap this gate itself accepts. The
connection-string runs are therefore bounded to
:data:`~llm_ingestion_guard.calibration.MAX_CONNSTR_VALUE`; the same defect in
the active-content table is fixed there by excluding the anchor character. Both
are pinned by ``tests/test_output.py::test_crafted_redos_payload_stays_bounded``.
"""
from __future__ import annotations
@ -56,6 +67,7 @@ from dataclasses import dataclass, replace
from typing import Optional, Union
from .active_content import scan_active_content
from .calibration import MAX_CONNSTR_VALUE
from .entropy import scan_entropy
from .lexicon import MAX_SCAN_CHARS, scan_lexicon
from .report import Finding, Report, Severity, Source
@ -122,19 +134,25 @@ _SECRET_PATTERNS: list[_SecretPattern] = [
_SecretPattern("pkcs8-private-key", _p(r"-{5}BEGIN PRIVATE KEY-{5}"),
Severity.CRITICAL, "PEM PKCS#8 private key header"),
# --- DB connection strings (suppress placeholder passwords) -------------
# The password run is bounded at MAX_CONNSTR_VALUE per the ReDoS note on
# _SECRET_PATTERNS above. Unlike the active-content table, excluding the
# anchor character is NOT available here: the anchor opens with `/`, and a
# password containing `/` is the common case (a base64-ish secret), so
# excluding it would drop real credentials. The bound is the lesser loss.
_SecretPattern("postgres-connstr",
_p(r"postgres(?:ql)?://[^:@\s]+:(?P<val>[^@\s]+)@[^\s'\"]+"),
_p(r"postgres(?:ql)?://[^:@\s]+:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
Severity.CRITICAL, "PostgreSQL connection string with credentials",
value_group="val"),
_SecretPattern("mongodb-connstr",
_p(r"mongodb(?:\+srv)?://[^:@\s]+:(?P<val>[^@\s]+)@[^\s'\"]+"),
_p(r"mongodb(?:\+srv)?://[^:@\s]+:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
Severity.CRITICAL, "MongoDB connection string with credentials",
value_group="val"),
_SecretPattern("mysql-connstr",
_p(r"mysql(?:2)?://[^:@\s]+:(?P<val>[^@\s]+)@[^\s'\"]+"),
_p(r"mysql(?:2)?://[^:@\s]+:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
Severity.CRITICAL, "MySQL/MariaDB connection string with credentials",
value_group="val"),
_SecretPattern("redis-connstr", _p(r"redis://:(?P<val>[^@\s]+)@[^\s'\"]+"),
_SecretPattern("redis-connstr",
_p(r"redis://:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
Severity.HIGH, "Redis connection string with password",
value_group="val"),
# --- JWT (high false-positive rate -> MEDIUM, flag for review) ----------