fix(output): 19 quadratic regex runs on the output path, worst ~5.7h at the cap
The output gate claimed LLM10 self-safety on the grounds that its patterns have no nested quantifiers. True, and irrelevant: nesting is not what makes these blow up. A run in front of a REQUIRED literal, reachable from a short anchor, is enough -- crafted input repeats the anchor and never supplies the literal, so every start position rescans the tail. Quadratic, not exponential, and the max_scan_chars cap does not help: it bounds the input, and quadratic work on a bounded input is still hours. Measured, not argued. `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone; the composed gate on that payload took 458.7s, extrapolating to ~5.7 hours at the 1_000_000-char input the gate itself accepts. Size-matched ordinary prose runs 0.31s, so the separation is 18x-660x -- unlike the blob in the neighbouring test, which is the *faster* side of prose and never exercised backtracking. Two fixes, chosen per pattern rather than uniformly: - active_content + lexicon JSON (15 runs): exclude the character that opens the pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot reach past the next start position and the per-start costs telescope. Verified to cost no recall: long URLs, long alt text, and `<` inside a quoted attribute all still match. Bounding instead would have been linear too but wrong here -- the content is attacker-controlled, so padding past a bound would be a one-line bypass of the EchoLeak class this table exists to catch. - connstr egress (4 runs): bound the password at MAX_CONNSTR_VALUE. The exclusion fix is unavailable -- the anchor character is `/` and passwords containing `/` are the common case (measured: they match today). The residual miss is a credential over 256 chars; a token that long is still caught by egress:jwt-token. hybrid-xss:script-tag had neither option: its run is the script BODY, which may legitimately contain `<`. It now matches the opening tag and drops the `</script>` requirement. That also closes a fail-open -- `<script>alert(1)` unclosed was silently missed -- at the cost of flagging prose that merely mentions `<script>`, now documented. Found by the composed-gate test staying red after every individual scanner was already linear: the lexicon's six html-obfuscation patterns were the remaining 813x. A per-scanner test alone would have shipped that. 662 passed (was 642), and faster than before the fix.
This commit is contained in:
parent
8deca93ee1
commit
cff043787d
10 changed files with 220 additions and 30 deletions
|
|
@ -104,31 +104,52 @@ def redact(s: str, show_start: int = 16, show_end: int = 6) -> str:
|
|||
|
||||
|
||||
# --- active-content constructs (the shared pattern table) ---------------------
|
||||
#
|
||||
# ReDoS note (OWASP LLM10) — every run below excludes the character that OPENS
|
||||
# this pattern's own anchor: `[` for the markdown forms, `<` for the autolink and
|
||||
# the raw tag. That exclusion is what keeps the table linear, and it is not
|
||||
# cosmetic. Each of these is a run followed by a REQUIRED literal (`]`, `)`,
|
||||
# `>`); if the run may cross the next anchor, then crafted input that repeats the
|
||||
# anchor and never supplies the literal makes every start position rescan the
|
||||
# whole tail — quadratic time, no nested quantifier needed. Measured before the
|
||||
# exclusions: `<a:` x 100_000 took 23.4s in AUTOLINK_RE alone and ~5.7 HOURS
|
||||
# extrapolated to the 1_000_000-char cap the gate accepts. With the exclusion a
|
||||
# run cannot reach past the next anchor, so the per-start costs telescope.
|
||||
# Bounding the runs instead ({0,256}) would also be linear but is the WRONG fix
|
||||
# here: the content is attacker-controlled, so padding past the bound would be a
|
||||
# one-line detection bypass of the very EchoLeak class this table exists to
|
||||
# catch. See tests/test_output.py::test_crafted_redos_payload_stays_bounded.
|
||||
#
|
||||
# Markdown image / inline link: `[text](url "title")`. `url` stops at the first
|
||||
# `)` or whitespace (balanced-paren URLs matched conservatively — see the
|
||||
# neutralize scope note).
|
||||
# neutralize scope note). `[` is excluded per the ReDoS note above; a URL that
|
||||
# needs a literal `[` (an IPv6 host literal) must percent-encode it anyway.
|
||||
MD_IMAGE_RE = re.compile(
|
||||
r"!\[(?P<alt>[^\]]*)\]\(\s*(?P<url>[^)\s]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
|
||||
r"!\[(?P<alt>[^\]\[]*)\]\(\s*(?P<url>[^)\s\[]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
|
||||
)
|
||||
MD_LINK_RE = re.compile(
|
||||
r"(?<!!)\[(?P<text>[^\]]*)\]\(\s*(?P<url>[^)\s]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
|
||||
r"(?<!!)\[(?P<text>[^\]\[]*)\]\(\s*(?P<url>[^)\s\[]+)(?P<title>(?:\s+\"[^\"]*\")?)\s*\)"
|
||||
)
|
||||
# Reference-style link definition: `[label]: destination`. Only fires when the
|
||||
# destination is absolute (has a scheme or is protocol-relative) — a footnote
|
||||
# `[1]: some plain text` is not a link target and is left alone.
|
||||
MD_REFDEF_RE = re.compile(
|
||||
r"(?m)^(?P<pre>[ ]{0,3}\[[^\]]+\]:\s*)(?P<url>[A-Za-z][\w+.\-]*:\S+|//\S+)"
|
||||
r"(?m)^(?P<pre>[ ]{0,3}\[[^\]\[]+\]:\s*)(?P<url>[A-Za-z][\w+.\-]*:\S+|//\S+)"
|
||||
)
|
||||
# Angle-bracket autolink: `<scheme:...>`.
|
||||
AUTOLINK_RE = re.compile(r"<(?P<url>[A-Za-z][A-Za-z0-9+.\-]*:[^>\s]+)>")
|
||||
# Angle-bracket autolink: `<scheme:...>`. A URL inside `<...>` cannot contain a
|
||||
# raw `<`, so excluding it costs no recall (verified) and bounds the run.
|
||||
AUTOLINK_RE = re.compile(r"<(?P<url>[A-Za-z][A-Za-z0-9+.\-]*:[^>\s<]+)>")
|
||||
# Standalone `data:` URI in prose (not preceded by a letter/digit -> "metadata:"
|
||||
# is not a match), consuming to the next whitespace / quote / bracket.
|
||||
DATA_URI_RE = re.compile(r"(?<![A-Za-z0-9])data:[^\s'\"<>)]+", re.IGNORECASE)
|
||||
|
||||
# Raw HTML tag. Attribute values may hold `>` inside quotes, so quoted runs are
|
||||
# consumed atomically. A tag is *active* if it is an inherently-executing element,
|
||||
# carries an event handler, or carries a URL-bearing attribute.
|
||||
HTML_TAG_RE = re.compile(r"<(?P<slash>/?)(?P<name>[A-Za-z][A-Za-z0-9:-]*)(?P<attrs>(?:[^>\"']|\"[^\"]*\"|'[^']*')*)>")
|
||||
# carries an event handler, or carries a URL-bearing attribute. The unquoted-char
|
||||
# branch excludes `<` per the ReDoS note above — a raw `<` cannot appear in an
|
||||
# unquoted attribute region anyway, and a `<` inside a QUOTED value is still
|
||||
# consumed by the quoted branches, so this costs no recall (verified).
|
||||
HTML_TAG_RE = re.compile(r"<(?P<slash>/?)(?P<name>[A-Za-z][A-Za-z0-9:-]*)(?P<attrs>(?:[^>\"'<]|\"[^\"]*\"|'[^']*')*)>")
|
||||
_EVENT_ATTR_RE = re.compile(r"\bon[a-z]+\s*=", re.IGNORECASE)
|
||||
_URL_ATTR_RE = re.compile(
|
||||
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)\s*=",
|
||||
|
|
|
|||
|
|
@ -40,10 +40,25 @@ ENTROPY_HEX_FLOOR_LEN = 64
|
|||
|
||||
# --- lexicon: self-safety + variant thresholds ------------------------------
|
||||
# Input-size cap (OWASP LLM10): large enough for a real ingested document;
|
||||
# beyond it the scanner reads the prefix and flags, so runtime stays bounded
|
||||
# even on a decompression-bomb-sized input.
|
||||
# beyond it the scanner reads the prefix and flags, so every sub-scanner sees a
|
||||
# bounded input. Note what this cap does NOT buy: bounded input is only bounded
|
||||
# runtime if the patterns are linear in it. A quadratic pattern turns this cap
|
||||
# into hours of work, which is what crafted input against the output path was
|
||||
# measured to do before the ReDoS fix (see active_content's pattern-table note).
|
||||
MAX_SCAN_CHARS = 1_000_000
|
||||
|
||||
# --- output: secret-egress self-safety --------------------------------------
|
||||
# Longest password a connection-string pattern will match. A bound is required
|
||||
# (not merely nice) because the run sits in front of a mandatory `@`: unbounded,
|
||||
# crafted input repeating `redis://:` and never supplying the `@` makes every
|
||||
# start position rescan the tail — quadratic. Excluding the anchor character the
|
||||
# way the active-content table does is not available here, since that character
|
||||
# is `/` and passwords containing `/` are the common case.
|
||||
# 256 is generous for a password and cheap to scan; the residual miss is a
|
||||
# credential longer than this, which for the realistic case (a token used as a
|
||||
# DB password) is still caught by the jwt-token / high-specificity patterns.
|
||||
MAX_CONNSTR_VALUE = 256
|
||||
|
||||
# Minimum length before the rot13 variant is scanned — shorter strings hit
|
||||
# rot13-look-alike false positives.
|
||||
ROT13_MIN_LEN = 40
|
||||
|
|
|
|||
|
|
@ -236,7 +236,7 @@
|
|||
},
|
||||
{
|
||||
"id": "html-obfuscation:display-none",
|
||||
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*display\\s*:\\s*none[^\"]*\"[^>]*>",
|
||||
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*display\\s*:\\s*none[^\"]*\"[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -244,7 +244,7 @@
|
|||
},
|
||||
{
|
||||
"id": "html-obfuscation:visibility-hidden",
|
||||
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*visibility\\s*:\\s*hidden[^\"]*\"[^>]*>",
|
||||
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*visibility\\s*:\\s*hidden[^\"]*\"[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -252,7 +252,7 @@
|
|||
},
|
||||
{
|
||||
"id": "html-obfuscation:offscreen",
|
||||
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*position\\s*:\\s*absolute[^\"]*-\\d{3,}px[^\"]*\"[^>]*>",
|
||||
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*position\\s*:\\s*absolute[^\"]*-\\d{3,}px[^\"]*\"[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -260,7 +260,7 @@
|
|||
},
|
||||
{
|
||||
"id": "html-obfuscation:zero-font",
|
||||
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*font-size\\s*:\\s*0[^\"]*\"[^>]*>",
|
||||
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*font-size\\s*:\\s*0[^\"]*\"[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -268,7 +268,7 @@
|
|||
},
|
||||
{
|
||||
"id": "html-obfuscation:zero-opacity",
|
||||
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*opacity\\s*:\\s*0[^\"]*\"[^>]*>",
|
||||
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*opacity\\s*:\\s*0[^\"]*\"[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -276,7 +276,7 @@
|
|||
},
|
||||
{
|
||||
"id": "html-obfuscation:zero-size-overflow",
|
||||
"regex": "<[^>]+style\\s*=\\s*\"[^\"]*(?:height|width)\\s*:\\s*0[^\"]*overflow\\s*:\\s*hidden[^\"]*\"[^>]*>",
|
||||
"regex": "<[^><]+style\\s*=\\s*\"[^\"]*(?:height|width)\\s*:\\s*0[^\"]*overflow\\s*:\\s*hidden[^\"]*\"[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -460,7 +460,7 @@
|
|||
},
|
||||
{
|
||||
"id": "hybrid-xss:script-tag",
|
||||
"regex": "<script\\b[^>]*>[\\s\\S]*?</script>",
|
||||
"regex": "<script\\b[^><]*>",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
@ -484,7 +484,7 @@
|
|||
},
|
||||
{
|
||||
"id": "hybrid-xss:iframe-src",
|
||||
"regex": "<iframe\\b[^>]*src\\s*=\\s*[\"\\'][^\"\\']*(?:javascript:|data:text/html)",
|
||||
"regex": "<iframe\\b[^><]*src\\s*=\\s*[\"\\'][^\"\\']*(?:javascript:|data:text/html)",
|
||||
"flags": "i",
|
||||
"severity": "high",
|
||||
"owasp": "LLM01",
|
||||
|
|
|
|||
|
|
@ -46,8 +46,19 @@ length. The report is meant to be logged; it must not become the leak.
|
|||
|
||||
**Self-safety (OWASP LLM10).** The output is capped once to ``max_scan_chars``
|
||||
and a single ``output:oversize-input`` finding is emitted if it was truncated;
|
||||
every sub-scanner then sees bounded input. The egress patterns are linear
|
||||
(anchored prefixes / negated character classes — no nested quantifiers).
|
||||
every sub-scanner then sees bounded input.
|
||||
|
||||
Bounded input is not by itself bounded runtime, and this module used to claim it
|
||||
was. The egress patterns have no nested quantifiers — that part was true — but
|
||||
absence of nesting does not imply linearity. A run in front of a *required*
|
||||
literal (here: the password run before ``@``) makes every start position rescan
|
||||
the tail when the literal never arrives, which is quadratic in the scanned
|
||||
length. Crafted input repeating ``redis://:`` measured 8.2s at 100_000 chars and
|
||||
extrapolated to hours at the 1_000_000-char cap this gate itself accepts. The
|
||||
connection-string runs are therefore bounded to
|
||||
:data:`~llm_ingestion_guard.calibration.MAX_CONNSTR_VALUE`; the same defect in
|
||||
the active-content table is fixed there by excluding the anchor character. Both
|
||||
are pinned by ``tests/test_output.py::test_crafted_redos_payload_stays_bounded``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -56,6 +67,7 @@ from dataclasses import dataclass, replace
|
|||
from typing import Optional, Union
|
||||
|
||||
from .active_content import scan_active_content
|
||||
from .calibration import MAX_CONNSTR_VALUE
|
||||
from .entropy import scan_entropy
|
||||
from .lexicon import MAX_SCAN_CHARS, scan_lexicon
|
||||
from .report import Finding, Report, Severity, Source
|
||||
|
|
@ -122,19 +134,25 @@ _SECRET_PATTERNS: list[_SecretPattern] = [
|
|||
_SecretPattern("pkcs8-private-key", _p(r"-{5}BEGIN PRIVATE KEY-{5}"),
|
||||
Severity.CRITICAL, "PEM PKCS#8 private key header"),
|
||||
# --- DB connection strings (suppress placeholder passwords) -------------
|
||||
# The password run is bounded at MAX_CONNSTR_VALUE per the ReDoS note on
|
||||
# _SECRET_PATTERNS above. Unlike the active-content table, excluding the
|
||||
# anchor character is NOT available here: the anchor opens with `/`, and a
|
||||
# password containing `/` is the common case (a base64-ish secret), so
|
||||
# excluding it would drop real credentials. The bound is the lesser loss.
|
||||
_SecretPattern("postgres-connstr",
|
||||
_p(r"postgres(?:ql)?://[^:@\s]+:(?P<val>[^@\s]+)@[^\s'\"]+"),
|
||||
_p(r"postgres(?:ql)?://[^:@\s]+:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
|
||||
Severity.CRITICAL, "PostgreSQL connection string with credentials",
|
||||
value_group="val"),
|
||||
_SecretPattern("mongodb-connstr",
|
||||
_p(r"mongodb(?:\+srv)?://[^:@\s]+:(?P<val>[^@\s]+)@[^\s'\"]+"),
|
||||
_p(r"mongodb(?:\+srv)?://[^:@\s]+:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
|
||||
Severity.CRITICAL, "MongoDB connection string with credentials",
|
||||
value_group="val"),
|
||||
_SecretPattern("mysql-connstr",
|
||||
_p(r"mysql(?:2)?://[^:@\s]+:(?P<val>[^@\s]+)@[^\s'\"]+"),
|
||||
_p(r"mysql(?:2)?://[^:@\s]+:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
|
||||
Severity.CRITICAL, "MySQL/MariaDB connection string with credentials",
|
||||
value_group="val"),
|
||||
_SecretPattern("redis-connstr", _p(r"redis://:(?P<val>[^@\s]+)@[^\s'\"]+"),
|
||||
_SecretPattern("redis-connstr",
|
||||
_p(r"redis://:(?P<val>[^@\s]{1,%d})@[^\s'\"]+" % MAX_CONNSTR_VALUE),
|
||||
Severity.HIGH, "Redis connection string with password",
|
||||
value_group="val"),
|
||||
# --- JWT (high false-positive rate -> MEDIUM, flag for review) ----------
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue