fix(active-content): raw-html graded two inert shapes HIGH, and the fix moved a second surface
`is_active_tag`'s URL-attribute branch was a presence test: any element carrying `href=`/`src=`/`action=` graded HIGH regardless of where the URL pointed. An MDX `<Card href="/en/agent-sdk/quickstart">` reaches no attacker-controlled host, and neither does APIM policy XML's `<set-header>`. It now requires an external target -- the rule the markdown paths have applied since 0.3.1. `<base>` left the active name set in the same change: HTML's `<base>` has its whole affordance in an `href` the attribute branch still catches, and APIM's attribute-less `<base />` is inert. Measured before and after in ONE session against one corpus state, because two of the three corpora are living and a split would mix this with re-harvest drift: reference-corpus 389 docs 133 -> 108 (ceiling 107) vendor-harvest 187 docs 100 -> 98 (ceiling 62) generated-notes 550 docs 90 -> 88 (ceiling 49) 96% of the achievable reduction in reference-corpus, 5% in the wiki corpora. The two classes had to be measured TOGETHER -- alone they free 3 and 13 documents, together 25, because a document carrying one usually carries the other. The second surface: `neutralize` imported `is_active_tag` by name, so this would have silently narrowed the opt-in mutator too -- and no test discriminated the two halves, since every `neutralize:raw-html` payload stays active under any narrowing considered. That test is written first here. The predicates are now separate symbols; the mutator keeps defanging anything, because over-defanging is auditable and blocks nothing while under-defanging hands a human a live construct. Behaviour change: a document whose only finding was one of these classes now WARNs instead of holding. Detection is unchanged -- 128/128 classes, 6/6 gaps hold. Self-safety: reading an attribute VALUE needs a pattern the presence test lacks. It reuses the same literal alternation so no new run shape enters the table; its `_REDOS_PAYLOADS` row denies the `=` the pattern requires, since a unit supplying it matches at once and never exercises the run (the lexicon's `script-tag` row is the cautionary case). 0.031-0.046s across five attack shapes at 100_000 chars against a 2.0s bound; `docs/redos-sweep.py` reports 0 candidates of 152. An attribute the presence test saw but the value parser cannot read counts as external -- fail secure. `docs/rawhtml-census.py` gains a PRODUCTION row that re-measures the shipped predicate rather than a hypothesis, so a published number and the code cannot drift apart unnoticed. README's limitation count moves 34 -> 33. 727 passed (was 717).
This commit is contained in:
parent
e671edb96f
commit
736f370cfb
9 changed files with 287 additions and 72 deletions
|
|
@ -16,13 +16,29 @@ consumers share it:
|
|||
* :func:`~llm_ingestion_guard.neutralize.neutralize` — the separate, opt-in
|
||||
**mutator** that defangs the same constructs for human audit.
|
||||
|
||||
One deliberate asymmetry between the two: the scanner flags markdown images and
|
||||
links only when the URL is absolute or protocol-relative. A relative in-document
|
||||
link has no attacker-reachable endpoint, and flagging it would silently
|
||||
over-block legitimate wiki/OKF content (design principle 5) — cross-linking is
|
||||
those formats' core mechanism. ``neutralize`` keeps its broader defang-anything
|
||||
behavior: it is opt-in, and bracketed dots in a relative path are auditable,
|
||||
not blocking.
|
||||
One deliberate asymmetry between the two: the scanner flags a construct only when
|
||||
its URL is absolute or protocol-relative. A relative in-document link has no
|
||||
attacker-reachable endpoint, and flagging it would silently over-block legitimate
|
||||
wiki/OKF content (design principle 5) — cross-linking is those formats' core
|
||||
mechanism. ``neutralize`` keeps its broader defang-anything behavior: it is
|
||||
opt-in, and bracketed dots in a relative path are auditable, not blocking.
|
||||
|
||||
The two predicates are therefore separate symbols — :func:`is_active_tag` for the
|
||||
scanner, :func:`is_defangable_tag` for the mutator. They were one symbol until
|
||||
0.6.0, imported by name across modules, so narrowing the scanner would have moved
|
||||
the mutator silently.
|
||||
|
||||
**The asymmetry covers raw HTML too** (0.6.0). It previously applied only to the
|
||||
markdown paths: a tag was active if it carried a URL attribute *at all*, so an MDX
|
||||
``<Card href="/en/quickstart">`` — a doc-relative route on a name outside the
|
||||
active set — carried HIGH. It now requires an external target, the rule the
|
||||
markdown paths have applied since 0.3.1. ``<base>`` left the active *name* set in
|
||||
the same change: its whole affordance is its ``href``, which the URL-attribute
|
||||
branch still catches, while the attribute-less ``<base />`` of Azure APIM policy
|
||||
XML has no affordance in any renderer. Measured together rather than one at a time
|
||||
— the classes co-occur — the pair frees 25 of 133 non-WARN documents on the
|
||||
reference corpus and 2 each on the two wiki corpora, at unchanged recall. Method
|
||||
and numbers: ``docs/rawhtml-census.py``; residuals: ``docs/LIMITATIONS.md``.
|
||||
|
||||
**Severity grades on URL shape, not construct type** (0.3.1). The exfiltration
|
||||
primitive is not "an image" — it is a URL that moves bytes to a host the
|
||||
|
|
@ -180,10 +196,61 @@ _ACTIVE_TAGS = frozenset({
|
|||
"form", "img", "input", "button", "video", "audio", "source", "track", "a",
|
||||
"area", "frame", "frameset", "applet", "style",
|
||||
})
|
||||
# The SCANNER's name set. `<base>`'s only affordance is its `href`, which the
|
||||
# URL-attribute branch still catches; `<base />` without one is inert. The mutator
|
||||
# keeps the full set — see the module docstring.
|
||||
_SCANNER_ACTIVE_TAGS = _ACTIVE_TAGS - {"base"}
|
||||
|
||||
# `_URL_ATTR_RE` above is a presence test and deliberately captures no value.
|
||||
# Reading the value needs the same literal alternation with the value attached, so
|
||||
# no new run shape enters the table: every run here sits in front of a required
|
||||
# literal that the alternation has already anchored. (Self-safety, OWASP LLM10 —
|
||||
# `tests/test_output.py::_REDOS_PAYLOADS` carries the measured row.)
|
||||
_URL_ATTR_VALUE_RE = re.compile(
|
||||
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)"
|
||||
r"\s*=\s*(?P<v>\"[^\"]*\"|'[^']*'|[^\s>]+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
# `srcset` holds a comma-separated candidate list, so an attribute value is not
|
||||
# always one URL. Splitting means a relative first candidate cannot mask an
|
||||
# external one behind it.
|
||||
_URL_CANDIDATE_SPLIT_RE = re.compile(r"[,\s]+")
|
||||
|
||||
|
||||
def _url_attr_is_external(attrs: str) -> bool:
|
||||
"""True if a URL-bearing attribute names an attacker-reachable target.
|
||||
|
||||
Fail-secure: an attribute ``_URL_ATTR_RE`` saw but whose value cannot be read
|
||||
here counts as external, so a gap between the two patterns over-blocks rather
|
||||
than under-blocks.
|
||||
"""
|
||||
seen = False
|
||||
for m in _URL_ATTR_VALUE_RE.finditer(attrs):
|
||||
seen = True
|
||||
value = m.group("v")
|
||||
if value[:1] in "\"'":
|
||||
value = value[1:-1]
|
||||
if any(_has_external_target(c)
|
||||
for c in _URL_CANDIDATE_SPLIT_RE.split(value.strip()) if c):
|
||||
return True
|
||||
return not seen
|
||||
|
||||
|
||||
def is_active_tag(name: str, attrs: str) -> bool:
|
||||
"""True if an HTML tag is active: executing element, event handler, or URL attr."""
|
||||
"""True if a tag is active for the SCANNER: executing element, event handler,
|
||||
or a URL attribute pointing at an external target."""
|
||||
if name.lower() in _SCANNER_ACTIVE_TAGS or _EVENT_ATTR_RE.search(attrs):
|
||||
return True
|
||||
return bool(_URL_ATTR_RE.search(attrs)) and _url_attr_is_external(attrs)
|
||||
|
||||
|
||||
def is_defangable_tag(name: str, attrs: str) -> bool:
|
||||
"""True if the MUTATOR should defang a tag — deliberately broader than
|
||||
:func:`is_active_tag`: any URL attribute, and the full name set.
|
||||
|
||||
Over-defanging costs nothing here (``neutralize`` is opt-in and blocks no
|
||||
disposition), while under-defanging would hand a human a live construct.
|
||||
"""
|
||||
return bool(
|
||||
name.lower() in _ACTIVE_TAGS
|
||||
or _EVENT_ATTR_RE.search(attrs)
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ from .active_content import (
|
|||
MD_REFDEF_RE,
|
||||
URL_IN_TEXT_RE,
|
||||
defang_url,
|
||||
is_active_tag,
|
||||
is_defangable_tag,
|
||||
redact,
|
||||
)
|
||||
from .calibration import MAX_INPUT_CHARS
|
||||
|
|
@ -146,7 +146,7 @@ def neutralize(
|
|||
|
||||
def _html(m: re.Match[str]) -> str:
|
||||
tag = m.group(0)
|
||||
if not is_active_tag(m.group("name"), m.group("attrs") or ""):
|
||||
if not is_defangable_tag(m.group("name"), m.group("attrs") or ""):
|
||||
return tag
|
||||
html_state["count"] += 1
|
||||
if not html_state["ev"]:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue