1
0
Fork 0

fix(active-content): raw-html graded two inert shapes HIGH, and the fix moved a second surface

`is_active_tag`'s URL-attribute branch was a presence test: any element carrying
`href=`/`src=`/`action=` graded HIGH regardless of where the URL pointed. An MDX
`<Card href="/en/agent-sdk/quickstart">` reaches no attacker-controlled host, and
neither does APIM policy XML's `<set-header>`. It now requires an external target
-- the rule the markdown paths have applied since 0.3.1. `<base>` left the active
name set in the same change: HTML's `<base>` has its whole affordance in an `href`
the attribute branch still catches, and APIM's attribute-less `<base />` is inert.

Measured before and after in ONE session against one corpus state, because two of
the three corpora are living and a split would mix this with re-harvest drift:

  reference-corpus  389 docs   133 -> 108   (ceiling 107)
  vendor-harvest    187 docs   100 ->  98   (ceiling  62)
  generated-notes   550 docs    90 ->  88   (ceiling  49)

96% of the achievable reduction in reference-corpus, 5% in the wiki corpora. The
two classes had to be measured TOGETHER -- alone they free 3 and 13 documents,
together 25, because a document carrying one usually carries the other.

The second surface: `neutralize` imported `is_active_tag` by name, so this would
have silently narrowed the opt-in mutator too -- and no test discriminated the two
halves, since every `neutralize:raw-html` payload stays active under any narrowing
considered. That test is written first here. The predicates are now separate
symbols; the mutator keeps defanging anything, because over-defanging is auditable
and blocks nothing while under-defanging hands a human a live construct.

Behaviour change: a document whose only finding was one of these classes now WARNs
instead of holding. Detection is unchanged -- 128/128 classes, 6/6 gaps hold.

Self-safety: reading an attribute VALUE needs a pattern the presence test lacks. It
reuses the same literal alternation so no new run shape enters the table; its
`_REDOS_PAYLOADS` row denies the `=` the pattern requires, since a unit supplying it
matches at once and never exercises the run (the lexicon's `script-tag` row is the
cautionary case). 0.031-0.046s across five attack shapes at 100_000 chars against a
2.0s bound; `docs/redos-sweep.py` reports 0 candidates of 152. An attribute the
presence test saw but the value parser cannot read counts as external -- fail secure.

`docs/rawhtml-census.py` gains a PRODUCTION row that re-measures the shipped
predicate rather than a hypothesis, so a published number and the code cannot drift
apart unnoticed. README's limitation count moves 34 -> 33.

727 passed (was 717).
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 16:56:31 +02:00
commit 736f370cfb
9 changed files with 287 additions and 72 deletions

View file

@ -16,13 +16,29 @@ consumers share it:
* :func:`~llm_ingestion_guard.neutralize.neutralize` the separate, opt-in
**mutator** that defangs the same constructs for human audit.
One deliberate asymmetry between the two: the scanner flags markdown images and
links only when the URL is absolute or protocol-relative. A relative in-document
link has no attacker-reachable endpoint, and flagging it would silently
over-block legitimate wiki/OKF content (design principle 5) cross-linking is
those formats' core mechanism. ``neutralize`` keeps its broader defang-anything
behavior: it is opt-in, and bracketed dots in a relative path are auditable,
not blocking.
One deliberate asymmetry between the two: the scanner flags a construct only when
its URL is absolute or protocol-relative. A relative in-document link has no
attacker-reachable endpoint, and flagging it would silently over-block legitimate
wiki/OKF content (design principle 5) cross-linking is those formats' core
mechanism. ``neutralize`` keeps its broader defang-anything behavior: it is
opt-in, and bracketed dots in a relative path are auditable, not blocking.
The two predicates are therefore separate symbols :func:`is_active_tag` for the
scanner, :func:`is_defangable_tag` for the mutator. They were one symbol until
0.6.0, imported by name across modules, so narrowing the scanner would have moved
the mutator silently.
**The asymmetry covers raw HTML too** (0.6.0). It previously applied only to the
markdown paths: a tag was active if it carried a URL attribute *at all*, so an MDX
``<Card href="/en/quickstart">`` a doc-relative route on a name outside the
active set carried HIGH. It now requires an external target, the rule the
markdown paths have applied since 0.3.1. ``<base>`` left the active *name* set in
the same change: its whole affordance is its ``href``, which the URL-attribute
branch still catches, while the attribute-less ``<base />`` of Azure APIM policy
XML has no affordance in any renderer. Measured together rather than one at a time
the classes co-occur the pair frees 25 of 133 non-WARN documents on the
reference corpus and 2 each on the two wiki corpora, at unchanged recall. Method
and numbers: ``docs/rawhtml-census.py``; residuals: ``docs/LIMITATIONS.md``.
**Severity grades on URL shape, not construct type** (0.3.1). The exfiltration
primitive is not "an image" it is a URL that moves bytes to a host the
@ -180,10 +196,61 @@ _ACTIVE_TAGS = frozenset({
"form", "img", "input", "button", "video", "audio", "source", "track", "a",
"area", "frame", "frameset", "applet", "style",
})
# The SCANNER's name set. `<base>`'s only affordance is its `href`, which the
# URL-attribute branch still catches; `<base />` without one is inert. The mutator
# keeps the full set — see the module docstring.
_SCANNER_ACTIVE_TAGS = _ACTIVE_TAGS - {"base"}
# `_URL_ATTR_RE` above is a presence test and deliberately captures no value.
# Reading the value needs the same literal alternation with the value attached, so
# no new run shape enters the table: every run here sits in front of a required
# literal that the alternation has already anchored. (Self-safety, OWASP LLM10 —
# `tests/test_output.py::_REDOS_PAYLOADS` carries the measured row.)
_URL_ATTR_VALUE_RE = re.compile(
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)"
r"\s*=\s*(?P<v>\"[^\"]*\"|'[^']*'|[^\s>]+)",
re.IGNORECASE,
)
# `srcset` holds a comma-separated candidate list, so an attribute value is not
# always one URL. Splitting means a relative first candidate cannot mask an
# external one behind it.
_URL_CANDIDATE_SPLIT_RE = re.compile(r"[,\s]+")
def _url_attr_is_external(attrs: str) -> bool:
"""True if a URL-bearing attribute names an attacker-reachable target.
Fail-secure: an attribute ``_URL_ATTR_RE`` saw but whose value cannot be read
here counts as external, so a gap between the two patterns over-blocks rather
than under-blocks.
"""
seen = False
for m in _URL_ATTR_VALUE_RE.finditer(attrs):
seen = True
value = m.group("v")
if value[:1] in "\"'":
value = value[1:-1]
if any(_has_external_target(c)
for c in _URL_CANDIDATE_SPLIT_RE.split(value.strip()) if c):
return True
return not seen
def is_active_tag(name: str, attrs: str) -> bool:
"""True if an HTML tag is active: executing element, event handler, or URL attr."""
"""True if a tag is active for the SCANNER: executing element, event handler,
or a URL attribute pointing at an external target."""
if name.lower() in _SCANNER_ACTIVE_TAGS or _EVENT_ATTR_RE.search(attrs):
return True
return bool(_URL_ATTR_RE.search(attrs)) and _url_attr_is_external(attrs)
def is_defangable_tag(name: str, attrs: str) -> bool:
"""True if the MUTATOR should defang a tag — deliberately broader than
:func:`is_active_tag`: any URL attribute, and the full name set.
Over-defanging costs nothing here (``neutralize`` is opt-in and blocks no
disposition), while under-defanging would hand a human a live construct.
"""
return bool(
name.lower() in _ACTIVE_TAGS
or _EVENT_ATTR_RE.search(attrs)

View file

@ -50,7 +50,7 @@ from .active_content import (
MD_REFDEF_RE,
URL_IN_TEXT_RE,
defang_url,
is_active_tag,
is_defangable_tag,
redact,
)
from .calibration import MAX_INPUT_CHARS
@ -146,7 +146,7 @@ def neutralize(
def _html(m: re.Match[str]) -> str:
tag = m.group(0)
if not is_active_tag(m.group("name"), m.group("attrs") or ""):
if not is_defangable_tag(m.group("name"), m.group("attrs") or ""):
return tag
html_state["count"] += 1
if not html_state["ev"]: