1
0
Fork 0

fix(active-content): raw-html graded two inert shapes HIGH, and the fix moved a second surface

`is_active_tag`'s URL-attribute branch was a presence test: any element carrying
`href=`/`src=`/`action=` graded HIGH regardless of where the URL pointed. An MDX
`<Card href="/en/agent-sdk/quickstart">` reaches no attacker-controlled host, and
neither does APIM policy XML's `<set-header>`. It now requires an external target
-- the rule the markdown paths have applied since 0.3.1. `<base>` left the active
name set in the same change: HTML's `<base>` has its whole affordance in an `href`
the attribute branch still catches, and APIM's attribute-less `<base />` is inert.

Measured before and after in ONE session against one corpus state, because two of
the three corpora are living and a split would mix this with re-harvest drift:

  reference-corpus  389 docs   133 -> 108   (ceiling 107)
  vendor-harvest    187 docs   100 ->  98   (ceiling  62)
  generated-notes   550 docs    90 ->  88   (ceiling  49)

96% of the achievable reduction in reference-corpus, 5% in the wiki corpora. The
two classes had to be measured TOGETHER -- alone they free 3 and 13 documents,
together 25, because a document carrying one usually carries the other.

The second surface: `neutralize` imported `is_active_tag` by name, so this would
have silently narrowed the opt-in mutator too -- and no test discriminated the two
halves, since every `neutralize:raw-html` payload stays active under any narrowing
considered. That test is written first here. The predicates are now separate
symbols; the mutator keeps defanging anything, because over-defanging is auditable
and blocks nothing while under-defanging hands a human a live construct.

Behaviour change: a document whose only finding was one of these classes now WARNs
instead of holding. Detection is unchanged -- 128/128 classes, 6/6 gaps hold.

Self-safety: reading an attribute VALUE needs a pattern the presence test lacks. It
reuses the same literal alternation so no new run shape enters the table; its
`_REDOS_PAYLOADS` row denies the `=` the pattern requires, since a unit supplying it
matches at once and never exercises the run (the lexicon's `script-tag` row is the
cautionary case). 0.031-0.046s across five attack shapes at 100_000 chars against a
2.0s bound; `docs/redos-sweep.py` reports 0 candidates of 152. An attribute the
presence test saw but the value parser cannot read counts as external -- fail secure.

`docs/rawhtml-census.py` gains a PRODUCTION row that re-measures the shipped
predicate rather than a hypothesis, so a published number and the code cannot drift
apart unnoticed. README's limitation count moves 34 -> 33.

727 passed (was 717).
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 16:56:31 +02:00
commit 736f370cfb
9 changed files with 287 additions and 72 deletions

View file

@ -7,7 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
Nothing yet. ### Changed — `active:raw-html` stops firing on two things that carry no affordance
`is_active_tag` had two over-reaching branches, both measured on consumer corpora
rather than argued from the code:
- **The URL-attribute branch was a presence test.** Any element carrying `href=`,
`src=`, `action=` … graded HIGH regardless of where the URL pointed. An MDX
`<Card href="/en/agent-sdk/quickstart">` — an internal doc route — reaches no
attacker-controlled host, and neither does Azure APIM policy XML's `<set-header>`.
The branch now requires an **external** target (absolute scheme or
protocol-relative), the rule the markdown paths have applied since 0.3.1.
- **`<base>` left the active name set.** HTML's `<base>` has its entire affordance in
its `href`, which the URL-attribute branch still catches. APIM's attribute-less
`<base />` means "run the inherited policy" and is inert in every renderer.
**Measured before and after in one session, against one corpus state** — the two
wiki corpora are living, so a before/after split across sessions would mix this
change with re-harvest drift:
| population | n | before | after | ceiling (raw-HTML off) |
|---|---|---|---|---|
| reference-corpus | 389 | 133 | **108** | 107 |
| vendor-harvest | 187 | 100 | **98** | 62 |
| generated-notes | 550 | 90 | **88** | 49 |
96% of the achievable reduction in reference-corpus, 5% in the two wiki corpora:
the over-reach was nearly the whole raw-html cost in APIM policy XML and nearly none
of it in vendor documentation, where what remains is real HTML — `<a>` 298, `<frame>`
94, `<img>` 63 — caught correctly by the name branch.
**The two classes had to be measured together.** Alone they free 3 and 13 documents
in reference-corpus; together, 25. A document carrying one usually carries the other,
so closing either alone leaves it blocked by its twin. `docs/rawhtml-census.py` now
carries a `PRODUCTION` row that re-measures the shipped predicate instead of a
hypothetical, so a doc number and the code cannot drift apart unnoticed.
### Known behaviour change
**Documents whose only finding was one of these two classes now persist unattended.**
On `PRESET_USER_UPLOAD` they move from `fail_secure` / `quarantine_review` to `WARN`
— 25 documents in the reference corpus, 2 in each wiki corpus. This is a deliberate
loosening of the *upload door*, not of detection: recall is unchanged at 128/128
demonstrated classes with 6/6 documented gaps holding, and a tag that is active by
name, carries an `on*=` handler, or points anywhere external is graded exactly as
before. An element outside the active name set whose only URL attribute is
doc-relative is the whole of what changed.
### Fixed — the scanner and the mutator no longer share one predicate
`neutralize` imported `is_active_tag` from `active_content` by name, so narrowing the
scanner would have silently narrowed the opt-in mutator as well — and **no test in
the suite discriminated the two halves**: every `neutralize:raw-html` payload stayed
active under each narrowing considered. The predicates are now separate symbols,
`is_active_tag` (scanner, external-target rule) and `is_defangable_tag` (mutator,
unchanged broad behaviour), and the mutator half is pinned by its own test. Over-
defanging costs nothing there — `neutralize` is opt-in and blocks no disposition —
while under-defanging would hand a human a live construct.
### Self-safety (OWASP LLM10)
Reading a URL attribute's *value* needs a pattern the presence test does not provide.
It reuses the same literal alternation with the value attached, so no new run shape
enters the table, and `_REDOS_PAYLOADS` gains a row (`active-url-attr-value`) whose
unit **denies** the `=` the pattern requires — a unit supplying it matches at once and
never exercises the run. Measured at 100_000 chars: 0.0310.046s across five attack
shapes, against the suite's 2.0s bound. A gap between the two patterns fails secure:
an attribute seen by the presence test but unreadable by the value parser counts as
external, so it over-blocks rather than under-blocks.
727 tests pass (was 717).
## [0.5.0] — 2026-08-11 ## [0.5.0] — 2026-08-11

View file

@ -241,8 +241,12 @@ a green scan means safe content. The highest-impact items:
other than WARN. Technical documentation is the expensive case: it is dense in the other than WARN. Technical documentation is the expensive case: it is dense in the
exact constructs the gate grades. Budget human review, or run a source you actually exact constructs the gate grades. Budget human review, or run a source you actually
trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py). trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py).
Those three are the published pre-0.6.0 numbers; the raw-HTML narrowing in 0.6.0
moves them to **108**, **98** and **88** measured against current corpus state —
two of the three corpora are living, so the cells are not rewritten in place.
Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
**Full list — 34 items, each with the mechanism, plus the out-of-scope boundary:** **Full list — 33 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces. consumer corpora, including the false positives the URL-shape rule actually produces.

View file

@ -216,55 +216,42 @@ items; this is the full list, each with the mechanism.
where this would bite — the two corpora that reported a carrier breakdown contained where this would bite — the two corpora that reported a carrier breakdown contained
zero remote images and the third did not report one, so the image row of this zero remote images and the third did not report one, so the image row of this
limitation remains unmeasured in the field. limitation remains unmeasured in the field.
- **Raw HTML with a *relative* URL attribute is HIGH, though it can reach no - **What `active:raw-html` still costs benign documentation is mostly ordinary HTML,
attacker-controlled host.** The markdown paths test for an external target before not over-reach — and that residual is not a narrowing away.** Two over-reach
flagging; the raw-HTML path deliberately does not, because an active element needs classes were closed in 0.6.0: the URL-attribute branch now requires an *external*
no URL at all (an `on*=` handler executes on its own). That reasoning covers event target (an element outside the active name set carrying `href="/en/agent-sdk/quickstart"`
handlers but over-reaches on the URL-attribute branch: an element outside the active reaches no attacker-controlled host), and `<base>` left the active *name* set (APIM's
name set carrying `href="/en/agent-sdk/quickstart"` — an internal doc route — grades attribute-less `<base />` means "run the inherited policy"; HTML's `<base>` has its
HIGH. Measured on a vendor-docs corpus, where it lands on MDX components: whole affordance in an `href` the attribute branch still catches). Measured before
`<Card href="/…">` fires this way, and `<Frame>` fires on the *name* branch alone and after in one session against one corpus state — `docs/rawhtml-census.py`, whose
because names are lower-cased and `frame` is in the active set — legacy HTML `PRODUCTION` row re-measures the shipped predicate rather than a hypothesis:
framesets, which appear in essentially no modern documentation, while `Frame` is a reference-corpus 133 → **108** of 389, vendor-harvest 100 → **98** of 187,
common component name. Case is not an available discriminator: HTML is generated-notes 90 → **88** of 550. Against a ceiling of 26 / 38 / 41 (raw-HTML
case-insensitive, so PascalCase cannot be treated as "component, not tag". detection switched off entirely) that is **96% of the achievable reduction in
**Correction (2026-08-11): this bullet was named as the cause of the raw-html reference-corpus and 5% in the two wiki corpora** — the over-reach was nearly the
false-positive rate, and it is not.** The bullet below used to attribute whole raw-html cost in APIM policy XML and nearly none of it in vendor
`active:raw-html` in *52 of vendor-harvest's 98* to this over-reach. Measured by documentation.
re-running the gate with the URL-attribute branch narrowed to *external* targets **The classes had to be measured together.** Alone, the URL-attribute branch frees
only, it frees **1 document in vendor-harvest, 1 in generated-notes, 3 in 3 documents in reference-corpus and `<base>` frees 13; together they free 25. A
reference-corpus**, because the documents carrying it almost always also carry a document carrying one usually carries the other, so closing either alone leaves it
tag that is active by *name*, and one raw-html finding per document is all it blocked by its twin — one-at-a-time measurement understates both, and reading
takes. What actually drives the rate is the **name branch**: per wiki corpus `<a>` "frees 3" as "this over-reach is cheap" is how this document got the attribution
298 occurrences, `<frame>` 94, `<img>` 63; in reference-corpus `<base>` 113. wrong for three releases.
**Do not read "frees 3" as "this over-reach is cheap" — the over-reach classes **What remains is real HTML in vendor documentation**, flagged by the *name*
co-occur, and measuring them one at a time understates both.** In reference-corpus, branch, correctly: per wiki corpus `<a>` 298 occurrences, `<frame>` 94, `<img>` 63
narrowing the URL-attribute branch alone frees 3 and taking `<base>` off the name — identical in both because 184 of their documents share ancestors. `<frame>` is
branch alone frees 13, but doing **both** frees **25** — because a document the one arguable member: names are lower-cased and `frame` is in the active set for
carrying one class usually carries the other, so closing either alone leaves the legacy HTML framesets, which appear in essentially no modern documentation, while
document blocked by its twin. Against a ceiling of 26 (raw-html detection switched `Frame` is a common MDX component name. Case is not an available discriminator:
off entirely) the pair captures 96% of what this detector costs that population. HTML is case-insensitive, so PascalCase cannot be treated as "component, not tag".
The same pair frees only 2 of a 38-document ceiling in vendor-harvest and 2 of 41 Recovering the rest is **not** a further narrowing — it needs a carrier split and a
in generated-notes: the over-reach is nearly the whole raw-html cost in APIM policy new label (`active:raw-html-link`), because raw HTML has no ordinary form and the
XML and nearly none of it in vendor documentation. class collapses to one finding per document.
**And the over-reach is not only MDX.** In reference-corpus it lands on Azure APIM **The scanner and the mutator no longer share a predicate.** Until 0.6.0 `neutralize`
policy XML — `<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2 — which imported `is_active_tag` by name, so any narrowing moved the opt-in mutator too, and
shares nothing with the `<Card href="/…">` shape beyond having a URL attribute. no test discriminated the two halves. They are now `is_active_tag` and
**Any fix here moves two surfaces, not one:** `neutralize` imports `is_active_tag` `is_defangable_tag`; the mutator kept the broader behaviour deliberately, pinned by
from this module, so narrowing it also stops the opt-in *mutator* from defanging `tests/test_neutralize.py::test_mutator_still_defangs_what_the_scanner_now_lets_pass`.
the same tags. No test covers that half — the suite's `neutralize:raw-html` cases
use payloads that stay active under every narrowing considered — so the mutator
would change behaviour silently. Whoever closes this writes that test first.
- **`<base />` in Azure APIM policy XML grades HIGH on the name branch alone.**
APIM's `<base />` means "run the inherited policy" and carries no attributes; it
collides with HTML's `<base>`, whose entire affordance is its `href`. It appears in
**25 of reference-corpus's 389** documents (`grep -rlE '<base\s*/?>'` — a loose
`grep '<base'` says 30 and is wrong: it also matches the literal `<base64_string>`
placeholder, which is not a tag this detector fires on). An attribute-less `<base>` has no
affordance in any renderer, and a `<base href="…">` would still be caught by the
URL-attribute branch — so the name-branch entry is doing no work here that the
attribute branch does not already do. This is a distinct over-reach class from the
MDX/APIM one above, and it was unnamed until the 2026-08-11 census.
- **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a - **Raw-HTML findings count end tags.** `</a>` is active by name on its own, so a
corpus census that counts only opening tags understates what this detector reports corpus census that counts only opening tags understates what this detector reports
by roughly the ratio of closing to opening active tags (measured at 1.6× on one by roughly the ratio of closing to opening active tags (measured at 1.6× on one
@ -284,6 +271,15 @@ items; this is the full list, each with the mechanism.
| generated-notes | model-written notes at their own persist gate | 547 | **88 (16.1%)** — 61 fail-secure, 27 held | | generated-notes | model-written notes at their own persist gate | 547 | **88 (16.1%)** — 61 fail-secure, 27 held |
| reference-corpus | first-party authored reference material | 389 | **133 (34.2%)** — 80 fail-secure, 53 held | | reference-corpus | first-party authored reference material | 389 | **133 (34.2%)** — 80 fail-secure, 53 held |
**These three numbers predate the 0.6.0 raw-HTML narrowing and are left as
published**, because two of the three corpora are living — re-harvested by their
owning repo, now 187 and 550 documents against the 185 and 547 measured here — so
rewriting the cells would mix a code change with corpus drift. The narrowing's
effect was measured separately, before and after against one corpus state:
reference-corpus (static, and reproduced at exactly 389/133) drops to **108**, the
two wiki rows to **98** and **88** on their current state. The bullet above carries
the method; `docs/rawhtml-census.py LABEL=<path>` reproduces it.
**The unit is a document and the gate is the strict one:** `screen_output(doc, **The unit is a document and the gate is the strict one:** `screen_output(doc,
PRESET_USER_UPLOAD)`, counting `disposition is not WARN`. WARN is the benign PRESET_USER_UPLOAD)`, counting `disposition is not WARN`. WARN is the benign
outcome (persisted, with a note), so a *finding* is not a false positive — only a outcome (persisted, with a note), so a *finding* is not a false positive — only a

View file

@ -22,10 +22,15 @@ TWO METHOD TRAPS IT EXISTS TO AVOID:
effect with corpus drift. Every candidate here runs against the same corpus state effect with corpus drift. Every candidate here runs against the same corpus state
in one process, and `base` is re-measured rather than quoted from the doc. in one process, and `base` is re-measured rather than quoted from the doc.
The candidates are applied by replacing `active_content.is_active_tag` in-process. The candidates are applied by replacing `active_content.is_active_tag` in-process,
That mirrors a real edit for the *scanner* path only: `neutralize` imports the which mirrors a real edit to the *scanner*. Since 0.6.0 that is the whole story:
symbol by name, so a real edit would also change the mutator, which this script does `neutralize` calls its own `is_defangable_tag`, so patching this symbol cannot
not simulate. See the raw-HTML bullets in `docs/LIMITATIONS.md`. move the mutator. Before 0.6.0 the two shared one symbol and this caveat read the
other way. See the raw-HTML bullets in `docs/LIMITATIONS.md`.
The `PRODUCTION` row is the only one that is not a hypothetical: it leaves the
shipped predicate in place. A shipped narrowing must equal its candidate row, and
saying so in the output is what keeps the doc's numbers checkable after the fact.
USAGE corpus roots are arguments, never hardcoded; the corpora live in private USAGE corpus roots are arguments, never hardcoded; the corpora live in private
consumer repos and their paths must not reach a public mirror: consumer repos and their paths must not reach a public mirror:
@ -88,7 +93,7 @@ def _variant(*, drop: frozenset[str] = frozenset(), external_only: bool = False)
CANDIDATES = [ CANDIDATES = [
("base", _variant()), ("pre-0.6.0 (no narrowing)", _variant()),
# The URL-attribute branch requires an EXTERNAL target — the rule the markdown # The URL-attribute branch requires an EXTERNAL target — the rule the markdown
# paths already apply. A relative `href` reaches no attacker-controlled host. # paths already apply. A relative `href` reaches no attacker-controlled host.
("A: url-attr external-only", _variant(external_only=True)), ("A: url-attr external-only", _variant(external_only=True)),
@ -98,6 +103,10 @@ CANDIDATES = [
("base-url: <base> needs a URL", _variant(drop=frozenset({"base"}))), ("base-url: <base> needs a URL", _variant(drop=frozenset({"base"}))),
("A + base-url (both)", ("A + base-url (both)",
_variant(drop=frozenset({"base"}), external_only=True)), _variant(drop=frozenset({"base"}), external_only=True)),
# Not a hypothetical: the shipped predicate, unpatched. `A + base-url` is what
# 0.6.0 shipped, so these two rows must agree — a mismatch means the code and
# this script have drifted apart and every number below is suspect.
("PRODUCTION (as shipped)", None),
# The CEILING: no narrowing can free more than switching the detector off. # The CEILING: no narrowing can free more than switching the detector off.
("NONE (ceiling)", lambda name, attrs: False), ("NONE (ceiling)", lambda name, attrs: False),
] ]
@ -180,7 +189,7 @@ def main() -> None:
baseline = None baseline = None
for name, fn in CANDIDATES: for name, fn in CANDIDATES:
ac.is_active_tag = fn ac.is_active_tag = original if fn is None else fn
non_warn = sum( non_warn = sum(
1 for t in texts 1 for t in texts
if screen_output(t, PRESET_USER_UPLOAD).disposition is not BENIGN if screen_output(t, PRESET_USER_UPLOAD).disposition is not BENIGN

View file

@ -16,13 +16,29 @@ consumers share it:
* :func:`~llm_ingestion_guard.neutralize.neutralize` the separate, opt-in * :func:`~llm_ingestion_guard.neutralize.neutralize` the separate, opt-in
**mutator** that defangs the same constructs for human audit. **mutator** that defangs the same constructs for human audit.
One deliberate asymmetry between the two: the scanner flags markdown images and One deliberate asymmetry between the two: the scanner flags a construct only when
links only when the URL is absolute or protocol-relative. A relative in-document its URL is absolute or protocol-relative. A relative in-document link has no
link has no attacker-reachable endpoint, and flagging it would silently attacker-reachable endpoint, and flagging it would silently over-block legitimate
over-block legitimate wiki/OKF content (design principle 5) cross-linking is wiki/OKF content (design principle 5) cross-linking is those formats' core
those formats' core mechanism. ``neutralize`` keeps its broader defang-anything mechanism. ``neutralize`` keeps its broader defang-anything behavior: it is
behavior: it is opt-in, and bracketed dots in a relative path are auditable, opt-in, and bracketed dots in a relative path are auditable, not blocking.
not blocking.
The two predicates are therefore separate symbols :func:`is_active_tag` for the
scanner, :func:`is_defangable_tag` for the mutator. They were one symbol until
0.6.0, imported by name across modules, so narrowing the scanner would have moved
the mutator silently.
**The asymmetry covers raw HTML too** (0.6.0). It previously applied only to the
markdown paths: a tag was active if it carried a URL attribute *at all*, so an MDX
``<Card href="/en/quickstart">`` a doc-relative route on a name outside the
active set carried HIGH. It now requires an external target, the rule the
markdown paths have applied since 0.3.1. ``<base>`` left the active *name* set in
the same change: its whole affordance is its ``href``, which the URL-attribute
branch still catches, while the attribute-less ``<base />`` of Azure APIM policy
XML has no affordance in any renderer. Measured together rather than one at a time
the classes co-occur the pair frees 25 of 133 non-WARN documents on the
reference corpus and 2 each on the two wiki corpora, at unchanged recall. Method
and numbers: ``docs/rawhtml-census.py``; residuals: ``docs/LIMITATIONS.md``.
**Severity grades on URL shape, not construct type** (0.3.1). The exfiltration **Severity grades on URL shape, not construct type** (0.3.1). The exfiltration
primitive is not "an image" it is a URL that moves bytes to a host the primitive is not "an image" it is a URL that moves bytes to a host the
@ -180,10 +196,61 @@ _ACTIVE_TAGS = frozenset({
"form", "img", "input", "button", "video", "audio", "source", "track", "a", "form", "img", "input", "button", "video", "audio", "source", "track", "a",
"area", "frame", "frameset", "applet", "style", "area", "frame", "frameset", "applet", "style",
}) })
# The SCANNER's name set. `<base>`'s only affordance is its `href`, which the
# URL-attribute branch still catches; `<base />` without one is inert. The mutator
# keeps the full set — see the module docstring.
_SCANNER_ACTIVE_TAGS = _ACTIVE_TAGS - {"base"}
# `_URL_ATTR_RE` above is a presence test and deliberately captures no value.
# Reading the value needs the same literal alternation with the value attached, so
# no new run shape enters the table: every run here sits in front of a required
# literal that the alternation has already anchored. (Self-safety, OWASP LLM10 —
# `tests/test_output.py::_REDOS_PAYLOADS` carries the measured row.)
_URL_ATTR_VALUE_RE = re.compile(
r"\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)"
r"\s*=\s*(?P<v>\"[^\"]*\"|'[^']*'|[^\s>]+)",
re.IGNORECASE,
)
# `srcset` holds a comma-separated candidate list, so an attribute value is not
# always one URL. Splitting means a relative first candidate cannot mask an
# external one behind it.
_URL_CANDIDATE_SPLIT_RE = re.compile(r"[,\s]+")
def _url_attr_is_external(attrs: str) -> bool:
"""True if a URL-bearing attribute names an attacker-reachable target.
Fail-secure: an attribute ``_URL_ATTR_RE`` saw but whose value cannot be read
here counts as external, so a gap between the two patterns over-blocks rather
than under-blocks.
"""
seen = False
for m in _URL_ATTR_VALUE_RE.finditer(attrs):
seen = True
value = m.group("v")
if value[:1] in "\"'":
value = value[1:-1]
if any(_has_external_target(c)
for c in _URL_CANDIDATE_SPLIT_RE.split(value.strip()) if c):
return True
return not seen
def is_active_tag(name: str, attrs: str) -> bool: def is_active_tag(name: str, attrs: str) -> bool:
"""True if an HTML tag is active: executing element, event handler, or URL attr.""" """True if a tag is active for the SCANNER: executing element, event handler,
or a URL attribute pointing at an external target."""
if name.lower() in _SCANNER_ACTIVE_TAGS or _EVENT_ATTR_RE.search(attrs):
return True
return bool(_URL_ATTR_RE.search(attrs)) and _url_attr_is_external(attrs)
def is_defangable_tag(name: str, attrs: str) -> bool:
"""True if the MUTATOR should defang a tag — deliberately broader than
:func:`is_active_tag`: any URL attribute, and the full name set.
Over-defanging costs nothing here (``neutralize`` is opt-in and blocks no
disposition), while under-defanging would hand a human a live construct.
"""
return bool( return bool(
name.lower() in _ACTIVE_TAGS name.lower() in _ACTIVE_TAGS
or _EVENT_ATTR_RE.search(attrs) or _EVENT_ATTR_RE.search(attrs)

View file

@ -50,7 +50,7 @@ from .active_content import (
MD_REFDEF_RE, MD_REFDEF_RE,
URL_IN_TEXT_RE, URL_IN_TEXT_RE,
defang_url, defang_url,
is_active_tag, is_defangable_tag,
redact, redact,
) )
from .calibration import MAX_INPUT_CHARS from .calibration import MAX_INPUT_CHARS
@ -146,7 +146,7 @@ def neutralize(
def _html(m: re.Match[str]) -> str: def _html(m: re.Match[str]) -> str:
tag = m.group(0) tag = m.group(0)
if not is_active_tag(m.group("name"), m.group("attrs") or ""): if not is_defangable_tag(m.group("name"), m.group("attrs") or ""):
return tag return tag
html_state["count"] += 1 html_state["count"] += 1
if not html_state["ev"]: if not html_state["ev"]:

View file

@ -268,11 +268,6 @@ def test_default_source_is_output_and_override_respected():
# closed over-block should fail here and force the doc to be updated. # closed over-block should fail here and force the doc to be updated.
@pytest.mark.parametrize("cid,text", [ @pytest.mark.parametrize("cid,text", [
# Fires on the URL-attr branch although the target is a relative doc route,
# which cannot reach an attacker-controlled host. `Card` is not in the active
# name set — the href alone carries it.
("relative-href-on-inactive-name",
'<Card title="Quickstart" icon="play" href="/en/agent-sdk/quickstart">'),
# Fires on the *name* branch: names are lower-cased and `frame` is in the # Fires on the *name* branch: names are lower-cased and `frame` is in the
# active set (legacy HTML framesets), while `Frame` is a common MDX component. # active set (legacy HTML framesets), while `Frame` is a common MDX component.
("mdx-component-named-like-a-tag", "<Frame>"), ("mdx-component-named-like-a-tag", "<Frame>"),
@ -284,6 +279,51 @@ def test_raw_html_overblocks_are_still_high(cid, text):
assert finding[0].severity is Severity.HIGH, f"{cid}: {finding[0].severity}" assert finding[0].severity is Severity.HIGH, f"{cid}: {finding[0].severity}"
# --- the two over-blocks CLOSED in 0.6.0 (the `A + base-url` narrowing) -------
# Measured together, never one at a time: the classes co-occur, so a document
# blocked by both is freed by neither alone. On the reference corpus the pair
# frees 25 of 133 non-WARN documents; on the two wiki corpora, 2 each. Recall is
# unchanged (128/128 + 6/6). Method and numbers: `docs/rawhtml-census.py`.
def test_relative_url_attr_on_an_inactive_name_is_not_active():
# `_URL_ATTR_RE` is a presence test, so a doc-relative route on a name outside
# the active set used to carry HIGH on its own. A relative target resolves
# against the rendering host and reaches nothing attacker-controlled — the rule
# the markdown paths have applied since 0.3.1.
assert not [
f for f in scan_active_content(
'<Card title="Quickstart" icon="play" href="/en/agent-sdk/quickstart">'
).findings if f.label == "active:raw-html"
]
def test_attributeless_base_is_not_active():
# `<base>`'s whole affordance is its `href`, which the URL-attribute branch
# still catches (below). An attribute-less `<base />` — Azure APIM policy XML,
# 25 documents in the reference corpus — has no affordance in any renderer.
assert not [f for f in scan_active_content("<base />").findings
if f.label == "active:raw-html"]
@pytest.mark.parametrize("cid,text", [
("absolute", '<Card title="Docs" href="https://evil.example/leak?d=x">'),
("protocol-relative", '<Card href="//evil.example/leak">'),
("non-http-scheme", '<Card href="file:///etc/passwd">'),
("base-keeps-its-href", '<base href="https://evil.example/">'),
# `srcset` is a comma-separated candidate list. A relative FIRST candidate must
# not mask an external one behind it — the value is not one URL.
("srcset-second-candidate", '<Card srcset="a.png 1x, https://evil.example/b.png 2x">'),
# The value parser saw the attribute but can resolve no value. The gap must
# over-block, never under-block.
("unreadable-value-fails-secure", "<Card href= >"),
])
def test_external_url_attr_is_still_active(cid, text):
finding = [f for f in scan_active_content(text).findings
if f.label == "active:raw-html"]
assert len(finding) == 1, f"{cid}: raw-html not reported"
assert finding[0].severity is Severity.HIGH, f"{cid}: {finding[0].severity}"
def test_raw_html_counts_end_tags(): def test_raw_html_counts_end_tags():
# `</a>` is active by name on its own, so a corpus census counting only opening # `</a>` is active by name on its own, so a corpus census counting only opening
# tags understates this detector's `count`. The class still collapses to ONE # tags understates this detector's `count`. The class still collapses to ONE

View file

@ -13,6 +13,8 @@ The transform is pure ``text -> (defanged_text, report)`` — no I/O, no globals
""" """
import time import time
import pytest
from llm_ingestion_guard.neutralize import neutralize from llm_ingestion_guard.neutralize import neutralize
from llm_ingestion_guard.report import Severity, Source from llm_ingestion_guard.report import Severity, Source
@ -97,6 +99,27 @@ def test_raw_active_html_is_escaped():
assert html[0].severity is Severity.HIGH assert html[0].severity is Severity.HIGH
@pytest.mark.parametrize("cid,text", [
("relative-href-on-inactive-name", '<Card href="/en/agent-sdk/quickstart">'),
("attributeless-base", "<base />"),
])
def test_mutator_still_defangs_what_the_scanner_now_lets_pass(cid, text):
# The deliberate asymmetry, extended to raw HTML in 0.6.0: the SCANNER narrowed
# its URL-attribute branch to external targets and dropped `<base>` from the name
# set; the opt-in MUTATOR keeps defanging anything. Over-defanging costs nothing
# here — it is auditable and blocks no disposition — while under-defanging would
# hand a human a live construct.
#
# Pinned because the two predicates are separate symbols as of this change
# (`is_active_tag` vs `is_defangable_tag`). Before the split, `neutralize`
# imported the scanner's predicate by name, so narrowing it would have moved the
# mutator silently — no test in this suite discriminated the two.
result = neutralize(text)
assert result.report.found is True, cid
assert any(f.label == "neutralize:raw-html" for f in result.report.findings), cid
assert "&lt;" in result.text, f"{cid}: not escaped -- {result.text!r}"
def test_benign_formatting_html_is_left_untouched(): def test_benign_formatting_html_is_left_untouched():
text = "This is **bold** and <b>strong</b> and <em>emph</em> text." text = "This is **bold** and <b>strong</b> and <em>emph</em> text."
result = neutralize(text) result = neutralize(text)

View file

@ -392,6 +392,13 @@ _REDOS_PAYLOADS = [
("active-md-image-url", scan_active_content, "![a]("), ("active-md-image-url", scan_active_content, "![a]("),
("active-md-link-url", scan_active_content, "[a]("), ("active-md-link-url", scan_active_content, "[a]("),
("active-html-tag-attrs", scan_active_content, "<a"), ("active-html-tag-attrs", scan_active_content, "<a"),
# 0.6.0 put a value parser behind the URL-attribute presence test. Its one run
# is the `\s*` in front of the required `=`, so the unit has to DENY the `=`:
# a unit that supplies it matches immediately and never exercises the run. The
# lexicon's `script-tag` row is the cautionary case — its `<script>` unit hands
# the pattern the `>` it needs, so that row stayed green under a form measured
# at ~84s on the input this gate accepts.
("active-url-attr-value", scan_active_content, "<a href >"),
# The lexicon is on the output path too, and it had the same defect in the # The lexicon is on the output path too, and it had the same defect in the
# JSON pattern table -- found only because the composed-gate test below # JSON pattern table -- found only because the composed-gate test below
# stayed red after every scanner above was already linear. `<a:` drove the # stayed red after every scanner above was already linear. `<a:` drove the