open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 3 Activity

measure(rawhtml): the over-reach classes co-occur, so one-at-a-time understates both

Browse source 
Yesterday's correction fixed a wrong causal claim and published a second one.
It reported that narrowing the URL-attribute branch frees 3 documents in
reference-corpus and left the reader to conclude the over-reach is cheap.
Measured together with the `<base>` name-branch fix it frees 25, against a
ceiling of 26 — 96% of what the detector costs that population. A document
blocked by two over-reach classes is freed by neither alone.

The same pair frees 2 of a 38-document ceiling in vendor-harvest and 2 of 41
in generated-notes. The over-reach is nearly the whole raw-html cost in Azure
APIM policy XML and nearly none of it in vendor documentation.

Three further corrections:

- `<base />` appears in 25 reference-corpus documents, not 30. The published
  count came from `grep '<base'`, which also matched the literal
  `<base64_string>` placeholder — not a tag this detector fires on.
- Any fix moves TWO surfaces: `neutralize` imports `is_active_tag` by name, so
  narrowing it also stops the opt-in mutator defanging the same tags. No test
  covers that half; the suite's `neutralize:raw-html` payloads stay active
  under every narrowing considered.
- Adds `docs/rawhtml-census.py` so the ladder is reproducible instead of
  living in a session scratchpad. Corpus roots are arguments, never hardcoded.

Measured under the real edit, not reasoned: the suite fails exactly one test,
`test_raw_html_overblocks_are_still_high[relative-href-on-inactive-name]`,
which exists to force this doc update when an over-block closes. Recall holds
at 128/128 and all 6 documented gaps still hold.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:50:29 +02:00
commit e671edb96f
2 changed files with 232 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/LIMITATIONS.md
View file

@ -233,18 +233,34 @@ items; this is the full list, each with the mechanism.
`active:raw-html` in *52 of vendor-harvest's 98* to this over-reach. Measured by
re-running the gate with the URL-attribute branch narrowed to *external* targets
only, it frees **1 document in vendor-harvest, 1 in generated-notes, 3 in
reference-corpus** — the over-reach is real and it costs almost nothing, because
the documents carrying it almost always also carry a tag that is active by *name*,
and one raw-html finding per document is all it takes. What actually drives the
rate is the **name branch**: per wiki corpus `<a>` 298 occurrences, `<frame>` 94,
`<img>` 63; in reference-corpus `<base>` 113.
reference-corpus**, because the documents carrying it almost always also carry a
tag that is active by *name*, and one raw-html finding per document is all it
takes. What actually drives the rate is the **name branch**: per wiki corpus `<a>`
298 occurrences, `<frame>` 94, `<img>` 63; in reference-corpus `<base>` 113.
**Do not read "frees 3" as "this over-reach is cheap" — the over-reach classes
co-occur, and measuring them one at a time understates both.** In reference-corpus,
narrowing the URL-attribute branch alone frees 3 and taking `<base>` off the name
branch alone frees 13, but doing **both** frees **25** — because a document
carrying one class usually carries the other, so closing either alone leaves the
document blocked by its twin. Against a ceiling of 26 (raw-html detection switched
off entirely) the pair captures 96% of what this detector costs that population.
The same pair frees only 2 of a 38-document ceiling in vendor-harvest and 2 of 41
in generated-notes: the over-reach is nearly the whole raw-html cost in APIM policy
XML and nearly none of it in vendor documentation.
**And the over-reach is not only MDX.** In reference-corpus it lands on Azure APIM
policy XML — `<set-header>` 50, `<ip-filter>` 3, `<set-query-parameter>` 2 — which
shares nothing with the `<Card href="/…">` shape beyond having a URL attribute.
**Any fix here moves two surfaces, not one:** `neutralize` imports `is_active_tag`
from this module, so narrowing it also stops the opt-in *mutator* from defanging
the same tags. No test covers that half — the suite's `neutralize:raw-html` cases
use payloads that stay active under every narrowing considered — so the mutator
would change behaviour silently. Whoever closes this writes that test first.
- **`<base />` in Azure APIM policy XML grades HIGH on the name branch alone.**
APIM's `<base />` means "run the inherited policy" and carries no attributes; it
collides with HTML's `<base>`, whose entire affordance is its `href`. It appears in
**30 of reference-corpus's 389** documents. An attribute-less `<base>` has no
**25 of reference-corpus's 389** documents (`grep -rlE '<base\s*/?>'` — a loose
`grep '<base'` says 30 and is wrong: it also matches the literal `<base64_string>`
placeholder, which is not a tag this detector fires on). An attribute-less `<base>` has no
affordance in any renderer, and a `<base href="…">` would still be caught by the
URL-attribute branch — so the name-branch entry is doing no work here that the
attribute branch does not already do. This is a distinct over-reach class from the

210
docs/rawhtml-census.py Normal file
View file

@ -0,0 +1,210 @@
"""raw-HTML census — which branch of `is_active_tag` fires, and what narrowing it costs.
`docs/fp-sweep.py` answers *how often* the upload door costs a human. This answers
*why*, for the one detector that drives most of it, and *what a proposed narrowing
would actually buy* — end to end, as a change in `screen_output` disposition, not as
a count of regex hits.
WHY THIS EXISTS. `docs/LIMITATIONS.md` once attributed `active:raw-html` in 52 of
vendor-harvest's 98 non-WARN documents to the relative-URL-attribute over-reach.
Re-measured, that over-reach frees **one** document there. The claim was reasoning,
not measurement, and it stood in a published file for three releases. This script is
the measurement, so the next person changing that detector argues with numbers.
TWO METHOD TRAPS IT EXISTS TO AVOID:
- **Over-reach classes co-occur.** In one corpus, narrowing the URL-attribute branch
alone frees 3 documents and taking `<base>` off the name branch alone frees 13 —
but both together free 25. A document blocked by two classes is freed by neither
alone. Measure candidates you intend to ship *together*, never one at a time.
- **Two of the three published populations are LIVING corpora**, re-harvested by
their owning repo. A before/after split across two sessions mixes the narrowing's
effect with corpus drift. Every candidate here runs against the same corpus state
in one process, and `base` is re-measured rather than quoted from the doc.
The candidates are applied by replacing `active_content.is_active_tag` in-process.
That mirrors a real edit for the *scanner* path only: `neutralize` imports the
symbol by name, so a real edit would also change the mutator, which this script does
not simulate. See the raw-HTML bullets in `docs/LIMITATIONS.md`.
USAGE — corpus roots are arguments, never hardcoded; the corpora live in private
consumer repos and their paths must not reach a public mirror:
python docs/rawhtml-census.py LABEL=/path/to/corpus [LABEL=/path ...]
[--ext=.md,.txt] [--include=/subtree/]
"""
from __future__ import annotations
import re
import sys
from collections import Counter
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src"))
from llm_ingestion_guard import ( # noqa: E402
PRESET_USER_UPLOAD, Disposition, screen_output,
)
from llm_ingestion_guard import active_content as ac # noqa: E402
BENIGN = Disposition.WARN
# Attribute parser — only needed to read a URL attribute's VALUE, which
# `_URL_ATTR_RE` (a presence test) deliberately does not capture.
_ATTR_KV_RE = re.compile(
r"""\b(?P<k>[A-Za-z_:][\w:.\-]*)\s*=\s*(?P<v>"[^"]*"|'[^']*'|[^\s>]+)"""
)
_URL_ATTR_NAMES = frozenset({
"src", "href", "xlink:href", "srcset", "data", "poster", "formaction",
"action", "background", "cite", "codebase", "longdesc",
})
def has_external_url_attr(attrs: str) -> bool:
"""True if any URL-bearing attribute points at an attacker-reachable target."""
for m in _ATTR_KV_RE.finditer(attrs):
if m.group("k").lower() not in _URL_ATTR_NAMES:
continue
value = m.group("v")
if value[:1] in "\"'":
value = value[1:-1]
if ac._has_external_target(value.strip()):
return True
return False
def _variant(*, drop: frozenset[str] = frozenset(), external_only: bool = False):
"""Build an `is_active_tag` replacement: names minus ``drop``, URL branch gated."""
keep = frozenset(ac._ACTIVE_TAGS - drop)
def is_active_tag(name: str, attrs: str) -> bool:
if name.lower() in keep or ac._EVENT_ATTR_RE.search(attrs):
return True
if not ac._URL_ATTR_RE.search(attrs):
return False
return has_external_url_attr(attrs) if external_only else True
return is_active_tag
CANDIDATES = [
("base", _variant()),
# The URL-attribute branch requires an EXTERNAL target — the rule the markdown
# paths already apply. A relative `href` reaches no attacker-controlled host.
("A: url-attr external-only", _variant(external_only=True)),
# `<base>` off the NAME set. HTML's `<base>` has its whole affordance in `href`,
# which the URL-attribute branch still catches; APIM policy XML's `<base />` is
# attribute-less and has no affordance in any renderer.
("base-url: <base> needs a URL", _variant(drop=frozenset({"base"}))),
("A + base-url (both)",
_variant(drop=frozenset({"base"}), external_only=True)),
# The CEILING: no narrowing can free more than switching the detector off.
("NONE (ceiling)", lambda name, attrs: False),
]
def documents(root: Path, exts: tuple[str, ...], include: str) -> list[Path]:
files = []
for p in sorted(root.rglob("*")):
if not p.is_file() or p.suffix not in exts:
continue
rel = p.relative_to(root)
if any(part.startswith(".") for part in rel.parts):
continue
if include and include not in f"/{rel}":
continue
files.append(p)
return files
def masked_text(text: str) -> str:
"""Reproduce `scan_active_content`'s masking up to the raw-HTML pass.
Order matters: an image is not also a link, and a construct already consumed
by an earlier pass must not be re-counted as a tag.
"""
masked = text[: ac.MAX_SCAN_CHARS]
for pattern in (ac.MD_IMAGE_RE, ac.MD_LINK_RE, ac.MD_REFDEF_RE, ac.AUTOLINK_RE):
masked = pattern.sub(lambda m: " " * len(m.group(0)), masked)
return masked
def branch_census(texts: list[str]) -> tuple[Counter, Counter, Counter]:
"""Count active-tag occurrences by the branch that fires first."""
names: Counter[str] = Counter()
relative: Counter[str] = Counter()
external: Counter[str] = Counter()
for text in texts:
for m in ac.HTML_TAG_RE.finditer(masked_text(text)):
name, attrs = m.group("name"), m.group("attrs") or ""
if name.lower() in ac._ACTIVE_TAGS:
names[name.lower()] += 1
elif ac._EVENT_ATTR_RE.search(attrs):
names["(on*= handler)"] += 1
elif ac._URL_ATTR_RE.search(attrs):
(external if has_external_url_attr(attrs) else relative)[name] += 1
return names, relative, external
def main() -> None:
exts: tuple[str, ...] = (".md", ".txt")
include = ""
specs = []
for arg in sys.argv[1:]:
if arg.startswith("--ext="):
exts = tuple(e if e.startswith(".") else f".{e}"
for e in arg.split("=", 1)[1].split(","))
elif arg.startswith("--include="):
include = arg.split("=", 1)[1]
else:
specs.append(arg)
if not specs:
print(__doc__)
raise SystemExit(2)
original = ac.is_active_tag
try:
for spec in specs:
if "=" not in spec:
raise SystemExit(f"expected LABEL=PATH, got {spec!r}")
label, _, path = spec.partition("=")
root = Path(path).expanduser()
if not root.is_dir():
raise SystemExit(f"{label}: {root} is not a directory")
texts = [t for t in (p.read_text(encoding="utf-8", errors="replace")
for p in documents(root, exts, include)) if t.strip()]
n = len(texts)
print(f"\n## {label} — {n} documents", flush=True)
baseline = None
for name, fn in CANDIDATES:
ac.is_active_tag = fn
non_warn = sum(
1 for t in texts
if screen_output(t, PRESET_USER_UPLOAD).disposition is not BENIGN
)
if baseline is None:
baseline = non_warn
print(f" {name:>28}: {non_warn:4d} ({non_warn / n:5.1%})", flush=True)
else:
print(f" {name:>28}: {non_warn:4d} ({non_warn / n:5.1%})"
f" frees {baseline - non_warn}", flush=True)
ac.is_active_tag = original
names, relative, external = branch_census(texts)
print(f" name branch : {dict(names.most_common(8))}")
print(f" url-attr relative: {dict(relative.most_common(8))} <- A frees these")
print(f" url-attr external: {dict(external.most_common(8))} <- A keeps these")
finally:
ac.is_active_tag = original
print("\n---")
print("Candidates are measured TOGETHER as well as alone: over-reach classes\n"
"co-occur, and a document blocked by two of them is freed by neither\n"
"alone. Populations are never summed — each has its own denominator.")
if __name__ == "__main__":
main()