Kjell Tore Guttormsen
e6ca5ae5ee
Adopting `active:raw-html-link` was one id. Publishing it honestly was the whole of `active_tag_class` — one function, three branches, no way to state the split without the no-URL narrowing and the 0.6.0 external-target rule. On the old predicate a bare `</a>` is active by name, so a consumer implementing from the hybrid would emit the new label where the seed runtime emits nothing. The file is now two pins, stated as two: v0.3.4/0bf0729 everywhere except the raw-HTML classifier, v0.7.0/be9759b there. The drift between them was measured field by field against the imported module rather than assumed, after stripping inline-flag rendering and applying the file's own declared quote normalisation so a spelling difference could not masquerade as drift. Exactly one published field had moved, and not the one this release was about: `html.active_tags` carried the MUTATOR's 23-name set where the gate means the SCANNER's 22. Correct at the 0.3.4 pin, wrong from 0.6.0 on. Kept as `html.mutator_tags`. The sweep covered 93 cases, not the 6 obvious ones. The narrowing can silence an `active:` finding inside the `observed_out_of_scope` evidence of a LEXICON case, and that field is guarded by no test anywhere — stale entries there survive forever. One case moved: html-obfuscation__aria-label, whose `<a aria-label=…>` carries no URL attribute. Its fixture is deliberately not rewritten; the residue is true at the commit `measurement` pins, and rewriting one of 83 would leave two commits under a header naming one. Recorded, dated and pinned in the manifest. The strongest check is not the digest: the checker rebuilds the published classifier from the JSON alone, importing nothing from the runtime, and differential-tests it against `active_tag_class` over 42 probe tags. 0 disagreements. That is what licenses shipping a classifier as data. No `aliases.llm_security` published, on this file or on carriers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LTTaT4quwNPwBYVqmgAt8t
437 lines
39 KiB
JSON
437 lines
39 KiB
JSON
{
|
|
"version": "0.3.0",
|
|
"id": "active-content",
|
|
"description": "Active content in persisted text: constructs that fetch, render or execute on their own when a renderer touches them. The EchoLeak class (CVE-2025-32711). Report-only data: no construct here is defined as something to remove.",
|
|
"owasp": "LLM05",
|
|
"detector": "active_content",
|
|
"label_format": "active:{class}",
|
|
|
|
"pattern_id_space": {
|
|
"$comment": "Stable, commons-owned identifier for each construct, so that a conformance fixture can name an active-content finding. Unlike the injection lexicon's id space, nothing was constructed here: `label_format` and the `constructs` keys were already extracted verbatim from the seed runtime, and their concatenation IS what that runtime emits. This block states an id space that the file already had implicitly rather than introducing one.",
|
|
"field": "\"active:\" + the key of the entry in `constructs`",
|
|
"ids": [
|
|
"active:autolink",
|
|
"active:data-uri",
|
|
"active:markdown-image",
|
|
"active:markdown-link",
|
|
"active:raw-html",
|
|
"active:raw-html-link",
|
|
"active:reference-link"
|
|
],
|
|
"ids_added_in_0_3_0": {
|
|
"$comment": "One id was ADDED, and no id was changed or removed, which is what keeps this a minor bump. What did narrow is `active:raw-html`'s membership — see the top-level `behaviour_change` block, which a consumer upgrading from 0.2.0 must read.",
|
|
"added": [
|
|
"active:raw-html-link"
|
|
],
|
|
"adopted_verbatim": "Not named here. The seed runtime shipped the class string `raw-html-link` in its 0.7.0 release, and this file's `field` rule turns it into the id mechanically, exactly as it did for the original six. No naming decision was taken and neither runtime was asked for one — unlike `codepoints/carriers.json`, where the id had to be named because the runtimes disagreed. Only one runtime implements this table, so there is no second namer to consult."
|
|
},
|
|
"verified": {
|
|
"method": "The `constructs` keys were compared to the class strings the seed runtime passes to its finding emitter: the `_flag(cls, …)` call sites in active_content.py, whose emitter builds `label=f\"active:{cls}\"`. Set-equal, exact strings, no normalisation applied.",
|
|
"recheck_0_3_0": {
|
|
"$comment": "Re-run for the seventh id rather than extrapolated from the six. At v0.7.0 the raw-HTML pass no longer reaches its emitter through a `_flag` call site with a literal class argument — it collects into a two-key dict and loops `for cls in (\"raw-html\", \"raw-html-link\")`, so the comparison method above had to be re-established rather than repeated.",
|
|
"date": "2026-08-13",
|
|
"at_commit": "be9759b",
|
|
"source_version": "0.7.0",
|
|
"method": "The seven `constructs` keys were compared to the seven class strings reachable at the emitter: the five literal `_flag(...)` call sites (`markdown-image`, `markdown-link`, `reference-link`, `autolink`, `data-uri`) plus the two-member loop tuple that drives the raw-HTML pass, cross-checked against `active_tag_class`'s three possible non-None return values (`\"raw-html\"`, `\"raw-html-link\"`, and `\"raw-html\"` again from the event-handler and outside-the-name-set branches). Set-equal, exact strings.",
|
|
"measured_through_the_entry_point_too": "Not only read off the source. Each of the seven ids was produced by running the seed runtime's own payload for it through `scan_output(text, source=Source.OUTPUT)` at this commit; all seven labels came back verbatim. The payloads were extracted from `coverage.py`'s AST rather than retyped."
|
|
},
|
|
"line_numbers": {
|
|
"$comment": "Given per commit rather than once, because the two differ and a reader checking against this file's provenance pin would otherwise land on the wrong lines. The 23-line insert between the two commits shifts everything below it by 21.",
|
|
"de09711": "_flag call sites at 337, 341, 345, 349, 363, 369; emitter at 309 — where the comparison was run",
|
|
"0bf0729": "the same six call sites at 316, 320, 324, 328, 342, 348; emitter at 288 — the commit `provenance` pins",
|
|
"symbols": "`_flag` and the `label=f\"active:{cls}\"` assignment inside it — stable across both, unlike the line numbers"
|
|
},
|
|
"date": "2026-08-10",
|
|
"at_commit": "de09711",
|
|
"at_commit_note": "Named because that is where the comparison was actually run, which is NOT the commit this file's provenance pins (0bf0729). active_content.py did change over that range: `git diff 0bf0729..de09711` adds 23 lines. Measured, not assumed - the change is a self-safety length cap (a new `max_scan_chars` parameter, and an `active:oversize-input` finding when the input exceeds it). It adds no construct, removes none, and leaves every `_flag` call site and every construct pattern untouched, so the six-way comparison above holds at both commits. The drift is recorded rather than smoothed over: a reader checking this file against 0bf0729 will see a different active_content.py and is entitled to know why that does not invalidate the check."
|
|
},
|
|
"not_every_active_label": {
|
|
"$comment": "The seed runtime emits at least one `active:`-prefixed label that is NOT in this id space, and a consumer that matches by prefix rather than by membership will mis-scope it.",
|
|
"known_example": "active:oversize-input",
|
|
"what_it_is": "A self-safety flag (OWASP LLM10) raised when the input exceeds the runtime's scan cap, announcing that the tail went unread. It reports the detector's own coverage, not a construct found in the text, which is why it is not a member of this table.",
|
|
"rule": "Membership in this id space is by presence in `ids`, never by the `active:` prefix. A finding carrying that prefix but absent from `ids` does not belong to this data file, and is therefore OUT of scope for a conformance case scoped here - see spec/conformance-corpus.md section 4.",
|
|
"introduced": "Between 0bf0729 and de09711; absent at the commit this file's provenance pins."
|
|
},
|
|
"stability": "An id is a stable identifier. Changing one is a BREAKING change for any consumer matching on it, and requires a major bump of this file.",
|
|
"single_runtime": {
|
|
"$comment": "This id space is ratified by ONE runtime, not two, and that is a weaker status than the injection lexicon's - stated here rather than left to be discovered by a consumer who assumes the two files carry equal weight.",
|
|
"implemented_by": [
|
|
"llm_ingestion_guard"
|
|
],
|
|
"not_implemented_by": [
|
|
"llm_security"
|
|
],
|
|
"absence_evidence": "Measured in llm-security at b0de0ca, not assumed: a case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation - only prose in `knowledge/` and an entropy-scanner test that mentions the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, so the table was never present and later removed either.",
|
|
"not_a_coverage_claim": "That runtime having no active-content TABLE is not the same as it being blind to these constructs. Its entropy scanner reaches markdown-image URLs from a different angle (rule E18, host- and query-aware). What is absent is this table, which is all `not_implemented_by` asserts.",
|
|
"conformance_consequence": "Under spec/conformance-corpus.md section 1.1, a runtime that does not declare this file reports cases scoped to it as `not-applicable` - neither pass nor error."
|
|
},
|
|
"aliases": {
|
|
"$comment": "No `aliases` object is carried per construct, and the omission is deliberate rather than pending. The lexicon needs aliases because two runtimes name the same pattern differently; here the sole implementing runtime's emitted label IS the id, so an alias would be a copy of the key. A second runtime adopting this table registers its alias then - see the lexicon's aliases block for the shape."
|
|
}
|
|
},
|
|
|
|
"$comment": "Extracted verbatim, without behaviour change, from llm-ingestion-pipeline-security v0.3.4 (commit 0bf0729, 2026-08-03): src/llm_ingestion_guard/active_content.py (pattern table, pass order, URL-shape analysis, defang) and src/llm_ingestion_guard/calibration.py (per-construct severities and opacity floors). That module documents itself as the canonical home of this table with the Node port sharing its severities, which is why the guard is the seed source here rather than llm-security. As of 0.3.0 the raw-HTML CLASSIFIER alone is re-pinned forward to v0.7.0 — see `provenance.raw_html_branch_repinned`.",
|
|
|
|
"provenance": {
|
|
"source_repo": "llm-ingestion-pipeline-security",
|
|
"source_version": "0.3.4",
|
|
"source_commit": "0bf0729",
|
|
"source_date": "2026-08-03",
|
|
"source_files": [
|
|
"src/llm_ingestion_guard/active_content.py",
|
|
"src/llm_ingestion_guard/calibration.py"
|
|
],
|
|
"verified": true,
|
|
"raw_html_branch_repinned": {
|
|
"$comment": "Two pins, not one, because this file is a hybrid as of 0.3.0 and a single pin would misdescribe both halves. Everything above stays at 0.3.4 / 0bf0729; the raw-HTML classifier (`html`, `gates.active_tag`, the `raw-html` and `raw-html-link` construct entries and the raw-HTML part of `pass_order`) is pinned at v0.7.0 / be9759b.",
|
|
"source_version": "0.7.0",
|
|
"source_commit": "be9759b4b3bdb7e6b013fd8e3b4f4f0344ee5833",
|
|
"source_date": "2026-08-13",
|
|
"measured_from": "A scratch tree extracted with `git archive v0.7.0`, so the TAG was read rather than whatever the working copy happened to be on — its HEAD was two commits further along at the time (5667063).",
|
|
"why_the_whole_classifier_and_not_just_the_new_id": "Publishing `active:raw-html-link` on top of the 0.3.4 predicate would have shipped a contract that looks complete and is wrong. Under the 0.3.4 rule a bare `</a>` is active by NAME, and a consumer implementing from that hybrid would emit `active:raw-html-link` for it where the seed runtime at v0.7.0 emits nothing. `active_tag_class` is one function with three branches; the carrier split, the no-URL narrowing and the 0.6.0 external-target requirement cannot be stated separately.",
|
|
"drift_measured_not_assumed": {
|
|
"method": "Every regex, severity, tag list and numeric floor this file publishes was compared field-by-field against the v0.7.0 module by importing it and reading the compiled objects — after stripping Python's inline-flag rendering and applying this file's own declared `redundant-quote-escape` normalisation, so a spelling difference could not masquerade as drift.",
|
|
"date": "2026-08-13",
|
|
"result": "Exactly ONE published field had drifted: `html.active_tags`. Every construct pattern, every severity, both gate patterns, the whole of `url_shape`, `defang`, `evidence` and the opacity floors are byte-identical at 0.3.4 and at 0.7.0. That is why the re-pin is scoped to the classifier rather than applied to the file as a whole.",
|
|
"the_one_drift": "`html.active_tags` published the MUTATOR's name set (23 entries, including `base`). At 0.3.4 there was only one set and the name was unambiguous; 0.6.0 split the predicate in two, and the SCANNER's set — which is the one this file's `active_tag` gate means — drops `base` to 22. The published value was therefore naming the wrong set of the two from 0.6.0 onward. Corrected in 0.3.0, with the mutator set kept under `html.mutator_tags` rather than deleted, because a consumer that also defangs needs it and it is the value this file used to publish."
|
|
}
|
|
}
|
|
},
|
|
|
|
"dialect": {
|
|
"name": "ecmascript-named-groups",
|
|
"$comment": "Patterns are written in the ECMAScript / PCRE spelling of named groups. Flags are declared in a `flags` field and never inlined as `(?m)` / `(?i)`, because inline flags are not portable and not always position-independent.",
|
|
"flags": {
|
|
"i": "case-insensitive",
|
|
"m": "multiline: ^ and $ match at line boundaries"
|
|
},
|
|
"features_used": [
|
|
"named capture groups: (?<name>...)",
|
|
"negative lookbehind: (?<!...)",
|
|
"negative lookahead: (?!...)",
|
|
"lookahead assertion: (?=...)"
|
|
],
|
|
"translation_notes": [
|
|
"Python (`re`): rewrite `(?<name>` to `(?P<name>`. Nothing else changes; `re` supports the lookarounds used here.",
|
|
"Engines without lookbehind (RE2, Go `regexp`, most POSIX): `markdown-link`, `data-uri` and the dot-defang rule cannot be expressed as written. A consumer on such an engine MUST report the pattern as unsupported rather than skip it silently — a skipped pattern is an invisible false negative.",
|
|
"`\\w` appears once, in the `reference-link` scheme run. Python's `\\w` is Unicode-aware on `str`, ECMAScript's is ASCII-only. RFC 3986 restricts a URI scheme to ASCII, so no reachable input distinguishes them; the difference is recorded rather than normalised away, to keep the pattern byte-identical to its source."
|
|
]
|
|
},
|
|
|
|
"normalisations": [
|
|
{
|
|
"id": "redundant-quote-escape",
|
|
"affects": ["markdown-image", "markdown-link", "raw-html", "data-uri", "defang.url_in_text_pattern"],
|
|
"from": "\\\"",
|
|
"to": "\"",
|
|
"reason": "The source patterns are Python raw strings, so `\\\"` reaches the engine as backslash-quote — a redundant escape of a character that needs none. Python `re` and ECMAScript without flags accept it, but ECMAScript REJECTS it under the `u` and `v` flags (verified: `new RegExp('\\\\\"', 'u')` throws `Invalid escape`), which would lock every unicode-mode consumer out of five patterns.",
|
|
"equivalence_evidence": "Differential match-set comparison against the source objects over an adversarial 30-input corpus (bare quotes, escaped quotes, markdown titles containing quotes, quoted and unquoted HTML attributes, quote runs of length 1-5): 5 patterns x 30 inputs, zero differences in match offsets or captured text.",
|
|
"byte_identical_to_source": false
|
|
}
|
|
],
|
|
|
|
"redos_note": "Every character run in this table excludes the character that opens its own anchor (`[` for the markdown forms, `<` for the autolink and the raw tag). That exclusion is load-bearing, not cosmetic: without it, crafted input that repeats the anchor and never supplies the required closing literal makes every start position rescan the tail — quadratic. Measured on the source implementation before the exclusions: `<a:` repeated 100000 times cost 23.4s in the autolink pattern alone. Bounding the runs instead would also be linear but is the WRONG fix, because padding past a bound would be a one-line bypass of the very class this table detects. A consumer that rewrites a pattern here MUST re-establish this property.",
|
|
|
|
"pass_order": {
|
|
"$comment": "Constructs are scanned in this order, and every match is masked out of the working text before the next pass — replaced by spaces of the same length, so line structure and later offsets survive. Masking is what makes a construct count once under its most specific class: an image is not also a link, an autolink is not also raw HTML.",
|
|
"order": [
|
|
"markdown-image",
|
|
"markdown-link",
|
|
"reference-link",
|
|
"autolink",
|
|
"raw-html",
|
|
"data-uri"
|
|
],
|
|
"order_lists_passes_not_constructs": {
|
|
"$comment": "SIX passes, SEVEN constructs, and the difference is load-bearing as of 0.3.0. Through 0.2.0 the two lists were the same list and a reader could safely treat `order` as an enumeration of `constructs`. That stopped being true when the raw-HTML pass gained a second class. The entries here are unchanged strings — no consumer pinning a value in `order` breaks — but the RELATIONSHIP to `constructs` changed, so it is stated rather than left to be inferred from a length mismatch.",
|
|
"passes": 6,
|
|
"constructs": 7,
|
|
"classes_per_pass": {
|
|
"markdown-image": ["markdown-image"],
|
|
"markdown-link": ["markdown-link"],
|
|
"reference-link": ["reference-link"],
|
|
"autolink": ["autolink"],
|
|
"raw-html": ["raw-html", "raw-html-link"],
|
|
"data-uri": ["data-uri"]
|
|
},
|
|
"pass_name_collides_with_a_class_name": "The raw-HTML pass and one of its two classes are both spelled `raw-html`. That is the seed runtime's own spelling and it is reproduced rather than disambiguated, because renaming either would break a published string. Read `order` as passes and `constructs` as classes; `classes_per_pass` above is the only mapping between them."
|
|
},
|
|
"raw_html_pass_is_one_scan_two_buckets": {
|
|
"$comment": "The single most misreadable thing in this file, and the reason it is spelled out. `raw-html` and `raw-html-link` share ONE pattern and ONE scan of the text. A consumer that gives the new class its own pattern and its own pass would scan the tag pattern twice, mask twice, and double-count every document holding both a `<script>` and an `<a href>`.",
|
|
"rule": "Run the `raw-html` pattern ONCE. For each match, classify it with the `active_tag` gate, which returns `raw-html`, `raw-html-link`, or inert. Append the match to that class's bucket and mask it. Inert matches are NOT masked — they are left in the working text, exactly as the seed runtime leaves them.",
|
|
"then": "After the single scan, emit one finding per NON-EMPTY bucket, in the order `raw-html` then `raw-html-link`. A document holding both a `<script>` and an `<a href>` therefore produces TWO findings from one pass — the script must not be graded down to the anchor's severity, nor the anchor lost behind the script.",
|
|
"emission_order": ["raw-html", "raw-html-link"],
|
|
"one_finding_per_class_still_holds": "See `finding_semantics.one_finding_per_class`. It is per CLASS, not per pass, and was already worded that way — no change was needed there."
|
|
},
|
|
"mask_with": "space",
|
|
"mask_preserves_length": true
|
|
},
|
|
|
|
"constructs": {
|
|
"markdown-image": {
|
|
"syntax": "![alt](url \"title\")",
|
|
"pattern": "!\\[(?<alt>[^\\]\\[]*)\\]\\(\\s*(?<url>[^)\\s\\[]+)(?<title>(?:\\s+\"[^\"]*\")?)\\s*\\)",
|
|
"flags": [],
|
|
"url_group": "url",
|
|
"gate": "external_target_only",
|
|
"severity": "high",
|
|
"severity_reason": "zero-click: a renderer auto-fetches the URL with no user action",
|
|
"has_ordinary_form": true
|
|
},
|
|
"markdown-link": {
|
|
"syntax": "[text](url \"title\")",
|
|
"pattern": "(?<!!)\\[(?<text>[^\\]\\[]*)\\]\\(\\s*(?<url>[^)\\s\\[]+)(?<title>(?:\\s+\"[^\"]*\")?)\\s*\\)",
|
|
"flags": [],
|
|
"url_group": "url",
|
|
"gate": "external_target_only",
|
|
"severity": "medium",
|
|
"severity_reason": "click-required: the fetch needs a human action",
|
|
"has_ordinary_form": true
|
|
},
|
|
"reference-link": {
|
|
"syntax": "[label]: destination",
|
|
"pattern": "^(?<pre>[ ]{0,3}\\[[^\\]\\[]+\\]:\\s*)(?<url>[A-Za-z][\\w+.\\-]*:\\S+|//\\S+)",
|
|
"flags": ["m"],
|
|
"url_group": "url",
|
|
"gate": "always",
|
|
"gate_reason": "the pattern is absolute-only by construction: a footnote such as `[1]: some plain text` has no scheme and does not match",
|
|
"severity": "medium",
|
|
"has_ordinary_form": true
|
|
},
|
|
"autolink": {
|
|
"syntax": "<scheme:...>",
|
|
"pattern": "<(?<url>[A-Za-z][A-Za-z0-9+.\\-]*:[^>\\s<]+)>",
|
|
"flags": [],
|
|
"url_group": "url",
|
|
"gate": "always",
|
|
"gate_reason": "carries a scheme by construction",
|
|
"severity": "medium",
|
|
"has_ordinary_form": true
|
|
},
|
|
"raw-html": {
|
|
"syntax": "<tag attr=...>",
|
|
"pattern": "<(?<slash>/?)(?<name>[A-Za-z][A-Za-z0-9:-]*)(?<attrs>(?:[^>\"'<]|\"[^\"]*\"|'[^']*')*)>",
|
|
"flags": [],
|
|
"url_group": null,
|
|
"gate": "active_tag",
|
|
"gate_returns": "raw-html",
|
|
"gate_reason": "see `html`: the execute-class carrier — what a renderer acts on with no human action. An event handler, an inherently-executing or resource-loading element, or a name outside the active set carrying a URL attribute that points somewhere external.",
|
|
"severity": "high",
|
|
"has_ordinary_form": false,
|
|
"ordinary_exempt_reason": "active whatever the URL looks like — an event handler needs no URL at all",
|
|
"membership_narrowed_in_0_3_0": "`<a>` and `<area>` moved OUT of this class into `raw-html-link`, and tags whose whole affordance is a URL they do not carry (`</a>`, `<Frame>`, `<video />`, `<img alt=…>` with no `src`) left both classes. See the top-level `behaviour_change` block."
|
|
},
|
|
"raw-html-link": {
|
|
"syntax": "<a href=...>, <area href=...>",
|
|
"pattern_shared_with": "raw-html",
|
|
"pattern_shared_note": "This entry carries NO `pattern` key of its own, and the omission is the point. It is the same pattern, scanned once — see `pass_order.raw_html_pass_is_one_scan_two_buckets`. Copying the pattern here would read as a second pass and double-count.",
|
|
"flags": [],
|
|
"url_group": null,
|
|
"gate": "active_tag",
|
|
"gate_returns": "raw-html-link",
|
|
"gate_reason": "see `html`: the click-required carrier — a name in `link_tags` that carries a URL attribute and no event handler. Following it needs a human.",
|
|
"severity": "medium",
|
|
"severity_reason": "click-required: the fetch needs a human action — the same grade `markdown-link` has carried since the seed runtime's 0.3.1, and the reason this class exists. Before it, the same URL was MEDIUM as `[t](url)` and HIGH as `<a href=\"url\">`, an asymmetry produced by syntax rather than by affordance.",
|
|
"has_ordinary_form": false,
|
|
"ordinary_exempt_reason": "Same as `raw-html`: the raw-HTML classes are graded on carrier, never on URL shape. Applying the ordinary-URL test to raw tags was considered and rejected upstream — real vendor-doc image URLs are largely not ordinary, so it buys little, and it would add a third tier. A consumer that applies `url_shape` here would grade `<a href=\"https://example.com/guide\">` LOW, which no runtime does.",
|
|
"added_in": "0.3.0"
|
|
},
|
|
"data-uri": {
|
|
"syntax": "data:...",
|
|
"pattern": "(?<![A-Za-z0-9])data:[^\\s'\"<>)]+",
|
|
"flags": ["i"],
|
|
"url_group": 0,
|
|
"gate": "always",
|
|
"gate_reason": "the lookbehind is what keeps `metadata:` from matching; a `data:` URI is its own scheme",
|
|
"severity": "high",
|
|
"has_ordinary_form": false,
|
|
"ordinary_exempt_reason": "carries its own payload; the scheme is rejected by the ordinary-URL test outright"
|
|
}
|
|
},
|
|
|
|
"gates": {
|
|
"external_target_only": {
|
|
"$comment": "Fires only when the URL is absolute or protocol-relative. A relative in-document link resolves against the rendering host and has no attacker-reachable endpoint; flagging it would over-block cross-linked wiki/OKF content, whose core mechanism is relative links.",
|
|
"pattern": "^(?:[A-Za-z][A-Za-z0-9+.\\-]*:|//)",
|
|
"flags": []
|
|
},
|
|
"always": {
|
|
"$comment": "Every match is kept; the construct's own pattern already guarantees an external target."
|
|
},
|
|
"active_tag": {
|
|
"$comment": "See the `html` section. As of 0.3.0 this gate is a CLASSIFIER, not a predicate: it returns `raw-html`, `raw-html-link`, or inert. A consumer that keeps it boolean can express the narrowing but not the split, and would collapse both classes into `raw-html` — silently, and in the direction that reads as no change."
|
|
}
|
|
},
|
|
|
|
"html": {
|
|
"$comment": "The raw-HTML classifier. Given a tag's NAME and its ATTRIBUTE run, it returns which of the two carrier classes the tag belongs to, or that it is inert. Quoted attribute runs are consumed atomically in the tag pattern, because an attribute value may hold `>`.",
|
|
"algorithm": {
|
|
"$comment": "Ordered, and the order is load-bearing at step 1. Reproduced as steps rather than as prose because three of the four 0.6.0/0.7.0 corrections live in the ORDER and the branch structure, not in the tag lists.",
|
|
"steps": [
|
|
"1. If the attribute run matches `event_attr_pattern`, return `raw-html`. This test runs FIRST, before any name test, so an `<a onclick=…>` is graded as the execute-class carrier it is rather than downgraded with the anchors.",
|
|
"2. Compute `has_url_attr` = whether the attribute run matches `url_attr_pattern`. This is a PRESENCE test on the attribute name; it deliberately does not read the value.",
|
|
"3. If the name is in `active_tags`: if it is also in `url_affordance_tags` and `has_url_attr` is false, return inert. Otherwise return `raw-html-link` if the name is in `link_tags`, else `raw-html`.",
|
|
"4. Otherwise the name is outside the active set. Return `raw-html` if `has_url_attr` is true AND the URL attribute names an external target (see `url_attr_external_test`); else inert."
|
|
],
|
|
"step_1_order_note": "Swapping steps 1 and 3 would downgrade `<a onclick=\"steal()\">` from HIGH to MEDIUM — an execute-class construct graded as click-required.",
|
|
"step_3_narrowing_note": "A tag whose ENTIRE active affordance is the URL it names, carrying no URL attribute at all, names no target and no renderer can fetch or follow it. `<base />` made this argument first (upstream 0.6.0) and left the scanner's name set entirely; 0.7.0 applied the same argument to the rest of the name branch via `url_affordance_tags`. Every other member of `active_tags` does something a URL cannot describe — `<script>` executes its body, `<style>` restyles, `<form>` submits — and stays active with no attributes at all.",
|
|
"step_4_stays_high_note": "The URL-attribute branch deliberately does NOT participate in the split. A name outside the active set has unknown rendering and `href` is not the only URL attribute it may carry, so grading `<Card src=\"…\">` as click-required would be reasoning rather than measurement.",
|
|
"inert_matches_are_not_masked": "A tag classified inert is left in the working text rather than replaced with spaces, so a later pass can still see it. See `pass_order.raw_html_pass_is_one_scan_two_buckets`."
|
|
},
|
|
"active_tags": [
|
|
"a", "applet", "area", "audio", "button", "embed", "form", "frame",
|
|
"frameset", "iframe", "img", "input", "link", "math", "meta", "object",
|
|
"script", "source", "style", "svg", "track", "video"
|
|
],
|
|
"active_tags_note": "The SCANNER's name set: 22 entries. Through 0.2.0 this key published 23 — the extra one being `base` — which was correct at the 0.3.4 pin, when the runtime had a single name set, and stopped being correct at 0.6.0, when the scanner's and the mutator's sets diverged. `<base>`'s only affordance is its `href`, which step 4 still catches; the attribute-less `<base />` of Azure APIM policy XML has no affordance in any renderer. The full 23 are kept below as `mutator_tags`.",
|
|
"mutator_tags": [
|
|
"a", "applet", "area", "audio", "base", "button", "embed", "form", "frame",
|
|
"frameset", "iframe", "img", "input", "link", "math", "meta", "object",
|
|
"script", "source", "style", "svg", "track", "video"
|
|
],
|
|
"mutator_tags_note": "The set a DEFANGER should use — deliberately broader than the scanner's, and it applies to no finding in this file. Over-defanging costs nothing (the upstream mutator is opt-in and blocks no disposition), while under-defanging would hand a human a live construct. Recorded because it is the value this key published through 0.2.0 and because a consumer that both scans and defangs needs both sets; a consumer that only scans MUST use `active_tags`.",
|
|
"url_affordance_tags": [
|
|
"a", "area", "audio", "frame", "frameset", "img", "source", "track", "video"
|
|
],
|
|
"url_affordance_tags_note": "Subset of `active_tags` whose entire active affordance IS the URL they name. Used only by step 3. Added in 0.3.0.",
|
|
"link_tags": ["a", "area"],
|
|
"link_tags_note": "Subset of `url_affordance_tags` that is click-required: following one needs a human. These and only these produce `active:raw-html-link`. Added in 0.3.0.",
|
|
"event_attr_pattern": "\\bon[a-z]+\\s*=",
|
|
"event_attr_flags": ["i"],
|
|
"url_attr_pattern": "\\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)\\s*=",
|
|
"url_attr_flags": ["i"],
|
|
"url_attr_external_test": {
|
|
"$comment": "Step 4 only. Reads the VALUE of each URL-bearing attribute and asks whether any of them names an attacker-reachable target, using the same `external_target_only` gate the markdown paths use. Added in 0.3.0; before upstream 0.6.0 the mere presence of a URL attribute was enough, which graded an MDX `<Card href=\"/en/quickstart\">` — a doc-relative route — as HIGH.",
|
|
"value_pattern": "\\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)\\s*=\\s*(?<v>\"[^\"]*\"|'[^']*'|[^\\s>]+)",
|
|
"value_flags": ["i"],
|
|
"strip_quotes": "If the captured value begins with `\"` or `'`, drop the first and last character.",
|
|
"candidate_split_pattern": "[,\\s]+",
|
|
"candidate_split_note": "`srcset` holds a comma-separated candidate list, so an attribute value is not always one URL. Splitting means a relative first candidate cannot mask an external one behind it. Empty candidates are discarded.",
|
|
"rule": "True if ANY candidate of ANY URL-bearing attribute matches the `external_target_only` gate pattern.",
|
|
"fail_secure": "If `url_attr_pattern` matched but `value_pattern` matched nothing — a gap between the presence test and the value test — the result is TRUE, not false. A value this test cannot read must over-block rather than under-block."
|
|
},
|
|
"tag_name_match": "case-insensitive"
|
|
},
|
|
|
|
"url_shape": {
|
|
"$comment": "The exfiltration primitive is not a construct type — it is a URL that moves bytes to a host the attacker controls. A URL that merely NAMES a remote document carries nothing outward and is graded `ordinary_severity` instead of the construct's own severity. Without this split, an ordinary document holding one remote image graded the same as ``, which fail-secured ordinary documents (measured on the source implementation v0.3.0).",
|
|
"ordinary_severity": "low",
|
|
"ordinary_scheme_pattern": "^(?:https?://|//)",
|
|
"ordinary_scheme_flags": ["i"],
|
|
"ordinary_requires": [
|
|
"an http(s) or protocol-relative scheme",
|
|
"no query component",
|
|
"no userinfo (username or password)",
|
|
"no percent-escape in host or path",
|
|
"no opaque host label or path segment"
|
|
],
|
|
"fragment_excluded": true,
|
|
"fragment_excluded_reason": "A fragment is never sent to the server, so it cannot carry data to the host a renderer auto-fetches. `.../overview#prerequisites` is the most common shape in real documentation.",
|
|
"host_case_preserved": true,
|
|
"host_case_preserved_reason": "Compare on the raw authority, not a lowercased hostname: lowercasing destroys the mixed case a base64 payload smuggled into a subdomain depends on.",
|
|
"malformed_authority": "never_ordinary",
|
|
"token_separator_pattern": "[/._\\-~+,;:=&$!*'()]+",
|
|
"token_separator_note": "Host labels and path segments are split on the separators that delimit a NAME. A token that survives the split and still looks like a blob is carried data.",
|
|
"opacity": {
|
|
"$comment": "Three signals, cheapest first. Shared with the entropy detector rather than reinvented, and a backstop rather than the main line of defence: a literal credential in a URL is caught by the secret-egress patterns in the same pass regardless of the severity assigned here.",
|
|
"tests": [
|
|
{
|
|
"id": "base64-decodes-to-printable",
|
|
"rule": "the token base64-decodes to printable text",
|
|
"min_length": null
|
|
},
|
|
{
|
|
"id": "hex-blob",
|
|
"rule": "the token is a hex blob at or above the URL-token hex floor",
|
|
"min_length": 32,
|
|
"min_length_note": "Deliberately lower than the prose hex floor (64): in prose a 32-char hex run is usually a checksum, but as a whole path segment or host label it is an opaque id — the md5/uuid length an exfil path uses."
|
|
},
|
|
{
|
|
"id": "shannon-entropy",
|
|
"rule": "length-paired Shannon entropy at or above the floor",
|
|
"min_length": 24,
|
|
"min_entropy_bits_per_char": 4.4,
|
|
"calibration_note": "Measured 2026-07-25 against real documentation URLs (Microsoft Learn, Wikipedia, GitHub raw, regjeringen.no): the worst legitimate token scored H=4.08 at length 44, while base64/hex payload segments scored 4.36-4.54 and random base62 averages 4.23 at length 24. The floor sits above every measured legitimate token with margin."
|
|
}
|
|
],
|
|
"any_test_sufficient": true
|
|
},
|
|
"accepted_false_positive": "A percent-escape counts as carrying, which grades a legitimate `%20` in a path as data-carrying. Accepted deliberately: obfuscated encoding is a core exfiltration primitive, and the ambiguous case belongs on the review side."
|
|
},
|
|
|
|
"defang": {
|
|
"$comment": "Evidence hygiene: a finding's evidence carries the DEFANGED URL, so a report is safe to log and render without recreating the affordance it flagged. Idempotent — applying it twice changes nothing.",
|
|
"danger_scheme_pattern": "^(javascript|data|vbscript|file|blob)(?=:)",
|
|
"danger_scheme_flags": ["i"],
|
|
"danger_scheme_action": "replace the scheme's trailing colon with `[:]`",
|
|
"scheme_substitutions": [
|
|
{ "pattern": "^https", "flags": ["i"], "replacement": "hxxps" },
|
|
{ "pattern": "^http", "flags": ["i"], "replacement": "hxxp" },
|
|
{ "pattern": "^ftp", "flags": ["i"], "replacement": "fxp" }
|
|
],
|
|
"scheme_substitutions_note": "Ordered, first match wins, and applied only when the danger-scheme rule did not fire.",
|
|
"dot_pattern": "(?<!\\[)\\.(?!\\])",
|
|
"dot_replacement": "[.]",
|
|
"dot_note": "The lookarounds are what make the rule idempotent: a `.` already inside `[.]` is never touched.",
|
|
"url_in_text_pattern": "[A-Za-z][A-Za-z0-9+.\\-]{0,63}://[^\\s'\"<>]+",
|
|
"url_in_text_flags": [],
|
|
"url_in_text_note": "Used to defang bare URLs inside an already-flagged raw HTML tag. The scheme run is bounded here — unlike the detector patterns, where bounding would be a bypass — because this is a defanger applied inside a tag that has already been flagged, and a 64+ character scheme is not resolvable by any renderer. Measured 0.185s at the 1000000-char cap; unbounded it cost 12.99s on 100000 characters."
|
|
},
|
|
|
|
"evidence": {
|
|
"redact_show_start": 16,
|
|
"redact_show_end": 6,
|
|
"redact_applies_when": "length > show_start + show_end + 3",
|
|
"redact_joiner": "..."
|
|
},
|
|
|
|
"finding_semantics": {
|
|
"one_finding_per_class": true,
|
|
"count": "number of kept matches in that class",
|
|
"graded_by": "worst member",
|
|
"graded_by_note": "A class collapses to one finding, so an exfil-shaped URL hiding behind an ordinary one must set BOTH the severity and the evidence — otherwise the report shows an innocent URL next to a HIGH verdict.",
|
|
"evidence_selection": "the first data-carrying hit if any, otherwise the first hit",
|
|
"mutates_input": false,
|
|
"renders_disposition": false
|
|
},
|
|
|
|
"behaviour_change": {
|
|
"$comment": "0.3.0 is a MINOR bump under this repository's rule — one id added, none changed or removed, no key removed, no structural rearrangement — but it is NOT behaviour-preserving, and a consumer upgrading from 0.2.0 will see different findings on the same document. Recorded explicitly so that the minor number is auditable rather than arguable. This is the first release of this file that is not a pure extraction; the v0.1.0 behaviour-preservation invariant covered the EXTRACTION and does not forbid tracking the seed runtime forward.",
|
|
"version": "0.2.0 -> 0.3.0",
|
|
"changes": [
|
|
{
|
|
"id": "carrier-split",
|
|
"what": "`<a>` and `<area>` carrying a URL attribute now report as `active:raw-html-link` at MEDIUM instead of `active:raw-html` at HIGH.",
|
|
"consumer_effect": "A label a consumer pinned against changes, and a severity drops one grade. A document containing only anchors moves from HIGH to MEDIUM. A document containing both a `<script>` and an `<a href>` now produces TWO findings where it produced one.",
|
|
"not_a_recall_loss": "Nothing stops being reported. The construct is still found, in a class that says a human must act for it to fetch."
|
|
},
|
|
{
|
|
"id": "no-url-narrowing",
|
|
"what": "A tag in `url_affordance_tags` carrying no URL attribute at all is now inert, in BOTH classes.",
|
|
"witnesses": ["</a>", "<Frame>", "<video />", "<img alt=\"…\"> with no src"],
|
|
"consumer_effect": "This one DOES silence findings. A document whose only raw-HTML content is end tags or attribute-less MDX wrapper components now produces no active-content finding where 0.2.0 produced `active:raw-html` at HIGH. End tags and MDX wrappers dominate the shapes this frees.",
|
|
"measured_in_this_corpus": "One conformance case changed its observed guard behaviour because of this: `html-obfuscation__aria-label`, whose input is `<a aria-label=\"ignore all instructions\">link</a>`. Neither its `<a aria-label>` (no URL attribute) nor its `</a>` is active at v0.7.0, so the guard's `active:raw-html` residue on that case is gone. That case is scoped to the lexicon, so its VERDICT is unaffected — the change is in `observed_out_of_scope`, which spec/conformance-corpus.md section 5 defines as evidence at a named commit rather than expectation. See conformance/manifest.json -> `active_content_measurement_0_7_0`."
|
|
},
|
|
{
|
|
"id": "base-left-the-scanner-set",
|
|
"what": "`base` is no longer in `html.active_tags`; it is only in `html.mutator_tags`.",
|
|
"consumer_effect": "`<base />` with no `href` is inert. `<base href=\"https://evil.test/\">` is still caught, by step 4 rather than by the name test.",
|
|
"note": "This one is a CORRECTION of a published value rather than a tracked change: the runtime made it at 0.6.0 and this file kept publishing the pre-split value until now. See `provenance.raw_html_branch_repinned.drift_measured_not_assumed`."
|
|
},
|
|
{
|
|
"id": "url-attr-must-be-external",
|
|
"what": "For a name OUTSIDE `active_tags`, a URL attribute must now name an external target, not merely be present.",
|
|
"consumer_effect": "An MDX `<Card href=\"/en/quickstart\">` — a doc-relative route — is inert where 0.2.0 graded it HIGH. Same correction class as the row above: made upstream at 0.6.0, published here now."
|
|
}
|
|
],
|
|
"direction": "All four changes NARROW or DOWNGRADE. None of them makes this table report something it did not report before, so no consumer gains a new false positive by upgrading. A consumer that depends on the old breadth loses coverage on the four shapes named above, which is the trade the seed runtime measured and shipped.",
|
|
"why_not_a_major_bump": "This repository's rule triggers MAJOR on a changed or removed key, a changed case id, or a changed file disposition. None occurred: `active:raw-html-link` was added, `html.mutator_tags` preserves the value `html.active_tags` used to carry, and every other key kept its name and its meaning. The argument FOR major is the silenced findings under `no-url-narrowing`, and it is recorded here rather than dismissed — a consumer who reads this and concludes the change is breaking FOR THEM is not misreading the data."
|
|
},
|
|
|
|
"known_gaps": [
|
|
"Pure beaconing: a fetch whose only signal is that it happened. No URL shape distinguishes it from an ordinary remote image.",
|
|
"Short opaque segments: a carried value below the opacity length floors grades as ordinary.",
|
|
"Hex-wrapped payloads below the URL-token hex floor.",
|
|
"Anchors are graded on carrier, never on URL shape: `<a href=\"https://example.com/guide\">` and `<a href=\"https://evil.test/x?d=SECRET\">` both grade MEDIUM. The upstream runtime considered applying the ordinary-URL test to raw tags and rejected it — see `constructs.raw-html-link.ordinary_exempt_reason`."
|
|
]
|
|
}
|