299 lines
20 KiB
JSON
299 lines
20 KiB
JSON
{
|
|
"version": "0.2.0",
|
|
"id": "active-content",
|
|
"description": "Active content in persisted text: constructs that fetch, render or execute on their own when a renderer touches them. The EchoLeak class (CVE-2025-32711). Report-only data: no construct here is defined as something to remove.",
|
|
"owasp": "LLM05",
|
|
"detector": "active_content",
|
|
"label_format": "active:{class}",
|
|
|
|
"pattern_id_space": {
|
|
"$comment": "Stable, commons-owned identifier for each construct, so that a conformance fixture can name an active-content finding. Unlike the injection lexicon's id space, nothing was constructed here: `label_format` and the `constructs` keys were already extracted verbatim from the seed runtime, and their concatenation IS what that runtime emits. This block states an id space that the file already had implicitly rather than introducing one.",
|
|
"field": "\"active:\" + the key of the entry in `constructs`",
|
|
"ids": [
|
|
"active:autolink",
|
|
"active:data-uri",
|
|
"active:markdown-image",
|
|
"active:markdown-link",
|
|
"active:raw-html",
|
|
"active:reference-link"
|
|
],
|
|
"verified": {
|
|
"method": "The six `constructs` keys were compared to the six class strings the seed runtime passes to its finding emitter: the `_flag(cls, …)` call sites in active_content.py, whose emitter builds `label=f\"active:{cls}\"`. Set-equal, exact strings, no normalisation applied.",
|
|
"line_numbers": {
|
|
"$comment": "Given per commit rather than once, because the two differ and a reader checking against this file's provenance pin would otherwise land on the wrong lines. The 23-line insert between the two commits shifts everything below it by 21.",
|
|
"de09711": "_flag call sites at 337, 341, 345, 349, 363, 369; emitter at 309 — where the comparison was run",
|
|
"0bf0729": "the same six call sites at 316, 320, 324, 328, 342, 348; emitter at 288 — the commit `provenance` pins",
|
|
"symbols": "`_flag` and the `label=f\"active:{cls}\"` assignment inside it — stable across both, unlike the line numbers"
|
|
},
|
|
"date": "2026-08-10",
|
|
"at_commit": "de09711",
|
|
"at_commit_note": "Named because that is where the comparison was actually run, which is NOT the commit this file's provenance pins (0bf0729). active_content.py did change over that range: `git diff 0bf0729..de09711` adds 23 lines. Measured, not assumed - the change is a self-safety length cap (a new `max_scan_chars` parameter, and an `active:oversize-input` finding when the input exceeds it). It adds no construct, removes none, and leaves every `_flag` call site and every construct pattern untouched, so the six-way comparison above holds at both commits. The drift is recorded rather than smoothed over: a reader checking this file against 0bf0729 will see a different active_content.py and is entitled to know why that does not invalidate the check."
|
|
},
|
|
"not_every_active_label": {
|
|
"$comment": "The seed runtime emits at least one `active:`-prefixed label that is NOT in this id space, and a consumer that matches by prefix rather than by membership will mis-scope it.",
|
|
"known_example": "active:oversize-input",
|
|
"what_it_is": "A self-safety flag (OWASP LLM10) raised when the input exceeds the runtime's scan cap, announcing that the tail went unread. It reports the detector's own coverage, not a construct found in the text, which is why it is not a member of this table.",
|
|
"rule": "Membership in this id space is by presence in `ids`, never by the `active:` prefix. A finding carrying that prefix but absent from `ids` does not belong to this data file, and is therefore OUT of scope for a conformance case scoped here - see spec/conformance-corpus.md section 4.",
|
|
"introduced": "Between 0bf0729 and de09711; absent at the commit this file's provenance pins."
|
|
},
|
|
"stability": "An id is a stable identifier. Changing one is a BREAKING change for any consumer matching on it, and requires a major bump of this file.",
|
|
"single_runtime": {
|
|
"$comment": "This id space is ratified by ONE runtime, not two, and that is a weaker status than the injection lexicon's - stated here rather than left to be discovered by a consumer who assumes the two files carry equal weight.",
|
|
"implemented_by": [
|
|
"llm_ingestion_guard"
|
|
],
|
|
"not_implemented_by": [
|
|
"llm_security"
|
|
],
|
|
"absence_evidence": "Measured in llm-security at b0de0ca, not assumed: a case-insensitive tree-wide search for an active-content detector, class table or emitted `active:` label returns no implementation - only prose in `knowledge/` and an entropy-scanner test that mentions the markdown-image construct. `git log -S 'active_content'` and `git log -S 'active:markdown'` over `--all` return zero commits, so the table was never present and later removed either.",
|
|
"not_a_coverage_claim": "That runtime having no active-content TABLE is not the same as it being blind to these constructs. Its entropy scanner reaches markdown-image URLs from a different angle (rule E18, host- and query-aware). What is absent is this table, which is all `not_implemented_by` asserts.",
|
|
"conformance_consequence": "Under spec/conformance-corpus.md section 1.1, a runtime that does not declare this file reports cases scoped to it as `not-applicable` - neither pass nor error."
|
|
},
|
|
"aliases": {
|
|
"$comment": "No `aliases` object is carried per construct, and the omission is deliberate rather than pending. The lexicon needs aliases because two runtimes name the same pattern differently; here the sole implementing runtime's emitted label IS the id, so an alias would be a copy of the key. A second runtime adopting this table registers its alias then - see the lexicon's aliases block for the shape."
|
|
}
|
|
},
|
|
|
|
"$comment": "Extracted verbatim, without behaviour change, from llm-ingestion-pipeline-security v0.3.4 (commit 0bf0729, 2026-08-03): src/llm_ingestion_guard/active_content.py (pattern table, pass order, URL-shape analysis, defang) and src/llm_ingestion_guard/calibration.py (per-construct severities and opacity floors). That module documents itself as the canonical home of this table with the Node port sharing its severities, which is why the guard is the seed source here rather than llm-security.",
|
|
|
|
"provenance": {
|
|
"source_repo": "llm-ingestion-pipeline-security",
|
|
"source_version": "0.3.4",
|
|
"source_commit": "0bf0729",
|
|
"source_date": "2026-08-03",
|
|
"source_files": [
|
|
"src/llm_ingestion_guard/active_content.py",
|
|
"src/llm_ingestion_guard/calibration.py"
|
|
],
|
|
"verified": true
|
|
},
|
|
|
|
"dialect": {
|
|
"name": "ecmascript-named-groups",
|
|
"$comment": "Patterns are written in the ECMAScript / PCRE spelling of named groups. Flags are declared in a `flags` field and never inlined as `(?m)` / `(?i)`, because inline flags are not portable and not always position-independent.",
|
|
"flags": {
|
|
"i": "case-insensitive",
|
|
"m": "multiline: ^ and $ match at line boundaries"
|
|
},
|
|
"features_used": [
|
|
"named capture groups: (?<name>...)",
|
|
"negative lookbehind: (?<!...)",
|
|
"negative lookahead: (?!...)",
|
|
"lookahead assertion: (?=...)"
|
|
],
|
|
"translation_notes": [
|
|
"Python (`re`): rewrite `(?<name>` to `(?P<name>`. Nothing else changes; `re` supports the lookarounds used here.",
|
|
"Engines without lookbehind (RE2, Go `regexp`, most POSIX): `markdown-link`, `data-uri` and the dot-defang rule cannot be expressed as written. A consumer on such an engine MUST report the pattern as unsupported rather than skip it silently — a skipped pattern is an invisible false negative.",
|
|
"`\\w` appears once, in the `reference-link` scheme run. Python's `\\w` is Unicode-aware on `str`, ECMAScript's is ASCII-only. RFC 3986 restricts a URI scheme to ASCII, so no reachable input distinguishes them; the difference is recorded rather than normalised away, to keep the pattern byte-identical to its source."
|
|
]
|
|
},
|
|
|
|
"normalisations": [
|
|
{
|
|
"id": "redundant-quote-escape",
|
|
"affects": ["markdown-image", "markdown-link", "raw-html", "data-uri", "defang.url_in_text_pattern"],
|
|
"from": "\\\"",
|
|
"to": "\"",
|
|
"reason": "The source patterns are Python raw strings, so `\\\"` reaches the engine as backslash-quote — a redundant escape of a character that needs none. Python `re` and ECMAScript without flags accept it, but ECMAScript REJECTS it under the `u` and `v` flags (verified: `new RegExp('\\\\\"', 'u')` throws `Invalid escape`), which would lock every unicode-mode consumer out of five patterns.",
|
|
"equivalence_evidence": "Differential match-set comparison against the source objects over an adversarial 30-input corpus (bare quotes, escaped quotes, markdown titles containing quotes, quoted and unquoted HTML attributes, quote runs of length 1-5): 5 patterns x 30 inputs, zero differences in match offsets or captured text.",
|
|
"byte_identical_to_source": false
|
|
}
|
|
],
|
|
|
|
"redos_note": "Every character run in this table excludes the character that opens its own anchor (`[` for the markdown forms, `<` for the autolink and the raw tag). That exclusion is load-bearing, not cosmetic: without it, crafted input that repeats the anchor and never supplies the required closing literal makes every start position rescan the tail — quadratic. Measured on the source implementation before the exclusions: `<a:` repeated 100000 times cost 23.4s in the autolink pattern alone. Bounding the runs instead would also be linear but is the WRONG fix, because padding past a bound would be a one-line bypass of the very class this table detects. A consumer that rewrites a pattern here MUST re-establish this property.",
|
|
|
|
"pass_order": {
|
|
"$comment": "Constructs are scanned in this order, and every match is masked out of the working text before the next pass — replaced by spaces of the same length, so line structure and later offsets survive. Masking is what makes a construct count once under its most specific class: an image is not also a link, an autolink is not also raw HTML.",
|
|
"order": [
|
|
"markdown-image",
|
|
"markdown-link",
|
|
"reference-link",
|
|
"autolink",
|
|
"raw-html",
|
|
"data-uri"
|
|
],
|
|
"mask_with": "space",
|
|
"mask_preserves_length": true
|
|
},
|
|
|
|
"constructs": {
|
|
"markdown-image": {
|
|
"syntax": "![alt](url \"title\")",
|
|
"pattern": "!\\[(?<alt>[^\\]\\[]*)\\]\\(\\s*(?<url>[^)\\s\\[]+)(?<title>(?:\\s+\"[^\"]*\")?)\\s*\\)",
|
|
"flags": [],
|
|
"url_group": "url",
|
|
"gate": "external_target_only",
|
|
"severity": "high",
|
|
"severity_reason": "zero-click: a renderer auto-fetches the URL with no user action",
|
|
"has_ordinary_form": true
|
|
},
|
|
"markdown-link": {
|
|
"syntax": "[text](url \"title\")",
|
|
"pattern": "(?<!!)\\[(?<text>[^\\]\\[]*)\\]\\(\\s*(?<url>[^)\\s\\[]+)(?<title>(?:\\s+\"[^\"]*\")?)\\s*\\)",
|
|
"flags": [],
|
|
"url_group": "url",
|
|
"gate": "external_target_only",
|
|
"severity": "medium",
|
|
"severity_reason": "click-required: the fetch needs a human action",
|
|
"has_ordinary_form": true
|
|
},
|
|
"reference-link": {
|
|
"syntax": "[label]: destination",
|
|
"pattern": "^(?<pre>[ ]{0,3}\\[[^\\]\\[]+\\]:\\s*)(?<url>[A-Za-z][\\w+.\\-]*:\\S+|//\\S+)",
|
|
"flags": ["m"],
|
|
"url_group": "url",
|
|
"gate": "always",
|
|
"gate_reason": "the pattern is absolute-only by construction: a footnote such as `[1]: some plain text` has no scheme and does not match",
|
|
"severity": "medium",
|
|
"has_ordinary_form": true
|
|
},
|
|
"autolink": {
|
|
"syntax": "<scheme:...>",
|
|
"pattern": "<(?<url>[A-Za-z][A-Za-z0-9+.\\-]*:[^>\\s<]+)>",
|
|
"flags": [],
|
|
"url_group": "url",
|
|
"gate": "always",
|
|
"gate_reason": "carries a scheme by construction",
|
|
"severity": "medium",
|
|
"has_ordinary_form": true
|
|
},
|
|
"raw-html": {
|
|
"syntax": "<tag attr=...>",
|
|
"pattern": "<(?<slash>/?)(?<name>[A-Za-z][A-Za-z0-9:-]*)(?<attrs>(?:[^>\"'<]|\"[^\"]*\"|'[^']*')*)>",
|
|
"flags": [],
|
|
"url_group": null,
|
|
"gate": "active_tag",
|
|
"gate_reason": "see `html`: an inherently-executing element, an event handler, or a URL-bearing attribute",
|
|
"severity": "high",
|
|
"has_ordinary_form": false,
|
|
"ordinary_exempt_reason": "active whatever the URL looks like — an event handler needs no URL at all"
|
|
},
|
|
"data-uri": {
|
|
"syntax": "data:...",
|
|
"pattern": "(?<![A-Za-z0-9])data:[^\\s'\"<>)]+",
|
|
"flags": ["i"],
|
|
"url_group": 0,
|
|
"gate": "always",
|
|
"gate_reason": "the lookbehind is what keeps `metadata:` from matching; a `data:` URI is its own scheme",
|
|
"severity": "high",
|
|
"has_ordinary_form": false,
|
|
"ordinary_exempt_reason": "carries its own payload; the scheme is rejected by the ordinary-URL test outright"
|
|
}
|
|
},
|
|
|
|
"gates": {
|
|
"external_target_only": {
|
|
"$comment": "Fires only when the URL is absolute or protocol-relative. A relative in-document link resolves against the rendering host and has no attacker-reachable endpoint; flagging it would over-block cross-linked wiki/OKF content, whose core mechanism is relative links.",
|
|
"pattern": "^(?:[A-Za-z][A-Za-z0-9+.\\-]*:|//)",
|
|
"flags": []
|
|
},
|
|
"always": {
|
|
"$comment": "Every match is kept; the construct's own pattern already guarantees an external target."
|
|
},
|
|
"active_tag": {
|
|
"$comment": "See the `html` section."
|
|
}
|
|
},
|
|
|
|
"html": {
|
|
"$comment": "A raw HTML tag is active if it is an inherently-executing or resource-loading element, carries an event handler, or carries a URL-bearing attribute. Quoted attribute runs are consumed atomically in the tag pattern, because an attribute value may hold `>`.",
|
|
"active_tags": [
|
|
"a", "applet", "area", "audio", "base", "button", "embed", "form", "frame",
|
|
"frameset", "iframe", "img", "input", "link", "math", "meta", "object",
|
|
"script", "source", "style", "svg", "track", "video"
|
|
],
|
|
"event_attr_pattern": "\\bon[a-z]+\\s*=",
|
|
"event_attr_flags": ["i"],
|
|
"url_attr_pattern": "\\b(?:src|href|xlink:href|srcset|data|poster|formaction|action|background|cite|codebase|longdesc)\\s*=",
|
|
"url_attr_flags": ["i"],
|
|
"tag_name_match": "case-insensitive"
|
|
},
|
|
|
|
"url_shape": {
|
|
"$comment": "The exfiltration primitive is not a construct type — it is a URL that moves bytes to a host the attacker controls. A URL that merely NAMES a remote document carries nothing outward and is graded `ordinary_severity` instead of the construct's own severity. Without this split, an ordinary document holding one remote image graded the same as ``, which fail-secured ordinary documents (measured on the source implementation v0.3.0).",
|
|
"ordinary_severity": "low",
|
|
"ordinary_scheme_pattern": "^(?:https?://|//)",
|
|
"ordinary_scheme_flags": ["i"],
|
|
"ordinary_requires": [
|
|
"an http(s) or protocol-relative scheme",
|
|
"no query component",
|
|
"no userinfo (username or password)",
|
|
"no percent-escape in host or path",
|
|
"no opaque host label or path segment"
|
|
],
|
|
"fragment_excluded": true,
|
|
"fragment_excluded_reason": "A fragment is never sent to the server, so it cannot carry data to the host a renderer auto-fetches. `.../overview#prerequisites` is the most common shape in real documentation.",
|
|
"host_case_preserved": true,
|
|
"host_case_preserved_reason": "Compare on the raw authority, not a lowercased hostname: lowercasing destroys the mixed case a base64 payload smuggled into a subdomain depends on.",
|
|
"malformed_authority": "never_ordinary",
|
|
"token_separator_pattern": "[/._\\-~+,;:=&$!*'()]+",
|
|
"token_separator_note": "Host labels and path segments are split on the separators that delimit a NAME. A token that survives the split and still looks like a blob is carried data.",
|
|
"opacity": {
|
|
"$comment": "Three signals, cheapest first. Shared with the entropy detector rather than reinvented, and a backstop rather than the main line of defence: a literal credential in a URL is caught by the secret-egress patterns in the same pass regardless of the severity assigned here.",
|
|
"tests": [
|
|
{
|
|
"id": "base64-decodes-to-printable",
|
|
"rule": "the token base64-decodes to printable text",
|
|
"min_length": null
|
|
},
|
|
{
|
|
"id": "hex-blob",
|
|
"rule": "the token is a hex blob at or above the URL-token hex floor",
|
|
"min_length": 32,
|
|
"min_length_note": "Deliberately lower than the prose hex floor (64): in prose a 32-char hex run is usually a checksum, but as a whole path segment or host label it is an opaque id — the md5/uuid length an exfil path uses."
|
|
},
|
|
{
|
|
"id": "shannon-entropy",
|
|
"rule": "length-paired Shannon entropy at or above the floor",
|
|
"min_length": 24,
|
|
"min_entropy_bits_per_char": 4.4,
|
|
"calibration_note": "Measured 2026-07-25 against real documentation URLs (Microsoft Learn, Wikipedia, GitHub raw, regjeringen.no): the worst legitimate token scored H=4.08 at length 44, while base64/hex payload segments scored 4.36-4.54 and random base62 averages 4.23 at length 24. The floor sits above every measured legitimate token with margin."
|
|
}
|
|
],
|
|
"any_test_sufficient": true
|
|
},
|
|
"accepted_false_positive": "A percent-escape counts as carrying, which grades a legitimate `%20` in a path as data-carrying. Accepted deliberately: obfuscated encoding is a core exfiltration primitive, and the ambiguous case belongs on the review side."
|
|
},
|
|
|
|
"defang": {
|
|
"$comment": "Evidence hygiene: a finding's evidence carries the DEFANGED URL, so a report is safe to log and render without recreating the affordance it flagged. Idempotent — applying it twice changes nothing.",
|
|
"danger_scheme_pattern": "^(javascript|data|vbscript|file|blob)(?=:)",
|
|
"danger_scheme_flags": ["i"],
|
|
"danger_scheme_action": "replace the scheme's trailing colon with `[:]`",
|
|
"scheme_substitutions": [
|
|
{ "pattern": "^https", "flags": ["i"], "replacement": "hxxps" },
|
|
{ "pattern": "^http", "flags": ["i"], "replacement": "hxxp" },
|
|
{ "pattern": "^ftp", "flags": ["i"], "replacement": "fxp" }
|
|
],
|
|
"scheme_substitutions_note": "Ordered, first match wins, and applied only when the danger-scheme rule did not fire.",
|
|
"dot_pattern": "(?<!\\[)\\.(?!\\])",
|
|
"dot_replacement": "[.]",
|
|
"dot_note": "The lookarounds are what make the rule idempotent: a `.` already inside `[.]` is never touched.",
|
|
"url_in_text_pattern": "[A-Za-z][A-Za-z0-9+.\\-]{0,63}://[^\\s'\"<>]+",
|
|
"url_in_text_flags": [],
|
|
"url_in_text_note": "Used to defang bare URLs inside an already-flagged raw HTML tag. The scheme run is bounded here — unlike the detector patterns, where bounding would be a bypass — because this is a defanger applied inside a tag that has already been flagged, and a 64+ character scheme is not resolvable by any renderer. Measured 0.185s at the 1000000-char cap; unbounded it cost 12.99s on 100000 characters."
|
|
},
|
|
|
|
"evidence": {
|
|
"redact_show_start": 16,
|
|
"redact_show_end": 6,
|
|
"redact_applies_when": "length > show_start + show_end + 3",
|
|
"redact_joiner": "..."
|
|
},
|
|
|
|
"finding_semantics": {
|
|
"one_finding_per_class": true,
|
|
"count": "number of kept matches in that class",
|
|
"graded_by": "worst member",
|
|
"graded_by_note": "A class collapses to one finding, so an exfil-shaped URL hiding behind an ordinary one must set BOTH the severity and the evidence — otherwise the report shows an innocent URL next to a HIGH verdict.",
|
|
"evidence_selection": "the first data-carrying hit if any, otherwise the first hit",
|
|
"mutates_input": false,
|
|
"renders_disposition": false
|
|
},
|
|
|
|
"known_gaps": [
|
|
"Pure beaconing: a fetch whose only signal is that it happened. No URL shape distinguishes it from an ordinary remote image.",
|
|
"Short opaque segments: a carried value below the opacity length floors grades as ordinary.",
|
|
"Hex-wrapped payloads below the URL-token hex floor."
|
|
]
|
|
}
|