fix(calibration): grade active content on URL shape, not construct type
v0.3.0 made the untrusted upload path unusable: measured on both doors, an ordinary remote image fail_secure'd and an ordinary link/autolink/refdef quarantined, so only documents without external references persisted. Two independent defects compounded; neither fix works alone: 1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a URL that moves bytes outward, not an image. `is_ordinary_url` now grades on shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape, no opaque host label or path segment -> LOW; anything data-carrying keeps the carrier's severity. raw-html and data: URIs stay HIGH unconditionally. Opacity reuses entropy's primitives; floors calibrated against real doc URLs (worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in calibration. 2. The quarantine_default floor fired on ANY finding, a premise that broke when every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what makes this a patch rather than a minor. The corpus blind spot that let this pass 522 green tests is closed: the FP corpus carries realistic markdown and is asserted on the OUTPUT gate under PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still block. Beaconing and short opaque segments are conceded in LIMITATIONS and asserted by the coverage matrix rather than papered over. No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
This commit is contained in:
parent
da7421e6c8
commit
6e9b8168e3
13 changed files with 533 additions and 46 deletions
|
|
@ -14,11 +14,15 @@ items; this is the full list, each with the mechanism.
|
|||
HIGH injection reproduced verbatim under a *trusted* policy persists with a WARN.
|
||||
This is by design: if your "trusted" sources can carry attacker-influenced text,
|
||||
run them as untrusted (or add a quarantine floor).
|
||||
- **The upload preset's quarantine floor is currently vacuous.**
|
||||
`PRESET_USER_UPLOAD` sets `quarantine_default`, but detectors emit
|
||||
CRITICAL/HIGH/MEDIUM only, and under untrusted trust a MEDIUM already escalates to
|
||||
QUARANTINE_REVIEW — so the floor changes no outcome today. It is headroom for a
|
||||
future LOW/INFO finding, documented so the preset is not over-read.
|
||||
- **The quarantine floor fires at MEDIUM+, and is a no-op under the shipped upload
|
||||
preset.** Through 0.3.0 `quarantine_default` floored *any* finding to
|
||||
QUARANTINE_REVIEW. That premise ("a finding is the exception") broke when the
|
||||
active-content detector made every ordinary markdown link a finding, so 0.3.1
|
||||
raised the floor to MEDIUM+. Under `PRESET_USER_UPLOAD` (untrusted) a MEDIUM
|
||||
already escalates on trust alone, so the floor still changes no outcome there; it
|
||||
is live only for a caller-defined *trusted* policy that opts into
|
||||
`quarantine_default`. Documented so the preset is not over-read: a LOW finding on
|
||||
an upload now disposes WARN.
|
||||
- **Semantic / factual poisoning is invisible** to lexicon + entropy: a false claim
|
||||
in clean prose carries no suspicious token. **Highest impact for a wiki.** The
|
||||
`grounding` module ships only a `SourceGroundingCheck` *seam* — the deterministic
|
||||
|
|
@ -68,13 +72,39 @@ items; this is the full list, each with the mechanism.
|
|||
- **Lexicon findings are deduplicated by pattern id** — `count=1` and the first
|
||||
offset are reported, so a class matched across several channels collapses to one
|
||||
finding at its first location: a deliberate readability tradeoff.
|
||||
- **Active-content severity grades on URL shape, so a pure *beacon* is only LOW.**
|
||||
Since 0.3.1 a URL that merely names a remote document (bare path, no query, no
|
||||
userinfo, no percent-escapes, no opaque segment) is LOW, and only a URL that can
|
||||
move bytes outward keeps HIGH/MEDIUM. The deliberate hole: `` on an attacker-controlled host still *fetches* when a
|
||||
renderer touches it, leaking reader IP, user-agent and timing. Grading the fetch
|
||||
itself would re-block every ordinary document, which is precisely the 0.3.0
|
||||
regression this replaced — so beaconing is conceded, not covered.
|
||||
- **Short opaque URL segments slip through the same grading.** Opacity is decided by
|
||||
`entropy`'s primitives: base64 that decodes to text (≥20 chars), a hex id
|
||||
(≥32 chars), or Shannon entropy ≥4.4 at ≥24 chars. A shorter payload segment —
|
||||
`https://evil.test/aGVsbG8gd29ybGQ` — cannot be told from a name, because entropy
|
||||
is bounded by `log2(length)` at short lengths. Mitigation in depth, not in this
|
||||
detector: a literal credential in a URL is still caught by the LLM02 egress
|
||||
patterns in the same `scan_output` pass, whatever severity the carrier gets.
|
||||
- **Percent-escapes count as data-carrying — a `%20` in a path is a false positive.**
|
||||
An ordinary link with an encoded space grades as carrying and reaches
|
||||
QUARANTINE_REVIEW / FAIL_SECURE on an untrusted upload. Obfuscated encoding is a
|
||||
core exfil primitive and the ambiguous case is put on the review side deliberately;
|
||||
it is listed here because it is the same *class* of over-block that 0.3.1 fixed,
|
||||
in a rarer shape, and it is the first thing to re-measure against a real corpus.
|
||||
- **URL fragments are not graded.** A fragment is never sent to the server, so it
|
||||
cannot carry data to the host a renderer auto-fetches, and `…/overview#section` is
|
||||
the most common shape in real documentation. The residual: a *clicked* link to an
|
||||
attacker-controlled page can have its `location.hash` read by that page's script,
|
||||
so a fragment payload on a link (not an image) is uncovered.
|
||||
- **Secret egress: base64-wrapped is caught, hex-wrapped is not.** The output gate
|
||||
decodes base64 blobs and re-scans the plaintext, so a base64-*wrapped* secret
|
||||
surfaces as `decoded:egress:*`. `entropy` exposes decoded plaintext for base64
|
||||
only, so hex (and other encodings, or nested wraps) is a deliberate boundary —
|
||||
decode the transport layer first if you need it scanned.
|
||||
|
||||
## The four documented gaps (tracked by the coverage matrix)
|
||||
## The six documented gaps (tracked by the coverage matrix)
|
||||
|
||||
These are asserted to *still hold* by `tests/test_coverage_matrix.py` — a closed gap
|
||||
fails the test, forcing this doc to be updated:
|
||||
|
|
@ -83,6 +113,8 @@ fails the test, forcing this doc to be updated:
|
|||
2. **Semantic / factual poisoning** — invisible to token analysis (above).
|
||||
3. **A lone HIGH in trusted prose → WARN** — the §4.7 trust-scaling design (above).
|
||||
4. **Lexicon dedup (`count=1`)** — first offset only, by design (above).
|
||||
5. **Pure beaconing** — a bare-path remote image on a hostile host is LOW (above).
|
||||
6. **Short opaque URL segment (<24 chars)** — below what entropy can resolve (above).
|
||||
|
||||
## Out-of-scope (documented boundary)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue