fix(calibration): grade active content on URL shape, not construct type
v0.3.0 made the untrusted upload path unusable: measured on both doors, an ordinary remote image fail_secure'd and an ordinary link/autolink/refdef quarantined, so only documents without external references persisted. Two independent defects compounded; neither fix works alone: 1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a URL that moves bytes outward, not an image. `is_ordinary_url` now grades on shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape, no opaque host label or path segment -> LOW; anything data-carrying keeps the carrier's severity. raw-html and data: URIs stay HIGH unconditionally. Opacity reuses entropy's primitives; floors calibrated against real doc URLs (worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in calibration. 2. The quarantine_default floor fired on ANY finding, a premise that broke when every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what makes this a patch rather than a minor. The corpus blind spot that let this pass 522 green tests is closed: the FP corpus carries realistic markdown and is asserted on the OUTPUT gate under PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still block. Beaconing and short opaque segments are conceded in LIMITATIONS and asserted by the coverage matrix rather than papered over. No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
This commit is contained in:
parent
da7421e6c8
commit
6e9b8168e3
13 changed files with 533 additions and 46 deletions
56
CHANGELOG.md
56
CHANGELOG.md
|
|
@ -7,6 +7,62 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.3.1] — 2026-07-25
|
||||
|
||||
> **Regression fix. Upgrade from 0.3.0.** v0.3.0 made the high-untrust upload path
|
||||
> unusable for ordinary documents — measured, not projected. `llm-ingestion-okf`
|
||||
> projected the consequence from the 0.3.0 changelog text *before* the tag was cut;
|
||||
> the release went out without the inbox being read. The v0.3.0 tag is not moved.
|
||||
|
||||
### Fixed — the upload path is usable again without losing EchoLeak detection
|
||||
|
||||
Measured on v0.3.0, both doors (`screen_output` under `PRESET_USER_UPLOAD` and
|
||||
`okf.import_bundle` with `origin=EXTERNAL`): a document with **one ordinary remote
|
||||
image** disposed `fail_secure`; one ordinary link, autolink or reference definition
|
||||
disposed `quarantine_review`. Only documents with no external references persisted.
|
||||
Two independent defects compounded, and both had to be fixed — either alone leaves
|
||||
the path blocked:
|
||||
|
||||
- **Severity graded on construct type instead of URL shape.** `markdown-image` was
|
||||
HIGH for *any* external image, but the exfiltration primitive is not "an image" —
|
||||
it is a URL that moves bytes to a host the attacker controls.
|
||||
`` carries nothing. Severity now grades on
|
||||
shape: a URL that only *names* a remote document (http(s) or protocol-relative,
|
||||
no query, no userinfo, no percent-escapes, no opaque host label or path segment)
|
||||
is **LOW**; anything that can carry a value keeps the carrier's full severity.
|
||||
`raw-html` and `data:` URIs have no ordinary form and stay HIGH unconditionally.
|
||||
Opacity reuses `entropy`'s primitives — decodable base64 (≥20 chars), hex id
|
||||
(≥32), or Shannon entropy ≥4.4 at ≥24 chars — calibrated 2026-07-25 against real
|
||||
documentation URLs (worst legitimate token H=4.08; exfil payload segments
|
||||
4.36-4.54). New constants live in `calibration` with the rest.
|
||||
- **The `quarantine_default` floor fired on *any* finding.** It rested on the premise
|
||||
that a finding is the exception; adding the active-content detector in 0.3.0 made
|
||||
every ordinary markdown link a finding, and the floor then held ordinary documents
|
||||
for review. The floor now fires at **MEDIUM+**. This is a no-op for every detector
|
||||
that shipped before 0.3.0 — the lexicon holds no LOW/INFO pattern and no other
|
||||
detector emits LOW (asserted in `tests/test_calibration.py`) — which is why this is
|
||||
a patch and not a minor.
|
||||
|
||||
**Unchanged, deliberately:** no new public API and no new preset (a middle tier is
|
||||
0.4.0 work); the `allow_reserved=True` mode-b default stands — two independent
|
||||
consumers document it as load-bearing; the gate still never rewrites content.
|
||||
|
||||
### Added
|
||||
|
||||
- **False-positive corpus covers ordinary markdown.** The 0.3.0 corpus had zero
|
||||
markdown links or images, asserted only under `PRESET_TRUSTED_SOURCE` (where every
|
||||
non-CRITICAL finding WARNs anyway), and drove the *input* path — so `scan_output`
|
||||
step 6, where active content actually lives, was never reached. That is how a
|
||||
regression this size passed 522 green tests. The corpus now carries realistic
|
||||
documents and asserts them on the **output gate under the upload preset**, plus a
|
||||
counter-corpus of exfil-shaped URLs (query, base64/hex path segment,
|
||||
percent-encoded payload, opaque subdomain, userinfo) that must still block.
|
||||
- **Two new documented gaps** in `docs/LIMITATIONS.md`, both asserted by the coverage
|
||||
matrix: **pure beaconing** (a bare-path image on a hostile host still fetches, and
|
||||
the fetch is not graded) and **short opaque URL segments** (<24 chars, below what
|
||||
entropy can resolve). Percent-escapes counting as data-carrying is recorded there
|
||||
as a known false positive.
|
||||
|
||||
## [0.3.0] — 2026-07-25
|
||||
|
||||
> **A minor bump, not a patch — deliberately.** The changes under *Changed* alter what
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue