The scanners cap by truncating: they return findings, so reading a prefix costs detection in the tail and nothing else. The three transform surfaces return *content*, where the same move is not available — a shortened document is silent data loss, and a transformed prefix followed by an untransformed tail is a bypass, since the attacker chooses where in the document the payload sits. So they fail secure instead. Above MAX_INPUT_CHARS (1 000 000) sanitize, fence and neutralize raise OversizeInputError. sanitize is step 1 of prepare_input and only ever removes, so that one refusal bounds the whole input path. OversizeInputError subclasses ContractViolation: a pipeline already bracketing its quarantined stage keeps failing closed rather than meeting a type it has never heard of. It inherits the alert-routable property too — sizes in the message, refusing surface in details, no input in either. Invariant now pinned across all three: returned text is always fully transformed, or not returned at all. Still uncapped and recorded in LIMITATIONS: scan_active_content called directly (through scan_output it inherits that cap) and the okf link graph. Both are detection-shaped, so truncate-and-flag transfers unchanged — mechanical, not policy. 699 tests (+23), coverage 128/128 + 6/6, ReDoS sweep 0 candidates / 150.
491 lines
27 KiB
Markdown
491 lines
27 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to this project will be documented in this file.
|
||
|
||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||
|
||
## [Unreleased]
|
||
|
||
> **Behaviour change, not a pure fix.** The three transform surfaces gain a
|
||
> refusal path they did not have. A caller that today passes a document larger
|
||
> than 1 000 000 characters gets an exception where it previously got a result.
|
||
> Both pinned consumers were measured against this before it was written; git
|
||
> pins are exact, so nobody picks it up without re-pinning.
|
||
|
||
### Added — input-size cap on the transform surfaces (OWASP LLM10)
|
||
|
||
`sanitize`, `fence` and `neutralize` now raise `OversizeInputError` above
|
||
`MAX_INPUT_CHARS` (1 000 000) instead of accepting text of any length. Since
|
||
`sanitize` is step 1 of `prepare_input` and only ever *removes*, that single
|
||
refusal bounds the whole input path.
|
||
|
||
They **reject** where the scanners **truncate**, and the asymmetry is the point:
|
||
|
||
- `scan_lexicon` / `scan_output` return findings. Reading a prefix costs
|
||
detection in the tail and nothing else — a lossy answer, but an answer.
|
||
- `sanitize` / `fence` / `neutralize` return *content*. Truncating would return
|
||
a shortened document (silent data loss for anything that persists the result)
|
||
or a transformed prefix followed by an untransformed tail — a bypass, since an
|
||
attacker chooses where in the document the payload sits.
|
||
|
||
The invariant the three now keep: **returned text is always fully transformed,
|
||
or not returned at all.**
|
||
|
||
`OversizeInputError` subclasses `ContractViolation`, so a pipeline already
|
||
bracketing its quarantined stage in `except ContractViolation` keeps failing
|
||
closed. Like its parent it is alert-routable: the message carries the size and
|
||
the cap, `details` names the refusing surface, and neither carries input.
|
||
`max_input_chars` is a per-call parameter, defaulting to the single calibrated
|
||
constant.
|
||
|
||
### Still uncapped, and deliberately
|
||
|
||
`scan_active_content` **called directly** and the okf link graph. Reached through
|
||
`scan_output`, `scan_active_content` inherits that function's cap. Both are
|
||
detection-shaped, so the truncate-and-flag mechanism transfers to them unchanged
|
||
— mechanical follow-up work, not a policy question. Recorded in
|
||
`docs/LIMITATIONS.md`.
|
||
|
||
|
||
## [0.3.4] — 2026-08-01
|
||
|
||
> **Denial-of-service fix on the INPUT path. Upgrade from 0.3.3.** 0.3.3 swept
|
||
> the 83 lexicon patterns arm by arm and left every other table on 0.3.2's
|
||
> hand-written rows. Generalising the sweep over all eleven regex-bearing modules
|
||
> found three more quadratic patterns — two of them on the input path, one in
|
||
> `sanitize`, the first thing every ingested document touches. No disposition
|
||
> changes: recall was measured case by case and nothing was lost. Earlier tags
|
||
> are not moved.
|
||
|
||
### Fixed — three quadratic patterns, two on the input path
|
||
|
||
Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a **required**
|
||
literal, so crafted input that never supplies the literal makes every start
|
||
position rescan the tail. Each exponent is read across four doublings, not from a
|
||
two-point ratio.
|
||
|
||
| Pattern | Crafted payload | Measured @ 100 000 | Exponent |
|
||
|---|---|---|---|
|
||
| `sanitize._HTML_COMMENT_RE` | `<!--` × N | **20.1 s** | 1.96–2.14 |
|
||
| `active_content.URL_IN_TEXT_RE` | `<a ` + `A` × N + `>` | 12.99 s / 14.9 s | 1.87–2.22 |
|
||
| `okf._MD_LINK_RE` | `[` × N | 7.1 s | 1.99–2.05 |
|
||
|
||
These are worse than the 0.3.3 findings, and the reason is a separate finding of
|
||
its own: `MAX_SCAN_CHARS` is applied in `scan_lexicon` and `scan_output` **only**.
|
||
`sanitize`, `neutralize`, `scan_active_content` and the okf link graph accept
|
||
input of any size, so there is no cap to extrapolate to. Now documented as a
|
||
residual in `docs/LIMITATIONS.md`; extending the cap into the input path changes
|
||
the contract for existing callers and is deliberately not done in a ReDoS patch.
|
||
|
||
Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix
|
||
choice must not be copied blindly from a neighbouring table:
|
||
|
||
- **`sanitize`** drops the regex for a `str.find` scan, semantically identical to
|
||
the lazy `<!--.*?-->` it replaces. Excluding `<` from the run would lose every
|
||
comment containing markup (`<!-- <b>x</b> -->` is the ordinary case); bounding
|
||
the run would be a one-line carrier bypass of the exact construct the stripper
|
||
exists to remove. The module's own "no catastrophic backtracking" comment was
|
||
wrong in the same way `output`'s was before 0.3.2, and is corrected in place.
|
||
- **`URL_IN_TEXT_RE`** bounds its scheme run to an RFC 3986 scheme (`{0,63}`).
|
||
Bounding is safe *here* only because this is a defanger applied inside a tag
|
||
already flagged `active:raw-html`, so padding shifts where the match starts
|
||
rather than evading detection. A lookbehind killing interior start positions
|
||
was measured too and **rejected**: it drops `-http://evil.com` and
|
||
`.http://x.com`, a one-character evasion of the defanger. Bounded, the pattern
|
||
runs in 0.185 s at the full 1 000 000-char cap.
|
||
- **`okf._MD_LINK_RE`** excludes `[`, matching `active_content.MD_LINK_RE`
|
||
exactly, including the nested-label trade already documented there.
|
||
|
||
### Changed — the sweep covers every regex surface, not one table
|
||
|
||
`docs/redos-sweep.py` now sweeps **150 patterns across 11 tables** (0.3.3 covered
|
||
83 in one). The collector is mechanical on both axes so no one has to remember to
|
||
list anything: it walks each module's namespace for compiled patterns, and it
|
||
derives each pattern's call mode from the module source, because `.sub()` and
|
||
`.finditer()` visit every start position where `.match()` cannot. A pattern
|
||
reachable only through a helper parameter gets the worst mode, marked `*` — the
|
||
fallback can over-measure but never miss.
|
||
|
||
Two arm shapes the generator cannot express are pinned by hand as a result: a tag
|
||
that *closes* around a long body (repeating-unit payloads never close it), and a
|
||
run of plain characters carrying no anchor at all. The `okf` destination run gets
|
||
no row on purpose: `[^)\s]+` cannot fail, so a pin for it could never go red.
|
||
|
||
A lexicon candidate flagged at ×2.8 measured **linear** across four doublings
|
||
(exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring
|
||
warns about, confirmed a second time.
|
||
|
||
676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.
|
||
|
||
## [0.3.3] — 2026-07-31
|
||
|
||
> **Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2.** The
|
||
> sweep 0.3.2 shipped was incomplete, and it said otherwise. Two lexicon patterns
|
||
> were still quadratic — reachable through `scan_output`, not only on the input
|
||
> path. No disposition changes: recall was measured case by case and nothing was
|
||
> lost. The v0.3.2 tag is not moved.
|
||
|
||
### Fixed — two quadratic patterns in the lexicon table
|
||
|
||
`8deca93` scoped the remaining ReDoS duty to the lexicon path, and this is that
|
||
work: all 83 patterns measured, arm by arm. Two are quadratic, same shape as
|
||
everything 0.3.2 fixed — a run in front of a **required** literal, where the run
|
||
may cross the pattern's own opening anchor.
|
||
|
||
| Pattern (arm) | Crafted unit | Measured | At the 1 000 000-char cap |
|
||
|---|---|---|---|
|
||
| `markdown:link-anchor-injection` (anchor text) | `[` | 1.91 s @ 8 000 | **~8.3 hours** |
|
||
| `markdown:link-anchor-injection` (URL run) | `[system](` | 0.006 s @ 8 000 | ~89 seconds |
|
||
| `markdown:link-ref-comment` (`.*` run) | `[//]: # (` | 0.22 s @ 8 000 | **~1.0 hour** |
|
||
|
||
Exponent measured over five points (1 000 → 16 000): **1.98** — quadratic, not
|
||
exponential. Legitimate content of the same size is unaffected: 0.316 s at
|
||
N=100 000 (prose 0.316 / html 0.315 / markdown 0.297 / connection-string 0.296).
|
||
|
||
**These were not input-path-only, and that is the correction.** `scan_lexicon`
|
||
runs on the output path too, so 0.3.2's *"the last quadratic-backtracking site on
|
||
the output path"* was false when written. Measured through the public gate before
|
||
this fix: `scan_output("[" * 100_000)` took **334.7 s**. The claim was too broad
|
||
because the sweep behind it drove the `[` payload only through
|
||
`scan_active_content` — no row ever drove it through the lexicon. The statement is
|
||
corrected in `docs/LIMITATIONS.md`.
|
||
|
||
The fix is anchor exclusion, per the rule `active_content` already documents —
|
||
bounding attacker-controlled content would be a one-line detection bypass. The
|
||
excluded character is `(`, not `[`:
|
||
|
||
```
|
||
markdown:link-anchor-injection
|
||
\[[^\]\[]*(?:system|…)[^\]\[]*\]\([^)(]+\)
|
||
markdown:link-ref-comment
|
||
\[//\]:\s*#\s*\([^(\n]*(?:ignore|…)
|
||
```
|
||
|
||
`[` was the obvious choice and it was measurably worse. Excluding `[` from the
|
||
URL run drops `[override your rules](https://[::1]/x)` — still covered, three
|
||
other patterns fire on it — but excluding `[` from the link-ref comment run drops
|
||
`[//]: # (see [x] then ignore this)`, which **no other pattern catches**. The
|
||
anchors contain `(` as well, so excluding `(` telescopes just as effectively at
|
||
zero measured recall cost. Both forms verified linear (×1.99–2.02 on doubling).
|
||
|
||
### Known behaviour changes
|
||
|
||
- **None measured.** Every case that matched before still matches, except URLs
|
||
containing a literal `(` inside a markdown link target and comment bodies
|
||
containing a literal `(` before the keyword. No corpus, showcase, or coverage
|
||
row moved; 666 tests pass.
|
||
|
||
### Tests
|
||
|
||
Four rows added. Three name the guilty pattern per arm
|
||
(`test_crafted_redos_payload_stays_bounded_in_the_lexicon`), one covers the
|
||
composed gate (`test_gate_is_bounded_on_the_payload_the_first_sweep_missed`).
|
||
Pre-fix they failed at 297 s, 8.1 s, 55 s and 334.7 s.
|
||
|
||
`N` is per row deliberately. The URL arm is quadratic with a small constant and
|
||
ran 0.9 s **unfixed** at N=100 000 — under the 2.0 s bound, so that row would have
|
||
passed whether or not the pattern was fixed. It is measured at N=300 000 instead,
|
||
where crafted (8.10 s) and legitimate (0.926 s) separate 8.8×.
|
||
|
||
### Residual
|
||
|
||
The sweep flags on timing and ignores measurements below a 1.5 ms noise floor at
|
||
N=8 000. An arm hiding just under it could still cost **~23 s** at the cap, so what
|
||
this supports is *"no arm worse than ~23 s"*, not *"no quadratic arm remains"*.
|
||
The blind spot is not hypothetical: a generic-payload pass found only one of the
|
||
two patterns. The second appeared only once payloads were synthesised per run
|
||
from each pattern's own skeleton. Recorded in `docs/LIMITATIONS.md`.
|
||
|
||
## [0.3.2] — 2026-07-31
|
||
|
||
> **Denial-of-service fix. Upgrade from 0.3.1.** The output gate could be made to
|
||
> spend hours on a single call by crafted input it accepts by design. No
|
||
> disposition changes for ordinary documents — the one measured exception is
|
||
> listed under *Known behaviour changes* below. The v0.3.1 tag is not moved.
|
||
|
||
### Fixed — 19 quadratic regex runs on the output path
|
||
|
||
`scan_output` claimed LLM10 self-safety on the grounds that its patterns contain no
|
||
nested quantifiers. That is true and it is not the property that matters. A run in
|
||
front of a **required** literal, reachable from a short anchor, is enough: crafted
|
||
input repeats the anchor and never supplies the literal, so every start position
|
||
rescans the tail. Quadratic, not exponential — and quadratic is sufficient here.
|
||
|
||
Measured, not argued (Python 3.14, this machine):
|
||
|
||
| Input | Time through `scan_output` |
|
||
|---|---|
|
||
| `<a:` × 100 000 (300 KB) | **458.7 s** |
|
||
| size-matched ordinary prose | 0.31 s |
|
||
| same payload extrapolated to the 1 000 000-char input the gate itself accepts | **~5.7 hours for one call** |
|
||
|
||
`max_scan_chars` does not mitigate this. It bounds the *input*; quadratic work on a
|
||
bounded input is still hours. That claim was stated in both `output.py` and
|
||
`calibration.py` and is corrected in both.
|
||
|
||
The fix is per pattern, not uniform:
|
||
|
||
- **`active_content` + the lexicon table (15 runs)** — exclude the character that
|
||
opens the pattern's own anchor (`[` for markdown, `<` for tags), so a run cannot
|
||
reach past the next start position and the per-start costs telescope. Verified to
|
||
cost no recall: long URLs, long alt text, and `<` inside a quoted attribute all
|
||
still match. Bounding instead would have been linear too, but wrong here — the
|
||
content is attacker-controlled, so padding past a bound would be a one-line bypass
|
||
of the EchoLeak class this table exists to catch.
|
||
- **`*-connstr` secret egress (4 runs)** — bound the password at the new
|
||
`MAX_CONNSTR_VALUE` (256) in `calibration`. The exclusion fix is unavailable: the
|
||
anchor character is `/`, and passwords containing `/` are the common case
|
||
(measured — they match today).
|
||
- **`hybrid-xss:script-tag`** — had neither option, since its run is the script
|
||
*body*, which may legitimately contain `<`. It now matches the opening tag and no
|
||
longer requires `</script>`.
|
||
|
||
### Known behaviour changes
|
||
|
||
Two, both measured against the v0.3.1 tag rather than reasoned about:
|
||
|
||
- **A JWT used as a DB password, over 256 chars, is no longer CRITICAL.** The
|
||
remaining detections (`entropy:base64-blob` HIGH, `egress:jwt-token` MEDIUM) top
|
||
out below CRITICAL, so the any-tier block is lost: under `PRESET_TRUSTED_SOURCE`
|
||
such a document moves from `fail_secure` to `quarantine_review`. Under
|
||
`PRESET_USER_UPLOAD` it still `fail_secure`s, and a *generic* long password still
|
||
trips `entropy:base64-blob` at CRITICAL with no change at all. The credential is
|
||
never silently missed; on one preset it is held for review instead of halted.
|
||
- **`hybrid-xss:script-tag` now fires on prose that merely mentions `<script>`** —
|
||
and this costs no consumer a disposition. Any text containing a literal
|
||
`<script>` already produced `active:raw-html` at HIGH on 0.3.1, so the same
|
||
document disposed identically before and after. The label is new; the outcome is
|
||
not. By the same measurement, the fail-open this closed (an unclosed
|
||
`<script>alert(1)`) was confined to `scan_lexicon` called on its own — through
|
||
either composed gate, `active:raw-html` already caught it.
|
||
|
||
Both are recorded in `docs/LIMITATIONS.md` (still 29 items — these replace nothing).
|
||
|
||
### Method note
|
||
|
||
The defect was found by a composed-gate DoS test that stayed red after every
|
||
individual scanner had been made linear; the remaining 813× was the lexicon's six
|
||
`html-obfuscation` patterns. A per-scanner test alone would have shipped it. The
|
||
static shape analysis used to find candidates also missed `[\s\S]*?` in
|
||
`script-tag` — the sweep that matters is measurement, not a regex over regexes.
|
||
|
||
662 tests pass (was 642), and the suite is faster than before the fix.
|
||
|
||
## [0.3.1] — 2026-07-25
|
||
|
||
> **Regression fix. Upgrade from 0.3.0.** v0.3.0 made the high-untrust upload path
|
||
> unusable for ordinary documents — measured, not projected. `llm-ingestion-okf`
|
||
> projected the consequence from the 0.3.0 changelog text *before* the tag was cut;
|
||
> the release went out without the inbox being read. The v0.3.0 tag is not moved.
|
||
|
||
### Fixed — the upload path is usable again without losing EchoLeak detection
|
||
|
||
Measured on v0.3.0, both doors (`screen_output` under `PRESET_USER_UPLOAD` and
|
||
`okf.import_bundle` with `origin=EXTERNAL`): a document with **one ordinary remote
|
||
image** disposed `fail_secure`; one ordinary link, autolink or reference definition
|
||
disposed `quarantine_review`. Only documents with no external references persisted.
|
||
Two independent defects compounded, and both had to be fixed — either alone leaves
|
||
the path blocked:
|
||
|
||
- **Severity graded on construct type instead of URL shape.** `markdown-image` was
|
||
HIGH for *any* external image, but the exfiltration primitive is not "an image" —
|
||
it is a URL that moves bytes to a host the attacker controls.
|
||
`` carries nothing. Severity now grades on
|
||
shape: a URL that only *names* a remote document (http(s) or protocol-relative,
|
||
no query, no userinfo, no percent-escapes, no opaque host label or path segment)
|
||
is **LOW**; anything that can carry a value keeps the carrier's full severity.
|
||
`raw-html` and `data:` URIs have no ordinary form and stay HIGH unconditionally.
|
||
Opacity reuses `entropy`'s primitives — decodable base64 (≥20 chars), hex id
|
||
(≥32), or Shannon entropy ≥4.4 at ≥24 chars — calibrated 2026-07-25 against real
|
||
documentation URLs (worst legitimate token H=4.08; exfil payload segments
|
||
4.36-4.54). New constants live in `calibration` with the rest.
|
||
- **The `quarantine_default` floor fired on *any* finding.** It rested on the premise
|
||
that a finding is the exception; adding the active-content detector in 0.3.0 made
|
||
every ordinary markdown link a finding, and the floor then held ordinary documents
|
||
for review. The floor now fires at **MEDIUM+**. This is a no-op for every detector
|
||
that shipped before 0.3.0 — the lexicon holds no LOW/INFO pattern and no other
|
||
detector emits LOW (asserted in `tests/test_calibration.py`) — which is why this is
|
||
a patch and not a minor.
|
||
|
||
**Unchanged, deliberately:** no new public API and no new preset (a middle tier is
|
||
0.4.0 work); the `allow_reserved=True` mode-b default stands — two independent
|
||
consumers document it as load-bearing; the gate still never rewrites content.
|
||
|
||
### Added
|
||
|
||
- **False-positive corpus covers ordinary markdown.** The 0.3.0 corpus had zero
|
||
markdown links or images, asserted only under `PRESET_TRUSTED_SOURCE` (where every
|
||
non-CRITICAL finding WARNs anyway), and drove the *input* path — so `scan_output`
|
||
step 6, where active content actually lives, was never reached. That is how a
|
||
regression this size passed 522 green tests. The corpus now carries realistic
|
||
documents and asserts them on the **output gate under the upload preset**, plus a
|
||
counter-corpus of exfil-shaped URLs (query, base64/hex path segment,
|
||
percent-encoded payload, opaque subdomain, userinfo) that must still block.
|
||
- **Two new documented gaps** in `docs/LIMITATIONS.md`, both asserted by the coverage
|
||
matrix: **pure beaconing** (a bare-path image on a hostile host still fetches, and
|
||
the fetch is not graded) and **short opaque URL segments** (<24 chars, below what
|
||
entropy can resolve). Percent-escapes counting as data-carrying is recorded there
|
||
as a known false positive.
|
||
|
||
## [0.3.0] — 2026-07-25
|
||
|
||
> **A minor bump, not a patch — deliberately.** The changes under *Changed* alter what
|
||
> an existing caller observes with no code change on their side, so a `>=0.2,<0.3` pin
|
||
> stops here rather than absorbing them silently. Re-test that branch before widening
|
||
> the pin. Still alpha: the public API may change again before 1.0.
|
||
|
||
### Changed — observable gate behaviour (re-test before upgrading)
|
||
|
||
Three commits since v0.2.0 change dispositions for an unchanged caller. Two tighten
|
||
the gate; one loosens it.
|
||
|
||
- **`okf.import_bundle` no longer path-rejects reserved basenames.** At v0.2.0,
|
||
`index.md` / `log.md` anywhere in a received bundle was an unconditional
|
||
per-concept hard reject (FAIL_SECURE), and `import_bundle` took no keyword for it.
|
||
The new `allow_reserved` keyword **defaults to `True`** on this mode-b
|
||
*received-bundle* path, so those files are scanned — their body is the
|
||
highest-priority injection surface — rather than refused, and may clear the floor
|
||
and become mergeable. **This is the one loosening change:** content a v0.2.0
|
||
consumer never saw can now reach it, so a consumer whose tests pin the v0.2.0
|
||
reject must re-check, not just bump. A front-end materialising individual
|
||
*uploads* must pass `allow_reserved=False` to keep the shadow-reject there;
|
||
`validate_concept_path` still defaults to `False`.
|
||
- **Active content now reaches the disposition engine.** `scan_output` step 6 runs
|
||
`scan_active_content`, so markdown images/links, reference definitions, autolinks,
|
||
raw active HTML and `data:` URIs surface as `active:*` findings (OWASP LLM05 — the
|
||
EchoLeak / CVE-2025-32711 class) instead of being admitted with `findings=[]`.
|
||
These carry real severities (zero-click auto-fetch/execute HIGH, click-required
|
||
MEDIUM), and two MEDIUM+ findings compound-escalate one tier, so a document that
|
||
passed clean at v0.2.0 can now WARN, quarantine, or fail secure in both
|
||
`screen_output` and `okf.import_bundle`.
|
||
- **Base64-wrapped secrets are now caught as egress.** The output gate's
|
||
decode-and-rescan feeds decoded base64 plaintext through both the lexicon and the
|
||
LLM02 secret-egress detector, so a base64-wrapped credential surfaces as
|
||
`decoded:egress:*` instead of disappearing. Hex-wrapped remains a documented gap
|
||
(`docs/LIMITATIONS.md`).
|
||
|
||
### Added — runnable threat-coverage matrix
|
||
|
||
A single declarative manifest (`llm_ingestion_guard.coverage`) that proves, in one
|
||
place, every vulnerability class the guard stops — and the documented gaps it does
|
||
not. Two consumers of the same source of truth:
|
||
|
||
- `python -m llm_ingestion_guard.coverage` — a narrated matrix
|
||
(`class -> OWASP -> expected -> observed -> verdict`); exit 0 iff every caught
|
||
class is caught and every documented gap holds. Stdlib-only, CI-usable.
|
||
- `tests/test_coverage_matrix.py` — asserts total recall over the core matrix
|
||
(carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the
|
||
contract asserters, the disposition engine, OKF T1–T7), asserts every documented
|
||
gap still holds, and guards completeness (every lexicon pattern id, and every
|
||
OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress
|
||
set and the container-layer front-end classes (CSV formula-injection, zip-slip,
|
||
zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.
|
||
|
||
This is the real-case validation gate ahead of a v1.0 freeze.
|
||
|
||
### Documentation — consumer adoption + README value proposition
|
||
|
||
- `docs/ADOPTION-BRIEF.md` — a self-contained brief a consumer repo (OKF
|
||
second-brain / LLM wiki) can plan an inclusion from: the write-time trust-boundary
|
||
argument, the two bookends + 8-step contract, the shipped OKF adapter
|
||
(`import_bundle` mode-b), how to verify (coverage matrix), how to depend
|
||
(stdlib-only core), and a checklist for *when/where* to wire it.
|
||
- README rewritten to lead with the write-time trust-boundary framing, add a
|
||
first-class **OKF / LLM-wiki support (shipped)** section for `import_bundle`, a
|
||
concrete **What it protects against** catalogue (attack classes grouped by OWASP
|
||
anchor, driven by the coverage matrix), and correct the test badge (357 → 522).
|
||
Every claim verified against the code.
|
||
- `docs/LIMITATIONS.md` — the full honest-limitations list (15 items + the four
|
||
documented gaps + out-of-scope) moved out of the README, which now carries a
|
||
high-impact summary + link, so protection and limits read in balance.
|
||
|
||
## [0.2.0] — 2026-07-06
|
||
|
||
### Added — OKF adapter (stream 1)
|
||
|
||
An OKF (Google Open Knowledge Format v0.1) adapter *on top of* the
|
||
format-agnostic core (`llm_ingestion_guard.okf`). The core stays `text ->
|
||
findings`; the adapter knows OKF structure and routes scannable regions into the
|
||
existing machinery. All TDD (failing test first), +61 tests. Verified against the
|
||
OKF `SPEC.md` (2026-07-06). See `docs/OKF-INGESTION-BRIEF.md` §8.
|
||
|
||
- `parse_frontmatter` — strict, reject-by-default frontmatter loader; refuses
|
||
anchors, aliases, explicit tags, merge keys, block scalars and flow collections
|
||
by construction, so YAML anchor/alias DoS and `!!python/object` coercion cannot
|
||
occur (not a general YAML parser, by design). (T2)
|
||
- `scan_concept` — whole-concept scan surface: frontmatter values (incl.
|
||
`description`, read first under progressive disclosure), `resource` and body all
|
||
go through `scan_output`. (T1)
|
||
- `validate_concept_path` — path / reserved-name gate: rejects `..` traversal,
|
||
absolute paths and `index.md` / `log.md` shadowing; returns the concept-ID. (T4)
|
||
- `validate_resource_url` — `resource` https allowlist: rejects non-https before
|
||
commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
|
||
- `stamp_concept` / `format_log_entry` — provenance stamping: origin × channel →
|
||
trust × disposition per concept, emitted as `log.md` lines. Trust follows the
|
||
origin, never the insertion channel. (T6)
|
||
- `import_bundle` — received-bundle iterator (mode b): validates each concept
|
||
(path, frontmatter, resource, scan, stamp) independently; one bad concept is
|
||
rejected fail-secure while the rest are still checked; the aggregate disposition
|
||
is the most severe. (T7)
|
||
- `link_graph` / `resolve_link` / `extract_link_targets` — in-import cross-link
|
||
graph: resolves `.md` links (bundle-absolute or relative) to concept-IDs, flags
|
||
dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme
|
||
or bundle-escaping targets. (T5a)
|
||
|
||
### Deferred
|
||
|
||
- Cross-run persisted link graph (T5b) — catching a link planted in one run whose
|
||
poisoned target is written in a *later* run (§7.2) needs durable graph state
|
||
whose storage/ownership depends on the consuming pipeline. Deferred to the
|
||
consumer-wiring stream; cross-run dormant links remain a documented residual
|
||
risk (README honest-limitations).
|
||
|
||
## [0.1.0] — 2026-07-06 (alpha)
|
||
|
||
The stdlib-only core, built test-first (TDD) per `docs/PLAN.md`. Tagged `v0.1.0`.
|
||
|
||
### Added
|
||
|
||
- `report` — shared `Finding` / `Report` / `Severity` / `Source` types.
|
||
- `sanitize` — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment,
|
||
`data:`); byte-identical on clean input.
|
||
- `entropy` — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
|
||
- `lexicon` — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants;
|
||
ReDoS-bounded, size-capped.
|
||
- `fence` — randomized per-call spotlight delimiter; attacker marker-strip.
|
||
- `neutralize` — opt-in defang of active-content output (byte-identical when clean).
|
||
- `output` — compose lexicon + entropy + decode-rescan over emitted text; secret
|
||
egress patterns (OWASP LLM02); report-only, never mutates.
|
||
- `disposition` — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust
|
||
policy; compound-signal escalation; fail-**closed** when the scanner errors.
|
||
- `contract` — write-time asserters that raise: `assert_tool_less`,
|
||
`assert_credential_allowlist`, `scoped_env`.
|
||
- `grounding` — the `SourceGroundingCheck` seam for semantic poisoning (interface
|
||
only; `[judge]` implementation plugs in behind an extra).
|
||
- Top-level wiring — the `prepare_input` / `screen_output` §6 bookends plus the
|
||
full public surface; end-to-end showcase and adversarial + false-positive corpora.
|
||
|
||
### Security
|
||
|
||
Pre-release hardening from an independent adversarial review (all TDD, failing
|
||
test first):
|
||
|
||
- `entropy` — decode-and-rescan now runs **before** false-positive suppression,
|
||
so an injection blob prefixed with an SRI/media marker (to dodge the entropy
|
||
finding) is still decoded and rescanned by the lexicon. Suppression gates only
|
||
the entropy finding, never the decode.
|
||
- `output` — the invisible-carrier invariant now holds on the persist gate:
|
||
`scan_output` flags zero-width / BIDI presence (`output:zero-width-present`,
|
||
`output:bidi-present`) and `disposition` treats those plus
|
||
`lexicon:unicode-tags-present` as any-tier carriers, so a carrier in model
|
||
output fails secure even under a trusted policy.
|
||
- `contract` — `assert_credential_allowlist` catches a bare `<PROVIDER>_KEY`
|
||
(e.g. `STRIPE_KEY`); the previous regex silently missed it (fail-open). The
|
||
rule is deliberately broad (also flags `PARTITION_KEY`/`SORT_KEY` as loud,
|
||
allowlistable false positives) — fail-loud beats fail-silent for isolation.
|
||
- `disposition` — `guard` runs `decide` inside its guarded block, so a malformed
|
||
report can no longer escape the fail-closed guarantee.
|
||
- `output` — secret-egress placeholder suppression anchors word markers
|
||
(`example`, `todo`, …) to a word boundary, so a real secret that merely
|
||
*contains* such a word is no longer suppressed (fail-open egress miss closed).
|