llm-ingestion-pipeline-security/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Behaviour change, not a pure fix. The three transform surfaces gain a refusal path they did not have. A caller that today passes a document larger than 1 000 000 characters gets an exception where it previously got a result. Both pinned consumers were measured against this before it was written; git pins are exact, so nobody picks it up without re-pinning.

Added — input-size cap on the transform surfaces (OWASP LLM10)

sanitize, fence and neutralize now raise OversizeInputError above MAX_INPUT_CHARS (1 000 000) instead of accepting text of any length. Since sanitize is step 1 of prepare_input and only ever removes, that single refusal bounds the whole input path.

They reject where the scanners truncate, and the asymmetry is the point:

• scan_lexicon / scan_output return findings. Reading a prefix costs detection in the tail and nothing else — a lossy answer, but an answer.
• sanitize / fence / neutralize return content. Truncating would return a shortened document (silent data loss for anything that persists the result) or a transformed prefix followed by an untransformed tail — a bypass, since an attacker chooses where in the document the payload sits.

The invariant the three now keep: returned text is always fully transformed, or not returned at all.

OversizeInputError subclasses ContractViolation, so a pipeline already bracketing its quarantined stage in except ContractViolation keeps failing closed. Like its parent it is alert-routable: the message carries the size and the cap, details names the refusing surface, and neither carries input. max_input_chars is a per-call parameter, defaulting to the single calibrated constant.

Still uncapped, and deliberately

scan_active_content called directly and the okf link graph. Reached through scan_output, scan_active_content inherits that function's cap. Both are detection-shaped, so the truncate-and-flag mechanism transfers to them unchanged — mechanical follow-up work, not a policy question. Recorded in docs/LIMITATIONS.md.

[0.3.4] — 2026-08-01

Denial-of-service fix on the INPUT path. Upgrade from 0.3.3. 0.3.3 swept the 83 lexicon patterns arm by arm and left every other table on 0.3.2's hand-written rows. Generalising the sweep over all eleven regex-bearing modules found three more quadratic patterns — two of them on the input path, one in sanitize, the first thing every ingested document touches. No disposition changes: recall was measured case by case and nothing was lost. Earlier tags are not moved.

Fixed — three quadratic patterns, two on the input path

Same class as everything 0.3.2 and 0.3.3 fixed: a run in front of a required literal, so crafted input that never supplies the literal makes every start position rescan the tail. Each exponent is read across four doublings, not from a two-point ratio.

Pattern	Crafted payload	Measured @ 100 000	Exponent
sanitize._HTML_COMMENT_RE	<!-- × N	20.1 s	1.96–2.14
active_content.URL_IN_TEXT_RE	<a + A × N + >	12.99 s / 14.9 s	1.87–2.22
okf._MD_LINK_RE	[ × N	7.1 s	1.99–2.05

These are worse than the 0.3.3 findings, and the reason is a separate finding of its own: MAX_SCAN_CHARS is applied in scan_lexicon and scan_output only. sanitize, neutralize, scan_active_content and the okf link graph accept input of any size, so there is no cap to extrapolate to. Now documented as a residual in docs/LIMITATIONS.md; extending the cap into the input path changes the contract for existing callers and is deliberately not done in a ReDoS patch.

Each fix is the one the pattern's own shape allows — the 0.3.3 lesson that a fix choice must not be copied blindly from a neighbouring table:

• sanitize drops the regex for a str.find scan, semantically identical to the lazy <!--.*?--> it replaces. Excluding < from the run would lose every comment containing markup (<!-- <b>x</b> --> is the ordinary case); bounding the run would be a one-line carrier bypass of the exact construct the stripper exists to remove. The module's own "no catastrophic backtracking" comment was wrong in the same way output's was before 0.3.2, and is corrected in place.
• URL_IN_TEXT_RE bounds its scheme run to an RFC 3986 scheme ({0,63}). Bounding is safe here only because this is a defanger applied inside a tag already flagged active:raw-html, so padding shifts where the match starts rather than evading detection. A lookbehind killing interior start positions was measured too and rejected: it drops -http://evil.com and .http://x.com, a one-character evasion of the defanger. Bounded, the pattern runs in 0.185 s at the full 1 000 000-char cap.
• okf._MD_LINK_RE excludes [, matching active_content.MD_LINK_RE exactly, including the nested-label trade already documented there.

Changed — the sweep covers every regex surface, not one table

docs/redos-sweep.py now sweeps 150 patterns across 11 tables (0.3.3 covered 83 in one). The collector is mechanical on both axes so no one has to remember to list anything: it walks each module's namespace for compiled patterns, and it derives each pattern's call mode from the module source, because .sub() and .finditer() visit every start position where .match() cannot. A pattern reachable only through a helper parameter gets the worst mode, marked * — the fallback can over-measure but never miss.

Two arm shapes the generator cannot express are pinned by hand as a result: a tag that closes around a long body (repeating-unit payloads never close it), and a run of plain characters carrying no anchor at all. The okf destination run gets no row on purpose: [^)\s]+ cannot fail, so a pin for it could never go red.

A lexicon candidate flagged at ×2.8 measured linear across four doublings (exponent 0.96–1.03) — the near-noise-floor false flag the script's own docstring warns about, confirmed a second time.

676 tests (+10). Coverage matrix unchanged at 128/128 caught, 6/6 gaps holding.

[0.3.3] — 2026-07-31

Denial-of-service fix, and a correction to 0.3.2. Upgrade from 0.3.2. The sweep 0.3.2 shipped was incomplete, and it said otherwise. Two lexicon patterns were still quadratic — reachable through scan_output, not only on the input path. No disposition changes: recall was measured case by case and nothing was lost. The v0.3.2 tag is not moved.

Fixed — two quadratic patterns in the lexicon table

8deca93 scoped the remaining ReDoS duty to the lexicon path, and this is that work: all 83 patterns measured, arm by arm. Two are quadratic, same shape as everything 0.3.2 fixed — a run in front of a required literal, where the run may cross the pattern's own opening anchor.

Pattern (arm)	Crafted unit	Measured	At the 1 000 000-char cap
markdown:link-anchor-injection (anchor text)	[	1.91 s @ 8 000	~8.3 hours
markdown:link-anchor-injection (URL run)	[system](	0.006 s @ 8 000	~89 seconds
markdown:link-ref-comment (.* run)	[//]: # (	0.22 s @ 8 000	~1.0 hour

Exponent measured over five points (1 000 → 16 000): 1.98 — quadratic, not exponential. Legitimate content of the same size is unaffected: 0.316 s at N=100 000 (prose 0.316 / html 0.315 / markdown 0.297 / connection-string 0.296).

These were not input-path-only, and that is the correction. scan_lexicon runs on the output path too, so 0.3.2's "the last quadratic-backtracking site on the output path" was false when written. Measured through the public gate before this fix: scan_output("[" * 100_000) took 334.7 s. The claim was too broad because the sweep behind it drove the [ payload only through scan_active_content — no row ever drove it through the lexicon. The statement is corrected in docs/LIMITATIONS.md.

The fix is anchor exclusion, per the rule active_content already documents — bounding attacker-controlled content would be a one-line detection bypass. The excluded character is (, not [:

markdown:link-anchor-injection
  \[[^\]\[]*(?:system|…)[^\]\[]*\]\([^)(]+\)
markdown:link-ref-comment
  \[//\]:\s*#\s*\([^(\n]*(?:ignore|…)

[ was the obvious choice and it was measurably worse. Excluding [ from the URL run drops [override your rules](https://[::1]/x) — still covered, three other patterns fire on it — but excluding [ from the link-ref comment run drops [//]: # (see [x] then ignore this), which no other pattern catches. The anchors contain ( as well, so excluding ( telescopes just as effectively at zero measured recall cost. Both forms verified linear (×1.99–2.02 on doubling).

Known behaviour changes

• None measured. Every case that matched before still matches, except URLs containing a literal ( inside a markdown link target and comment bodies containing a literal ( before the keyword. No corpus, showcase, or coverage row moved; 666 tests pass.

Tests

Four rows added. Three name the guilty pattern per arm (test_crafted_redos_payload_stays_bounded_in_the_lexicon), one covers the composed gate (test_gate_is_bounded_on_the_payload_the_first_sweep_missed). Pre-fix they failed at 297 s, 8.1 s, 55 s and 334.7 s.

N is per row deliberately. The URL arm is quadratic with a small constant and ran 0.9 s unfixed at N=100 000 — under the 2.0 s bound, so that row would have passed whether or not the pattern was fixed. It is measured at N=300 000 instead, where crafted (8.10 s) and legitimate (0.926 s) separate 8.8×.

Residual

The sweep flags on timing and ignores measurements below a 1.5 ms noise floor at N=8 000. An arm hiding just under it could still cost ~23 s at the cap, so what this supports is "no arm worse than ~23 s", not "no quadratic arm remains". The blind spot is not hypothetical: a generic-payload pass found only one of the two patterns. The second appeared only once payloads were synthesised per run from each pattern's own skeleton. Recorded in docs/LIMITATIONS.md.

[0.3.2] — 2026-07-31

Denial-of-service fix. Upgrade from 0.3.1. The output gate could be made to spend hours on a single call by crafted input it accepts by design. No disposition changes for ordinary documents — the one measured exception is listed under Known behaviour changes below. The v0.3.1 tag is not moved.

Fixed — 19 quadratic regex runs on the output path

scan_output claimed LLM10 self-safety on the grounds that its patterns contain no nested quantifiers. That is true and it is not the property that matters. A run in front of a required literal, reachable from a short anchor, is enough: crafted input repeats the anchor and never supplies the literal, so every start position rescans the tail. Quadratic, not exponential — and quadratic is sufficient here.

Measured, not argued (Python 3.14, this machine):

Input	Time through scan_output
<a: × 100 000 (300 KB)	458.7 s
size-matched ordinary prose	0.31 s
same payload extrapolated to the 1 000 000-char input the gate itself accepts	~5.7 hours for one call

max_scan_chars does not mitigate this. It bounds the input; quadratic work on a bounded input is still hours. That claim was stated in both output.py and calibration.py and is corrected in both.

The fix is per pattern, not uniform:

• active_content + the lexicon table (15 runs) — exclude the character that opens the pattern's own anchor ([ for markdown, < for tags), so a run cannot reach past the next start position and the per-start costs telescope. Verified to cost no recall: long URLs, long alt text, and < inside a quoted attribute all still match. Bounding instead would have been linear too, but wrong here — the content is attacker-controlled, so padding past a bound would be a one-line bypass of the EchoLeak class this table exists to catch.
• *-connstr secret egress (4 runs) — bound the password at the new MAX_CONNSTR_VALUE (256) in calibration. The exclusion fix is unavailable: the anchor character is /, and passwords containing / are the common case (measured — they match today).
• hybrid-xss:script-tag — had neither option, since its run is the script body, which may legitimately contain <. It now matches the opening tag and no longer requires </script>.

Known behaviour changes

Two, both measured against the v0.3.1 tag rather than reasoned about:

• A JWT used as a DB password, over 256 chars, is no longer CRITICAL. The remaining detections (entropy:base64-blob HIGH, egress:jwt-token MEDIUM) top out below CRITICAL, so the any-tier block is lost: under PRESET_TRUSTED_SOURCE such a document moves from fail_secure to quarantine_review. Under PRESET_USER_UPLOAD it still fail_secures, and a generic long password still trips entropy:base64-blob at CRITICAL with no change at all. The credential is never silently missed; on one preset it is held for review instead of halted.
• hybrid-xss:script-tag now fires on prose that merely mentions <script> — and this costs no consumer a disposition. Any text containing a literal <script> already produced active:raw-html at HIGH on 0.3.1, so the same document disposed identically before and after. The label is new; the outcome is not. By the same measurement, the fail-open this closed (an unclosed <script>alert(1)) was confined to scan_lexicon called on its own — through either composed gate, active:raw-html already caught it.

Both are recorded in docs/LIMITATIONS.md (still 29 items — these replace nothing).

Method note

The defect was found by a composed-gate DoS test that stayed red after every individual scanner had been made linear; the remaining 813× was the lexicon's six html-obfuscation patterns. A per-scanner test alone would have shipped it. The static shape analysis used to find candidates also missed [\s\S]*? in script-tag — the sweep that matters is measurement, not a regex over regexes.

662 tests pass (was 642), and the suite is faster than before the fix.

[0.3.1] — 2026-07-25

Regression fix. Upgrade from 0.3.0. v0.3.0 made the high-untrust upload path unusable for ordinary documents — measured, not projected. llm-ingestion-okf projected the consequence from the 0.3.0 changelog text before the tag was cut; the release went out without the inbox being read. The v0.3.0 tag is not moved.

Fixed — the upload path is usable again without losing EchoLeak detection

Measured on v0.3.0, both doors (screen_output under PRESET_USER_UPLOAD and okf.import_bundle with origin=EXTERNAL): a document with one ordinary remote image disposed fail_secure; one ordinary link, autolink or reference definition disposed quarantine_review. Only documents with no external references persisted. Two independent defects compounded, and both had to be fixed — either alone leaves the path blocked:

• Severity graded on construct type instead of URL shape. markdown-image was HIGH for any external image, but the exfiltration primitive is not "an image" — it is a URL that moves bytes to a host the attacker controls. ![diagram](https://example.com/arch.png) carries nothing. Severity now grades on shape: a URL that only names a remote document (http(s) or protocol-relative, no query, no userinfo, no percent-escapes, no opaque host label or path segment) is LOW; anything that can carry a value keeps the carrier's full severity. raw-html and data: URIs have no ordinary form and stay HIGH unconditionally. Opacity reuses entropy's primitives — decodable base64 (≥20 chars), hex id (≥32), or Shannon entropy ≥4.4 at ≥24 chars — calibrated 2026-07-25 against real documentation URLs (worst legitimate token H=4.08; exfil payload segments 4.36-4.54). New constants live in calibration with the rest.
• The quarantine_default floor fired on any finding. It rested on the premise that a finding is the exception; adding the active-content detector in 0.3.0 made every ordinary markdown link a finding, and the floor then held ordinary documents for review. The floor now fires at MEDIUM+. This is a no-op for every detector that shipped before 0.3.0 — the lexicon holds no LOW/INFO pattern and no other detector emits LOW (asserted in tests/test_calibration.py) — which is why this is a patch and not a minor.

Unchanged, deliberately: no new public API and no new preset (a middle tier is 0.4.0 work); the allow_reserved=True mode-b default stands — two independent consumers document it as load-bearing; the gate still never rewrites content.

Added

• False-positive corpus covers ordinary markdown. The 0.3.0 corpus had zero markdown links or images, asserted only under PRESET_TRUSTED_SOURCE (where every non-CRITICAL finding WARNs anyway), and drove the input path — so scan_output step 6, where active content actually lives, was never reached. That is how a regression this size passed 522 green tests. The corpus now carries realistic documents and asserts them on the output gate under the upload preset, plus a counter-corpus of exfil-shaped URLs (query, base64/hex path segment, percent-encoded payload, opaque subdomain, userinfo) that must still block.
• Two new documented gaps in docs/LIMITATIONS.md, both asserted by the coverage matrix: pure beaconing (a bare-path image on a hostile host still fetches, and the fetch is not graded) and short opaque URL segments (<24 chars, below what entropy can resolve). Percent-escapes counting as data-carrying is recorded there as a known false positive.

[0.3.0] — 2026-07-25

A minor bump, not a patch — deliberately. The changes under Changed alter what an existing caller observes with no code change on their side, so a >=0.2,<0.3 pin stops here rather than absorbing them silently. Re-test that branch before widening the pin. Still alpha: the public API may change again before 1.0.

Changed — observable gate behaviour (re-test before upgrading)

Three commits since v0.2.0 change dispositions for an unchanged caller. Two tighten the gate; one loosens it.

• okf.import_bundle no longer path-rejects reserved basenames. At v0.2.0, index.md / log.md anywhere in a received bundle was an unconditional per-concept hard reject (FAIL_SECURE), and import_bundle took no keyword for it. The new allow_reserved keyword defaults to True on this mode-b received-bundle path, so those files are scanned — their body is the highest-priority injection surface — rather than refused, and may clear the floor and become mergeable. This is the one loosening change: content a v0.2.0 consumer never saw can now reach it, so a consumer whose tests pin the v0.2.0 reject must re-check, not just bump. A front-end materialising individual uploads must pass allow_reserved=False to keep the shadow-reject there; validate_concept_path still defaults to False.
• Active content now reaches the disposition engine. scan_output step 6 runs scan_active_content, so markdown images/links, reference definitions, autolinks, raw active HTML and data: URIs surface as active:* findings (OWASP LLM05 — the EchoLeak / CVE-2025-32711 class) instead of being admitted with findings=[]. These carry real severities (zero-click auto-fetch/execute HIGH, click-required MEDIUM), and two MEDIUM+ findings compound-escalate one tier, so a document that passed clean at v0.2.0 can now WARN, quarantine, or fail secure in both screen_output and okf.import_bundle.
• Base64-wrapped secrets are now caught as egress. The output gate's decode-and-rescan feeds decoded base64 plaintext through both the lexicon and the LLM02 secret-egress detector, so a base64-wrapped credential surfaces as decoded:egress:* instead of disappearing. Hex-wrapped remains a documented gap (docs/LIMITATIONS.md).

Added — runnable threat-coverage matrix

A single declarative manifest (llm_ingestion_guard.coverage) that proves, in one place, every vulnerability class the guard stops — and the documented gaps it does not. Two consumers of the same source of truth:

• python -m llm_ingestion_guard.coverage — a narrated matrix (class -> OWASP -> expected -> observed -> verdict); exit 0 iff every caught class is caught and every documented gap holds. Stdlib-only, CI-usable.
• tests/test_coverage_matrix.py — asserts total recall over the core matrix (carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the contract asserters, the disposition engine, OKF T1–T7), asserts every documented gap still holds, and guards completeness (every lexicon pattern id, and every OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress set and the container-layer front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.

This is the real-case validation gate ahead of a v1.0 freeze.

Documentation — consumer adoption + README value proposition

• docs/ADOPTION-BRIEF.md — a self-contained brief a consumer repo (OKF second-brain / LLM wiki) can plan an inclusion from: the write-time trust-boundary argument, the two bookends + 8-step contract, the shipped OKF adapter (import_bundle mode-b), how to verify (coverage matrix), how to depend (stdlib-only core), and a checklist for when/where to wire it.
• README rewritten to lead with the write-time trust-boundary framing, add a first-class OKF / LLM-wiki support (shipped) section for import_bundle, a concrete What it protects against catalogue (attack classes grouped by OWASP anchor, driven by the coverage matrix), and correct the test badge (357 → 522). Every claim verified against the code.
• docs/LIMITATIONS.md — the full honest-limitations list (15 items + the four documented gaps + out-of-scope) moved out of the README, which now carries a high-impact summary + link, so protection and limits read in balance.

[0.2.0] — 2026-07-06

Added — OKF adapter (stream 1)

An OKF (Google Open Knowledge Format v0.1) adapter on top of the format-agnostic core (llm_ingestion_guard.okf). The core stays text -> findings; the adapter knows OKF structure and routes scannable regions into the existing machinery. All TDD (failing test first), +61 tests. Verified against the OKF SPEC.md (2026-07-06). See docs/OKF-INGESTION-BRIEF.md §8.

• parse_frontmatter — strict, reject-by-default frontmatter loader; refuses anchors, aliases, explicit tags, merge keys, block scalars and flow collections by construction, so YAML anchor/alias DoS and !!python/object coercion cannot occur (not a general YAML parser, by design). (T2)
• scan_concept — whole-concept scan surface: frontmatter values (incl. description, read first under progressive disclosure), resource and body all go through scan_output. (T1)
• validate_concept_path — path / reserved-name gate: rejects .. traversal, absolute paths and index.md / log.md shadowing; returns the concept-ID. (T4)
• validate_resource_url — resource https allowlist: rejects non-https before commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
• stamp_concept / format_log_entry — provenance stamping: origin × channel → trust × disposition per concept, emitted as log.md lines. Trust follows the origin, never the insertion channel. (T6)
• import_bundle — received-bundle iterator (mode b): validates each concept (path, frontmatter, resource, scan, stamp) independently; one bad concept is rejected fail-secure while the rest are still checked; the aggregate disposition is the most severe. (T7)
• link_graph / resolve_link / extract_link_targets — in-import cross-link graph: resolves .md links (bundle-absolute or relative) to concept-IDs, flags dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme or bundle-escaping targets. (T5a)

Deferred

• Cross-run persisted link graph (T5b) — catching a link planted in one run whose poisoned target is written in a later run (§7.2) needs durable graph state whose storage/ownership depends on the consuming pipeline. Deferred to the consumer-wiring stream; cross-run dormant links remain a documented residual risk (README honest-limitations).

[0.1.0] — 2026-07-06 (alpha)

The stdlib-only core, built test-first (TDD) per docs/PLAN.md. Tagged v0.1.0.

Added

• report — shared Finding / Report / Severity / Source types.
• sanitize — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment, data:); byte-identical on clean input.
• entropy — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
• lexicon — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants; ReDoS-bounded, size-capped.
• fence — randomized per-call spotlight delimiter; attacker marker-strip.
• neutralize — opt-in defang of active-content output (byte-identical when clean).
• output — compose lexicon + entropy + decode-rescan over emitted text; secret egress patterns (OWASP LLM02); report-only, never mutates.
• disposition — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust policy; compound-signal escalation; fail-closed when the scanner errors.
• contract — write-time asserters that raise: assert_tool_less, assert_credential_allowlist, scoped_env.
• grounding — the SourceGroundingCheck seam for semantic poisoning (interface only; [judge] implementation plugs in behind an extra).
• Top-level wiring — the prepare_input / screen_output §6 bookends plus the full public surface; end-to-end showcase and adversarial + false-positive corpora.

Security

Pre-release hardening from an independent adversarial review (all TDD, failing test first):

• entropy — decode-and-rescan now runs before false-positive suppression, so an injection blob prefixed with an SRI/media marker (to dodge the entropy finding) is still decoded and rescanned by the lexicon. Suppression gates only the entropy finding, never the decode.
• output — the invisible-carrier invariant now holds on the persist gate: scan_output flags zero-width / BIDI presence (output:zero-width-present, output:bidi-present) and disposition treats those plus lexicon:unicode-tags-present as any-tier carriers, so a carrier in model output fails secure even under a trusted policy.
• contract — assert_credential_allowlist catches a bare <PROVIDER>_KEY (e.g. STRIPE_KEY); the previous regex silently missed it (fail-open). The rule is deliberately broad (also flags PARTITION_KEY/SORT_KEY as loud, allowlistable false positives) — fail-loud beats fail-silent for isolation.
• disposition — guard runs decide inside its guarded block, so a malformed report can no longer escape the fail-closed guarantee.
• output — secret-egress placeholder suppression anchors word markers (example, todo, …) to a word boundary, so a real secret that merely contains such a word is no longer suppressed (fail-open egress miss closed).