llm-ingestion-pipeline-security/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added — runnable threat-coverage matrix

A single declarative manifest (llm_ingestion_guard.coverage) that proves, in one place, every vulnerability class the guard stops — and the documented gaps it does not. Two consumers of the same source of truth:

• python -m llm_ingestion_guard.coverage — a narrated matrix (class -> OWASP -> expected -> observed -> verdict); exit 0 iff every caught class is caught and every documented gap holds. Stdlib-only, CI-usable.
• tests/test_coverage_matrix.py — asserts total recall over the core matrix (carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the contract asserters, the disposition engine, OKF T1–T7), asserts every documented gap still holds, and guards completeness (every lexicon pattern id, and every OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress set and the container-layer front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.

This is the real-case validation gate ahead of a v1.0 freeze.

Documentation — consumer adoption + README value proposition

• docs/ADOPTION-BRIEF.md — a self-contained brief a consumer repo (OKF second-brain / LLM wiki) can plan an inclusion from: the write-time trust-boundary argument, the two bookends + 8-step contract, the shipped OKF adapter (import_bundle mode-b), how to verify (coverage matrix), how to depend (stdlib-only core), and a checklist for when/where to wire it.
• README rewritten to lead with the write-time trust-boundary framing, add a first-class OKF / LLM-wiki support (shipped) section for import_bundle, and correct the test badge (357 → 522). Every claim verified against the code; all honest-limitations items retained.

[0.2.0] — 2026-07-06

Added — OKF adapter (stream 1)

An OKF (Google Open Knowledge Format v0.1) adapter on top of the format-agnostic core (llm_ingestion_guard.okf). The core stays text -> findings; the adapter knows OKF structure and routes scannable regions into the existing machinery. All TDD (failing test first), +61 tests. Verified against the OKF SPEC.md (2026-07-06). See docs/OKF-INGESTION-BRIEF.md §8.

• parse_frontmatter — strict, reject-by-default frontmatter loader; refuses anchors, aliases, explicit tags, merge keys, block scalars and flow collections by construction, so YAML anchor/alias DoS and !!python/object coercion cannot occur (not a general YAML parser, by design). (T2)
• scan_concept — whole-concept scan surface: frontmatter values (incl. description, read first under progressive disclosure), resource and body all go through scan_output. (T1)
• validate_concept_path — path / reserved-name gate: rejects .. traversal, absolute paths and index.md / log.md shadowing; returns the concept-ID. (T4)
• validate_resource_url — resource https allowlist: rejects non-https before commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
• stamp_concept / format_log_entry — provenance stamping: origin × channel → trust × disposition per concept, emitted as log.md lines. Trust follows the origin, never the insertion channel. (T6)
• import_bundle — received-bundle iterator (mode b): validates each concept (path, frontmatter, resource, scan, stamp) independently; one bad concept is rejected fail-secure while the rest are still checked; the aggregate disposition is the most severe. (T7)
• link_graph / resolve_link / extract_link_targets — in-import cross-link graph: resolves .md links (bundle-absolute or relative) to concept-IDs, flags dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme or bundle-escaping targets. (T5a)

Deferred

• Cross-run persisted link graph (T5b) — catching a link planted in one run whose poisoned target is written in a later run (§7.2) needs durable graph state whose storage/ownership depends on the consuming pipeline. Deferred to the consumer-wiring stream; cross-run dormant links remain a documented residual risk (README honest-limitations).

[0.1.0] — 2026-07-06 (alpha)

The stdlib-only core, built test-first (TDD) per docs/PLAN.md. Tagged v0.1.0.

Added

• report — shared Finding / Report / Severity / Source types.
• sanitize — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment, data:); byte-identical on clean input.
• entropy — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
• lexicon — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants; ReDoS-bounded, size-capped.
• fence — randomized per-call spotlight delimiter; attacker marker-strip.
• neutralize — opt-in defang of active-content output (byte-identical when clean).
• output — compose lexicon + entropy + decode-rescan over emitted text; secret egress patterns (OWASP LLM02); report-only, never mutates.
• disposition — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust policy; compound-signal escalation; fail-closed when the scanner errors.
• contract — write-time asserters that raise: assert_tool_less, assert_credential_allowlist, scoped_env.
• grounding — the SourceGroundingCheck seam for semantic poisoning (interface only; [judge] implementation plugs in behind an extra).
• Top-level wiring — the prepare_input / screen_output §6 bookends plus the full public surface; end-to-end showcase and adversarial + false-positive corpora.

Security

Pre-release hardening from an independent adversarial review (all TDD, failing test first):

• entropy — decode-and-rescan now runs before false-positive suppression, so an injection blob prefixed with an SRI/media marker (to dodge the entropy finding) is still decoded and rescanned by the lexicon. Suppression gates only the entropy finding, never the decode.
• output — the invisible-carrier invariant now holds on the persist gate: scan_output flags zero-width / BIDI presence (output:zero-width-present, output:bidi-present) and disposition treats those plus lexicon:unicode-tags-present as any-tier carriers, so a carrier in model output fails secure even under a trusted policy.
• contract — assert_credential_allowlist catches a bare <PROVIDER>_KEY (e.g. STRIPE_KEY); the previous regex silently missed it (fail-open). The rule is deliberately broad (also flags PARTITION_KEY/SORT_KEY as loud, allowlistable false positives) — fail-loud beats fail-silent for isolation.
• disposition — guard runs decide inside its guarded block, so a malformed report can no longer escape the fail-closed guarantee.
• output — secret-egress placeholder suppression anchors word markers (example, todo, …) to a word boundary, so a real secret that merely contains such a word is no longer suppressed (fail-open egress miss closed).