llm-ingestion-pipeline-security/CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added — runnable threat-coverage matrix

A single declarative manifest (`llm_ingestion_guard.coverage`) that proves, in one
place, every vulnerability class the guard stops — and the documented gaps it does
not. Two consumers of the same source of truth:

- `python -m llm_ingestion_guard.coverage` — a narrated matrix
  (`class -> OWASP -> expected -> observed -> verdict`); exit 0 iff every caught
  class is caught and every documented gap holds. Stdlib-only, CI-usable.
- `tests/test_coverage_matrix.py` — asserts total recall over the core matrix
  (carriers, all 83 lexicon patterns, entropy/decode-rescan, active content, the
  contract asserters, the disposition engine, OKF T1–T7), asserts every documented
  gap still holds, and guards completeness (every lexicon pattern id, and every
  OWASP anchor claimed, has a case). Adds the full 25-pattern LLM02 secret-egress
  set and the container-layer front-end classes (CSV formula-injection, zip-slip,
  zip-bomb, symlink). +165 tests (357 → 522), no core dependency added.

This is the real-case validation gate ahead of a v1.0 freeze.

### Documentation — consumer adoption + README value proposition

- `docs/ADOPTION-BRIEF.md` — a self-contained brief a consumer repo (OKF
  second-brain / LLM wiki) can plan an inclusion from: the write-time trust-boundary
  argument, the two bookends + 8-step contract, the shipped OKF adapter
  (`import_bundle` mode-b), how to verify (coverage matrix), how to depend
  (stdlib-only core), and a checklist for *when/where* to wire it.
- README rewritten to lead with the write-time trust-boundary framing, add a
  first-class **OKF / LLM-wiki support (shipped)** section for `import_bundle`, and
  correct the test badge (357 → 522). Every claim verified against the code; all
  honest-limitations items retained.

## [0.2.0] — 2026-07-06

### Added — OKF adapter (stream 1)

An OKF (Google Open Knowledge Format v0.1) adapter *on top of* the
format-agnostic core (`llm_ingestion_guard.okf`). The core stays `text ->
findings`; the adapter knows OKF structure and routes scannable regions into the
existing machinery. All TDD (failing test first), +61 tests. Verified against the
OKF `SPEC.md` (2026-07-06). See `docs/OKF-INGESTION-BRIEF.md` §8.

- `parse_frontmatter` — strict, reject-by-default frontmatter loader; refuses
  anchors, aliases, explicit tags, merge keys, block scalars and flow collections
  by construction, so YAML anchor/alias DoS and `!!python/object` coercion cannot
  occur (not a general YAML parser, by design). (T2)
- `scan_concept` — whole-concept scan surface: frontmatter values (incl.
  `description`, read first under progressive disclosure), `resource` and body all
  go through `scan_output`. (T1)
- `validate_concept_path` — path / reserved-name gate: rejects `..` traversal,
  absolute paths and `index.md` / `log.md` shadowing; returns the concept-ID. (T4)
- `validate_resource_url` — `resource` https allowlist: rejects non-https before
  commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
- `stamp_concept` / `format_log_entry` — provenance stamping: origin × channel →
  trust × disposition per concept, emitted as `log.md` lines. Trust follows the
  origin, never the insertion channel. (T6)
- `import_bundle` — received-bundle iterator (mode b): validates each concept
  (path, frontmatter, resource, scan, stamp) independently; one bad concept is
  rejected fail-secure while the rest are still checked; the aggregate disposition
  is the most severe. (T7)
- `link_graph` / `resolve_link` / `extract_link_targets` — in-import cross-link
  graph: resolves `.md` links (bundle-absolute or relative) to concept-IDs, flags
  dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme
  or bundle-escaping targets. (T5a)

### Deferred

- Cross-run persisted link graph (T5b) — catching a link planted in one run whose
  poisoned target is written in a *later* run (§7.2) needs durable graph state
  whose storage/ownership depends on the consuming pipeline. Deferred to the
  consumer-wiring stream; cross-run dormant links remain a documented residual
  risk (README honest-limitations).

## [0.1.0] — 2026-07-06 (alpha)

The stdlib-only core, built test-first (TDD) per `docs/PLAN.md`. Tagged `v0.1.0`.

### Added

- `report` — shared `Finding` / `Report` / `Severity` / `Source` types.
- `sanitize` — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment,
  `data:`); byte-identical on clean input.
- `entropy` — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
- `lexicon` — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants;
  ReDoS-bounded, size-capped.
- `fence` — randomized per-call spotlight delimiter; attacker marker-strip.
- `neutralize` — opt-in defang of active-content output (byte-identical when clean).
- `output` — compose lexicon + entropy + decode-rescan over emitted text; secret
  egress patterns (OWASP LLM02); report-only, never mutates.
- `disposition` — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust
  policy; compound-signal escalation; fail-**closed** when the scanner errors.
- `contract` — write-time asserters that raise: `assert_tool_less`,
  `assert_credential_allowlist`, `scoped_env`.
- `grounding` — the `SourceGroundingCheck` seam for semantic poisoning (interface
  only; `[judge]` implementation plugs in behind an extra).
- Top-level wiring — the `prepare_input` / `screen_output` §6 bookends plus the
  full public surface; end-to-end showcase and adversarial + false-positive corpora.

### Security

Pre-release hardening from an independent adversarial review (all TDD, failing
test first):

- `entropy` — decode-and-rescan now runs **before** false-positive suppression,
  so an injection blob prefixed with an SRI/media marker (to dodge the entropy
  finding) is still decoded and rescanned by the lexicon. Suppression gates only
  the entropy finding, never the decode.
- `output` — the invisible-carrier invariant now holds on the persist gate:
  `scan_output` flags zero-width / BIDI presence (`output:zero-width-present`,
  `output:bidi-present`) and `disposition` treats those plus
  `lexicon:unicode-tags-present` as any-tier carriers, so a carrier in model
  output fails secure even under a trusted policy.
- `contract` — `assert_credential_allowlist` catches a bare `<PROVIDER>_KEY`
  (e.g. `STRIPE_KEY`); the previous regex silently missed it (fail-open). The
  rule is deliberately broad (also flags `PARTITION_KEY`/`SORT_KEY` as loud,
  allowlistable false positives) — fail-loud beats fail-silent for isolation.
- `disposition` — `guard` runs `decide` inside its guarded block, so a malformed
  report can no longer escape the fail-closed guarantee.
- `output` — secret-egress placeholder suppression anchors word markers
  (`example`, `todo`, …) to a word boundary, so a real secret that merely
  *contains* such a word is no longer suppressed (fail-open egress miss closed).