llm-ingestion-pipeline-secu.../CHANGELOG.md

5.4 KiB
Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added — OKF adapter (v0.2, stream 1)

An OKF (Google Open Knowledge Format v0.1) adapter on top of the format-agnostic core (llm_ingestion_guard.okf). The core stays text -> findings; the adapter knows OKF structure and routes scannable regions into the existing machinery. All TDD (failing test first), +61 tests. Verified against the OKF SPEC.md (2026-07-06). See docs/OKF-INGESTION-BRIEF.md §8.

  • parse_frontmatter — strict, reject-by-default frontmatter loader; refuses anchors, aliases, explicit tags, merge keys, block scalars and flow collections by construction, so YAML anchor/alias DoS and !!python/object coercion cannot occur (not a general YAML parser, by design). (T2)
  • scan_concept — whole-concept scan surface: frontmatter values (incl. description, read first under progressive disclosure), resource and body all go through scan_output. (T1)
  • validate_concept_path — path / reserved-name gate: rejects .. traversal, absolute paths and index.md / log.md shadowing; returns the concept-ID. (T4)
  • validate_resource_urlresource https allowlist: rejects non-https before commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
  • stamp_concept / format_log_entry — provenance stamping: origin × channel → trust × disposition per concept, emitted as log.md lines. Trust follows the origin, never the insertion channel. (T6)
  • import_bundle — received-bundle iterator (mode b): validates each concept (path, frontmatter, resource, scan, stamp) independently; one bad concept is rejected fail-secure while the rest are still checked; the aggregate disposition is the most severe. (T7)
  • link_graph / resolve_link / extract_link_targets — in-import cross-link graph: resolves .md links (bundle-absolute or relative) to concept-IDs, flags dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme or bundle-escaping targets. (T5a)

Deferred

  • Cross-run persisted link graph (T5b) — catching a link planted in one run whose poisoned target is written in a later run (§7.2) needs durable graph state whose storage/ownership depends on the consuming pipeline. Deferred to the consumer-wiring stream; cross-run dormant links remain a documented residual risk (README honest-limitations).

[0.1.0] — 2026-07-06 (alpha)

The stdlib-only core, built test-first (TDD) per docs/PLAN.md. Tagged v0.1.0.

Added

  • report — shared Finding / Report / Severity / Source types.
  • sanitize — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment, data:); byte-identical on clean input.
  • entropy — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
  • lexicon — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants; ReDoS-bounded, size-capped.
  • fence — randomized per-call spotlight delimiter; attacker marker-strip.
  • neutralize — opt-in defang of active-content output (byte-identical when clean).
  • output — compose lexicon + entropy + decode-rescan over emitted text; secret egress patterns (OWASP LLM02); report-only, never mutates.
  • disposition — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust policy; compound-signal escalation; fail-closed when the scanner errors.
  • contract — write-time asserters that raise: assert_tool_less, assert_credential_allowlist, scoped_env.
  • grounding — the SourceGroundingCheck seam for semantic poisoning (interface only; [judge] implementation plugs in behind an extra).
  • Top-level wiring — the prepare_input / screen_output §6 bookends plus the full public surface; end-to-end showcase and adversarial + false-positive corpora.

Security

Pre-release hardening from an independent adversarial review (all TDD, failing test first):

  • entropy — decode-and-rescan now runs before false-positive suppression, so an injection blob prefixed with an SRI/media marker (to dodge the entropy finding) is still decoded and rescanned by the lexicon. Suppression gates only the entropy finding, never the decode.
  • output — the invisible-carrier invariant now holds on the persist gate: scan_output flags zero-width / BIDI presence (output:zero-width-present, output:bidi-present) and disposition treats those plus lexicon:unicode-tags-present as any-tier carriers, so a carrier in model output fails secure even under a trusted policy.
  • contractassert_credential_allowlist catches a bare <PROVIDER>_KEY (e.g. STRIPE_KEY); the previous regex silently missed it (fail-open). The rule is deliberately broad (also flags PARTITION_KEY/SORT_KEY as loud, allowlistable false positives) — fail-loud beats fail-silent for isolation.
  • dispositionguard runs decide inside its guarded block, so a malformed report can no longer escape the fail-closed guarantee.
  • output — secret-egress placeholder suppression anchors word markers (example, todo, …) to a word boundary, so a real secret that merely contains such a word is no longer suppressed (fail-open egress miss closed).