open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Pull requests Releases Activity
llm-ingestion-pipeline-secu.../CHANGELOG.md
Kjell Tore Guttormsen 5397ba15a1 fix(security): harden 5 adversarial-review findings (M1/M2/M3 + m4/m6) via TDD 
Pre-release hardening from an independent adversarial review; each fixed
test-first (failing test -> fix -> green). 214 tests pass.

- entropy (M1): decode-and-rescan now runs BEFORE false-positive suppression,
  so an SRI/media-prefixed injection blob is still decoded and lexicon-rescanned.
  Suppression gates only the entropy finding, never the decode.
- output/disposition (M3): the invisible-carrier invariant now holds on the
  persist gate. scan_output flags zero-width/BIDI presence and disposition
  treats those + lexicon:unicode-tags-present as any-tier carriers, so a carrier
  in model output fails secure even under a trusted policy.
- contract (M2): assert_credential_allowlist catches a bare <PROVIDER>_KEY
  (e.g. STRIPE_KEY) that the old regex silently missed (fail-open). Deliberately
  broad: also flags PARTITION_KEY/SORT_KEY as loud, allowlistable FPs -- fail-loud
  beats fail-silent for an isolation control.
- disposition (m6): guard runs decide inside its guarded block -> total
  fail-closed even on a malformed report.
- output (m4): egress placeholder suppression anchors word markers (example,
  todo, ...) to a word boundary, closing a fail-open where a real secret merely
  containing such a word was suppressed.

Docs: CHANGELOG Security subsection; README honest-limit for lexicon dedup (m5,
documented tradeoff, not fixed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-05 10:45:05 +02:00

3 KiB
Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased] — v0.1.0 (alpha)

The stdlib-only core, built test-first (TDD) per docs/PLAN.md.

Added

  • report — shared Finding / Report / Severity / Source types.
  • sanitize — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment, data:); byte-identical on clean input.
  • entropy — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
  • lexicon — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants; ReDoS-bounded, size-capped.
  • fence — randomized per-call spotlight delimiter; attacker marker-strip.
  • neutralize — opt-in defang of active-content output (byte-identical when clean).
  • output — compose lexicon + entropy + decode-rescan over emitted text; secret egress patterns (OWASP LLM02); report-only, never mutates.
  • disposition — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust policy; compound-signal escalation; fail-closed when the scanner errors.
  • contract — write-time asserters that raise: assert_tool_less, assert_credential_allowlist, scoped_env.
  • grounding — the SourceGroundingCheck seam for semantic poisoning (interface only; [judge] implementation plugs in behind an extra).
  • Top-level wiring — the prepare_input / screen_output §6 bookends plus the full public surface; end-to-end showcase and adversarial + false-positive corpora.

Security

Pre-release hardening from an independent adversarial review (all TDD, failing test first):

  • entropy — decode-and-rescan now runs before false-positive suppression, so an injection blob prefixed with an SRI/media marker (to dodge the entropy finding) is still decoded and rescanned by the lexicon. Suppression gates only the entropy finding, never the decode.
  • output — the invisible-carrier invariant now holds on the persist gate: scan_output flags zero-width / BIDI presence (output:zero-width-present, output:bidi-present) and disposition treats those plus lexicon:unicode-tags-present as any-tier carriers, so a carrier in model output fails secure even under a trusted policy.
  • contract — assert_credential_allowlist catches a bare <PROVIDER>_KEY (e.g. STRIPE_KEY); the previous regex silently missed it (fail-open). The rule is deliberately broad (also flags PARTITION_KEY/SORT_KEY as loud, allowlistable false positives) — fail-loud beats fail-silent for isolation.
  • disposition — guard runs decide inside its guarded block, so a malformed report can no longer escape the fail-closed guarantee.
  • output — secret-egress placeholder suppression anchors word markers (example, todo, …) to a word boundary, so a real secret that merely contains such a word is no longer suppressed (fail-open egress miss closed).