open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions
Reusable, minimal, dependency-light defensive layer for LLM ingestion (write-time) pipelines — sanitize, fence, tool-less quarantined transform, capability isolation, output-scan-before-persist, fail-secure.
6 commits 1 branch 2 tags 443 KiB
  • Python 100%
Find a file
Kjell Tore Guttormsen e5e91df369 feat(entropy): high-entropy/encoded-blob detection with decode-and-rescan (TDD) 
Build order step 3. Pure text -> findings detector ported from the
llm-security entropy-scanner seed:

- Length-calibrated Shannon-entropy classification (CRITICAL 5.4/128,
  HIGH 5.1/64, MEDIUM 4.7/40).
- Shape floor: base64-like (len>100) / hex (len>64) reach at least MEDIUM
  even when entropy alone does not trigger — the only path that catches hex
  (16-symbol alphabet caps H at 4.0 < 4.7).
- Decode-and-rescan (must-have): base64 blobs that decode to printable text
  are exposed on EntropyResult.decoded for a later lexicon rescan.
- FP suppression scoped to the text-relevant subset (base64 media data-URI
  prefixes + SRI sha*- prefix); source-code-specific seed rules omitted.

Exposes ported primitives shannon_entropy / is_base64_like / is_hex_blob /
try_decode_base64. 16 new tests; full suite 31 green. Stdlib-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 09:43:24 +02:00
docs feat: scaffold package + report and sanitize modules (TDD) 2026-07-04 09:24:20 +02:00
src/llm_ingestion_guard feat(entropy): high-entropy/encoded-blob detection with decode-and-rescan (TDD) 2026-07-04 09:43:24 +02:00
tests feat(entropy): high-entropy/encoded-blob detection with decode-and-rescan (TDD) 2026-07-04 09:43:24 +02:00
.gitignore chore: repo scaffolding via /repo-init 2026-07-04 08:39:23 +02:00
CHANGELOG.md chore: repo scaffolding via /repo-init 2026-07-04 08:39:23 +02:00
CLAUDE.md chore: repo scaffolding via /repo-init 2026-07-04 08:39:23 +02:00
LICENSE feat: scaffold package + report and sanitize modules (TDD) 2026-07-04 09:24:20 +02:00
pyproject.toml feat: scaffold package + report and sanitize modules (TDD) 2026-07-04 09:24:20 +02:00
README.md docs: design brief for reusable LLM ingestion-pipeline security library 2026-07-04 06:18:32 +02:00

README.md

llm-ingestion-pipeline-security

A reusable, minimal, dependency-light defensive layer for LLM ingestion pipelines — the write-time siblings of query-time chatbot guardrails.

Where mature guardrails (LLM Guard, NeMo Guardrails, Rebuff, Vigil, …) sit between a user and a model at query time, this library hardens the other shape: untrusted content flowing through an LLM enrichment/summarization/extraction step into a persisted, downstream-consumed artifact (RAG corpus, knowledge base, wiki). It packages the architectural contract — sanitize → fence → tool-less quarantined transform → per-stage capability isolation → scan output before commit → fail-secure — as composable, framework-agnostic code.

Status: brief / pre-implementation. Start with the design brief:

The contract is extracted from a working reference implementation (the claude-code-llm-wiki Stage B enrichment pipeline).