open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions
Reusable, minimal, dependency-light defensive layer for LLM ingestion (write-time) pipelines — sanitize, fence, tool-less quarantined transform, capability isolation, output-scan-before-persist, fail-secure.
3 commits 1 branch 2 tags 443 KiB
  • Python 100%
Find a file
Kjell Tore Guttormsen d983aa3c95 docs(brief): fold in ms-ai-architect Layer B learnings (first output-side impl) 
Based on the working, tested Layer B ingestion-gate in ms-ai-architect (2026-07-04):
- A: correct consumer 2 — it fetches authored Learn docs + code samples via the
  microsoft-learn MCP, NOT the open Q&A forum / MSDN / Stack Overflow (that claim
  was unverified/overstated); low-trust surface is intra-document. Repo/name
  reconciled: the consumer is ms-ai-architect, no separate "MS AI Security plugin".
- B: §4.7 — trust tiers WITHIN a document (code sample / localized string vs
  authored prose), not only across sources; carriers + critical block in any tier.
- C: §12 — consume the lexicon as pure imported functions, not the llm-security
  scan CLI (which under-covers Markdown prose + base64-in-code-block, verified).
- D: §4.6 — fail-secure extends to scanner-unavailable: un-scannable ⇒ BLOCK.
Adds ms-ai-architect as the second reference implementation (output side).
2026-07-04 08:32:38 +02:00
docs docs(brief): fold in ms-ai-architect Layer B learnings (first output-side impl) 2026-07-04 08:32:38 +02:00
CLAUDE.md docs: design brief for reusable LLM ingestion-pipeline security library 2026-07-04 06:18:32 +02:00
README.md docs: design brief for reusable LLM ingestion-pipeline security library 2026-07-04 06:18:32 +02:00

README.md

llm-ingestion-pipeline-security

A reusable, minimal, dependency-light defensive layer for LLM ingestion pipelines — the write-time siblings of query-time chatbot guardrails.

Where mature guardrails (LLM Guard, NeMo Guardrails, Rebuff, Vigil, …) sit between a user and a model at query time, this library hardens the other shape: untrusted content flowing through an LLM enrichment/summarization/extraction step into a persisted, downstream-consumed artifact (RAG corpus, knowledge base, wiki). It packages the architectural contract — sanitize → fence → tool-less quarantined transform → per-stage capability isolation → scan output before commit → fail-secure — as composable, framework-agnostic code.

Status: brief / pre-implementation. Start with the design brief:

The contract is extracted from a working reference implementation (the claude-code-llm-wiki Stage B enrichment pipeline).