docs: design brief for reusable LLM ingestion-pipeline security library

Brief-stage repo. docs/BRIEF.md defines a minimal, framework-agnostic
library that packages the write-time ingestion contract (sanitize ->
fence -> tool-less quarantined transform -> per-stage capability
isolation -> scan output before persist -> fail-secure) as reusable
code. Positioned honestly against query-time guardrails (LLM Guard,
NeMo, Rebuff, Vigil) with a prior-art verification log. Reference
implementation: claude-code-llm-wiki Stage B.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPAmFyEVWbwvmSNVdXTu4d

CLAUDE.md

@@ -0,0 +1,21 @@
+# llm-ingestion-pipeline-security
+
+## Kontekst
+
+Gjenbrukbar, minimal defensiv layer for LLM **ingestion**-pipelines (write-time),
+til forskjell fra query-time chatbot-guardrails. Pakker det arkitektoniske
+kontraktet — sanitize → fence → tool-less karantenert transform → per-stadium
+capability-isolasjon → scan output før commit → fail-secure — som komponerbar,
+framework-agnostisk kode.
+
+Referanse-implementasjon: `claude-code-llm-wiki` Stage B (`tools/wiki_ingest/`).
+Lexikon-seed: `injection-patterns.mjs` fra `llm-security`-pluginen.
+
+Repoet er på **brief-stadiet**. Start med `docs/BRIEF.md`.
+
+## Konvensjoner
+
+- Norsk for dialog og planer, engelsk for kode og innhold (repoet er ment publisert).
+- Ingen GitHub — kun Forgejo (`git.fromaitochitta.com`) hvis/når publisert.
+- Ingen remote satt ennå; ingen push før operatøren bestemmer publisering.
+- Minimal-dependency: stdlib-first kjerne; ML/judge-detektorer bak extras.

README.md

@@ -0,0 +1,19 @@
+# llm-ingestion-pipeline-security
+
+A reusable, minimal, dependency-light defensive layer for **LLM ingestion
+pipelines** — the write-time siblings of query-time chatbot guardrails.
+
+Where mature guardrails (LLM Guard, NeMo Guardrails, Rebuff, Vigil, …) sit
+between a user and a model at query time, this library hardens the other shape:
+untrusted content flowing through an LLM enrichment/summarization/extraction step
+into a **persisted, downstream-consumed artifact** (RAG corpus, knowledge base,
+wiki). It packages the architectural contract — sanitize → fence → tool-less
+quarantined transform → per-stage capability isolation → scan output before
+commit → fail-secure — as composable, framework-agnostic code.
+
+**Status:** brief / pre-implementation. Start with the design brief:
+
+- [Design brief](docs/BRIEF.md) — what this repo should contain and why.
+
+The contract is extracted from a working reference implementation (the
+`claude-code-llm-wiki` Stage B enrichment pipeline).

docs/BRIEF.md

@@ -0,0 +1,237 @@
+# Brief: `llm-ingestion-pipeline-security`
+
+**A reusable, minimal, dependency-light defensive layer for LLM *ingestion*
+pipelines — the write-time siblings of query-time chatbot guardrails.**
+
+Status: brief / pre-implementation. This document defines what the repo should
+contain and why. No code yet.
+
+---
+
+## 1. One-line purpose
+
+Give any pipeline that runs untrusted content through a large language model and
+then **persists the result into a downstream-consumed artifact** (RAG corpus,
+knowledge base, wiki, embeddings store) a small, framework-agnostic library that
+packages the *architectural contract* for doing it safely: sanitize → fence →
+tool-less quarantined transform → capability isolation → scan output before
+commit → fail-secure.
+
+## 2. The problem, stated precisely
+
+The mature tooling for prompt injection is **query-time**: it sits between a user
+(or an agent) and a model, at inference, on live traffic. Ingestion is a
+different shape and is underserved:
+
+- **Direction.** The danger flows from *content* (documents, changelogs, scraped
+  web pages, user uploads, tickets) through an LLM *enrichment/summarization/
+  extraction/classification* step, into a **persisted artifact** that other
+  systems and agents later read.
+- **Two distinct failure modes**, only the first of which query-time guardrails
+  address:
+  1. **Injection steers the enrichment step** — untrusted content contains
+     instructions that hijack the summarizer/extractor.
+  2. **Poisoned content is published** — the artifact itself becomes the attack.
+     A verbatim-quoted payload, or an LLM-emitted instruction, lands in the
+     knowledge base and later poisons a *downstream* agent's context. This is
+     RAG/memory poisoning committed at *write* time, and a query-time guardrail
+     on the downstream reader never sees where it came from.
+- **Non-interactive, unattended.** Ingestion typically runs headless on a
+  schedule. There is no human in the loop to notice a weird answer. Failure
+  discipline (fail-secure, alerting, no silent verbatim fallback under attack)
+  matters more here than in a chat UI.
+
+## 3. Prior art and honest positioning
+
+Injection *detection* is a crowded space. This repo must not pretend otherwise.
+
+| Tool | Orientation |
+|------|-------------|
+| LLM Guard, Guardrails AI, NeMo Guardrails | Query-time I/O guardrails / dialogue policy |
+| Rebuff | Self-hardening query-time detector (vector store of past attacks) |
+| Vigil, Lakera Guard, Vijil | Query-time injection classifiers / detection servers |
+| LlamaFirewall (Meta) | Agent guardrail framework (I/O, tool calls) |
+| Resk | Python guardrail lib for LLM API calls |
+| CleanBase, DataFilter (research) | Detecting/cleaning malicious docs in RAG DBs |
+
+The gap this repo fills is **not** "no one detects injection." It is:
+
+> A small **library** (not a hosted service, not a fine-tuned model) that
+> packages the **write-time ingestion contract** — quarantined tool-less
+> transform, per-stage capability isolation, output-scan-before-persist,
+> fail-secure disposition — as composable, framework-agnostic, minimal-dependency
+> code, with corpus-aware false-positive control.
+
+**Necessary honesty (verified, see §11):** 2025 research shows character-level
+and adversarial-ML evasion defeats most pattern- and classifier-based detectors
+while keeping the payload legible to the target model. Therefore the value of
+this library is **architecture and defense-in-depth**, not a detector that
+"solves" injection. The lexicon is one WARN-class layer; the *contract* (a
+tool-less call that cannot act on a successful injection, credentials the
+hijacked step never holds, output scanned before it is trusted) is the load-
+bearing part. The brief must lead with the contract, not the regex.
+
+## 4. Design principles
+
+1. **Minimal dependencies.** Stdlib-first core. Optional pluggable detectors
+   (an embedding classifier, an LLM-judge) live behind extras, never in the core
+   path. A pipeline should be able to adopt the core with zero new runtime deps.
+2. **Framework-agnostic.** Works with any SDK (Anthropic, OpenAI, local). The
+   library never makes the model call; it hardens the call the pipeline makes.
+3. **Pure functions.** Detection is `text -> findings`. No network, no telemetry,
+   no global state. Portable, testable, auditable.
+4. **Disposition belongs to the caller.** The library reports; the pipeline
+   decides WARN vs quarantine-review vs fail-secure-halt. Defaults are provided,
+   but blocking is never imposed — see principle 5.
+5. **Corpus-aware false-positive control.** A corpus that *legitimately discusses*
+   prompt injection (security docs, changelogs, a wiki about AI safety) will trip
+   any broad lexicon. The library must make WARN-not-block the easy default and
+   hard-fail an explicit, calibrated opt-in. Silent over-blocking of legitimate
+   content is a failure mode, not a safe default.
+6. **Fail-secure under compound signals.** Injection-scan hit *and* a transform
+   failure is treated as a probable forced-fallback attack: halt + alert, never
+   an auto-committed verbatim entry.
+
+## 5. Proposed module layout
+
+```
+src/llm_ingest_security/
+  sanitize/      Carrier stripping: invisible chars (Unicode Tags, zero-width,
+                 bidi), HTML comments, data: URIs. Byte-identical on clean input;
+                 removes only, never rewrites. Per-class counts for a WARN gate.
+  fence/         Spotlighting helpers: wrap untrusted input in markers, and strip
+                 the markers FROM the input first so it cannot break out of its
+                 own fence.
+  lexicon/       Semantic injection patterns as DATA (regex + label + severity),
+                 tiered CRITICAL/HIGH/MEDIUM: override, spoofed headers/tags,
+                 identity redefinition, config attacks, NL-indirection
+                 (fetch-and-execute, send-to-external, read-dotfiles,
+                 extract-and-exfiltrate), sub-agent spawn, leetspeak/homoglyph,
+                 multi-language. Curated, versioned, re-syncable.
+  entropy/       base64/hex blob detection for smuggled encoded instructions or
+                 encoded exfil (complements secret-pattern scanning).
+  contract/      The quarantine asserters — the differentiator:
+                   - assert the model request carries NO tools
+                   - assert per-stage credential allowlist (the enrichment step
+                     sees exactly the one key it needs, and none of the publish
+                     credentials)
+                   - capability-isolation helpers (env scoping)
+  output/        Output-side scanner: run the lexicon + entropy over the model's
+                 OUTPUT (summary, verbatim quotes, extracted fields) BEFORE the
+                 artifact is persisted. This is the RAG-poisoning gate.
+  disposition/   Policy object: WARN | QUARANTINE_REVIEW | FAIL_SECURE per gate,
+                 plus the compound-signal fail-secure rule.
+  report/        Structured findings: class, count, severity, offset, source
+                 (input|output), disposition.
+```
+
+## 6. The reusable contract (adopt-this checklist)
+
+The actual product is this checklist, encoded as code a pipeline wires in order:
+
+1. **Sanitize before fence.** Strip carrier classes from untrusted input first.
+2. **Fence untrusted input.** Spotlight-mark it; strip markers from the payload.
+3. **Tool-less transform.** Call the model with zero tools. A successful
+   injection then has nothing to act with.
+4. **Per-stage capability isolation.** The enrichment stage holds only the model
+   key; the publish stage holds only the publish credential; no stage holds both.
+5. **Treat output as data.** Parse to a frozen schema; reject on structural
+   violation. The output never reaches a shell, git, or a filesystem path.
+6. **Scan output before persist.** Run the lexicon + entropy over the emitted
+   text. Verbatim-carried payloads and model-emitted instructions are caught here.
+7. **Fail-secure on compound signals.** Injection hit + transform failure = halt
+   + alert, never a silent verbatim commit.
+8. **Minimal alert payloads.** Alert with a gate code + run ID, never content.
+   Treat the alert channel/topic as a secret.
+
+## 7. Non-goals
+
+- **Not** a query-time chatbot / agent guardrail (that space is served).
+- **Not** a fine-tuned model or a required ML classifier — deterministic core;
+  ML detectors are optional, pluggable extras.
+- **Not** a hosted service or SaaS.
+- **Not** a silver bullet — explicitly a defense-in-depth layer whose lexicon is
+  bypassable in isolation; the contract is what carries the security.
+
+## 8. Language, packaging, licensing
+
+- **Python first** — most ingestion pipelines are Python. Stdlib-only core;
+  `pyproject.toml` with optional extras (`[ml]`, `[judge]`).
+- Framework-agnostic public API; no SDK imported by the core.
+- A Node/TypeScript port is a plausible follow-on (a shared JSON lexicon as the
+  single source of truth across both).
+- Permissive license (MIT or Apache-2.0) so it is droppable into any pipeline.
+
+## 9. Test strategy
+
+- **Adversarial corpus.** A seeded set of payloads (one per lexicon class,
+  including obfuscated and multi-language variants); measure and report *recall*.
+- **False-positive corpus.** Real content that legitimately discusses injection
+  (security docs, changelogs); assert the default disposition is WARN, not block,
+  and that hard-fail is an explicit opt-in.
+- **Sanitizer invariant.** Clean input returns byte-identical with an all-zero
+  report; the sanitizer only removes.
+- **Contract asserters.** A tool-carrying request and a credential-leaking stage
+  env both raise; the happy path passes.
+- **No network in any test.**
+
+## 10. Threat-model grounding
+
+- OWASP LLM Top 10: **LLM01 Prompt Injection**, output-handling, and the RAG /
+  knowledge-poisoning vectors.
+- Google DeepMind "AI Agent Traps" (Franklin et al., 2025): Latent Memory
+  Poisoning and RAG Knowledge Poisoning map directly onto the write-time artifact.
+- Research anchors: CleanBase (malicious docs in RAG DBs), DataFilter, and the
+  2025 guardrail-evasion literature that motivates the defense-in-depth framing.
+
+## 11. Verification log (prior-art claims + sources)
+
+Per the operator's verification duty — key claims in this brief and where they
+are grounded:
+
+- *Injection detection is crowded and query-time oriented* — LLM Guard, NeMo
+  Guardrails, Guardrails AI, Rebuff, Vigil, Vijil, LlamaFirewall, Resk, verified
+  via the 2026 guardrails comparison and tool docs.
+  - https://dev.to/agdex_ai/best-ai-agent-security-guardrails-tools-in-2026-llm-guard-vs-nemo-vs-guardrails-ai-5e5d
+- *Pattern/classifier detection alone is bypassable (defense-in-depth framing)* —
+  "Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks", arXiv
+  2504.11168 (character injection defeats most guardrails).
+  - https://arxiv.org/abs/2504.11168
+- *Write-time RAG-DB poisoning is a researched but under-tooled vector* —
+  CleanBase (detecting malicious documents in RAG knowledge databases).
+  - https://arxiv.org/pdf/2605.00460
+- *Multi-agent output-guard-before-release architecture exists in research* —
+  "A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks",
+  arXiv 2509.14285.
+  - https://arxiv.org/abs/2509.14285
+- *LlamaFirewall as an open-source guardrail reference* — arXiv 2505.03574.
+  - https://arxiv.org/pdf/2505.03574
+
+Marked **assumed, not verified**: the specific claim that no existing *library*
+packages the full write-time contract as minimal-dependency code. The search
+found no such library, but absence of evidence is not proof; a focused survey of
+PyPI + GitHub topics should confirm before the README makes a novelty claim.
+
+## 12. Reference implementation
+
+The contract in §6 is not hypothetical — it is extracted from a working pipeline:
+the `claude-code-llm-wiki` Stage B enrichment path (`tools/wiki_ingest/`),
+which already implements sanitize + spotlight-fence + tool-less quarantined SDK
+call + per-stage credential isolation + schema-validated output + fail-secure
+`ForcedFallbackHalt`. That pipeline is the reference implementation and the first
+consumer; this repo generalizes its pattern for reuse. The semantic lexicon seed
+is the `injection-patterns.mjs` table from the `llm-security` plugin (a pure
+regex+label+severity dataset, portable as data).
+
+## 13. Open decisions for the operator
+
+1. **Name** — `llm-ingestion-pipeline-security` is descriptive but long; a
+   shorter package name (e.g. `ingestguard`) may be worth it before first publish.
+2. **License** — MIT vs Apache-2.0.
+3. **Relationship to `llm-security`** — sibling repos sharing one lexicon dataset
+   as the single source of truth, or fully independent? Avoid two drifting copies
+   of the pattern table.
+4. **Publish target** — Forgejo (`git.fromaitochitta.com`), and whether a public
+   `open/` mirror. No GitHub per house policy.
+5. **Python-only vs polyglot** from day one.
+6. **Maintenance model** — solo fork-and-own (as `llm-security`), or open to PRs.