llm-ingestion-guard

Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.

Write-time ingestion is the trust boundary that query-time guardrails structurally cannot see. When untrusted content passes through an LLM enrichment/summarization/extraction step into a persisted artifact — a RAG corpus, a knowledge base, an LLM wiki — the poisoned result is read later by a downstream agent as trusted context. That agent's guardrail never sees where the content came from. The only place the provenance still exists is the write.

This library packages that write-time contract — sanitize → fence → tool-less quarantined transform → per-stage capability isolation → scan-before-commit → fail-secure — as composable, stdlib-first, framework-agnostic code. It is the write-time sibling of query-time tools (LLM Guard, NeMo Guardrails, Rebuff, Vigil), not a competitor: those harden material as it enters the model; this hardens it as it is committed for a later reader. Where existing OSS tooling is mostly single-stage detectors — a risk verdict, with quarantine, capability isolation, scan-before-persist, and fail-secure left to the integrator — this packages the full contract as code. (Neighbours surveyed in docs/BRIEF.md §11.)

Why an LLM wiki (e.g. Google OKF) needs this specifically. OKF and second-brain formats have no schema registry, no central authority, and no signing — a bundle's claimed origin is not verifiable at the format level. So your ingestion pipeline is the trust boundary: provenance must be stamped by you at write time, never assumed from the format. Any pipeline ingesting external data into an agent-read store has this shape; an OKF wiki is its canonical form — which is why the guard ships a first-class OKF adapter (below).

Status: v1.6.0. The stdlib-only core — its detector, contract, and OKF-adapter modules plus the top-level wiring — is built and tested, exercised by an end-to-end showcase and adversarial + false-positive corpora. The exported Python surface is now frozen under semver: nothing exported is removed, renamed or given a different meaning without a 2.0.0. Detection behaviour is not frozen — severities, thresholds and lexicon entries are calibration and move in 1.x. There are real limitations, stated plainly below; read them.

Table of Contents

• Install
• Quickstart — the two bookends
• OKF / LLM-wiki support (shipped)
• What it protects against
• The reusable contract (adopt-this checklist)
• Known limitations
• Non-goals
• Design & threat model
• License

Install

Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:

pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.6.0"

Or take a version range from the Forgejo package index, which carries 1.6.0, 1.5.0, 1.4.2, 1.4.1 and 1.1.0 (1.2.0–1.4.0 were never published there):

pip install --index-url https://git.fromaitochitta.com/api/packages/open/pypi/simple/ "llm-ingestion-guard>=1.6.0,<2"

The open/ mirror is anonymously readable, so CI needs no deploy key, token, or other credential. The core is stdlib-only with zero dependencies, so nothing else resolves. Optional ML/judge detectors live behind extras ([ml], [judge]) and are not required — the core is deterministic and dependency-free.

Tags are the stable contract. Every release is version-synced before tagging, and published tags are never moved.

Verify it yourself — nothing runs the suite automatically. There is no CI runner on the forge, so the test count is not something a badge can honestly assert. From a clean clone:

pip install -e ".[dev]" && pytest        # the whole suite

Two consequences worth knowing before you depend on this:

• A git URL is a PEP 508 direct reference: it pins one exact tag, not a range like >=1.0,<2.0. Range pinning — and therefore automatic pickup of patch releases — comes from the Forgejo package index above, the durable channel from 1.4.1 (the first 1.x patch release) on. The distribution name (llm-ingestion-guard) and the version scheme are the same on both channels, so a tag pin written earlier keeps its meaning.
• Vendoring the source into a consumer is not supported. It severs the patch channel that a shared security dependency exists to provide: a copied guard keeps running the vulnerabilities the original has already fixed.

Quickstart — the two bookends

The library never makes the model call itself. It gives you the two library-side halves around your own tool-less transform:

from llm_ingestion_guard import (
    prepare_input, screen_output, Disposition, PRESET_USER_UPLOAD,
)

prepared = prepare_input(untrusted_content)              # sanitize + fence
enriched = your_model(prepared.fenced)                   # tool-less — YOUR call
decision = screen_output(enriched, PRESET_USER_UPLOAD)   # scan + dispose

if decision.disposition is Disposition.FAIL_SECURE:
    alert(gate_code=decision.reasons)                    # minimal payload, no content
    raise SystemExit                                     # halt — never persist

screen_output fails closed: if the scanner itself errors on crafted input, the disposition is FAIL_SECURE, never a silent persist. Pass transform_failed=True when your model call raised or fell back — a scan hit together with a transform failure is treated as a probable forced-fallback attack and halts regardless of trust tier.

New in v0.4.0, and the reason for the major-line bump. prepare_input fails closed on size too: above MAX_INPUT_CHARS (1 000 000) it raises OversizeInputError, a ContractViolation subclass, rather than returning a half-sanitized document. If you call sanitize / fence / neutralize on documents that large, this is the upgrade that needs a try — everything else in 0.4.0 is additive. The scanners bound their work differently: they read a prefix and flag, which costs only detection in the tail. A transform returns content, where the same move would either drop your data silently or hand back an untransformed tail — the exact place an attacker would put the payload. Catch it where you catch your other ingest refusals; the exception carries sizes and the refusing surface, never any of the input.

Every primitive is also exported for pipelines that compose the checklist themselves — sanitize, scan_lexicon, scan_entropy, scan_output, scan_active_content, neutralize, the decide / guard disposition machinery, and the contract asserters assert_tool_less / assert_credential_allowlist / scoped_env. See the end-to-end showcase for a full worked pipeline.

OKF / LLM-wiki support (shipped)

For a bundle-shaped store, the okf submodule sits on top of the format-agnostic core: it knows OKF structure (frontmatter, paths, links, resource, bundles) and feeds scannable regions into the same sanitize / scan_output / disposition machinery — no YAML/format awareness leaks into the core. Two ingestion modes:

• (a) Own enrichment output — your agent writes concepts. Run the two bookends above per concept before commit.
• (b) Received external bundle — you merge a whole third-party OKF bundle. import_bundle iterates concept-by-concept and runs the full per-concept gate:

from llm_ingestion_guard.okf import import_bundle, Origin, Channel

# bundle: {concept_path -> raw document text}, e.g. {"tables/users.md": "---\n..."}
result = import_bundle(bundle, origin=Origin.EXTERNAL, channel=Channel.AUTOMATIC)

for c in result.concepts:
    if c.error:            # hard reject: bad path, unsafe frontmatter, non-https resource
        skip(c.path)       # disposition is FAIL_SECURE; do not merge this concept
# result.disposition = most-severe across concepts
# result.links       = the cross-link graph (dangling/rejected/resolved edges)
# result.log()       = the log.md body (one provenance-stamped line per concept)

Per-concept gates: path / reserved-name (rejects .. traversal and reserved index.md/log.md shadowing); frontmatter parse-safety — a strict, reject-by-default loader that refuses anchors, aliases, and explicit tags by construction, so a billion-laughs alias expansion or a !!python/object coercion cannot occur (it is deliberately not a general YAML engine, whose own features are the attack surface). A mapping has four carriers: the flow mapping as a value (generated: { by: x, at: y }) or as a block-list item, a flow sequence of flow mappings (sources: [{ id: a, resource: x }]), and a block sequence of block mappings (SPEC §5.1's own form). Every carrier is admitted key-by-key against a nine-name allowlist (by, at, from, to, id, title, author, usage_count, last_modified) with plain-scalar leaves only; resource and usage_window are admitted inside a sources entry only, so an executor or attester resource is refused through every carrier. A key off that list, a nested collection or a duplicate key is refused; a top-level block mapping and the dotted and inline-colon routes to a mapping still raise. A sequence value has two carriers — the block list, and (as of 1.4.0) the flow sequence tags: [a, b, c], which is SPEC §4.1's own skeleton — whose elements are either all plain scalars or all flow mappings, never a mix. A scalar element carrying any of { } [ ] , " ' is refused rather than guessed at, and so is a # or : where YAML reads it (a # opening the element or following whitespace, a : opening or ending it or preceding whitespace); since 1.4.1 [/docs/a#anchor, vscode://x, https://e.com:8443/a] parses, as it does in YAML. See LIMITATIONS for what that admits and what it still walls off; resource https-allowlist (hard-rejects data:/javascript:/file: before commit — a reject-gate, not defang — on the top-level resource only: a sources[].resource is never URL-validated, because SPEC §5.1 allows bundle-relative paths there, so a consumer that follows it calls validate_resource_url itself); whole-concept scan (frontmatter values + body through scan_output); cross-link graph (surfaces dangling targets, the dormant-injection signal, and rejects bundle-escaping links); provenance stamping (Origin × Channel → tier + disposition, emitted to log.md).

What it protects against

Concrete attack classes, grouped by OWASP Top 10 for LLM Applications 2026 anchor. Numbers carry the year, OWASP's own form, because the 2026 edition renumbered the list; a class no entry describes carries no number. Every row is driven by a live payload in the coverage matrix — run it to watch all 145 pass in your own environment:

python -m llm_ingestion_guard.coverage    # 139/139 classes; exit 0 = all as documented

Anchor	Attack classes it stops (representative)
LLM01:2026 · prompt injection	83 instruction-override lexicon classes (ignore / forget / disregard / suspend-constraints, role-play, jailbreak, …); hidden carriers — every invisible codepoint in Unicode 18.0 (zero-width, BIDI override, Unicode-tag, variation-selector and Sneaky-Bits smuggling, private use; legitimate emoji, CJK, Arabic-script, Indic and right-to-left uses exempt on context), HTML-comment, data: URI — stripped on input and re-checked at the persist gate (a comment the model leaves in its artifact is reported unless it sits in code); text hidden by styling — a font too small to read, or text coloured like its own background; high-entropy base64/hex blobs; an injection hidden inside a base64 blob (decoded, then re-scanned)
LLM02:2026 · sensitive-info disclosure	Secret egress in output — AWS keys, tokens, private keys, the credential set; a base64-wrapped secret (decoded → decoded:egress:*)
LLM10:2026 · improper output handling	Zero-click exfil carriers (EchoLeak, CVE-2025-32711) — markdown-image auto-fetch, inline / reference / autolink links, raw active HTML, standalone data: URIs
LLM03:2026 · excessive agency	A tool surface on the quarantined transform; credential use beyond the per-stage allowlist; capability isolation (scoped_env)
Fail-secure (the guard's own failure semantics)	Forced-fallback attack (scan hit + transform failure → halt); compound weak-signal escalation; an un-scannable artifact fails closed — never a silent persist
OKF structure (T1–T6)	Body + frontmatter-value injection; YAML anchor / merge / nested / block DoS (parse-safety); non-https / active-scheme resource URLs and the resource allowlist; path traversal / reserved-name shadow; cross-link graph; provenance stamping
Container / upload layer	CSV/XLSX formula-injection (lead = + - @); zip-slip path escape; zip-bomb size cap; symlink refusal (upload front-end — see the showcase)

The manifest (coverage.py) is the single source of truth for tests/test_coverage_matrix.py, which asserts total recall over every class, that every documented gap still holds (a closed gap fails the test), and that every lexicon pattern has a case — so the matrix cannot fall behind the code. The four classes it deliberately does not stop are called out in Known limitations.

The persisted store: OWASP ASI06 and MITRE ATLAS

The store this guard writes to is what the OWASP Top 10 for Agentic Applications for 2026 calls ASI06 · Memory & Context Poisoning: persistent corruption of agent memory, RAG stores or contextual knowledge that keeps acting long after the write. The rows below map it, and the MITRE ATLAS v2026.10 techniques that reach a write-time pipeline, onto what the guard does. The right-hand column is the part it does not stop. Each one was measured or is a documented gap.

Catalog entry	What the guard stops at the write	What it does not stop
ASI06 · Memory & Context Poisoning	Instruction-carrying and hidden-carrier content, before it is persisted (scan-before-persist); a trust tier from Origin × Channel; one provenance line per concept in log.md	A false claim in clean prose — semantic poisoning, the highest-impact gap
AML.T0070 · RAG Poisoning	Prompt injections and hidden carriers in a document headed for the index	The "false or misleading information" half — same gap
AML.T0080.000 · AI Agent Context Poisoning: Memory	A memory entry that carries instructions the lexicon or a carrier check sees	An entry that only steers in plain prose
AML.T0143 · LLM Prompt Propagation	Instructions that the lexicon or a carrier check sees are caught at the output gate, so they never reach the store that would pass them on	Propagation over a channel the guard does not sit on
AML.T0051.001 · LLM Prompt Injection: Indirect	The LLM01:2026 row above	Novel phrasings with no lexicon hit; the tool-less transform carries that case
AML.T0068 · LLM Prompt Obfuscation	Invisible characters, HTML comments, a font too small to read, text coloured like the background declared on the same element, base64 and rot13 (decoded, then re-scanned)	White text whose background sits on a parent element or in a stylesheet: a plain-prose instruction hidden that way gives 0 findings
AML.T0077 · LLM Response Rendering	The LLM10:2026 row above: markdown-image auto-fetch, links, raw active HTML	Pure beaconing to a bare path on a hostile host (a documented gap)
AML.T0066 · Retrieval Content Crafting	Only the crafted content that also carries an injection	The crafted content itself — semantic gap

Every catalog the guard cites is on its newest edition, with the date it was last checked and a link. The full list is the register in docs/threat-2026/catalogs.json, and tests/test_catalog_register.py fails on an entry without a date or a link. The main ones, checked 2026-10-31:

Catalog	Edition	Published
OWASP Top 10 for LLM Applications	2026	2026-08-04
OWASP Top 10 for Agentic Applications	for 2026	2025-12-09
MITRE ATLAS	v2026.10	2026-10-28
OWASP RAG Security Cheat Sheet	living, last changed 2026-10-04	2026-05-06
The Unicode Standard	18.0.0	2026-09-16
CVE-2025-32711 (EchoLeak)	NVD record, modified 2026-06-17	2025-06-11

The reusable contract (adopt-this checklist)

The actual product is this checklist, encoded as code you wire in order:

1. Sanitize before fence. Strip carrier classes (every invisible codepoint not exempt on its context, HTML comment, data:) from untrusted input first.
2. Fence untrusted input. Spotlight-mark it in a randomized per-call delimiter; strip attacker fence markers from the payload.
3. Tool-less transform. Call the model with zero tools. A successful injection then has nothing to act with.
4. Per-stage capability isolation. The enrichment stage holds only the model key; the publish stage holds only the publish credential; no stage holds both.
5. Treat output as data. Parse to a frozen schema; reject on structural violation. The output never reaches a shell, git, or a filesystem path.
6. Scan output before persist. Run the lexicon + entropy + active-content scan over the emitted text. Verbatim-carried payloads, model-emitted instructions, and zero-click exfil carriers (EchoLeak-class markdown images/links, raw active HTML, data: URIs) are caught here; neutralize additionally defangs them, opt-in.
7. Fail-secure on compound signals. Injection hit + transform failure = halt
   • alert, never a silent verbatim commit.
8. Minimal alert payloads. Alert with a gate code + run ID, never content.

Steps 1-2 are prepare_input; steps 6-7 are screen_output; steps 3-5 are yours; the contract asserters harden step 3-4.

Known limitations

Conceding these plainly is itself a control — it prevents the false assurance that a green scan means safe content. The highest-impact items:

• The contract carries the security, not the lexicon. Pattern detection is bypassable in isolation (character-injection, novel phrasings); the tool-less transform + capability isolation + fail-secure are the wall.
• Semantic / factual poisoning is invisible. A false claim in clean prose carries no suspicious token — the highest-impact gap for a wiki. Needs a [judge] plugged into the grounding seam.
• Text-only, extracted-text-only. The core parses no files; OCR-embedded instructions, macros, and multimodal stego are out of scope. .pdf is refused as unsupported, not half-scanned.
• A lone HIGH in trusted prose disposes to WARN, and insider in-place edits are outside the untrusted-content threat model — run genuinely untrusted sources as untrusted.
• Active-content severity grades on URL shape, not construct type. A URL that only names a remote document is LOW; one that can carry a value outward keeps HIGH/MEDIUM. The conceded hole: a bare-path image on a hostile host still fetches when rendered, so pure beaconing (reader IP, timing) is not graded.
• Six documented gaps the coverage matrix keeps honest: hex-wrapped secret egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (count=1), pure beaconing, and short opaque URL segments.
• The upload door is a review queue, not an auto-persist path — measured. On three benign document populations, PRESET_USER_UPLOAD disposed 98 of 185 (53.0%), 88 of 547 (16.1%) and 133 of 389 (34.2%) documents to something other than WARN. Technical documentation is the expensive case: it is dense in the exact constructs the gate grades. Budget human review, or run a source you actually trust as trusted. Re-run it yourself with docs/fp-sweep.py. Those three are the published pre-0.6.0 numbers; the raw-HTML narrowing in 0.6.0 moves them to 108, 98 and 88 measured against current corpus state — two of the three corpora are living, so the cells are not rewritten in place. Method and before/after: docs/rawhtml-census.py.

Full list — 48 items, each with the mechanism, plus the out-of-scope boundary: docs/LIMITATIONS.md. Several carry field measurements from consumer corpora, including the false positives the URL-shape rule actually produces.

Non-goals

Embedding/vector-layer defenses (OWASP LLM09:2026, downstream of persist); multimodal steganography; query-time / runtime guardrails; semantic factuality verification.

Design & threat model

• Design brief — what this repo contains and why.
• URL-shape rule — is_ordinary_url stated precisely enough to reconstruct, with worked examples asserted against the implementation. Read this before reasoning about the rule from prose: three consumers reconstructed it from summaries and each produced a different wrong number on a real corpus.
• Build plan — module build order and the reuse map.
• Adoption brief — wiring the guard into an OKF second-brain / LLM wiki, and a checklist for when to include it.

The contract is extracted from a working reference implementation (the claude-code-llm-wiki Stage B enrichment pipeline). Threat-model anchors: OWASP Top 10 for LLM Applications 2026 (LLM01:2026, LLM02:2026, LLM03:2026, LLM05:2026, LLM10:2026 strongest; LLM06:2026 self-safety; LLM09:2026 boundary); ASI06 from the OWASP Top 10 for Agentic Applications for 2026; MITRE ATLAS v2026.10 (the mapping above); the OWASP RAG Security Cheat Sheet (checked 2026-10-31); PoisonedRAG; guardrail evasion (arXiv 2504.11168); EchoLeak (CVE-2025-32711). Edition, check date and link for every source: the register.

License

MIT — see LICENSE.