Metadata-Version: 2.5
Name: llm-ingestion-guard
Version: 1.4.1
Summary: Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
Author: Kjell Tore Guttormsen
License: MIT License
        
        Copyright (c) 2026 Kjell Tore Guttormsen
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: guardrails,ingestion,llm,prompt-injection,rag,security,write-time
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: openpyxl>=3.1; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: python-docx>=1.2; extra == 'dev'
Requires-Dist: python-pptx>=1.0; extra == 'dev'
Provides-Extra: judge
Provides-Extra: ml
Description-Content-Type: text/markdown

# llm-ingestion-guard

Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.

![Version](https://img.shields.io/badge/version-1.4.1-blue)
![Status](https://img.shields.io/badge/status-stable-brightgreen)
![Python](https://img.shields.io/badge/python-3.10%2B-purple)
![License](https://img.shields.io/badge/license-MIT-lightgrey)

**Write-time ingestion is the trust boundary that query-time guardrails
structurally cannot see.** When untrusted content passes through an LLM
enrichment/summarization/extraction step into a *persisted* artifact — a RAG
corpus, a knowledge base, an LLM wiki — the poisoned result is read *later* by a
downstream agent as **trusted context**. That agent's guardrail never sees where
the content came from. The only place the provenance still exists is the write.

This library packages that write-time contract — sanitize → fence → tool-less
quarantined transform → per-stage capability isolation → scan-before-commit →
fail-secure — as composable, stdlib-first, framework-agnostic code. It is the
write-time **sibling** of query-time tools (LLM Guard, NeMo Guardrails, Rebuff,
Vigil), not a competitor: those harden material as it enters the model; this
hardens it as it is committed for a *later* reader. Where existing OSS tooling is
mostly single-stage *detectors* — a risk verdict, with quarantine, capability
isolation, scan-before-persist, and fail-secure left to the integrator — this
packages the full contract as code. (Neighbours surveyed in
[`docs/BRIEF.md`](docs/BRIEF.md) §11.)

**Why an LLM wiki (e.g. Google OKF) needs this specifically.** OKF and
second-brain formats have no schema registry, no central authority, and no
signing — a bundle's claimed origin is not verifiable at the format level. So
*your ingestion pipeline is the trust boundary*: provenance must be stamped by you
at write time, never assumed from the format. Any pipeline ingesting external data
into an agent-read store has this shape; an OKF wiki is its canonical form — which
is why the guard ships a first-class OKF adapter (below).

**Status:** `v1.4.1`. The stdlib-only core — its detector, contract, and
OKF-adapter modules plus the top-level wiring — is built and tested, exercised by
an end-to-end showcase and adversarial + false-positive corpora. The exported
Python surface is now frozen under semver: nothing exported is removed, renamed or
given a different meaning without a `2.0.0`. **Detection behaviour is not frozen** —
severities, thresholds and lexicon entries are calibration and move in `1.x`. There
are real limitations, stated plainly below; read them.

## Table of Contents

- [Install](#install)
- [Quickstart — the two bookends](#quickstart--the-two-bookends)
- [OKF / LLM-wiki support (shipped)](#okf--llm-wiki-support-shipped)
- [What it protects against](#what-it-protects-against)
- [The reusable contract (adopt-this checklist)](#the-reusable-contract-adopt-this-checklist)
- [Known limitations](#known-limitations)
- [Non-goals](#non-goals)
- [Design & threat model](#design--threat-model)
- [License](#license)

## Install

Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:

```bash
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v1.4.1"
```

The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or
other credential. The core is stdlib-only with zero dependencies, so nothing else
resolves. Optional ML/judge detectors live behind extras (`[ml]`, `[judge]`) and
are not required — the core is deterministic and dependency-free.

**Tags are the stable contract.** Every release is version-synced before tagging,
and published tags are never moved.

**Verify it yourself — nothing runs the suite automatically.** There is no CI
runner on the forge, so the test count is not something a badge can honestly
assert. From a clean clone:

```bash
pip install -e ".[dev]" && pytest        # the whole suite
```

Two consequences worth knowing before you depend on this:

- A git URL is a PEP 508 *direct reference*: it pins one exact tag, not a range
  like `>=1.0,<2.0`. Real range pinning — and therefore automatic pickup of patch
  releases — arrives with a Forgejo PyPI registry, which becomes the durable
  channel at the first patch release or the second downstream consumer, whichever
  comes first. The distribution name (`llm-ingestion-guard`) and the version
  scheme are unchanged by that move, so pins written today keep their meaning.
- **Vendoring the source into a consumer is not supported.** It severs the patch
  channel that a shared security dependency exists to provide: a copied guard
  keeps running the vulnerabilities the original has already fixed.

## Quickstart — the two bookends

The library never makes the model call itself. It gives you the two library-side
halves around your own **tool-less** transform:

```python
from llm_ingestion_guard import (
    prepare_input, screen_output, Disposition, PRESET_USER_UPLOAD,
)

prepared = prepare_input(untrusted_content)              # sanitize + fence
enriched = your_model(prepared.fenced)                   # tool-less — YOUR call
decision = screen_output(enriched, PRESET_USER_UPLOAD)   # scan + dispose

if decision.disposition is Disposition.FAIL_SECURE:
    alert(gate_code=decision.reasons)                    # minimal payload, no content
    raise SystemExit                                     # halt — never persist
```

`screen_output` fails **closed**: if the scanner itself errors on crafted input,
the disposition is `FAIL_SECURE`, never a silent persist. Pass
`transform_failed=True` when your model call raised or fell back — a scan hit
together with a transform failure is treated as a probable forced-fallback attack
and halts regardless of trust tier.

**New in `v0.4.0`, and the reason for the major-line bump.** `prepare_input` fails
**closed** on size too: above `MAX_INPUT_CHARS` (1 000 000) it raises
`OversizeInputError`, a `ContractViolation` subclass, rather than returning a
half-sanitized document. If you call `sanitize` / `fence` / `neutralize` on
documents that large, this is the upgrade that needs a `try` — everything else in
0.4.0 is additive. The scanners bound their work differently: they read a prefix
and flag, which costs only detection in the tail. A transform returns *content*,
where the same move would either drop your data silently or hand back an
untransformed tail — the exact place an attacker would put the payload. Catch it
where you catch your other ingest refusals; the exception carries sizes and the
refusing surface, never any of the input.

Every primitive is also exported for pipelines that compose the checklist
themselves — `sanitize`, `scan_lexicon`, `scan_entropy`, `scan_output`,
`scan_active_content`, `neutralize`, the `decide` / `guard` disposition
machinery, and the contract asserters `assert_tool_less` /
`assert_credential_allowlist` / `scoped_env`. See
[the end-to-end showcase](tests/test_showcase.py) for a full worked pipeline.

## OKF / LLM-wiki support (shipped)

For a bundle-shaped store, the `okf` submodule sits **on top of** the
format-agnostic core: it knows OKF structure (frontmatter, paths, links,
`resource`, bundles) and feeds scannable regions into the same `sanitize` /
`scan_output` / disposition machinery — no YAML/format awareness leaks into the
core. Two ingestion modes:

- **(a) Own enrichment output** — your agent writes concepts. Run the two bookends
  above per concept before commit.
- **(b) Received external bundle** — you merge a whole third-party OKF bundle.
  `import_bundle` iterates concept-by-concept and runs the full per-concept gate:

```python
from llm_ingestion_guard.okf import import_bundle, Origin, Channel

# bundle: {concept_path -> raw document text}, e.g. {"tables/users.md": "---\n..."}
result = import_bundle(bundle, origin=Origin.EXTERNAL, channel=Channel.AUTOMATIC)

for c in result.concepts:
    if c.error:            # hard reject: bad path, unsafe frontmatter, non-https resource
        skip(c.path)       # disposition is FAIL_SECURE; do not merge this concept
# result.disposition = most-severe across concepts
# result.links       = the cross-link graph (dangling/rejected/resolved edges)
# result.log()       = the log.md body (one provenance-stamped line per concept)
```

Per-concept gates: **path / reserved-name** (rejects `..` traversal and reserved
`index.md`/`log.md` shadowing); **frontmatter parse-safety** — a strict,
reject-by-default loader that refuses anchors, aliases, and explicit tags *by
construction*, so a billion-laughs alias expansion or a `!!python/object` coercion
cannot occur (it is deliberately **not** a general YAML engine, whose own features
are the attack surface). A mapping has four carriers: the flow mapping as a value
(`generated: { by: x, at: y }`) or as a block-list item, a flow sequence of flow
mappings (`sources: [{ id: a, resource: x }]`), and a block sequence of block
mappings (SPEC §5.1's own form). Every carrier is admitted key-by-key against a
nine-name allowlist (`by`, `at`, `from`, `to`, `id`, `title`, `author`,
`usage_count`, `last_modified`) with plain-scalar leaves only; `resource` and
`usage_window` are admitted *inside a `sources` entry only*, so an `executor` or
`attester` `resource` is refused through every carrier. A key off that list, a
nested collection or a duplicate key is refused; a top-level block mapping and
the dotted and inline-colon routes to a mapping still raise. A
*sequence* value has two carriers — the block list, and (as of `1.4.0`) the flow
sequence `tags: [a, b, c]`, which is SPEC §4.1's own skeleton — whose elements
are either all plain scalars or all flow mappings, never a mix. A scalar element
carrying any of `{ } [ ] , " '` is refused rather than guessed at, and so is a
`#` or `:` where YAML reads it (a `#` opening the element or following
whitespace, a `:` opening or ending it or preceding whitespace); since `1.4.1`
`[/docs/a#anchor, vscode://x, https://e.com:8443/a]` parses, as it does in
YAML. See
[LIMITATIONS](docs/LIMITATIONS.md) for what that admits and what it still walls
off; **`resource` https-allowlist** (hard-rejects
`data:`/`javascript:`/`file:` before commit — a reject-gate, not defang — on the
top-level `resource` only: a `sources[].resource` is never URL-validated, because
SPEC §5.1 allows bundle-relative paths there, so a consumer that follows it calls
`validate_resource_url` itself);
**whole-concept scan** (frontmatter *values* + body through `scan_output`);
**cross-link graph** (surfaces dangling targets, the dormant-injection signal, and
rejects bundle-escaping links); **provenance stamping** (`Origin` × `Channel` →
tier + disposition, emitted to `log.md`).

## What it protects against

Concrete attack classes, grouped by OWASP LLM Top-10 (2025) anchor. Every row is
driven by a **live payload** in the coverage matrix — run it to watch all 134 pass
in your own environment:

```bash
python -m llm_ingestion_guard.coverage    # 130/130 classes; exit 0 = all as documented
```

| Anchor | Attack classes it stops (representative) |
|---|---|
| **LLM01 · prompt injection** | 83 instruction-override lexicon classes (ignore / forget / disregard / suspend-constraints, role-play, jailbreak, …); hidden carriers — zero-width stego, BIDI override, Unicode-tag stego, HTML-comment, `data:` URI — stripped on input **and** re-checked at the persist gate; high-entropy base64/hex blobs; an injection hidden inside a base64 blob (decoded, then re-scanned) |
| **LLM02 · sensitive-info disclosure** | Secret egress in output — AWS keys, tokens, private keys, the credential set; a base64-*wrapped* secret (decoded → `decoded:egress:*`) |
| **LLM05 · improper output handling** | Zero-click exfil carriers (EchoLeak, CVE-2025-32711) — markdown-image auto-fetch, inline / reference / autolink links, raw active HTML, standalone `data:` URIs; non-`https` `resource` URLs |
| **LLM06 · excessive agency** | A tool surface on the quarantined transform; credential use beyond the per-stage allowlist; capability isolation (`scoped_env`) |
| **LLM10 · fail-secure** | Forced-fallback attack (scan hit + transform failure → halt); compound weak-signal escalation; an un-scannable artifact fails **closed** — never a silent persist |
| **OKF structure (T1–T6)** | Body + frontmatter-value injection; YAML anchor / merge / nested / block DoS (parse-safety); `resource` allowlist; path traversal / reserved-name shadow; cross-link graph; provenance stamping |
| **Container / upload layer** | CSV/XLSX formula-injection (lead `= + - @`); zip-slip path escape; zip-bomb size cap; symlink refusal *(upload front-end — see the showcase)* |

The manifest ([`coverage.py`](src/llm_ingestion_guard/coverage.py)) is the single
source of truth for
[`tests/test_coverage_matrix.py`](tests/test_coverage_matrix.py), which asserts
total recall over every class, that every documented gap still holds (a closed gap
fails the test), and that every lexicon pattern has a case — so the matrix cannot
fall behind the code. The four classes it deliberately does **not** stop are called
out in [Known limitations](#known-limitations).

## The reusable contract (adopt-this checklist)

The actual product is this checklist, encoded as code you wire in order:

1. **Sanitize before fence.** Strip carrier classes (zero-width, BIDI,
   Unicode-tag, HTML comment, `data:`) from untrusted input first.
2. **Fence untrusted input.** Spotlight-mark it in a randomized per-call
   delimiter; strip attacker fence markers from the payload.
3. **Tool-less transform.** Call the model with zero tools. A successful
   injection then has nothing to act with.
4. **Per-stage capability isolation.** The enrichment stage holds only the model
   key; the publish stage holds only the publish credential; no stage holds both.
5. **Treat output as data.** Parse to a frozen schema; reject on structural
   violation. The output never reaches a shell, git, or a filesystem path.
6. **Scan output before persist.** Run the lexicon + entropy + active-content
   scan over the emitted text. Verbatim-carried payloads, model-emitted
   instructions, and zero-click exfil carriers (EchoLeak-class markdown
   images/links, raw active HTML, `data:` URIs) are caught here; `neutralize`
   additionally defangs them, opt-in.
7. **Fail-secure on compound signals.** Injection hit + transform failure = halt
   + alert, never a silent verbatim commit.
8. **Minimal alert payloads.** Alert with a gate code + run ID, never content.

Steps 1-2 are `prepare_input`; steps 6-7 are `screen_output`; steps 3-5 are
yours; the contract asserters harden step 3-4.

## Known limitations

Conceding these plainly is itself a control — it prevents the false assurance that
a green scan means safe content. The highest-impact items:

- **The contract carries the security, not the lexicon.** Pattern detection is
  bypassable in isolation (character-injection, novel phrasings); the tool-less
  transform + capability isolation + fail-secure are the wall.
- **Semantic / factual poisoning is invisible.** A false claim in clean prose
  carries no suspicious token — the highest-impact gap for a wiki. Needs a `[judge]`
  plugged into the `grounding` seam.
- **Text-only, extracted-text-only.** The core parses no files; OCR-embedded
  instructions, macros, and multimodal stego are out of scope. `.pdf` is refused as
  unsupported, not half-scanned.
- **A lone HIGH in *trusted* prose disposes to WARN**, and **insider in-place
  edits** are outside the untrusted-content threat model — run genuinely untrusted
  sources as untrusted.
- **Active-content severity grades on URL shape, not construct type.** A URL that
  only *names* a remote document is LOW; one that can carry a value outward keeps
  HIGH/MEDIUM. The conceded hole: a bare-path image on a hostile host still *fetches*
  when rendered, so pure beaconing (reader IP, timing) is not graded.
- **Six documented gaps** the coverage matrix keeps honest: hex-wrapped secret
  egress, semantic poisoning, trusted-prose lone-HIGH, lexicon dedup (`count=1`),
  pure beaconing, and short opaque URL segments.
- **The upload door is a review queue, not an auto-persist path — measured.** On
  three benign document populations, `PRESET_USER_UPLOAD` disposed **98 of 185**
  (53.0%), **88 of 547** (16.1%) and **133 of 389** (34.2%) documents to something
  other than WARN. Technical documentation is the expensive case: it is dense in the
  exact constructs the gate grades. Budget human review, or run a source you actually
  trust as trusted. Re-run it yourself with [`docs/fp-sweep.py`](docs/fp-sweep.py).
  Those three are the published pre-0.6.0 numbers; the raw-HTML narrowing in 0.6.0
  moves them to **108**, **98** and **88** measured against current corpus state —
  two of the three corpora are living, so the cells are not rewritten in place.
  Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).

**Full list — 45 items, each with the mechanism, plus the out-of-scope boundary:**
[`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
consumer corpora, including the false positives the URL-shape rule actually produces.

## Non-goals

Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal
steganography; query-time / runtime guardrails; semantic factuality verification.

## Design & threat model

- [Design brief](docs/BRIEF.md) — what this repo contains and why.
- [URL-shape rule](docs/URL-SHAPE.md) — `is_ordinary_url` stated precisely enough to
  reconstruct, with worked examples asserted against the implementation. **Read this
  before reasoning about the rule from prose:** three consumers reconstructed it from
  summaries and each produced a different wrong number on a real corpus.
- [Build plan](docs/PLAN.md) — module build order and the reuse map.
- [Adoption brief](docs/ADOPTION-BRIEF.md) — wiring the guard into an OKF
  second-brain / LLM wiki, and a checklist for *when* to include it.

The contract is extracted from a working reference implementation (the
`claude-code-llm-wiki` Stage B enrichment pipeline). Threat-model anchors: OWASP
LLM Top-10 2025 (LLM01/02/04/05/06 strongest, LLM08 boundary, LLM09/10),
PoisonedRAG, guardrail-evasion (arXiv 2504.11168), EchoLeak (CVE-2025-32711).

## License

MIT — see [LICENSE](LICENSE).
