llm-ingestion-pipeline-security/docs/ADOPTION-BRIEF.md

# Adoption brief — wiring `llm-ingestion-guard` into an OKF second-brain / LLM wiki

**Audience:** a repo that is building (or planning) an LLM wiki / second-brain —
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
to decide **when** and **where** to add a write-time ingestion guard.

**Status of the guard:** `v0.6.1` (alpha). Stdlib-only core, framework-agnostic.
Public API may still change. Read the known-limitations section before you rely
on it.

This brief is self-contained: you can plan an inclusion from it alone. Every
technical claim below is checkable against the guard repo (commands given inline).

---

## 1. What this is — and what it is *not*

`llm-ingestion-guard` is the **write-time** sibling of query-time chatbot
guardrails. It does **not** sit between a user and a model at query time (that is
LLM Guard / NeMo Guardrails / Rebuff / Vigil territory). It hardens the other
shape: **untrusted content flowing through an LLM enrichment/summarization/
extraction step into a *persisted, downstream-consumed* artifact** — a RAG corpus,
a knowledge base, a wiki, an OKF bundle.

Why this matters for a second-brain: a poisoned concept committed at **write**
time is later read by a *downstream* agent as **trusted context**. That agent's
query-time guardrail never sees where the concept came from. The write gate is the
only place the provenance is still known. **Your ingestion pipeline *is* the trust
boundary** — OKF has no schema registry, no central authority, and no signing, so
a received bundle's claimed origin is not verifiable at the format level.

The library never makes the model call itself. It gives you the two library-side
halves around your own **tool-less** transform, plus an OKF adapter for bundles.

## 2. The two bookends (the minimal integration)

```python
from llm_ingestion_guard import (
    prepare_input, screen_output, Disposition, PRESET_USER_UPLOAD,
)

prepared = prepare_input(untrusted_content)              # sanitize + fence
enriched = your_model(prepared.fenced)                   # tool-less — YOUR call
decision = screen_output(enriched, PRESET_USER_UPLOAD)   # scan + dispose

if decision.disposition is Disposition.FAIL_SECURE:
    alert(gate_code=decision.reasons)                    # minimal payload, no content
    raise SystemExit                                     # halt — never persist
```

`screen_output` fails **closed**: if the scanner itself errors on crafted input,
the disposition is `FAIL_SECURE`, never a silent persist. Pass
`transform_failed=True` when your model call raised or fell back — a scan hit
together with a transform failure is treated as a probable forced-fallback attack
and halts regardless of trust tier.

## 3. The reusable contract (the actual product — an adopt-this checklist)

The library is this checklist encoded as composable code you wire in order. Steps
1–2 are `prepare_input`; steps 6–7 are `screen_output`; steps 3–5 are yours (the
contract asserters harden 3–4):

1. **Sanitize before fence.** Strip carrier classes (zero-width, BIDI,
   Unicode-tag, HTML comment, `data:`) from untrusted input first.
2. **Fence untrusted input.** Spotlight-mark it in a randomized per-call delimiter;
   strip attacker fence markers from the payload.
3. **Tool-less transform.** Call the model with zero tools. A successful injection
   then has nothing to act with.
4. **Per-stage capability isolation.** The enrichment stage holds only the model
   key; the publish stage holds only the publish credential; no stage holds both.
5. **Treat output as data.** Parse to a frozen schema; reject on structural
   violation. Output never reaches a shell, git, or a filesystem path.
6. **Scan output before persist.** Lexicon + entropy + active-content scan over the
   emitted text (catches verbatim-carried payloads, model-emitted instructions,
   EchoLeak-class markdown images/links, raw active HTML, `data:` URIs).
7. **Fail-secure on compound signals.** Injection hit + transform failure = halt +
   alert, never a silent verbatim commit.
8. **Minimal alert payloads.** Alert with a gate code + run ID, never content.

You do not have to take all eight at once — every primitive is exported
(`sanitize`, `scan_lexicon`, `scan_entropy`, `scan_output`, `scan_active_content`,
`neutralize`, the `decide`/`guard` disposition machinery, and the contract
asserters `assert_tool_less` / `assert_credential_allowlist` / `scoped_env`).

## 4. The OKF adapter (for bundle-shaped ingestion)

If your second-brain is (or is converging on) OKF, the `okf` submodule sits **on
top of** the format-agnostic core: it knows OKF structure (frontmatter, paths,
links, `resource`, bundles) and feeds scannable regions into the same
`sanitize` / `scan_output` / disposition machinery. No YAML/format awareness leaks
into the core.

Two ingestion modes:

- **(a) Own enrichment output** — your agent writes concepts. Run the two bookends
  (§2) per concept before commit.
- **(b) Received external bundle** — you merge a whole third-party OKF bundle.
  `import_bundle` iterates concept-by-concept and runs the full per-concept gate:

```python
from llm_ingestion_guard.okf import import_bundle, Origin, Channel

# bundle: {concept_path -> raw document text}, e.g. {"tables/users.md": "---\n..."}
result = import_bundle(bundle, origin=Origin.EXTERNAL, channel=Channel.AUTOMATIC)

for c in result.concepts:
    if c.error:            # hard reject: bad path, unsafe frontmatter, non-https resource
        skip(c.path)       # disposition is FAIL_SECURE; do not merge this concept
# result.disposition = most-severe across concepts; result.links = cross-link graph
# result.log()          = the log.md body (one provenance-stamped line per concept)
```

Per-concept gates the adapter applies (each maps to a named control):

| Gate | What it does |
|---|---|
| **Path / reserved-name** | Rejects `..` traversal, absolute paths, and reserved-name shadowing. `validate_concept_path`. |
| **Frontmatter parse-safety** | `parse_frontmatter` is a strict, reject-by-default loader for the minimal OKF subset — anchors, aliases, and explicit tags are refused *by construction*, so billion-laughs alias DoS and `!!python/object` coercion cannot occur. It is deliberately **not** a general YAML engine (that engine's features *are* the attack surface). |
| **`resource` URL allowlist** | `validate_resource_url` hard-rejects non-`https` (`data:`/`javascript:`/`file:`) before commit — a reject-gate, not defang. |
| **Whole-concept scan** | Frontmatter *values* + body run through the core `scan_output`. |
| **Cross-link graph** | `link_graph` resolves in-bundle links, flags dangling targets (the dormant-injection signal), and rejects dangerous-scheme / bundle-escaping targets. |
| **Provenance stamping** | `Origin` × `Channel` → trust tier + disposition per concept, emitted to `log.md`. |

**Reserved files (`index.md` / `log.md`) — a deliberate mode difference.** In a
*received* bundle these are legitimate structure (directory listing, update log),
so `import_bundle` defaults to `allow_reserved=True`: it **scans their body and
frontmatter** (a directory listing is a high-priority injection surface) rather
than path-rejecting an otherwise-conformant third-party bundle. A front-end that
materialises **individual uploads** should pass `allow_reserved=False` instead —
there a reserved basename is a shadow of the listing and must be refused. Pick the
rule that matches your channel.

## 5. Verify what it stops (before you wire it in)

The guard ships a runnable coverage matrix — every vulnerability class it stops,
and the ones it deliberately does not, each row driving the **real** guard with a
live payload:

```bash
python -m llm_ingestion_guard.coverage    # exit 0 = all as documented
```

As of `v0.6.1`: **128 / 128 defended classes demonstrated (recall 100%)** and **6 /
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
update). The matrix is the single source of truth for the test suite (**736
passing**), which also asserts total recall, that every lexicon pattern has a
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,
symlink). Run it once; it tells you exactly what assurance you are buying.

## 6. How to depend on it

```bash
pip install llm-ingestion-guard          # stdlib-only core, zero dependencies
```

- **Core is stdlib-only** (`dependencies = []`), Python **3.10+**. Nothing to
  vet for supply-chain beyond the package itself; it parses no files and makes no
  network calls.
- **Optional extras**, none required: `[ml]` / `[judge]` (heavier detectors, e.g.
  a semantic-poisoning judge behind the `grounding` seam), `[dev]` (file-extraction
  libs used only by the dev-scoped upload showcase — `python-docx`/`python-pptx`/
  `openpyxl`/`lxml`/`Pillow`; never core dependencies).
- The core is `text -> findings`. If you ingest files, **extract text first**, then
  scan the extracted text with high-untrust upload provenance.

## 7. Planning checklist — *when* to include the guard

Score your ingestion pipeline. The guard earns its place at the **persist gate**
when the untrusted-ingest condition holds:

- [ ] You persist LLM-enriched or externally-received content into a store a
  *downstream* agent later reads as trusted (RAG / KB / wiki / OKF bundle).
- [ ] **At least one ingest path takes UNTRUSTED content** — an external URL, an
  uploaded file, a received third-party bundle, or auto-fetched web content.
  *(This is the decisive one.)*
- [ ] An LLM step (summarize / extract / classify / rewrite) sits between the
  untrusted source and the store.
- [ ] You want **fail-secure** (halt before persist) rather than best-effort
  detection with a silent commit on error.

**Where it applies vs. where it doesn't.** A second-brain that ingests primarily
the **user's own context** (onboarding writes conformant concepts, the user edits
their own notes) is a *first-party* path — the guard's untrusted-content threat
model does **not** target it, and trusted-author in-place edits are out of scope
by design. Wire the guard specifically at the **untrusted boundary**: a
"react-to-URL" command, an inbox that accepts external drops, a manual-import of a
foreign file, an auto-fetch of web/vendor content, or a received third-party OKF
bundle. Trust follows the data's **origin**, not the insertion channel — a manual
paste of external material is still external.

**When in your roadmap.** It is a persist-gate, so include it *before the first
untrusted ingest path goes live*, wired at the point where enriched content is
committed. If today you only have first-party ingest, note the guard as a
dependency to add when (not if) you open an external/inbox/received-bundle path.

## 8. Known limitations (read these — a green scan is not "safe")

Conceding these is itself a control. The full list is in the guard's `README.md`
("Known limitations"); the ones that matter most for a wiki/second-brain:

- **Semantic / factual poisoning is invisible** to lexicon + entropy: a
  plausible-but-wrong concept (wrong join-path, wrong metric, wrong runbook step)
  carries no suspicious token and passes clean. **Highest impact for a wiki.**
  Needs human review or source verification — the deterministic core does not judge
  semantics; a `[judge]` implementation plugs into the `grounding` seam.
- **Dormant / broken-link injection**: a link to a not-yet-existing target passes a
  per-concept write-time scan; the payload is planted later when that target is
  written. `link_graph` surfaces the *dangling* edge as the signal, but whether to
  block is your disposition call, and cross-write re-scan over time is your
  responsibility.
- **A document that *describes* attacks is a false positive.** Security notes that
  legitimately document injection payloads trip carrier-strip / fail-secure. At the
  text layer "about an attack" is indistinguishable from "carrying an attack" — such
  content needs a deliberate, explicitly-marked escaped path, never a silent allow.
- **Structural unsolvability at the text layer.** Pattern/lexicon detection is
  bypassable in isolation; novel phrasings and character-injection evade it. The
  *contract* (tool-less transform, capability isolation, fail-secure) carries the
  security — the lexicon is defense-in-depth, not a wall.
- **Text-only, extracted-text-only.** No file parsing in the core; what survives
  text extraction (macros, OLE objects, OCR-embedded instructions, render/font
  stego, encrypted files) is out of scope beyond the sanitizer's character layer.
- **Secret egress: base64-wrapped is caught, hex-wrapped is not** (a documented
  boundary — decode the transport layer first if you need it scanned).

## 9. Where to read more (in the guard repo)

- `README.md` — usage, the full contract, and the complete known-limitations list.
- `docs/BRIEF.md` — design rationale and the nearest-neighbour survey (§11).
- `docs/OKF-INGESTION-BRIEF.md` — the OKF threat-surface analysis (frontmatter,
  `resource`, cross-link graph, reserved names, provenance) the adapter implements.
- `python -m llm_ingestion_guard.coverage` — the runnable "verify what it stops".

Threat-model anchors: OWASP LLM Top-10 2025 (LLM01/02/04/05/06 strongest, LLM08
boundary), PoisonedRAG, guardrail-evasion (arXiv 2504.11168), EchoLeak
(CVE-2025-32711). OKF: Google Cloud Open Knowledge Format v0.1 (announced
2026-06-12) — `GoogleCloudPlatform/knowledge-catalog/okf/SPEC.md`.