portfolio-optimiser/docs/plan/2026-07-16-llm-ingestion-guard-adoption.md

# Adoption plan — `llm-ingestion-guard` at the ingest persist gate

> **Status:** plan only. This document decides *when* and *where* the write-time
> ingestion guard (`llm-ingestion-guard`, alpha — this plan pins no version; see
> §3.2 for the `allow_reserved` default, which the guard changed in v0.3.0) earns its place in the
> architecture this commons specifies. **It does NOT wire the guard in** — the
> executable wiring lands in the two reference implementations; this document is
> the framework-neutral contract they build to, same as [method-spec.md](../../method-spec.md)
> and [ingest-spec.md](../../ingest-spec.md).
>
> Source brief: `llm-ingestion-pipeline-security/docs/ADOPTION-BRIEF.md` (§7 checklist
> is the template used below). Guard repo: `git.fromaitochitta.com/open/llm-ingestion-pipeline-security`.

## 1. What this commons is (relevant framing)

`portfolio-optimiser-commons` is the **framework-neutral shared core** consumed as a
`shared/` git subtree by both reference implementations (MAF and Claude Agents SDK).
It contains **normative specs + fixtures, no executable pipeline.** Two specs matter here:

- **[ingest-spec.md](../../ingest-spec.md)** — a *deterministic, zero-model-call* step that
  materializes real data sources into OKF bundles BEFORE the loop. Source types: `file`
  (local CSV catalogue), `sql` (read-only SELECT), and `http` (an **OPTIONAL extension
  point** — remote endpoint / MCP connector, §4).
- **[method-spec.md](../../method-spec.md)** — the 8-step agentic loop that *reads* the OKF
  bundle, enriches via a proposer/checker model, gates numbers with a mandatory
  deterministic validator, takes human expert verdicts, and **promotes approved verdicts
  back into the bundle wiki** (the promotion gate, §6 / Step 8).

The bundle *is* an OKF second-brain a downstream run later reads as trusted context — the
exact shape the guard's brief targets.

## 2. §7 checklist, scored against this architecture

| §7 condition | Verdict for the commons-specified system |
|---|---|
| Persist LLM-enriched / externally-received content into a store a *downstream* agent reads as trusted | **Partial.** The **promotion gate** (method §6) lifts model-*proposed*, expert-*approved* verdicts into the OKF wiki; the next run's seeding fold reads them as trusted. Ingest materializes *raw source extracts* — **no LLM step at ingest time**. |
| **≥1 ingest path takes UNTRUSTED content** *(the decisive box)* | **Not on any path implemented today.** The two conformance-required sources — `file` and `sql` — are **first-party by origin** (the operator's own project files / database). The untrusted boundary is the **`http`/MCP source type** (ingest §4, an explicit not-yet-built extension point) and any future **received-external-OKF-bundle** merge (not yet specified). |
| An LLM step sits between the untrusted source and the store | Ingest: **no** (zero model calls, ingest §1). Loop: yes, but over *first-party* bundle context. |
| You want fail-secure (halt before persist) | **Yes, already the design ethos** — ingest is fail-fast/fail-closed throughout (§3 verdict-layer reservation, §4 manifest validation, §8 network opt-in). |

**Verdict: the decisive box is currently NO** — every live/required ingest path is
first-party. But the architecture has *designed, foreseeable* untrusted extension points
(`http`/MCP; received bundles). Per brief §7 ("note the guard as a dependency to add when —
not if — you open an external/inbox/received-bundle path"), the honest status is
**`planned`**, not `not-applicable`.

## 3. Untrusted boundaries (guard wires here) vs first-party paths (it does not)

### 3.1 Untrusted boundary A — the `http`/MCP connector (ingest §4 extension point)

When a connector fetches remote/vendor/web content, the extracted body is untrusted, and
ingest §5 persists it **verbatim inside a fenced code block** into a concept file the loop
later reads as trusted context. That verbatim persist of remote content into an agent-read
bundle is precisely the write-time boundary the guard defends.

**Minimal wiring (at ingest materialization, before writing `ingest-{id}.md`):** ingest
makes *no model call*, so only the **scan-before-persist half** of the contract applies
(brief §3 steps 1, 6, 7 — the fence-a-model-call half is N/A here):

```python
from llm_ingestion_guard import sanitize, scan_output, Disposition  # primitives are exported per-piece

raw       = connector.fetch(extraction)          # untrusted remote body
cleaned   = sanitize(raw)                         # strip carrier classes (zero-width/BIDI/tag/comment/data:)
decision  = scan_output(cleaned)                  # lexicon + entropy + active-content, high-untrust
if decision.disposition is Disposition.FAIL_SECURE:
    alert(gate_code=decision.reasons, run_id=run_id)   # minimal payload, no content (brief §2/§3.8)
    raise SystemExit                                    # halt — never materialize this concept
# else: write the concept file exactly as ingest §5 specifies today
```

### 3.2 Untrusted boundary B — a received external OKF bundle (`okf.import_bundle`)

Distinct from the first-party `commons` subtree sync: if either implementation ever adds a
"merge a *foreign* OKF bundle" path, run the guard's OKF adapter at that persist gate.

```python
from llm_ingestion_guard.okf import import_bundle, Origin, Channel

result = import_bundle(bundle, origin=Origin.EXTERNAL, channel=Channel.AUTOMATIC)
for c in result.concepts:
    if c.error:            # FAIL_SECURE per concept: bad path, unsafe frontmatter, non-https resource
        skip(c.path)       # do not merge this concept
# then materialize the surviving concepts
```

This maps directly onto ingest-spec gates the commons *already* mandates — path
boundary-check (§4), **verdict-layer reservation (§3 — a received bundle MUST NOT inject
`type: verdict` files)**, and the index cross-link graph (§6). The adapter implements a
superset (frontmatter parse-safety, `resource` URL allowlist, dangling-link detection). Pick
`allow_reserved=True` for a whole received bundle (scan `index.md`/`log.md` bodies),
`allow_reserved=False` if the path ever materialises *individual* uploads.

### 3.3 First-party path — the promotion gate (method §6). **Do NOT wire the guard here.**

The promotion gate lifts model-proposed, expert-approved verdicts into the wiki. Its content
originates from the **first-party bundle context**, passes the **deterministic validator**,
and crosses only on **human/persona approval** (method §6 fail-closed). Per brief §7,
trusted-author approvals are **out of the guard's threat model by design**. Wiring the guard
here would trip on legitimate content and add no security the human + validator do not
already provide. *Recorded explicitly so a future session does not over-wire.*

### 3.4 Sharp edge — origin vs channel for `file`/`sql`

`file`/`sql` are first-party *channels*, but their *content origin* can be external (a
vendor's CSV, a subcontractor's report the operator ingests). Brief §7: **"trust follows the
data's origin, not the insertion channel."** A natural — but **not-yet-decided** — extension
is a per-source `origin: first-party | external` field in the ingest manifest; a source
declared external-origin would route its extracted text through the §3.1 scan-before-persist
gate. Flagged as an open design question, **not prescribed here** (avoid scope creep); it
keeps the decisive-box judgment honest.

## 4. When — roadmap placement

- **T0 (now):** first-party only (`file` + `sql`, curated bundles, first-party `commons`
  subtree). Guard = **documented dependency, not wired.** Marker status `planned`.
- **Trigger A — `http`/MCP source type implemented** (in either reference impl): wire §3.1
  at ingest materialization **before that path handles any real remote content**, and author
  the normative gate as a new ingest-spec section (below) so both stacks wire it identically.
- **Trigger B — a received-external-OKF-bundle merge path is added:** wire §3.2
  (`okf.import_bundle`) at that persist gate.
- **Trigger C (weaker) — a manifest per-source `origin: external` field is added:** route
  external-origin `file`/`sql` extracts through the same §3.1 gate.

**Commons' specific role.** Because this is the normative shared core, the *contract* for the
untrusted-source gate ("untrusted extracted text MUST be sanitized + scanned before persist;
FAIL_SECURE halts materialization, never a silent commit") should be authored **here** — a
future ingest-spec section, added when Trigger A/B nears — so MAF and the Claude-SDK
implementation wire the identical guard. The executable wiring lives in each implementation
repo; the *rule* lives in commons, mirroring how method-spec/ingest-spec already work.

**Coordination with consuming implementation repos (recorded 2026-07-16).** The Claude
Agents SDK implementation (`claude-code-llm-wiki`) is running this same adoption task in
parallel and has drawn the reciprocal boundary: its `shared/` is a **pull-only subtree of
commons**, so guard wiring that touches `shared/` or the **ingest-spec** is owned by *this*
commons session, not there — its plan covers only its **repo-local modules** (`ingest.py`,
`verdicts.py`, `okf.py`). Division of labor, both directions:

- **Commons (here)** authors the normative untrusted-source *gate contract* (a future
  ingest-spec section) and MUST NOT edit implementation-repo modules.
- **Each implementation repo** wires the guard in its own runtime modules and pulls the spec
  via subtree; it MUST NOT edit the shared ingest-spec in place (edits land in commons first,
  then `git subtree pull`, per README).

## 5. Honest limitations to carry into any future wiring (brief §8)

- **Semantic / factual poisoning is invisible** to the deterministic core — highest impact
  for a wiki. A plausible-but-wrong extracted value (wrong metric, wrong cost) passes clean.
  This system already has a strong mitigation: the **deterministic validator** (method §4)
  and **human expert gate** anchor the *numbers* the wiki feeds — the guard is defense in
  depth over the *text carrier*, not the semantic truth.
- **Dormant / broken-link injection** — the guard's `link_graph` surfaces the dangling edge;
  blocking is a disposition call. Ingest's index-link contract (§6) is the natural place.
- **Text-only, extracted-text-only** — if ingest ever pulls binary/file sources, extract
  text first, then scan with high-untrust provenance (brief §6).

## 6. Verification (testable criteria)

Run from the repo root:

1. Plan exists: `test -f docs/plan/2026-07-16-llm-ingestion-guard-adoption.md`
2. Marker present, status first token: `grep -Eq '^llm-ingestion-guard: (planned|integrated|not-applicable) — ' STATE.md`
3. STATE.md is LOCAL-ONLY: `git check-ignore -q STATE.md` (exit 0 = ignored, never committed)
4. Both persist gates named, promotion gate excluded:
   `grep -q 'import_bundle' docs/plan/2026-07-16-llm-ingestion-guard-adoption.md` and
   `grep -q 'Do NOT wire the guard here' docs/plan/2026-07-16-llm-ingestion-guard-adoption.md`
5. Scope guard — guard NOT wired yet (this repo has no runtime code):
   `! grep -rq 'llm_ingestion_guard' --include='*.py' .` (no Python files exist; the only
   `llm_ingestion_guard` references are the illustrative snippets in this plan)

## 7. Explicitly out of scope for this task

Implementing the guard, adding it as a dependency, editing ingest-spec.md/method-spec.md, or
touching either implementation repo. This task **plans and records only.**