Adoption plan — `llm-ingestion-guard` at the ingest persist gate

Status: plan only. This document decides when and where the write-time ingestion guard (llm-ingestion-guard, alpha — this plan pins no version; see §3.2 for the allow_reserved default, which the guard changed in v0.3.0) earns its place in the architecture this commons specifies. It does NOT wire the guard in — the executable wiring lands in the two reference implementations; this document is the framework-neutral contract they build to, same as method-spec.md and ingest-spec.md.

Source brief: llm-ingestion-pipeline-security/docs/ADOPTION-BRIEF.md (§7 checklist is the template used below). Guard repo: git.fromaitochitta.com/open/llm-ingestion-pipeline-security.

1. What this commons is (relevant framing)

portfolio-optimiser-commons is the framework-neutral shared core consumed as a shared/ git subtree by both reference implementations (MAF and Claude Agents SDK). It contains normative specs + fixtures, no executable pipeline. Two specs matter here:

ingest-spec.md — a deterministic, zero-model-call step that materializes real data sources into OKF bundles BEFORE the loop. Source types: file (local CSV catalogue), sql (read-only SELECT), and http (an OPTIONAL extension point — remote endpoint / MCP connector, §4).
method-spec.md — the 8-step agentic loop that reads the OKF bundle, enriches via a proposer/checker model, gates numbers with a mandatory deterministic validator, takes human expert verdicts, and promotes approved verdicts back into the bundle wiki (the promotion gate, §6 / Step 8).

The bundle is an OKF second-brain a downstream run later reads as trusted context — the exact shape the guard's brief targets.

2. §7 checklist, scored against this architecture

§7 condition	Verdict for the commons-specified system
Persist LLM-enriched / externally-received content into a store a downstream agent reads as trusted	Partial. The promotion gate (method §6) lifts model-proposed, expert-approved verdicts into the OKF wiki; the next run's seeding fold reads them as trusted. Ingest materializes raw source extracts — no LLM step at ingest time.
≥1 ingest path takes UNTRUSTED content (the decisive box)	Not on any path implemented today. The two conformance-required sources — `file` and `sql` — are first-party by origin (the operator's own project files / database). The untrusted boundary is the `http`/MCP source type (ingest §4, an explicit not-yet-built extension point) and any future received-external-OKF-bundle merge (not yet specified).
An LLM step sits between the untrusted source and the store	Ingest: no (zero model calls, ingest §1). Loop: yes, but over first-party bundle context.
You want fail-secure (halt before persist)	Yes, already the design ethos — ingest is fail-fast/fail-closed throughout (§3 verdict-layer reservation, §4 manifest validation, §8 network opt-in).

Verdict: the decisive box is currently NO — every live/required ingest path is first-party. But the architecture has designed, foreseeable untrusted extension points (http/MCP; received bundles). Per brief §7 ("note the guard as a dependency to add when — not if — you open an external/inbox/received-bundle path"), the honest status is planned, not not-applicable.

3. Untrusted boundaries (guard wires here) vs first-party paths (it does not)

3.1 Untrusted boundary A — the `http`/MCP connector (ingest §4 extension point)

When a connector fetches remote/vendor/web content, the extracted body is untrusted, and ingest §5 persists it verbatim inside a fenced code block into a concept file the loop later reads as trusted context. That verbatim persist of remote content into an agent-read bundle is precisely the write-time boundary the guard defends.

Minimal wiring (at ingest materialization, before writing ingest-{id}.md): ingest makes no model call, so only the scan-before-persist half of the contract applies (brief §3 steps 1, 6, 7 — the fence-a-model-call half is N/A here):

from llm_ingestion_guard import sanitize, scan_output, Disposition  # primitives are exported per-piece

raw       = connector.fetch(extraction)          # untrusted remote body
cleaned   = sanitize(raw)                         # strip carrier classes (zero-width/BIDI/tag/comment/data:)
decision  = scan_output(cleaned)                  # lexicon + entropy + active-content, high-untrust
if decision.disposition is Disposition.FAIL_SECURE:
    alert(gate_code=decision.reasons, run_id=run_id)   # minimal payload, no content (brief §2/§3.8)
    raise SystemExit                                    # halt — never materialize this concept
# else: write the concept file exactly as ingest §5 specifies today

3.2 Untrusted boundary B — a received external OKF bundle (`okf.import_bundle`)

Distinct from the first-party commons subtree sync: if either implementation ever adds a "merge a foreign OKF bundle" path, run the guard's OKF adapter at that persist gate.

from llm_ingestion_guard.okf import import_bundle, Origin, Channel

result = import_bundle(bundle, origin=Origin.EXTERNAL, channel=Channel.AUTOMATIC)
for c in result.concepts:
    if c.error:            # FAIL_SECURE per concept: bad path, unsafe frontmatter, non-https resource
        skip(c.path)       # do not merge this concept
# then materialize the surviving concepts

This maps directly onto ingest-spec gates the commons already mandates — path boundary-check (§4), verdict-layer reservation (§3 — a received bundle MUST NOT inject type: verdict files), and the index cross-link graph (§6). The adapter implements a superset (frontmatter parse-safety, resource URL allowlist, dangling-link detection). Pick allow_reserved=True for a whole received bundle (scan index.md/log.md bodies), allow_reserved=False if the path ever materialises individual uploads.

3.3 First-party path — the promotion gate (method §6). Do NOT wire the guard here.

The promotion gate lifts model-proposed, expert-approved verdicts into the wiki. Its content originates from the first-party bundle context, passes the deterministic validator, and crosses only on human/persona approval (method §6 fail-closed). Per brief §7, trusted-author approvals are out of the guard's threat model by design. Wiring the guard here would trip on legitimate content and add no security the human + validator do not already provide. Recorded explicitly so a future session does not over-wire.

3.4 Sharp edge — origin vs channel for `file`/`sql`

file/sql are first-party channels, but their content origin can be external (a vendor's CSV, a subcontractor's report the operator ingests). Brief §7: "trust follows the data's origin, not the insertion channel." A natural — but not-yet-decided — extension is a per-source origin: first-party | external field in the ingest manifest; a source declared external-origin would route its extracted text through the §3.1 scan-before-persist gate. Flagged as an open design question, not prescribed here (avoid scope creep); it keeps the decisive-box judgment honest.

4. When — roadmap placement

T0 (now): first-party only (file + sql, curated bundles, first-party commons subtree). Guard = documented dependency, not wired. Marker status planned.
Trigger A — http/MCP source type implemented (in either reference impl): wire §3.1 at ingest materialization before that path handles any real remote content, and author the normative gate as a new ingest-spec section (below) so both stacks wire it identically.
Trigger B — a received-external-OKF-bundle merge path is added: wire §3.2 (okf.import_bundle) at that persist gate.
Trigger C (weaker) — a manifest per-source origin: external field is added: route external-origin file/sql extracts through the same §3.1 gate.

Commons' specific role. Because this is the normative shared core, the contract for the untrusted-source gate ("untrusted extracted text MUST be sanitized + scanned before persist; FAIL_SECURE halts materialization, never a silent commit") should be authored here — a future ingest-spec section, added when Trigger A/B nears — so MAF and the Claude-SDK implementation wire the identical guard. The executable wiring lives in each implementation repo; the rule lives in commons, mirroring how method-spec/ingest-spec already work.

Coordination with consuming implementation repos (recorded 2026-07-16). The Claude Agents SDK implementation (claude-code-llm-wiki) is running this same adoption task in parallel and has drawn the reciprocal boundary: its shared/ is a pull-only subtree of commons, so guard wiring that touches shared/ or the ingest-spec is owned by this commons session, not there — its plan covers only its repo-local modules (ingest.py, verdicts.py, okf.py). Division of labor, both directions:

Commons (here) authors the normative untrusted-source gate contract (a future ingest-spec section) and MUST NOT edit implementation-repo modules.
Each implementation repo wires the guard in its own runtime modules and pulls the spec via subtree; it MUST NOT edit the shared ingest-spec in place (edits land in commons first, then git subtree pull, per README).

5. Honest limitations to carry into any future wiring (brief §8)

Semantic / factual poisoning is invisible to the deterministic core — highest impact for a wiki. A plausible-but-wrong extracted value (wrong metric, wrong cost) passes clean. This system already has a strong mitigation: the deterministic validator (method §4) and human expert gate anchor the numbers the wiki feeds — the guard is defense in depth over the text carrier, not the semantic truth.
Dormant / broken-link injection — the guard's link_graph surfaces the dangling edge; blocking is a disposition call. Ingest's index-link contract (§6) is the natural place.
Text-only, extracted-text-only — if ingest ever pulls binary/file sources, extract text first, then scan with high-untrust provenance (brief §6).

6. Verification (testable criteria)

Run from the repo root:

Plan exists: test -f docs/plan/2026-07-16-llm-ingestion-guard-adoption.md
Marker present, status first token: grep -Eq '^llm-ingestion-guard: (planned|integrated|not-applicable) — ' STATE.md
STATE.md is LOCAL-ONLY: git check-ignore -q STATE.md (exit 0 = ignored, never committed)
Both persist gates named, promotion gate excluded: grep -q 'import_bundle' docs/plan/2026-07-16-llm-ingestion-guard-adoption.md and grep -q 'Do NOT wire the guard here' docs/plan/2026-07-16-llm-ingestion-guard-adoption.md
Scope guard — guard NOT wired yet (this repo has no runtime code): ! grep -rq 'llm_ingestion_guard' --include='*.py' . (no Python files exist; the only llm_ingestion_guard references are the illustrative snippets in this plan)

7. Explicitly out of scope for this task

Implementing the guard, adding it as a dependency, editing ingest-spec.md/method-spec.md, or touching either implementation repo. This task plans and records only.

11 KiB Raw Blame History

Adoption plan — llm-ingestion-guard at the ingest persist gate