llm-ingestion-okf/docs/plan/phase-2-doors-b-c.md

Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)

Status: approved roadmap phase (see CLAUDE.md); details settled here before code. Depends on: Phase 1 (materialization + index primitives are reused, never duplicated). This phase adds the library's first — and only permitted — runtime dependency: llm-ingestion-guard (pinned >=0.2,<0.3 when this plan was written; the window moved to >=0.3,<0.4 after measurement — see the settled note below).

Goal

Two guard-gated persist paths on top of the Phase 1 plumbing:

• Door B — bundle inbox: convert operator-dropped files into OKF concept files. All file-type→text extraction lives here (the guard is text-only).
• Door C — external bundle import: assess a third-party OKF bundle per concept via the guard's okf.import_bundle; merge, materialize, and index only the non-rejected concepts.

The boundary rule is absolute: the guard answers "is this content safe to persist?"; this library only connects, extracts, materializes, and indexes. No scanning, sanitizing, or quarantine logic is implemented here.

Door B — deliverables

1. Extraction registry — file extension → extractor:
   • Core (stdlib only): md/txt passthrough, csv → markdown table (reuses the Phase 1 renderer), json → verbatim inside a fenced block, html → text via html.parser.
   • Optional ([extract] extra only): pdf, docx, xlsx. Without the extra installed those types are rejected fail-fast with a typed error naming the extra — never a silent skip, never a bundled parser in core.
2. Inbox flow — an explicit operator command (mirrors the spec §9 HITL rule: never automatic, no watcher, no scheduler): process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult. Per file: read bytes → extract text → guard gate → materialize concept → index link. ingested_at explicit as in Phase 1; no wall-clock anywhere.
3. Guard gate (persist gate #1): extracted text passes through the guard (prepare_input/screen_output bookends with an upload-grade policy) before any write. Disposition handling: blocking dispositions → the file is NOT persisted and is reported in InboxResult; quarantine-grade dispositions → reported for operator review, not persisted (a quarantine directory is an extension point, not v1); pass/warn → persisted with the report attached to the result.
4. Inbox provenance layer — machine-generated files carry the honesty marker, analogous to spec §7: type, title, source_file (inbox-relative name), source_sha256 (of the original bytes), ingested_at, generated: true. Filenames namespaced inbox-{slug}.md with the slug reduced to the Phase 1 id grammar — disjoint from index.md, promoted-verdict-*, and ingest-*. The verdict reservation and the pre-mutation collision gate apply unchanged.

Door C — deliverables

1. Import flow — explicit operator command: import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult. Reads the external bundle as {relative path -> text}, hands it to the guard's okf.import_bundle(bundle, origin=…, channel=…) (the guard v0.2 signature — reserved-name rejection of index.md/log.md is unconditional, there is no allow_reserved toggle), and merges ONLY concepts whose per-concept result carries no error.
2. Merge + index: accepted concepts go through the Phase 1 staging/collision gate and index primitives (idempotent linking, curated content preserved). Rejected concepts are reported per concept with the guard's reason — never partially written.
3. Import report: the guard's log body (BundleResult.log()) is returned to the caller in ImportResult; persisting it into the bundle is NOT done in v1 (reserved-file policy differs per consumer and becomes configurable in Phase 3).

Settled during implementation (step 4)

• Door B calls screen_output alone, and this supersedes deliverable 3's "prepare_input/screen_output bookends". The bookends assume a model call between them; this library makes none, and prepare_input returns prompt-shaped text (sanitized and spotlight-fenced with a per-call nonce) that must never reach disk. The adapter therefore screens the extracted text as it stands and persists that same string, so the verdict is a statement about the bytes actually written.
• The gate refuses; it does not repair. Sanitizing before persisting would write a document differing invisibly from the operator's file while source_sha256 still points at the original bytes. A file carrying an invisible carrier is rejected instead — the guard's own doctrine is that a carrier has no legitimate place in a reference file, and this library's posture everywhere else is fail-fast, never repair. Operator decision.
• The policy is PRESET_USER_UPLOAD (untrusted tier, quarantine floor): an inbox drop is an untrusted upload, so any finding at all is held rather than written. A caller needing another tier injects their own adapter.
• guard_adapter.py is the only module that imports the guard, and the package __init__ does not import it, so a Door A consumer's import path is unaffected by the dependency's state.
• Assumption B1 pins what the adapters call, not the whole guard. prepare_input is dropped from the smoke test: drift there cannot reach this library. What is pinned: screen_output and okf.import_bundle signatures, the Disposition values both doors compare against, the Origin/Channel vocabularies Door C validates, the result fields the adapters read, and the upload preset's shape.
• The pin stays a range; the git URL is an install channel. A PEP 508 direct reference pins one tag and cannot express a range, but it is an install-time channel rather than a dependency declaration: the range is what pyproject.toml carries, it is satisfied by the tag install today, and it resolves normally once the package index exists (confirmed by the guard repo, superseding an earlier reading that the range had to go).
• Door C reasons are derived, not carried. The guard's stamp_concept keeps a concept's disposition and drops the reason strings behind it, so the adapter reports severity:label per finding — the audit trail actually available at that seam.

Settled during implementation (step 5)

• Verbatim merge, no frontmatter of ours. A merged concept is written exactly as the gate saw it. This is a correctness constraint, not a preference: the guard's frontmatter parser accepts block lists, which this library's line-oriented parse_frontmatter cannot round-trip, so stamping provenance into an external concept would silently drop sender data — and would persist bytes the guard never screened.

• Ownership by content identity. With no stamp available, an occupied target name is re-used only when the bytes already there are identical (a no-op re-merge, so re-import of an unchanged bundle is idempotent). Anything else at the name — curated content or an updated version of the same concept — is refused under collision_unstamped; the operator removes the file to accept an update. A configurable stamping/reserved-file policy is Phase 3's.

• The merge floor is fail-closed, not "no error". Deliverable 1's wording is the looser reading: a concept carrying no error but a FAIL_SECURE or unrecognised disposition is refused, and a concept the gate returned no verdict for at all is refused. Only the guard's non-blocking floor merges — the same floor Door B applies, with quarantine_review reported as its own bucket rather than folded into rejection.

• Upstream has released past the pin, and the decision is now taken. main's allow_reserved=True kwarg was first observed by us in guard v0.3.3 (a 19-fixture measurement against a scratch venv, unrelated to the pinned install). It did not ship there — the signature was measured across every 0.3 tag at bump time and the kwarg is present from v0.3.0 onward, absent in v0.2.0. The original wording read "first version we ran the suite against" as "version it was introduced in"; the two coincided only because v0.3.3 was the first 0.3 we measured at all. The conclusion it supported was right and the reason was wrong, so the reason is corrected rather than the outcome quietly kept. It also decides the pin's floor: >=0.3 is exactly right, and would have been wrong either way if the kwarg had really arrived in 0.3.3. The kwarg defaults True, so an unqualified call now merges index.md/log.md in a mode-b import instead of path-rejecting them — reversing this plan's original "no allow_reserved toggle, rejection is unconditional" reading. Decided: when the pin bumps into the 0.3.x line, guard_adapter.import_gate passes allow_reserved=False explicitly, keeping the reserved-name refusal this plan committed to.

  Done. The pin moved to >=0.3,<0.4 (resolved v0.3.4, not v0.3.3 — v0.3.4 shipped first and repairs a quadratic regex on Door C's own call path). The 19-fixture suite was re-run against v0.3.4 before the bump and reproduced the v0.3.3 deltas exactly, with none added. guard_adapter.import_gate now passes allow_reserved=False, and test_door_c_pins_allow_reserved_false_against_the_guards_default pins both halves: that the guard still defaults True (without which the override is a no-op that would pass forever over nothing) and that Door C overrides it.

  The recorded justification is worth sharpening now that it is code: the guard's True default is right for the guard, and this library does not dispute the safety reasoning behind it. Door C's refusal is structural — it generates the merged bundle's index.md from what it merged and writes every merged concept verbatim, so a sender's index.md is a second and irreconcilable claim on one path, not merely a risk to be scanned.

• origin/channel are validated against the guard's pinned vocabulary. The guard derives trust from origin by enum identity, so an unrecognised string would be silently downgraded to untrusted. The library refuses to carry a provenance declaration it cannot recognise rather than let a typo decide trust — refusing to guess, not deciding trust itself.

Design decisions fixed by this plan

• Trust follows origin, never channel (guard doctrine) — callers must supply both; no defaults.
• Extraction failure (corrupt file, missing extra) is a typed error on that file; the run continues with the remaining files and the result reports per-file outcomes. A guard FAIL-grade disposition on one file likewise never aborts the whole inbox run.
• Determinism: same inbox bytes + same ingested_at + same guard version ⇒ byte-identical persisted output. Guard verdicts are part of that input surface.

TDD order

1. Extraction registry: core types (fixtures per type), fail-fast on unknown extensions, fail-fast on optional types without [extract].
2. Inbox provenance rendering + filename slugging (pure functions).
3. Door B flow against a stub guard (in tests only — a test double standing in for the pinned guard API, so persist/reject branches are exercised deterministically; the real guard is exercised in integration tests).
4. Door B integration with the real guard: benign fixture persists; a fixture the guard fails-secure on is not persisted.
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only accepted concepts merged; collision and curated-preservation semantics reused from Phase 1 tests.

Key assumptions (each with its test)

#	Assumption	Test
B1	Guard 0.2 API matches the pinned surface (screen_output, okf.import_bundle, disposition enum, origin/channel vocabularies)	Import-and-signature smoke test that fails on upgrade drift (tests/test_guard_adapter.py) — CLOSED at step 4
B2	Guard is installable where this library's CI runs	CLOSED: git+https tag install over anonymously readable HTTPS, no credential; Forgejo package index becomes the durable channel later without a pyproject edit
B3	html.parser-based extraction is adequate for v1	Golden fixtures for representative HTML; anything richer is explicitly out of scope
B4	Guard verdicts are deterministic for fixed input + version	Same fixture run twice → identical InboxResult

Non-goals

• Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
• Quarantine storage/workflow and re-scan over time — extension points; the guard's companion-expectations list is revisited in Phase 3.
• Model-assisted enrichment of inbox content — never in the run path.
• Watchers, schedulers, implicit triggers.

Verification

1. pytest green with the guard installed; mypy --strict src/; ruff check . and ruff format --check . clean.
2. pdf file without [extract] → typed error naming the extra; with pip install .[extract] the same file extracts (integration test, may be skipped in environments without the extra — but must run in CI).
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new files in the bundle and an explicit rejection entry in the result (named test asserts the bundle directory is byte-identical before/after).
4. Door C partial-merge proof: mixed fixture → accepted concepts present, rejected absent, index links only for accepted, curated lines untouched.
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
6. Grep-gate: grep -rn "sanitize\|quarantine\|lexicon" src/ shows no local security reimplementation (guard imports only).
7. pyproject.toml runtime dependencies == exactly one range on llm-ingestion-guard (automated: test_the_only_runtime_dependency_is_the_security_boundary; the range itself is >=0.3,<0.4 since the bump).