llm-ingestion-okf/docs/plan/phase-2-doors-b-c.md

# Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)

Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
This phase adds the library's first — and only permitted — runtime dependency:
`llm-ingestion-guard>=0.2,<0.3`.

## Goal

Two guard-gated persist paths on top of the Phase 1 plumbing:

- **Door B — bundle inbox:** convert operator-dropped files into OKF concept
  files. All file-type→text extraction lives here (the guard is text-only).
- **Door C — external bundle import:** assess a third-party OKF bundle per
  concept via the guard's `okf.import_bundle`; merge, materialize, and index
  only the non-rejected concepts.

The boundary rule is absolute: the guard answers "is this content safe to
persist?"; this library only connects, extracts, materializes, and indexes.
No scanning, sanitizing, or quarantine logic is implemented here.

## Door B — deliverables

1. **Extraction registry** — file extension → extractor:
   - Core (stdlib only): `md`/`txt` passthrough, `csv` → markdown table (reuses
     the Phase 1 renderer), `json` → verbatim inside a fenced block, `html` →
     text via `html.parser`.
   - Optional (`[extract]` extra only): `pdf`, `docx`, `xlsx`. Without the extra
     installed those types are rejected fail-fast with a typed error naming the
     extra — never a silent skip, never a bundled parser in core.
2. **Inbox flow** — an explicit operator command (mirrors the spec §9 HITL rule:
   never automatic, no watcher, no scheduler):
   `process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult`.
   Per file: read bytes → extract text → guard gate → materialize concept →
   index link. `ingested_at` explicit as in Phase 1; no wall-clock anywhere.
3. **Guard gate (persist gate #1):** extracted text passes through the guard
   (`prepare_input`/`screen_output` bookends with an upload-grade policy) before
   any write. Disposition handling: blocking dispositions → the file is NOT
   persisted and is reported in `InboxResult`; quarantine-grade dispositions →
   reported for operator review, not persisted (a quarantine directory is an
   extension point, not v1); pass/warn → persisted with the report attached to
   the result.
4. **Inbox provenance layer** — machine-generated files carry the honesty
   marker, analogous to spec §7: `type`, `title`, `source_file` (inbox-relative
   name), `source_sha256` (of the original bytes), `ingested_at`, `generated: true`.
   Filenames namespaced `inbox-{slug}.md` with the slug reduced to the Phase 1
   id grammar — disjoint from `index.md`, `promoted-verdict-*`, and `ingest-*`.
   The verdict reservation and the pre-mutation collision gate apply unchanged.

## Door C — deliverables

1. **Import flow** — explicit operator command:
   `import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult`.
   Reads the external bundle as `{relative path -> text}`, hands it to the
   guard's `okf.import_bundle(bundle, origin=…, channel=…, allow_reserved=False)`,
   and merges ONLY concepts whose per-concept result carries no error.
2. **Merge + index:** accepted concepts go through the Phase 1 staging/collision
   gate and index primitives (idempotent linking, curated content preserved).
   Rejected concepts are reported per concept with the guard's reason — never
   partially written.
3. **Import report:** the guard's log body (`BundleResult.log()`) is returned to
   the caller in `ImportResult`; persisting it into the bundle is NOT done in v1
   (reserved-file policy differs per consumer and becomes configurable in
   Phase 3).

## Design decisions fixed by this plan

- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
  both; no defaults.
- Extraction failure (corrupt file, missing extra) is a typed error on that file;
  the run continues with the remaining files and the result reports per-file
  outcomes. A guard FAIL-grade disposition on one file likewise never aborts the
  whole inbox run.
- Determinism: same inbox bytes + same `ingested_at` + same guard version ⇒
  byte-identical persisted output. Guard verdicts are part of that input surface.

## TDD order

1. Extraction registry: core types (fixtures per type), fail-fast on unknown
   extensions, fail-fast on optional types without `[extract]`.
2. Inbox provenance rendering + filename slugging (pure functions).
3. Door B flow against a stub guard (in tests only — a test double standing in
   for the pinned guard API, so persist/reject branches are exercised
   deterministically; the real guard is exercised in integration tests).
4. Door B integration with the real guard: benign fixture persists; a fixture
   the guard fails-secure on is not persisted.
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only
   accepted concepts merged; collision and curated-preservation semantics reused
   from Phase 1 tests.

## Key assumptions (each with its test)

| # | Assumption | Test |
|---|---|---|
| B1 | Guard 0.2 API matches the pinned surface (`prepare_input`, `screen_output`, `okf.import_bundle`, disposition enum) | Import-and-signature smoke test that fails on upgrade drift |
| B2 | Guard is installable where this library's CI runs | Resolve the distribution channel with the operator before implementation starts (risk: not yet decided) |
| B3 | `html.parser`-based extraction is adequate for v1 | Golden fixtures for representative HTML; anything richer is explicitly out of scope |
| B4 | Guard verdicts are deterministic for fixed input + version | Same fixture run twice → identical `InboxResult` |

## Non-goals

- Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
- Quarantine storage/workflow and re-scan over time — extension points; the
  guard's companion-expectations list is revisited in Phase 3.
- Model-assisted enrichment of inbox content — never in the run path.
- Watchers, schedulers, implicit triggers.

## Verification

1. `pytest` green with the guard installed; `mypy --strict src/`; `ruff check .`
   and `ruff format --check .` clean.
2. `pdf` file without `[extract]` → typed error naming the extra; with
   `pip install .[extract]` the same file extracts (integration test, may be
   skipped in environments without the extra — but must run in CI).
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new
   files in the bundle and an explicit rejection entry in the result (named
   test asserts the bundle directory is byte-identical before/after).
4. Door C partial-merge proof: mixed fixture → accepted concepts present,
   rejected absent, index links only for accepted, curated lines untouched.
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
   security reimplementation (guard imports only).
7. `pyproject.toml` runtime dependencies == exactly `llm-ingestion-guard>=0.2,<0.3`.