llm-ingestion-okf/docs/plan/phase-2-doors-b-c.md

# Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)

Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
This phase adds the library's first — and only permitted — runtime dependency:
`llm-ingestion-guard>=0.2,<0.3`.

## Goal

Two guard-gated persist paths on top of the Phase 1 plumbing:

- **Door B — bundle inbox:** convert operator-dropped files into OKF concept
  files. All file-type→text extraction lives here (the guard is text-only).
- **Door C — external bundle import:** assess a third-party OKF bundle per
  concept via the guard's `okf.import_bundle`; merge, materialize, and index
  only the non-rejected concepts.

The boundary rule is absolute: the guard answers "is this content safe to
persist?"; this library only connects, extracts, materializes, and indexes.
No scanning, sanitizing, or quarantine logic is implemented here.

## Door B — deliverables

1. **Extraction registry** — file extension → extractor:
   - Core (stdlib only): `md`/`txt` passthrough, `csv` → markdown table (reuses
     the Phase 1 renderer), `json` → verbatim inside a fenced block, `html` →
     text via `html.parser`.
   - Optional (`[extract]` extra only): `pdf`, `docx`, `xlsx`. Without the extra
     installed those types are rejected fail-fast with a typed error naming the
     extra — never a silent skip, never a bundled parser in core.
2. **Inbox flow** — an explicit operator command (mirrors the spec §9 HITL rule:
   never automatic, no watcher, no scheduler):
   `process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult`.
   Per file: read bytes → extract text → guard gate → materialize concept →
   index link. `ingested_at` explicit as in Phase 1; no wall-clock anywhere.
3. **Guard gate (persist gate #1):** extracted text passes through the guard
   (`prepare_input`/`screen_output` bookends with an upload-grade policy) before
   any write. Disposition handling: blocking dispositions → the file is NOT
   persisted and is reported in `InboxResult`; quarantine-grade dispositions →
   reported for operator review, not persisted (a quarantine directory is an
   extension point, not v1); pass/warn → persisted with the report attached to
   the result.
4. **Inbox provenance layer** — machine-generated files carry the honesty
   marker, analogous to spec §7: `type`, `title`, `source_file` (inbox-relative
   name), `source_sha256` (of the original bytes), `ingested_at`, `generated: true`.
   Filenames namespaced `inbox-{slug}.md` with the slug reduced to the Phase 1
   id grammar — disjoint from `index.md`, `promoted-verdict-*`, and `ingest-*`.
   The verdict reservation and the pre-mutation collision gate apply unchanged.

## Door C — deliverables

1. **Import flow** — explicit operator command:
   `import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult`.
   Reads the external bundle as `{relative path -> text}`, hands it to the
   guard's `okf.import_bundle(bundle, origin=…, channel=…)` (the guard v0.2
   signature — reserved-name rejection of `index.md`/`log.md` is unconditional,
   there is no `allow_reserved` toggle), and merges ONLY concepts whose
   per-concept result carries no error.
2. **Merge + index:** accepted concepts go through the Phase 1 staging/collision
   gate and index primitives (idempotent linking, curated content preserved).
   Rejected concepts are reported per concept with the guard's reason — never
   partially written.
3. **Import report:** the guard's log body (`BundleResult.log()`) is returned to
   the caller in `ImportResult`; persisting it into the bundle is NOT done in v1
   (reserved-file policy differs per consumer and becomes configurable in
   Phase 3).

### Settled during implementation (step 4)

- **Door B calls `screen_output` alone, and this supersedes deliverable 3's
  "`prepare_input`/`screen_output` bookends".** The bookends assume a model
  call between them; this library makes none, and `prepare_input` returns
  prompt-shaped text (sanitized *and* spotlight-fenced with a per-call nonce)
  that must never reach disk. The adapter therefore screens the extracted
  text as it stands and persists that same string, so the verdict is a
  statement about the bytes actually written.
- **The gate refuses; it does not repair.** Sanitizing before persisting
  would write a document differing invisibly from the operator's file while
  `source_sha256` still points at the original bytes. A file carrying an
  invisible carrier is rejected instead — the guard's own doctrine is that a
  carrier has no legitimate place in a reference file, and this library's
  posture everywhere else is fail-fast, never repair. Operator decision.
- **The policy is `PRESET_USER_UPLOAD`** (untrusted tier, quarantine floor):
  an inbox drop is an untrusted upload, so any finding at all is held rather
  than written. A caller needing another tier injects their own adapter.
- **`guard_adapter.py` is the only module that imports the guard**, and the
  package `__init__` does not import it, so a Door A consumer's import path
  is unaffected by the dependency's state.
- **Assumption B1 pins what the adapters call, not the whole guard.**
  `prepare_input` is dropped from the smoke test: drift there cannot reach
  this library. What is pinned: `screen_output` and `okf.import_bundle`
  signatures, the `Disposition` values both doors compare against, the
  `Origin`/`Channel` vocabularies Door C validates, the result fields the
  adapters read, and the upload preset's shape.
- **The pin stays a range; the git URL is an install channel.** A PEP 508
  direct reference pins one tag and cannot express `>=0.2,<0.3`, but it is an
  install-time channel rather than a dependency declaration: the range is
  what `pyproject.toml` carries, it is satisfied by the tag install today,
  and it resolves normally once the package index exists (confirmed by the
  guard repo, superseding an earlier reading that the range had to go).
- **Door C reasons are derived, not carried.** The guard's `stamp_concept`
  keeps a concept's disposition and drops the reason strings behind it, so
  the adapter reports `severity:label` per finding — the audit trail actually
  available at that seam.

### Settled during implementation (step 5)

- **Verbatim merge, no frontmatter of ours.** A merged concept is written
  exactly as the gate saw it. This is a correctness constraint, not a
  preference: the guard's frontmatter parser accepts block lists, which this
  library's line-oriented `parse_frontmatter` cannot round-trip, so stamping
  provenance into an external concept would silently drop sender data — and
  would persist bytes the guard never screened.
- **Ownership by content identity.** With no stamp available, an occupied
  target name is re-used only when the bytes already there are identical (a
  no-op re-merge, so re-import of an unchanged bundle is idempotent).
  Anything else at the name — curated content or an updated version of the
  same concept — is refused under `collision_unstamped`; the operator removes
  the file to accept an update. A configurable stamping/reserved-file policy
  is Phase 3's.
- **The merge floor is fail-closed, not "no error".** Deliverable 1's wording
  is the looser reading: a concept carrying no error but a FAIL_SECURE or
  unrecognised disposition is refused, and a concept the gate returned no
  verdict for at all is refused. Only the guard's non-blocking floor merges —
  the same floor Door B applies, with `quarantine_review` reported as its own
  bucket rather than folded into rejection.
- **Upstream has already moved past the pin (observed, not acted on).** The
  guard's `main` carries a commit that adds `allow_reserved=True` to
  `okf.import_bundle` and *scans* `index.md`/`log.md` in a mode-b import
  instead of path-rejecting them. The pin is the `v0.2.0` tag, where the
  kwarg does not exist and rejection is unconditional, so this door is built
  against the tag. Whoever bumps the pin owns re-checking that branch: today
  a reserved name in a received bundle arrives as a per-concept rejection.
- **`origin`/`channel` are validated against the guard's pinned vocabulary.**
  The guard derives trust from `origin` by enum *identity*, so an unrecognised
  string would be silently downgraded to untrusted. The library refuses to
  carry a provenance declaration it cannot recognise rather than let a typo
  decide trust — refusing to guess, not deciding trust itself.

## Design decisions fixed by this plan

- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
  both; no defaults.
- Extraction failure (corrupt file, missing extra) is a typed error on that file;
  the run continues with the remaining files and the result reports per-file
  outcomes. A guard FAIL-grade disposition on one file likewise never aborts the
  whole inbox run.
- Determinism: same inbox bytes + same `ingested_at` + same guard version ⇒
  byte-identical persisted output. Guard verdicts are part of that input surface.

## TDD order

1. Extraction registry: core types (fixtures per type), fail-fast on unknown
   extensions, fail-fast on optional types without `[extract]`.
2. Inbox provenance rendering + filename slugging (pure functions).
3. Door B flow against a stub guard (in tests only — a test double standing in
   for the pinned guard API, so persist/reject branches are exercised
   deterministically; the real guard is exercised in integration tests).
4. Door B integration with the real guard: benign fixture persists; a fixture
   the guard fails-secure on is not persisted.
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only
   accepted concepts merged; collision and curated-preservation semantics reused
   from Phase 1 tests.

## Key assumptions (each with its test)

| # | Assumption | Test |
|---|---|---|
| B1 | Guard 0.2 API matches the pinned surface (`screen_output`, `okf.import_bundle`, disposition enum, origin/channel vocabularies) | Import-and-signature smoke test that fails on upgrade drift (`tests/test_guard_adapter.py`) — CLOSED at step 4 |
| B2 | Guard is installable where this library's CI runs | CLOSED: git+https tag install over anonymously readable HTTPS, no credential; Forgejo package index becomes the durable channel later without a pyproject edit |
| B3 | `html.parser`-based extraction is adequate for v1 | Golden fixtures for representative HTML; anything richer is explicitly out of scope |
| B4 | Guard verdicts are deterministic for fixed input + version | Same fixture run twice → identical `InboxResult` |

## Non-goals

- Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
- Quarantine storage/workflow and re-scan over time — extension points; the
  guard's companion-expectations list is revisited in Phase 3.
- Model-assisted enrichment of inbox content — never in the run path.
- Watchers, schedulers, implicit triggers.

## Verification

1. `pytest` green with the guard installed; `mypy --strict src/`; `ruff check .`
   and `ruff format --check .` clean.
2. `pdf` file without `[extract]` → typed error naming the extra; with
   `pip install .[extract]` the same file extracts (integration test, may be
   skipped in environments without the extra — but must run in CI).
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new
   files in the bundle and an explicit rejection entry in the result (named
   test asserts the bundle directory is byte-identical before/after).
4. Door C partial-merge proof: mixed fixture → accepted concepts present,
   rejected absent, index links only for accepted, curated lines untouched.
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
   security reimplementation (guard imports only).
7. `pyproject.toml` runtime dependencies == exactly `llm-ingestion-guard>=0.2,<0.3`
   (automated: `test_the_only_runtime_dependency_is_the_security_boundary`).