llm-ingestion-okf/docs/plan/phase-2-doors-b-c.md

# Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)

Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
This phase adds the library's first — and only permitted — runtime dependency:
`llm-ingestion-guard` (pinned `>=0.2,<0.3` when this plan was written; the
window moved to `>=0.3,<0.4` after measurement — see the settled note below).

## Goal

Two guard-gated persist paths on top of the Phase 1 plumbing:

- **Door B — bundle inbox:** convert operator-dropped files into OKF concept
  files. All file-type→text extraction lives here (the guard is text-only).
- **Door C — external bundle import:** assess a third-party OKF bundle per
  concept via the guard's `okf.import_bundle`; merge, materialize, and index
  only the non-rejected concepts.

The boundary rule is absolute: the guard answers "is this content safe to
persist?"; this library only connects, extracts, materializes, and indexes.
No scanning, sanitizing, or quarantine logic is implemented here.

## Door B — deliverables

1. **Extraction registry** — file extension → extractor:
   - Core (stdlib only): `md`/`txt` passthrough, `csv` → markdown table (reuses
     the Phase 1 renderer), `json` → verbatim inside a fenced block, `html` →
     text via `html.parser`.
   - Optional (`[extract]` extra only): `pdf`, `docx`, `xlsx`. Without the extra
     installed those types are rejected fail-fast with a typed error naming the
     extra — never a silent skip, never a bundled parser in core.
2. **Inbox flow** — an explicit operator command (mirrors the spec §9 HITL rule:
   never automatic, no watcher, no scheduler):
   `process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult`.
   Per file: read bytes → extract text → guard gate → materialize concept →
   index link. `ingested_at` explicit as in Phase 1; no wall-clock anywhere.
3. **Guard gate (persist gate #1):** extracted text passes through the guard
   (`prepare_input`/`screen_output` bookends with an upload-grade policy) before
   any write. Disposition handling: blocking dispositions → the file is NOT
   persisted and is reported in `InboxResult`; quarantine-grade dispositions →
   reported for operator review, not persisted (a quarantine directory is an
   extension point, not v1); pass/warn → persisted with the report attached to
   the result.
4. **Inbox provenance layer** — machine-generated files carry the honesty
   marker, analogous to spec §7: `type`, `title`, `source_file` (inbox-relative
   name), `source_sha256` (of the original bytes), `ingested_at`, `generated: true`.
   Filenames namespaced `inbox-{slug}.md` with the slug reduced to the Phase 1
   id grammar — disjoint from `index.md`, `promoted-verdict-*`, and `ingest-*`.
   The verdict reservation and the pre-mutation collision gate apply unchanged.

## Door C — deliverables

1. **Import flow** — explicit operator command:
   `import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult`.
   Reads the external bundle as `{relative path -> text}`, hands it to the
   guard's `okf.import_bundle(bundle, origin=…, channel=…)` (the guard v0.2
   signature — reserved-name rejection of `index.md`/`log.md` is unconditional,
   there is no `allow_reserved` toggle), and merges ONLY concepts whose
   per-concept result carries no error.
2. **Merge + index:** accepted concepts go through the Phase 1 staging/collision
   gate and index primitives (idempotent linking, curated content preserved).
   Rejected concepts are reported per concept with the guard's reason — never
   partially written.
3. **Import report:** the guard's log body (`BundleResult.log()`) is returned to
   the caller in `ImportResult`; persisting it into the bundle is NOT done in v1
   (reserved-file policy differs per consumer and becomes configurable in
   Phase 3).

### Settled during implementation (step 4)

- **Door B calls `screen_output` alone, and this supersedes deliverable 3's
  "`prepare_input`/`screen_output` bookends".** The bookends assume a model
  call between them; this library makes none, and `prepare_input` returns
  prompt-shaped text (sanitized *and* spotlight-fenced with a per-call nonce)
  that must never reach disk. The adapter therefore screens the extracted
  text as it stands and persists that same string, so the verdict is a
  statement about the bytes actually written.
- **The gate refuses; it does not repair.** Sanitizing before persisting
  would write a document differing invisibly from the operator's file while
  `source_sha256` still points at the original bytes. A file carrying an
  invisible carrier is rejected instead — the guard's own doctrine is that a
  carrier has no legitimate place in a reference file, and this library's
  posture everywhere else is fail-fast, never repair. Operator decision.
- **The policy is `PRESET_USER_UPLOAD`** (untrusted tier, quarantine floor):
  an inbox drop is an untrusted upload, so any finding at all is held rather
  than written. A caller needing another tier injects their own adapter.
- **`guard_adapter.py` is the only module that imports the guard**, and the
  package `__init__` does not import it, so a Door A consumer's import path
  is unaffected by the dependency's state.
- **Assumption B1 pins what the adapters call, not the whole guard.**
  `prepare_input` is dropped from the smoke test: drift there cannot reach
  this library. What is pinned: `screen_output` and `okf.import_bundle`
  signatures, the `Disposition` values both doors compare against, the
  `Origin`/`Channel` vocabularies Door C validates, the result fields the
  adapters read, and the upload preset's shape.
- **The pin stays a range; the git URL is an install channel.** A PEP 508
  direct reference pins one tag and cannot express a range, but it is an
  install-time channel rather than a dependency declaration: the range is
  what `pyproject.toml` carries, it is satisfied by the tag install today,
  and it resolves normally once the package index exists (confirmed by the
  guard repo, superseding an earlier reading that the range had to go).
- **Door C reasons are derived, not carried.** The guard's `stamp_concept`
  keeps a concept's disposition and drops the reason strings behind it, so
  the adapter reports `severity:label` per finding — the audit trail actually
  available at that seam.

### Settled during implementation (step 5)

- **Verbatim merge, no frontmatter of ours.** A merged concept is written
  exactly as the gate saw it. This is a correctness constraint, not a
  preference: the guard's frontmatter parser accepts block lists, which this
  library's line-oriented `parse_frontmatter` cannot round-trip, so stamping
  provenance into an external concept would silently drop sender data — and
  would persist bytes the guard never screened.
- **Ownership by content identity.** With no stamp available, an occupied
  target name is re-used only when the bytes already there are identical (a
  no-op re-merge, so re-import of an unchanged bundle is idempotent).
  Anything else at the name — curated content or an updated version of the
  same concept — is refused under `collision_unstamped`; the operator removes
  the file to accept an update. A configurable stamping/reserved-file policy
  is Phase 3's.
- **The merge floor is fail-closed, not "no error".** Deliverable 1's wording
  is the looser reading: a concept carrying no error but a FAIL_SECURE or
  unrecognised disposition is refused, and a concept the gate returned no
  verdict for at all is refused. Only the guard's non-blocking floor merges —
  the same floor Door B applies, with `quarantine_review` reported as its own
  bucket rather than folded into rejection.
- **Upstream has released past the pin, and the decision is now taken.** `main`'s
  `allow_reserved=True` kwarg was first *observed* by us in guard `v0.3.3` (a
  19-fixture measurement against a scratch venv, unrelated to the pinned
  install). **It did not ship there** — the signature was measured across every
  0.3 tag at bump time and the kwarg is present from `v0.3.0` onward, absent in
  `v0.2.0`. The original wording read "first version we ran the suite against"
  as "version it was introduced in"; the two coincided only because `v0.3.3`
  was the first 0.3 we measured at all. The conclusion it supported was right
  and the reason was wrong, so the reason is corrected rather than the outcome
  quietly kept. It also decides the pin's floor: `>=0.3` is exactly right, and
  would have been wrong either way if the kwarg had really arrived in `0.3.3`.
  The kwarg defaults `True`, so an unqualified call now *merges*
  `index.md`/`log.md` in a mode-b import instead of path-rejecting them —
  reversing this plan's original "no allow_reserved toggle, rejection is
  unconditional" reading. Decided: when the pin bumps into the `0.3.x` line,
  `guard_adapter.import_gate` passes `allow_reserved=False` explicitly,
  keeping the reserved-name refusal this plan committed to.

  **Done.** The pin moved to `>=0.3,<0.4` (resolved `v0.3.4`, not `v0.3.3` —
  `v0.3.4` shipped first and repairs a quadratic regex on Door C's own call
  path). The 19-fixture suite was re-run against `v0.3.4` before the bump and
  reproduced the `v0.3.3` deltas exactly, with none added.
  `guard_adapter.import_gate` now passes `allow_reserved=False`, and
  `test_door_c_pins_allow_reserved_false_against_the_guards_default` pins both
  halves: that the guard still defaults `True` (without which the override is
  a no-op that would pass forever over nothing) and that Door C overrides it.

  The recorded justification is worth sharpening now that it is code: the
  guard's `True` default is right *for the guard*, and this library does not
  dispute the safety reasoning behind it. Door C's refusal is structural — it
  generates the merged bundle's `index.md` from what it merged and writes every
  merged concept verbatim, so a sender's `index.md` is a second and
  irreconcilable claim on one path, not merely a risk to be scanned.
- **`origin`/`channel` are validated against the guard's pinned vocabulary.**
  The guard derives trust from `origin` by enum *identity*, so an unrecognised
  string would be silently downgraded to untrusted. The library refuses to
  carry a provenance declaration it cannot recognise rather than let a typo
  decide trust — refusing to guess, not deciding trust itself.

## Design decisions fixed by this plan

- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
  both; no defaults.
- Extraction failure (corrupt file, missing extra) is a typed error on that file;
  the run continues with the remaining files and the result reports per-file
  outcomes. A guard FAIL-grade disposition on one file likewise never aborts the
  whole inbox run.
- Determinism: same inbox bytes + same `ingested_at` + same guard version ⇒
  byte-identical persisted output. Guard verdicts are part of that input surface.

## TDD order

1. Extraction registry: core types (fixtures per type), fail-fast on unknown
   extensions, fail-fast on optional types without `[extract]`.
2. Inbox provenance rendering + filename slugging (pure functions).
3. Door B flow against a stub guard (in tests only — a test double standing in
   for the pinned guard API, so persist/reject branches are exercised
   deterministically; the real guard is exercised in integration tests).
4. Door B integration with the real guard: benign fixture persists; a fixture
   the guard fails-secure on is not persisted.
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only
   accepted concepts merged; collision and curated-preservation semantics reused
   from Phase 1 tests.

## Key assumptions (each with its test)

| # | Assumption | Test |
|---|---|---|
| B1 | Guard 0.2 API matches the pinned surface (`screen_output`, `okf.import_bundle`, disposition enum, origin/channel vocabularies) | Import-and-signature smoke test that fails on upgrade drift (`tests/test_guard_adapter.py`) — CLOSED at step 4 |
| B2 | Guard is installable where this library's CI runs | CLOSED: git+https tag install over anonymously readable HTTPS, no credential; Forgejo package index becomes the durable channel later without a pyproject edit |
| B3 | `html.parser`-based extraction is adequate for v1 | Golden fixtures for representative HTML; anything richer is explicitly out of scope |
| B4 | Guard verdicts are deterministic for fixed input + version | Same fixture run twice → identical `InboxResult` |

## Non-goals

- Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
- Quarantine storage/workflow and re-scan over time — extension points; the
  guard's companion-expectations list is revisited in Phase 3.
- Model-assisted enrichment of inbox content — never in the run path.
- Watchers, schedulers, implicit triggers.

## Verification

1. `pytest` green with the guard installed; `mypy --strict src/`; `ruff check .`
   and `ruff format --check .` clean.
2. `pdf` file without `[extract]` → typed error naming the extra; with
   `pip install .[extract]` the same file extracts (integration test, may be
   skipped in environments without the extra — but must run in CI).
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new
   files in the bundle and an explicit rejection entry in the result (named
   test asserts the bundle directory is byte-identical before/after).
4. Door C partial-merge proof: mixed fixture → accepted concepts present,
   rejected absent, index links only for accepted, curated lines untouched.
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
   security reimplementation (guard imports only).
7. `pyproject.toml` runtime dependencies == exactly one range on
   `llm-ingestion-guard` (automated:
   `test_the_only_runtime_dependency_is_the_security_boundary`; the range
   itself is `>=0.3,<0.4` since the bump).