Kjell Tore Guttormsen
f10fc60de2
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
159 lines
9.3 KiB
Markdown
159 lines
9.3 KiB
Markdown
# Phase 2 plan — Door B (bundle inbox) + Door C (external bundle import)
|
|
|
|
Status: approved roadmap phase (see `CLAUDE.md`); details settled here before code.
|
|
Depends on: Phase 1 (materialization + index primitives are reused, never duplicated).
|
|
This phase adds the library's first — and only permitted — runtime dependency:
|
|
`llm-ingestion-guard>=0.2,<0.3`.
|
|
|
|
## Goal
|
|
|
|
Two guard-gated persist paths on top of the Phase 1 plumbing:
|
|
|
|
- **Door B — bundle inbox:** convert operator-dropped files into OKF concept
|
|
files. All file-type→text extraction lives here (the guard is text-only).
|
|
- **Door C — external bundle import:** assess a third-party OKF bundle per
|
|
concept via the guard's `okf.import_bundle`; merge, materialize, and index
|
|
only the non-rejected concepts.
|
|
|
|
The boundary rule is absolute: the guard answers "is this content safe to
|
|
persist?"; this library only connects, extracts, materializes, and indexes.
|
|
No scanning, sanitizing, or quarantine logic is implemented here.
|
|
|
|
## Door B — deliverables
|
|
|
|
1. **Extraction registry** — file extension → extractor:
|
|
- Core (stdlib only): `md`/`txt` passthrough, `csv` → markdown table (reuses
|
|
the Phase 1 renderer), `json` → verbatim inside a fenced block, `html` →
|
|
text via `html.parser`.
|
|
- Optional (`[extract]` extra only): `pdf`, `docx`, `xlsx`. Without the extra
|
|
installed those types are rejected fail-fast with a typed error naming the
|
|
extra — never a silent skip, never a bundled parser in core.
|
|
2. **Inbox flow** — an explicit operator command (mirrors the spec §9 HITL rule:
|
|
never automatic, no watcher, no scheduler):
|
|
`process_inbox(inbox_dir, bundle_dir, ingested_at, ...) -> InboxResult`.
|
|
Per file: read bytes → extract text → guard gate → materialize concept →
|
|
index link. `ingested_at` explicit as in Phase 1; no wall-clock anywhere.
|
|
3. **Guard gate (persist gate #1):** extracted text passes through the guard
|
|
(`prepare_input`/`screen_output` bookends with an upload-grade policy) before
|
|
any write. Disposition handling: blocking dispositions → the file is NOT
|
|
persisted and is reported in `InboxResult`; quarantine-grade dispositions →
|
|
reported for operator review, not persisted (a quarantine directory is an
|
|
extension point, not v1); pass/warn → persisted with the report attached to
|
|
the result.
|
|
4. **Inbox provenance layer** — machine-generated files carry the honesty
|
|
marker, analogous to spec §7: `type`, `title`, `source_file` (inbox-relative
|
|
name), `source_sha256` (of the original bytes), `ingested_at`, `generated: true`.
|
|
Filenames namespaced `inbox-{slug}.md` with the slug reduced to the Phase 1
|
|
id grammar — disjoint from `index.md`, `promoted-verdict-*`, and `ingest-*`.
|
|
The verdict reservation and the pre-mutation collision gate apply unchanged.
|
|
|
|
## Door C — deliverables
|
|
|
|
1. **Import flow** — explicit operator command:
|
|
`import_bundle(source_dir, bundle_dir, ingested_at, *, origin, channel) -> ImportResult`.
|
|
Reads the external bundle as `{relative path -> text}`, hands it to the
|
|
guard's `okf.import_bundle(bundle, origin=…, channel=…)` (the guard v0.2
|
|
signature — reserved-name rejection of `index.md`/`log.md` is unconditional,
|
|
there is no `allow_reserved` toggle), and merges ONLY concepts whose
|
|
per-concept result carries no error.
|
|
2. **Merge + index:** accepted concepts go through the Phase 1 staging/collision
|
|
gate and index primitives (idempotent linking, curated content preserved).
|
|
Rejected concepts are reported per concept with the guard's reason — never
|
|
partially written.
|
|
3. **Import report:** the guard's log body (`BundleResult.log()`) is returned to
|
|
the caller in `ImportResult`; persisting it into the bundle is NOT done in v1
|
|
(reserved-file policy differs per consumer and becomes configurable in
|
|
Phase 3).
|
|
|
|
### Settled during implementation (step 5)
|
|
|
|
- **Verbatim merge, no frontmatter of ours.** A merged concept is written
|
|
exactly as the gate saw it. This is a correctness constraint, not a
|
|
preference: the guard's frontmatter parser accepts block lists, which this
|
|
library's line-oriented `parse_frontmatter` cannot round-trip, so stamping
|
|
provenance into an external concept would silently drop sender data — and
|
|
would persist bytes the guard never screened.
|
|
- **Ownership by content identity.** With no stamp available, an occupied
|
|
target name is re-used only when the bytes already there are identical (a
|
|
no-op re-merge, so re-import of an unchanged bundle is idempotent).
|
|
Anything else at the name — curated content or an updated version of the
|
|
same concept — is refused under `collision_unstamped`; the operator removes
|
|
the file to accept an update. A configurable stamping/reserved-file policy
|
|
is Phase 3's.
|
|
- **The merge floor is fail-closed, not "no error".** Deliverable 1's wording
|
|
is the looser reading: a concept carrying no error but a FAIL_SECURE or
|
|
unrecognised disposition is refused, and a concept the gate returned no
|
|
verdict for at all is refused. Only the guard's non-blocking floor merges —
|
|
the same floor Door B applies, with `quarantine_review` reported as its own
|
|
bucket rather than folded into rejection.
|
|
- **Upstream has already moved past the pin (observed, not acted on).** The
|
|
guard's `main` carries a commit that adds `allow_reserved=True` to
|
|
`okf.import_bundle` and *scans* `index.md`/`log.md` in a mode-b import
|
|
instead of path-rejecting them. The pin is the `v0.2.0` tag, where the
|
|
kwarg does not exist and rejection is unconditional, so this door is built
|
|
against the tag. Whoever bumps the pin owns re-checking that branch: today
|
|
a reserved name in a received bundle arrives as a per-concept rejection.
|
|
- **`origin`/`channel` are validated against the guard's pinned vocabulary.**
|
|
The guard derives trust from `origin` by enum *identity*, so an unrecognised
|
|
string would be silently downgraded to untrusted. The library refuses to
|
|
carry a provenance declaration it cannot recognise rather than let a typo
|
|
decide trust — refusing to guess, not deciding trust itself.
|
|
|
|
## Design decisions fixed by this plan
|
|
|
|
- Trust follows `origin`, never `channel` (guard doctrine) — callers must supply
|
|
both; no defaults.
|
|
- Extraction failure (corrupt file, missing extra) is a typed error on that file;
|
|
the run continues with the remaining files and the result reports per-file
|
|
outcomes. A guard FAIL-grade disposition on one file likewise never aborts the
|
|
whole inbox run.
|
|
- Determinism: same inbox bytes + same `ingested_at` + same guard version ⇒
|
|
byte-identical persisted output. Guard verdicts are part of that input surface.
|
|
|
|
## TDD order
|
|
|
|
1. Extraction registry: core types (fixtures per type), fail-fast on unknown
|
|
extensions, fail-fast on optional types without `[extract]`.
|
|
2. Inbox provenance rendering + filename slugging (pure functions).
|
|
3. Door B flow against a stub guard (in tests only — a test double standing in
|
|
for the pinned guard API, so persist/reject branches are exercised
|
|
deterministically; the real guard is exercised in integration tests).
|
|
4. Door B integration with the real guard: benign fixture persists; a fixture
|
|
the guard fails-secure on is not persisted.
|
|
5. Door C flow: mixed fixture bundle (accepted + rejected concepts) → only
|
|
accepted concepts merged; collision and curated-preservation semantics reused
|
|
from Phase 1 tests.
|
|
|
|
## Key assumptions (each with its test)
|
|
|
|
| # | Assumption | Test |
|
|
|---|---|---|
|
|
| B1 | Guard 0.2 API matches the pinned surface (`prepare_input`, `screen_output`, `okf.import_bundle`, disposition enum) | Import-and-signature smoke test that fails on upgrade drift |
|
|
| B2 | Guard is installable where this library's CI runs | Resolve the distribution channel with the operator before implementation starts (risk: not yet decided) |
|
|
| B3 | `html.parser`-based extraction is adequate for v1 | Golden fixtures for representative HTML; anything richer is explicitly out of scope |
|
|
| B4 | Guard verdicts are deterministic for fixed input + version | Same fixture run twice → identical `InboxResult` |
|
|
|
|
## Non-goals
|
|
|
|
- Any security decision logic (scan, sanitize, lexicons, fencing) — guard only.
|
|
- Quarantine storage/workflow and re-scan over time — extension points; the
|
|
guard's companion-expectations list is revisited in Phase 3.
|
|
- Model-assisted enrichment of inbox content — never in the run path.
|
|
- Watchers, schedulers, implicit triggers.
|
|
|
|
## Verification
|
|
|
|
1. `pytest` green with the guard installed; `mypy --strict src/`; `ruff check .`
|
|
and `ruff format --check .` clean.
|
|
2. `pdf` file without `[extract]` → typed error naming the extra; with
|
|
`pip install .[extract]` the same file extracts (integration test, may be
|
|
skipped in environments without the extra — but must run in CI).
|
|
3. Persist-gate proof: a fixture the guard fails-secure on results in ZERO new
|
|
files in the bundle and an explicit rejection entry in the result (named
|
|
test asserts the bundle directory is byte-identical before/after).
|
|
4. Door C partial-merge proof: mixed fixture → accepted concepts present,
|
|
rejected absent, index links only for accepted, curated lines untouched.
|
|
5. Phase 1 golden suite still passes byte-for-byte (no regression from reuse).
|
|
6. Grep-gate: `grep -rn "sanitize\|quarantine\|lexicon" src/` shows no local
|
|
security reimplementation (guard imports only).
|
|
7. `pyproject.toml` runtime dependencies == exactly `llm-ingestion-guard>=0.2,<0.3`.
|