open/llm-ingestion-okf
1
0
Fork 0
Code Issues Pull requests Releases Activity

feat(import): Door C flow against an injected import gate (Phase 2 step 5)

Browse source 
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.

Two constraints shaped the design and are pinned by tests:

- A merged concept is written VERBATIM. Stamping provenance into it would
  require round-tripping its frontmatter through this library's line-oriented
  parser, which cannot represent the block lists the guard's parser accepts --
  silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
  target name are a no-op re-merge (re-import of an unchanged bundle is
  idempotent), and anything else at the name is refused. Curated content and
  an updated concept are refused alike; refusing is what never destroys.

The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.

Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.

README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.

405 tests green; ruff, ruff format and mypy --strict clean.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 06:57:25 +02:00
commit f10fc60de2
11 changed files with 1250 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

32
README.md
View file

@ -4,7 +4,10 @@ Shared ingestion library for OKF (Open Knowledge Format) bundles.
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
the `file`/`sql`/`http` connectors, deterministic materialization, index
generation, and the golden fixture suite under `examples/`. Phases 2–4 are
generation, and the golden fixture suite under `examples/`. Phase 2 is in
progress: the bundle inbox (`process_inbox`) and external-bundle import
(`import_bundle`) are implemented against an **injected** persist gate; the
integration with the real guard is not done yet (see below). Phases 3–4 are
planned (see `docs/plan/`).
## Planned scope (v1)
@ -40,18 +43,27 @@ Security is owned by the sibling package
No security functionality is reimplemented here.
### What is gated today: nothing
### What is gated today: read this before trusting a door
Door A — the only door shipped so far — is **ungated**. The package has zero
runtime dependencies and calls no guard function before writing to disk.
`materialize_bundle` writes what it is given.
The package still has **zero runtime dependencies** and imports no guard
function anywhere. That has a direct consequence for what "gated" means here:
If you materialize external or otherwise untrusted content, gating is **your**
responsibility at the call site: `okf.import_bundle` for received bundles, or
`prepare_input`/`screen_output` around extracted text. The guard-calling
persist gates are part of Doors B and C (phase 2), not of Door A.
- **Door A (`materialize_bundle`) is ungated.** It calls nothing before
writing to disk and writes what it is given.
- **Doors B and C (`process_inbox`, `import_bundle`) gate through an adapter
you supply.** Each takes a `gate` argument; the flow hands it the content
and obeys the verdict, refusing to persist anything that does not clear the
guard's non-blocking floor — including a disposition it does not recognise,
and (at Door C) a concept the gate returned no verdict for. What it cannot
do is check that your adapter is a real guard. A permissive stub approves
everything, and the flow will believe it.
This is stated plainly because the earlier wording ("calls the guard at every
So gating remains **your** responsibility at the call site: pass an adapter
over `prepare_input`/`screen_output` (Door B) or `okf.import_bundle`
(Door C). Wiring the doors to the real guard, and pinning it as a dependency,
is step 4 of phase 2 and is not done yet.
This is stated plainly because earlier wording ("calls the guard at every
persist gate") described the intended end state in the present tense, and a
consumer reasonably read it as safe-by-default.