Kjell Tore Guttormsen
f10fc60de2
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
4.8 KiB
llm-ingestion-okf
Shared ingestion library for OKF (Open Knowledge Format) bundles.
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
the file/sql/http connectors, deterministic materialization, index
generation, and the golden fixture suite under examples/. Phase 2 is in
progress: the bundle inbox (process_inbox) and external-bundle import
(import_bundle) are implemented against an injected persist gate; the
integration with the real guard is not done yet (see below). Phases 3–4 are
planned (see docs/plan/).
Planned scope (v1)
The library provides three entry points for getting content into an OKF bundle:
- Spec-based ingestion. An implementation of the normative ingest
specification owned by
portfolio-optimiser-commons: manifest →file/sql/httpconnector → deterministic materialization ofingest-{id}.mdconcept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures. - Bundle inbox. A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
md,txt,csv,json, andhtmlare handled by the stdlib core;pdf,docx, andxlsxrequire the optional[extract]extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted. - External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.
Boundary: security is delegated
Security is owned by the sibling package
llm-ingestion-guard
(pinned >=0.2,<0.3). The division is strict:
- guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
- this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index.
No security functionality is reimplemented here.
What is gated today: read this before trusting a door
The package still has zero runtime dependencies and imports no guard function anywhere. That has a direct consequence for what "gated" means here:
- Door A (
materialize_bundle) is ungated. It calls nothing before writing to disk and writes what it is given. - Doors B and C (
process_inbox,import_bundle) gate through an adapter you supply. Each takes agateargument; the flow hands it the content and obeys the verdict, refusing to persist anything that does not clear the guard's non-blocking floor — including a disposition it does not recognise, and (at Door C) a concept the gate returned no verdict for. What it cannot do is check that your adapter is a real guard. A permissive stub approves everything, and the flow will believe it.
So gating remains your responsibility at the call site: pass an adapter
over prepare_input/screen_output (Door B) or okf.import_bundle
(Door C). Wiring the doors to the real guard, and pinning it as a dependency,
is step 4 of phase 2 and is not done yet.
This is stated plainly because earlier wording ("calls the guard at every persist gate") described the intended end state in the present tense, and a consumer reasonably read it as safe-by-default.
Roadmap
The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:
- Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
- Bundle inbox and external-bundle import (Python), guard-gated — plan.
- Configurable bundle contract (types, layers, frontmatter sets, and
reserved-file policy as configuration), enabling stricter bundle
profiles such as
strict-v1— plan. - A
node/half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.
Non-goals
- Verdict/feedback machinery from the method specification (stays in the consuming repositories).
- Embedding- or retrieval-layer functionality.
- Security functionality, in either runtime — that is always
llm-ingestion-guard's domain.
Requirements
Python 3.10+. The core has zero runtime dependencies. The planned Node half targets Node/ESM with zero npm dependencies.
License
MIT — see LICENSE.