- Python 100%
Kjell Tore Guttormsen
241e00f27a
Adds llm_ingestion_guard>=0.2,<0.3 as this library's first and only runtime dependency, and guard_adapter.py -- the one module that imports it. The flows themselves are unchanged: they still take an injected gate, and importing the package still does not import the guard, so a Door A consumer is unaffected by the dependency's state. Door B screens the exact bytes it persists. The guard's §6 bookends (prepare_input -> model -> screen_output) assume a model call in between; this library makes none, and prepare_input returns prompt-shaped text (sanitized AND spotlight-fenced with a per-call nonce) that must never reach disk. So the adapter calls screen_output alone, on the extracted text as it stands, and hands that same text back -- the verdict is then a statement about the bytes actually written. This supersedes the plan's "bookends" wording, recorded there under "Settled during implementation (step 4)". It follows that the gate refuses rather than repairs: a file carrying an invisible carrier is rejected, not stripped and persisted. Sanitizing first would write a document differing invisibly from the operator's file while source_sha256 still points at the original bytes. Operator decision; the policy is PRESET_USER_UPLOAD, so any finding at all is held back. Door C hands the bundle over whole to okf.import_bundle, which resolves the cross-link graph across concepts. Per-concept reasons are derived from the scan findings (severity:label) because stamp_concept keeps the disposition and drops the reason strings behind it. Assumption B1 closes as a signature smoke test over what the adapters actually call -- screen_output and okf.import_bundle signatures, the Disposition values both doors compare by value, the Origin/Channel vocabularies Door C validates, the result fields read, and the upload preset's shape. prepare_input is not pinned: drift there cannot reach this library. Behaviour is pinned against the real scanner too, including the persist-gate proof that a fail-secure fixture leaves the bundle byte-identical. B2 closes with it: git+https tag install over anonymously readable HTTPS, no credential. A direct reference is an install-time channel, not the pin -- the range stays in pyproject, is satisfied by the tag install today, and resolves normally once the package index exists. A packaging test enforces that the guard remains the only runtime dependency (verified by hand-mutation). |
||
|---|---|---|
| docs/plan | ||
| examples | ||
| src/llm_ingestion_okf | ||
| tests | ||
| .gitignore | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
llm-ingestion-okf
Shared ingestion library for OKF (Open Knowledge Format) bundles.
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
the file/sql/http connectors, deterministic materialization, index
generation, and the golden fixture suite under examples/. Phase 2 is in
progress: the bundle inbox (process_inbox) and external-bundle import
(import_bundle) are implemented against an injected persist gate, and
llm_ingestion_okf.guard_adapter wires that gate to the real guard (see
below). Phases 3–4 are planned (see docs/plan/).
Planned scope (v1)
The library provides three entry points for getting content into an OKF bundle:
- Spec-based ingestion. An implementation of the normative ingest
specification owned by
portfolio-optimiser-commons: manifest →file/sql/httpconnector → deterministic materialization ofingest-{id}.mdconcept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures. - Bundle inbox. A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
md,txt,csv,json, andhtmlare handled by the stdlib core;pdf,docx, andxlsxrequire the optional[extract]extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted. - External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.
Boundary: security is delegated
Security is owned by the sibling package
llm-ingestion-guard
(pinned >=0.2,<0.3). The division is strict:
- guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
- this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index.
No security functionality is reimplemented here.
What is gated today: read this before trusting a door
- Door A (
materialize_bundle) is ungated. It calls nothing before writing to disk and writes what it is given. A caller materializing untrusted content is responsible for gating it. - Doors B and C (
process_inbox,import_bundle) gate through an adapter you pass in. Each takes agateargument; the flow hands it the content and obeys the verdict, refusing to persist anything that does not clear the guard's non-blocking floor — including a disposition it does not recognise, and (at Door C) a concept the gate returned no verdict for. What it cannot do is check that your adapter is a real guard: a permissive stub approves everything, and the flow will believe it.
llm_ingestion_okf.guard_adapter is the adapter over the real guard, and the
only module here that imports it — importing the package itself does not:
from llm_ingestion_okf import process_inbox
from llm_ingestion_okf.guard_adapter import inbox_gate
result = process_inbox(inbox_dir, bundle_dir, "2026-07-25T12:00:00Z",
okf_type="reference", gate=inbox_gate)
Two properties of that adapter are worth knowing before you rely on it.
It screens the exact bytes it persists — the guard's prepare_input
bookend prepares text for a model call, which this library never makes, so
only screen_output is used and the screened string is the written string.
And it refuses rather than repairs: a file carrying an invisible
zero-width or bidi character is rejected, not silently stripped and written.
Door B screens under the untrusted-upload policy, so any finding at all is
held back rather than persisted.
This section is stated plainly because earlier wording ("calls the guard at every persist gate") described the intended end state in the present tense, and a consumer reasonably read it as safe-by-default.
Roadmap
The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:
- Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
- Bundle inbox and external-bundle import (Python), guard-gated — plan.
- Configurable bundle contract (types, layers, frontmatter sets, and
reserved-file policy as configuration), enabling stricter bundle
profiles such as
strict-v1— plan. - A
node/half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.
Non-goals
- Verdict/feedback machinery from the method specification (stays in the consuming repositories).
- Embedding- or retrieval-layer functionality.
- Security functionality, in either runtime — that is always
llm-ingestion-guard's domain.
Requirements
Python 3.10+, and exactly one runtime dependency — the security boundary,
llm-ingestion-guard>=0.2,<0.3. Everything else is stdlib. Until that
package is published to an index, install it from its tag:
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
install-time channel, not the pin: the range above stays the declared
dependency and resolves normally once the package index exists. The optional
[extract] extra (pdf/docx/xlsx parsers) is not populated yet. The planned
Node half targets Node/ESM with zero npm dependencies.
License
MIT — see LICENSE.