Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import. Security delegated to llm-ingestion-guard.
- Python 100%
Kjell Tore Guttormsen
c94ed6d525
TDD step 4: read_sql executes one read-only SELECT against the sqlite database whose path is resolved at run time from the env var named by connection_ref (credentials never in the manifest). Read-only enforced by the connection mode (file:...?mode=ro), single statement via Connection.execute, §5 cell text conversion, streaming max_rows cap, and every sqlite failure wrapped in SourceError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg |
||
|---|---|---|
| docs/plan | ||
| src/llm_ingestion_okf | ||
| tests | ||
| .gitignore | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
llm-ingestion-okf
Shared ingestion library for OKF (Open Knowledge Format) bundles.
Status: pre-implementation. This repository currently contains the project
scaffold, the scope definition, and detailed plans for each phase (under
docs/plan/); no pipeline code has been written yet.
Planned scope (v1)
The library provides three entry points for getting content into an OKF bundle:
- Spec-based ingestion. An implementation of the normative ingest
specification owned by
portfolio-optimiser-commons: manifest →file/sql/httpconnector → deterministic materialization ofingest-{id}.mdconcept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures. - Bundle inbox. A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
md,txt,csv,json, andhtmlare handled by the stdlib core;pdf,docx, andxlsxrequire the optional[extract]extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted. - External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.
Boundary: security is delegated
Security is owned by the sibling package
llm-ingestion-guard
(pinned >=0.2,<0.3). The division is strict:
- guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
- this library does the plumbing — connect a source, materialize a
deterministic OKF bundle, generate the index — and calls the guard at every
persist gate (
prepare_input/screen_outputfor extracted text,okf.import_bundlefor received bundles).
No security functionality is reimplemented here.
Roadmap
The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:
- Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
- Bundle inbox and external-bundle import (Python), guard-gated — plan.
- Configurable bundle contract (types, layers, frontmatter sets, and
reserved-file policy as configuration), enabling stricter bundle
profiles such as
strict-v1— plan. - A
node/half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.
Non-goals
- Verdict/feedback machinery from the method specification (stays in the consuming repositories).
- Embedding- or retrieval-layer functionality.
- Security functionality, in either runtime — that is always
llm-ingestion-guard's domain.
Requirements
Python 3.10+. The core has zero runtime dependencies. The planned Node half targets Node/ESM with zero npm dependencies.
License
MIT — see LICENSE.