open/llm-ingestion-okf
1
0
Fork 0
Code Issues Pull requests Releases Activity
Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import. Security delegated to llm-ingestion-guard.
3 commits 1 branch 0 tags 1.3 MiB
  • Python 100%
Find a file
Kjell Tore Guttormsen b0ad2aedfb docs(plan): add detailed phase 1-4 implementation plans 
Phase 1 (Door A): ingest-spec v1 implementation + §11 golden fixtures,
pinned to the current spec revision, with TDD order and byte-exact
verification criteria. Phase 2 (Doors B/C): guard-gated inbox and
external bundle import. Phase 3: configurable bundle contract via a
profile object, default profile locked to the golden suite. Phase 4:
zero-dep Node half, coordination-first with owner sign-offs.

Each plan carries explicit key assumptions with tests and a
verification section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QeqhJpYQyghASjiJo5EhGg
2026-07-16 10:58:08 +02:00
docs/plan docs(plan): add detailed phase 1-4 implementation plans 2026-07-16 10:58:08 +02:00
src/llm_ingestion_okf feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
.gitignore feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
CHANGELOG.md feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
CLAUDE.md docs(scope): expand roadmap to cover all eight OKF surfaces 2026-07-16 10:32:58 +02:00
LICENSE feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
pyproject.toml feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
README.md docs(scope): expand roadmap to cover all eight OKF surfaces 2026-07-16 10:32:58 +02:00

README.md

llm-ingestion-okf

Shared ingestion library for OKF (Open Knowledge Format) bundles.

Status: pre-implementation. This repository currently contains the project scaffold and scope definition only; no pipeline code has been written yet.

Planned scope (v1)

The library provides three entry points for getting content into an OKF bundle:

  1. Spec-based ingestion. An implementation of the normative ingest specification owned by portfolio-optimiser-commons: manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md concept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures.
  2. Bundle inbox. A drop directory where common file types are converted to OKF concept files. All file-type→text extraction lives in this library: md, txt, csv, json, and html are handled by the stdlib core; pdf, docx, and xlsx require the optional [extract] extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted.
  3. External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.

Boundary: security is delegated

Security is owned by the sibling package llm-ingestion-guard (pinned >=0.2,<0.3). The division is strict:

  • guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
  • this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index — and calls the guard at every persist gate (prepare_input/screen_output for extracted text, okf.import_bundle for received bundles).

No security functionality is reimplemented here.

Roadmap

The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered:

  1. Spec-based ingestion (Python) with byte-exact golden fixtures.
  2. Bundle inbox and external-bundle import (Python), guard-gated.
  3. Configurable bundle contract (types, layers, frontmatter sets, and reserved-file policy as configuration), enabling stricter bundle profiles such as strict-v1.
  4. A node/ half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code.

Non-goals

  • Verdict/feedback machinery from the method specification (stays in the consuming repositories).
  • Embedding- or retrieval-layer functionality.
  • Security functionality, in either runtime — that is always llm-ingestion-guard's domain.

Requirements

Python 3.10+. The core has zero runtime dependencies. The planned Node half targets Node/ESM with zero npm dependencies.

License

MIT — see LICENSE.