open/llm-ingestion-okf
1
0
Fork 0
Code Issues Pull requests Releases Activity
Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import. Security delegated to llm-ingestion-guard.
27 commits 1 branch 0 tags 1.3 MiB
  • Python 100%
Find a file
Kjell Tore Guttormsen 1747238a83 fix(materialize): scope §5 replacement to the running manifest's stem 
The §3 ownership scan classified every ingest-stamped file as replaceable,
so materializing one manifest into a bundle deleted the ingest files another
manifest had stamped there — and removed their index links too.

Narrow ownership to files whose stamp names the running manifest by stem.
The stamp is `{stem}@{sha256[:16]}`; matching on the stem, not the whole
stamp, lets an edited manifest still reclaim the files a prior run of the
same manifest wrote (the stem is stable across content edits), while a
different manifest sharing the bundle keeps its own files. Implements the
trinn-e §10.2 decision. This does not close the operator-copy restriction
(a generated file copied into curated content), which stays documented in
ingest-spec.md:69-72, not enforced.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
2026-07-23 07:00:27 +02:00
docs/plan docs(plan): add detailed phase 1-4 implementation plans 2026-07-16 10:58:08 +02:00
examples feat(examples): ship the §11 golden fixtures with byte-exact conformance 2026-07-16 20:09:03 +02:00
src/llm_ingestion_okf fix(materialize): scope §5 replacement to the running manifest's stem 2026-07-23 07:00:27 +02:00
tests fix(materialize): scope §5 replacement to the running manifest's stem 2026-07-23 07:00:27 +02:00
.gitignore feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
CHANGELOG.md chore(release): v0.3.2 2026-07-23 06:42:30 +02:00
CLAUDE.md docs: refresh the golden-fixtures note in CLAUDE.md 2026-07-23 06:41:27 +02:00
LICENSE feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
pyproject.toml chore(release): v0.3.2 2026-07-23 06:42:30 +02:00
README.md docs(security): state plainly that Door A is ungated 2026-07-19 10:09:25 +02:00
uv.lock chore(release): v0.3.2 2026-07-23 06:42:30 +02:00

README.md

llm-ingestion-okf

Shared ingestion library for OKF (Open Knowledge Format) bundles.

Status: phase 1 (spec-based ingestion) is implemented — manifest validation, the file/sql/http connectors, deterministic materialization, index generation, and the golden fixture suite under examples/. Phases 2–4 are planned (see docs/plan/).

Planned scope (v1)

The library provides three entry points for getting content into an OKF bundle:

  1. Spec-based ingestion. An implementation of the normative ingest specification owned by portfolio-optimiser-commons: manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md concept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures.
  2. Bundle inbox. A drop directory where common file types are converted to OKF concept files. All file-type→text extraction lives in this library: md, txt, csv, json, and html are handled by the stdlib core; pdf, docx, and xlsx require the optional [extract] extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted.
  3. External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.

Boundary: security is delegated

Security is owned by the sibling package llm-ingestion-guard (pinned >=0.2,<0.3). The division is strict:

  • guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
  • this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index.

No security functionality is reimplemented here.

What is gated today: nothing

Door A — the only door shipped so far — is ungated. The package has zero runtime dependencies and calls no guard function before writing to disk. materialize_bundle writes what it is given.

If you materialize external or otherwise untrusted content, gating is your responsibility at the call site: okf.import_bundle for received bundles, or prepare_input/screen_output around extracted text. The guard-calling persist gates are part of Doors B and C (phase 2), not of Door A.

This is stated plainly because the earlier wording ("calls the guard at every persist gate") described the intended end state in the present tense, and a consumer reasonably read it as safe-by-default.

Roadmap

The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:

  1. Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
  2. Bundle inbox and external-bundle import (Python), guard-gated — plan.
  3. Configurable bundle contract (types, layers, frontmatter sets, and reserved-file policy as configuration), enabling stricter bundle profiles such as strict-v1 — plan.
  4. A node/ half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.

Non-goals

  • Verdict/feedback machinery from the method specification (stays in the consuming repositories).
  • Embedding- or retrieval-layer functionality.
  • Security functionality, in either runtime — that is always llm-ingestion-guard's domain.

Requirements

Python 3.10+. The core has zero runtime dependencies. The planned Node half targets Node/ESM with zero npm dependencies.

License

MIT — see LICENSE.