open/llm-ingestion-okf
1
0
Fork 0
Code Issues Pull requests Releases Activity
Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import. Security delegated to llm-ingestion-guard.
46 commits 1 branch 0 tags 1.3 MiB
  • Python 100%
Find a file
Kjell Tore Guttormsen 848e3953fc feat(phase-3): STRICT_V1's type enum and frontmatter schema, with the reader 
Phase 3 step 3, frontmatter half. FrontmatterSchema grows the four judging
fields the proving consumer's contract needs — required, allowed, nullable,
key_pattern — and `violations` is what reads them, so no field lands as an
untested claim.

STRICT_V1 requires the eight keys their operator ratified on 2026-07-25 inside
the eleven-key allowlist that doubles as emission order, both derived from one
tuple so they cannot drift; `summary` is the single nullable key; a value is a
string scalar or a NON-EMPTY LIST OF STRINGS, because their Release pages carry
block lists and a scalars-only schema would reject pages they ship today.

Measured, not assumed: DEFAULT must NOT grow a required set. Door A emits seven
keys, Door B six, and Door C writes an imported concept verbatim with whatever
the sender wrote, so any required set on DEFAULT would declare invalid the
bundles this library itself produces. The plan's other cross-profile direction —
a wiki bundle rejected under DEFAULT — therefore has to come from index and path
shape, not from frontmatter. Recorded here because the plan implied otherwise.

Violations are reported, not raised, like TypeRejection, and sorted by
(key, code): a report that depended on mapping order would not be reproducible.
An unsatisfiable schema — a required or nullable key outside its own allowlist —
fails at construction.

Two boundaries kept explicit. `violations` judges an already-parsed mapping;
parsing strict frontmatter needs a parser this library does not have, which is
the same reason Door C writes verbatim. And STRICT_V1's index policy is still
DEFAULT's, pinned by a test as a known gap: per-directory indexes and an entry
description have no field on IndexPolicy yet, and nothing consumes a profile's
index policy today.

C1 re-proven: 443 existing tests unmodified and green (468 total), and
`git diff --stat examples/` empty.
2026-07-25 15:29:26 +02:00
docs docs(phase-3): the four open questions are answered; required set is eight 2026-07-25 15:21:59 +02:00
examples feat(examples): ship the §11 golden fixtures with byte-exact conformance 2026-07-16 20:09:03 +02:00
src/llm_ingestion_okf feat(phase-3): STRICT_V1's type enum and frontmatter schema, with the reader 2026-07-25 15:29:26 +02:00
tests feat(phase-3): STRICT_V1's type enum and frontmatter schema, with the reader 2026-07-25 15:29:26 +02:00
.gitignore feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
CHANGELOG.md docs: two corrections from the convention owner and the spec owner 2026-07-25 12:32:21 +02:00
CLAUDE.md feat(guard): wire Doors B and C to the real guard (Phase 2 step 4) 2026-07-25 07:32:45 +02:00
LICENSE feat: initial commit — repo scaffold and v1 scope 2026-07-16 10:12:59 +02:00
pyproject.toml docs(install): correct how the guard reaches a consumer, measured both ways 2026-07-25 12:24:54 +02:00
README.md docs(install): correct how the guard reaches a consumer, measured both ways 2026-07-25 12:24:54 +02:00
uv.lock chore(release): v0.4.0 2026-07-25 12:08:48 +02:00

README.md

llm-ingestion-okf

Shared ingestion library for OKF (Open Knowledge Format) bundles.

Status: phases 1 and 2 are implemented. Phase 1 (spec-based ingestion) covers manifest validation, the file/sql/http connectors, deterministic materialization, index generation, and the golden fixture suite under examples/. Phase 2 adds the bundle inbox (process_inbox) and external-bundle import (import_bundle), both against an injected persist gate, with llm_ingestion_okf.guard_adapter wiring that gate to the real guard (see below). One phase-2 item is deliberately outstanding: binary extraction (pdf/docx/xlsx behind the [extract] extra) is unimplemented, so those types are rejected fail-fast. Phases 3–4 are planned (see docs/plan/).

Planned scope (v1)

The library provides three entry points for getting content into an OKF bundle:

  1. Spec-based ingestion. An implementation of the normative ingest specification owned by portfolio-optimiser-commons: manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md concept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures.
  2. Bundle inbox. A drop directory where common file types are converted to OKF concept files. All file-type→text extraction lives in this library: md, txt, csv, json, and html are handled by the stdlib core; pdf, docx, and xlsx require the optional [extract] extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted.
  3. External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.

Boundary: security is delegated

Security is owned by the sibling package llm-ingestion-guard (pinned >=0.2,<0.3). The division is strict:

  • guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
  • this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index.

No security functionality is reimplemented here.

What is gated today: read this before trusting a door

  • Door A (materialize_bundle) is ungated. It calls nothing before writing to disk and writes what it is given. A caller materializing untrusted content is responsible for gating it.
  • Doors B and C (process_inbox, import_bundle) gate through an adapter you pass in. Each takes a gate argument; the flow hands it the content and obeys the verdict, refusing to persist anything that does not clear the guard's non-blocking floor — including a disposition it does not recognise, and (at Door C) a concept the gate returned no verdict for. What it cannot do is check that your adapter is a real guard: a permissive stub approves everything, and the flow will believe it.

llm_ingestion_okf.guard_adapter is the adapter over the real guard, and the only module here that imports it — importing the package itself does not:

from llm_ingestion_okf import process_inbox
from llm_ingestion_okf.guard_adapter import inbox_gate

result = process_inbox(inbox_dir, bundle_dir, "2026-07-25T12:00:00Z",
                       okf_type="reference", gate=inbox_gate)

Two properties of that adapter are worth knowing before you rely on it. It screens the exact bytes it persists — the guard's prepare_input bookend prepares text for a model call, which this library never makes, so only screen_output is used and the screened string is the written string. And it refuses rather than repairs: a file carrying an invisible zero-width or bidi character is rejected, not silently stripped and written. Door B screens under the untrusted-upload policy, so any finding at all is held back rather than persisted.

This section is stated plainly because earlier wording ("calls the guard at every persist gate") described the intended end state in the present tense, and a consumer reasonably read it as safe-by-default.

Roadmap

The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:

  1. Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
  2. Bundle inbox and external-bundle import (Python), guard-gated — plan.
  3. Configurable bundle contract (types, layers, frontmatter sets, and reserved-file policy as configuration), enabling stricter bundle profiles such as strict-v1 — plan.
  4. A node/ half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.

Non-goals

  • Verdict/feedback machinery from the method specification (stays in the consuming repositories).
  • Embedding- or retrieval-layer functionality.
  • Security functionality, in either runtime — that is always llm-ingestion-guard's domain.

Requirements

Python 3.10+, and exactly one runtime dependency — the security boundary, llm-ingestion-guard>=0.2,<0.3. Everything else is stdlib.

That guard is not on a package index yet, so with pip, install it first — otherwise installing this package fails with No matching distribution found for llm-ingestion-guard:

pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"

With uv, one command is enough — uv pip install "llm-ingestion-okf @ git+…@v0.4.0" resolves the guard from the tag on its own, because uv reads the [tool.uv.sources] entry in this project's pyproject.toml when it builds from the source tree. Both paths were measured on 2026-07-25.

A git URL is a PEP 508 direct reference and pins one exact tag, so it is an install-time channel, not the pin: the range above stays the declared dependency — the built wheel carries Requires-Dist: llm-ingestion-guard<0.3,>=0.2 — and resolves normally once the package index exists. The optional [extract] extra (pdf/docx/xlsx parsers) is not populated yet. The planned Node half targets Node/ESM with zero npm dependencies.

License

MIT — see LICENSE.