llm-ingestion-okf/README.md

llm-ingestion-okf

Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import. Security delegated to llm-ingestion-guard.

Status: phases 1–3 are implemented. Phase 1 (spec-based ingestion) covers manifest validation, the file/sql/http connectors, deterministic materialization, index generation, and the golden fixture suite under examples/. Phase 2 adds the bundle inbox (process_inbox) and external-bundle import (import_bundle), both against an injected persist gate, with llm_ingestion_okf.guard_adapter wiring that gate to the real guard (see below). Phase 3 makes the bundle contract configurable, so types, layers, frontmatter sets, index shape, and reserved-file policy are carried by a profile rather than by constants (see Upstream OKF versions). Binary extraction is partial: pdf is implemented behind the optional [extract] extra, while docx/xlsx remain unimplemented and are rejected fail-fast. Phase 4 (the Node half) is planned (see docs/plan/).

Install

Python 3.10+. Neither this package nor the guard it depends on is on a package index yet. With uv, one command is enough:

uv pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"

uv resolves the guard on its own, because it reads the [tool.uv.sources] entry in the pyproject.toml of the tag it is installing, and v0.4.0 points that entry at the guard tag below. Measured 2026-07-25 and re-measured 2026-08-20 with an empty uv cache; both runs installed llm-ingestion-guard==0.2.0 + llm-ingestion-okf==0.4.0 and imported clean.

With plain pip, the transitive git dependency does not resolve on its own — install the guard first, or installing this package fails with No matching distribution found for llm-ingestion-guard:

pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
pip install "llm-ingestion-okf @ git+https://git.fromaitochitta.com/open/llm-ingestion-okf.git@v0.4.0"

The guard tag is paired to the okf tag, not to this branch: v0.4.0 declares llm-ingestion-guard>=0.2,<0.3, which v0.2.0 satisfies and later guard tags do not. main has since moved its own pin to >=0.3,<0.4 (see Requirements); that pin reaches you in the next stable tag, not in the commands above. Reading a pin off this branch and installing it against v0.4.0 is the one combination that fails.

v0.4.0 is the current stable tag. v0.5.0a2 is a pre-release for the named OKF v0.2 pilot set only; pin it only if you are one of them (see Upstream OKF versions).

Planned scope (v1)

The library provides three entry points for getting content into an OKF bundle:

1. Spec-based ingestion. An implementation of the normative ingest specification owned by portfolio-optimiser-commons: manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md concept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures.
2. Bundle inbox. A drop directory where common file types are converted to OKF concept files. All file-type→text extraction lives in this library: md, txt, csv, json, and html are handled by the stdlib core; pdf requires the optional [extract] extra and is rejected fail-fast without it; docx and xlsx ship no parser yet and are always rejected. Extracted text passes the security gate before anything is persisted.
3. External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.

Boundary: security is delegated

Security is owned by the sibling package llm-ingestion-guard (pinned >=0.3,<0.4). The division is strict:

• guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
• this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index.

No security functionality is reimplemented here.

What is gated today: read this before trusting a door

• Door A (materialize_bundle) is ungated. It calls nothing before writing to disk and writes what it is given. A caller materializing untrusted content is responsible for gating it.
• Doors B and C (process_inbox, import_bundle) gate through an adapter you pass in. Each takes a gate argument; the flow hands it the content and obeys the verdict, refusing to persist anything that does not clear the guard's non-blocking floor — including a disposition it does not recognise, and (at Door C) a concept the gate returned no verdict for. What it cannot do is check that your adapter is a real guard: a permissive stub approves everything, and the flow will believe it.

llm_ingestion_okf.guard_adapter is the adapter over the real guard, and the only module here that imports it — importing the package itself does not:

from llm_ingestion_okf import process_inbox
from llm_ingestion_okf.guard_adapter import inbox_gate

result = process_inbox(inbox_dir, bundle_dir, "2026-07-25T12:00:00Z",
                       okf_type="reference", gate=inbox_gate)

Two properties of that adapter are worth knowing before you rely on it. It screens the exact bytes it persists — the guard's prepare_input bookend prepares text for a model call, which this library never makes, so only screen_output is used and the screened string is the written string. And it refuses rather than repairs: a file carrying an invisible zero-width or bidi character is rejected, not silently stripped and written. Door B screens under the untrusted-upload policy, so any finding at all is held back rather than persisted.

This section is stated plainly because earlier wording ("calls the guard at every persist gate") described the intended end state in the present tense, and a consumer reasonably read it as safe-by-default.

Roadmap

The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:

1. Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
2. Bundle inbox and external-bundle import (Python), guard-gated — plan.
3. Configurable bundle contract (types, layers, frontmatter sets, index shape, and reserved-file policy as configuration), enabling stricter bundle profiles such as strict-v1 — plan.
4. A node/ half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.

Upstream OKF versions

The library targets the current latest version of Google's OKF. Support is additive — a new upstream version arrives as a new profile, never as a migration of an existing one — so an upstream release does not change the bytes an existing profile emits.

That guarantee is about upstream, and one profile tracks a second contract as well. DEFAULT states the ingest-spec owned by portfolio-optimiser-commons, so when they change that spec, DEFAULT follows them. It happened on 2026-08-09: generated moved from true to { by: process:okf-ingest, at: <ingested_at> }, one changed line per generated file. Upgrading across it costs a re-run and nothing more — a profile still recognises bundles stamped by earlier versions, so re-running writes in place instead of refusing. DEFAULT remains OKF v0.1 on every axis upstream owns.

Profile	Contract	Status
DEFAULT	commons' ingest-spec layer (OKF v0.1 semantics)	stable
STRICT_V1	a consumer's ratified v0.1 contract	stable
OKF_V0_2	OKF v0.2	provisional, pre-release only
OKF_LATEST	alias for the latest version supported as stable	currently DEFAULT

OKF_V0_2 ships first as a pre-release to a named pilot set and may change on their feedback without a deprecation cycle. Pin the versioned constant rather than OKF_LATEST unless you have explicitly opted into tracking; OKF_LATEST moves at general availability, which is a deliberate release event rather than a side effect of an upgrade.

Selecting a profile is keyword-only, so existing call sites are unaffected:

materialize_bundle(manifest, bundle_dir, ingested_at, profile=OKF_V0_2)

A bundle may declare the version it targets. OKF v0.2 §12 makes this a MAY, and puts the declaration in the bundle-root index.md's frontmatter block. The profile names the key; the caller supplies the value, because that value tracks the upstream version and is not this library's to decide:

materialize_bundle(
    manifest, bundle_dir, ingested_at,
    profile=OKF_V0_2,
    root_frontmatter_values={"okf_version": "0.2"},
)

Omit the argument and no frontmatter block is written. Offering a key the profile does not name is refused before anything is written to disk.

Attested computations (v0.2 §10)

OKF_V0_2 supports the Attested Computation type as a format: its five contract fields — runtime, parameters, computation, executor, attester — are emitted in canonical position, judged, and round-tripped. runtime is required for that type and for no other, which the profile expresses through FrontmatterSchema.required_by_type; a type the mapping does not name carries no extra requirement, because §14 forbids a consumer to reject on an unknown type.

Nothing here executes a computation or checks an attestation. Upstream defers the receipt and verdict wire formats, so there is no contract to implement, and the question an attestation answers — was this value produced the sanctioned way — is not this library's. It re-enters scope when upstream specifies the protocol.

On the import side, a third-party concept may name an executor or attester resource pointing at executable code. Door C imports the pointer and never the code — it writes concepts verbatim and skips every non-.md file — so such a reference may not resolve, or may resolve to a file the destination tree already holds under that path. Each one is reported in ImportResult.unverified_references; the concept still merges, because §14 forbids rejecting a bundle over a broken cross-link while §10.5 asks a consumer to surface rather than silently drop. The report names the pointer key, not the resource it points at: recovering the resource needs the structured reader.

One limit worth knowing before you write such a concept: §10.2 presents executor and attester as nested block mappings, and this library's frontmatter parser is line-oriented. It reads inline flow mappings (executor: { resource: …, receipt: [ … ] }) as opaque values that round-trip unchanged, but it cannot read the block form — two block mappings that both carry a resource collapse into one namespace and the first is lost. Write the flow form; both are valid YAML, and a real YAML consumer recovers the same structure from either.

Non-goals

• Verdict/feedback machinery from the method specification (stays in the consuming repositories).
• Embedding- or retrieval-layer functionality.
• Security functionality, in either runtime — that is always llm-ingestion-guard's domain.

Requirements

Python 3.10+, and exactly one runtime dependency — the security boundary, llm-ingestion-guard>=0.3,<0.4. Everything else is stdlib. The commands are under Install; what follows is why they look the way they do.

From a checkout, the test suite runs with:

.venv/bin/python -m pytest

The suite is the verification surface for everything above: 596 tests, run on 2026-08-21 against this branch with the [extract] extra installed. Without the extra the same suite is 589 passed and 7 skipped, measured the same day: the seven cover the parser path, and the tests holding the fail-fast rejection for an uninstalled extra run in both. It is not shipped in an installed distribution — tests/ lives at the repository root, so this command needs a clone rather than a pip install.

A git URL is a PEP 508 direct reference and pins one exact tag, so it is an install-time channel, not the pin: the range above stays the declared dependency — a wheel built from this branch carries Requires-Dist: llm-ingestion-guard<0.4,>=0.3, measured 2026-08-10 — and resolves normally once the package index exists. A wheel built from a tag carries that tag's range instead, which is why the install commands pair tag with tag.

The optional [extract] extra ships one parser, pdfplumber (MIT), for pdf; docx/xlsx are still unimplemented. It is opt-in because it pulls binary wheels (pillow, pypdfium2), which the default install must never do.

Request it by appending [extract] to the package name in whichever install command from Install you are using — this package is not on an index, so a bare pip install 'llm-ingestion-okf[extract]' does not work today, and the error message naming that command is written for the day it does. The extra is unreleased: it reaches a consumer through a tag that contains it, and no such tag exists yet.

Two properties of the extra are worth knowing before depending on its output:

• Extracted text is pinned to an exact parser version. pdfplumber pins pdfminer.six==20260107 exactly, and pdfminer.six ships date-stamped releases with no stability contract. Extraction is deterministic within a parser version and not guaranteed across one, so a golden fixture built on extracted PDF text is a fixture migration away from any parser upgrade.
• Text extraction recovers text, and nothing that is drawn. Figures, diagrams and images have no text to recover — only their captions survive — so a bundle built from drawn documents is incomplete by construction. The library says so itself: every pdf extraction emits an ExtractionWarning. Structured table recovery is separately out of scope; PDFs enter as prose.

The planned Node half targets Node/ESM with zero npm dependencies.

License

MIT — see LICENSE.