- Python 100%
Kjell Tore Guttormsen
d812a839be
`process_inbox(inbox_dir, bundle_dir, ingested_at, *, okf_type, gate)`: per dropped file, bytes -> extract_text -> guard gate -> render -> collision gate -> write -> index link. Returns an InboxResult whose four buckets (persisted / quarantined / rejected / failed) are disjoint and complete, so a file that vanished shows up as a missing entry rather than as nothing. The guard is INJECTED rather than imported. The library calls no guard function and makes no security decision: the Gate adapter returns a GateDecision carrying the guard's own disposition value and the sanitized text, and the flow only branches on it. That keeps the core dependency-free while B2 (the guard's CI channel) is still open, and lets the persist and refuse branches run deterministically against a test double. Decisions taken with the operator this session: - What is persisted is the gate's SANITIZED text, not the extracted text. Screening one string and writing another would make the verdict a statement about bytes nobody kept. `sanitize` is exported by the guard precisely for callers composing the checklist themselves, so this is sanctioned API, not a reimplementation. Door B has no model call, so the fenced text the bookends produce for a transform is never persisted. - Quarantine is reported apart from rejection. QUARANTINE_REVIEW means "hold for human review" — an operator queue — where FAIL_SECURE is a decision. No quarantine directory in v1; that stays an extension point. Fails closed by construction: only the guard's non-blocking floor (`warn`) persists, so a renamed member, a future disposition or an adapter typo lands in `rejected` rather than being guessed safe. One bad file never aborts the run. Only three conditions fail the whole run, and each is wrong for every file at once: an invalid `ingested_at`, a reserved `okf_type`, and a missing inbox directory. New code `inbox_slug_collision`: two dropped files reducing to one generated name are BOTH refused. Persisting one would let iteration order decide the winner, and overwriting would lose the other's content. Phase 1 primitives are reused, never duplicated, so four helpers become package-internal names (write_bytes, link_in_index, parse_frontmatter, INDEX_NAME) and link_in_index learns to append to an empty index — Door A always seeds its index with bundle_summary, but Door B has no summary to invent and must not open with a blank line. |
||
|---|---|---|
| docs/plan | ||
| examples | ||
| src/llm_ingestion_okf | ||
| tests | ||
| .gitignore | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
llm-ingestion-okf
Shared ingestion library for OKF (Open Knowledge Format) bundles.
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
the file/sql/http connectors, deterministic materialization, index
generation, and the golden fixture suite under examples/. Phases 2–4 are
planned (see docs/plan/).
Planned scope (v1)
The library provides three entry points for getting content into an OKF bundle:
- Spec-based ingestion. An implementation of the normative ingest
specification owned by
portfolio-optimiser-commons: manifest →file/sql/httpconnector → deterministic materialization ofingest-{id}.mdconcept files → index generation. Zero model calls in the run path; output is reproducible byte-for-byte against golden fixtures. - Bundle inbox. A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
md,txt,csv,json, andhtmlare handled by the stdlib core;pdf,docx, andxlsxrequire the optional[extract]extra and are rejected fail-fast without it. Extracted text passes the security gate before anything is persisted. - External bundle import. Import and merge of third-party OKF bundles: each concept is assessed via the security gate, and only concepts that pass are merged, materialized, and linked into the index.
Boundary: security is delegated
Security is owned by the sibling package
llm-ingestion-guard
(pinned >=0.2,<0.3). The division is strict:
- guard answers "is this content safe to persist?" — scan, sanitize, quarantine, fail-secure, provenance stamping.
- this library does the plumbing — connect a source, materialize a deterministic OKF bundle, generate the index.
No security functionality is reimplemented here.
What is gated today: nothing
Door A — the only door shipped so far — is ungated. The package has zero
runtime dependencies and calls no guard function before writing to disk.
materialize_bundle writes what it is given.
If you materialize external or otherwise untrusted content, gating is your
responsibility at the call site: okf.import_bundle for received bundles, or
prepare_input/screen_output around extracted text. The guard-calling
persist gates are part of Doors B and C (phase 2), not of Door A.
This is stated plainly because the earlier wording ("calls the guard at every persist gate") described the intended end state in the present tense, and a consumer reasonably read it as safe-by-default.
Roadmap
The library is built in four phases so that every known OKF surface in the ecosystem is eventually covered. Each phase has a detailed plan with verification criteria:
- Spec-based ingestion (Python) with byte-exact golden fixtures — plan.
- Bundle inbox and external-bundle import (Python), guard-gated — plan.
- Configurable bundle contract (types, layers, frontmatter sets, and
reserved-file policy as configuration), enabling stricter bundle
profiles such as
strict-v1— plan. - A
node/half: a zero-dependency Node/ESM package (importable and CLI-invokable, vendored per consumer) providing bundle checking, index generation, inbox processing, and document conversion for the OKF second-brain plugin ecosystem. The Python and Node halves share the OKF contract and fixture suite, not code — plan.
Non-goals
- Verdict/feedback machinery from the method specification (stays in the consuming repositories).
- Embedding- or retrieval-layer functionality.
- Security functionality, in either runtime — that is always
llm-ingestion-guard's domain.
Requirements
Python 3.10+. The core has zero runtime dependencies. The planned Node half targets Node/ESM with zero npm dependencies.
License
MIT — see LICENSE.