# llm-ingestion-okf

Shared ingestion library for OKF (Open Knowledge Format) bundles.

Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
the `file`/`sql`/`http` connectors, deterministic materialization, index
generation, and the golden fixture suite under `examples/`. Phases 2–4 are
planned (see `docs/plan/`).

## Planned scope (v1)

The library provides three entry points for getting content into an OKF
bundle:

1. **Spec-based ingestion.** An implementation of the normative ingest
   specification owned by `portfolio-optimiser-commons`: manifest →
   `file`/`sql`/`http` connector → deterministic materialization of
   `ingest-{id}.md` concept files → index generation. Zero model calls in the
   run path; output is reproducible byte-for-byte against golden fixtures.
2. **Bundle inbox.** A drop directory where common file types are converted
   to OKF concept files. All file-type→text extraction lives in this library:
   `md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
   `pdf`, `docx`, and `xlsx` require the optional `[extract]` extra and are
   rejected fail-fast without it. Extracted text passes the security gate
   before anything is persisted.
3. **External bundle import.** Import and merge of third-party OKF bundles:
   each concept is assessed via the security gate, and only concepts that
   pass are merged, materialized, and linked into the index.

## Boundary: security is delegated

Security is owned by the sibling package
[`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
(pinned `>=0.2,<0.3`). The division is strict:

- **guard** answers "is this content safe to persist?" — scan, sanitize,
  quarantine, fail-secure, provenance stamping.
- **this library** does the plumbing — connect a source, materialize a
  deterministic OKF bundle, generate the index.

No security functionality is reimplemented here.

### What is gated today: nothing

Door A — the only door shipped so far — is **ungated**. The package has zero
runtime dependencies and calls no guard function before writing to disk.
`materialize_bundle` writes what it is given.

If you materialize external or otherwise untrusted content, gating is **your**
responsibility at the call site: `okf.import_bundle` for received bundles, or
`prepare_input`/`screen_output` around extracted text. The guard-calling
persist gates are part of Doors B and C (phase 2), not of Door A.

This is stated plainly because the earlier wording ("calls the guard at every
persist gate") described the intended end state in the present tense, and a
consumer reasonably read it as safe-by-default.

## Roadmap

The library is built in four phases so that every known OKF surface in the
ecosystem is eventually covered. Each phase has a detailed plan with
verification criteria:

1. Spec-based ingestion (Python) with byte-exact golden fixtures —
   [plan](docs/plan/phase-1-door-a.md).
2. Bundle inbox and external-bundle import (Python), guard-gated —
   [plan](docs/plan/phase-2-doors-b-c.md).
3. Configurable bundle contract (types, layers, frontmatter sets, and
   reserved-file policy as configuration), enabling stricter bundle
   profiles such as `strict-v1` —
   [plan](docs/plan/phase-3-configurable-contract.md).
4. A `node/` half: a zero-dependency Node/ESM package (importable and
   CLI-invokable, vendored per consumer) providing bundle checking, index
   generation, inbox processing, and document conversion for the OKF
   second-brain plugin ecosystem. The Python and Node halves share the OKF
   contract and fixture suite, not code —
   [plan](docs/plan/phase-4-node-half.md).

## Non-goals

- Verdict/feedback machinery from the method specification (stays in the
  consuming repositories).
- Embedding- or retrieval-layer functionality.
- Security functionality, in either runtime — that is always
  `llm-ingestion-guard`'s domain.

## Requirements

Python 3.10+. The core has zero runtime dependencies. The planned Node half
targets Node/ESM with zero npm dependencies.

## License

MIT — see [LICENSE](LICENSE).