Kjell Tore Guttormsen
7fe073ac61
The module docstring and README claimed the library "calls the guard at every persist gate". That described the intended end state in the present tense. Door A -- the only door shipped -- has zero runtime dependencies and calls no guard function before writing to disk. Both places now say so, and state that gating external or untrusted content is the caller's responsibility (okf.import_bundle, or prepare_input / screen_output) until the persist gates land with Doors B and C. Reported as F2 in a dogfood review by claude-playlist-corpus, which read the earlier wording as safe-by-default and had to wire the gating itself. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WdVgowYC4LARgvNdNMiuvz
94 lines
4 KiB
Markdown
94 lines
4 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
Shared ingestion library for OKF (Open Knowledge Format) bundles.
|
||
|
||
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
|
||
the `file`/`sql`/`http` connectors, deterministic materialization, index
|
||
generation, and the golden fixture suite under `examples/`. Phases 2–4 are
|
||
planned (see `docs/plan/`).
|
||
|
||
## Planned scope (v1)
|
||
|
||
The library provides three entry points for getting content into an OKF
|
||
bundle:
|
||
|
||
1. **Spec-based ingestion.** An implementation of the normative ingest
|
||
specification owned by `portfolio-optimiser-commons`: manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` concept files → index generation. Zero model calls in the
|
||
run path; output is reproducible byte-for-byte against golden fixtures.
|
||
2. **Bundle inbox.** A drop directory where common file types are converted
|
||
to OKF concept files. All file-type→text extraction lives in this library:
|
||
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
|
||
`pdf`, `docx`, and `xlsx` require the optional `[extract]` extra and are
|
||
rejected fail-fast without it. Extracted text passes the security gate
|
||
before anything is persisted.
|
||
3. **External bundle import.** Import and merge of third-party OKF bundles:
|
||
each concept is assessed via the security gate, and only concepts that
|
||
pass are merged, materialized, and linked into the index.
|
||
|
||
## Boundary: security is delegated
|
||
|
||
Security is owned by the sibling package
|
||
[`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
|
||
(pinned `>=0.2,<0.3`). The division is strict:
|
||
|
||
- **guard** answers "is this content safe to persist?" — scan, sanitize,
|
||
quarantine, fail-secure, provenance stamping.
|
||
- **this library** does the plumbing — connect a source, materialize a
|
||
deterministic OKF bundle, generate the index.
|
||
|
||
No security functionality is reimplemented here.
|
||
|
||
### What is gated today: nothing
|
||
|
||
Door A — the only door shipped so far — is **ungated**. The package has zero
|
||
runtime dependencies and calls no guard function before writing to disk.
|
||
`materialize_bundle` writes what it is given.
|
||
|
||
If you materialize external or otherwise untrusted content, gating is **your**
|
||
responsibility at the call site: `okf.import_bundle` for received bundles, or
|
||
`prepare_input`/`screen_output` around extracted text. The guard-calling
|
||
persist gates are part of Doors B and C (phase 2), not of Door A.
|
||
|
||
This is stated plainly because the earlier wording ("calls the guard at every
|
||
persist gate") described the intended end state in the present tense, and a
|
||
consumer reasonably read it as safe-by-default.
|
||
|
||
## Roadmap
|
||
|
||
The library is built in four phases so that every known OKF surface in the
|
||
ecosystem is eventually covered. Each phase has a detailed plan with
|
||
verification criteria:
|
||
|
||
1. Spec-based ingestion (Python) with byte-exact golden fixtures —
|
||
[plan](docs/plan/phase-1-door-a.md).
|
||
2. Bundle inbox and external-bundle import (Python), guard-gated —
|
||
[plan](docs/plan/phase-2-doors-b-c.md).
|
||
3. Configurable bundle contract (types, layers, frontmatter sets, and
|
||
reserved-file policy as configuration), enabling stricter bundle
|
||
profiles such as `strict-v1` —
|
||
[plan](docs/plan/phase-3-configurable-contract.md).
|
||
4. A `node/` half: a zero-dependency Node/ESM package (importable and
|
||
CLI-invokable, vendored per consumer) providing bundle checking, index
|
||
generation, inbox processing, and document conversion for the OKF
|
||
second-brain plugin ecosystem. The Python and Node halves share the OKF
|
||
contract and fixture suite, not code —
|
||
[plan](docs/plan/phase-4-node-half.md).
|
||
|
||
## Non-goals
|
||
|
||
- Verdict/feedback machinery from the method specification (stays in the
|
||
consuming repositories).
|
||
- Embedding- or retrieval-layer functionality.
|
||
- Security functionality, in either runtime — that is always
|
||
`llm-ingestion-guard`'s domain.
|
||
|
||
## Requirements
|
||
|
||
Python 3.10+. The core has zero runtime dependencies. The planned Node half
|
||
targets Node/ESM with zero npm dependencies.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE).
|