Kjell Tore Guttormsen
241e00f27a
Adds llm_ingestion_guard>=0.2,<0.3 as this library's first and only runtime dependency, and guard_adapter.py -- the one module that imports it. The flows themselves are unchanged: they still take an injected gate, and importing the package still does not import the guard, so a Door A consumer is unaffected by the dependency's state. Door B screens the exact bytes it persists. The guard's §6 bookends (prepare_input -> model -> screen_output) assume a model call in between; this library makes none, and prepare_input returns prompt-shaped text (sanitized AND spotlight-fenced with a per-call nonce) that must never reach disk. So the adapter calls screen_output alone, on the extracted text as it stands, and hands that same text back -- the verdict is then a statement about the bytes actually written. This supersedes the plan's "bookends" wording, recorded there under "Settled during implementation (step 4)". It follows that the gate refuses rather than repairs: a file carrying an invisible carrier is rejected, not stripped and persisted. Sanitizing first would write a document differing invisibly from the operator's file while source_sha256 still points at the original bytes. Operator decision; the policy is PRESET_USER_UPLOAD, so any finding at all is held back. Door C hands the bundle over whole to okf.import_bundle, which resolves the cross-link graph across concepts. Per-concept reasons are derived from the scan findings (severity:label) because stamp_concept keeps the disposition and drops the reason strings behind it. Assumption B1 closes as a signature smoke test over what the adapters actually call -- screen_output and okf.import_bundle signatures, the Disposition values both doors compare by value, the Origin/Channel vocabularies Door C validates, the result fields read, and the upload preset's shape. prepare_input is not pinned: drift there cannot reach this library. Behaviour is pinned against the real scanner too, including the persist-gate proof that a fail-secure fixture leaves the bundle byte-identical. B2 closes with it: git+https tag install over anonymously readable HTTPS, no credential. A direct reference is an install-time channel, not the pin -- the range stays in pyproject, is satisfied by the tag install today, and resolves normally once the package index exists. A packaging test enforces that the guard remains the only runtime dependency (verified by hand-mutation).
130 lines
6 KiB
Markdown
130 lines
6 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
Shared ingestion library for OKF (Open Knowledge Format) bundles.
|
||
|
||
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
|
||
the `file`/`sql`/`http` connectors, deterministic materialization, index
|
||
generation, and the golden fixture suite under `examples/`. Phase 2 is in
|
||
progress: the bundle inbox (`process_inbox`) and external-bundle import
|
||
(`import_bundle`) are implemented against an **injected** persist gate, and
|
||
`llm_ingestion_okf.guard_adapter` wires that gate to the real guard (see
|
||
below). Phases 3–4 are planned (see `docs/plan/`).
|
||
|
||
## Planned scope (v1)
|
||
|
||
The library provides three entry points for getting content into an OKF
|
||
bundle:
|
||
|
||
1. **Spec-based ingestion.** An implementation of the normative ingest
|
||
specification owned by `portfolio-optimiser-commons`: manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` concept files → index generation. Zero model calls in the
|
||
run path; output is reproducible byte-for-byte against golden fixtures.
|
||
2. **Bundle inbox.** A drop directory where common file types are converted
|
||
to OKF concept files. All file-type→text extraction lives in this library:
|
||
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
|
||
`pdf`, `docx`, and `xlsx` require the optional `[extract]` extra and are
|
||
rejected fail-fast without it. Extracted text passes the security gate
|
||
before anything is persisted.
|
||
3. **External bundle import.** Import and merge of third-party OKF bundles:
|
||
each concept is assessed via the security gate, and only concepts that
|
||
pass are merged, materialized, and linked into the index.
|
||
|
||
## Boundary: security is delegated
|
||
|
||
Security is owned by the sibling package
|
||
[`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
|
||
(pinned `>=0.2,<0.3`). The division is strict:
|
||
|
||
- **guard** answers "is this content safe to persist?" — scan, sanitize,
|
||
quarantine, fail-secure, provenance stamping.
|
||
- **this library** does the plumbing — connect a source, materialize a
|
||
deterministic OKF bundle, generate the index.
|
||
|
||
No security functionality is reimplemented here.
|
||
|
||
### What is gated today: read this before trusting a door
|
||
|
||
- **Door A (`materialize_bundle`) is ungated.** It calls nothing before
|
||
writing to disk and writes what it is given. A caller materializing
|
||
untrusted content is responsible for gating it.
|
||
- **Doors B and C (`process_inbox`, `import_bundle`) gate through an adapter
|
||
you pass in.** Each takes a `gate` argument; the flow hands it the content
|
||
and obeys the verdict, refusing to persist anything that does not clear the
|
||
guard's non-blocking floor — including a disposition it does not recognise,
|
||
and (at Door C) a concept the gate returned no verdict for. What it cannot
|
||
do is check that your adapter is a real guard: a permissive stub approves
|
||
everything, and the flow will believe it.
|
||
|
||
`llm_ingestion_okf.guard_adapter` is the adapter over the real guard, and the
|
||
only module here that imports it — importing the package itself does not:
|
||
|
||
```python
|
||
from llm_ingestion_okf import process_inbox
|
||
from llm_ingestion_okf.guard_adapter import inbox_gate
|
||
|
||
result = process_inbox(inbox_dir, bundle_dir, "2026-07-25T12:00:00Z",
|
||
okf_type="reference", gate=inbox_gate)
|
||
```
|
||
|
||
Two properties of that adapter are worth knowing before you rely on it.
|
||
It screens the **exact bytes it persists** — the guard's `prepare_input`
|
||
bookend prepares text for a model call, which this library never makes, so
|
||
only `screen_output` is used and the screened string is the written string.
|
||
And it **refuses rather than repairs**: a file carrying an invisible
|
||
zero-width or bidi character is rejected, not silently stripped and written.
|
||
Door B screens under the untrusted-upload policy, so any finding at all is
|
||
held back rather than persisted.
|
||
|
||
This section is stated plainly because earlier wording ("calls the guard at
|
||
every persist gate") described the intended end state in the present tense,
|
||
and a consumer reasonably read it as safe-by-default.
|
||
|
||
## Roadmap
|
||
|
||
The library is built in four phases so that every known OKF surface in the
|
||
ecosystem is eventually covered. Each phase has a detailed plan with
|
||
verification criteria:
|
||
|
||
1. Spec-based ingestion (Python) with byte-exact golden fixtures —
|
||
[plan](docs/plan/phase-1-door-a.md).
|
||
2. Bundle inbox and external-bundle import (Python), guard-gated —
|
||
[plan](docs/plan/phase-2-doors-b-c.md).
|
||
3. Configurable bundle contract (types, layers, frontmatter sets, and
|
||
reserved-file policy as configuration), enabling stricter bundle
|
||
profiles such as `strict-v1` —
|
||
[plan](docs/plan/phase-3-configurable-contract.md).
|
||
4. A `node/` half: a zero-dependency Node/ESM package (importable and
|
||
CLI-invokable, vendored per consumer) providing bundle checking, index
|
||
generation, inbox processing, and document conversion for the OKF
|
||
second-brain plugin ecosystem. The Python and Node halves share the OKF
|
||
contract and fixture suite, not code —
|
||
[plan](docs/plan/phase-4-node-half.md).
|
||
|
||
## Non-goals
|
||
|
||
- Verdict/feedback machinery from the method specification (stays in the
|
||
consuming repositories).
|
||
- Embedding- or retrieval-layer functionality.
|
||
- Security functionality, in either runtime — that is always
|
||
`llm-ingestion-guard`'s domain.
|
||
|
||
## Requirements
|
||
|
||
Python 3.10+, and exactly one runtime dependency — the security boundary,
|
||
`llm-ingestion-guard>=0.2,<0.3`. Everything else is stdlib. Until that
|
||
package is published to an index, install it from its tag:
|
||
|
||
```
|
||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"
|
||
```
|
||
|
||
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
|
||
install-time *channel*, not the pin: the range above stays the declared
|
||
dependency and resolves normally once the package index exists. The optional
|
||
`[extract]` extra (pdf/docx/xlsx parsers) is not populated yet. The planned
|
||
Node half targets Node/ESM with zero npm dependencies.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE).
|