Kjell Tore Guttormsen
241e00f27a
Adds llm_ingestion_guard>=0.2,<0.3 as this library's first and only runtime dependency, and guard_adapter.py -- the one module that imports it. The flows themselves are unchanged: they still take an injected gate, and importing the package still does not import the guard, so a Door A consumer is unaffected by the dependency's state. Door B screens the exact bytes it persists. The guard's §6 bookends (prepare_input -> model -> screen_output) assume a model call in between; this library makes none, and prepare_input returns prompt-shaped text (sanitized AND spotlight-fenced with a per-call nonce) that must never reach disk. So the adapter calls screen_output alone, on the extracted text as it stands, and hands that same text back -- the verdict is then a statement about the bytes actually written. This supersedes the plan's "bookends" wording, recorded there under "Settled during implementation (step 4)". It follows that the gate refuses rather than repairs: a file carrying an invisible carrier is rejected, not stripped and persisted. Sanitizing first would write a document differing invisibly from the operator's file while source_sha256 still points at the original bytes. Operator decision; the policy is PRESET_USER_UPLOAD, so any finding at all is held back. Door C hands the bundle over whole to okf.import_bundle, which resolves the cross-link graph across concepts. Per-concept reasons are derived from the scan findings (severity:label) because stamp_concept keeps the disposition and drops the reason strings behind it. Assumption B1 closes as a signature smoke test over what the adapters actually call -- screen_output and okf.import_bundle signatures, the Disposition values both doors compare by value, the Origin/Channel vocabularies Door C validates, the result fields read, and the upload preset's shape. prepare_input is not pinned: drift there cannot reach this library. Behaviour is pinned against the real scanner too, including the persist-gate proof that a fail-secure fixture leaves the bundle byte-identical. B2 closes with it: git+https tag install over anonymously readable HTTPS, no credential. A direct reference is an install-time channel, not the pin -- the range stays in pyproject, is satisfied by the tag install today, and resolves normally once the package index exists. A packaging test enforces that the guard remains the only runtime dependency (verified by hand-mutation).
139 lines
6.7 KiB
Markdown
139 lines
6.7 KiB
Markdown
# llm-ingestion-okf
|
|
|
|
## Context
|
|
|
|
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
|
one boundary rule:
|
|
|
|
- **Door A — spec-based ingestion:** implements the normative
|
|
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
|
`file`/`sql`/`http` connector → deterministic materialization of
|
|
`ingest-{id}.md` → index generation; zero model calls). This repo
|
|
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
|
needs go via commons, never edited locally. The library ships the §11
|
|
golden fixtures (byte-exact) for the three door-A source types
|
|
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
|
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
|
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
|
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
|
the optional `[extract]` extra; without it those types are rejected
|
|
fail-fast.
|
|
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
|
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
|
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
|
load-bearing: a merged concept is written **verbatim** (this library's
|
|
line-oriented frontmatter parser cannot round-trip the block lists the
|
|
guard's parser accepts, so stamping an external concept would destroy sender
|
|
data and persist bytes the guard never screened), and ownership is therefore
|
|
proven by **content identity** — an occupied target name is re-used only
|
|
when the bytes there are already identical, never overwritten otherwise.
|
|
|
|
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
|
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
|
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
|
plumbing: connect source → materialize deterministic OKF bundle → generate
|
|
index. Never reimplement security; call the guard at persist gates
|
|
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
|
side of the boundary something belongs on: ask the operator.
|
|
|
|
**Implementation baseline:** the stricter behaviors from
|
|
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
|
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
|
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
|
|
|
### Roadmap (committed phases — all known OKF surfaces get covered)
|
|
|
|
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
|
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
|
`portfolio-optimiser`.
|
|
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
|
import, guard-gated.
|
|
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
|
sets, and reserved-file policy become config instead of constants;
|
|
proving consumer is `claude-code-llm-wiki` (`strict-v1` profile).
|
|
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
|
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
|
marketplace precedent) for the second-brain world: bundle check, index
|
|
generation, inbox split/frontmatter/write, and doc conversion
|
|
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
|
and the marketplace catalog.
|
|
|
|
The two halves share the OKF contract and fixture suite, **not code**.
|
|
|
|
Phase 4 preconditions (coordination, not unilateral moves):
|
|
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
|
innboks libs) in agreement with okr and the marketplace catalog; the
|
|
catalog remains the convention owner and re-pins its shared gate here.
|
|
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
|
plugin-local by design (different lifecycle) — do not normalize it.
|
|
- The Node-side persist gate remains security territory: guard-as-contract
|
|
(per okr's adoption doc) until a Node guard exists in the security repo.
|
|
No security reimplementation here, in either runtime.
|
|
|
|
### Non-goals (all phases)
|
|
|
|
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
|
- Embedding/RAG/retrieval layers.
|
|
- Security functionality — always the guard's domain.
|
|
|
|
## Stack
|
|
|
|
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
|
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
|
|
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
|
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
|
guard; importing the package does not. Install channel until the package
|
|
index exists (a direct reference is a channel, not the pin):
|
|
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
|
|
Binary extraction parsers live behind the `[extract]` extra only.
|
|
|
|
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
|
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
|
vendoring per plugin rather than npm publishing. The halves share contract
|
|
and fixtures, never code.
|
|
|
|
## Conventions
|
|
|
|
- Type hints everywhere; `mypy --strict` target.
|
|
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
|
(no wall-clock defaults); LF-only output; golden fixtures compared
|
|
byte-for-byte.
|
|
- No model calls anywhere in the run path.
|
|
- Credentials only as env-var *references* resolved at runtime; never in
|
|
manifests, logs, or frontmatter.
|
|
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
|
otherwise.
|
|
- Conventional Commits: `type(scope): description`.
|
|
- English for all code, docs, and commit messages (public repo).
|
|
|
|
## Commands
|
|
|
|
- Test: `pytest`
|
|
- Lint: `ruff check .` + `ruff format --check .`
|
|
- Type check: `mypy --strict src/`
|
|
|
|
## Workflow
|
|
|
|
- TDD: no production code without a failing test first.
|
|
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
|
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
|
session state or internal briefs. No secrets, sober English prose, no
|
|
marketing language.
|
|
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
|
Never GitHub.
|
|
|
|
## Communication patterns
|
|
|
|
### Linking to local files
|
|
|
|
When pointing to local files in responses, always use markdown link syntax
|
|
with a descriptive name:
|
|
|
|
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
|
`file:///...` URLs or autolinks `<file://...>`.
|
|
- Always use absolute paths. Never `~/` or relative paths.
|
|
- For multiple files, render as a bullet list of named markdown links.
|
|
|
|
Why: bare `file://` URLs only render the first as clickable across multiple
|
|
lines. Named markdown links make each entry independently clickable and look
|
|
cleaner.
|