Kjell Tore Guttormsen 241e00f27a feat(guard): wire Doors B and C to the real guard (Phase 2 step 4)

Adds llm_ingestion_guard>=0.2,<0.3 as this library's first and only runtime
dependency, and guard_adapter.py -- the one module that imports it. The flows
themselves are unchanged: they still take an injected gate, and importing the
package still does not import the guard, so a Door A consumer is unaffected by
the dependency's state.

Door B screens the exact bytes it persists. The guard's §6 bookends
(prepare_input -> model -> screen_output) assume a model call in between; this
library makes none, and prepare_input returns prompt-shaped text (sanitized
AND spotlight-fenced with a per-call nonce) that must never reach disk. So the
adapter calls screen_output alone, on the extracted text as it stands, and
hands that same text back -- the verdict is then a statement about the bytes
actually written. This supersedes the plan's "bookends" wording, recorded
there under "Settled during implementation (step 4)".

It follows that the gate refuses rather than repairs: a file carrying an
invisible carrier is rejected, not stripped and persisted. Sanitizing first
would write a document differing invisibly from the operator's file while
source_sha256 still points at the original bytes. Operator decision; the
policy is PRESET_USER_UPLOAD, so any finding at all is held back.

Door C hands the bundle over whole to okf.import_bundle, which resolves the
cross-link graph across concepts. Per-concept reasons are derived from the
scan findings (severity:label) because stamp_concept keeps the disposition and
drops the reason strings behind it.

Assumption B1 closes as a signature smoke test over what the adapters actually
call -- screen_output and okf.import_bundle signatures, the Disposition values
both doors compare by value, the Origin/Channel vocabularies Door C validates,
the result fields read, and the upload preset's shape. prepare_input is not
pinned: drift there cannot reach this library. Behaviour is pinned against the
real scanner too, including the persist-gate proof that a fail-secure fixture
leaves the bundle byte-identical.

B2 closes with it: git+https tag install over anonymously readable HTTPS, no
credential. A direct reference is an install-time channel, not the pin -- the
range stays in pyproject, is satisfied by the tag install today, and resolves
normally once the package index exists. A packaging test enforces that the
guard remains the only runtime dependency (verified by hand-mutation).

2026-07-25 07:32:45 +02:00

6.7 KiB

Raw Blame History

llm-ingestion-okf

Context

Shared OKF (Open Knowledge Format) ingestion library. Three entry doors, one boundary rule:

Door A — spec-based ingestion: implements the normative ingest-spec.md owned by portfolio-optimiser-commons (manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md → index generation; zero model calls). This repo IMPLEMENTS the spec; commons keeps authorship. Spec changes the library needs go via commons, never edited locally. The library ships the §11 golden fixtures (byte-exact) for the three door-A source types (ingest-golden-{file,sql,http}/, shipped in 9dd86b1).
Door B — bundle inbox: converts dropped files to OKF concepts. All file-type→text extraction lives HERE (the guard is text-only). v1 core: md, txt, csv, json, html (stdlib). pdf/docx/xlsx only via the optional [extract] extra; without it those types are rejected fail-fast.
Door C — external bundle import: third-party OKF bundles are assessed per concept via the guard's okf.import_bundle; only concepts clearing the guard's non-blocking floor are merged/indexed here. Two invariants, both load-bearing: a merged concept is written verbatim (this library's line-oriented frontmatter parser cannot round-trip the block lists the guard's parser accepts, so stamping an external concept would destroy sender data and persist bytes the guard never screened), and ownership is therefore proven by content identity — an occupied target name is re-used only when the bytes there are already identical, never overwritten otherwise.

Boundary rule (non-negotiable, zero overlap): llm-ingestion-guard (pinned >=0.2,<0.3) answers "is this content safe to persist?" — scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is plumbing: connect source → materialize deterministic OKF bundle → generate index. Never reimplement security; call the guard at persist gates (prepare_input/screen_output, okf.import_bundle). When in doubt which side of the boundary something belongs on: ask the operator.

Implementation baseline: the stricter behaviors from portfolio-optimiser (streaming row caps, utf-8-sig, in-memory staging with pre-mutation collision gate, validated ingested_at, typed IngestError) are the library baseline. First consumer: portfolio-optimiser-claude.

Roadmap (committed phases — all known OKF surfaces get covered)

Phase 1 — Door A (Python). ingest-spec implementation + the §11 golden fixtures. Consumers: portfolio-optimiser-claude first, then portfolio-optimiser.
Phase 2 — Doors B/C (Python). Bundle inbox and external-bundle import, guard-gated.
Phase 3 — Configurable bundle contract. Types, layers, frontmatter sets, and reserved-file policy become config instead of constants; proving consumer is claude-code-llm-wiki (strict-v1 profile).
Phase 4 — Node half (node/). Zero-dependency Node/ESM package (importable and CLI-invokable, vendorable per plugin — matching the marketplace precedent) for the second-brain world: bundle check, index generation, inbox split/frontmatter/write, and doc conversion (docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect, and the marketplace catalog.

The two halves share the OKF contract and fixture suite, not code.

Phase 4 preconditions (coordination, not unilateral moves):

Lifts okr's reference implementations (okf-check.mjs, okf-index.mjs, innboks libs) in agreement with okr and the marketplace catalog; the catalog remains the convention owner and re-pins its shared gate here.
linkedin-studio's ingest/published/ provenance-record grammar stays plugin-local by design (different lifecycle) — do not normalize it.
The Node-side persist gate remains security territory: guard-as-contract (per okr's adoption doc) until a Node guard exists in the security repo. No security reimplementation here, in either runtime.

Non-goals (all phases)

Verdict/feedback machinery (method-spec) — stays in consumer repos.
Embedding/RAG/retrieval layers.
Security functionality — always the guard's domain.

Stack

Python 3.10+. Package llm_ingestion_okf (src layout, hatchling). Exactly one runtime dependency, ever: llm-ingestion-guard>=0.2,<0.3 (itself zero-dep), landed with the Door B/C persist gates. Everything else is stdlib, and a packaging test enforces it. Only guard_adapter.py imports the guard; importing the package does not. Install channel until the package index exists (a direct reference is a channel, not the pin): pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0". Binary extraction parsers live behind the [extract] extra only.

Phase 4 adds a node/ half: Node/ESM with zero npm dependencies (node: builtins only), both importable and CLI-invokable, consumed by vendoring per plugin rather than npm publishing. The halves share contract and fixtures, never code.

Conventions

Type hints everywhere; mypy --strict target.
Determinism is bit-exact: ingested_at is an explicit required argument (no wall-clock defaults); LF-only output; golden fixtures compared byte-for-byte.
No model calls anywhere in the run path.
Credentials only as env-var references resolved at runtime; never in manifests, logs, or frontmatter.
Network access requires an explicit per-run opt-in flag; refuse fail-fast otherwise.
Conventional Commits: type(scope): description.
English for all code, docs, and commit messages (public repo).

Commands

Test: pytest
Lint: ruff check . + ruff format --check .
Type check: mypy --strict src/

Workflow

TDD: no production code without a failing test first.
This repo is published PUBLICLY (open/ namespace on Forgejo). STATE.md and docs/oppstartsprompt.md are LOCAL-ONLY (gitignored) — never commit session state or internal briefs. No secrets, sober English prose, no marketing language.
After git commit: push to Forgejo (git push origin) immediately. Never GitHub.

Communication patterns

Linking to local files

When pointing to local files in responses, always use markdown link syntax with a descriptive name:

Use [Human-friendly name](file:///absolute/path) — never bare file:///... URLs or autolinks <file://...>.
Always use absolute paths. Never ~/ or relative paths.
For multiple files, render as a bullet list of named markdown links.

Why: bare file:// URLs only render the first as clickable across multiple lines. Named markdown links make each entry independently clickable and look cleaner.

6.7 KiB Raw Blame History