llm-ingestion-okf/CLAUDE.md

llm-ingestion-okf

Context

Shared OKF (Open Knowledge Format) ingestion library. Three entry doors, one boundary rule:

• Door A — spec-based ingestion: implements the normative ingest-spec.md owned by portfolio-optimiser-commons (manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md → index generation; zero model calls). This repo IMPLEMENTS the spec; commons keeps authorship. Spec changes the library needs go via commons, never edited locally. The library ships the §11 golden fixtures (byte-exact) that are missing today.
• Door B — bundle inbox: converts dropped files to OKF concepts. All file-type→text extraction lives HERE (the guard is text-only). v1 core: md, txt, csv, json, html (stdlib). pdf/docx/xlsx only via the optional [extract] extra; without it those types are rejected fail-fast.
• Door C — external bundle import: third-party OKF bundles are assessed per concept via the guard's okf.import_bundle; only non-rejected concepts are merged/materialized/indexed here.

Boundary rule (non-negotiable, zero overlap): llm-ingestion-guard (pinned >=0.2,<0.3) answers "is this content safe to persist?" — scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is plumbing: connect source → materialize deterministic OKF bundle → generate index. Never reimplement security; call the guard at persist gates (prepare_input/screen_output, okf.import_bundle). When in doubt which side of the boundary something belongs on: ask the operator.

Implementation baseline: the stricter behaviors from portfolio-optimiser (streaming row caps, utf-8-sig, in-memory staging with pre-mutation collision gate, validated ingested_at, typed IngestError) are the library baseline. First consumer: portfolio-optimiser-claude.

Roadmap (committed phases — all known OKF surfaces get covered)

1. Phase 1 — Door A (Python). ingest-spec implementation + the §11 golden fixtures. Consumers: portfolio-optimiser-claude first, then portfolio-optimiser.
2. Phase 2 — Doors B/C (Python). Bundle inbox and external-bundle import, guard-gated.
3. Phase 3 — Configurable bundle contract. Types, layers, frontmatter sets, and reserved-file policy become config instead of constants; proving consumer is claude-code-llm-wiki (strict-v1 profile).
4. Phase 4 — Node half (node/). Zero-dependency Node/ESM package (importable and CLI-invokable, vendorable per plugin — matching the marketplace precedent) for the second-brain world: bundle check, index generation, inbox split/frontmatter/write, and doc conversion (docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect, and the marketplace catalog.

The two halves share the OKF contract and fixture suite, not code.

Phase 4 preconditions (coordination, not unilateral moves):

• Lifts okr's reference implementations (okf-check.mjs, okf-index.mjs, innboks libs) in agreement with okr and the marketplace catalog; the catalog remains the convention owner and re-pins its shared gate here.
• linkedin-studio's ingest/published/ provenance-record grammar stays plugin-local by design (different lifecycle) — do not normalize it.
• The Node-side persist gate remains security territory: guard-as-contract (per okr's adoption doc) until a Node guard exists in the security repo. No security reimplementation here, in either runtime.

Non-goals (all phases)

• Verdict/feedback machinery (method-spec) — stays in consumer repos.
• Embedding/RAG/retrieval layers.
• Security functionality — always the guard's domain.

Stack

Python 3.10+. Package llm_ingestion_okf (src layout, hatchling). Zero runtime dependencies in the core (stdlib only, like the guard). The only permitted future runtime dependency is llm-ingestion-guard (itself zero-dep), added when the first persist gate is implemented. Binary extraction parsers live behind the [extract] extra only.

Phase 4 adds a node/ half: Node/ESM with zero npm dependencies (node: builtins only), both importable and CLI-invokable, consumed by vendoring per plugin rather than npm publishing. The halves share contract and fixtures, never code.

Conventions

• Type hints everywhere; mypy --strict target.
• Determinism is bit-exact: ingested_at is an explicit required argument (no wall-clock defaults); LF-only output; golden fixtures compared byte-for-byte.
• No model calls anywhere in the run path.
• Credentials only as env-var references resolved at runtime; never in manifests, logs, or frontmatter.
• Network access requires an explicit per-run opt-in flag; refuse fail-fast otherwise.
• Conventional Commits: type(scope): description.
• English for all code, docs, and commit messages (public repo).

Commands

• Test: pytest
• Lint: ruff check . + ruff format --check .
• Type check: mypy --strict src/

Workflow

• TDD: no production code without a failing test first.
• This repo is published PUBLICLY (open/ namespace on Forgejo). STATE.md and docs/oppstartsprompt.md are LOCAL-ONLY (gitignored) — never commit session state or internal briefs. No secrets, sober English prose, no marketing language.
• After git commit: push to Forgejo (git push origin) immediately. Never GitHub.

Communication patterns

Linking to local files

When pointing to local files in responses, always use markdown link syntax with a descriptive name:

• Use [Human-friendly name](file:///absolute/path) — never bare file:///... URLs or autolinks <file://...>.
• Always use absolute paths. Never ~/ or relative paths.
• For multiple files, render as a bullet list of named markdown links.

Why: bare file:// URLs only render the first as clickable across multiple lines. Named markdown links make each entry independently clickable and look cleaner.