llm-ingestion-okf/CLAUDE.md

llm-ingestion-okf

Context

Shared OKF (Open Knowledge Format) ingestion library. Three entry doors, one boundary rule:

• Door A — spec-based ingestion: implements the normative ingest-spec.md owned by portfolio-optimiser-commons (manifest → file/sql/http connector → deterministic materialization of ingest-{id}.md → index generation; zero model calls). This repo IMPLEMENTS the spec; commons keeps authorship. Spec changes the library needs go via commons, never edited locally. The library ships the §11 golden fixtures (byte-exact) for the three door-A source types (ingest-golden-{file,sql,http}/, shipped in 9dd86b1).
• Door B — bundle inbox: converts dropped files to OKF concepts. All file-type→text extraction lives HERE (the guard is text-only). v1 core: md, txt, csv, json, html (stdlib). pdf/docx/xlsx only via the optional [extract] extra; without it those types are rejected fail-fast.
• Door C — external bundle import: third-party OKF bundles are assessed per concept via the guard's okf.import_bundle; only concepts clearing the guard's non-blocking floor are merged/indexed here. Two invariants, both load-bearing: a merged concept is written verbatim (this library's line-oriented frontmatter parser cannot round-trip the block lists the guard's parser accepts, so stamping an external concept would destroy sender data and persist bytes the guard never screened), and ownership is therefore proven by content identity — an occupied target name is re-used only when the bytes there are already identical, never overwritten otherwise.

Boundary rule (non-negotiable, zero overlap): llm-ingestion-guard (pinned >=0.2,<0.3) answers "is this content safe to persist?" — scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is plumbing: connect source → materialize deterministic OKF bundle → generate index. Never reimplement security; call the guard at persist gates (prepare_input/screen_output, okf.import_bundle). When in doubt which side of the boundary something belongs on: ask the operator.

Implementation baseline: the stricter behaviors from portfolio-optimiser (streaming row caps, utf-8-sig, in-memory staging with pre-mutation collision gate, validated ingested_at, typed IngestError) are the library baseline. First consumer: portfolio-optimiser-claude.

Roadmap (committed phases — all known OKF surfaces get covered)

1. Phase 1 — Door A (Python). ingest-spec implementation + the §11 golden fixtures. Consumers: portfolio-optimiser-claude first, then portfolio-optimiser.

2. Phase 2 — Doors B/C (Python). Bundle inbox and external-bundle import, guard-gated.

3. Phase 3 — Configurable bundle contract. Types, layers, frontmatter sets, index shape, and reserved-file policy become config instead of constants; proving consumer is claude-code-llm-wiki (strict-v1 profile). Two consumers hold opposite postures on whether an index is authored or directory-derived, so neither is a library invariant and nothing here enumerates a directory unless the profile says derived.

4. Phase 4 — Node half (node/). Zero-dependency Node/ESM package (importable and CLI-invokable, vendorable per plugin — matching the marketplace precedent) for the second-brain world: bundle check, index generation, inbox split/frontmatter/write, and doc conversion (docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect, and the marketplace catalog.

5. Phase 5 — MCP as a way to populate a bundle. Committed by the operator 2026-07-27; the design is open, the scope is not. Plan: docs/plan/mcp-bundle-population.md. The fork that must be settled before any code — whether we are the MCP server (an agent calls our doors as tools) or an MCP client (a manifest source type that pulls from someone else's server) — decides both the owner and the dependency question, and the two are not variants of one feature.

The two halves share the OKF contract and fixture suite, not code.

Upstream version policy (standing, non-negotiable)

The library always supports the current latest version of Google OKF. Set by the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped 2026-07-25, so v0.2 support is committed work — not contingent on a consumer asking for it. Plan: docs/plan/okf-v0.2-alignment.md.

Support is additive, expressed as a new profile, never a migration of existing ones. This is what makes the policy sustainable instead of a recurring crisis, and it is bounded by three facts that do not yield to it:

• DEFAULT states commons' ingest-spec §5 layer — its generated shape is commons' call, raised there, never patched locally.
• STRICT_V1 mirrors the proving consumer's ratified contract — changing another repo's contract from here violates O2.
• okf_version's value belongs to catalog (decision E1).

Rollout is pilot-first. A new upstream version reaches a small pilot set on a pre-release tag and is revised on their feedback before general availability — consumers testing real data find what fixtures cannot. OKF_LATEST means the latest version supported as stable, so flipping that alias is the GA event, not a merge side effect.

Two invariants fall out: no profile hard-codes an upstream version, and no bundle declares a version its shape has not earned. The first has a mechanism, not just an intention: a profile names a key, a caller owns its value. okf_version is declared through materialize_bundle(..., root_frontmatter_values=...) because its value tracks the upstream Google version and belongs to catalog (decision E1) — a constant here would claim a decision we do not own, and would be the one thing to chase on every upstream release. Where upstream itself defers a contract — v0.2's attestation receipt and verdict wire formats — the format is supported and the unspecified runtime is not; it re-enters scope when upstream specifies it. Because "always latest" decays silently, the release checklist carries an upstream-version re-check.

Structured frontmatter values are emitted in YAML flow form, never block. Both are valid YAML and an upstream reader recovers the same structure from either, but this library's parser is line-oriented: it round-trips a flow mapping as an opaque value and cannot read the block form at all — two block mappings sharing an inner key (§10.2's executor and attester, both carrying resource) collapse into one namespace and the first is lost silently. Emitting block would produce bundles we cannot read back. Reading it needs the structured reader (D1b); until then the constraint binds what we write.

Every upstream release runs docs/upstream-okf-upgrade-runbook.md. Pin the commit, enumerate the whole okf/ tree, read the shipped example bundles and not only SPEC.md, classify the diff, measure our exposure and each consumer's, plan additively, pilot before GA, then inform every OKF-consuming repo. The runbook is not optional and not a summary of good intentions: each step names the concrete failure it prevents, and all of them are failures that happened during v0.1 → v0.2.

This repo is a black box for its consumers. The target cost of an upstream release to a consuming repo is a re-run, nothing more: support is additive (a new profile, never a migration), existing profiles stay byte-stable, new public parameters are keyword-only with defaults so positional call sites stay source-compatible, and consumer golden fixtures must not churn. The boundary is stated every time rather than glossed — the library absorbs shape changes, not upstream changes to content a consumer authored (v0.2's timestamp and # Citations supersessions). For that class the deliverable is a measured exposure report per consumer, sent before they ask.

Phase 4 preconditions (coordination, not unilateral moves):

• Lifts okr's reference implementations (okf-check.mjs, okf-index.mjs, innboks libs) in agreement with okr and the marketplace catalog; the catalog remains the convention owner and re-pins its shared gate here.
• linkedin-studio's ingest/published/ provenance-record grammar stays plugin-local by design (different lifecycle) — do not normalize it.
• The Node-side persist gate remains security territory: guard-as-contract (per okr's adoption doc) until a Node guard exists in the security repo. No security reimplementation here, in either runtime.

Non-goals (all phases)

• Verdict/feedback machinery (method-spec) — stays in consumer repos.
• Embedding/RAG/retrieval layers.
• Security functionality — always the guard's domain.

Stack

Python 3.10+. Package llm_ingestion_okf (src layout, hatchling). Exactly one runtime dependency, ever: llm-ingestion-guard>=0.2,<0.3 (itself zero-dep), landed with the Door B/C persist gates. Everything else is stdlib, and a packaging test enforces it. Only guard_adapter.py imports the guard; importing the package does not. Install channel until the package index exists (a direct reference is a channel, not the pin): pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0". Binary extraction parsers live behind the [extract] extra only.

Phase 4 adds a node/ half: Node/ESM with zero npm dependencies (node: builtins only), both importable and CLI-invokable, consumed by vendoring per plugin rather than npm publishing. The halves share contract and fixtures, never code.

Conventions

• Type hints everywhere; mypy --strict target.
• Determinism is bit-exact: ingested_at is an explicit required argument (no wall-clock defaults); LF-only output; golden fixtures compared byte-for-byte.
• No model calls anywhere in the run path.
• Credentials only as env-var references resolved at runtime; never in manifests, logs, or frontmatter.
• Network access requires an explicit per-run opt-in flag; refuse fail-fast otherwise.
• Conventional Commits: type(scope): description.
• English for all code, docs, and commit messages (public repo).

Commands

• Test: pytest
• Lint: ruff check . + ruff format --check .
• Type check: mypy --strict src/

Workflow

• TDD: no production code without a failing test first.
• This repo is published PUBLICLY (open/ namespace on Forgejo). STATE.md and docs/oppstartsprompt.md are LOCAL-ONLY (gitignored) — never commit session state or internal briefs. No secrets, sober English prose, no marketing language.
• Consumer content stays at form level in public files. Some consumers we read are private (claude-code-llm-wiki is, pending an Anthropic ToS assessment). Key names, counts, gate names, profile fields and contract shapes are publishable; page bodies, full title or path lists from a consumer's bundle, and Anthropic-derived prose are not. Findings about a private consumer's data go back to them through coord, never as a file here. This costs nothing — every question this library asks of a consumer is about shapes and key sets — and it is not reversible once pushed.
• After git commit: push to Forgejo (git push origin) immediately. Never GitHub.

Communication patterns

Linking to local files

When pointing to local files in responses, always use markdown link syntax with a descriptive name:

• Use [Human-friendly name](file:///absolute/path) — never bare file:///... URLs or autolinks <file://...>.
• Always use absolute paths. Never ~/ or relative paths.
• For multiple files, render as a bullet list of named markdown links.

Why: bare file:// URLs only render the first as clickable across multiple lines. Named markdown links make each entry independently clickable and look cleaner.