llm-ingestion-okf/CLAUDE.md

# llm-ingestion-okf

## Context

Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
one boundary rule:

- **Door A — spec-based ingestion:** implements the normative
  `ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
  `file`/`sql`/`http` connector → deterministic materialization of
  `ingest-{id}.md` → index generation; zero model calls). This repo
  IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
  needs go via commons, never edited locally. The library ships the §11
  golden fixtures (byte-exact) for the three door-A source types
  (`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
  file-type→text extraction lives HERE (the guard is text-only). v1 core:
  `md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
  the optional `[extract]` extra; without it those types are rejected
  fail-fast.
- **Door C — external bundle import:** third-party OKF bundles are assessed
  per concept via the guard's `okf.import_bundle`; only concepts clearing the
  guard's non-blocking floor are merged/indexed here. Two invariants, both
  load-bearing: a merged concept is written **verbatim** (this library's
  line-oriented frontmatter parser cannot round-trip the block lists the
  guard's parser accepts, so stamping an external concept would destroy sender
  data and persist bytes the guard never screened), and ownership is therefore
  proven by **content identity** — an occupied target name is re-used only
  when the bytes there are already identical, never overwritten otherwise.

**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
plumbing: connect source → materialize deterministic OKF bundle → generate
index. Never reimplement security; call the guard at persist gates
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
side of the boundary something belongs on: ask the operator.

**Implementation baseline:** the stricter behaviors from
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
are the library baseline. First consumer: `portfolio-optimiser-claude`.

### Roadmap (committed phases — all known OKF surfaces get covered)

1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
   golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
   `portfolio-optimiser`.
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
   import, guard-gated.
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
   sets, index shape, and reserved-file policy become config instead of
   constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
   profile). Two consumers hold opposite postures on whether an index is
   authored or directory-derived, so neither is a library invariant and
   nothing here enumerates a directory unless the profile says derived.
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
   (importable *and* CLI-invokable, vendorable per plugin — matching the
   marketplace precedent) for the second-brain world: bundle check, index
   generation, inbox split/frontmatter/write, and doc conversion
   (docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
   and the marketplace catalog.

The two halves share the OKF contract and fixture suite, **not code**.

**Upstream version: everything above targets OKF v0.1.** Google published v0.2
on 2026-07-25 (two supersessions, both with documented consumer fallbacks; all
new fields optional). Alignment is a policy track between Phases 3 and 4, not a
phase: `docs/plan/okf-v0.2-alignment.md`. Two constraints bind here — the
`generated` marker `DEFAULT` emits sits in commons' ingest-spec §5 layer, so its
shape is commons' call, not ours; and `okf_version`'s value belongs to catalog
(decision E1). No profile may hard-code an upstream version, and no bundle
claims a version its shape has not earned.

Phase 4 preconditions (coordination, not unilateral moves):
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
  innboks libs) in agreement with okr and the marketplace catalog; the
  catalog remains the convention owner and re-pins its shared gate here.
- linkedin-studio's `ingest/published/` provenance-record grammar stays
  plugin-local by design (different lifecycle) — do not normalize it.
- The Node-side persist gate remains security territory: guard-as-contract
  (per okr's adoption doc) until a Node guard exists in the security repo.
  No security reimplementation here, in either runtime.

### Non-goals (all phases)

- Verdict/feedback machinery (method-spec) — stays in consumer repos.
- Embedding/RAG/retrieval layers.
- Security functionality — always the guard's domain.

## Stack

Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
guard; importing the package does not. Install channel until the package
index exists (a direct reference is a channel, not the pin):
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
Binary extraction parsers live behind the `[extract]` extra only.

Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
(`node:` builtins only), both importable and CLI-invokable, consumed by
vendoring per plugin rather than npm publishing. The halves share contract
and fixtures, never code.

## Conventions

- Type hints everywhere; `mypy --strict` target.
- Determinism is bit-exact: `ingested_at` is an explicit required argument
  (no wall-clock defaults); LF-only output; golden fixtures compared
  byte-for-byte.
- No model calls anywhere in the run path.
- Credentials only as env-var *references* resolved at runtime; never in
  manifests, logs, or frontmatter.
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
  otherwise.
- Conventional Commits: `type(scope): description`.
- English for all code, docs, and commit messages (public repo).

## Commands

- Test: `pytest`
- Lint: `ruff check .` + `ruff format --check .`
- Type check: `mypy --strict src/`

## Workflow

- TDD: no production code without a failing test first.
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
  and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
  session state or internal briefs. No secrets, sober English prose, no
  marketing language.
- After `git commit`: push to Forgejo (`git push origin`) immediately.
  Never GitHub.

## Communication patterns

### Linking to local files

When pointing to local files in responses, always use markdown link syntax
with a descriptive name:

- Use `[Human-friendly name](file:///absolute/path)` — never bare
  `file:///...` URLs or autolinks `<file://...>`.
- Always use absolute paths. Never `~/` or relative paths.
- For multiple files, render as a bullet list of named markdown links.

Why: bare `file://` URLs only render the first as clickable across multiple
lines. Named markdown links make each entry independently clickable and look
cleaner.