Kjell Tore Guttormsen
f7385fafdb
All three pilots accepted. Their measurements land in the plan now rather than as failed runs later, which is the entire return on asking before building. Six corrections, each measured by the consumer rather than reasoned by us: - A-E6 specified `okf_version: "0.2"` quoted. Catalog measured the quoted form failing their shape regex with exit 1, and a UTF-8 BOM making the marker invisible while still exiting 0. Both now stated as raw-byte expectations. - A-E2 said the diff lies only *inside* the frontmatter block. No bundle whose root index has no frontmatter block can satisfy that together with A-E6. - B-E3 covers one of catalog's three gates; the other two cannot see our fixture. Recorded so the report is not read as broader than it is. - Our claim that no gate keys on the version value was wrong as formulated: check-okf-parity.mjs puts the raw value in its signature. It compares without judging -- verdict-invariant, not value-blind. - C-E1's ground truth is 524 non-reserved documents, not 522; 522 was correct at d2c12d2. - `materialize_bundle` gains keyword-only `profile=DEFAULT` on po-claude's request. The signature already has a `*`, so their three-positional call path stays source-compatible and additivity becomes a property of the signature instead of something a consumer measures. Also recorded: po-claude closed their index-parser defect themselves (f41264f), so do not design around it; the assumption under it -- "a generated index has no frontmatter" -- is not peculiar to them. Dropping the superseded `timestamp` draws one catalog warning per concept file, decided after the pilot, not before. The estimate's open caveat is discharged by reading profiles.py:197: emit is one formatting path, the validator already admits list values, so only the emitter side is missing and the sizing stands. CLAUDE.md gains a standing constraint: consumer content stays at form level in public files. The wiki's bundle is private pending an Anthropic ToS assessment, their report goes back through coord, and a push is not reversible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
180 lines
9.1 KiB
Markdown
180 lines
9.1 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
## Context
|
||
|
||
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
||
one boundary rule:
|
||
|
||
- **Door A — spec-based ingestion:** implements the normative
|
||
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` → index generation; zero model calls). This repo
|
||
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
||
needs go via commons, never edited locally. The library ships the §11
|
||
golden fixtures (byte-exact) for the three door-A source types
|
||
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
||
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
||
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
||
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
||
the optional `[extract]` extra; without it those types are rejected
|
||
fail-fast.
|
||
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
||
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
||
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
||
load-bearing: a merged concept is written **verbatim** (this library's
|
||
line-oriented frontmatter parser cannot round-trip the block lists the
|
||
guard's parser accepts, so stamping an external concept would destroy sender
|
||
data and persist bytes the guard never screened), and ownership is therefore
|
||
proven by **content identity** — an occupied target name is re-used only
|
||
when the bytes there are already identical, never overwritten otherwise.
|
||
|
||
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
||
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
||
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
||
plumbing: connect source → materialize deterministic OKF bundle → generate
|
||
index. Never reimplement security; call the guard at persist gates
|
||
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
||
side of the boundary something belongs on: ask the operator.
|
||
|
||
**Implementation baseline:** the stricter behaviors from
|
||
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
||
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
||
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
||
|
||
### Roadmap (committed phases — all known OKF surfaces get covered)
|
||
|
||
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
||
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
||
`portfolio-optimiser`.
|
||
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
||
import, guard-gated.
|
||
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
||
sets, index shape, and reserved-file policy become config instead of
|
||
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
|
||
profile). Two consumers hold opposite postures on whether an index is
|
||
authored or directory-derived, so neither is a library invariant and
|
||
nothing here enumerates a directory unless the profile says derived.
|
||
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
||
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
||
marketplace precedent) for the second-brain world: bundle check, index
|
||
generation, inbox split/frontmatter/write, and doc conversion
|
||
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
||
and the marketplace catalog.
|
||
|
||
The two halves share the OKF contract and fixture suite, **not code**.
|
||
|
||
### Upstream version policy (standing, non-negotiable)
|
||
|
||
**The library always supports the current latest version of Google OKF.** Set by
|
||
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
|
||
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
|
||
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
|
||
|
||
Support is **additive, expressed as a new profile**, never a migration of
|
||
existing ones. This is what makes the policy sustainable instead of a recurring
|
||
crisis, and it is bounded by three facts that do not yield to it:
|
||
|
||
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
|
||
commons' call, raised there, never patched locally.
|
||
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
|
||
repo's contract from here violates O2.
|
||
- `okf_version`'s *value* belongs to catalog (decision E1).
|
||
|
||
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
|
||
pre-release tag and is revised on their feedback before general availability —
|
||
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
|
||
latest version supported as *stable*, so flipping that alias is the GA event, not
|
||
a merge side effect.
|
||
|
||
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
|
||
declares a version its shape has not earned. Where upstream itself defers a
|
||
contract — v0.2's attestation receipt and verdict wire formats — the format is
|
||
supported and the unspecified runtime is not; it re-enters scope when upstream
|
||
specifies it. Because "always latest" decays silently, the release checklist
|
||
carries an upstream-version re-check.
|
||
|
||
Phase 4 preconditions (coordination, not unilateral moves):
|
||
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
||
innboks libs) in agreement with okr and the marketplace catalog; the
|
||
catalog remains the convention owner and re-pins its shared gate here.
|
||
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
||
plugin-local by design (different lifecycle) — do not normalize it.
|
||
- The Node-side persist gate remains security territory: guard-as-contract
|
||
(per okr's adoption doc) until a Node guard exists in the security repo.
|
||
No security reimplementation here, in either runtime.
|
||
|
||
### Non-goals (all phases)
|
||
|
||
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
||
- Embedding/RAG/retrieval layers.
|
||
- Security functionality — always the guard's domain.
|
||
|
||
## Stack
|
||
|
||
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
||
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
|
||
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
||
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
||
guard; importing the package does not. Install channel until the package
|
||
index exists (a direct reference is a channel, not the pin):
|
||
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
|
||
Binary extraction parsers live behind the `[extract]` extra only.
|
||
|
||
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
||
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
||
vendoring per plugin rather than npm publishing. The halves share contract
|
||
and fixtures, never code.
|
||
|
||
## Conventions
|
||
|
||
- Type hints everywhere; `mypy --strict` target.
|
||
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
||
(no wall-clock defaults); LF-only output; golden fixtures compared
|
||
byte-for-byte.
|
||
- No model calls anywhere in the run path.
|
||
- Credentials only as env-var *references* resolved at runtime; never in
|
||
manifests, logs, or frontmatter.
|
||
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
||
otherwise.
|
||
- Conventional Commits: `type(scope): description`.
|
||
- English for all code, docs, and commit messages (public repo).
|
||
|
||
## Commands
|
||
|
||
- Test: `pytest`
|
||
- Lint: `ruff check .` + `ruff format --check .`
|
||
- Type check: `mypy --strict src/`
|
||
|
||
## Workflow
|
||
|
||
- TDD: no production code without a failing test first.
|
||
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
||
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
||
session state or internal briefs. No secrets, sober English prose, no
|
||
marketing language.
|
||
- **Consumer content stays at form level in public files.** Some consumers we
|
||
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
|
||
assessment). Key names, counts, gate names, profile fields and contract shapes
|
||
are publishable; page bodies, full title or path lists from a consumer's
|
||
bundle, and Anthropic-derived prose are not. Findings about a private
|
||
consumer's data go back to them through coord, never as a file here. This
|
||
costs nothing — every question this library asks of a consumer is about shapes
|
||
and key sets — and it is not reversible once pushed.
|
||
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
||
Never GitHub.
|
||
|
||
## Communication patterns
|
||
|
||
### Linking to local files
|
||
|
||
When pointing to local files in responses, always use markdown link syntax
|
||
with a descriptive name:
|
||
|
||
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
||
`file:///...` URLs or autolinks `<file://...>`.
|
||
- Always use absolute paths. Never `~/` or relative paths.
|
||
- For multiple files, render as a bullet list of named markdown links.
|
||
|
||
Why: bare `file://` URLs only render the first as clickable across multiple
|
||
lines. Named markdown links make each entry independently clickable and look
|
||
cleaner.
|