Kjell Tore Guttormsen
a27017e635
Operator directive 2026-07-26: consumers are better served by getting the latest version early and reporting back than by us holding it until we judge it finished. This closes a real gap. Every test in the plan -- including V-A8 against upstream's reference implementation -- asks whether output is conformant. None asks whether it is usable: whether a bundle is awkward to construct, whether a rejection message is actionable, whether the profile can express what a consumer's actual data needs. Only real data surfaces that. Pilot set is three repos, one axis each, chosen for signal: portfolio-optimiser-claude for the producer path (one real manifest run), catalog for gate acceptance (their gate on our fixture, measured rather than inferred from the regex), and claude-code-llm-wiki for expressiveness across 522 real documents. The load-bearing design detail is that most of the pilot is read-only on our side: we run the v0.2 reader over real consumer bundles and send the report. That needs no adoption, no writes into their trees (O2 holds), and no change to a contract their operator ratified. Only the producer axis asks a consumer to do anything, and it asks for one run. Excluded with reasons rather than silently: okr (Node side not yet lifted), linkedin-studio (v0.2's provenance families would put implicit pressure on the ingest/published carve-out we agreed not to normalize), commons (they are deciding V1 -- a participant, not a test site). Shipping a provisional surface without owing stability rests on three rules, not on saying "provisional": OKF_LATEST does not point at v0.2 until GA, so flipping it is the GA event rather than a merge side effect; the docstring and CHANGELOG name the pilot repos; and breaking changes during the pilot get no deprecation cycle. Stating that last one up front is what buys the freedom to act on feedback -- discovering it later is what turns a pilot into a de-facto release. Feedback is requested in five named parts, because unstructured reports are not comparable across three repos. The one that matters most is "what was awkward but worked": workarounds are the highest-value signal and never appear as a failure. GA exit criteria are testable, and carry one honest limit: a three-repo pilot exercises only what those three use, so `sources` with usage_window, multi-verifier `verified`, and Attested Computation will likely go unexercised. Those stay marked provisional at GA instead of being silently promoted -- claiming otherwise would be the same unearned-claim pattern that "conform first, claim after" exists to prevent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
172 lines
8.5 KiB
Markdown
172 lines
8.5 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
## Context
|
||
|
||
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
||
one boundary rule:
|
||
|
||
- **Door A — spec-based ingestion:** implements the normative
|
||
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` → index generation; zero model calls). This repo
|
||
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
||
needs go via commons, never edited locally. The library ships the §11
|
||
golden fixtures (byte-exact) for the three door-A source types
|
||
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
||
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
||
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
||
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
||
the optional `[extract]` extra; without it those types are rejected
|
||
fail-fast.
|
||
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
||
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
||
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
||
load-bearing: a merged concept is written **verbatim** (this library's
|
||
line-oriented frontmatter parser cannot round-trip the block lists the
|
||
guard's parser accepts, so stamping an external concept would destroy sender
|
||
data and persist bytes the guard never screened), and ownership is therefore
|
||
proven by **content identity** — an occupied target name is re-used only
|
||
when the bytes there are already identical, never overwritten otherwise.
|
||
|
||
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
||
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
||
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
||
plumbing: connect source → materialize deterministic OKF bundle → generate
|
||
index. Never reimplement security; call the guard at persist gates
|
||
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
||
side of the boundary something belongs on: ask the operator.
|
||
|
||
**Implementation baseline:** the stricter behaviors from
|
||
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
||
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
||
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
||
|
||
### Roadmap (committed phases — all known OKF surfaces get covered)
|
||
|
||
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
||
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
||
`portfolio-optimiser`.
|
||
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
||
import, guard-gated.
|
||
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
||
sets, index shape, and reserved-file policy become config instead of
|
||
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
|
||
profile). Two consumers hold opposite postures on whether an index is
|
||
authored or directory-derived, so neither is a library invariant and
|
||
nothing here enumerates a directory unless the profile says derived.
|
||
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
||
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
||
marketplace precedent) for the second-brain world: bundle check, index
|
||
generation, inbox split/frontmatter/write, and doc conversion
|
||
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
||
and the marketplace catalog.
|
||
|
||
The two halves share the OKF contract and fixture suite, **not code**.
|
||
|
||
### Upstream version policy (standing, non-negotiable)
|
||
|
||
**The library always supports the current latest version of Google OKF.** Set by
|
||
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
|
||
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
|
||
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
|
||
|
||
Support is **additive, expressed as a new profile**, never a migration of
|
||
existing ones. This is what makes the policy sustainable instead of a recurring
|
||
crisis, and it is bounded by three facts that do not yield to it:
|
||
|
||
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
|
||
commons' call, raised there, never patched locally.
|
||
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
|
||
repo's contract from here violates O2.
|
||
- `okf_version`'s *value* belongs to catalog (decision E1).
|
||
|
||
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
|
||
pre-release tag and is revised on their feedback before general availability —
|
||
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
|
||
latest version supported as *stable*, so flipping that alias is the GA event, not
|
||
a merge side effect.
|
||
|
||
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
|
||
declares a version its shape has not earned. Where upstream itself defers a
|
||
contract — v0.2's attestation receipt and verdict wire formats — the format is
|
||
supported and the unspecified runtime is not; it re-enters scope when upstream
|
||
specifies it. Because "always latest" decays silently, the release checklist
|
||
carries an upstream-version re-check.
|
||
|
||
Phase 4 preconditions (coordination, not unilateral moves):
|
||
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
||
innboks libs) in agreement with okr and the marketplace catalog; the
|
||
catalog remains the convention owner and re-pins its shared gate here.
|
||
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
||
plugin-local by design (different lifecycle) — do not normalize it.
|
||
- The Node-side persist gate remains security territory: guard-as-contract
|
||
(per okr's adoption doc) until a Node guard exists in the security repo.
|
||
No security reimplementation here, in either runtime.
|
||
|
||
### Non-goals (all phases)
|
||
|
||
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
||
- Embedding/RAG/retrieval layers.
|
||
- Security functionality — always the guard's domain.
|
||
|
||
## Stack
|
||
|
||
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
||
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
|
||
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
||
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
||
guard; importing the package does not. Install channel until the package
|
||
index exists (a direct reference is a channel, not the pin):
|
||
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
|
||
Binary extraction parsers live behind the `[extract]` extra only.
|
||
|
||
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
||
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
||
vendoring per plugin rather than npm publishing. The halves share contract
|
||
and fixtures, never code.
|
||
|
||
## Conventions
|
||
|
||
- Type hints everywhere; `mypy --strict` target.
|
||
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
||
(no wall-clock defaults); LF-only output; golden fixtures compared
|
||
byte-for-byte.
|
||
- No model calls anywhere in the run path.
|
||
- Credentials only as env-var *references* resolved at runtime; never in
|
||
manifests, logs, or frontmatter.
|
||
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
||
otherwise.
|
||
- Conventional Commits: `type(scope): description`.
|
||
- English for all code, docs, and commit messages (public repo).
|
||
|
||
## Commands
|
||
|
||
- Test: `pytest`
|
||
- Lint: `ruff check .` + `ruff format --check .`
|
||
- Type check: `mypy --strict src/`
|
||
|
||
## Workflow
|
||
|
||
- TDD: no production code without a failing test first.
|
||
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
||
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
||
session state or internal briefs. No secrets, sober English prose, no
|
||
marketing language.
|
||
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
||
Never GitHub.
|
||
|
||
## Communication patterns
|
||
|
||
### Linking to local files
|
||
|
||
When pointing to local files in responses, always use markdown link syntax
|
||
with a descriptive name:
|
||
|
||
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
||
`file:///...` URLs or autolinks `<file://...>`.
|
||
- Always use absolute paths. Never `~/` or relative paths.
|
||
- For multiple files, render as a bullet list of named markdown links.
|
||
|
||
Why: bare `file://` URLs only render the first as clickable across multiple
|
||
lines. Named markdown links make each entry independently clickable and look
|
||
cleaner.
|