Kjell Tore Guttormsen
499e96ed92
The standing always-latest policy had no procedure, and v0.1 -> v0.2 showed what that costs: the spec was read, the shipped example bundles were not, and five findings a spec reading had settled wrongly surfaced only after someone asked whether we had studied the examples. One of them re-sized the release itself. docs/upstream-okf-upgrade-runbook.md is that procedure. Every step names the concrete failure it prevents, and each is a failure that actually happened: - Pin before reading. Two repos read main independently, which felt like two confirmations and was one unstable reference read twice -- and SPEC.md was edited after the migration commit, so neither alone was the whole answer. - Enumerate the whole okf/ tree. bundles/ held the real examples and was on nobody's list until the tree was listed. - Read the shipped examples, not only the normative text. A directory named samples/ turned out to be a runner config, not samples of the format. And markdown-converting fetch tooling strips exactly the byte-level properties (BOM, trailing newline, quoting) that break downstream gates. - Name fixtures by repo AND path. Two repos had examples/ingest-golden-file/ holding different fixtures with zero content overlap, and a pilot baseline was agreed by a name that pointed at two files. The black-box commitment is now an invariant in CLAUDE.md rather than an emergent property: a consumer's cost for an upstream release is a re-run. Additive profiles, byte-stable existing profiles, keyword-only new parameters so positional call sites survive, no churn in consumer fixtures. The boundary is stated rather than glossed -- shape changes are absorbed, upstream changes to consumer-authored content are not, and those get a measured exposure report per consumer instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
197 lines
10 KiB
Markdown
197 lines
10 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
## Context
|
||
|
||
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
||
one boundary rule:
|
||
|
||
- **Door A — spec-based ingestion:** implements the normative
|
||
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` → index generation; zero model calls). This repo
|
||
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
||
needs go via commons, never edited locally. The library ships the §11
|
||
golden fixtures (byte-exact) for the three door-A source types
|
||
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
||
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
||
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
||
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
||
the optional `[extract]` extra; without it those types are rejected
|
||
fail-fast.
|
||
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
||
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
||
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
||
load-bearing: a merged concept is written **verbatim** (this library's
|
||
line-oriented frontmatter parser cannot round-trip the block lists the
|
||
guard's parser accepts, so stamping an external concept would destroy sender
|
||
data and persist bytes the guard never screened), and ownership is therefore
|
||
proven by **content identity** — an occupied target name is re-used only
|
||
when the bytes there are already identical, never overwritten otherwise.
|
||
|
||
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
||
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
||
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
||
plumbing: connect source → materialize deterministic OKF bundle → generate
|
||
index. Never reimplement security; call the guard at persist gates
|
||
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
||
side of the boundary something belongs on: ask the operator.
|
||
|
||
**Implementation baseline:** the stricter behaviors from
|
||
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
||
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
||
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
||
|
||
### Roadmap (committed phases — all known OKF surfaces get covered)
|
||
|
||
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
||
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
||
`portfolio-optimiser`.
|
||
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
||
import, guard-gated.
|
||
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
||
sets, index shape, and reserved-file policy become config instead of
|
||
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
|
||
profile). Two consumers hold opposite postures on whether an index is
|
||
authored or directory-derived, so neither is a library invariant and
|
||
nothing here enumerates a directory unless the profile says derived.
|
||
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
||
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
||
marketplace precedent) for the second-brain world: bundle check, index
|
||
generation, inbox split/frontmatter/write, and doc conversion
|
||
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
||
and the marketplace catalog.
|
||
|
||
The two halves share the OKF contract and fixture suite, **not code**.
|
||
|
||
### Upstream version policy (standing, non-negotiable)
|
||
|
||
**The library always supports the current latest version of Google OKF.** Set by
|
||
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
|
||
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
|
||
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
|
||
|
||
Support is **additive, expressed as a new profile**, never a migration of
|
||
existing ones. This is what makes the policy sustainable instead of a recurring
|
||
crisis, and it is bounded by three facts that do not yield to it:
|
||
|
||
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
|
||
commons' call, raised there, never patched locally.
|
||
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
|
||
repo's contract from here violates O2.
|
||
- `okf_version`'s *value* belongs to catalog (decision E1).
|
||
|
||
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
|
||
pre-release tag and is revised on their feedback before general availability —
|
||
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
|
||
latest version supported as *stable*, so flipping that alias is the GA event, not
|
||
a merge side effect.
|
||
|
||
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
|
||
declares a version its shape has not earned. Where upstream itself defers a
|
||
contract — v0.2's attestation receipt and verdict wire formats — the format is
|
||
supported and the unspecified runtime is not; it re-enters scope when upstream
|
||
specifies it. Because "always latest" decays silently, the release checklist
|
||
carries an upstream-version re-check.
|
||
|
||
**Every upstream release runs `docs/upstream-okf-upgrade-runbook.md`.** Pin the
|
||
commit, enumerate the whole `okf/` tree, **read the shipped example bundles and not
|
||
only `SPEC.md`**, classify the diff, measure our exposure and each consumer's, plan
|
||
additively, pilot before GA, then inform every OKF-consuming repo. The runbook is
|
||
not optional and not a summary of good intentions: each step names the concrete
|
||
failure it prevents, and all of them are failures that happened during v0.1 → v0.2.
|
||
|
||
**This repo is a black box for its consumers.** The target cost of an upstream
|
||
release to a consuming repo is **a re-run, nothing more**: support is additive
|
||
(a new profile, never a migration), existing profiles stay byte-stable, new public
|
||
parameters are keyword-only with defaults so positional call sites stay
|
||
source-compatible, and consumer golden fixtures must not churn. The boundary is
|
||
stated every time rather than glossed — the library absorbs *shape* changes, not
|
||
upstream changes to content a consumer authored (v0.2's `timestamp` and
|
||
`# Citations` supersessions). For that class the deliverable is a measured exposure
|
||
report per consumer, sent before they ask.
|
||
|
||
Phase 4 preconditions (coordination, not unilateral moves):
|
||
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
||
innboks libs) in agreement with okr and the marketplace catalog; the
|
||
catalog remains the convention owner and re-pins its shared gate here.
|
||
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
||
plugin-local by design (different lifecycle) — do not normalize it.
|
||
- The Node-side persist gate remains security territory: guard-as-contract
|
||
(per okr's adoption doc) until a Node guard exists in the security repo.
|
||
No security reimplementation here, in either runtime.
|
||
|
||
### Non-goals (all phases)
|
||
|
||
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
||
- Embedding/RAG/retrieval layers.
|
||
- Security functionality — always the guard's domain.
|
||
|
||
## Stack
|
||
|
||
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
||
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
|
||
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
||
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
||
guard; importing the package does not. Install channel until the package
|
||
index exists (a direct reference is a channel, not the pin):
|
||
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
|
||
Binary extraction parsers live behind the `[extract]` extra only.
|
||
|
||
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
||
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
||
vendoring per plugin rather than npm publishing. The halves share contract
|
||
and fixtures, never code.
|
||
|
||
## Conventions
|
||
|
||
- Type hints everywhere; `mypy --strict` target.
|
||
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
||
(no wall-clock defaults); LF-only output; golden fixtures compared
|
||
byte-for-byte.
|
||
- No model calls anywhere in the run path.
|
||
- Credentials only as env-var *references* resolved at runtime; never in
|
||
manifests, logs, or frontmatter.
|
||
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
||
otherwise.
|
||
- Conventional Commits: `type(scope): description`.
|
||
- English for all code, docs, and commit messages (public repo).
|
||
|
||
## Commands
|
||
|
||
- Test: `pytest`
|
||
- Lint: `ruff check .` + `ruff format --check .`
|
||
- Type check: `mypy --strict src/`
|
||
|
||
## Workflow
|
||
|
||
- TDD: no production code without a failing test first.
|
||
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
||
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
||
session state or internal briefs. No secrets, sober English prose, no
|
||
marketing language.
|
||
- **Consumer content stays at form level in public files.** Some consumers we
|
||
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
|
||
assessment). Key names, counts, gate names, profile fields and contract shapes
|
||
are publishable; page bodies, full title or path lists from a consumer's
|
||
bundle, and Anthropic-derived prose are not. Findings about a private
|
||
consumer's data go back to them through coord, never as a file here. This
|
||
costs nothing — every question this library asks of a consumer is about shapes
|
||
and key sets — and it is not reversible once pushed.
|
||
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
||
Never GitHub.
|
||
|
||
## Communication patterns
|
||
|
||
### Linking to local files
|
||
|
||
When pointing to local files in responses, always use markdown link syntax
|
||
with a descriptive name:
|
||
|
||
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
||
`file:///...` URLs or autolinks `<file://...>`.
|
||
- Always use absolute paths. Never `~/` or relative paths.
|
||
- For multiple files, render as a bullet list of named markdown links.
|
||
|
||
Why: bare `file://` URLs only render the first as clickable across multiple
|
||
lines. Named markdown links make each entry independently clickable and look
|
||
cleaner.
|