Kjell Tore Guttormsen
2504011010
Placement settled by catalog's own reading of upstream at the pinned commit
3fcbb9f: SS8:509-510 and SS12:773-775 both put `okf_version` in a bundle-root
`index.md` frontmatter block, and SS12 calls it the only place frontmatter is
permitted in an index. Catalog's spec says the opposite about the same file;
that divergence is theirs against upstream, and we conform to upstream.
The value never touches a profile. `OKF_V0_2.index.root_frontmatter` names the
key; the caller supplies the value through a new keyword-only
`root_frontmatter_values` mapping. That keeps V4/V-A5 intact - `okf_version`'s
value tracks the upstream Google version and belongs to catalog (E1), so a
constant here would claim a decision we do not own and would have to be chased
on every upstream release. In the fixture the value is fixture DATA
(`okf-version.txt`), not a literal in our source.
Ordering comes from the policy, not the caller's mapping: a dict preserves
insertion order, so two callers passing the same keys would otherwise emit
different bytes. A key the policy does not name is refused fail-fast, before
any disk mutation. Omitting the argument emits no block at all - SS12 is a MAY
and none of upstream's four reference bundles declares the key.
The block is written only when the index is CREATED, so a re-run into an
existing bundle stays byte-identical (A-E5).
Raw-byte assertions rather than parsed ones, on the committed fixture as well
as on fresh runs: catalog measured that a quoted value fails their shape regex
with exit 1 and that a BOM hides the marker while still exiting 0.
`yaml.safe_load` returns "0.2" either way and strips a BOM first, so a parsed
assertion masks exactly those two defects. Asserting the frozen fixture catches
what a self-comparison cannot - regenerating from a broken emitter moves both
sides together.
A-E6 is now placement-explicit (promised catalog in 99cf987), and separates the
two byte properties: BOM-free is a property of the file, unquoted is a property
of CATALOG'S GATE and not of OKF v0.2 - upstream's own SS12 example is quoted,
so their gate rejects the spec's canonical form.
README gains the upstream-version section it was missing; CLAUDE.md gains the
mechanism behind "no profile hard-codes an upstream version": a profile names a
key, a caller owns its value.
550 -> 559 tests. test_profile_threading's `OKF_V0_2.index is DEFAULT.index`
assertion is replaced rather than deleted: object identity was a proxy for "the
shipped profiles differ in no NAME-bearing field", which is what makes the
synthetic test profile necessary, so the guard now asserts that directly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dgkSPjkLpACjMayd9R5jx
210 lines
11 KiB
Markdown
210 lines
11 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
## Context
|
||
|
||
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
||
one boundary rule:
|
||
|
||
- **Door A — spec-based ingestion:** implements the normative
|
||
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` → index generation; zero model calls). This repo
|
||
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
||
needs go via commons, never edited locally. The library ships the §11
|
||
golden fixtures (byte-exact) for the three door-A source types
|
||
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
||
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
||
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
||
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
||
the optional `[extract]` extra; without it those types are rejected
|
||
fail-fast.
|
||
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
||
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
||
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
||
load-bearing: a merged concept is written **verbatim** (this library's
|
||
line-oriented frontmatter parser cannot round-trip the block lists the
|
||
guard's parser accepts, so stamping an external concept would destroy sender
|
||
data and persist bytes the guard never screened), and ownership is therefore
|
||
proven by **content identity** — an occupied target name is re-used only
|
||
when the bytes there are already identical, never overwritten otherwise.
|
||
|
||
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
||
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
||
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
||
plumbing: connect source → materialize deterministic OKF bundle → generate
|
||
index. Never reimplement security; call the guard at persist gates
|
||
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
||
side of the boundary something belongs on: ask the operator.
|
||
|
||
**Implementation baseline:** the stricter behaviors from
|
||
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
||
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
||
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
||
|
||
### Roadmap (committed phases — all known OKF surfaces get covered)
|
||
|
||
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
||
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
||
`portfolio-optimiser`.
|
||
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
||
import, guard-gated.
|
||
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
||
sets, index shape, and reserved-file policy become config instead of
|
||
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
|
||
profile). Two consumers hold opposite postures on whether an index is
|
||
authored or directory-derived, so neither is a library invariant and
|
||
nothing here enumerates a directory unless the profile says derived.
|
||
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
||
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
||
marketplace precedent) for the second-brain world: bundle check, index
|
||
generation, inbox split/frontmatter/write, and doc conversion
|
||
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
||
and the marketplace catalog.
|
||
|
||
5. **Phase 5 — MCP as a way to populate a bundle.** Committed by the operator
|
||
2026-07-27; the design is open, the scope is not. Plan:
|
||
`docs/plan/mcp-bundle-population.md`. The fork that must be settled before
|
||
any code — whether we are the MCP **server** (an agent calls our doors as
|
||
tools) or an MCP **client** (a manifest source type that pulls from someone
|
||
else's server) — decides both the owner and the dependency question, and the
|
||
two are not variants of one feature.
|
||
|
||
The two halves share the OKF contract and fixture suite, **not code**.
|
||
|
||
### Upstream version policy (standing, non-negotiable)
|
||
|
||
**The library always supports the current latest version of Google OKF.** Set by
|
||
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
|
||
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
|
||
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
|
||
|
||
Support is **additive, expressed as a new profile**, never a migration of
|
||
existing ones. This is what makes the policy sustainable instead of a recurring
|
||
crisis, and it is bounded by three facts that do not yield to it:
|
||
|
||
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
|
||
commons' call, raised there, never patched locally.
|
||
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
|
||
repo's contract from here violates O2.
|
||
- `okf_version`'s *value* belongs to catalog (decision E1).
|
||
|
||
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
|
||
pre-release tag and is revised on their feedback before general availability —
|
||
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
|
||
latest version supported as *stable*, so flipping that alias is the GA event, not
|
||
a merge side effect.
|
||
|
||
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
|
||
declares a version its shape has not earned. The first has a mechanism, not just
|
||
an intention: **a profile names a key, a caller owns its value.** `okf_version`
|
||
is declared through `materialize_bundle(..., root_frontmatter_values=...)`
|
||
because its value tracks the upstream Google version and belongs to catalog
|
||
(decision E1) — a constant here would claim a decision we do not own, and would
|
||
be the one thing to chase on every upstream release. Where upstream itself defers a
|
||
contract — v0.2's attestation receipt and verdict wire formats — the format is
|
||
supported and the unspecified runtime is not; it re-enters scope when upstream
|
||
specifies it. Because "always latest" decays silently, the release checklist
|
||
carries an upstream-version re-check.
|
||
|
||
**Every upstream release runs `docs/upstream-okf-upgrade-runbook.md`.** Pin the
|
||
commit, enumerate the whole `okf/` tree, **read the shipped example bundles and not
|
||
only `SPEC.md`**, classify the diff, measure our exposure and each consumer's, plan
|
||
additively, pilot before GA, then inform every OKF-consuming repo. The runbook is
|
||
not optional and not a summary of good intentions: each step names the concrete
|
||
failure it prevents, and all of them are failures that happened during v0.1 → v0.2.
|
||
|
||
**This repo is a black box for its consumers.** The target cost of an upstream
|
||
release to a consuming repo is **a re-run, nothing more**: support is additive
|
||
(a new profile, never a migration), existing profiles stay byte-stable, new public
|
||
parameters are keyword-only with defaults so positional call sites stay
|
||
source-compatible, and consumer golden fixtures must not churn. The boundary is
|
||
stated every time rather than glossed — the library absorbs *shape* changes, not
|
||
upstream changes to content a consumer authored (v0.2's `timestamp` and
|
||
`# Citations` supersessions). For that class the deliverable is a measured exposure
|
||
report per consumer, sent before they ask.
|
||
|
||
Phase 4 preconditions (coordination, not unilateral moves):
|
||
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
||
innboks libs) in agreement with okr and the marketplace catalog; the
|
||
catalog remains the convention owner and re-pins its shared gate here.
|
||
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
||
plugin-local by design (different lifecycle) — do not normalize it.
|
||
- The Node-side persist gate remains security territory: guard-as-contract
|
||
(per okr's adoption doc) until a Node guard exists in the security repo.
|
||
No security reimplementation here, in either runtime.
|
||
|
||
### Non-goals (all phases)
|
||
|
||
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
||
- Embedding/RAG/retrieval layers.
|
||
- Security functionality — always the guard's domain.
|
||
|
||
## Stack
|
||
|
||
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
||
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
|
||
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
||
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
||
guard; importing the package does not. Install channel until the package
|
||
index exists (a direct reference is a channel, not the pin):
|
||
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
|
||
Binary extraction parsers live behind the `[extract]` extra only.
|
||
|
||
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
||
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
||
vendoring per plugin rather than npm publishing. The halves share contract
|
||
and fixtures, never code.
|
||
|
||
## Conventions
|
||
|
||
- Type hints everywhere; `mypy --strict` target.
|
||
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
||
(no wall-clock defaults); LF-only output; golden fixtures compared
|
||
byte-for-byte.
|
||
- No model calls anywhere in the run path.
|
||
- Credentials only as env-var *references* resolved at runtime; never in
|
||
manifests, logs, or frontmatter.
|
||
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
||
otherwise.
|
||
- Conventional Commits: `type(scope): description`.
|
||
- English for all code, docs, and commit messages (public repo).
|
||
|
||
## Commands
|
||
|
||
- Test: `pytest`
|
||
- Lint: `ruff check .` + `ruff format --check .`
|
||
- Type check: `mypy --strict src/`
|
||
|
||
## Workflow
|
||
|
||
- TDD: no production code without a failing test first.
|
||
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
||
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
||
session state or internal briefs. No secrets, sober English prose, no
|
||
marketing language.
|
||
- **Consumer content stays at form level in public files.** Some consumers we
|
||
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
|
||
assessment). Key names, counts, gate names, profile fields and contract shapes
|
||
are publishable; page bodies, full title or path lists from a consumer's
|
||
bundle, and Anthropic-derived prose are not. Findings about a private
|
||
consumer's data go back to them through coord, never as a file here. This
|
||
costs nothing — every question this library asks of a consumer is about shapes
|
||
and key sets — and it is not reversible once pushed.
|
||
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
||
Never GitHub.
|
||
|
||
## Communication patterns
|
||
|
||
### Linking to local files
|
||
|
||
When pointing to local files in responses, always use markdown link syntax
|
||
with a descriptive name:
|
||
|
||
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
||
`file:///...` URLs or autolinks `<file://...>`.
|
||
- Always use absolute paths. Never `~/` or relative paths.
|
||
- For multiple files, render as a bullet list of named markdown links.
|
||
|
||
Why: bare `file://` URLs only render the first as clickable across multiple
|
||
lines. Named markdown links make each entry independently clickable and look
|
||
cleaner.
|