Kjell Tore Guttormsen
f536e1384d
Measure first, widen after. The 19-fixture guard-surface suite was re-run against v0.3.4 in a scratch venv before the range moved, and reproduced the three deltas measured against v0.3.3 exactly, with none added. v0.3.4 is the tag pinned rather than v0.3.3 because it shipped first and repairs a quadratic regex (okf._MD_LINK_RE) that sits on Door C's own call path. Door C now passes allow_reserved=False explicitly. The guard added the keyword in the 0.3 line and defaults it True for received bundles, which would merge a sender's index.md / log.md instead of rejecting them. The override keeps the unconditional reserved-name refusal committed to before the keyword existed, and the reason is structural rather than a second opinion on the guard's scan: Door C generates the merged bundle's index.md from what it merged and writes every merged concept verbatim, so a sender's index.md would be a second and irreconcilable claim on one path. This is not a behaviour change for anyone on the previous pin: under v0.2.0 the keyword did not exist and reserved names were refused by construction. The floor is >=0.3 and not >=0.2 for a measured reason. allow_reserved is absent in v0.2.0 and present from v0.3.0 onward, checked across all five tags: a >=0.2 floor would admit a version that raises TypeError on every Door C import. That measurement also corrects a recorded premise -- the plan said the keyword "shipped in v0.3.3", which read the first version we ran the suite against as the version it was introduced in. The conclusion held; the reason did not, and the reason is what a future bump would have relied on. test_door_c_pins_allow_reserved_false_against_the_guards_default locks both halves: that the guard still defaults True, without which the override is a no-op that would pass forever over nothing, and that Door C overrides it. 586 tests, mypy --strict clean, goldens byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V2v1hrDhrff2H3y2TNJHkF
226 lines
12 KiB
Markdown
226 lines
12 KiB
Markdown
# llm-ingestion-okf
|
||
|
||
## Context
|
||
|
||
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
||
one boundary rule:
|
||
|
||
- **Door A — spec-based ingestion:** implements the normative
|
||
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
||
`file`/`sql`/`http` connector → deterministic materialization of
|
||
`ingest-{id}.md` → index generation; zero model calls). This repo
|
||
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
||
needs go via commons, never edited locally. The library ships the §11
|
||
golden fixtures (byte-exact) for the three door-A source types
|
||
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
||
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
||
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
||
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
||
the optional `[extract]` extra; without it those types are rejected
|
||
fail-fast.
|
||
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
||
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
||
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
||
load-bearing: a merged concept is written **verbatim** (this library's
|
||
line-oriented frontmatter parser cannot round-trip the block lists the
|
||
guard's parser accepts, so stamping an external concept would destroy sender
|
||
data and persist bytes the guard never screened), and ownership is therefore
|
||
proven by **content identity** — an occupied target name is re-used only
|
||
when the bytes there are already identical, never overwritten otherwise.
|
||
|
||
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
||
(pinned `>=0.3,<0.4`) answers "is this content safe to persist?" —
|
||
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
||
plumbing: connect source → materialize deterministic OKF bundle → generate
|
||
index. Never reimplement security; call the guard at persist gates
|
||
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
||
side of the boundary something belongs on: ask the operator.
|
||
|
||
**Implementation baseline:** the stricter behaviors from
|
||
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
||
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
||
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
||
|
||
### Roadmap (committed phases — all known OKF surfaces get covered)
|
||
|
||
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
||
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
||
`portfolio-optimiser`.
|
||
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
||
import, guard-gated.
|
||
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
||
sets, index shape, and reserved-file policy become config instead of
|
||
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
|
||
profile). Two consumers hold opposite postures on whether an index is
|
||
authored or directory-derived, so neither is a library invariant and
|
||
nothing here enumerates a directory unless the profile says derived.
|
||
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
||
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
||
marketplace precedent) for the second-brain world: bundle check, index
|
||
generation, inbox split/frontmatter/write, and doc conversion
|
||
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
||
and the marketplace catalog.
|
||
|
||
5. **Phase 5 — MCP as a way to populate a bundle.** Committed by the operator
|
||
2026-07-27; the design is open, the scope is not. Plan:
|
||
`docs/plan/mcp-bundle-population.md`. The fork that must be settled before
|
||
any code — whether we are the MCP **server** (an agent calls our doors as
|
||
tools) or an MCP **client** (a manifest source type that pulls from someone
|
||
else's server) — decides both the owner and the dependency question, and the
|
||
two are not variants of one feature.
|
||
|
||
The two halves share the OKF contract and fixture suite, **not code**.
|
||
|
||
### Upstream version policy (standing, non-negotiable)
|
||
|
||
**The library always supports the current latest version of Google OKF.** Set by
|
||
the operator 2026-07-26. Phases 1–3 were built against v0.1; v0.2 shipped
|
||
2026-07-25, so v0.2 support is committed work — not contingent on a consumer
|
||
asking for it. Plan: `docs/plan/okf-v0.2-alignment.md`.
|
||
|
||
Support is **additive, expressed as a new profile**, never a migration of
|
||
existing ones. This is what makes the policy sustainable instead of a recurring
|
||
crisis, and it is bounded by three facts that do not yield to it:
|
||
|
||
- `DEFAULT` states commons' ingest-spec §5 layer — its `generated` shape is
|
||
commons' call, raised there, never patched locally.
|
||
- `STRICT_V1` mirrors the proving consumer's ratified contract — changing another
|
||
repo's contract from here violates O2.
|
||
- `okf_version`'s *value* belongs to catalog (decision E1).
|
||
|
||
**Rollout is pilot-first.** A new upstream version reaches a small pilot set on a
|
||
pre-release tag and is revised on their feedback before general availability —
|
||
consumers testing real data find what fixtures cannot. `OKF_LATEST` means the
|
||
latest version supported as *stable*, so flipping that alias is the GA event, not
|
||
a merge side effect.
|
||
|
||
Two invariants fall out: no profile hard-codes an upstream version, and no bundle
|
||
declares a version its shape has not earned. The first has a mechanism, not just
|
||
an intention: **a profile names a key, a caller owns its value.** `okf_version`
|
||
is declared through `materialize_bundle(..., root_frontmatter_values=...)`
|
||
because its value tracks the upstream Google version and belongs to catalog
|
||
(decision E1) — a constant here would claim a decision we do not own, and would
|
||
be the one thing to chase on every upstream release. Where upstream itself defers a
|
||
contract — v0.2's attestation receipt and verdict wire formats — the format is
|
||
supported and the unspecified runtime is not; it re-enters scope when upstream
|
||
specifies it. Because "always latest" decays silently, the release checklist
|
||
carries an upstream-version re-check.
|
||
|
||
**Structured frontmatter values are emitted in YAML *flow* form, never block.**
|
||
Both are valid YAML and an upstream reader recovers the same structure from
|
||
either, but this library's parser is line-oriented: it round-trips a flow
|
||
mapping as an opaque value and cannot read the block form at all — two block
|
||
mappings sharing an inner key (§10.2's `executor` and `attester`, both carrying
|
||
`resource`) collapse into one namespace and the first is lost silently.
|
||
Emitting block would produce bundles we cannot read back. Reading it needs the
|
||
structured reader (D1b); until then the constraint binds what we write.
|
||
|
||
**Every upstream release runs `docs/upstream-okf-upgrade-runbook.md`.** Pin the
|
||
commit, enumerate the whole `okf/` tree, **read the shipped example bundles and not
|
||
only `SPEC.md`**, classify the diff, measure our exposure and each consumer's, plan
|
||
additively, pilot before GA, then inform every OKF-consuming repo. The runbook is
|
||
not optional and not a summary of good intentions: each step names the concrete
|
||
failure it prevents, and all of them are failures that happened during v0.1 → v0.2.
|
||
|
||
**This repo is a black box for its consumers.** The target cost of an upstream
|
||
release to a consuming repo is **a re-run, nothing more**: support is additive
|
||
(a new profile, never a migration), existing profiles stay byte-stable, new public
|
||
parameters are keyword-only with defaults so positional call sites stay
|
||
source-compatible, and consumer golden fixtures must not churn. The boundary is
|
||
stated every time rather than glossed — the library absorbs *shape* changes, not
|
||
upstream changes to content a consumer authored (v0.2's `timestamp` and
|
||
`# Citations` supersessions). For that class the deliverable is a measured exposure
|
||
report per consumer, sent before they ask.
|
||
|
||
Phase 4 preconditions (coordination, not unilateral moves):
|
||
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
||
innboks libs) in agreement with okr and the marketplace catalog; the
|
||
catalog remains the convention owner and re-pins its shared gate here.
|
||
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
||
plugin-local by design (different lifecycle) — do not normalize it.
|
||
- The Node-side persist gate remains security territory: guard-as-contract
|
||
(per okr's adoption doc) until a Node guard exists in the security repo.
|
||
No security reimplementation here, in either runtime.
|
||
|
||
### Non-goals (all phases)
|
||
|
||
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
||
- Embedding/RAG/retrieval layers.
|
||
- Security functionality — always the guard's domain.
|
||
|
||
## Stack
|
||
|
||
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
||
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.3,<0.4`
|
||
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
||
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
||
guard; importing the package does not. Install channel until the package
|
||
index exists (a direct reference is a channel, not the pin):
|
||
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.3.4"`.
|
||
Binary extraction parsers live behind the `[extract]` extra only.
|
||
|
||
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
||
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
||
vendoring per plugin rather than npm publishing. The halves share contract
|
||
and fixtures, never code.
|
||
|
||
## Conventions
|
||
|
||
- Type hints everywhere; `mypy --strict` target.
|
||
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
||
(no wall-clock defaults); LF-only output; golden fixtures compared
|
||
byte-for-byte.
|
||
- Filenames and titles are normalized to Unicode NFC before use
|
||
(`materialize.reduce_to_id_grammar`, `inbox.process_inbox`): macOS/APFS
|
||
hands filenames over in decomposed form, so an `é` arrives as `e` +
|
||
combining acute. Without normalizing first, the same visual name (e.g. a
|
||
Norwegian slugger title like "linkedin-studio") reduces differently
|
||
depending on which form it arrived in, splitting one title into two
|
||
generated filenames.
|
||
- No model calls anywhere in the run path.
|
||
- Credentials only as env-var *references* resolved at runtime; never in
|
||
manifests, logs, or frontmatter.
|
||
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
||
otherwise.
|
||
- Conventional Commits: `type(scope): description`.
|
||
- English for all code, docs, and commit messages (public repo).
|
||
|
||
## Commands
|
||
|
||
- Test: `pytest`
|
||
- Lint: `ruff check .` + `ruff format --check .`
|
||
- Type check: `mypy --strict src/`
|
||
|
||
## Workflow
|
||
|
||
- TDD: no production code without a failing test first.
|
||
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
||
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
||
session state or internal briefs. No secrets, sober English prose, no
|
||
marketing language.
|
||
- **Consumer content stays at form level in public files.** Some consumers we
|
||
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
|
||
assessment). Key names, counts, gate names, profile fields and contract shapes
|
||
are publishable; page bodies, full title or path lists from a consumer's
|
||
bundle, and Anthropic-derived prose are not. Findings about a private
|
||
consumer's data go back to them through coord, never as a file here. This
|
||
costs nothing — every question this library asks of a consumer is about shapes
|
||
and key sets — and it is not reversible once pushed.
|
||
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
||
Never GitHub.
|
||
|
||
## Communication patterns
|
||
|
||
### Linking to local files
|
||
|
||
When pointing to local files in responses, always use markdown link syntax
|
||
with a descriptive name:
|
||
|
||
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
||
`file:///...` URLs or autolinks `<file://...>`.
|
||
- Always use absolute paths. Never `~/` or relative paths.
|
||
- For multiple files, render as a bullet list of named markdown links.
|
||
|
||
Why: bare `file://` URLs only render the first as clickable across multiple
|
||
lines. Named markdown links make each entry independently clickable and look
|
||
cleaner.
|