llm-ingestion-okf/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Changed

• DEFAULT now stamps generated: { by: process:okf-ingest, at: <ingested_at> } instead of generated: true. This is a byte change in every concept file Door A writes under DEFAULT, so a consumer's own golden fixtures will show one changed line per generated file.

  The trigger is not an upstream Google release. DEFAULT states the ingest-spec layer owned by portfolio-optimiser-commons, and they ratified this shape (2026-08-02) and executed it in their spec on 2026-08-09. §7 defines the value: the actor is the fixed process:okf-ingest, at repeats ingested_at verbatim, and it is unquoted because frontmatter is parsed line-oriented — a quote would be a character in the value rather than syntax a reader strips.

  What it costs a consumer is a re-run, and nothing else. Ownership recognition is one-way: a profile carrying an actor still owns the older literal stamp, so a bundle written by an earlier version re-runs in place rather than tripping the §3 collision gate. No call signature changed and no key was added or removed.

  Two things it is NOT. It is not a migration of DEFAULT onto OKF v0.2 — the profile remains v0.1 on every axis upstream owns, and still emits no sources. And it does not make DEFAULT and OKF_V0_2 the same profile; they now agree on the stamp and continue to differ in index root frontmatter, type-conditional requirements, and sources derivation.

  Door B (process_inbox) is deliberately unchanged: its marker is generated plus source_file, it is not governed by the ingest-spec, and it stays disjoint from Door A's ingest_manifest.

• The v0.5.0 pilot set gained a fourth member: portfolio-optimiser. Admitted 2026-08-09 on their request. v0.5.0a1's entry below says "do not pin this tag outside the pilot set" and names three repos; that entry is left as written, because it records what was true when the tag shipped. This is the amendment, and the sentence still binds — the set is now those three plus portfolio-optimiser.

  The reason is the producer axis, not the count. portfolio-optimiser-claude is parked, and with it parked no original member could emit a v0.2 bundle at all: claude-code-llm-wiki is read-only in the pilot and catalog is gate-side. The new member consumes the same Door A. Nothing about the provisional surface changes: OKF_LATEST still points at DEFAULT, and the v0.2 surface may still move on pilot feedback without a deprecation cycle.

  If you pin v0.5.0a2, its guard pin is >=0.2,<0.3 — that tag resolves against guard v0.2.0, not the v0.3.4 this repo's main now uses. main moved to >=0.3,<0.4 after the tag. Also note tool.uv.sources is not inherited transitively: a consumer supplies the guard's source itself.

• The guard pin moved to >=0.3,<0.4, resolved against v0.3.4. The window is widened only after measurement, never before: the 19-fixture guard-surface suite was run against v0.3.4 in a scratch venv first, and reproduced exactly the three deltas measured against v0.3.3 — no new ones. v0.3.4's own fixes are regex-complexity repairs, one of them (okf._MD_LINK_RE) on Door C's call path, with no disposition changes.

• Door C now passes allow_reserved=False to okf.import_bundle. The guard added the keyword in the 0.3.x line and defaults it True for the received-bundle path, which would merge a sender's index.md / log.md instead of rejecting them. Door C overrides it, keeping the unconditional reserved-name refusal committed to before the keyword existed. The reason is structural rather than a second opinion on the guard's scan: Door C generates the merged bundle's index.md from what it merged and writes every merged concept verbatim, so a sender's index.md would be a second, irreconcilable claim on one path.

  This is not a behaviour change for anyone on the previous pin. Under v0.2.0 the keyword did not exist and reserved names were refused by construction; the explicit argument preserves that outcome across the bump. A consumer sees the same rejections, with the same reasons, before and after.

0.5.0a2 — 2026-07-31

This is the pre-release the pilots pin. v0.5.0a1 was tagged and abandoned unused — do not pin it. It carried a generated.by actor that the spec owner had already excluded, and it was caught before any pilot was notified.

Fixed

• OKF_V0_2's generated.by actor is process:okf-ingest, not process:llm-ingestion-okf. Commons decided the fixed id on this repo's own proposal 2026-07-31, superseding option (d) chosen here 2026-07-27. The exclusion is ingest-spec.md:7-8, frozen on the spec being framework-neutral: normalising our repo name into the id would force every other conformant implementation to write it into its own output.

  Nothing in the wild carried the excluded value — OKF_V0_2 did not exist at v0.4.0, and no pilot had been notified — so this costs a tag rather than a migration. It is recorded rather than quietly folded in because the failure it avoids is specific: actor is both the stamp written and the value owned back (OwnershipPolicy), and recognition is one-way, so a pilot holding bundles stamped with the excluded id would have hit collision_unstamped on its own files the moment the id was corrected.

0.5.0a1 — 2026-07-31 — ABANDONED, do not pin

Pre-release. PROVISIONAL surface. Shipped to a named pilot set — portfolio-optimiser-claude, the plugin marketplace catalog, and claude-code-llm-wiki — so that real data can find what fixtures cannot. The v0.2 surface may change on their feedback without a deprecation cycle. Saying so up front is what buys the freedom to act on the feedback; discovering it later is what would make a pilot a de-facto release. Do not pin this tag outside the pilot set.

Support for a new upstream version is additive — a new profile, never a migration. OKF_LATEST still points at DEFAULT; flipping it is the GA event, not a side effect of this tag.

Added

• OKF_V0_2 profile. Google OKF v0.2 as a profile alongside DEFAULT and STRICT_V1. It closes nothing: §14 forbids a conformant consumer to reject on an unknown type value or on unknown additional keys, so type is the only required key. It NAMES okf_version but never carries its value — that value tracks the upstream Google version and belongs to catalog (decision E1), so the caller supplies it via materialize_bundle(..., root_frontmatter_values=…) and the profile fixes only the key and its position.
• root_frontmatter_values on materialize_bundle, keyword-only: the mechanism behind "a profile names a key, a caller owns its value".
• sources emitted as an inline flow sequence, populated from the manifest.
• The v0.2 golden fixture, compared byte-for-byte like the others.

Changed

• profile on materialize_bundle is keyword-only, so existing positional call sites stay source-compatible.
• _is_ingest_owned is profile-aware. Ownership is a policy on the profile, not a literal; recognition stays one-way.

Fixed

• root_frontmatter permitted a key without requiring it — the two had been the same profile field. Upstream says MAY where the field said MUST, so a root index that legally omitted an optional key was rejected. root_frontmatter now permits and orders; the new root_frontmatter_required requires. STRICT_V1 keeps requiring its three (unchanged for its consumer), OKF_V0_2 requires none, DEFAULT is untouched. Measured against the eight root indexes available locally: 7 of 8 failed before, 0 of 8 after.

Notes

• DEFAULT and STRICT_V1 are byte-stable across this release. The emit path is byte-identical; the golden suite is the check.
• Known, deliberately unfixed: TypePolicy's closed branch does not strip quotes from a declared type. All three call sites pass DEFAULT.types, and both DEFAULT and OKF_V0_2 set allowed=None, so the branch is unreachable in shipped code — it fires only for a caller constructing STRICT_V1 and calling rejection directly. Fixing it would repair a write path that never sees quotes, and would fix the meaning of a quote character without a value model that can express one, in a line-oriented format.

0.4.0 — 2026-07-25

Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship, and with them this library's first — and only ever — runtime dependency.

Added

• Door B — bundle inbox (process_inbox). Per dropped file: bytes → extraction → persist gate → render → collision gate → write → index link. Returns an InboxResult whose four buckets (persisted, quarantined, rejected, failed) are disjoint and complete, so a file that vanished mid-run surfaces as a missing entry rather than as nothing. One bad file never aborts the run: only an invalid ingested_at, a reserved okf_type, and a missing inbox directory fail the whole run, each being wrong for every file at once. Concept files are named inbox-{slug}.md, disjoint from index.md, Door A's ingest-*, and promoted-verdict-*; source_sha256 is taken over the original dropped bytes, so provenance stays re-verifiable against the operator's file.
• Door C — external bundle import (import_bundle). Reads an external OKF bundle and hands it whole to an injected gate over the guard's okf.import_bundle — a bundle-level call, because it resolves the cross-link graph across concepts — merging only concepts that clear the non-blocking floor. Two invariants, both load-bearing: a merged concept is written verbatim (stamping it would require round-tripping frontmatter through this library's line-oriented parser, which cannot represent the block lists the guard's parser accepts, and would persist bytes the guard never screened), and ownership is therefore proven by content identity — an occupied target name is re-used only when the bytes there are already identical, never overwritten. Re-importing an unchanged bundle is a no-op.
• extract_text — the Door B extraction registry. All file-type → text extraction lives in this library, because the guard is text-only. md/txt pass through, csv renders the markdown table, json is fenced verbatim, and html/htm reduce to text with html.parser — stdlib throughout. pdf, docx, and xlsx are gated behind the [extract] extra, which ships no parser yet: those types fail fast with a typed error naming the extra, never a silent skip.
• llm_ingestion_okf.guard_adapter — the shipped gate over the real guard (inbox_gate, import_gate), and the only module here that imports it. Door B screens the exact bytes it persists: the guard's prepare_input bookend prepares text for a model call this library never makes, so screen_output alone is used and the screened string is the written string. It follows that the gate refuses rather than repairs — a file carrying an invisible carrier is rejected, not stripped and persisted. The policy is PRESET_USER_UPLOAD, so any finding at all is held back.
• 15 new stable error codes, each registered in the errors.py docstrings (the stability contract) and pinned by the error-code conformance suite: extractor_decode_error, extractor_empty_csv, extractor_extra_missing, extractor_unknown, import_label_invalid, import_path_empty, import_path_too_long, import_provenance_invalid, import_slug_collision, inbox_slug_collision, inbox_slug_empty, inbox_slug_too_long, inbox_source_file_invalid, inbox_title_invalid, okf_type_reserved.

Changed

• llm-ingestion-guard>=0.2,<0.3 is now a mandatory runtime dependency. Installing this package installs the guard. Importing it does not: only guard_adapter imports the guard, so a Door A consumer keeps working whatever state the dependency is in, and the doors themselves still take an injected gate. Until the guard is published to a package index, install it from its tag — see the README. A packaging test enforces that this stays the only runtime dependency.
• An extraction title containing [ or ] is rejected at manifest load (ingest-spec §4, ratified D1). Previously only single-line was validated. A manifest that loaded before and carries a bracket in a title now fails fast with code manifest_schema: the title renders verbatim into the index link label - [title](target), where a bracket breaks index-link and navigation parsing downstream.

Fixed

• Materializing one manifest no longer deletes the files another manifest stamped into the same bundle. The §3 ownership scan classified every ingest-stamped file as replaceable, so a second manifest sharing a bundle removed the first one's concept files and their index links. Ownership is now narrowed to files whose stamp names the running manifest by stem — the stem is stable across content edits, so an edited manifest still reclaims what a prior run of itself wrote. The operator-copy restriction (a generated file copied into curated content) remains documented, not enforced.

Notes

• Phase 2's binary extraction is not in this release: the [extract] extra is declared but empty, and pdf/docx/xlsx therefore fail fast. That is the one outstanding item from the phase, and it ships separately.
• The ownership change above is an extension point, not a spec fix. Filed under "Fixed" because it stops silent data loss, but the spec owner (portfolio-optimiser-commons) has since pointed out that removing every ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes one manifest per bundle and defers multiple manifests feeding one bundle as a named extension point. So this release implements that extension point and outruns the frozen text rather than conforming to it. The mechanic itself is not at risk — matching by manifest stem follows from the spec's own {stem}@{h} stamp definition, since {h} changes on every content edit — but the surrounding prose is queued for amendment in commons and is not ratified. Treat multi-manifest bundles as ahead of the spec until it is.
• Exception __cause__ preservation is now pinned by a conformance suite, one test per fail-fast wrap site, alongside the existing .code suite.

0.3.2 — 2026-07-23

Fixed

• Frontmatter and index-label values are emitted verbatim; only source_query is whitespace-collapsed. Earlier releases collapsed every whitespace run in every frontmatter value and index link label to a single space. §5 of ingest-spec.md mandates that collapse for source_query alone — where a multi-line SQL SELECT must render on one line — while every other value is validated single-line at manifest load and passed through unchanged: validation, not repair. A title carrying an internal whitespace run now survives byte-for-byte at both the title frontmatter and the index link label, instead of being silently altered. Output bytes change only for values that contained a collapsible whitespace run; the shipped golden fixtures and both consumers are unaffected.

0.3.1 — 2026-07-19

Fixed

• Documentation corrected a security claim that did not hold. The module docstring and README stated that this library "calls the guard at every persist gate". That described the intended end state in the present tense. Door A — the only door shipped — is ungated: the package has zero runtime dependencies and calls no guard function before writing to disk. Both places now say so plainly, and state that gating external or untrusted content is the caller's responsibility (okf.import_bundle, or prepare_input/screen_output) until the persist gates land with Doors B and C.

  No behavior changed in this release. The correction is published because a consumer read the earlier wording as safe-by-default and would have persisted ungated content on that basis.

0.3.0 — 2026-07-17

Added

• Stable machine-readable error codes. IngestError gained a keyword-only code attribute (default "unspecified"). Roughly 24 codes are documented in the errors.py docstrings, and those docstrings are the stability contract. Consumers should assert on exc.value.code, not on message text.

Changed

• Exception message text is explicitly declared unstable. It may change in any release. Tests matching on message strings (pytest.raises(match=...)) should migrate to code comparisons.

Notes

• Generic schema shape violations deliberately share the single code manifest_schema. One code per validation rule would have frozen an unnecessarily large surface. Finer resolution is a separate decision, not an assumed requirement.

0.2.0 — 2026-07-17

Added

• PEP 561 support. The py.typed marker ships with the package, so consumers get the inline type hints without a mypy override.
• IngestResult.stamp exposes the spec §5 provenance stamp on the result object.

0.1.0 — 2026-07-16

Phase 1 (Door A) implemented against the normative ingest-spec.md owned by portfolio-optimiser-commons. Never tagged; consumers pinned the commit dae0bd1a directly.

Added

• Fail-fast manifest validation (spec §3–§4).
• Spec §5 body renderers as pure functions.
• The file connector (CSV, fail-closed path boundary), the sql connector (read-only sqlite, env-resolved credentials), and the http connector behind an explicit per-run network opt-in.
• Spec §5 materialization with an in-memory staging collision gate.
• Index maintenance on re-materialization (spec §6).
• The spec §11 golden fixtures, compared byte-for-byte.
• The Door A public surface: materialize_bundle plus the typed error hierarchy rooted in IngestError.