llm-ingestion-okf/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• The corpus harness (tools/okf_corpus_run.py) can replay segmentation plans, and it writes the bundle's log.md. --plans-dir names the proposals to replay and the profile follows from it; --bundle-id and --okf-version are arguments, never constants, because a profile names a key and the caller owns its value. log.md is written in SPEC section 9 form and dated from ingested_at, so merged + sum(coded rejections) == N is checkable from the bundle alone rather than only from a report that does not travel with it. Without --plans-dir a run is unchanged.

• The corpus harness links that log.md from the bundle's root index.md. Measured on the K2 artifact by a consumer: the log was on disk and no index named it, so a reader entering the bundle at index.md -- the walk section 8 exists to support -- never reached the one file carrying N. A LOCAL choice rather than conformance, and stated as one: upstream's own bundles do not link their log (measured at 9a15b13, 0 of 24 shipped index.md files name the single log.md in the set), which shows the link is not required, not that it is disallowed. It lives in the harness because the library cannot make it -- the log's content is the run's outcome, so an index that enumerated it off the directory would gain the link only from the second run and break rebuild-equals-incremental. Written only when absent, because the two reprojections differ: the per-directory one drops the line as a managed entry, the flat one keeps it (its target is not an owned concept), and appending unconditionally doubled it on the second unsegmented run.

  Measured cause: the harness passed STRUCTURED_V1 and no plans, so a 43-document corpus arrived as 39 flat concepts with no adjudication key anywhere. Rebuilt, the same corpus yields 629 concepts, 618 of them adjudication: proposed. Record: docs/2026-09-03-k2-bundle-rebuild.md.

• tools/okf_propose_segments.py takes --path-prefix. Section numbering is document-local, so across 39 documents 618 proposed entries claimed only 601 distinct paths -- 17 collisions that Door B's gate refuses per document. Scoping each document's entries under a caller-supplied prefix removes all 17. Without the flag every artifact already produced is byte-identical.

Changed

• The pinned llm-ingestion-guard moves to v1.3.0 ([tool.uv.sources] and uv.lock; the >=1.2,<2.0 range in [project.dependencies] already covered it and is unchanged). What this fixes is that the guard could not read back what this library WRITES: at 1.2.0 the flow-form sources in the OKF v0.2 golden was refused outright, and flow is the only form this library is able to emit, because its own line-oriented parser cannot round-trip the block form. Measured before the bump so the test discriminates rather than merely passes, and pinned by tests/test_guard_adapter.py::test_the_guard_parses_the_flow_form_sources_our_goldens_emit. Two rows of the gate table in docs/okf-nokkelinventar.md moved, not one: the BLOCK form of sources now passes too, which retires G30 -- though it changes nothing about what we emit, since our own parser is still the binding constraint. resource is allowlisted only inside a sources entry, so section 10.2's executor/attester resource stays rejected through every carrier and the Door C boundary is unmoved. No public API of this library changes; a consumer's cost is a re-run.

• uv.lock also picks up pypandoc-binary==1.17, which is a stale lockfile being corrected rather than a new dependency: the package was already declared in the [extract] extra, and uv lock --check reports the lockfile as out of date on the untouched tree. Core still has exactly one runtime dependency.

• tools/okf_propose_segments.py writes no artifact when it has nothing to propose, exiting 1 (distinct from 2, "could not do the job") instead of 0 with an empty plan. An empty plan cannot be replayed -- process_inbox refuses one, because a plan naming no entry would persist nothing for a document that was dropped -- so the file's only possible use was to fail a run later, and it did. Measured: 11 of 39 documents in the K2 corpus propose zero segments.

• A concept can now record more than one source. sources renders a flow sequence of N flow mappings on one line, so a v0.2 profile can express multi-source provenance instead of the single entry that was the measured ceiling on SPEC 5.1 coverage. A single source is byte-identical to before, so every golden is unmoved and no existing bundle changes.

  The form is flow, not the block list PM decision B6 prescribed, and the reason is measured: this library's frontmatter parser is line-oriented and skips indented lines, so a block list round-trips to an EMPTY value with every entry silently gone -- and the consumer the decision was written for accepts the multi-entry flow sequence while classifying a block sequence as unreadable provenance. Emitting block would have produced records neither side can read. A negative-control test pins the block form's data loss so the reason stays measurable rather than remembered.

  New error code sources_empty: an empty list is refused, because sources: [] reads as a measured absence when it is the absence of a measurement.

• The consumption contract, stated normatively in docs/consumption-contract.md, with a copyable skill template (skills/okf-consume-template/) and a checker (tools/okf_contract_check.py) that reads its mechanically checkable half: payload shape, source marking per excerpt, the closed adjudication and trust_tier state sets, denominator identity, and the budget gate with its validated known-positive. The checker ships outside src/, so no consumer's install surface changes.

• Five office formats through a vendored converter, behind the same optional [extract] extra: docx, xlsx, pptx, odt, rtf. The converter binary travels inside the wheel and is resolved by path rather than found on PATH, with its version asserted against a pin -- pypandoc searches PATH first and takes the highest version it finds, so a vendored binary buys nothing until something resolves it explicitly. A host carrying a different converter is refused rather than silently used.

  Two of the five rows are measured; three are not. The corpus this work was measured on contains zero pptx, odt and rtf files, so those rows work by construction and have never met a document anyone wrote. The distinction is asserted in the suite, not left in a comment.

  .doc (Word 97) stays out -- the converter does not read it. Drawn content does not survive extraction in any format, and every conversion warns about it.

• Four error codes for the converter path: extractor_binary_missing, extractor_binary_version, extractor_convert_error and extractor_empty_conversion.

• tests/test_docs_promises.py, which asserts the README's published format list against the registries it describes. This exists because the README's previous promise -- that docx and xlsx always fail fast -- went false silently when the converter landed. A published guarantee is a test obligation.

• Door B extracts pdf behind the optional [extract] extra. The extra is populated for the first time, with one parser: pdfplumber>=0.11.10,<0.12 (MIT). The default install is unchanged — still exactly one runtime dependency, still stdlib otherwise — and a packaging test enforces that.

  The parser choice was forced by a measurement, not by preference (docs/2026-08-21-g2-pdf-extraction-measurement.md). On a real Vegnormalene requirement table, pdfplumber keeps 4 of 4 rows with label and value on the same line; pypdf, pdfminer.six and pymupdf each keep 0 of 4, emitting all labels and then all values. A downstream reader can only re-pair those by guessing, and in a requirements document a wrong pairing looks right. pymupdf was additionally excluded on licence (AGPL-3.0 or commercial): this package is MIT and an optional extra must not hand a consumer copyleft they did not choose.

  docx and xlsx were unchanged at this release, shipping no parser and failing fast with extractor_extra_missing. That is no longer true: a later release added a vendored converter reaching five office formats. This entry is left as written — a changelog records what a release did — and the correction is stated here so a reader arriving at this line is not misled.

• ExtractionWarning, exported from the package. Every pdf extraction emits one. Text extraction recovers text; anything a PDF draws — figures, diagrams, images — has no text to recover, so only captions survive and a bundle built from drawn documents is incomplete by construction. That is categorically true rather than document-specific, so it is stated rather than detected: deciding "is there a figure on this page" is a layout heuristic this library does not own. A named class so it can be filtered deliberately.

• Two error codes, both mirroring existing patterns rather than inventing behaviour: extractor_empty_pdf (a PDF yielded no text on any page — a scanned or image-only document; refused rather than persisted as an empty concept, which would be the silent skip this registry exists to prevent) and extractor_pdf_error (the parser failed on the bytes; the third-party exception is wrapped, never leaked).

Changed

• The [extract] gate for pdf is now an import probe rather than a membership test. The rejection did not change: without the extra installed, pdf still raises ExtractionError with code extractor_extra_missing and the same message naming the remedy. A consumer that has not installed the extra sees no difference at all. That behaviour is asserted unconditionally, including on machines where the parser is installed, so it cannot rot into a skipped test.

  Known limitation, stated rather than worked around: structured table recovery is out of scope. pdfplumber.extract_tables() and PyMuPDF.find_tables() — two independent implementations — return the same wrong shape for the measured requirement table, and across the whole handbook only 45 of 196 detected table objects are clean enough to hand to render_table unchanged. The breakage is in the documents' ruling geometry, not in either library. PDFs therefore enter this library as prose, with table lines correctly paired.

  Extracted PDF text is pinned to an exact parser version. pdfplumber pins pdfminer.six==20260107 exactly, and pdfminer.six ships date-stamped releases with no stability contract. Extraction is deterministic within a parser version (measured across five configurations) and not guaranteed across one, so any golden fixture built on extracted PDF text is a fixture migration away from a parser upgrade. tests/test_extract.py freezes the expected text of a committed fixture so that upgrade breaks something visible instead of drifting silently; see tests/fixtures/README.md.

• DEFAULT now stamps generated: { by: process:okf-ingest, at: <ingested_at> } instead of generated: true. This is a byte change in every concept file Door A writes under DEFAULT, so a consumer's own golden fixtures will show one changed line per generated file.

  The trigger is not an upstream Google release. DEFAULT states the ingest-spec layer owned by portfolio-optimiser-commons, and they ratified this shape (2026-08-02) and executed it in their spec on 2026-08-09. §7 defines the value: the actor is the fixed process:okf-ingest, at repeats ingested_at verbatim, and it is unquoted because frontmatter is parsed line-oriented — a quote would be a character in the value rather than syntax a reader strips.

  What it costs a consumer is a re-run, and nothing else. Ownership recognition is one-way: a profile carrying an actor still owns the older literal stamp, so a bundle written by an earlier version re-runs in place rather than tripping the §3 collision gate. No call signature changed and no key was added or removed.

  Two things it is NOT. It is not a migration of DEFAULT onto OKF v0.2 — the profile remains v0.1 on every axis upstream owns, and still emits no sources. And it does not make DEFAULT and OKF_V0_2 the same profile; they now agree on the stamp and continue to differ in index root frontmatter, type-conditional requirements, and sources derivation.

  Door B (process_inbox) is deliberately unchanged: its marker is generated plus source_file, it is not governed by the ingest-spec, and it stays disjoint from Door A's ingest_manifest.

• The v0.5.0 pilot set gained a fourth member: portfolio-optimiser. Admitted 2026-08-09 on their request. v0.5.0a1's entry below says "do not pin this tag outside the pilot set" and names three repos; that entry is left as written, because it records what was true when the tag shipped. This is the amendment, and the sentence still binds — the set is now those three plus portfolio-optimiser.

  The reason is the producer axis, not the count. portfolio-optimiser-claude is parked, and with it parked no original member could emit a v0.2 bundle at all: claude-code-llm-wiki is read-only in the pilot and catalog is gate-side. The new member consumes the same Door A. Nothing about the provisional surface changes: OKF_LATEST still points at DEFAULT, and the v0.2 surface may still move on pilot feedback without a deprecation cycle.

  If you pin v0.5.0a2, its guard pin is >=0.2,<0.3 — that tag resolves against guard v0.2.0, not the v0.3.4 this repo's main now uses. main moved to >=0.3,<0.4 after the tag. Also note tool.uv.sources is not inherited transitively: a consumer supplies the guard's source itself.

• The guard pin moved to >=0.3,<0.4, resolved against v0.3.4. The window is widened only after measurement, never before: the 19-fixture guard-surface suite was run against v0.3.4 in a scratch venv first, and reproduced exactly the three deltas measured against v0.3.3 — no new ones. v0.3.4's own fixes are regex-complexity repairs, one of them (okf._MD_LINK_RE) on Door C's call path, with no disposition changes.

• Door C now passes allow_reserved=False to okf.import_bundle. The guard added the keyword in the 0.3.x line and defaults it True for the received-bundle path, which would merge a sender's index.md / log.md instead of rejecting them. Door C overrides it, keeping the unconditional reserved-name refusal committed to before the keyword existed. The reason is structural rather than a second opinion on the guard's scan: Door C generates the merged bundle's index.md from what it merged and writes every merged concept verbatim, so a sender's index.md would be a second, irreconcilable claim on one path.

  This is not a behaviour change for anyone on the previous pin. Under v0.2.0 the keyword did not exist and reserved names were refused by construction; the explicit argument preserves that outcome across the bump. A consumer sees the same rejections, with the same reasons, before and after.

0.5.0a2 — 2026-07-31

This is the pre-release the pilots pin. v0.5.0a1 was tagged and abandoned unused — do not pin it. It carried a generated.by actor that the spec owner had already excluded, and it was caught before any pilot was notified.

Fixed

• OKF_V0_2's generated.by actor is process:okf-ingest, not process:llm-ingestion-okf. Commons decided the fixed id on this repo's own proposal 2026-07-31, superseding option (d) chosen here 2026-07-27. The exclusion is ingest-spec.md:7-8, frozen on the spec being framework-neutral: normalising our repo name into the id would force every other conformant implementation to write it into its own output.

  Nothing in the wild carried the excluded value — OKF_V0_2 did not exist at v0.4.0, and no pilot had been notified — so this costs a tag rather than a migration. It is recorded rather than quietly folded in because the failure it avoids is specific: actor is both the stamp written and the value owned back (OwnershipPolicy), and recognition is one-way, so a pilot holding bundles stamped with the excluded id would have hit collision_unstamped on its own files the moment the id was corrected.

0.5.0a1 — 2026-07-31 — ABANDONED, do not pin

Pre-release. PROVISIONAL surface. Shipped to a named pilot set — portfolio-optimiser-claude, the plugin marketplace catalog, and claude-code-llm-wiki — so that real data can find what fixtures cannot. The v0.2 surface may change on their feedback without a deprecation cycle. Saying so up front is what buys the freedom to act on the feedback; discovering it later is what would make a pilot a de-facto release. Do not pin this tag outside the pilot set.

Support for a new upstream version is additive — a new profile, never a migration. OKF_LATEST still points at DEFAULT; flipping it is the GA event, not a side effect of this tag.

Added

• OKF_V0_2 profile. Google OKF v0.2 as a profile alongside DEFAULT and STRICT_V1. It closes nothing: §14 forbids a conformant consumer to reject on an unknown type value or on unknown additional keys, so type is the only required key. It NAMES okf_version but never carries its value — that value tracks the upstream Google version and belongs to catalog (decision E1), so the caller supplies it via materialize_bundle(..., root_frontmatter_values=…) and the profile fixes only the key and its position.
• root_frontmatter_values on materialize_bundle, keyword-only: the mechanism behind "a profile names a key, a caller owns its value".
• sources emitted as an inline flow sequence, populated from the manifest.
• The v0.2 golden fixture, compared byte-for-byte like the others.

Changed

• profile on materialize_bundle is keyword-only, so existing positional call sites stay source-compatible.
• _is_ingest_owned is profile-aware. Ownership is a policy on the profile, not a literal; recognition stays one-way.

Fixed

• root_frontmatter permitted a key without requiring it — the two had been the same profile field. Upstream says MAY where the field said MUST, so a root index that legally omitted an optional key was rejected. root_frontmatter now permits and orders; the new root_frontmatter_required requires. STRICT_V1 keeps requiring its three (unchanged for its consumer), OKF_V0_2 requires none, DEFAULT is untouched. Measured against the eight root indexes available locally: 7 of 8 failed before, 0 of 8 after.

Notes

• DEFAULT and STRICT_V1 are byte-stable across this release. The emit path is byte-identical; the golden suite is the check.
• Known, deliberately unfixed: TypePolicy's closed branch does not strip quotes from a declared type. All three call sites pass DEFAULT.types, and both DEFAULT and OKF_V0_2 set allowed=None, so the branch is unreachable in shipped code — it fires only for a caller constructing STRICT_V1 and calling rejection directly. Fixing it would repair a write path that never sees quotes, and would fix the meaning of a quote character without a value model that can express one, in a line-oriented format.

0.4.0 — 2026-07-25

Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship, and with them this library's first — and only ever — runtime dependency.

Added

• Door B — bundle inbox (process_inbox). Per dropped file: bytes → extraction → persist gate → render → collision gate → write → index link. Returns an InboxResult whose four buckets (persisted, quarantined, rejected, failed) are disjoint and complete, so a file that vanished mid-run surfaces as a missing entry rather than as nothing. One bad file never aborts the run: only an invalid ingested_at, a reserved okf_type, and a missing inbox directory fail the whole run, each being wrong for every file at once. Concept files are named inbox-{slug}.md, disjoint from index.md, Door A's ingest-*, and promoted-verdict-*; source_sha256 is taken over the original dropped bytes, so provenance stays re-verifiable against the operator's file.
• Door C — external bundle import (import_bundle). Reads an external OKF bundle and hands it whole to an injected gate over the guard's okf.import_bundle — a bundle-level call, because it resolves the cross-link graph across concepts — merging only concepts that clear the non-blocking floor. Two invariants, both load-bearing: a merged concept is written verbatim (stamping it would require round-tripping frontmatter through this library's line-oriented parser, which cannot represent the block lists the guard's parser accepts, and would persist bytes the guard never screened), and ownership is therefore proven by content identity — an occupied target name is re-used only when the bytes there are already identical, never overwritten. Re-importing an unchanged bundle is a no-op.
• extract_text — the Door B extraction registry. All file-type → text extraction lives in this library, because the guard is text-only. md/txt pass through, csv renders the markdown table, json is fenced verbatim, and html/htm reduce to text with html.parser — stdlib throughout. pdf, docx, and xlsx are gated behind the [extract] extra, which ships no parser yet: those types fail fast with a typed error naming the extra, never a silent skip.
• llm_ingestion_okf.guard_adapter — the shipped gate over the real guard (inbox_gate, import_gate), and the only module here that imports it. Door B screens the exact bytes it persists: the guard's prepare_input bookend prepares text for a model call this library never makes, so screen_output alone is used and the screened string is the written string. It follows that the gate refuses rather than repairs — a file carrying an invisible carrier is rejected, not stripped and persisted. The policy is PRESET_USER_UPLOAD, so any finding at all is held back.
• 15 new stable error codes, each registered in the errors.py docstrings (the stability contract) and pinned by the error-code conformance suite: extractor_decode_error, extractor_empty_csv, extractor_extra_missing, extractor_unknown, import_label_invalid, import_path_empty, import_path_too_long, import_provenance_invalid, import_slug_collision, inbox_slug_collision, inbox_slug_empty, inbox_slug_too_long, inbox_source_file_invalid, inbox_title_invalid, okf_type_reserved.

Changed

• llm-ingestion-guard>=0.2,<0.3 is now a mandatory runtime dependency. Installing this package installs the guard. Importing it does not: only guard_adapter imports the guard, so a Door A consumer keeps working whatever state the dependency is in, and the doors themselves still take an injected gate. Until the guard is published to a package index, install it from its tag — see the README. A packaging test enforces that this stays the only runtime dependency.
• An extraction title containing [ or ] is rejected at manifest load (ingest-spec §4, ratified D1). Previously only single-line was validated. A manifest that loaded before and carries a bracket in a title now fails fast with code manifest_schema: the title renders verbatim into the index link label - [title](target), where a bracket breaks index-link and navigation parsing downstream.

Fixed

• Materializing one manifest no longer deletes the files another manifest stamped into the same bundle. The §3 ownership scan classified every ingest-stamped file as replaceable, so a second manifest sharing a bundle removed the first one's concept files and their index links. Ownership is now narrowed to files whose stamp names the running manifest by stem — the stem is stable across content edits, so an edited manifest still reclaims what a prior run of itself wrote. The operator-copy restriction (a generated file copied into curated content) remains documented, not enforced.

Notes

• Phase 2's binary extraction is not in this release: the [extract] extra is declared but empty, and pdf/docx/xlsx therefore fail fast. That is the one outstanding item from the phase, and it ships separately.
• The ownership change above is an extension point, not a spec fix. Filed under "Fixed" because it stops silent data loss, but the spec owner (portfolio-optimiser-commons) has since pointed out that removing every ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes one manifest per bundle and defers multiple manifests feeding one bundle as a named extension point. So this release implements that extension point and outruns the frozen text rather than conforming to it. The mechanic itself is not at risk — matching by manifest stem follows from the spec's own {stem}@{h} stamp definition, since {h} changes on every content edit — but the surrounding prose is queued for amendment in commons and is not ratified. Treat multi-manifest bundles as ahead of the spec until it is.
• Exception __cause__ preservation is now pinned by a conformance suite, one test per fail-fast wrap site, alongside the existing .code suite.

0.3.2 — 2026-07-23

Fixed

• Frontmatter and index-label values are emitted verbatim; only source_query is whitespace-collapsed. Earlier releases collapsed every whitespace run in every frontmatter value and index link label to a single space. §5 of ingest-spec.md mandates that collapse for source_query alone — where a multi-line SQL SELECT must render on one line — while every other value is validated single-line at manifest load and passed through unchanged: validation, not repair. A title carrying an internal whitespace run now survives byte-for-byte at both the title frontmatter and the index link label, instead of being silently altered. Output bytes change only for values that contained a collapsible whitespace run; the shipped golden fixtures and both consumers are unaffected.

0.3.1 — 2026-07-19

Fixed

• Documentation corrected a security claim that did not hold. The module docstring and README stated that this library "calls the guard at every persist gate". That described the intended end state in the present tense. Door A — the only door shipped — is ungated: the package has zero runtime dependencies and calls no guard function before writing to disk. Both places now say so plainly, and state that gating external or untrusted content is the caller's responsibility (okf.import_bundle, or prepare_input/screen_output) until the persist gates land with Doors B and C.

  No behavior changed in this release. The correction is published because a consumer read the earlier wording as safe-by-default and would have persisted ungated content on that basis.

0.3.0 — 2026-07-17

Added

• Stable machine-readable error codes. IngestError gained a keyword-only code attribute (default "unspecified"). Roughly 24 codes are documented in the errors.py docstrings, and those docstrings are the stability contract. Consumers should assert on exc.value.code, not on message text.

Changed

• Exception message text is explicitly declared unstable. It may change in any release. Tests matching on message strings (pytest.raises(match=...)) should migrate to code comparisons.

Notes

• Generic schema shape violations deliberately share the single code manifest_schema. One code per validation rule would have frozen an unnecessarily large surface. Finer resolution is a separate decision, not an assumed requirement.

0.2.0 — 2026-07-17

Added

• PEP 561 support. The py.typed marker ships with the package, so consumers get the inline type hints without a mypy override.
• IngestResult.stamp exposes the spec §5 provenance stamp on the result object.

0.1.0 — 2026-07-16

Phase 1 (Door A) implemented against the normative ingest-spec.md owned by portfolio-optimiser-commons. Never tagged; consumers pinned the commit dae0bd1a directly.

Added

• Fail-fast manifest validation (spec §3–§4).
• Spec §5 body renderers as pure functions.
• The file connector (CSV, fail-closed path boundary), the sql connector (read-only sqlite, env-resolved credentials), and the http connector behind an explicit per-run network opt-in.
• Spec §5 materialization with an in-memory staging collision gate.
• Index maintenance on re-materialization (spec §6).
• The spec §11 golden fixtures, compared byte-for-byte.
• The Door A public surface: materialize_bundle plus the typed error hierarchy rooted in IngestError.