llm-ingestion-okf/CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Changed

- **`DEFAULT` now stamps `generated: { by: process:okf-ingest, at: <ingested_at> }`
  instead of `generated: true`.** This is a byte change in every concept file Door A
  writes under `DEFAULT`, so a consumer's own golden fixtures will show one changed
  line per generated file.

  The trigger is not an upstream Google release. `DEFAULT` states the ingest-spec
  layer owned by `portfolio-optimiser-commons`, and they ratified this shape
  (2026-08-02) and executed it in their spec on 2026-08-09. §7 defines the value:
  the actor is the fixed `process:okf-ingest`, `at` repeats `ingested_at` verbatim,
  and it is unquoted because frontmatter is parsed line-oriented — a quote would be
  a character in the value rather than syntax a reader strips.

  **What it costs a consumer is a re-run, and nothing else.** Ownership recognition
  is one-way: a profile carrying an actor still owns the older literal stamp, so a
  bundle written by an earlier version re-runs in place rather than tripping the §3
  collision gate. No call signature changed and no key was added or removed.

  Two things it is NOT. It is not a migration of `DEFAULT` onto OKF v0.2 — the
  profile remains v0.1 on every axis upstream owns, and still emits no `sources`.
  And it does not make `DEFAULT` and `OKF_V0_2` the same profile; they now agree on
  the stamp and continue to differ in index root frontmatter, type-conditional
  requirements, and `sources` derivation.

  Door B (`process_inbox`) is deliberately unchanged: its marker is `generated`
  plus `source_file`, it is not governed by the ingest-spec, and it stays disjoint
  from Door A's `ingest_manifest`.

- **The v0.5.0 pilot set gained a fourth member: `portfolio-optimiser`.** Admitted
  2026-08-09 on their request. `v0.5.0a1`'s entry below says "do not pin this tag
  outside the pilot set" and names three repos; that entry is left as written,
  because it records what was true when the tag shipped. This is the amendment,
  and the sentence still binds — the set is now those three plus
  `portfolio-optimiser`.

  The reason is the producer axis, not the count. `portfolio-optimiser-claude` is
  parked, and with it parked no original member could *emit* a v0.2 bundle at all:
  `claude-code-llm-wiki` is read-only in the pilot and `catalog` is gate-side. The
  new member consumes the same Door A. Nothing about the provisional surface
  changes: `OKF_LATEST` still points at `DEFAULT`, and the v0.2 surface may still
  move on pilot feedback without a deprecation cycle.

  **If you pin `v0.5.0a2`, its guard pin is `>=0.2,<0.3`** — that tag resolves
  against guard `v0.2.0`, *not* the `v0.3.4` this repo's `main` now uses. `main`
  moved to `>=0.3,<0.4` after the tag. Also note `tool.uv.sources` is not
  inherited transitively: a consumer supplies the guard's source itself.

- **The guard pin moved to `>=0.3,<0.4`, resolved against `v0.3.4`.** The
  window is widened only after measurement, never before: the 19-fixture
  guard-surface suite was run against `v0.3.4` in a scratch venv first, and
  reproduced exactly the three deltas measured against `v0.3.3` — no new ones.
  `v0.3.4`'s own fixes are regex-complexity repairs, one of them
  (`okf._MD_LINK_RE`) on Door C's call path, with no disposition changes.

- **Door C now passes `allow_reserved=False` to `okf.import_bundle`.** The
  guard added the keyword in the `0.3.x` line and defaults it `True` for the
  received-bundle path, which would merge a sender's `index.md` / `log.md`
  instead of rejecting them. Door C overrides it, keeping the unconditional
  reserved-name refusal committed to before the keyword existed. The reason is
  structural rather than a second opinion on the guard's scan: Door C generates
  the merged bundle's `index.md` from what it merged and writes every merged
  concept verbatim, so a sender's `index.md` would be a second, irreconcilable
  claim on one path.

  **This is not a behaviour change for anyone on the previous pin.** Under
  `v0.2.0` the keyword did not exist and reserved names were refused by
  construction; the explicit argument preserves that outcome across the bump.
  A consumer sees the same rejections, with the same reasons, before and after.

## [0.5.0a2] — 2026-07-31

**This is the pre-release the pilots pin. `v0.5.0a1` was tagged and abandoned
unused — do not pin it.** It carried a `generated.by` actor that the spec owner
had already excluded, and it was caught before any pilot was notified.

### Fixed

- **`OKF_V0_2`'s `generated.by` actor is `process:okf-ingest`**, not
  `process:llm-ingestion-okf`. Commons decided the fixed id on this repo's own
  proposal 2026-07-31, superseding option (d) chosen here 2026-07-27. The
  exclusion is `ingest-spec.md:7-8`, frozen on the spec being framework-neutral:
  normalising *our* repo name into the id would force every other conformant
  implementation to write it into its own output.

  Nothing in the wild carried the excluded value — `OKF_V0_2` did not exist at
  `v0.4.0`, and no pilot had been notified — so this costs a tag rather than a
  migration. It is recorded rather than quietly folded in because the failure it
  avoids is specific: `actor` is both the stamp written and the value owned back
  (`OwnershipPolicy`), and recognition is one-way, so a pilot holding bundles
  stamped with the excluded id would have hit `collision_unstamped` on its own
  files the moment the id was corrected.

## [0.5.0a1] — 2026-07-31 — ABANDONED, do not pin

**Pre-release. PROVISIONAL surface.** Shipped to a named pilot set —
`portfolio-optimiser-claude`, the plugin marketplace catalog, and
`claude-code-llm-wiki` — so that real data can find what fixtures cannot. The
v0.2 surface may change on their feedback **without a deprecation cycle**.
Saying so up front is what buys the freedom to act on the feedback; discovering
it later is what would make a pilot a de-facto release. Do not pin this tag
outside the pilot set.

Support for a new upstream version is **additive — a new profile, never a
migration**. `OKF_LATEST` still points at `DEFAULT`; flipping it is the GA
event, not a side effect of this tag.

### Added

- **`OKF_V0_2` profile.** Google OKF v0.2 as a profile alongside `DEFAULT` and
  `STRICT_V1`. It closes nothing: §14 forbids a conformant consumer to reject on
  an unknown `type` value or on unknown additional keys, so `type` is the only
  required key. It NAMES `okf_version` but never carries its value — that value
  tracks the upstream Google version and belongs to catalog (decision E1), so
  the caller supplies it via `materialize_bundle(..., root_frontmatter_values=…)`
  and the profile fixes only the key and its position.
- **`root_frontmatter_values`** on `materialize_bundle`, keyword-only: the
  mechanism behind "a profile names a key, a caller owns its value".
- **`sources` emitted as an inline flow sequence**, populated from the manifest.
- **The v0.2 golden fixture**, compared byte-for-byte like the others.

### Changed

- **`profile` on `materialize_bundle` is keyword-only**, so existing positional
  call sites stay source-compatible.
- **`_is_ingest_owned` is profile-aware.** Ownership is a policy on the profile,
  not a literal; recognition stays one-way.

### Fixed

- **`root_frontmatter` permitted a key without requiring it — the two had been
  the same profile field.** Upstream says MAY where the field said MUST, so a
  root index that legally omitted an optional key was rejected. `root_frontmatter`
  now permits and orders; the new `root_frontmatter_required` requires.
  `STRICT_V1` keeps requiring its three (unchanged for its consumer), `OKF_V0_2`
  requires none, `DEFAULT` is untouched. Measured against the eight root indexes
  available locally: 7 of 8 failed before, 0 of 8 after.

### Notes

- **`DEFAULT` and `STRICT_V1` are byte-stable across this release.** The emit
  path is byte-identical; the golden suite is the check.
- **Known, deliberately unfixed:** `TypePolicy`'s closed branch does not strip
  quotes from a declared type. All three call sites pass `DEFAULT.types`, and
  both `DEFAULT` and `OKF_V0_2` set `allowed=None`, so the branch is
  unreachable in shipped code — it fires only for a caller constructing
  `STRICT_V1` and calling `rejection` directly. Fixing it would repair a write
  path that never sees quotes, and would fix the meaning of a quote character
  without a value model that can express one, in a line-oriented format.

## [0.4.0] — 2026-07-25

Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
and with them this library's first — and only ever — runtime dependency.

### Added

- **Door B — bundle inbox (`process_inbox`).** Per dropped file: bytes →
  extraction → persist gate → render → collision gate → write → index link.
  Returns an `InboxResult` whose four buckets (`persisted`, `quarantined`,
  `rejected`, `failed`) are disjoint and complete, so a file that vanished
  mid-run surfaces as a missing entry rather than as nothing. One bad file
  never aborts the run: only an invalid `ingested_at`, a reserved `okf_type`,
  and a missing inbox directory fail the whole run, each being wrong for every
  file at once. Concept files are named `inbox-{slug}.md`, disjoint from
  `index.md`, Door A's `ingest-*`, and `promoted-verdict-*`; `source_sha256`
  is taken over the original dropped bytes, so provenance stays re-verifiable
  against the operator's file.
- **Door C — external bundle import (`import_bundle`).** Reads an external OKF
  bundle and hands it *whole* to an injected gate over the guard's
  `okf.import_bundle` — a bundle-level call, because it resolves the
  cross-link graph across concepts — merging only concepts that clear the
  non-blocking floor. Two invariants, both load-bearing: a merged concept is
  written **verbatim** (stamping it would require round-tripping frontmatter
  through this library's line-oriented parser, which cannot represent the
  block lists the guard's parser accepts, and would persist bytes the guard
  never screened), and ownership is therefore proven by **content identity** —
  an occupied target name is re-used only when the bytes there are already
  identical, never overwritten. Re-importing an unchanged bundle is a no-op.
- **`extract_text` — the Door B extraction registry.** All file-type → text
  extraction lives in this library, because the guard is text-only. `md`/`txt`
  pass through, `csv` renders the markdown table, `json` is fenced verbatim,
  and `html`/`htm` reduce to text with `html.parser` — stdlib throughout.
  `pdf`, `docx`, and `xlsx` are gated behind the `[extract]` extra, which
  ships no parser yet: those types fail fast with a typed error naming the
  extra, never a silent skip.
- **`llm_ingestion_okf.guard_adapter`** — the shipped gate over the real
  guard (`inbox_gate`, `import_gate`), and the only module here that imports
  it. Door B screens the exact bytes it persists: the guard's `prepare_input`
  bookend prepares text for a model call this library never makes, so
  `screen_output` alone is used and the screened string is the written string.
  It follows that the gate **refuses rather than repairs** — a file carrying
  an invisible carrier is rejected, not stripped and persisted. The policy is
  `PRESET_USER_UPLOAD`, so any finding at all is held back.
- **15 new stable error codes**, each registered in the `errors.py` docstrings
  (the stability contract) and pinned by the error-code conformance suite:
  `extractor_decode_error`, `extractor_empty_csv`, `extractor_extra_missing`,
  `extractor_unknown`, `import_label_invalid`, `import_path_empty`,
  `import_path_too_long`, `import_provenance_invalid`,
  `import_slug_collision`, `inbox_slug_collision`, `inbox_slug_empty`,
  `inbox_slug_too_long`, `inbox_source_file_invalid`, `inbox_title_invalid`,
  `okf_type_reserved`.

### Changed

- **`llm-ingestion-guard>=0.2,<0.3` is now a mandatory runtime dependency.**
  Installing this package installs the guard. Importing it does not: only
  `guard_adapter` imports the guard, so a Door A consumer keeps working
  whatever state the dependency is in, and the doors themselves still take an
  *injected* gate. Until the guard is published to a package index, install it
  from its tag — see the README. A packaging test enforces that this stays the
  only runtime dependency.
- **An extraction `title` containing `[` or `]` is rejected at manifest load**
  (ingest-spec §4, ratified D1). Previously only single-line was validated. A
  manifest that loaded before and carries a bracket in a title now fails fast
  with code `manifest_schema`: the title renders verbatim into the index link
  label `- [title](target)`, where a bracket breaks index-link and navigation
  parsing downstream.

### Fixed

- **Materializing one manifest no longer deletes the files another manifest
  stamped into the same bundle.** The §3 ownership scan classified every
  ingest-stamped file as replaceable, so a second manifest sharing a bundle
  removed the first one's concept files and their index links. Ownership is
  now narrowed to files whose stamp names the running manifest by stem — the
  stem is stable across content edits, so an edited manifest still reclaims
  what a prior run of itself wrote. The operator-copy restriction (a generated
  file copied into curated content) remains documented, not enforced.

### Notes

- Phase 2's binary extraction is **not** in this release: the `[extract]`
  extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
  That is the one outstanding item from the phase, and it ships separately.
- **The ownership change above is an extension point, not a spec fix.** Filed
  under "Fixed" because it stops silent data loss, but the spec owner
  (`portfolio-optimiser-commons`) has since pointed out that removing *every*
  ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes
  one manifest per bundle and defers multiple manifests feeding one bundle as
  a named extension point. So this release implements that extension point and
  outruns the frozen text rather than conforming to it. The mechanic itself is
  not at risk — matching by manifest stem follows from the spec's own
  `{stem}@{h}` stamp definition, since `{h}` changes on every content edit —
  but the surrounding prose is queued for amendment in commons and is not
  ratified. Treat multi-manifest bundles as ahead of the spec until it is.
- Exception `__cause__` preservation is now pinned by a conformance suite, one
  test per fail-fast wrap site, alongside the existing `.code` suite.

## [0.3.2] — 2026-07-23

### Fixed

- **Frontmatter and index-label values are emitted verbatim; only
  `source_query` is whitespace-collapsed.** Earlier releases collapsed every
  whitespace run in every frontmatter value and index link label to a single
  space. §5 of `ingest-spec.md` mandates that collapse for `source_query`
  alone — where a multi-line SQL `SELECT` must render on one line — while
  every other value is validated single-line at manifest load and passed
  through unchanged: validation, not repair. A `title` carrying an internal
  whitespace run now survives byte-for-byte at both the `title` frontmatter
  and the index link label, instead of being silently altered. Output bytes
  change only for values that contained a collapsible whitespace run; the
  shipped golden fixtures and both consumers are unaffected.

## [0.3.1] — 2026-07-19

### Fixed

- **Documentation corrected a security claim that did not hold.** The module
  docstring and README stated that this library "calls the guard at every persist
  gate". That described the intended end state in the present tense. Door A — the
  only door shipped — is ungated: the package has zero runtime dependencies and
  calls no guard function before writing to disk. Both places now say so plainly,
  and state that gating external or untrusted content is the caller's
  responsibility (`okf.import_bundle`, or `prepare_input`/`screen_output`) until
  the persist gates land with Doors B and C.

  No behavior changed in this release. The correction is published because a
  consumer read the earlier wording as safe-by-default and would have persisted
  ungated content on that basis.

## [0.3.0] — 2026-07-17

### Added

- **Stable machine-readable error codes.** `IngestError` gained a keyword-only
  `code` attribute (default `"unspecified"`). Roughly 24 codes are documented in
  the `errors.py` docstrings, and those docstrings are the stability contract.
  Consumers should assert on `exc.value.code`, not on message text.

### Changed

- **Exception message text is explicitly declared unstable.** It may change in any
  release. Tests matching on message strings (`pytest.raises(match=...)`) should
  migrate to code comparisons.

### Notes

- Generic schema shape violations deliberately share the single code
  `manifest_schema`. One code per validation rule would have frozen an
  unnecessarily large surface. Finer resolution is a separate decision, not an
  assumed requirement.

## [0.2.0] — 2026-07-17

### Added

- **PEP 561 support.** The `py.typed` marker ships with the package, so consumers
  get the inline type hints without a mypy override.
- **`IngestResult.stamp`** exposes the spec §5 provenance stamp on the result
  object.

## [0.1.0] — 2026-07-16

Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
`portfolio-optimiser-commons`. Never tagged; consumers pinned the commit
`dae0bd1a` directly.

### Added

- Fail-fast manifest validation (spec §3–§4).
- Spec §5 body renderers as pure functions.
- The `file` connector (CSV, fail-closed path boundary), the `sql` connector
  (read-only sqlite, env-resolved credentials), and the `http` connector behind an
  explicit per-run network opt-in.
- Spec §5 materialization with an in-memory staging collision gate.
- Index maintenance on re-materialization (spec §6).
- The spec §11 golden fixtures, compared byte-for-byte.
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
  rooted in `IngestError`.

[0.5.0a2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.5.0a1...v0.5.0a2
[0.5.0a1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.4.0...v0.5.0a1
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1
[0.3.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.2.0...v0.3.0
[0.2.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/dae0bd1a...v0.2.0
[0.1.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/src/commit/dae0bd1a