Kjell Tore Guttormsen
2d9fb0f934
The guard could not read back what this library WRITES. At 1.2.0,
`okf.parse_frontmatter` refused the OKF v0.2 golden outright --
`OKFFrontmatterError: value begins with a disallowed YAML indicator '['`
against `sources: [{ id: golden-v0-2-sales, resource: fixture }]`. Flow is
the only form this library can emit, because its own line-oriented parser
cannot round-trip the block form at all, so a gate that refuses flow
refuses everything Door A produces under `OKF_V0_2`.
The control was run BEFORE the bump, which is the only moment it exists:
the probe raised on 1.2.0, so the new test discriminates rather than
merely passes. `[project.dependencies]` already said `>=1.2,<2.0` and is
unchanged; only `[tool.uv.sources]` and `uv.lock` move.
TWO gate rows moved, not the one the work was scoped around, which is why
the whole documented probe was re-run instead of just the `sources` case:
the BLOCK form of `sources` now passes too, retiring G30. That changes
nothing about what we emit -- our own parser is still the binding
constraint on writing flow -- and `docs/okf-nokkelinventar.md` now carries
a `guard 1.3.0` column beside the 1.2.0 measurement rather than
overwriting it. A third row kept its verdict but changed its reason, so
the quoted message was corrected too.
The Door C boundary is unmoved, verified with a known-positive:
`resource` is allowlisted only inside a `sources` entry, so section
10.2's `executor.resource` and `attester.resource` are still rejected
("not on the OKF mapping allowlist under 'executor'") while top-level
`resource` passes.
`uv.lock` also gains `pypandoc-binary==1.17`. That is a stale lockfile
being corrected, not a new dependency: it was already declared in the
`[extract]` extra, and `uv lock --check` reports the lockfile out of date
on the untouched tree. Core keeps exactly one runtime dependency.
Not addressed, and recorded rather than built: the guard reports that
`sources[].resource` is scanned as text but never URL-validated, because
SPEC 5.1 permits bundle-relative paths and scope descriptions. No
consumer has asked for a gate there.
Guard 1.3.0 installed from 44e2b31, verified anonymously over https
against the remote tag. 1054 -> 1055 tests. `mypy --strict` clean, `ruff`
clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
532 lines
30 KiB
Markdown
532 lines
30 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to this project will be documented in this file.
|
||
|
||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||
|
||
## [Unreleased]
|
||
|
||
### Added
|
||
|
||
- **The corpus harness (`tools/okf_corpus_run.py`) can replay segmentation
|
||
plans, and it writes the bundle's `log.md`.** `--plans-dir` names the
|
||
proposals to replay and the profile follows from it; `--bundle-id` and
|
||
`--okf-version` are arguments, never constants, because a profile names a key
|
||
and the caller owns its value. `log.md` is written in SPEC section 9 form and
|
||
dated from `ingested_at`, so `merged + sum(coded rejections) == N` is
|
||
checkable from the bundle alone rather than only from a report that does not
|
||
travel with it. Without `--plans-dir` a run is unchanged.
|
||
- **The corpus harness links that `log.md` from the bundle's root `index.md`.**
|
||
Measured on the K2 artifact by a consumer: the log was on disk and no index
|
||
named it, so a reader entering the bundle at `index.md` -- the walk section 8
|
||
exists to support -- never reached the one file carrying `N`. A LOCAL choice
|
||
rather than conformance, and stated as one: upstream's own bundles do not
|
||
link their log (measured at `9a15b13`, 0 of 24 shipped `index.md` files name
|
||
the single `log.md` in the set), which shows the link is not required, not
|
||
that it is disallowed. It lives in the harness because the library cannot
|
||
make it -- the log's content is the run's outcome, so an index that
|
||
enumerated it off the directory would gain the link only from the second run
|
||
and break rebuild-equals-incremental. Written only when absent, because the
|
||
two reprojections differ: the per-directory one drops the line as a managed
|
||
entry, the flat one keeps it (its target is not an owned concept), and
|
||
appending unconditionally doubled it on the second unsegmented run.
|
||
|
||
Measured cause: the harness passed `STRUCTURED_V1` and no plans, so a
|
||
43-document corpus arrived as 39 flat concepts with no `adjudication` key
|
||
anywhere. Rebuilt, the same corpus yields 629 concepts, 618 of them
|
||
`adjudication: proposed`. Record: `docs/2026-09-03-k2-bundle-rebuild.md`.
|
||
|
||
- **`tools/okf_propose_segments.py` takes `--path-prefix`.** Section numbering
|
||
is document-local, so across 39 documents 618 proposed entries claimed only
|
||
601 distinct paths -- 17 collisions that Door B's gate refuses per document.
|
||
Scoping each document's entries under a caller-supplied prefix removes all
|
||
17. Without the flag every artifact already produced is byte-identical.
|
||
|
||
### Changed
|
||
|
||
- **The pinned `llm-ingestion-guard` moves to `v1.3.0`** (`[tool.uv.sources]`
|
||
and `uv.lock`; the `>=1.2,<2.0` range in `[project.dependencies]` already
|
||
covered it and is unchanged). What this fixes is that the guard could not
|
||
read back what this library WRITES: at `1.2.0` the flow-form `sources` in the
|
||
OKF v0.2 golden was refused outright, and flow is the only form this library
|
||
is able to emit, because its own line-oriented parser cannot round-trip the
|
||
block form. Measured before the bump so the test discriminates rather than
|
||
merely passes, and pinned by
|
||
`tests/test_guard_adapter.py::test_the_guard_parses_the_flow_form_sources_our_goldens_emit`.
|
||
Two rows of the gate table in `docs/okf-nokkelinventar.md` moved, not one:
|
||
the BLOCK form of `sources` now passes too, which retires G30 -- though it
|
||
changes nothing about what we emit, since our own parser is still the binding
|
||
constraint. `resource` is allowlisted only inside a `sources` entry, so
|
||
section 10.2's `executor`/`attester` resource stays rejected through every
|
||
carrier and the Door C boundary is unmoved. No public API of this library
|
||
changes; a consumer's cost is a re-run.
|
||
- **`uv.lock` also picks up `pypandoc-binary==1.17`**, which is a stale lockfile
|
||
being corrected rather than a new dependency: the package was already declared
|
||
in the `[extract]` extra, and `uv lock --check` reports the lockfile as out of
|
||
date on the untouched tree. Core still has exactly one runtime dependency.
|
||
- **`tools/okf_propose_segments.py` writes no artifact when it has nothing to
|
||
propose**, exiting `1` (distinct from `2`, "could not do the job") instead of
|
||
`0` with an empty plan. An empty plan cannot be replayed -- `process_inbox`
|
||
refuses one, because a plan naming no entry would persist nothing for a
|
||
document that was dropped -- so the file's only possible use was to fail a
|
||
run later, and it did. Measured: 11 of 39 documents in the K2 corpus propose
|
||
zero segments.
|
||
|
||
- **A concept can now record more than one source.** `sources` renders a flow
|
||
sequence of N flow mappings on one line, so a v0.2 profile can express
|
||
multi-source provenance instead of the single entry that was the measured
|
||
ceiling on SPEC 5.1 coverage. **A single source is byte-identical to before**,
|
||
so every golden is unmoved and no existing bundle changes.
|
||
|
||
The form is flow, not the block list PM decision B6 prescribed, and the
|
||
reason is measured: this library's frontmatter parser is line-oriented and
|
||
skips indented lines, so a block list round-trips to an EMPTY value with every
|
||
entry silently gone -- and the consumer the decision was written for accepts
|
||
the multi-entry flow sequence while classifying a block sequence as unreadable
|
||
provenance. Emitting block would have produced records neither side can read.
|
||
A negative-control test pins the block form's data loss so the reason stays
|
||
measurable rather than remembered.
|
||
|
||
New error code `sources_empty`: an empty list is refused, because
|
||
`sources: []` reads as a measured absence when it is the absence of a
|
||
measurement.
|
||
|
||
- **The consumption contract, stated normatively** in
|
||
`docs/consumption-contract.md`, with a copyable skill template
|
||
(`skills/okf-consume-template/`) and a checker (`tools/okf_contract_check.py`)
|
||
that reads its mechanically checkable half: payload shape, source marking per
|
||
excerpt, the closed `adjudication` and `trust_tier` state sets, denominator
|
||
identity, and the budget gate with its validated known-positive. The checker
|
||
ships outside `src/`, so no consumer's install surface changes.
|
||
|
||
- **Five office formats through a vendored converter**, behind the same
|
||
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
|
||
converter binary travels inside the wheel and is resolved by path rather than
|
||
found on `PATH`, with its version asserted against a pin -- `pypandoc`
|
||
searches `PATH` first and takes the highest version it finds, so a vendored
|
||
binary buys nothing until something resolves it explicitly. A host carrying a
|
||
different converter is refused rather than silently used.
|
||
|
||
**Two of the five rows are measured; three are not.** The corpus this work
|
||
was measured on contains zero `pptx`, `odt` and `rtf` files, so those rows
|
||
work by construction and have never met a document anyone wrote. The
|
||
distinction is asserted in the suite, not left in a comment.
|
||
|
||
`.doc` (Word 97) stays out -- the converter does not read it. Drawn content
|
||
does not survive extraction in any format, and every conversion warns about
|
||
it.
|
||
|
||
- **Four error codes** for the converter path: `extractor_binary_missing`,
|
||
`extractor_binary_version`, `extractor_convert_error` and
|
||
`extractor_empty_conversion`.
|
||
|
||
- **`tests/test_docs_promises.py`**, which asserts the README's published
|
||
format list against the registries it describes. This exists because the
|
||
README's previous promise -- that `docx` and `xlsx` always fail fast -- went
|
||
false silently when the converter landed. A published guarantee is a test
|
||
obligation.
|
||
|
||
|
||
- **Door B extracts `pdf` behind the optional `[extract]` extra.** The extra is
|
||
populated for the first time, with one parser: `pdfplumber>=0.11.10,<0.12`
|
||
(MIT). The default install is unchanged — still exactly one runtime
|
||
dependency, still stdlib otherwise — and a packaging test enforces that.
|
||
|
||
**The parser choice was forced by a measurement, not by preference**
|
||
(`docs/2026-08-21-g2-pdf-extraction-measurement.md`). On a real Vegnormalene
|
||
requirement table, `pdfplumber` keeps 4 of 4 rows with label and value on the
|
||
same line; `pypdf`, `pdfminer.six` and `pymupdf` each keep 0 of 4, emitting
|
||
all labels and then all values. A downstream reader can only re-pair those by
|
||
guessing, and in a requirements document a wrong pairing looks right.
|
||
`pymupdf` was additionally excluded on licence (AGPL-3.0 or commercial):
|
||
this package is MIT and an optional extra must not hand a consumer copyleft
|
||
they did not choose.
|
||
|
||
**`docx` and `xlsx` were unchanged at this release**, shipping no parser and
|
||
failing fast with `extractor_extra_missing`. That is no longer true: a later
|
||
release added a vendored converter reaching five office formats. This entry
|
||
is left as written — a changelog records what a release did — and the
|
||
correction is stated here so a reader arriving at this line is not misled.
|
||
|
||
- **`ExtractionWarning`**, exported from the package. Every `pdf` extraction
|
||
emits one. Text extraction recovers text; anything a PDF *draws* — figures,
|
||
diagrams, images — has no text to recover, so only captions survive and a
|
||
bundle built from drawn documents is **incomplete by construction**. That is
|
||
categorically true rather than document-specific, so it is stated rather than
|
||
detected: deciding "is there a figure on this page" is a layout heuristic this
|
||
library does not own. A named class so it can be filtered deliberately.
|
||
|
||
- **Two error codes**, both mirroring existing patterns rather than inventing
|
||
behaviour: `extractor_empty_pdf` (a PDF yielded no text on any page — a
|
||
scanned or image-only document; refused rather than persisted as an empty
|
||
concept, which would be the silent skip this registry exists to prevent) and
|
||
`extractor_pdf_error` (the parser failed on the bytes; the third-party
|
||
exception is wrapped, never leaked).
|
||
|
||
### Changed
|
||
|
||
- **The `[extract]` gate for `pdf` is now an import probe rather than a
|
||
membership test.** The rejection did not change: without the extra installed,
|
||
`pdf` still raises `ExtractionError` with code `extractor_extra_missing` and
|
||
the same message naming the remedy. A consumer that has not installed the
|
||
extra sees no difference at all. That behaviour is asserted unconditionally,
|
||
including on machines where the parser *is* installed, so it cannot rot into
|
||
a skipped test.
|
||
|
||
**Known limitation, stated rather than worked around:** structured table
|
||
recovery is out of scope. `pdfplumber.extract_tables()` and
|
||
`PyMuPDF.find_tables()` — two independent implementations — return the *same*
|
||
wrong shape for the measured requirement table, and across the whole handbook
|
||
only 45 of 196 detected table objects are clean enough to hand to
|
||
`render_table` unchanged. The breakage is in the documents' ruling geometry,
|
||
not in either library. PDFs therefore enter this library as **prose**, with
|
||
table lines correctly paired.
|
||
|
||
**Extracted PDF text is pinned to an exact parser version.** `pdfplumber`
|
||
pins `pdfminer.six==20260107` exactly, and `pdfminer.six` ships date-stamped
|
||
releases with no stability contract. Extraction is deterministic within a
|
||
parser version (measured across five configurations) and not guaranteed
|
||
across one, so any golden fixture built on extracted PDF text is a fixture
|
||
migration away from a parser upgrade. `tests/test_extract.py` freezes the
|
||
expected text of a committed fixture so that upgrade breaks something visible
|
||
instead of drifting silently; see `tests/fixtures/README.md`.
|
||
|
||
- **`DEFAULT` now stamps `generated: { by: process:okf-ingest, at: <ingested_at> }`
|
||
instead of `generated: true`.** This is a byte change in every concept file Door A
|
||
writes under `DEFAULT`, so a consumer's own golden fixtures will show one changed
|
||
line per generated file.
|
||
|
||
The trigger is not an upstream Google release. `DEFAULT` states the ingest-spec
|
||
layer owned by `portfolio-optimiser-commons`, and they ratified this shape
|
||
(2026-08-02) and executed it in their spec on 2026-08-09. §7 defines the value:
|
||
the actor is the fixed `process:okf-ingest`, `at` repeats `ingested_at` verbatim,
|
||
and it is unquoted because frontmatter is parsed line-oriented — a quote would be
|
||
a character in the value rather than syntax a reader strips.
|
||
|
||
**What it costs a consumer is a re-run, and nothing else.** Ownership recognition
|
||
is one-way: a profile carrying an actor still owns the older literal stamp, so a
|
||
bundle written by an earlier version re-runs in place rather than tripping the §3
|
||
collision gate. No call signature changed and no key was added or removed.
|
||
|
||
Two things it is NOT. It is not a migration of `DEFAULT` onto OKF v0.2 — the
|
||
profile remains v0.1 on every axis upstream owns, and still emits no `sources`.
|
||
And it does not make `DEFAULT` and `OKF_V0_2` the same profile; they now agree on
|
||
the stamp and continue to differ in index root frontmatter, type-conditional
|
||
requirements, and `sources` derivation.
|
||
|
||
Door B (`process_inbox`) is deliberately unchanged: its marker is `generated`
|
||
plus `source_file`, it is not governed by the ingest-spec, and it stays disjoint
|
||
from Door A's `ingest_manifest`.
|
||
|
||
- **The v0.5.0 pilot set gained a fourth member: `portfolio-optimiser`.** Admitted
|
||
2026-08-09 on their request. `v0.5.0a1`'s entry below says "do not pin this tag
|
||
outside the pilot set" and names three repos; that entry is left as written,
|
||
because it records what was true when the tag shipped. This is the amendment,
|
||
and the sentence still binds — the set is now those three plus
|
||
`portfolio-optimiser`.
|
||
|
||
The reason is the producer axis, not the count. `portfolio-optimiser-claude` is
|
||
parked, and with it parked no original member could *emit* a v0.2 bundle at all:
|
||
`claude-code-llm-wiki` is read-only in the pilot and `catalog` is gate-side. The
|
||
new member consumes the same Door A. Nothing about the provisional surface
|
||
changes: `OKF_LATEST` still points at `DEFAULT`, and the v0.2 surface may still
|
||
move on pilot feedback without a deprecation cycle.
|
||
|
||
**If you pin `v0.5.0a2`, its guard pin is `>=0.2,<0.3`** — that tag resolves
|
||
against guard `v0.2.0`, *not* the `v0.3.4` this repo's `main` now uses. `main`
|
||
moved to `>=0.3,<0.4` after the tag. Also note `tool.uv.sources` is not
|
||
inherited transitively: a consumer supplies the guard's source itself.
|
||
|
||
- **The guard pin moved to `>=0.3,<0.4`, resolved against `v0.3.4`.** The
|
||
window is widened only after measurement, never before: the 19-fixture
|
||
guard-surface suite was run against `v0.3.4` in a scratch venv first, and
|
||
reproduced exactly the three deltas measured against `v0.3.3` — no new ones.
|
||
`v0.3.4`'s own fixes are regex-complexity repairs, one of them
|
||
(`okf._MD_LINK_RE`) on Door C's call path, with no disposition changes.
|
||
|
||
- **Door C now passes `allow_reserved=False` to `okf.import_bundle`.** The
|
||
guard added the keyword in the `0.3.x` line and defaults it `True` for the
|
||
received-bundle path, which would merge a sender's `index.md` / `log.md`
|
||
instead of rejecting them. Door C overrides it, keeping the unconditional
|
||
reserved-name refusal committed to before the keyword existed. The reason is
|
||
structural rather than a second opinion on the guard's scan: Door C generates
|
||
the merged bundle's `index.md` from what it merged and writes every merged
|
||
concept verbatim, so a sender's `index.md` would be a second, irreconcilable
|
||
claim on one path.
|
||
|
||
**This is not a behaviour change for anyone on the previous pin.** Under
|
||
`v0.2.0` the keyword did not exist and reserved names were refused by
|
||
construction; the explicit argument preserves that outcome across the bump.
|
||
A consumer sees the same rejections, with the same reasons, before and after.
|
||
|
||
## [0.5.0a2] — 2026-07-31
|
||
|
||
**This is the pre-release the pilots pin. `v0.5.0a1` was tagged and abandoned
|
||
unused — do not pin it.** It carried a `generated.by` actor that the spec owner
|
||
had already excluded, and it was caught before any pilot was notified.
|
||
|
||
### Fixed
|
||
|
||
- **`OKF_V0_2`'s `generated.by` actor is `process:okf-ingest`**, not
|
||
`process:llm-ingestion-okf`. Commons decided the fixed id on this repo's own
|
||
proposal 2026-07-31, superseding option (d) chosen here 2026-07-27. The
|
||
exclusion is `ingest-spec.md:7-8`, frozen on the spec being framework-neutral:
|
||
normalising *our* repo name into the id would force every other conformant
|
||
implementation to write it into its own output.
|
||
|
||
Nothing in the wild carried the excluded value — `OKF_V0_2` did not exist at
|
||
`v0.4.0`, and no pilot had been notified — so this costs a tag rather than a
|
||
migration. It is recorded rather than quietly folded in because the failure it
|
||
avoids is specific: `actor` is both the stamp written and the value owned back
|
||
(`OwnershipPolicy`), and recognition is one-way, so a pilot holding bundles
|
||
stamped with the excluded id would have hit `collision_unstamped` on its own
|
||
files the moment the id was corrected.
|
||
|
||
## [0.5.0a1] — 2026-07-31 — ABANDONED, do not pin
|
||
|
||
**Pre-release. PROVISIONAL surface.** Shipped to a named pilot set —
|
||
`portfolio-optimiser-claude`, the plugin marketplace catalog, and
|
||
`claude-code-llm-wiki` — so that real data can find what fixtures cannot. The
|
||
v0.2 surface may change on their feedback **without a deprecation cycle**.
|
||
Saying so up front is what buys the freedom to act on the feedback; discovering
|
||
it later is what would make a pilot a de-facto release. Do not pin this tag
|
||
outside the pilot set.
|
||
|
||
Support for a new upstream version is **additive — a new profile, never a
|
||
migration**. `OKF_LATEST` still points at `DEFAULT`; flipping it is the GA
|
||
event, not a side effect of this tag.
|
||
|
||
### Added
|
||
|
||
- **`OKF_V0_2` profile.** Google OKF v0.2 as a profile alongside `DEFAULT` and
|
||
`STRICT_V1`. It closes nothing: §14 forbids a conformant consumer to reject on
|
||
an unknown `type` value or on unknown additional keys, so `type` is the only
|
||
required key. It NAMES `okf_version` but never carries its value — that value
|
||
tracks the upstream Google version and belongs to catalog (decision E1), so
|
||
the caller supplies it via `materialize_bundle(..., root_frontmatter_values=…)`
|
||
and the profile fixes only the key and its position.
|
||
- **`root_frontmatter_values`** on `materialize_bundle`, keyword-only: the
|
||
mechanism behind "a profile names a key, a caller owns its value".
|
||
- **`sources` emitted as an inline flow sequence**, populated from the manifest.
|
||
- **The v0.2 golden fixture**, compared byte-for-byte like the others.
|
||
|
||
### Changed
|
||
|
||
- **`profile` on `materialize_bundle` is keyword-only**, so existing positional
|
||
call sites stay source-compatible.
|
||
- **`_is_ingest_owned` is profile-aware.** Ownership is a policy on the profile,
|
||
not a literal; recognition stays one-way.
|
||
|
||
### Fixed
|
||
|
||
- **`root_frontmatter` permitted a key without requiring it — the two had been
|
||
the same profile field.** Upstream says MAY where the field said MUST, so a
|
||
root index that legally omitted an optional key was rejected. `root_frontmatter`
|
||
now permits and orders; the new `root_frontmatter_required` requires.
|
||
`STRICT_V1` keeps requiring its three (unchanged for its consumer), `OKF_V0_2`
|
||
requires none, `DEFAULT` is untouched. Measured against the eight root indexes
|
||
available locally: 7 of 8 failed before, 0 of 8 after.
|
||
|
||
### Notes
|
||
|
||
- **`DEFAULT` and `STRICT_V1` are byte-stable across this release.** The emit
|
||
path is byte-identical; the golden suite is the check.
|
||
- **Known, deliberately unfixed:** `TypePolicy`'s closed branch does not strip
|
||
quotes from a declared type. All three call sites pass `DEFAULT.types`, and
|
||
both `DEFAULT` and `OKF_V0_2` set `allowed=None`, so the branch is
|
||
unreachable in shipped code — it fires only for a caller constructing
|
||
`STRICT_V1` and calling `rejection` directly. Fixing it would repair a write
|
||
path that never sees quotes, and would fix the meaning of a quote character
|
||
without a value model that can express one, in a line-oriented format.
|
||
|
||
## [0.4.0] — 2026-07-25
|
||
|
||
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
|
||
and with them this library's first — and only ever — runtime dependency.
|
||
|
||
### Added
|
||
|
||
- **Door B — bundle inbox (`process_inbox`).** Per dropped file: bytes →
|
||
extraction → persist gate → render → collision gate → write → index link.
|
||
Returns an `InboxResult` whose four buckets (`persisted`, `quarantined`,
|
||
`rejected`, `failed`) are disjoint and complete, so a file that vanished
|
||
mid-run surfaces as a missing entry rather than as nothing. One bad file
|
||
never aborts the run: only an invalid `ingested_at`, a reserved `okf_type`,
|
||
and a missing inbox directory fail the whole run, each being wrong for every
|
||
file at once. Concept files are named `inbox-{slug}.md`, disjoint from
|
||
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*`; `source_sha256`
|
||
is taken over the original dropped bytes, so provenance stays re-verifiable
|
||
against the operator's file.
|
||
- **Door C — external bundle import (`import_bundle`).** Reads an external OKF
|
||
bundle and hands it *whole* to an injected gate over the guard's
|
||
`okf.import_bundle` — a bundle-level call, because it resolves the
|
||
cross-link graph across concepts — merging only concepts that clear the
|
||
non-blocking floor. Two invariants, both load-bearing: a merged concept is
|
||
written **verbatim** (stamping it would require round-tripping frontmatter
|
||
through this library's line-oriented parser, which cannot represent the
|
||
block lists the guard's parser accepts, and would persist bytes the guard
|
||
never screened), and ownership is therefore proven by **content identity** —
|
||
an occupied target name is re-used only when the bytes there are already
|
||
identical, never overwritten. Re-importing an unchanged bundle is a no-op.
|
||
- **`extract_text` — the Door B extraction registry.** All file-type → text
|
||
extraction lives in this library, because the guard is text-only. `md`/`txt`
|
||
pass through, `csv` renders the markdown table, `json` is fenced verbatim,
|
||
and `html`/`htm` reduce to text with `html.parser` — stdlib throughout.
|
||
`pdf`, `docx`, and `xlsx` are gated behind the `[extract]` extra, which
|
||
ships no parser yet: those types fail fast with a typed error naming the
|
||
extra, never a silent skip.
|
||
- **`llm_ingestion_okf.guard_adapter`** — the shipped gate over the real
|
||
guard (`inbox_gate`, `import_gate`), and the only module here that imports
|
||
it. Door B screens the exact bytes it persists: the guard's `prepare_input`
|
||
bookend prepares text for a model call this library never makes, so
|
||
`screen_output` alone is used and the screened string is the written string.
|
||
It follows that the gate **refuses rather than repairs** — a file carrying
|
||
an invisible carrier is rejected, not stripped and persisted. The policy is
|
||
`PRESET_USER_UPLOAD`, so any finding at all is held back.
|
||
- **15 new stable error codes**, each registered in the `errors.py` docstrings
|
||
(the stability contract) and pinned by the error-code conformance suite:
|
||
`extractor_decode_error`, `extractor_empty_csv`, `extractor_extra_missing`,
|
||
`extractor_unknown`, `import_label_invalid`, `import_path_empty`,
|
||
`import_path_too_long`, `import_provenance_invalid`,
|
||
`import_slug_collision`, `inbox_slug_collision`, `inbox_slug_empty`,
|
||
`inbox_slug_too_long`, `inbox_source_file_invalid`, `inbox_title_invalid`,
|
||
`okf_type_reserved`.
|
||
|
||
### Changed
|
||
|
||
- **`llm-ingestion-guard>=0.2,<0.3` is now a mandatory runtime dependency.**
|
||
Installing this package installs the guard. Importing it does not: only
|
||
`guard_adapter` imports the guard, so a Door A consumer keeps working
|
||
whatever state the dependency is in, and the doors themselves still take an
|
||
*injected* gate. Until the guard is published to a package index, install it
|
||
from its tag — see the README. A packaging test enforces that this stays the
|
||
only runtime dependency.
|
||
- **An extraction `title` containing `[` or `]` is rejected at manifest load**
|
||
(ingest-spec §4, ratified D1). Previously only single-line was validated. A
|
||
manifest that loaded before and carries a bracket in a title now fails fast
|
||
with code `manifest_schema`: the title renders verbatim into the index link
|
||
label `- [title](target)`, where a bracket breaks index-link and navigation
|
||
parsing downstream.
|
||
|
||
### Fixed
|
||
|
||
- **Materializing one manifest no longer deletes the files another manifest
|
||
stamped into the same bundle.** The §3 ownership scan classified every
|
||
ingest-stamped file as replaceable, so a second manifest sharing a bundle
|
||
removed the first one's concept files and their index links. Ownership is
|
||
now narrowed to files whose stamp names the running manifest by stem — the
|
||
stem is stable across content edits, so an edited manifest still reclaims
|
||
what a prior run of itself wrote. The operator-copy restriction (a generated
|
||
file copied into curated content) remains documented, not enforced.
|
||
|
||
### Notes
|
||
|
||
- Phase 2's binary extraction is **not** in this release: the `[extract]`
|
||
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
|
||
That is the one outstanding item from the phase, and it ships separately.
|
||
- **The ownership change above is an extension point, not a spec fix.** Filed
|
||
under "Fixed" because it stops silent data loss, but the spec owner
|
||
(`portfolio-optimiser-commons`) has since pointed out that removing *every*
|
||
ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes
|
||
one manifest per bundle and defers multiple manifests feeding one bundle as
|
||
a named extension point. So this release implements that extension point and
|
||
outruns the frozen text rather than conforming to it. The mechanic itself is
|
||
not at risk — matching by manifest stem follows from the spec's own
|
||
`{stem}@{h}` stamp definition, since `{h}` changes on every content edit —
|
||
but the surrounding prose is queued for amendment in commons and is not
|
||
ratified. Treat multi-manifest bundles as ahead of the spec until it is.
|
||
- Exception `__cause__` preservation is now pinned by a conformance suite, one
|
||
test per fail-fast wrap site, alongside the existing `.code` suite.
|
||
|
||
## [0.3.2] — 2026-07-23
|
||
|
||
### Fixed
|
||
|
||
- **Frontmatter and index-label values are emitted verbatim; only
|
||
`source_query` is whitespace-collapsed.** Earlier releases collapsed every
|
||
whitespace run in every frontmatter value and index link label to a single
|
||
space. §5 of `ingest-spec.md` mandates that collapse for `source_query`
|
||
alone — where a multi-line SQL `SELECT` must render on one line — while
|
||
every other value is validated single-line at manifest load and passed
|
||
through unchanged: validation, not repair. A `title` carrying an internal
|
||
whitespace run now survives byte-for-byte at both the `title` frontmatter
|
||
and the index link label, instead of being silently altered. Output bytes
|
||
change only for values that contained a collapsible whitespace run; the
|
||
shipped golden fixtures and both consumers are unaffected.
|
||
|
||
## [0.3.1] — 2026-07-19
|
||
|
||
### Fixed
|
||
|
||
- **Documentation corrected a security claim that did not hold.** The module
|
||
docstring and README stated that this library "calls the guard at every persist
|
||
gate". That described the intended end state in the present tense. Door A — the
|
||
only door shipped — is ungated: the package has zero runtime dependencies and
|
||
calls no guard function before writing to disk. Both places now say so plainly,
|
||
and state that gating external or untrusted content is the caller's
|
||
responsibility (`okf.import_bundle`, or `prepare_input`/`screen_output`) until
|
||
the persist gates land with Doors B and C.
|
||
|
||
No behavior changed in this release. The correction is published because a
|
||
consumer read the earlier wording as safe-by-default and would have persisted
|
||
ungated content on that basis.
|
||
|
||
## [0.3.0] — 2026-07-17
|
||
|
||
### Added
|
||
|
||
- **Stable machine-readable error codes.** `IngestError` gained a keyword-only
|
||
`code` attribute (default `"unspecified"`). Roughly 24 codes are documented in
|
||
the `errors.py` docstrings, and those docstrings are the stability contract.
|
||
Consumers should assert on `exc.value.code`, not on message text.
|
||
|
||
### Changed
|
||
|
||
- **Exception message text is explicitly declared unstable.** It may change in any
|
||
release. Tests matching on message strings (`pytest.raises(match=...)`) should
|
||
migrate to code comparisons.
|
||
|
||
### Notes
|
||
|
||
- Generic schema shape violations deliberately share the single code
|
||
`manifest_schema`. One code per validation rule would have frozen an
|
||
unnecessarily large surface. Finer resolution is a separate decision, not an
|
||
assumed requirement.
|
||
|
||
## [0.2.0] — 2026-07-17
|
||
|
||
### Added
|
||
|
||
- **PEP 561 support.** The `py.typed` marker ships with the package, so consumers
|
||
get the inline type hints without a mypy override.
|
||
- **`IngestResult.stamp`** exposes the spec §5 provenance stamp on the result
|
||
object.
|
||
|
||
## [0.1.0] — 2026-07-16
|
||
|
||
Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
|
||
`portfolio-optimiser-commons`. Never tagged; consumers pinned the commit
|
||
`dae0bd1a` directly.
|
||
|
||
### Added
|
||
|
||
- Fail-fast manifest validation (spec §3–§4).
|
||
- Spec §5 body renderers as pure functions.
|
||
- The `file` connector (CSV, fail-closed path boundary), the `sql` connector
|
||
(read-only sqlite, env-resolved credentials), and the `http` connector behind an
|
||
explicit per-run network opt-in.
|
||
- Spec §5 materialization with an in-memory staging collision gate.
|
||
- Index maintenance on re-materialization (spec §6).
|
||
- The spec §11 golden fixtures, compared byte-for-byte.
|
||
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
|
||
rooted in `IngestError`.
|
||
|
||
[0.5.0a2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.5.0a1...v0.5.0a2
|
||
[0.5.0a1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.4.0...v0.5.0a1
|
||
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
|
||
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
|
||
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1
|
||
[0.3.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.2.0...v0.3.0
|
||
[0.2.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/dae0bd1a...v0.2.0
|
||
[0.1.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/src/commit/dae0bd1a
|