open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity
llm-ingestion-okf/CHANGELOG.md
Kjell Tore Guttormsen 524ade78a6 chore(release): v0.4.0 
Phase 2 ships: Doors B and C, the extraction registry, the guard adapter,
and with them the first and only runtime dependency this library takes.
The minor bump is the news consumers are waiting on -- installing the
package now installs llm-ingestion-guard>=0.2,<0.3, though importing it
still does not import the guard.

Two behaviour changes carried in the same release are called out in the
changelog rather than buried: an extraction title containing `[` or `]`
now fails at manifest load (ingest-spec section 4), and the section 3
ownership scan no longer deletes files another manifest stamped into the
same bundle.

uv.lock was stale -- it predated the guard dependency and could not be
regenerated, because the guard is not on a package index. A
[tool.uv.sources] entry pins the same tag the README documents, so
`uv lock`/`uv sync` resolve. It is uv-specific dev metadata: the built
wheel still carries `Requires-Dist: llm-ingestion-guard<0.3,>=0.2`,
verified against the wheel, so the range remains the pin and the git URL
remains a channel.

Binary extraction (pdf/docx/xlsx behind [extract]) is deliberately not in
this release and is stated as outstanding in both README and changelog.

425 tests green; ruff, ruff format and mypy --strict clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
2026-07-25 12:08:48 +02:00

184 lines
9.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.4.0] — 2026-07-25
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
and with them this library's first — and only ever — runtime dependency.
### Added
- **Door B — bundle inbox (`process_inbox`).** Per dropped file: bytes →
extraction → persist gate → render → collision gate → write → index link.
Returns an `InboxResult` whose four buckets (`persisted`, `quarantined`,
`rejected`, `failed`) are disjoint and complete, so a file that vanished
mid-run surfaces as a missing entry rather than as nothing. One bad file
never aborts the run: only an invalid `ingested_at`, a reserved `okf_type`,
and a missing inbox directory fail the whole run, each being wrong for every
file at once. Concept files are named `inbox-{slug}.md`, disjoint from
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*`; `source_sha256`
is taken over the original dropped bytes, so provenance stays re-verifiable
against the operator's file.
- **Door C — external bundle import (`import_bundle`).** Reads an external OKF
bundle and hands it *whole* to an injected gate over the guard's
`okf.import_bundle` — a bundle-level call, because it resolves the
cross-link graph across concepts — merging only concepts that clear the
non-blocking floor. Two invariants, both load-bearing: a merged concept is
written **verbatim** (stamping it would require round-tripping frontmatter
through this library's line-oriented parser, which cannot represent the
block lists the guard's parser accepts, and would persist bytes the guard
never screened), and ownership is therefore proven by **content identity** —
an occupied target name is re-used only when the bytes there are already
identical, never overwritten. Re-importing an unchanged bundle is a no-op.
- **`extract_text` — the Door B extraction registry.** All file-type → text
extraction lives in this library, because the guard is text-only. `md`/`txt`
pass through, `csv` renders the markdown table, `json` is fenced verbatim,
and `html`/`htm` reduce to text with `html.parser` — stdlib throughout.
`pdf`, `docx`, and `xlsx` are gated behind the `[extract]` extra, which
ships no parser yet: those types fail fast with a typed error naming the
extra, never a silent skip.
- **`llm_ingestion_okf.guard_adapter`** — the shipped gate over the real
guard (`inbox_gate`, `import_gate`), and the only module here that imports
it. Door B screens the exact bytes it persists: the guard's `prepare_input`
bookend prepares text for a model call this library never makes, so
`screen_output` alone is used and the screened string is the written string.
It follows that the gate **refuses rather than repairs** — a file carrying
an invisible carrier is rejected, not stripped and persisted. The policy is
`PRESET_USER_UPLOAD`, so any finding at all is held back.
- **15 new stable error codes**, each registered in the `errors.py` docstrings
(the stability contract) and pinned by the error-code conformance suite:
`extractor_decode_error`, `extractor_empty_csv`, `extractor_extra_missing`,
`extractor_unknown`, `import_label_invalid`, `import_path_empty`,
`import_path_too_long`, `import_provenance_invalid`,
`import_slug_collision`, `inbox_slug_collision`, `inbox_slug_empty`,
`inbox_slug_too_long`, `inbox_source_file_invalid`, `inbox_title_invalid`,
`okf_type_reserved`.
### Changed
- **`llm-ingestion-guard>=0.2,<0.3` is now a mandatory runtime dependency.**
Installing this package installs the guard. Importing it does not: only
`guard_adapter` imports the guard, so a Door A consumer keeps working
whatever state the dependency is in, and the doors themselves still take an
*injected* gate. Until the guard is published to a package index, install it
from its tag — see the README. A packaging test enforces that this stays the
only runtime dependency.
- **An extraction `title` containing `[` or `]` is rejected at manifest load**
(ingest-spec §4, ratified D1). Previously only single-line was validated. A
manifest that loaded before and carries a bracket in a title now fails fast
with code `manifest_schema`: the title renders verbatim into the index link
label `- [title](target)`, where a bracket breaks index-link and navigation
parsing downstream.
### Fixed
- **Materializing one manifest no longer deletes the files another manifest
stamped into the same bundle.** The §3 ownership scan classified every
ingest-stamped file as replaceable, so a second manifest sharing a bundle
removed the first one's concept files and their index links. Ownership is
now narrowed to files whose stamp names the running manifest by stem — the
stem is stable across content edits, so an edited manifest still reclaims
what a prior run of itself wrote. The operator-copy restriction (a generated
file copied into curated content) remains documented, not enforced.
### Notes
- Phase 2's binary extraction is **not** in this release: the `[extract]`
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
That is the one outstanding item from the phase, and it ships separately.
- Exception `__cause__` preservation is now pinned by a conformance suite, one
test per fail-fast wrap site, alongside the existing `.code` suite.
## [0.3.2] — 2026-07-23
### Fixed
- **Frontmatter and index-label values are emitted verbatim; only
`source_query` is whitespace-collapsed.** Earlier releases collapsed every
whitespace run in every frontmatter value and index link label to a single
space. §5 of `ingest-spec.md` mandates that collapse for `source_query`
alone — where a multi-line SQL `SELECT` must render on one line — while
every other value is validated single-line at manifest load and passed
through unchanged: validation, not repair. A `title` carrying an internal
whitespace run now survives byte-for-byte at both the `title` frontmatter
and the index link label, instead of being silently altered. Output bytes
change only for values that contained a collapsible whitespace run; the
shipped golden fixtures and both consumers are unaffected.
## [0.3.1] — 2026-07-19
### Fixed
- **Documentation corrected a security claim that did not hold.** The module
docstring and README stated that this library "calls the guard at every persist
gate". That described the intended end state in the present tense. Door A — the
only door shipped — is ungated: the package has zero runtime dependencies and
calls no guard function before writing to disk. Both places now say so plainly,
and state that gating external or untrusted content is the caller's
responsibility (`okf.import_bundle`, or `prepare_input`/`screen_output`) until
the persist gates land with Doors B and C.
No behavior changed in this release. The correction is published because a
consumer read the earlier wording as safe-by-default and would have persisted
ungated content on that basis.
## [0.3.0] — 2026-07-17
### Added
- **Stable machine-readable error codes.** `IngestError` gained a keyword-only
`code` attribute (default `"unspecified"`). Roughly 24 codes are documented in
the `errors.py` docstrings, and those docstrings are the stability contract.
Consumers should assert on `exc.value.code`, not on message text.
### Changed
- **Exception message text is explicitly declared unstable.** It may change in any
release. Tests matching on message strings (`pytest.raises(match=...)`) should
migrate to code comparisons.
### Notes
- Generic schema shape violations deliberately share the single code
`manifest_schema`. One code per validation rule would have frozen an
unnecessarily large surface. Finer resolution is a separate decision, not an
assumed requirement.
## [0.2.0] — 2026-07-17
### Added
- **PEP 561 support.** The `py.typed` marker ships with the package, so consumers
get the inline type hints without a mypy override.
- **`IngestResult.stamp`** exposes the spec §5 provenance stamp on the result
object.
## [0.1.0] — 2026-07-16
Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
`portfolio-optimiser-commons`. Never tagged; consumers pinned the commit
`dae0bd1a` directly.
### Added
- Fail-fast manifest validation (spec §3–§4).
- Spec §5 body renderers as pure functions.
- The `file` connector (CSV, fail-closed path boundary), the `sql` connector
(read-only sqlite, env-resolved credentials), and the `http` connector behind an
explicit per-run network opt-in.
- Spec §5 materialization with an in-memory staging collision gate.
- Index maintenance on re-materialization (spec §6).
- The spec §11 golden fixtures, compared byte-for-byte.
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
rooted in `IngestError`.
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1
[0.3.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.2.0...v0.3.0
[0.2.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/dae0bd1a...v0.2.0
[0.1.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/src/commit/dae0bd1a
Reference in a new issue View git blame Copy permalink