Kjell Tore Guttormsen
678f9b2b07
Both arrived within minutes of the v0.4.0 announcement and both change
what we would otherwise have built or claimed.
CATALOG -- a profile field we must NOT add. Their steg 5 measured bundle
placement across four consumers: exactly one installs a bundle anywhere,
the other three take the bundle directory as a per-run argument with no
default, no discovery and no $HOME expansion. Placement is unmandated, so
BundleProfile gets no root-location or install-path field, "no
installation root" is a conforming consumer shape, and write policy must
not be coupled to placement. Recorded in the split table, which also now
spells the proving consumer's layer prefixes bundle-relative
(concepts/, releases/) rather than repeating the bundle/ prefix their own
repository layout carries -- that prefix was placement leaking into a
contract.
COMMONS -- our "Fixed" framing outruns their frozen text. Removing every
ingest-stamped file is verbatim what ingest-spec v1 section 5 mandates:
v1 assumes one manifest per bundle and defers the multi-manifest case as
a named extension point. So v0.4.0 implements that extension point rather
than fixing a defect against the spec. The mechanic survives ratification
either way -- stem matching follows from their own {stem}@{h} stamp
definition, since {h} changes on every content edit -- but the prose is
queued for amendment and is not ratified, so the changelog now says so
instead of claiming conformance it does not have.
195 lines
10 KiB
Markdown
195 lines
10 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to this project will be documented in this file.
|
||
|
||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||
|
||
## [0.4.0] — 2026-07-25
|
||
|
||
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
|
||
and with them this library's first — and only ever — runtime dependency.
|
||
|
||
### Added
|
||
|
||
- **Door B — bundle inbox (`process_inbox`).** Per dropped file: bytes →
|
||
extraction → persist gate → render → collision gate → write → index link.
|
||
Returns an `InboxResult` whose four buckets (`persisted`, `quarantined`,
|
||
`rejected`, `failed`) are disjoint and complete, so a file that vanished
|
||
mid-run surfaces as a missing entry rather than as nothing. One bad file
|
||
never aborts the run: only an invalid `ingested_at`, a reserved `okf_type`,
|
||
and a missing inbox directory fail the whole run, each being wrong for every
|
||
file at once. Concept files are named `inbox-{slug}.md`, disjoint from
|
||
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*`; `source_sha256`
|
||
is taken over the original dropped bytes, so provenance stays re-verifiable
|
||
against the operator's file.
|
||
- **Door C — external bundle import (`import_bundle`).** Reads an external OKF
|
||
bundle and hands it *whole* to an injected gate over the guard's
|
||
`okf.import_bundle` — a bundle-level call, because it resolves the
|
||
cross-link graph across concepts — merging only concepts that clear the
|
||
non-blocking floor. Two invariants, both load-bearing: a merged concept is
|
||
written **verbatim** (stamping it would require round-tripping frontmatter
|
||
through this library's line-oriented parser, which cannot represent the
|
||
block lists the guard's parser accepts, and would persist bytes the guard
|
||
never screened), and ownership is therefore proven by **content identity** —
|
||
an occupied target name is re-used only when the bytes there are already
|
||
identical, never overwritten. Re-importing an unchanged bundle is a no-op.
|
||
- **`extract_text` — the Door B extraction registry.** All file-type → text
|
||
extraction lives in this library, because the guard is text-only. `md`/`txt`
|
||
pass through, `csv` renders the markdown table, `json` is fenced verbatim,
|
||
and `html`/`htm` reduce to text with `html.parser` — stdlib throughout.
|
||
`pdf`, `docx`, and `xlsx` are gated behind the `[extract]` extra, which
|
||
ships no parser yet: those types fail fast with a typed error naming the
|
||
extra, never a silent skip.
|
||
- **`llm_ingestion_okf.guard_adapter`** — the shipped gate over the real
|
||
guard (`inbox_gate`, `import_gate`), and the only module here that imports
|
||
it. Door B screens the exact bytes it persists: the guard's `prepare_input`
|
||
bookend prepares text for a model call this library never makes, so
|
||
`screen_output` alone is used and the screened string is the written string.
|
||
It follows that the gate **refuses rather than repairs** — a file carrying
|
||
an invisible carrier is rejected, not stripped and persisted. The policy is
|
||
`PRESET_USER_UPLOAD`, so any finding at all is held back.
|
||
- **15 new stable error codes**, each registered in the `errors.py` docstrings
|
||
(the stability contract) and pinned by the error-code conformance suite:
|
||
`extractor_decode_error`, `extractor_empty_csv`, `extractor_extra_missing`,
|
||
`extractor_unknown`, `import_label_invalid`, `import_path_empty`,
|
||
`import_path_too_long`, `import_provenance_invalid`,
|
||
`import_slug_collision`, `inbox_slug_collision`, `inbox_slug_empty`,
|
||
`inbox_slug_too_long`, `inbox_source_file_invalid`, `inbox_title_invalid`,
|
||
`okf_type_reserved`.
|
||
|
||
### Changed
|
||
|
||
- **`llm-ingestion-guard>=0.2,<0.3` is now a mandatory runtime dependency.**
|
||
Installing this package installs the guard. Importing it does not: only
|
||
`guard_adapter` imports the guard, so a Door A consumer keeps working
|
||
whatever state the dependency is in, and the doors themselves still take an
|
||
*injected* gate. Until the guard is published to a package index, install it
|
||
from its tag — see the README. A packaging test enforces that this stays the
|
||
only runtime dependency.
|
||
- **An extraction `title` containing `[` or `]` is rejected at manifest load**
|
||
(ingest-spec §4, ratified D1). Previously only single-line was validated. A
|
||
manifest that loaded before and carries a bracket in a title now fails fast
|
||
with code `manifest_schema`: the title renders verbatim into the index link
|
||
label `- [title](target)`, where a bracket breaks index-link and navigation
|
||
parsing downstream.
|
||
|
||
### Fixed
|
||
|
||
- **Materializing one manifest no longer deletes the files another manifest
|
||
stamped into the same bundle.** The §3 ownership scan classified every
|
||
ingest-stamped file as replaceable, so a second manifest sharing a bundle
|
||
removed the first one's concept files and their index links. Ownership is
|
||
now narrowed to files whose stamp names the running manifest by stem — the
|
||
stem is stable across content edits, so an edited manifest still reclaims
|
||
what a prior run of itself wrote. The operator-copy restriction (a generated
|
||
file copied into curated content) remains documented, not enforced.
|
||
|
||
### Notes
|
||
|
||
- Phase 2's binary extraction is **not** in this release: the `[extract]`
|
||
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
|
||
That is the one outstanding item from the phase, and it ships separately.
|
||
- **The ownership change above is an extension point, not a spec fix.** Filed
|
||
under "Fixed" because it stops silent data loss, but the spec owner
|
||
(`portfolio-optimiser-commons`) has since pointed out that removing *every*
|
||
ingest-stamped file is verbatim what ingest-spec v1 §5 mandates: v1 assumes
|
||
one manifest per bundle and defers multiple manifests feeding one bundle as
|
||
a named extension point. So this release implements that extension point and
|
||
outruns the frozen text rather than conforming to it. The mechanic itself is
|
||
not at risk — matching by manifest stem follows from the spec's own
|
||
`{stem}@{h}` stamp definition, since `{h}` changes on every content edit —
|
||
but the surrounding prose is queued for amendment in commons and is not
|
||
ratified. Treat multi-manifest bundles as ahead of the spec until it is.
|
||
- Exception `__cause__` preservation is now pinned by a conformance suite, one
|
||
test per fail-fast wrap site, alongside the existing `.code` suite.
|
||
|
||
## [0.3.2] — 2026-07-23
|
||
|
||
### Fixed
|
||
|
||
- **Frontmatter and index-label values are emitted verbatim; only
|
||
`source_query` is whitespace-collapsed.** Earlier releases collapsed every
|
||
whitespace run in every frontmatter value and index link label to a single
|
||
space. §5 of `ingest-spec.md` mandates that collapse for `source_query`
|
||
alone — where a multi-line SQL `SELECT` must render on one line — while
|
||
every other value is validated single-line at manifest load and passed
|
||
through unchanged: validation, not repair. A `title` carrying an internal
|
||
whitespace run now survives byte-for-byte at both the `title` frontmatter
|
||
and the index link label, instead of being silently altered. Output bytes
|
||
change only for values that contained a collapsible whitespace run; the
|
||
shipped golden fixtures and both consumers are unaffected.
|
||
|
||
## [0.3.1] — 2026-07-19
|
||
|
||
### Fixed
|
||
|
||
- **Documentation corrected a security claim that did not hold.** The module
|
||
docstring and README stated that this library "calls the guard at every persist
|
||
gate". That described the intended end state in the present tense. Door A — the
|
||
only door shipped — is ungated: the package has zero runtime dependencies and
|
||
calls no guard function before writing to disk. Both places now say so plainly,
|
||
and state that gating external or untrusted content is the caller's
|
||
responsibility (`okf.import_bundle`, or `prepare_input`/`screen_output`) until
|
||
the persist gates land with Doors B and C.
|
||
|
||
No behavior changed in this release. The correction is published because a
|
||
consumer read the earlier wording as safe-by-default and would have persisted
|
||
ungated content on that basis.
|
||
|
||
## [0.3.0] — 2026-07-17
|
||
|
||
### Added
|
||
|
||
- **Stable machine-readable error codes.** `IngestError` gained a keyword-only
|
||
`code` attribute (default `"unspecified"`). Roughly 24 codes are documented in
|
||
the `errors.py` docstrings, and those docstrings are the stability contract.
|
||
Consumers should assert on `exc.value.code`, not on message text.
|
||
|
||
### Changed
|
||
|
||
- **Exception message text is explicitly declared unstable.** It may change in any
|
||
release. Tests matching on message strings (`pytest.raises(match=...)`) should
|
||
migrate to code comparisons.
|
||
|
||
### Notes
|
||
|
||
- Generic schema shape violations deliberately share the single code
|
||
`manifest_schema`. One code per validation rule would have frozen an
|
||
unnecessarily large surface. Finer resolution is a separate decision, not an
|
||
assumed requirement.
|
||
|
||
## [0.2.0] — 2026-07-17
|
||
|
||
### Added
|
||
|
||
- **PEP 561 support.** The `py.typed` marker ships with the package, so consumers
|
||
get the inline type hints without a mypy override.
|
||
- **`IngestResult.stamp`** exposes the spec §5 provenance stamp on the result
|
||
object.
|
||
|
||
## [0.1.0] — 2026-07-16
|
||
|
||
Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
|
||
`portfolio-optimiser-commons`. Never tagged; consumers pinned the commit
|
||
`dae0bd1a` directly.
|
||
|
||
### Added
|
||
|
||
- Fail-fast manifest validation (spec §3–§4).
|
||
- Spec §5 body renderers as pure functions.
|
||
- The `file` connector (CSV, fail-closed path boundary), the `sql` connector
|
||
(read-only sqlite, env-resolved credentials), and the `http` connector behind an
|
||
explicit per-run network opt-in.
|
||
- Spec §5 materialization with an in-memory staging collision gate.
|
||
- Index maintenance on re-materialization (spec §6).
|
||
- The spec §11 golden fixtures, compared byte-for-byte.
|
||
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
|
||
rooted in `IngestError`.
|
||
|
||
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
|
||
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
|
||
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1
|
||
[0.3.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.2.0...v0.3.0
|
||
[0.2.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/dae0bd1a...v0.2.0
|
||
[0.1.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/src/commit/dae0bd1a
|