chore(release): v0.4.0
Phase 2 ships: Doors B and C, the extraction registry, the guard adapter, and with them the first and only runtime dependency this library takes. The minor bump is the news consumers are waiting on -- installing the package now installs llm-ingestion-guard>=0.2,<0.3, though importing it still does not import the guard. Two behaviour changes carried in the same release are called out in the changelog rather than buried: an extraction title containing `[` or `]` now fails at manifest load (ingest-spec section 4), and the section 3 ownership scan no longer deletes files another manifest stamped into the same bundle. uv.lock was stale -- it predated the guard dependency and could not be regenerated, because the guard is not on a package index. A [tool.uv.sources] entry pins the same tag the README documents, so `uv lock`/`uv sync` resolve. It is uv-specific dev metadata: the built wheel still carries `Requires-Dist: llm-ingestion-guard<0.3,>=0.2`, verified against the wheel, so the range remains the pin and the git URL remains a channel. Binary extraction (pdf/docx/xlsx behind [extract]) is deliberately not in this release and is stated as outstanding in both README and changelog. 425 tests green; ruff, ruff format and mypy --strict clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
This commit is contained in:
parent
0b77c58a1c
commit
524ade78a6
5 changed files with 119 additions and 10 deletions
90
CHANGELOG.md
90
CHANGELOG.md
|
|
@ -5,6 +5,94 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.4.0] — 2026-07-25
|
||||
|
||||
Phase 2. The bundle inbox (Door B) and external-bundle import (Door C) ship,
|
||||
and with them this library's first — and only ever — runtime dependency.
|
||||
|
||||
### Added
|
||||
|
||||
- **Door B — bundle inbox (`process_inbox`).** Per dropped file: bytes →
|
||||
extraction → persist gate → render → collision gate → write → index link.
|
||||
Returns an `InboxResult` whose four buckets (`persisted`, `quarantined`,
|
||||
`rejected`, `failed`) are disjoint and complete, so a file that vanished
|
||||
mid-run surfaces as a missing entry rather than as nothing. One bad file
|
||||
never aborts the run: only an invalid `ingested_at`, a reserved `okf_type`,
|
||||
and a missing inbox directory fail the whole run, each being wrong for every
|
||||
file at once. Concept files are named `inbox-{slug}.md`, disjoint from
|
||||
`index.md`, Door A's `ingest-*`, and `promoted-verdict-*`; `source_sha256`
|
||||
is taken over the original dropped bytes, so provenance stays re-verifiable
|
||||
against the operator's file.
|
||||
- **Door C — external bundle import (`import_bundle`).** Reads an external OKF
|
||||
bundle and hands it *whole* to an injected gate over the guard's
|
||||
`okf.import_bundle` — a bundle-level call, because it resolves the
|
||||
cross-link graph across concepts — merging only concepts that clear the
|
||||
non-blocking floor. Two invariants, both load-bearing: a merged concept is
|
||||
written **verbatim** (stamping it would require round-tripping frontmatter
|
||||
through this library's line-oriented parser, which cannot represent the
|
||||
block lists the guard's parser accepts, and would persist bytes the guard
|
||||
never screened), and ownership is therefore proven by **content identity** —
|
||||
an occupied target name is re-used only when the bytes there are already
|
||||
identical, never overwritten. Re-importing an unchanged bundle is a no-op.
|
||||
- **`extract_text` — the Door B extraction registry.** All file-type → text
|
||||
extraction lives in this library, because the guard is text-only. `md`/`txt`
|
||||
pass through, `csv` renders the markdown table, `json` is fenced verbatim,
|
||||
and `html`/`htm` reduce to text with `html.parser` — stdlib throughout.
|
||||
`pdf`, `docx`, and `xlsx` are gated behind the `[extract]` extra, which
|
||||
ships no parser yet: those types fail fast with a typed error naming the
|
||||
extra, never a silent skip.
|
||||
- **`llm_ingestion_okf.guard_adapter`** — the shipped gate over the real
|
||||
guard (`inbox_gate`, `import_gate`), and the only module here that imports
|
||||
it. Door B screens the exact bytes it persists: the guard's `prepare_input`
|
||||
bookend prepares text for a model call this library never makes, so
|
||||
`screen_output` alone is used and the screened string is the written string.
|
||||
It follows that the gate **refuses rather than repairs** — a file carrying
|
||||
an invisible carrier is rejected, not stripped and persisted. The policy is
|
||||
`PRESET_USER_UPLOAD`, so any finding at all is held back.
|
||||
- **15 new stable error codes**, each registered in the `errors.py` docstrings
|
||||
(the stability contract) and pinned by the error-code conformance suite:
|
||||
`extractor_decode_error`, `extractor_empty_csv`, `extractor_extra_missing`,
|
||||
`extractor_unknown`, `import_label_invalid`, `import_path_empty`,
|
||||
`import_path_too_long`, `import_provenance_invalid`,
|
||||
`import_slug_collision`, `inbox_slug_collision`, `inbox_slug_empty`,
|
||||
`inbox_slug_too_long`, `inbox_source_file_invalid`, `inbox_title_invalid`,
|
||||
`okf_type_reserved`.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`llm-ingestion-guard>=0.2,<0.3` is now a mandatory runtime dependency.**
|
||||
Installing this package installs the guard. Importing it does not: only
|
||||
`guard_adapter` imports the guard, so a Door A consumer keeps working
|
||||
whatever state the dependency is in, and the doors themselves still take an
|
||||
*injected* gate. Until the guard is published to a package index, install it
|
||||
from its tag — see the README. A packaging test enforces that this stays the
|
||||
only runtime dependency.
|
||||
- **An extraction `title` containing `[` or `]` is rejected at manifest load**
|
||||
(ingest-spec §4, ratified D1). Previously only single-line was validated. A
|
||||
manifest that loaded before and carries a bracket in a title now fails fast
|
||||
with code `manifest_schema`: the title renders verbatim into the index link
|
||||
label `- [title](target)`, where a bracket breaks index-link and navigation
|
||||
parsing downstream.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Materializing one manifest no longer deletes the files another manifest
|
||||
stamped into the same bundle.** The §3 ownership scan classified every
|
||||
ingest-stamped file as replaceable, so a second manifest sharing a bundle
|
||||
removed the first one's concept files and their index links. Ownership is
|
||||
now narrowed to files whose stamp names the running manifest by stem — the
|
||||
stem is stable across content edits, so an edited manifest still reclaims
|
||||
what a prior run of itself wrote. The operator-copy restriction (a generated
|
||||
file copied into curated content) remains documented, not enforced.
|
||||
|
||||
### Notes
|
||||
|
||||
- Phase 2's binary extraction is **not** in this release: the `[extract]`
|
||||
extra is declared but empty, and `pdf`/`docx`/`xlsx` therefore fail fast.
|
||||
That is the one outstanding item from the phase, and it ships separately.
|
||||
- Exception `__cause__` preservation is now pinned by a conformance suite, one
|
||||
test per fail-fast wrap site, alongside the existing `.code` suite.
|
||||
|
||||
## [0.3.2] — 2026-07-23
|
||||
|
||||
### Fixed
|
||||
|
|
@ -88,6 +176,8 @@ Phase 1 (Door A) implemented against the normative `ingest-spec.md` owned by
|
|||
- The Door A public surface: `materialize_bundle` plus the typed error hierarchy
|
||||
rooted in `IngestError`.
|
||||
|
||||
[0.4.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.2...v0.4.0
|
||||
[0.3.2]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.1...v0.3.2
|
||||
[0.3.1]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.3.0...v0.3.1
|
||||
[0.3.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/v0.2.0...v0.3.0
|
||||
[0.2.0]: https://git.fromaitochitta.com/open/llm-ingestion-okf/compare/dae0bd1a...v0.2.0
|
||||
|
|
|
|||
17
README.md
17
README.md
|
|
@ -2,13 +2,16 @@
|
|||
|
||||
Shared ingestion library for OKF (Open Knowledge Format) bundles.
|
||||
|
||||
Status: phase 1 (spec-based ingestion) is implemented — manifest validation,
|
||||
the `file`/`sql`/`http` connectors, deterministic materialization, index
|
||||
generation, and the golden fixture suite under `examples/`. Phase 2 is in
|
||||
progress: the bundle inbox (`process_inbox`) and external-bundle import
|
||||
(`import_bundle`) are implemented against an **injected** persist gate, and
|
||||
`llm_ingestion_okf.guard_adapter` wires that gate to the real guard (see
|
||||
below). Phases 3–4 are planned (see `docs/plan/`).
|
||||
Status: phases 1 and 2 are implemented. Phase 1 (spec-based ingestion) covers
|
||||
manifest validation, the `file`/`sql`/`http` connectors, deterministic
|
||||
materialization, index generation, and the golden fixture suite under
|
||||
`examples/`. Phase 2 adds the bundle inbox (`process_inbox`) and
|
||||
external-bundle import (`import_bundle`), both against an **injected** persist
|
||||
gate, with `llm_ingestion_okf.guard_adapter` wiring that gate to the real
|
||||
guard (see below). One phase-2 item is deliberately outstanding: binary
|
||||
extraction (`pdf`/`docx`/`xlsx` behind the `[extract]` extra) is unimplemented,
|
||||
so those types are rejected fail-fast. Phases 3–4 are planned (see
|
||||
`docs/plan/`).
|
||||
|
||||
## Planned scope (v1)
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "llm-ingestion-okf"
|
||||
version = "0.3.2"
|
||||
version = "0.4.0"
|
||||
description = "Shared OKF (Open Knowledge Format) ingestion library: spec-based connectors, bundle inbox, and external-bundle import, with security delegated to llm-ingestion-guard."
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
|
|
@ -49,3 +49,10 @@ python_version = "3.10"
|
|||
[[tool.mypy.overrides]]
|
||||
module = ["llm_ingestion_guard", "llm_ingestion_guard.*"]
|
||||
ignore_missing_imports = true
|
||||
|
||||
# Development-time install CHANNEL for the guard, which is not on a package
|
||||
# index yet. It resolves `uv lock`/`uv sync` against the tag; it is uv-specific
|
||||
# and never reaches consumers — the built wheel carries the range from
|
||||
# [project.dependencies] above, which is the pin.
|
||||
[tool.uv.sources]
|
||||
llm-ingestion-guard = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git", tag = "v0.2.0" }
|
||||
|
|
|
|||
|
|
@ -73,7 +73,7 @@ from .manifest import (
|
|||
)
|
||||
from .materialize import IngestResult, materialize_bundle
|
||||
|
||||
__version__ = "0.3.2"
|
||||
__version__ = "0.4.0"
|
||||
|
||||
__all__ = [
|
||||
"BlockedFile",
|
||||
|
|
|
|||
11
uv.lock
generated
11
uv.lock
generated
|
|
@ -164,10 +164,18 @@ wheels = [
|
|||
{ url = "https://files.pythonhosted.org/packages/5f/5d/3dcec2884ba1b0806d1408612555c38dd5d68e90156b59f75f6e36435c3a/librt-0.13.0-cp314-cp314t-win_arm64.whl", hash = "sha256:2f281549a4c52ac7bb97997f14353f8bd0e53a34ca0dad1c905cfd0b4a58ae99", size = 110771, upload-time = "2026-07-08T12:26:12.303Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "llm-ingestion-guard"
|
||||
version = "0.2.0"
|
||||
source = { git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git?tag=v0.2.0#542ac92349cd0015972105ba66017b36cce1e708" }
|
||||
|
||||
[[package]]
|
||||
name = "llm-ingestion-okf"
|
||||
version = "0.3.2"
|
||||
version = "0.4.0"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "llm-ingestion-guard" },
|
||||
]
|
||||
|
||||
[package.dev-dependencies]
|
||||
dev = [
|
||||
|
|
@ -177,6 +185,7 @@ dev = [
|
|||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [{ name = "llm-ingestion-guard", git = "https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git?tag=v0.2.0" }]
|
||||
provides-extras = ["extract"]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue