docs(extract): the extra now ships an office converter

The README told consumers that `docx` and `xlsx` ship no parser and always fail
fast. True when written; false the moment the converter seam landed -- and
false SILENTLY, because prose has no test. This repository has been bitten by
that exact shape before: a published guarantee is a test obligation.

So the correction comes with `tests/test_docs_promises.py`, which compares the
README's declared format list against the registries it describes and fails on
a format added without touching the README, on the old claim reappearing in any
wording, on an unmeasured row going unnamed, and on the exclusions being
dropped. Negative control: removing one format from the README's marker turns
it red.

The README now states which rows are measured and which are not. Three of the
five office rows have denominator ZERO in the corpus -- they work by
construction and have never met a document anyone wrote. They are not known to
be broken and not known to be right, and a reader should not have to open the
source to learn which.

The CHANGELOG's shipped entry is left as written, because a changelog records
what a release did; the correction is stated at that line instead so a reader
arriving there is not misled.

Suite 913 -> 917.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 14:15:55 +02:00
commit 0170c526ad
3 changed files with 159 additions and 12 deletions

View file

@ -9,6 +9,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **Five office formats through a vendored converter**, behind the same
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
converter binary travels inside the wheel and is resolved by path rather than
found on `PATH`, with its version asserted against a pin -- `pypandoc`
searches `PATH` first and takes the highest version it finds, so a vendored
binary buys nothing until something resolves it explicitly. A host carrying a
different converter is refused rather than silently used.
**Two of the five rows are measured; three are not.** The corpus this work
was measured on contains zero `pptx`, `odt` and `rtf` files, so those rows
work by construction and have never met a document anyone wrote. The
distinction is asserted in the suite, not left in a comment.
`.doc` (Word 97) stays out -- the converter does not read it. Drawn content
does not survive extraction in any format, and every conversion warns about
it.
- **Four error codes** for the converter path: `extractor_binary_missing`,
`extractor_binary_version`, `extractor_convert_error` and
`extractor_empty_conversion`.
- **`tests/test_docs_promises.py`**, which asserts the README's published
format list against the registries it describes. This exists because the
README's previous promise -- that `docx` and `xlsx` always fail fast -- went
false silently when the converter landed. A published guarantee is a test
obligation.
- **Door B extracts `pdf` behind the optional `[extract]` extra.** The extra is
populated for the first time, with one parser: `pdfplumber>=0.11.10,<0.12`
(MIT). The default install is unchanged — still exactly one runtime
@ -24,8 +52,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
this package is MIT and an optional extra must not hand a consumer copyleft
they did not choose.
**`docx` and `xlsx` are unchanged.** They ship no parser and still fail fast
with `extractor_extra_missing`, so the extra names exactly what it delivers.
**`docx` and `xlsx` were unchanged at this release**, shipping no parser and
failing fast with `extractor_extra_missing`. That is no longer true: a later
release added a vendored converter reaching five office formats. This entry
is left as written — a changelog records what a release did — and the
correction is stated here so a reader arriving at this line is not misled.
- **`ExtractionWarning`**, exported from the package. Every `pdf` extraction
emits one. Text extraction recovers text; anything a PDF *draws* — figures,

View file

@ -11,10 +11,11 @@ gate, with `llm_ingestion_okf.guard_adapter` wiring that gate to the real
guard (see below). Phase 3 makes the bundle contract configurable, so types,
layers, frontmatter sets, index shape, and reserved-file policy are carried by
a profile rather than by constants (see [Upstream OKF
versions](#upstream-okf-versions)). Binary extraction is partial: `pdf` is
implemented behind the optional `[extract]` extra, while `docx`/`xlsx` remain
unimplemented and are rejected fail-fast. Phase 4 (the Node half) is planned
(see `docs/plan/`).
versions](#upstream-okf-versions)). Binary extraction runs behind the
optional `[extract]` extra: `pdf` through a PDF parser, and five office
formats through a vendored document converter. Three of those five office
rows are **unmeasured** — see [Binary extraction](#binary-extraction). Phase 4
(the Node half) is planned (see `docs/plan/`).
## Install
@ -64,9 +65,11 @@ bundle:
2. **Bundle inbox.** A drop directory where common file types are converted
to OKF concept files. All file-type→text extraction lives in this library:
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
`pdf` requires the optional `[extract]` extra and is rejected fail-fast
without it; `docx` and `xlsx` ship no parser yet and are always rejected.
Extracted text passes the security gate before anything is persisted.
`pdf` and the five office formats (`docx`, `xlsx`, `pptx`, `odt`, `rtf`)
require the optional `[extract]` extra and are rejected fail-fast without
it. Extracted text passes the security gate before anything is persisted.
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
3. **External bundle import.** Import and merge of third-party OKF bundles:
each concept is assessed via the security gate, and only concepts that
pass are merged, materialized, and linked into the index.
@ -277,9 +280,38 @@ llm-ingestion-guard<2.0,>=1.2`, measured 2026-08-23 — and resolves normally
once the package index exists. A wheel built from a *tag* carries that tag's
range instead, which is why the install commands pair tag with tag.
The optional `[extract]` extra ships one parser, `pdfplumber` (MIT), for `pdf`;
`docx`/`xlsx` are still unimplemented. It is opt-in because it pulls binary
wheels (`pillow`, `pypdfium2`), which the default install must never do.
### Binary extraction
The optional `[extract]` extra ships two things: `pdfplumber` (MIT) for `pdf`,
and `pypandoc-binary` for five office formats. It is opt-in because it pulls
binary wheels, which the default install must never do — the single runtime
dependency rule covers the default install and this extra sits outside it.
The converter **binary travels inside the wheel** and is resolved by path
rather than found on `PATH`, with its version asserted against a pin. A host
carrying a different converter is refused, not silently used: extraction is
deterministic within a converter version and not across one.
| Format | Reader | Evidence |
|---|---|---|
| `pdf` | `pdfplumber` | measured |
| `docx` | converter | measured |
| `xlsx` | converter | measured |
| `pptx` | converter | **unmeasured** |
| `odt` | converter | **unmeasured** |
| `rtf` | converter | **unmeasured** |
**`unmeasured` means what it says.** The corpus this work was measured on
contains **zero** `pptx`, `odt` and `rtf` files, so those three rows work by
construction and have never been checked against a document anyone wrote.
They are not known to be broken; they are not known to be right either, and
the distinction is the point.
**What stays out.** `.doc` (Word 97) is not supported — the converter does not
read it. Rastered or scanned PDFs are refused rather than persisted as empty
concepts, because this library does not do OCR. Drawn content — figures,
diagrams, shapes — does not survive extraction in any format here, and every
extraction says so with a warning. Structured table recovery is out of scope.
Request it by appending `[extract]` to the package name in whichever install
command from [Install](#install) you are using — this package is not on an

View file

@ -0,0 +1,84 @@
"""The published format promise, asserted rather than trusted.
`README.md` told consumers that `docx` and `xlsx` ship no parser and always
fail fast. That was true when it was written and became false the moment the
converter seam landed -- silently, because prose has no test.
This library already learned that lesson once: a published promise without a
test goes false without anyone noticing, and a guarantee made publicly is a
test obligation. So the README's claimed format list is compared against the
registries it describes. Adding a format without touching the README, or
describing one that does not exist, fails here.
"""
from __future__ import annotations
import re
from pathlib import Path
from llm_ingestion_okf.extract import (
_CORE_EXTRACTORS,
_EVIDENCE,
_OPTIONAL_EXTRACTORS,
_PANDOC_FORMATS,
)
PROJECT_ROOT = Path(__file__).resolve().parents[1]
README = PROJECT_ROOT / "README.md"
# The line the README carries, and the one place this list is written in prose.
_FORMAT_LINE = re.compile(r"^<!-- extract-formats: (.+) -->$", re.MULTILINE)
def _declared_formats() -> set[str]:
match = _FORMAT_LINE.search(README.read_text(encoding="utf-8"))
assert match is not None, (
"README.md carries no `<!-- extract-formats: ... -->` marker; without "
"it this test cannot check the promise and the promise can drift"
)
return {token.strip() for token in match.group(1).split(",")}
def test_the_readme_names_exactly_the_formats_that_exist() -> None:
assert _declared_formats() == set(_CORE_EXTRACTORS) | set(_OPTIONAL_EXTRACTORS)
def test_the_readme_no_longer_claims_docx_and_xlsx_fail_fast() -> None:
"""The specific false sentence, pinned so it cannot come back.
Written as a search for the claim rather than for its exact wording: the
sentence could be rephrased and stay just as wrong.
"""
text = README.read_text(encoding="utf-8").lower()
for claim in (
"docx` and `xlsx` ship no parser",
"docx`/`xlsx` remain\nunimplemented",
"docx`/`xlsx` are still unimplemented",
):
assert claim.lower() not in text, f"README still claims: {claim}"
def test_the_readme_states_which_rows_are_unmeasured() -> None:
"""An unmeasured row must not read as a supported one.
Three of the five office formats have denominator ZERO in the corpus this
work was measured on. A consumer reading the README should be able to see
that without reading the source.
"""
text = README.read_text(encoding="utf-8")
unmeasured = {s.lstrip(".") for s, e in _EVIDENCE.items() if e == "unmeasured"}
assert unmeasured, "the evidence table lists no unmeasured rows"
for suffix in unmeasured:
assert suffix in text, f"README does not mention the unmeasured row {suffix}"
assert "unmeasured" in text.lower()
def test_the_readme_still_states_what_stays_out() -> None:
"""`.doc` (Word 97) and rastered PDFs are out, and stay named.
A format list that grows without also saying what it excludes reads as a
promise to handle anything office-shaped.
"""
text = README.read_text(encoding="utf-8")
assert ".doc`" in text or "Word 97" in text
assert ".doc" not in set(_PANDOC_FORMATS)