docs(extract): the extra now ships an office converter
The README told consumers that `docx` and `xlsx` ship no parser and always fail fast. True when written; false the moment the converter seam landed -- and false SILENTLY, because prose has no test. This repository has been bitten by that exact shape before: a published guarantee is a test obligation. So the correction comes with `tests/test_docs_promises.py`, which compares the README's declared format list against the registries it describes and fails on a format added without touching the README, on the old claim reappearing in any wording, on an unmeasured row going unnamed, and on the exclusions being dropped. Negative control: removing one format from the README's marker turns it red. The README now states which rows are measured and which are not. Three of the five office rows have denominator ZERO in the corpus -- they work by construction and have never met a document anyone wrote. They are not known to be broken and not known to be right, and a reader should not have to open the source to learn which. The CHANGELOG's shipped entry is left as written, because a changelog records what a release did; the correction is stated at that line instead so a reader arriving there is not misled. Suite 913 -> 917. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
66a44f173b
commit
0170c526ad
3 changed files with 159 additions and 12 deletions
35
CHANGELOG.md
35
CHANGELOG.md
|
|
@ -9,6 +9,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
### Added
|
||||
|
||||
- **Five office formats through a vendored converter**, behind the same
|
||||
optional `[extract]` extra: `docx`, `xlsx`, `pptx`, `odt`, `rtf`. The
|
||||
converter binary travels inside the wheel and is resolved by path rather than
|
||||
found on `PATH`, with its version asserted against a pin -- `pypandoc`
|
||||
searches `PATH` first and takes the highest version it finds, so a vendored
|
||||
binary buys nothing until something resolves it explicitly. A host carrying a
|
||||
different converter is refused rather than silently used.
|
||||
|
||||
**Two of the five rows are measured; three are not.** The corpus this work
|
||||
was measured on contains zero `pptx`, `odt` and `rtf` files, so those rows
|
||||
work by construction and have never met a document anyone wrote. The
|
||||
distinction is asserted in the suite, not left in a comment.
|
||||
|
||||
`.doc` (Word 97) stays out -- the converter does not read it. Drawn content
|
||||
does not survive extraction in any format, and every conversion warns about
|
||||
it.
|
||||
|
||||
- **Four error codes** for the converter path: `extractor_binary_missing`,
|
||||
`extractor_binary_version`, `extractor_convert_error` and
|
||||
`extractor_empty_conversion`.
|
||||
|
||||
- **`tests/test_docs_promises.py`**, which asserts the README's published
|
||||
format list against the registries it describes. This exists because the
|
||||
README's previous promise -- that `docx` and `xlsx` always fail fast -- went
|
||||
false silently when the converter landed. A published guarantee is a test
|
||||
obligation.
|
||||
|
||||
|
||||
- **Door B extracts `pdf` behind the optional `[extract]` extra.** The extra is
|
||||
populated for the first time, with one parser: `pdfplumber>=0.11.10,<0.12`
|
||||
(MIT). The default install is unchanged — still exactly one runtime
|
||||
|
|
@ -24,8 +52,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
this package is MIT and an optional extra must not hand a consumer copyleft
|
||||
they did not choose.
|
||||
|
||||
**`docx` and `xlsx` are unchanged.** They ship no parser and still fail fast
|
||||
with `extractor_extra_missing`, so the extra names exactly what it delivers.
|
||||
**`docx` and `xlsx` were unchanged at this release**, shipping no parser and
|
||||
failing fast with `extractor_extra_missing`. That is no longer true: a later
|
||||
release added a vendored converter reaching five office formats. This entry
|
||||
is left as written — a changelog records what a release did — and the
|
||||
correction is stated here so a reader arriving at this line is not misled.
|
||||
|
||||
- **`ExtractionWarning`**, exported from the package. Every `pdf` extraction
|
||||
emits one. Text extraction recovers text; anything a PDF *draws* — figures,
|
||||
|
|
|
|||
52
README.md
52
README.md
|
|
@ -11,10 +11,11 @@ gate, with `llm_ingestion_okf.guard_adapter` wiring that gate to the real
|
|||
guard (see below). Phase 3 makes the bundle contract configurable, so types,
|
||||
layers, frontmatter sets, index shape, and reserved-file policy are carried by
|
||||
a profile rather than by constants (see [Upstream OKF
|
||||
versions](#upstream-okf-versions)). Binary extraction is partial: `pdf` is
|
||||
implemented behind the optional `[extract]` extra, while `docx`/`xlsx` remain
|
||||
unimplemented and are rejected fail-fast. Phase 4 (the Node half) is planned
|
||||
(see `docs/plan/`).
|
||||
versions](#upstream-okf-versions)). Binary extraction runs behind the
|
||||
optional `[extract]` extra: `pdf` through a PDF parser, and five office
|
||||
formats through a vendored document converter. Three of those five office
|
||||
rows are **unmeasured** — see [Binary extraction](#binary-extraction). Phase 4
|
||||
(the Node half) is planned (see `docs/plan/`).
|
||||
|
||||
## Install
|
||||
|
||||
|
|
@ -64,9 +65,11 @@ bundle:
|
|||
2. **Bundle inbox.** A drop directory where common file types are converted
|
||||
to OKF concept files. All file-type→text extraction lives in this library:
|
||||
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
|
||||
`pdf` requires the optional `[extract]` extra and is rejected fail-fast
|
||||
without it; `docx` and `xlsx` ship no parser yet and are always rejected.
|
||||
Extracted text passes the security gate before anything is persisted.
|
||||
`pdf` and the five office formats (`docx`, `xlsx`, `pptx`, `odt`, `rtf`)
|
||||
require the optional `[extract]` extra and are rejected fail-fast without
|
||||
it. Extracted text passes the security gate before anything is persisted.
|
||||
|
||||
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
|
||||
3. **External bundle import.** Import and merge of third-party OKF bundles:
|
||||
each concept is assessed via the security gate, and only concepts that
|
||||
pass are merged, materialized, and linked into the index.
|
||||
|
|
@ -277,9 +280,38 @@ llm-ingestion-guard<2.0,>=1.2`, measured 2026-08-23 — and resolves normally
|
|||
once the package index exists. A wheel built from a *tag* carries that tag's
|
||||
range instead, which is why the install commands pair tag with tag.
|
||||
|
||||
The optional `[extract]` extra ships one parser, `pdfplumber` (MIT), for `pdf`;
|
||||
`docx`/`xlsx` are still unimplemented. It is opt-in because it pulls binary
|
||||
wheels (`pillow`, `pypdfium2`), which the default install must never do.
|
||||
### Binary extraction
|
||||
|
||||
The optional `[extract]` extra ships two things: `pdfplumber` (MIT) for `pdf`,
|
||||
and `pypandoc-binary` for five office formats. It is opt-in because it pulls
|
||||
binary wheels, which the default install must never do — the single runtime
|
||||
dependency rule covers the default install and this extra sits outside it.
|
||||
|
||||
The converter **binary travels inside the wheel** and is resolved by path
|
||||
rather than found on `PATH`, with its version asserted against a pin. A host
|
||||
carrying a different converter is refused, not silently used: extraction is
|
||||
deterministic within a converter version and not across one.
|
||||
|
||||
| Format | Reader | Evidence |
|
||||
|---|---|---|
|
||||
| `pdf` | `pdfplumber` | measured |
|
||||
| `docx` | converter | measured |
|
||||
| `xlsx` | converter | measured |
|
||||
| `pptx` | converter | **unmeasured** |
|
||||
| `odt` | converter | **unmeasured** |
|
||||
| `rtf` | converter | **unmeasured** |
|
||||
|
||||
**`unmeasured` means what it says.** The corpus this work was measured on
|
||||
contains **zero** `pptx`, `odt` and `rtf` files, so those three rows work by
|
||||
construction and have never been checked against a document anyone wrote.
|
||||
They are not known to be broken; they are not known to be right either, and
|
||||
the distinction is the point.
|
||||
|
||||
**What stays out.** `.doc` (Word 97) is not supported — the converter does not
|
||||
read it. Rastered or scanned PDFs are refused rather than persisted as empty
|
||||
concepts, because this library does not do OCR. Drawn content — figures,
|
||||
diagrams, shapes — does not survive extraction in any format here, and every
|
||||
extraction says so with a warning. Structured table recovery is out of scope.
|
||||
|
||||
Request it by appending `[extract]` to the package name in whichever install
|
||||
command from [Install](#install) you are using — this package is not on an
|
||||
|
|
|
|||
84
tests/test_docs_promises.py
Normal file
84
tests/test_docs_promises.py
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
"""The published format promise, asserted rather than trusted.
|
||||
|
||||
`README.md` told consumers that `docx` and `xlsx` ship no parser and always
|
||||
fail fast. That was true when it was written and became false the moment the
|
||||
converter seam landed -- silently, because prose has no test.
|
||||
|
||||
This library already learned that lesson once: a published promise without a
|
||||
test goes false without anyone noticing, and a guarantee made publicly is a
|
||||
test obligation. So the README's claimed format list is compared against the
|
||||
registries it describes. Adding a format without touching the README, or
|
||||
describing one that does not exist, fails here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from llm_ingestion_okf.extract import (
|
||||
_CORE_EXTRACTORS,
|
||||
_EVIDENCE,
|
||||
_OPTIONAL_EXTRACTORS,
|
||||
_PANDOC_FORMATS,
|
||||
)
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
README = PROJECT_ROOT / "README.md"
|
||||
|
||||
# The line the README carries, and the one place this list is written in prose.
|
||||
_FORMAT_LINE = re.compile(r"^<!-- extract-formats: (.+) -->$", re.MULTILINE)
|
||||
|
||||
|
||||
def _declared_formats() -> set[str]:
|
||||
match = _FORMAT_LINE.search(README.read_text(encoding="utf-8"))
|
||||
assert match is not None, (
|
||||
"README.md carries no `<!-- extract-formats: ... -->` marker; without "
|
||||
"it this test cannot check the promise and the promise can drift"
|
||||
)
|
||||
return {token.strip() for token in match.group(1).split(",")}
|
||||
|
||||
|
||||
def test_the_readme_names_exactly_the_formats_that_exist() -> None:
|
||||
assert _declared_formats() == set(_CORE_EXTRACTORS) | set(_OPTIONAL_EXTRACTORS)
|
||||
|
||||
|
||||
def test_the_readme_no_longer_claims_docx_and_xlsx_fail_fast() -> None:
|
||||
"""The specific false sentence, pinned so it cannot come back.
|
||||
|
||||
Written as a search for the claim rather than for its exact wording: the
|
||||
sentence could be rephrased and stay just as wrong.
|
||||
"""
|
||||
text = README.read_text(encoding="utf-8").lower()
|
||||
for claim in (
|
||||
"docx` and `xlsx` ship no parser",
|
||||
"docx`/`xlsx` remain\nunimplemented",
|
||||
"docx`/`xlsx` are still unimplemented",
|
||||
):
|
||||
assert claim.lower() not in text, f"README still claims: {claim}"
|
||||
|
||||
|
||||
def test_the_readme_states_which_rows_are_unmeasured() -> None:
|
||||
"""An unmeasured row must not read as a supported one.
|
||||
|
||||
Three of the five office formats have denominator ZERO in the corpus this
|
||||
work was measured on. A consumer reading the README should be able to see
|
||||
that without reading the source.
|
||||
"""
|
||||
text = README.read_text(encoding="utf-8")
|
||||
unmeasured = {s.lstrip(".") for s, e in _EVIDENCE.items() if e == "unmeasured"}
|
||||
assert unmeasured, "the evidence table lists no unmeasured rows"
|
||||
for suffix in unmeasured:
|
||||
assert suffix in text, f"README does not mention the unmeasured row {suffix}"
|
||||
assert "unmeasured" in text.lower()
|
||||
|
||||
|
||||
def test_the_readme_still_states_what_stays_out() -> None:
|
||||
"""`.doc` (Word 97) and rastered PDFs are out, and stay named.
|
||||
|
||||
A format list that grows without also saying what it excludes reads as a
|
||||
promise to handle anything office-shaped.
|
||||
"""
|
||||
text = README.read_text(encoding="utf-8")
|
||||
assert ".doc`" in text or "Word 97" in text
|
||||
assert ".doc" not in set(_PANDOC_FORMATS)
|
||||
Loading…
Add table
Add a link
Reference in a new issue