Kjell Tore Guttormsen
658b7aafe0
Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.
The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.
`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.
The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.
Three facts from that measurement are now carried in code rather than in a
report:
- Extracted text is pinned to an exact transitive parser version
(pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
contract). tests/test_extract.py freezes the expected text of a committed
hand-written fixture so a parser upgrade breaks something visible instead of
drifting silently. Reasoning at the declaration site and in
tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
new `ExtractionWarning`: figures have no text to recover, so a bundle built
from drawn documents is incomplete by construction. Stated categorically
rather than detected — deciding "is there a figure here" is the layout
heuristic G2b declined.
Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).
Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.
Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
53 lines
2.5 KiB
Markdown
53 lines
2.5 KiB
Markdown
# Test fixtures
|
|
|
|
## The PDF fixtures
|
|
|
|
`two-line-krav.pdf` and `no-text-layer.pdf` are hand-written minimal PDFs,
|
|
regenerated by `make_fixtures.py` in this directory:
|
|
|
|
```
|
|
python3 tests/fixtures/make_fixtures.py
|
|
```
|
|
|
|
They carry no library's output — the objects are laid out by hand and the xref
|
|
offsets computed from the emitted bytes — so they are auditable byte for byte
|
|
and reproducible from that one file.
|
|
|
|
| Fixture | What it is for |
|
|
|---|---|
|
|
| `two-line-krav.pdf` | One heading plus one requirement row with label and value on the **same line**. That pairing is the property `pdfplumber` was chosen for. |
|
|
| `no-text-layer.pdf` | A structurally valid page with no text operators — the shape a scanned or image-only PDF presents. Must fail fast (`extractor_empty_pdf`), never persist as an empty concept. |
|
|
|
|
## Why the expected PDF text is frozen as a literal
|
|
|
|
`tests/test_extract.py` asserts the extracted text of `two-line-krav.pdf` as an
|
|
exact string. That is deliberate, and it is the mechanism behind a promise this
|
|
library makes everywhere else:
|
|
|
|
- Extraction is deterministic **within** a parser version. Measured 2026-08-21
|
|
across five configurations, two runs each, compared byte for byte
|
|
(`docs/2026-08-21-g2-pdf-extraction-measurement.md`).
|
|
- Extraction is **not** guaranteed stable **across** parser versions.
|
|
`pdfplumber` pins `pdfminer.six==20260107` exactly, and `pdfminer.six` ships
|
|
date-stamped releases with no stability contract. So the real pin on extracted
|
|
text is a transitive one, and it is exact.
|
|
|
|
The consequence is worth stating plainly: **any golden fixture built on
|
|
extracted PDF text is pinned to an exact parser version, and a parser upgrade
|
|
is a fixture migration, not a routine bump.** The frozen literal is what makes
|
|
that upgrade break something visible instead of drifting silently. If it goes
|
|
red after a dependency change, the correct response is to read the diff and
|
|
decide, not to re-record the expectation.
|
|
|
|
The version range that carries this lives in `pyproject.toml`'s
|
|
`[project.optional-dependencies] extract`, with the same reasoning at the
|
|
declaration site.
|
|
|
|
## What these fixtures do not cover
|
|
|
|
Structured table recovery. Measured on real Vegnormalene, only 45 of 196
|
|
detected table objects are clean enough to hand to `render_table` unchanged;
|
|
two independent parsers return the same wrong shape, because the breakage is in
|
|
the documents' ruling geometry rather than in either library. PDFs enter this
|
|
library as **prose**, and structured tables are out of scope until that is
|
|
decided separately.
|