History

Kjell Tore Guttormsen 658b7aafe0 feat(extract): implement pdf behind the [extract] extra with pdfplumber Order G2a. Populates the optional `[extract]` extra for the first time with one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it. The default install is untouched: exactly one runtime dependency, stdlib otherwise, enforced by test_packaging.py. The gate for `pdf` becomes an import probe rather than a frozenset membership test, exactly as extract.py's docstring had promised. The rejection does not change: without the extra, `pdf` still raises `extractor_extra_missing` with the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules monkeypatch, so it holds on machines where the parser is installed too — a skip would have preserved nothing there. Verified in a clean venv without the extra: 589 passed, 7 skipped; with it, 596 passed. `docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what it ships. The parser choice was forced by measurement, not preference (`b73dd9d`, docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf, pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on licence (AGPL-3.0), which an MIT package must not push onto a consumer. Three facts from that measurement are now carried in code rather than in a report: - Extracted text is pinned to an exact transitive parser version (pdfplumber pins pdfminer.six==20260107; date-stamped, no stability contract). tests/test_extract.py freezes the expected text of a committed hand-written fixture so a parser upgrade breaks something visible instead of drifting silently. Reasoning at the declaration site and in tests/fixtures/README.md. - Determinism within a version is now held by a test, not only measured once. - Drawn content does not survive extraction. Every pdf extraction emits the new `ExtractionWarning`: figures have no text to recover, so a bundle built from drawn documents is incomplete by construction. Stated categorically rather than detected — deciding "is there a figure here" is the layout heuristic G2b declined. Two new error codes, both mirroring existing patterns: `extractor_empty_pdf` (a scanned/image-only PDF, refused rather than persisted as an empty concept) and `extractor_pdf_error` (parser failure wrapped, never leaked). Structured table recovery (G2b) is NOT implemented and is documented as out of scope: two independent parsers return the same wrong shape, so the breakage is document geometry, not a library choice. PDFs enter as prose. Also corrects an install promise this change would otherwise have published: the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'` as working, because the package is not on an index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z		2026-08-21 20:22:39 +02:00
..
make_fixtures.py	feat(extract): implement pdf behind the [extract] extra with pdfplumber	2026-08-21 20:22:39 +02:00
no-text-layer.pdf	feat(extract): implement pdf behind the [extract] extra with pdfplumber	2026-08-21 20:22:39 +02:00
README.md	feat(extract): implement pdf behind the [extract] extra with pdfplumber	2026-08-21 20:22:39 +02:00
two-line-krav.pdf	feat(extract): implement pdf behind the [extract] extra with pdfplumber	2026-08-21 20:22:39 +02:00

README.md

Test fixtures

The PDF fixtures

two-line-krav.pdf and no-text-layer.pdf are hand-written minimal PDFs, regenerated by make_fixtures.py in this directory:

python3 tests/fixtures/make_fixtures.py

They carry no library's output — the objects are laid out by hand and the xref offsets computed from the emitted bytes — so they are auditable byte for byte and reproducible from that one file.

Fixture	What it is for
`two-line-krav.pdf`	One heading plus one requirement row with label and value on the same line. That pairing is the property `pdfplumber` was chosen for.
`no-text-layer.pdf`	A structurally valid page with no text operators — the shape a scanned or image-only PDF presents. Must fail fast (`extractor_empty_pdf`), never persist as an empty concept.

Why the expected PDF text is frozen as a literal

tests/test_extract.py asserts the extracted text of two-line-krav.pdf as an exact string. That is deliberate, and it is the mechanism behind a promise this library makes everywhere else:

Extraction is deterministic within a parser version. Measured 2026-08-21 across five configurations, two runs each, compared byte for byte (docs/2026-08-21-g2-pdf-extraction-measurement.md).
Extraction is not guaranteed stable across parser versions. pdfplumber pins pdfminer.six==20260107 exactly, and pdfminer.six ships date-stamped releases with no stability contract. So the real pin on extracted text is a transitive one, and it is exact.

The consequence is worth stating plainly: any golden fixture built on extracted PDF text is pinned to an exact parser version, and a parser upgrade is a fixture migration, not a routine bump. The frozen literal is what makes that upgrade break something visible instead of drifting silently. If it goes red after a dependency change, the correct response is to read the diff and decide, not to re-record the expectation.

The version range that carries this lives in pyproject.toml's [project.optional-dependencies] extract, with the same reasoning at the declaration site.

What these fixtures do not cover

Structured table recovery. Measured on real Vegnormalene, only 45 of 196 detected table objects are clean enough to hand to render_table unchanged; two independent parsers return the same wrong shape, because the breakage is in the documents' ruling geometry rather than in either library. PDFs enter this library as prose, and structured tables are out of scope until that is decided separately.