feat(extract): implement pdf behind the [extract] extra with pdfplumber
Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.
The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.
`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.
The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.
Three facts from that measurement are now carried in code rather than in a
report:
- Extracted text is pinned to an exact transitive parser version
(pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
contract). tests/test_extract.py freezes the expected text of a committed
hand-written fixture so a parser upgrade breaks something visible instead of
drifting silently. Reasoning at the declaration site and in
tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
new `ExtractionWarning`: figures have no text to recover, so a bundle built
from drawn documents is incomplete by construction. Stated categorically
rather than detected — deciding "is there a figure here" is the layout
heuristic G2b declined.
Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).
Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.
Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
This commit is contained in:
parent
b73dd9d6a4
commit
658b7aafe0
14 changed files with 1046 additions and 37 deletions
44
README.md
44
README.md
|
|
@ -11,10 +11,10 @@ gate, with `llm_ingestion_okf.guard_adapter` wiring that gate to the real
|
|||
guard (see below). Phase 3 makes the bundle contract configurable, so types,
|
||||
layers, frontmatter sets, index shape, and reserved-file policy are carried by
|
||||
a profile rather than by constants (see [Upstream OKF
|
||||
versions](#upstream-okf-versions)). One phase-2 item is deliberately
|
||||
outstanding: binary extraction (`pdf`/`docx`/`xlsx` behind the `[extract]`
|
||||
extra) is unimplemented, so those types are rejected fail-fast. Phase 4 (the
|
||||
Node half) is planned (see `docs/plan/`).
|
||||
versions](#upstream-okf-versions)). Binary extraction is partial: `pdf` is
|
||||
implemented behind the optional `[extract]` extra, while `docx`/`xlsx` remain
|
||||
unimplemented and are rejected fail-fast. Phase 4 (the Node half) is planned
|
||||
(see `docs/plan/`).
|
||||
|
||||
## Install
|
||||
|
||||
|
|
@ -64,9 +64,9 @@ bundle:
|
|||
2. **Bundle inbox.** A drop directory where common file types are converted
|
||||
to OKF concept files. All file-type→text extraction lives in this library:
|
||||
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
|
||||
`pdf`, `docx`, and `xlsx` require the optional `[extract]` extra and are
|
||||
rejected fail-fast without it. Extracted text passes the security gate
|
||||
before anything is persisted.
|
||||
`pdf` requires the optional `[extract]` extra and is rejected fail-fast
|
||||
without it; `docx` and `xlsx` ship no parser yet and are always rejected.
|
||||
Extracted text passes the security gate before anything is persisted.
|
||||
3. **External bundle import.** Import and merge of third-party OKF bundles:
|
||||
each concept is assessed via the security gate, and only concepts that
|
||||
pass are merged, materialized, and linked into the index.
|
||||
|
|
@ -258,9 +258,33 @@ install-time *channel*, not the pin: the range above stays the declared
|
|||
dependency — a wheel built from this branch carries `Requires-Dist:
|
||||
llm-ingestion-guard<0.4,>=0.3`, measured 2026-08-10 — and resolves normally
|
||||
once the package index exists. A wheel built from a *tag* carries that tag's
|
||||
range instead, which is why the install commands pair tag with tag. The optional `[extract]` extra (pdf/docx/xlsx parsers) is not
|
||||
populated yet. The planned Node half targets Node/ESM with zero npm
|
||||
dependencies.
|
||||
range instead, which is why the install commands pair tag with tag.
|
||||
|
||||
The optional `[extract]` extra ships one parser, `pdfplumber` (MIT), for `pdf`;
|
||||
`docx`/`xlsx` are still unimplemented. It is opt-in because it pulls binary
|
||||
wheels (`pillow`, `pypdfium2`), which the default install must never do.
|
||||
|
||||
Request it by appending `[extract]` to the package name in whichever install
|
||||
command from [Install](#install) you are using — this package is not on an
|
||||
index, so a bare `pip install 'llm-ingestion-okf[extract]'` does **not** work
|
||||
today, and the error message naming that command is written for the day it
|
||||
does. The extra is unreleased: it reaches a consumer through a tag that
|
||||
contains it, and no such tag exists yet.
|
||||
|
||||
Two properties of the extra are worth knowing before depending on its output:
|
||||
|
||||
- **Extracted text is pinned to an exact parser version.** `pdfplumber` pins
|
||||
`pdfminer.six==20260107` exactly, and `pdfminer.six` ships date-stamped
|
||||
releases with no stability contract. Extraction is deterministic within a
|
||||
parser version and not guaranteed across one, so a golden fixture built on
|
||||
extracted PDF text is a fixture migration away from any parser upgrade.
|
||||
- **Text extraction recovers text, and nothing that is drawn.** Figures,
|
||||
diagrams and images have no text to recover — only their captions survive —
|
||||
so a bundle built from drawn documents is incomplete by construction. The
|
||||
library says so itself: every `pdf` extraction emits an `ExtractionWarning`.
|
||||
Structured table recovery is separately out of scope; PDFs enter as prose.
|
||||
|
||||
The planned Node half targets Node/ESM with zero npm dependencies.
|
||||
|
||||
## License
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue