Kjell Tore Guttormsen
658b7aafe0
Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.
The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.
`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.
The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.
Three facts from that measurement are now carried in code rather than in a
report:
- Extracted text is pinned to an exact transitive parser version
(pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
contract). tests/test_extract.py freezes the expected text of a committed
hand-written fixture so a parser upgrade breaks something visible instead of
drifting silently. Reasoning at the declaration site and in
tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
new `ExtractionWarning`: figures have no text to recover, so a bundle built
from drawn documents is incomplete by construction. Stated categorically
rather than detected — deciding "is there a figure here" is the layout
heuristic G2b declined.
Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).
Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.
Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
64 lines
2.5 KiB
Python
64 lines
2.5 KiB
Python
"""Regenerate the committed PDF fixtures for tests/test_extract.py.
|
|
|
|
Hand-written minimal PDFs: objects laid out by hand, xref offsets computed
|
|
from the emitted bytes. No generator library, so the fixtures are auditable
|
|
byte for byte and reproducible from this file alone.
|
|
|
|
Run from the repository root: python3 tests/fixtures/make_fixtures.py
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).parent
|
|
|
|
# Two text lines: a heading, and one requirement row with label and value on
|
|
# the SAME line. That pairing is the property the parser choice was made on
|
|
# (see docs/2026-08-21-g2-pdf-extraction-measurement.md), so the fixture
|
|
# fails visibly if a parser upgrade ever breaks it. Byte 0xE5 is the Norwegian
|
|
# 'a-ring' in WinAnsiEncoding, which the font object below declares.
|
|
KRAV_CONTENT = (
|
|
b"BT /F1 12 Tf 20 160 Td (Krav til helning p\xe5 utkilingen) Tj ET\n"
|
|
b"BT /F1 12 Tf 20 140 Td (60 og 70 1:15) Tj ET\n"
|
|
)
|
|
|
|
# A structurally valid page carrying no text operators at all -- the shape a
|
|
# scanned or image-only PDF presents to a text extractor.
|
|
NO_TEXT_CONTENT = b"20 20 160 160 re S\n"
|
|
|
|
|
|
def build_pdf(content: bytes) -> bytes:
|
|
"""Assemble a one-page PDF around `content` as the page content stream."""
|
|
objects = [
|
|
b"<< /Type /Catalog /Pages 2 0 R >>",
|
|
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
|
|
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] "
|
|
b"/Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>",
|
|
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream",
|
|
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
|
|
]
|
|
|
|
out = bytearray(b"%PDF-1.4\n")
|
|
offsets = []
|
|
for number, body in enumerate(objects, start=1):
|
|
offsets.append(len(out))
|
|
out += str(number).encode() + b" 0 obj\n" + body + b"\nendobj\n"
|
|
|
|
xref_at = len(out)
|
|
size = str(len(objects) + 1).encode()
|
|
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
|
|
for offset in offsets:
|
|
out += ("%010d 00000 n \n" % offset).encode()
|
|
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
|
|
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
|
|
return bytes(out)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
for name, content in (
|
|
("two-line-krav.pdf", KRAV_CONTENT),
|
|
("no-text-layer.pdf", NO_TEXT_CONTENT),
|
|
):
|
|
(HERE / name).write_bytes(build_pdf(content))
|
|
print(f"wrote {name}")
|