feat(extract): implement pdf behind the [extract] extra with pdfplumber

Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.

The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.

`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.

The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.

Three facts from that measurement are now carried in code rather than in a
report:

- Extracted text is pinned to an exact transitive parser version
  (pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
  contract). tests/test_extract.py freezes the expected text of a committed
  hand-written fixture so a parser upgrade breaks something visible instead of
  drifting silently. Reasoning at the declaration site and in
  tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
  new `ExtractionWarning`: figures have no text to recover, so a bundle built
  from drawn documents is incomplete by construction. Stated categorically
  rather than detected — deciding "is there a figure here" is the layout
  heuristic G2b declined.

Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).

Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.

Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z

src/llm_ingestion_okf/extract.py

@@ -4,9 +4,10 @@ All file-type -> text extraction lives here (the guard is text-only). The core
 registry is stdlib-only and deterministic: `md`/`txt` pass through, `csv` renders
 the Phase 1 markdown table, `json` is fenced verbatim, and `html`/`htm` are
 reduced to text with `html.parser`. Binary types (`pdf`/`docx`/`xlsx`) are
-`[extract]`-gated and — until that optional extra ships a parser — fail fast with
-a typed error naming the extra, never a silent skip and never a bundled parser in
-core.
+`[extract]`-gated: `pdf` is extracted with `pdfplumber` when the extra is
+installed and rejected with the same typed error when it is not (the gate is an
+import probe, not a membership test), while `docx`/`xlsx` ship no parser yet and
+always fail fast. Never a silent skip and never a bundled parser in core.
 
 `extract_text` returns the extracted text *content*; final LF framing and the
 concept frontmatter are the materializer's concern (Phase 2 step 2), not this
@@ -17,17 +18,30 @@ from __future__ import annotations
 
 import csv
 import io
+import warnings
 from collections.abc import Callable
 from html.parser import HTMLParser
 from pathlib import Path
 
-from .errors import ExtractionError
+from .errors import ExtractionError, ExtractionWarning
 from .render import render_fenced_block, render_table
 
-# Binary types gated behind the optional `[extract]` extra. The extra ships no
-# parser yet, so these always fail fast for now; when a parser lands the gate
-# becomes an import probe, but the rejection code and message stay the same.
-_OPTIONAL_EXTENSIONS = frozenset({".pdf", ".docx", ".xlsx"})
+# Binary types gated behind the optional `[extract]` extra that it ships no
+# parser for. `.pdf` is no longer here: its gate is the import probe inside
+# `_extract_pdf`, which raises the same code and the same message when the
+# extra is absent. These two still fail fast unconditionally.
+_UNPARSED_OPTIONAL_EXTENSIONS = frozenset({".docx", ".xlsx"})
+
+# Text extraction recovers text. A figure is a vector drawing with no text to
+# recover — only its caption survives — so any bundle built from drawn
+# documents is incomplete by construction. Said out loud on every PDF rather
+# than detected per document: deciding "is there a figure here" is a layout
+# heuristic this library does not own.
+_PDF_LOSSY_WARNING = (
+    "PDF extraction recovers text only: figures, diagrams and images are not "
+    "represented in the output (their captions are). A bundle built from "
+    "drawn documents is incomplete by construction."
+)
 
 # Tags whose text content is never document prose.
 _SKIP_TAGS = frozenset({"script", "style"})
@@ -110,6 +124,64 @@ def _extract_html(data: bytes) -> str:
     return parser.text()
 
 
+def _extra_missing(suffix: str) -> ExtractionError:
+    """The one rejection for a `[extract]` type without the extra installed.
+
+    One constructor, one wording: the import probe and the still-unparsed
+    types must be indistinguishable to a consumer, because they are the same
+    fact — the extra is not installed.
+    """
+    return ExtractionError(
+        f"extracting {suffix!r} requires the optional 'extract' extra "
+        f"(pip install 'llm-ingestion-okf[extract]'); it is not installed",
+        code="extractor_extra_missing",
+    )
+
+
+def _extract_pdf(data: bytes) -> str:
+    """`pdf`: page text via `pdfplumber`, in page order, pages separated by a
+    blank line.
+
+    The gate is this import, not a membership test: without the `[extract]`
+    extra the very same typed rejection is raised as for the types that ship
+    no parser at all. Text is returned VERBATIM — no Unicode normalization,
+    matching `md`/`txt` passthrough; normalizing would edit source content,
+    and NFC folding belongs to filenames and titles, not to document bodies.
+
+    `pdfplumber` was chosen on ONE measured property (2026-08-21,
+    docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement
+    table it keeps label and value on the same line, where pypdf, pdfminer.six
+    and pymupdf each emit all labels then all values. Re-pairing those is
+    guesswork, and in a requirements document a wrong pairing looks right.
+    """
+    try:
+        import pdfplumber
+    except ImportError as exc:
+        raise _extra_missing(".pdf") from exc
+
+    try:
+        with pdfplumber.open(io.BytesIO(data)) as pdf:
+            pages = [(page.extract_text() or "").rstrip() for page in pdf.pages]
+    except ExtractionError:
+        raise
+    except Exception as exc:  # noqa: BLE001 - third-party parser, wrapped never leaked
+        raise ExtractionError(
+            f"the PDF parser failed on this file: {exc}", code="extractor_pdf_error"
+        ) from exc
+
+    text = "\n\n".join(page for page in pages if page)
+    if not text:
+        raise ExtractionError(
+            "the PDF yielded no text on any page; a scanned or image-only "
+            "document needs OCR, which this registry does not do",
+            code="extractor_empty_pdf",
+        )
+    # After the parse, not before: a run that produced no text has nothing to
+    # be lossy about, and warning there would just add noise to a failure.
+    warnings.warn(_PDF_LOSSY_WARNING, ExtractionWarning, stacklevel=3)
+    return text
+
+
 _CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
     ".md": _extract_passthrough,
     ".txt": _extract_passthrough,
@@ -119,6 +191,12 @@ _CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
     ".htm": _extract_html,
 }
 
+# Types the `[extract]` extra ships a parser for. Kept separate from the core
+# registry so "adds no runtime dependency" stays readable at a glance.
+_OPTIONAL_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
+    ".pdf": _extract_pdf,
+}
+
 
 def extract_text(filename: str, data: bytes) -> str:
     """Convert one dropped file's bytes to OKF concept text, dispatched by type.
@@ -126,18 +204,15 @@ def extract_text(filename: str, data: bytes) -> str:
     `filename` supplies the extension (case-insensitive); `data` is the raw
     bytes. A core stdlib type is extracted; a `[extract]`-gated binary type
     without the extra, and any unregistered extension, fail fast with a typed
-    :class:`ExtractionError`.
+    :class:`ExtractionError`. Extracting a `pdf` also emits an
+    :class:`ExtractionWarning`: drawn content has no text to recover.
     """
     suffix = Path(filename).suffix.lower()
-    extractor = _CORE_EXTRACTORS.get(suffix)
+    extractor = _CORE_EXTRACTORS.get(suffix) or _OPTIONAL_EXTRACTORS.get(suffix)
     if extractor is not None:
         return extractor(data)
-    if suffix in _OPTIONAL_EXTENSIONS:
-        raise ExtractionError(
-            f"extracting {suffix!r} requires the optional 'extract' extra "
-            f"(pip install 'llm-ingestion-okf[extract]'); it is not installed",
-            code="extractor_extra_missing",
-        )
+    if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
+        raise _extra_missing(suffix)
     raise ExtractionError(
         f"no extractor is registered for file extension {suffix!r} ({filename!r})",
         code="extractor_unknown",