Order G2a. Populates the optional `[extract]` extra for the first time with
one parser, `pdfplumber>=0.11.10,<0.12` (MIT), and wires `pdf` through it.
The default install is untouched: exactly one runtime dependency, stdlib
otherwise, enforced by test_packaging.py.
The gate for `pdf` becomes an import probe rather than a frozenset membership
test, exactly as extract.py's docstring had promised. The rejection does not
change: without the extra, `pdf` still raises `extractor_extra_missing` with
the same message. That behaviour is asserted UNCONDITIONALLY via a sys.modules
monkeypatch, so it holds on machines where the parser is installed too — a
skip would have preserved nothing there. Verified in a clean venv without the
extra: 589 passed, 7 skipped; with it, 596 passed.
`docx`/`xlsx` are unchanged and still fail fast — the extra names exactly what
it ships.
The parser choice was forced by measurement, not preference (b73dd9d,
docs/2026-08-21-g2-pdf-extraction-measurement.md): on a real requirement table
pdfplumber keeps 4 of 4 rows with label and value on one line, where pypdf,
pdfminer.six and pymupdf each keep 0 of 4. pymupdf is additionally out on
licence (AGPL-3.0), which an MIT package must not push onto a consumer.
Three facts from that measurement are now carried in code rather than in a
report:
- Extracted text is pinned to an exact transitive parser version
(pdfplumber pins pdfminer.six==20260107; date-stamped, no stability
contract). tests/test_extract.py freezes the expected text of a committed
hand-written fixture so a parser upgrade breaks something visible instead of
drifting silently. Reasoning at the declaration site and in
tests/fixtures/README.md.
- Determinism within a version is now held by a test, not only measured once.
- Drawn content does not survive extraction. Every pdf extraction emits the
new `ExtractionWarning`: figures have no text to recover, so a bundle built
from drawn documents is incomplete by construction. Stated categorically
rather than detected — deciding "is there a figure here" is the layout
heuristic G2b declined.
Two new error codes, both mirroring existing patterns: `extractor_empty_pdf`
(a scanned/image-only PDF, refused rather than persisted as an empty concept)
and `extractor_pdf_error` (parser failure wrapped, never leaked).
Structured table recovery (G2b) is NOT implemented and is documented as out of
scope: two independent parsers return the same wrong shape, so the breakage is
document geometry, not a library choice. PDFs enter as prose.
Also corrects an install promise this change would otherwise have published:
the README no longer presents a bare `pip install 'llm-ingestion-okf[extract]'`
as working, because the package is not on an index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtNhsdHnMGtMi7U2mvMU8z
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.
Two constraints shaped the design and are pinned by tests:
- A merged concept is written VERBATIM. Stamping provenance into it would
require round-tripping its frontmatter through this library's line-oriented
parser, which cannot represent the block lists the guard's parser accepts --
silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
target name are a no-op re-merge (re-import of an unchanged bundle is
idempotent), and anything else at the name is refused. Curated content and
an updated concept are refused alike; refusing is what never destroys.
The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.
Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.
README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.
405 tests green; ruff, ruff format and mypy --strict clean.
First, guard-independent step of Phase 2: a stdlib-only registry mapping a
dropped file's extension to its text extractor, with the fail-fast gates that
keep binary parsing out of core. `extract_text(filename, data)` dispatches
(case-insensitively) to:
- `md`/`txt` — utf-8-sig passthrough (BOM never leaks, baseline parity with
Door A's read_csv);
- `csv` — the Phase 1 `render_table` (renderer reused, not duplicated);
- `json` — verbatim inside `render_fenced_block`;
- `html`/`htm` — text via `html.parser`, `script`/`style` stripped, tags as
word boundaries (spec B3: adequate for v1, richer is out of scope).
`pdf`/`docx`/`xlsx` are `[extract]`-gated; until that extra ships a parser they
fail fast with a typed error naming the extra — never a silent skip, never a
bundled parser in core. New `ExtractionError(IngestError)` carries four stable
codes (`extractor_unknown`, `extractor_extra_missing`, `extractor_decode_error`,
`extractor_empty_csv`); a non-UTF-8 file is a typed corrupt-input failure, never
a leaked UnicodeDecodeError. `extract_text` returns text content only — LF
framing and concept frontmatter are the materializer's job (step 2).
No runtime dependency and no guard call yet (the guard pin and 0.4.0 land with
the persist gate in steps 4–5). TDD: test_extract.py + the four codes in the
test_error_codes.py registry precede the implementation; mypy --strict, ruff,
and the `sanitize|quarantine|lexicon` boundary grep-gate all clean; the Phase 1
golden suite still passes byte-for-byte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBbjgS5A55RVavoyjJC4FX
