open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(extract): measure one Vegnormalene PDF page against the extraction registry

Browse source 
Order 20260821T170054Z-486638087-from-.claude (gap G2). Measurement only: no
parser implemented, no version bump, no pin move.

Measured on Handbok N200 Vegbygging (juli 2018), 308 pages, page index 150:
the registry rejects .pdf with extractor_extra_missing while .md/.csv controls
pass in the same call, and process_inbox reports the file as failed without
aborting the run. pdfplumber recovers Tabell 524.1 as 4/4 correctly paired text
lines where pypdf, pdfminer.six and pymupdf all score 0/4; both structural
extractors return the same wrong 2x6 grid, so table structure is the document's
geometry rather than a library defect. Whole book: 308/308 pages yield text,
45 of 196 detected tables are clean enough for render_table.

Verdict: text extraction is a small, bounded job; structured table recovery is a
separate project that nothing currently waits on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013xTq1nbpz9x34udpDDExWM
This commit is contained in:
Kjell Tore Guttormsen 2026-08-21 19:16:15 +02:00
commit b73dd9d6a4
1 changed files with 274 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

274
docs/2026-08-21-g2-pdf-extraction-measurement.md Normal file
View file

@ -0,0 +1,274 @@
# G2 measurement: one Vegnormalene PDF page through the extraction registry
**Date:** 2026-08-21 · **Order:** `20260821T170054Z-486638087-from-.claude` (from `.claude`)
**Question this document answers:** is a PDF parser behind the `[extract]` extra a small,
bounded job, or is it a project?
This is a measurement, not an implementation. No production code was written, no version
bumped, no tag cut, no guard pin moved. Filename is English per this repo's docs convention;
the order suggested a Norwegian one. Grep anchor for the gap register: **G2**.
## Verdict
**Two answers, and the fork is the finding — but only one of the two branches is needed
to unblock the trial stone.**
1. **Text extraction is small.** One extractor function plus an import probe, at a seam that
is two source lines wide. One library (`pdfplumber`, MIT) reads 308 of 308 pages of the
measured handbook, is deterministic across runs, and recovers the requirement table on the
sample page as correctly paired text lines (4 of 4 rows). This is a bounded job.
2. **Recovering tables as *structured* markdown is a project, and it is not a parser choice.**
`pdfplumber.extract_tables()` and `PyMuPDF.find_tables()` — two independent
implementations — return the *same* wrong structure for the sample table: one 2x6 grid
where four logical rows collapse into a single cell with embedded newlines. The breakage
is in the document's ruling geometry, not in either parser. Fixing it means layout
heuristics this repo would own, specify and test. Across the whole handbook, only
**45 of 196** detected table objects are structurally clean enough to hand to
`render_table` unchanged.
**Recommendation: take branch 1, decline branch 2 for now.** Vegnormalene enter the chain as
prose with correctly paired table lines, which is what the trial stone needs to reach leg 3.
Branch 2 is a separate decision that nothing currently waits on.
**The choice of library is forced by branch 1, not free.** Cheaper candidates parse the page
but destroy the requirement table's row-to-value pairing — in a `krav` document that is a
correctness hazard, not a cosmetic one. See [Table pairing](#table-pairing-the-decisive-number).
## What was measured
| Item | Value |
|---|---|
| Source | Håndbok N200 Vegbygging (juli 2018), Statens vegvesen, public PDF |
| URL | `https://www.vegvesen.no/globalassets/fag/handboker/hb-n200-vegbygging-juli-2018.pdf` |
| Book bytes | 9 879 066 · sha256 `aec0825c305cc830f4e542062c51ec8e4499769495b7d39cb36e48079fe0fdb4` |
| Book pages | 308 |
| Sample page | page index 150 (printed page 151), ch. 5 "Dimensjonering av vegoverbygning" |
| Sample bytes | 316 356 · sha256 `95e1d28b42ec3f514a79e0ad568d91a7122530fba82828c05b859e324e0884ac` |
| Why this page | running prose + one numbered requirement table (`Tabell 524.1 Krav til helning på utkilingen`) + one real figure (`Figur 524.1`) + `skal`-requirements |
The PDF lives under a scratch path, not in this repo — it is test material, not content.
Everything below ran on Python 3.14.0 in throwaway virtualenvs; nothing was added to this
repo's environment or its dependency set.
## Probe 1 — the registry as it stands today
`extract_text` called on the page, with two known-positive controls **in the same call**, so
the negative result is measured rather than assumed:
```
n200-page.pdf: RAISED ExtractionError code='extractor_extra_missing' msg=extracting '.pdf'
requires the optional 'extract' extra (pip install 'llm-ingestion-okf[extract]');
it is not installed
control.md: OK 48 chars -> '# Kontroll\n\nDette er en kjent-positiv kontroll.\n'
control.csv: OK 54 chars -> '| krav | verdi |\n| --- | --- |\n| N200 5.4.1 | fritt |\n'
```
The probe can succeed; the `.pdf` rejection is real and typed, not a broken probe.
## Probe 2 — the full Door B path
The same page dropped in an inbox next to a `.md` control, through `process_inbox` with the
real guard adapter:
```
persisted : [PersistedFile(source_file='kontroll.md',
path=.../bundle/inbox-kontroll.md, reasons=('clean: no findings',))]
failed : [FailedFile(source_file='n200-page.pdf',
error=ExtractionError("extracting '.pdf' requires the optional 'extract' extra ..."))]
rejected : () quarantined: ()
```
The behaviour is exactly what the contract promises: one bad file does not abort the run, the
control persists and is indexed, and the PDF is reported per-file. Nothing here is a defect —
leg 2 stops because the extra ships no parser, by design.
## Candidates
License, release date and required dependencies read from PyPI metadata; package count and
disk size measured by installing each into a clean virtualenv and listing it.
| Candidate | Version | Released | License | Packages installed | site-packages |
|---|---|---|---|---|---|
| `pypdf` | 6.16.1 | 2026-08-14 | BSD-3-Clause | 1 | 16 MB |
| `pdfminer.six` | 20260107 | 2026-01-07 | MIT | 5 | 32 MB |
| `pdfplumber` | 0.11.10 | 2026-06-15 | MIT | 8 | 56 MB |
| `pymupdf` | 1.28.2 | 2026-08-06 | AGPL-3.0 **or** Artifex commercial | 1 | 71 MB |
Three facts that matter more than the table:
- **`pymupdf` is a licensing decision, not a technical one.** AGPL-3.0 propagates to whoever
installs the extra. This repo is MIT; shipping an AGPL default in an optional extra hands a
consumer a copyleft obligation they did not choose. It is out on that ground alone, and it
performs no better on the sample page than the MIT alternatives.
- **`pdfplumber` pins `pdfminer.six==20260107` exactly.** A hard `==` in a transitive
dependency is a co-installation hazard for any consumer that already uses `pdfminer.six`.
- **`pdfplumber` pulls binary wheels** (`pillow`, `pypdfium2`). Acceptable behind an opt-in
extra; it would not be acceptable in core, and this repo's "exactly one runtime dependency"
rule is untouched either way.
For completeness, the other two types the extra covers: `python-docx` 1.2.0 (MIT, 3 packages,
26 MB, pulls `lxml`) and `openpyxl` 3.1.5 (MIT, 2 packages, 15 MB). Both are materially
simpler problems than PDF — the source formats are already structured XML — so they do not
change the verdict, but they do mean `[extract]` is three parsers, not one.
## What each candidate does to this page
`pdfplumber.extract_text()`, first ~30 lines, verbatim:
```
Kapittel 5 – Dimensjonering av vegoverbygning
Utkiling
Utkiling i vegens lengderetning (overganger i undergrunn)
For ikke frostsikret veg skal det bygges en utkiling med ikke telefarlige materialer eller med
isolasjonsmaterialer for å unngå ujevne telehiv ved overgang mellom materialer med forskjellig
telefarlighet. Utkilingen går opp til underkant av forsterkningslaget. For frostsikret veg er det ikke
behov for utkilinger.
Isolasjonsmaterialene skal ligge drenert. Figur 524.1 viser prinsippskisser for utkiling med ikke
telefarlige materialer og isolasjonsplater av XPS. Også skumglass og lettklinker kan benyttes til
utkiling. Utkilingslengde L skal beregnes fra tabell 524.1 hvor frostsikringsdybden ved frostmengde
F danner utgangspunktet for beregning av utkilingslengden. Ved bruk av isolasjonsmaterialer skal
10
det benyttes et nedre frostsikringslag med tykkelse som gitt i tabell 521.2.
Utkilingslengder gjelder både for nyanlegg og utbedring av eksisterende veg.
Tabell 524.1 Krav til helning på utkilingen
Skiltet hastighet, km/t Helning på utkilingen, maks
≤ 50 og g/s – veger 1:10
60 og 70 1:15
80 og 90 1: 25
≥ 100 Frostsikret veg forutsettes
Figur 524.1 Utkiling ved overgang bergskjæring/telefarlig grunn eller underbygning
For å redusere fuktopptak i isolasjonsplater av XPS skal tykkelsen være minimum 5 cm. For å unngå å
bruke tynnere isolasjonsplater i avslutningen av en utkiling bør det vurderes å legge isolasjonsplatene
med økende mellomrom for gradvis å redusere isoleringsevnen.
Ved overgang mellom skjæring og fylling i telefarlig jord, bør utkilingen utføres med det materialet
fyllingen er bygget opp av, se figur 524.2. Største utkilingsdybde skal beregnes etter reglene i kapittel
521.
Håndbok N200 Vegbygging (juli 2018) 151
```
That is readable Norwegian technical prose, in reading order, with the table's rows intact as
lines. Two fidelity defects are visible and neither is fatal: the subscript in `F10` breaks
across two lines, and the figure is gone — only its caption survives, because the figure is a
vector drawing, not text. A figure whose content is only in the drawing is lost at leg 2
regardless of parser, and that is worth knowing before anyone promises a complete bundle.
Character counts on the same page: `pypdf` 1819, `pdfminer.six` 1834, `pdfplumber` 1761,
`pymupdf` 1820. **Volume is not the discriminator — order is.**
### Table pairing: the decisive number
`Tabell 524.1` has four requirement rows (speed class → maximum slope). Counting how many
survive with label and value on the same line:
| Parser | Paired rows |
|---|---|
| `pdfplumber.extract_text()` | **4 / 4** |
| `pypdf.extract_text()` | 0 / 4 |
| `pdfminer.six.extract_text()` | 0 / 4 |
| `pymupdf.get_text()` | 0 / 4 |
The three zero-scorers emit all four labels, then all four values, as separate blocks. A
downstream reader — human or model — can only re-pair them by guessing. In a requirements
document, `80 og 90 → 1:25` becoming `80 og 90 → 1:10` is a wrong answer that looks right.
This is why the cheapest candidate is not the correct one.
### What happens to the table as *structure*
Feeding `render_table` needs rows and cells, not lines. Both structural extractors return the
same thing for this table:
```
pdfplumber.extract_tables() -> 1 table, 2 rows x 6 cols
['', 'Skiltet hastighet, km/t', '', '', 'Helning på utkilingen, maks', '']
['≤ 50 og g/s – veger\n60 og 70\n80 og 90\n≥ 100', None, None,
'1:10\n1:15\n1: 25\nFrostsikret veg forutsettes', None, None]
pymupdf.find_tables() -> 1 table, 2 rows x 6 cols (byte-identical shape)
```
Four logical rows collapse into one cell each side; three of six columns are empty padding.
`render_table` would faithfully render a two-row table that misrepresents the requirement.
Two independent implementations agreeing on the same wrong shape is the evidence that this is
the document's geometry, not a library bug — N200 draws ruling lines around blocks, not rows.
**So: Vegnormalene are usable as prose, and are not usable as structured concept tables
without a layer this repo does not have.**
## Determinism
Two runs of each configuration, compared byte-for-byte:
```
pypdf identical=True
pdfminer.six identical=True
pdfplumber.text identical=True
pdfplumber.tables identical=True
pdfplumber slice[140:160] identical=True
```
Within one parser version, extraction is deterministic — the bit-exact promise survives.
**Across parser versions it is not guaranteed** (ASSUMED, not measured): `pdfminer.six` uses
date-stamped releases with no stability contract, so any golden fixture built on extracted
text would be pinned to an exact parser version, and a parser upgrade becomes a fixture
migration. That is a real, recurring maintenance cost and it belongs in the decision.
## Denominators
Whole-book run, all 308 pages:
| Measure | Count |
|---|---|
| Pages total | 308 |
| Pages yielding non-empty text | **308 / 308** |
| Lines matching `^Tabell <n>` (tables the document claims) | 149 |
| Table objects detected by `extract_tables()` | 196 |
| Detected tables clean enough for `render_table` unchanged | **45 / 196** |
"Clean enough" is defined in the measurement script, not by eye: at least two rows,
rectangular, no `None` cell, no cell containing a newline. Anything failing that would either
crash or silently misrepresent if handed to `render_table`.
Scope of these denominators: **one handbook, one edition (N200, July 2018)**. They are not a
claim about the N-series as a whole. Nothing here was measured on a scanned or image-only
PDF; every page of this book carries a text layer, and a scanned normal would score 0 and
need OCR — a different project entirely.
## Size of the wiring
The seam is small and already shaped for this. `grep` finds the gate at exactly two source
lines — `extract.py:30` (`_OPTIONAL_EXTENSIONS`) and `extract.py:135` (the raise) — plus two
test files asserting the current rejection (`tests/test_extract.py`,
`tests/test_error_codes.py`). Adding a type means one `_extract_pdf(data)` function in the
same shape as `_extract_csv`, an import probe replacing the frozenset membership test, and
the same error code kept for the still-uninstalled case. The module docstring already
describes this exact transition.
**The wiring is hours. The cost is the choice above it**: which library, which fidelity
promise, which fixtures, and what happens to figures and tables that do not survive.
## What this order deliberately did not do
- No parser implemented, no `[extract]` populated, no error message changed.
- No version bump, no tag, no publish, no guard pin move.
- No writes in any other repo. The PDF stays in scratch.
## Verification log
| # | Claim | Status | Evidence |
|---|---|---|---|
| 1 | The N200 PDF is public and downloadable from vegvesen.no | VERIFIED | `curl -L -w '%{http_code} %{content_type} %{size_download}'` → `200 application/pdf 9879066`; magic bytes `%PDF-1.6` |
| 2 | Book has 308 pages; sample is index 150 | VERIFIED | `pypdf.PdfReader(...).pages` length; `makepage.py` |
| 3 | `.pdf` fails with `extractor_extra_missing`, controls pass in the same call | VERIFIED | `probe_registry.py` output, quoted above |
| 4 | Full Door B path reports the PDF as `failed` and persists the control | VERIFIED | `probe_inbox.py` output, quoted above |
| 5 | `[extract]` is declared and empty | VERIFIED | `pyproject.toml` `[project.optional-dependencies] extract = []` |
| 6 | Candidate licenses and release dates | VERIFIED | PyPI JSON API per package (`pypi.org/pypi/<name>/json`) |
| 7 | Package counts and disk sizes | VERIFIED | one clean `python3 -m venv` per candidate; `pip list --format=freeze`, `du -sk` |
| 8 | `pdfplumber` pins `pdfminer.six==20260107` | VERIFIED | PyPI `requires_dist` |
| 9 | Table row pairing 4/4 vs 0/4 | VERIFIED | `pairing.py` / `pairing_mupdf.py`, ground truth read off the rendered page |
| 10 | `extract_tables()` and `find_tables()` return the same 2x6 shape | VERIFIED | `cand_pdfplumber.py`, `cand_pymupdf.py` |
| 11 | Determinism within a version, 5 configurations | VERIFIED | `determinism.py`, sha256 per run |
| 12 | 308/308 pages with text; 149 captions; 196 detected; 45 clean | VERIFIED | `denominator.py`, whole book, ~41 s |
| 13 | Gate seam is two source lines plus two test files | VERIFIED | `grep -rn '_OPTIONAL_EXTENSIONS\|extractor_extra_missing' src/ tests/` |
| 14 | Cross-version output stability is not guaranteed | ASSUMED | inferred from `pdfminer.six` date-stamped versioning; not measured across versions |
| 15 | Figures are lost because they are vector drawings | VERIFIED (this page) | caption present, no figure text in any parser's output; not generalised to the book |
| 16 | Denominators generalise beyond N200:2018 | NOT CLAIMED | one handbook, one edition measured |