feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.
`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.
It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).
`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.
Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.
Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
f6fea13299
commit
53d5c74c96
15 changed files with 1394 additions and 28 deletions
37
README.md
37
README.md
|
|
@ -88,12 +88,12 @@ did not pass them.
|
|||
|
||||
### The segmentation flags
|
||||
|
||||
Six rules are reachable from `okf build`. **Two of them are ON by default since
|
||||
2026-09-08** — `--outline-run 3` and `--unit-fold`, an operator decision — and
|
||||
each has an explicit opt-out, `--outline-run 0` and `--no-unit-fold`. Passing
|
||||
both opt-outs reproduces the pre-2026-09-08 bytes exactly. The other four are
|
||||
off. Each line below carries the number it was measured at, and nothing beyond
|
||||
it.
|
||||
Six rules are reachable from `okf build`. **Three of them are ON by default
|
||||
since 2026-09-08** — `--outline-run 3`, `--table-grid` and `--unit-fold`, an
|
||||
operator decision taken in two steps — and each has an explicit opt-out,
|
||||
`--outline-run 0`, `--no-table-grid` and `--no-unit-fold`. Passing all three
|
||||
opt-outs reproduces the pre-2026-09-08 bytes exactly. The other three are off.
|
||||
Each line below carries the number it was measured at, and nothing beyond it.
|
||||
|
||||
**A re-run is what this costs a consumer, and it is not a small one:** on the
|
||||
43-document reference corpus the default bundle goes from **629 concepts in
|
||||
|
|
@ -104,7 +104,7 @@ block still runs as written.
|
|||
| flag | what it does | measured |
|
||||
|---|---|---|
|
||||
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
|
||||
| `--table-grid` | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
|
||||
| `--table-grid` (**on** by default; opt out with `--no-table-grid`) | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
|
||||
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
|
||||
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
|
||||
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
|
||||
|
|
@ -128,6 +128,29 @@ is 9, so the default is a full arm behind what the proposer can do on that
|
|||
document — which is a statement about the default, not a licence to change it
|
||||
here.
|
||||
|
||||
### The two PDF reader flags
|
||||
|
||||
Separate from the six above, and they sit before every one of them: a
|
||||
segmentation flag changes how the proposer cuts a text, these change what the
|
||||
text says. **Both are off by default.**
|
||||
|
||||
| flag | what it does | measured |
|
||||
|---|---|---|
|
||||
| `--pdf-headings font` | a PDF carries no heading markup, so one is inferred from typography — a line whose dominant font size is above the document's character-weighted median AND whose dominant font name says bold — and emitted as an ATX heading in the same markdown the office path produces, so the existing heading rule reads it | on a tender PDF: **9 of 9** numbered chapters found, plus 4 extra candidates. Whole corpus: **25 of 32 `pdf`** change, **0 of 5 `docx`**, **0 of 2 `xlsx`**. **Off by measurement:** against the operator's unit worksheet it takes `pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those documents the outline rule already found the chapters and a second heading source can only add |
|
||||
| `--ocr` | read a PDF page as an image when its own text never arrived: the page extracts empty, or as `(cid:N)` placeholder codes at or above 10 % of its characters. Needs the optional `ocr` group | on the one corpus document with the failure: **95.07 % → 0 %** cid, **44 → 2561** words of four or more letters, 17 → **18** pages with text, 3.6 s/page. Whole corpus: **16 of 834** pages qualify, in **1 of 32** files |
|
||||
|
||||
```
|
||||
pip install "llm-ingestion-okf[extract,ocr]"
|
||||
```
|
||||
|
||||
Without that group `--ocr` is a typed refusal (`extractor_ocr_group_missing`)
|
||||
per file, never a crash, and the corpus run still reports
|
||||
`merged + coded rejections == N`. OCR text is a model's reading of an image: it
|
||||
is reproducible against the model version and rendering resolution it was
|
||||
produced with, and no dependency pin can promise more. The full measurement,
|
||||
including the per-page distribution the 10 % threshold was read off, is
|
||||
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`.
|
||||
|
||||
Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
|
||||
three files no reader accepts), one
|
||||
`okf build` invocation replacing the shell loop over `tools/` that produced the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue