open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off

Browse source 
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.

`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.

It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).

`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.

Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.

Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 23:10:47 +02:00
commit 53d5c74c96
15 changed files with 1394 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

32
CLAUDE.md
View file

@ -278,8 +278,10 @@ and fixtures, never code.
so the two layers disagree on purpose and `cli.DEFAULT_OUTLINE_RUN` /
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
(`9cd74519...`, the delivered 2026-09-03 tree) to **517 / 969**
(`42c8646d...`, default flags, default epoch stamp). **And the number the
(`9cd74519...`, the delivered 2026-09-03 tree) to **492 / 944**
(`bdefa679...`, default flags, default epoch stamp, measured 2026-09-08 on
`f6fea13`). That digest SUPERSEDES `42c8646d...` (517 / 969), which named the
round-3 default before Arm E joined it. **And the number the
decision cites belongs to another configuration:** Arm F's 5 of 12 was
measured with `--table-grid` ON; without it the same sample scores 2 of 12
and `docx` 0 of 3, because the fold's table clause has no joined table to
@ -307,6 +309,32 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **Two PDF READER flags, both off, and they sit BEFORE every segmentation
flag** -- an arm changes how the proposer cuts a text, these change what the
text says. `--pdf-headings font` infers a heading from typography (dominant
font size above the document's character-weighted body median AND a bold font
name -- the CONJUNCTION measured at recall 1.000 / precision 0.846, where
adding weight as a disjunct took precision 0.786 -> 0.524) and emits it as an
ATX heading in the SAME markdown the office path produces, so `_ATX` applies
unchanged and **no PDF-only heading grammar exists**. It is off **by
measurement, not by caution**: against the operator's unit worksheet it takes
`pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those
documents the outline rule already recovers the document's own numbered
chapters and a second heading source can only add. The cost of the ATX form is
named rather than hidden: a font-inferred heading carries `rule:heading` and is
indistinguishable in the artifact from one the document declared, which is why
`RULE_POPPLER_SIZE_AND_BOLD` was deliberately NOT assigned to it -- that name
records a poppler measurement on a path that cannot ship. `--ocr` reads a page
as an IMAGE when its own text never arrived (empty, or `(cid:N)` codes at or
above `OCR_CID_SHARE = 0.10`, a threshold READ OFF the measured per-page
distribution: 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing between). Its engine is the optional `ocr` group
(`rapidocr`/`onnxruntime`/`pypdfium2`) and **never** a runtime dependency; a
packaging test pins both halves, and without it every affected file is a coded
rejection (`extractor_ocr_group_missing`), never a crash. `--ocr` can never
become a default -- an optional dependency in the default path would make an
ordinary install fail on the first scanned page. Report:
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction

37
README.md
View file

@ -88,12 +88,12 @@ did not pass them.
### The segmentation flags
Six rules are reachable from `okf build`. **Two of them are ON by default since
2026-09-08** — `--outline-run 3` and `--unit-fold`, an operator decision — and
each has an explicit opt-out, `--outline-run 0` and `--no-unit-fold`. Passing
both opt-outs reproduces the pre-2026-09-08 bytes exactly. The other four are
off. Each line below carries the number it was measured at, and nothing beyond
it.
Six rules are reachable from `okf build`. **Three of them are ON by default
since 2026-09-08** — `--outline-run 3`, `--table-grid` and `--unit-fold`, an
operator decision taken in two steps — and each has an explicit opt-out,
`--outline-run 0`, `--no-table-grid` and `--no-unit-fold`. Passing all three
opt-outs reproduces the pre-2026-09-08 bytes exactly. The other three are off.
Each line below carries the number it was measured at, and nothing beyond it.
**A re-run is what this costs a consumer, and it is not a small one:** on the
43-document reference corpus the default bundle goes from **629 concepts in
@ -104,7 +104,7 @@ block still runs as written.
| flag | what it does | measured |
|---|---|---|
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
| `--table-grid` | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
| `--table-grid` (**on** by default; opt out with `--no-table-grid`) | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
@ -128,6 +128,29 @@ is 9, so the default is a full arm behind what the proposer can do on that
document — which is a statement about the default, not a licence to change it
here.
### The two PDF reader flags
Separate from the six above, and they sit before every one of them: a
segmentation flag changes how the proposer cuts a text, these change what the
text says. **Both are off by default.**
| flag | what it does | measured |
|---|---|---|
| `--pdf-headings font` | a PDF carries no heading markup, so one is inferred from typography — a line whose dominant font size is above the document's character-weighted median AND whose dominant font name says bold — and emitted as an ATX heading in the same markdown the office path produces, so the existing heading rule reads it | on a tender PDF: **9 of 9** numbered chapters found, plus 4 extra candidates. Whole corpus: **25 of 32 `pdf`** change, **0 of 5 `docx`**, **0 of 2 `xlsx`**. **Off by measurement:** against the operator's unit worksheet it takes `pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those documents the outline rule already found the chapters and a second heading source can only add |
| `--ocr` | read a PDF page as an image when its own text never arrived: the page extracts empty, or as `(cid:N)` placeholder codes at or above 10 % of its characters. Needs the optional `ocr` group | on the one corpus document with the failure: **95.07 % → 0 %** cid, **44 → 2561** words of four or more letters, 17 → **18** pages with text, 3.6 s/page. Whole corpus: **16 of 834** pages qualify, in **1 of 32** files |
```
pip install "llm-ingestion-okf[extract,ocr]"
```
Without that group `--ocr` is a typed refusal (`extractor_ocr_group_missing`)
per file, never a crash, and the corpus run still reports
`merged + coded rejections == N`. OCR text is a model's reading of an image: it
is reproducible against the model version and rendering resolution it was
produced with, and no dependency pin can promise more. The full measurement,
including the per-page distribution the 10 % threshold was read off, is
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`.
Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
three files no reader accepts), one
`okf build` invocation replacing the shell loop over `tools/` that produced the

483
docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md Normal file
View file

@ -0,0 +1,483 @@
# K3 round 4, `pdf`: typography as a heading source, and OCR for the pages whose text never came
Fourth iteration of the per-file-type directive of 2026-09-08 13:05Z. Order
`20260908T195728Z-8160520552-from-.claude`. HEAD at the start and at the end of
the session is O5's `f6fea13` plus this session's commit; the tag on `f6fea13`
is `v0.6.0`.
Everything below is measured on the same 43-document corpus and the same
12-document sample as `docs/2026-09-08-k3-runde3-per-filtype.md`, against the
same reference: the operator's filled unit worksheet, which is local-only and
gitignored. No document title or extracted text from it is reproduced here.
## Section 0: what was measured and what was not
**Measured.** The three reproduction numbers the order names, before any edit.
Two new readers, each red-first, each with a known-negative. The per-page
`(cid:N)` distribution over every K2 PDF, with both denominators, and the
threshold read off it rather than chosen. What OCR recovers on the one document
that has the failure, through the shipped code path and the shipped instrument.
The 12-document sample re-scored with session 109's criterion under seven
configurations. The whole 43-document corpus under both renderings from one
cached extraction. Two K2 bundles differing only in the new flag.
**Not measured.** Whether a live model answers better with any of this. Whether
OCR helps a corpus that is not this one -- the `(cid:N)` denominator that
matters is **1 document of 39**. Whether the font reader helps a corpus whose
typography is unlike this one. No o200k token counts (no encoder is installed
and the single-dependency rule does not admit one).
## 1. The reproduction, before any edit
`git rev-parse HEAD` -> `f6fea13299d92b959f497652ff19a7fd79546281`,
`git status --short` empty, tag `v0.6.0` on that commit.
| # | control | expected | result |
|---|---|---|---|
| a | `pytest -q` | 1427 collected | **1427 passed** in 38 s |
| b | `okf build ~/okf-test/dokumenter`, no flags | 30 | **30** markdown files |
| c | the same folder, `--no-table-grid` | 43 | **43** markdown files |
All three reproduce. (a) differs in FORM from the PM's figure and not in
substance: on a frozen export the same suite is 1424 passed, 1 skipped and 2
export artefacts; in a working tree all 1427 run and pass.
## 2. The root cause, verified rather than inherited
The order states a diagnosis. Each half was checked against the tree before any
code was written, because a premise is an input like any other:
| claim | check | result |
|---|---|---|
| PDF reaches the proposer as a plain text stream | `_pdf_pages` called `page.extract_text()` and nothing else | confirmed |
| the `docx` path emits ATX the proposer already cuts on | `_PANDOC_WRITER = "markdown"`, never `plain` | confirmed |
| the measured best deterministic heading rule is not on the run path | `RULE_POPPLER_SIZE_AND_BOLD` occurs **twice** in `src/`, both in `propose.py` -- its definition and its entry in `RULE_NAMES` -- and **zero times** in `tools/` or `tests/` | confirmed: no candidate has ever carried it |
A PDF has no notion of a heading. A heading in a PDF is a typographic fact, and
the text stream throws that fact away before anything downstream can read it.
## 3. Deliverable A: the font reader, behind `--pdf-headings font`
### The mechanism
`pdfplumber` -- already a dependency -- reports `size` and `fontname` per
character. Per line, the reader takes the dominant size and dominant font name
over the line's non-blank characters. A line is a heading when its dominant
size is **above the document's character-weighted median size** AND its
dominant font name contains `bold`. The heading sizes of the whole document are
ranked largest-first and that rank becomes the ATX level, clamped at six.
Three choices, each with a reason rather than a taste:
- **A conjunction, not a disjunction.** `propose.py` already records the
measurement: `size AND bold` scored recall 1.000 / precision 0.846, and
adding weight as a disjunct took precision from 0.786 to 0.524.
- **The body size is the document's, not the page's.** A title page is 100 %
heading, so a per-page median compares it with itself and marks nothing.
- **The output is ATX in the same markdown the office path produces**, so
`_ATX` reads it unchanged and no PDF-only heading grammar exists. A second
grammar would be a second definition of "heading" free to drift from the one
five office formats already use. The cost is stated in Honesty limits.
### The controls
| control | result |
|---|---|
| a PDF with one font at one size, flag ON | **byte-identical** to flag OFF (test) |
| `.docx` / `.xlsx`, flag ON, whole corpus | **0 of 5** and **0 of 2** changed |
| every readable corpus document: same line count, and every differing line is the cached line with a hash run prefixed | **39 of 39** |
That last control is the load-bearing one and it states three things at once:
the round-3 cache is still valid (`extract.py` is byte-identical between
`c3b645b` and `f6fea13`), joining `extract_text_lines()` reproduces
`extract_text()` on real documents rather than on the one it was spot-checked
on, and the reader **adds markers and changes nothing else**.
**The first version of that control was wrong and said so.** It stripped
`^#{1,6} ` from the marked text and compared: 38 of 39. The one failure is a
corpus document whose own extracted text carries lines beginning with `# `, and
the stripper removed those too. The corrected control compares line by line.
### The reach
Whole corpus, 39 readable documents, one cached extraction per rendering:
| figure | value | denominator |
|---|---|---|
| documents changed | **25** | 32 `pdf` |
| documents changed | **0** | 5 `docx` |
| documents changed | **0** | 2 `xlsx` |
| ATX headings added | **1279** | — |
| `pdf` documents where the reader finds nothing | **7** | 32 |
On the order's spike document the reader reproduces the PM's numbers exactly:
**9 of 9** numbered chapters found on 11 pages in 1.1 s, with **4 extra
candidates** -- two 20 pt title lines and two 16 pt contents lines.
### Per `pdf` position, against the unit worksheet
`N` is the shipped default (`--outline-run 3 --table-grid --unit-fold`); `G` is
that default plus `--pdf-headings font`. Concepts per document, reference from
the operator's worksheet:
| pos | reference | N | **G** | verdict |
|---|---|---|---|---|
| 0 | undefined | 1 | 1 | unchanged (the reader finds nothing in `(cid:N)`) |
| 1 | 15 | 17 | **5** | worse, distance 2 -> 10 |
| 2 | 9 | **9** `treffer` | 12 | **hit lost** |
| 4 | 1 | 4 | 11 | worse, distance 3 -> 10 |
| 6 | 6 | 7 | 11 | worse, distance 1 -> 5 |
| 7 | 30 | 33 | 33 | unchanged |
| 8 | 8 | **8** `treffer` | 16 | **hit lost** |
| 9 | 4 | 10 | 12 | worse, distance 6 -> 8 |
**`pdf` goes 2 of 8 to 0 of 8.** `docx` (3 of 3) and `xlsx` (0 of 1) do not
move, which is what the whole-corpus screen predicts: the reader is PDF-only.
**The mechanism of the loss, so the row is not just a number.** At positions 2
and 8 the outline rule already recovers the document's own numbered chapters,
and the font reader ADDS the title page and the contents lines on top of them --
the four extras above. It is not that typography is wrong; it is that on these
documents the document's own numbering is already right, and a second heading
source can only add.
### The variant that asks the question the other way round
If typography competes with numbering rather than adding to it, the comparison
is the font reader with `--outline-run 0`. Measured, same cache, same criterion:
| configuration | `pdf` | `docx` | `xlsx` | whole sample |
|---|---|---|---|---|
| N, the shipped default | **2 of 8** | 3 of 3 | 0 of 1 | 5 of 12 |
| G = N + font | 0 of 8 | 3 of 3 | 0 of 1 | 3 of 12 |
| X = font instead of Arm D (`--outline-run 0`) | 1 of 8 | 3 of 3 | 0 of 1 | 4 of 12 |
| F = the three round-3 rules on top of N | **3 of 8** | 3 of 3 | 0 of 1 | **6 of 12** |
| H = F + font | 0 of 8 | 3 of 3 | 0 of 1 | 3 of 12 |
**X is the one result worth reading twice.** It is worse overall, and it is the
first configuration in four rounds to land position 7 exactly on its reference
of **30** -- a document no arm has ever reached. Position 7's worksheet answer
(c) is `I`, "no arm is closest", so the criterion's LITERAL gate declares that
any `treffer` there is an instrument failure. **Both readings are stated
because the criterion's author never faced this case:** the gate's clause at
position 7 is a statement about the three arms the operator rated, and a
configuration that did not exist then reaching the reference count is new
information rather than evidence the instrument broke. It is not enough to make
X a default -- X loses positions 2, 6 and 8 to buy it.
## 4. Deliverable B: OCR behind `--ocr`, as an optional group
### The threshold, read off a measured distribution
Every page of every K2 PDF, `(cid:N)` characters as a share of the page's
extracted characters. Denominators: **32 `pdf` files, 834 pages that produced
text** (a page that produced none is not in the table; it is caught by the
other half of the trigger).
| share | pages |
|---|---|
| exactly 0.0 | **818** |
| 0 < s < 1 % | 0 |
| 1 % <= s < 10 % | 0 |
| 10 % <= s < 50 % | 0 |
| 50 % <= s < 90 % | 0 |
| s >= 90 % | **16** |
The distribution is bimodal with **nothing between 0.0 and 0.93**. The lowest
positive share measured is 0.9300 and the highest zero is 0.0.
**`OCR_CID_SHARE = 0.10`, and what happens at plus or minus ten points is not
symmetric.** At 0.20 the rule selects the identical 16 pages -- the gap is 93
percentage points wide, so every threshold inside it is the same rule. At 0.00
the rule stops being a rule: `share >= 0.0` is true of every page, and all 834
would be read as images. The threshold is therefore bounded by an empty region
on one side and by zero on the other, and 0.10 is the middle of the useful
range rather than a calibrated value. **This number is bounded by the corpus,
not by a property of the format**, and a corpus with intermediate pages would
have to set it again.
### The trigger has two conditions because there are two failures
A page is read as an image when its extracted text is **empty** or when its
`(cid:N)` share reaches the threshold. A rule written for one of them leaves
the other exactly where it was: a page with no text layer extracts as the empty
string and has no cid codes at all.
### What it recovers
The one document in the corpus with the failure, through the shipped code path
(`extract_text(..., ocr=True)`) and measured with the shipped instrument
(`tools/okf_cid_measure.measure`, imported rather than re-implemented):
| | before | after |
|---|---|---|
| characters | 217 470 | 34 534 |
| `(cid:N)` characters | 206 758 | **0** |
| cid share | **95.07 %** | **0.00 %** |
| words of 4+ letters | **44** | **2561** |
| pages that produced text | 17 of 18 | **18 of 18** |
| wall time | — | 65.3 s, 3.6 s/page |
The before column reproduces the published known-positive of
`docs/2026-09-04-cid-andel-k2.md` exactly on both figures (95.07 %, 44 words).
**The trigger fired on 16 of the 17 pages that had text, and on the one page
that had none.** The page it did not fire on carries 16 characters and no cid
codes -- text arrived, just very little of it. A length rule would catch it and
there is no measurement for one, so it is reported rather than repaired.
### Position 0 moves from "the text never came" to segmented
| configuration | concepts |
|---|---|
| position 0, shipped default, no OCR | **1** (no plan at all; the proposer finds no boundary) |
| position 0, shipped default, `--ocr` | **4** |
| position 0, the three round-3 rules, `--ocr` | **3** |
**The cell still cannot cross, and that is the reference's doing rather than
the reader's.** The worksheet's answer for position 0 is that no unit can be
decided, so the criterion makes `treffer` unreachable there by construction and
records the dominant error instead. The label moves from `for grovt` to `for
fint`; the `pdf` row is 8 positions either way.
### Why this engine, and what it costs
`rapidocr` on `onnxruntime`, in an optional group named `ocr` and never in
`project.dependencies` -- a packaging test asserts both halves. Install
measured at 31 s into an empty venv on this machine.
Docling was measured first and is out on a platform fact rather than a
preference: it needs torch, and torch stopped publishing macOS x86_64 wheels
after 2.2.2, with no `transformers` version inside Docling's own window that
works against that one.
The models ship **inside the `rapidocr` wheel** -- the run log names files under
`site-packages/rapidocr/models/` and no download step runs -- which matters more
here than it usually would: this library's network access is an explicit
per-run opt-in, and an engine that fetched a model on first use would walk
straight through that gate.
Without the group installed, `--ocr` is a typed refusal with its own code,
`extractor_ocr_group_missing`, distinct from `extractor_extra_missing` because
the `[extract]` extra can be fully installed with only the engine absent. In a
corpus run it is a coded rejection per file, so `merged + coded rejections == N`
still holds; a test pins that.
## 5. The default question, answered on numbers
**`--pdf-headings font` does not become the default.** The order's condition is
that it be at least as good on EVERY `pdf` position and regress nothing else.
It is worse on five positions, it loses two `treffer` cells outright, and the
label row goes 2 of 8 to 0 of 8. `DEFAULT_PDF_HEADINGS = False`, with a test.
**`--ocr` can never become a default**, and that is by construction rather than
by measurement: the engine is an optional dependency, so a default that needed
it would make an ordinary install fail on the first scanned page.
## 6. What round 3 left standing
**6.1 `pdf` position 1 (17 against 15).** The font axis is the first one not
tried here, and it is worse: 17 -> **5**, distance 2 -> 10. Declined for a
third round, now on three axes.
**6.2 positions 6, 7, 9.** Position 6 unchanged (7). Position 7 unchanged at 33
under the default, and reached exactly at **30** under X -- see § 3. Position 9
unchanged at 10 under the default, 8 under X. None of them crosses under a
configuration that does not lose more elsewhere.
**6.3 position 0.** Measured in § 4: 1 -> 4 concepts with `--ocr`, and the cell
still cannot cross because the reference says no unit can be decided.
**6.4 `xlsx` position 3.** Untouched: the font reader changes 0 of 2
spreadsheets. **The reading used here is the criterion as written** -- a
`rule:sheet-section` concept is neither `Tabell linje N` nor
`rule:table-block`/`rule:table-grid`, so the eleven cost groups count as
ordinary units and the distance is **1**, `for fint`. Under the opposite
reading the same plan scores distance 11. Round 3 stated both; this round
changes neither and picks the same one, out loud.
**6.5 the `docx` cell, re-measured on the current default.** Round 3's `docx`
**0 of 3** was measured on a default that no longer exists -- `--table-grid`
entered the default in `f6fea13`. On this HEAD the shipped default scores
`docx` **3 of 3**, as expected. Reported, not reversed.
**6.6 the order's premise here is false, and it is reported rather than acted
on.** The order says K3-3 § 0 promises "a third K2 bundle on the new default"
and that the report "contains the result nowhere". It contains it:
`docs/2026-09-08-k3-runde3-per-filtype.md` has a section titled *"The K2 bundle
on the new default"* whose table carries the row **517 concepts / 969 files /
`42c8646d…`**. Nothing was struck. What IS true, and is the useful half, is
that `42c8646d…` names the round-3 default (Arm D + Arm F) and the default
moved again in `f6fea13`, so the digest is stale. Section 8 supersedes it.
## 7. Per file type, after round 4
The 12-document sample, concepts per configuration, from one cached extraction
per rendering. **The instrument was verified before it was read**: re-run on the
archive plans the operator rated, session 109's criterion returns **LITERAL
7/12, DOMINANCE 11/12** -- its published gates.
| pos | type | reference | N (default) | **G (+font)** | F (round 3) | **H (F+font)** |
|---|---|---|---|---|---|---|
| 0 | pdf | — | 1 | 1 | 1 | 1 |
| 1 | pdf | 15 | 17 | 5 | 17 | 5 |
| 2 | pdf | 9 | **9** | 12 | **9** | 11 |
| 3 | xlsx | 11 | 1 | 1 | 12 | 12 |
| 4 | pdf | 1 | 4 | 11 | **1** | 8 |
| 5 | docx | 3 | **3** | **3** | **3** | **3** |
| 6 | pdf | 6 | 7 | 11 | 7 | 11 |
| 7 | pdf | 30 | 33 | 33 | 33 | 33 |
| 8 | pdf | 8 | **8** | 16 | **8** | 16 |
| 9 | pdf | 4 | 10 | 12 | 6 | 8 |
| 10 | docx | 2 | **2** | **2** | **2** | **2** |
| 11 | docx | 1 | **1** | **1** | **1** | **1** |
**The label row:**
| type | n | Arm B | N (default) | **G (+font)** | F (round 3) | **H (F+font)** |
|---|---|---|---|---|---|---|
| `pdf` | 8 | 2 | **2** | **0** | **3** | **0** |
| `docx` | 3 | 0 | **3** | **3** | **3** | **3** |
| `xlsx` | 1 | 0 | 0 | 0 | 0 | 0 |
| whole sample | 12 | 2 | 5 | 3 | **6** | 3 |
**One divergence between the rebuilt arms and the archive, named rather than
smoothed.** Arms B, D and E were rebuilt on this tree rather than read from the
archive, and they reproduce the archive's cells on 11 of 12 positions. Position
3 differs, and the reason is a known extraction change rather than an arm: the
spreadsheet writer moved in `56ae274`, after the archive plans were written. It
does not touch the N-versus-G comparison, which reads one cache with one code.
## 8. The K2 control
Two K2 bundles from the same 43-document corpus, same `--bundle-id
k2-trinn1-20260903 --okf-version 0.2`, the default epoch stamp, differing only
in the new flag. Both exit 0 and both report the conservation identity.
| | default | `--pdf-headings font` |
|---|---|---|
| `merged + coded rejections` | `39 + 4 = 43 = N` | idem |
| rejection codes | `extractor_empty_pdf` 1, `extractor_unknown` 3 | idem |
| concepts | **492** | 359 |
| markdown files | **944** | 595 |
| content digest | **`bdefa679957edd7d…`** | `a751b48f517f3ed4…` |
| wall time | 819.9 s | 814.7 s |
| concepts reachable from the index tree | 492 of 492 | 359 of 359 |
**The digest a consumer should compare against from today is
`bdefa679957edd7d…`, at 492 concepts in 944 files.** It supersedes round 3's
`42c8646d…` (517 / 969), which named the round-3 default before `--table-grid`
joined it in `f6fea13`. The command is the one above with no flags.
And the ranking, on the S7 control question, `--k 12`:
| | default | `--pdf-headings font` |
|---|---|---|
| priced concept present | yes, title `Prissammenstilling {#sheet-1}` | yes, same title |
| candidate rank, no vocabulary | 434 of 492 | 298 of 359 |
| candidate rank, `--cost-vocabulary` | **159 of 492** | 151 of 359 |
| delivered | **no** | **no** |
| payload bytes, vocabulary on | 123 681 | 95 446 |
**Nothing gets worse and nothing gets better: the flag moves the priced concept
from rank 159 of 492 to 151 of 359 and it is delivered in neither.** The bundle
it produces is 133 concepts smaller, which is a large change to buy nothing.
**One thing in that table is not about this flag and the operator should read
it.** Round 3 published the same question against a 629-concept Arm B bundle:
candidate rank **96 of 629** with the vocabulary on, **delivered at rank 10**.
On the shipped default it is rank 159 of 492 and **not delivered at all**. That
is the default move of `47f6619`/`f6fea13` showing up on the retrieval side for
the first time, not this round's flag -- the two columns above are the
controlled comparison, and they differ only in the flag. It is one question and
one bundle, and it is written down rather than left for the next round to
rediscover.
## 9. Tests
`1427 -> 1441`. Fourteen new, and which of them were red first is stated rather
than counted -- the first run of the new file was **8 failed, 1 passed**, and
the second, after the flags reached `okf build`, **3 failed, 9 passed**:
- **Red before the implementation (11):** the ATX emission and its level
mapping; the proposer segmenting it with no new rule; the known-negative (a
PDF with no font variation is byte-identical with the flag on); the locator
indexing the marked text; the OCR refusal without the group; OCR recovering a
page with no text; OCR leaving a page that has text alone; the cid threshold
and the two-condition trigger; the flag reaching `okf build`; the build-layer
default control; the coded-rejection accounting.
- **Green by construction, and named as such (3):** the "off by default"
extraction control, the new code's entry in the error-code registry suite, and
the `ocr` group's pin. All three were written after the code they describe.
One fixture is new -- `tests/fixtures/font-heading-krav.pdf`, a hand-laid
two-font PDF regenerated by `tests/fixtures/make_fixtures.py`. It gets its own
builder rather than a parameter on the existing one: a second font object
changes the object numbering, and every other PDF fixture's value is that its
bytes have not moved. Regenerating writes all nine fixtures and `git status`
shows only the new one.
No shipped expectation moved. The README's segmentation section did move, and
that is a correction rather than a change: it still said **two** rules were on
by default after `f6fea13` made it three.
## Reproducing
```
# 1. The three reproduction numbers.
pytest -q
okf build ~/okf-test/dokumenter --bundle /tmp/b --bundle-id t --okf-version 0.2
okf build ~/okf-test/dokumenter --bundle /tmp/b2 --bundle-id t --okf-version 0.2 \
--no-table-grid
# 2. The per-page cid distribution, both denominators.
python3 /tmp/k3r4/cidpages.py # 32 pdf files, 834 pages with text
# 3. The two renderings of the whole corpus, one cached extraction each.
python3 /tmp/k3r4/font43.py # writes /tmp/k3r4/font43/
# the control is line-by-line: same line count, every differing line is the
# cached line with a hash run prefixed -> 39 of 39
# 4. The 12-sample, scored with session 109's own criterion.
python3 /tmp/k3r4/plans4.py && python3 /tmp/k3r4/score4.py
python3 /tmp/k3r4/score4b.py # the X and Y variants
# 5. What OCR recovers. Needs the optional group in a venv of its own:
# uv pip install rapidocr onnxruntime pypdfium2 pdfplumber
python3 /tmp/k3r4/ocr91.py
# 6. The K2 pair. Each build is a foreground nohup with an rc file and a
# bounded poll -- the Bash tool caps at 600 s and a build takes ~15-25 min.
/tmp/k3r4/k2.sh default
/tmp/k3r4/k2.sh font --pdf-headings font
```
## Honesty limits
- **A font-inferred heading is indistinguishable from a declared one in the
artifact.** Emitting ATX is what makes `_ATX` apply unchanged, and it is also
what makes the plan say `rule:heading` for a heading no document ever
declared. `RULE_POPPLER_SIZE_AND_BOLD` was deliberately NOT assigned to it:
that name records a poppler measurement on a path that cannot ship, and
putting it on a pdfplumber inference would make the artifact claim a
provenance it does not have. The cost is real and it is the strongest
argument against this design; it is affordable only because the flag is off.
- **n = 12, one rater, one reference.** Per file type the denominators are 8, 3
and 1. The `xlsx` row is a `0/1`, which is not a rate.
- **The OCR denominator is 1.** One document of 39 has the `(cid:N)` failure and
one has no text layer at all. Every claim about what OCR buys is a claim about
those two documents.
- **OCR output is not deterministic across versions.** It is a model's reading
of an image at a rendering resolution. It is reproducible against the model
version and `OCR_DPI` it was produced with, and no dependency pin can promise
more. The `ocr` group therefore uses ranges and says why.
- **`--ocr` and `--pdf-headings font` do not compose on an OCR'd page.** The
engine reports text, not fonts, so a recovered page carries no typography and
can contribute no heading. Both flags together are valid and each does its own
work on its own pages.
- **The cid threshold is bounded by a corpus, not by the format.** Every value
between 0.01 and 0.93 selects the same 16 pages here. A corpus with pages in
between would need the number set again, from its own distribution.
- **The 43-document corpus and the 12-document sample are the same corpus.** The
sample is drawn from it, so the whole-corpus screen is not an independent
check of the sample result.

39
pyproject.toml
View file

@ -88,6 +88,36 @@ okf = "llm_ingestion_okf.cli:main"
# third entry cannot arrive unexamined.
extract = ["pdfplumber>=0.11.10,<0.12", "pypandoc-binary==1.17"]
# The OCR engine for `--ocr`, and NEVER a runtime dependency. It is a separate
# group from `extract` rather than three more entries in it, because it buys
# something categorically different: `extract` decides which file types can be
# read at all, while this one only changes how a PDF page is read when the
# page's own text never arrived. A consumer who installs `[extract]` gets every
# file type; a consumer who never meets a scanned document should never carry
# an inference runtime.
#
# WHY rapidocr ON onnxruntime, and why not the obvious alternative. Docling was
# measured first and is OUT on a platform fact, not a preference: it needs
# torch, and torch stopped publishing macOS x86_64 wheels after 2.2.2, with no
# `transformers` version inside Docling's own window that works against that
# one (4 tried, 2026-09-08). rapidocr on onnxruntime installs and runs on this
# machine, and it carries its ONNX models inside its own wheel, so `--ocr`
# needs no network at run time -- which matters here more than usual, since
# this library's network gate is an explicit per-run opt-in and an engine that
# downloaded a model on first use would walk straight through it.
#
# `pypdfium2` is named although `[extract]` already reaches it through
# pdfplumber: the OCR path RENDERS a page before reading it, and the renderer
# is a dependency of that path rather than a happy accident of another one.
#
# The pins are ranges rather than exact versions, and that is a weaker promise
# than `[extract]` makes on purpose: OCR output is a model's reading of an
# image, so it is deterministic within one model version and NOT across one,
# and no range can make it otherwise. A bundle built with `--ocr` is
# reproducible against the versions it was built with, which is stated in the
# report rather than implied by a pin.
ocr = ["rapidocr>=3.9,<4", "onnxruntime>=1.20,<2", "pypdfium2>=4,<6"]
[dependency-groups]
dev = ["pytest>=8", "mypy>=1.14", "ruff>=0.9"]
@ -129,6 +159,15 @@ ignore_missing_imports = true
module = ["pypandoc", "pypandoc.*"]
ignore_missing_imports = true
# `rapidocr` ships no py.typed marker either, and it is behind an OPTIONAL
# group -- so on a machine without that group installed the import does not
# resolve at all. Only `_ocr_reader` imports it, and the only value that
# crosses back is coerced to `str` there, the same discipline as the two
# overrides above.
[[tool.mypy.overrides]]
module = ["rapidocr", "rapidocr.*"]
ignore_missing_imports = true
# Install CHANNEL for the guard, which is not on a package index yet. It is
# uv-specific, and it reaches further than a dev-only setting: a consumer
# installing this package from git WITH UV picks the guard up from this tag

66
src/llm_ingestion_okf/cli.py
View file

@ -136,6 +136,20 @@ DEFAULT_OUTLINE_RUN = 3
DEFAULT_TABLE_GRID = True
DEFAULT_UNIT_FOLD = True
#: Round 4's two PDF READERS, both off. They are not arms and they are not in
#: the paragraph above: an arm changes how the proposer cuts a text, and these
#: two change what the text SAYS -- so they sit before every arm rather than
#: beside them, and a measurement of an arm is a measurement of it reading
#: whatever these produced.
#:
#: `DEFAULT_PDF_HEADINGS` is off BY MEASUREMENT and the measurement is in
#: `docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`, not by caution.
#: `DEFAULT_OCR` is off BY CONSTRUCTION and can never move: the engine is an
#: optional dependency group, so a default that needed it would make the
#: package's ordinary install fail on the first scanned page.
DEFAULT_PDF_HEADINGS = False
DEFAULT_OCR = False
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -156,6 +170,8 @@ def _propose_plans(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
pdf_headings: bool = False,
ocr: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -186,6 +202,8 @@ def _propose_plans(
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
pdf_headings=pdf_headings,
ocr=ocr,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -215,6 +233,8 @@ def build(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
ocr: bool = DEFAULT_OCR,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -233,7 +253,14 @@ def build(
if proposed_at is None:
proposed_at = ingested_at
if not segments:
report = measure(inbox, bundle, ingested_at=ingested_at, profile=STRUCTURED_V1)
report = measure(
inbox,
bundle,
ingested_at=ingested_at,
profile=STRUCTURED_V1,
pdf_headings=pdf_headings,
ocr=ocr,
)
_write_log(bundle, report, profile=STRUCTURED_V1)
return report
@ -264,6 +291,8 @@ def build(
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
pdf_headings=pdf_headings,
ocr=ocr,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -278,6 +307,8 @@ def build(
plans=plans,
profile=SEGMENTED_OKF_V0_2,
root_frontmatter_values={"okf_version": okf_version, "bundle_id": bundle_id},
pdf_headings=pdf_headings,
ocr=ocr,
)
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
return report
@ -474,6 +505,37 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"and the risk-table rows the operator wrote off"
),
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font"),
default="font" if DEFAULT_PDF_HEADINGS else "none",
help=(
"how a PDF's headings are recovered before any arm reads the text. "
"none (the default) is today's extraction: a PDF carries no heading "
"markup and none is inferred. font infers one from TYPOGRAPHY -- a "
"line whose dominant size is above the document's body median AND "
"whose dominant font name says bold -- and emits it as an ATX "
"heading in the same markdown the office path produces, so the "
"existing heading rule reads it and no PDF-only grammar exists. A "
"CONJUNCTION, measured: adding weight as a disjunct took precision "
"from 0.786 to 0.524"
),
)
build_parser.add_argument(
"--ocr",
action="store_true",
default=DEFAULT_OCR,
help=(
"read a PDF page as an IMAGE when its own text never arrived -- the "
"page extracts empty, or as a page of (cid:N) placeholder codes. "
"Requires the optional 'ocr' group (pip install "
"'llm-ingestion-okf[extract,ocr]'); without it every affected file "
"is a coded rejection, never a crash. Never a default and never "
"able to become one: the engine is an optional dependency. OCR text "
"is a reading of a picture, so it is deterministic within one model "
"version and not across one"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
@ -503,6 +565,8 @@ def main(argv: list[str] | None = None) -> int:
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
pdf_headings=args.pdf_headings == "font",
ocr=args.ocr,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

4
src/llm_ingestion_okf/corpus.py
View file

@ -280,6 +280,8 @@ def measure(
plans: Mapping[str, SegmentationPlan] | None = None,
profile: BundleProfile = STRUCTURED_V1,
root_frontmatter_values: Mapping[str, str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
) -> CorpusReport:
"""Run the corpus through the door and count what happened.
@ -301,6 +303,8 @@ def measure(
profile=profile,
root_frontmatter_values=root_frontmatter_values,
segmentations=plans,
pdf_headings=pdf_headings,
ocr=ocr,
)
elapsed = time.monotonic() - started

5
src/llm_ingestion_okf/errors.py
View file

@ -103,6 +103,11 @@ class ExtractionError(IngestError):
- `extractor_empty_conversion` — the converter returned no text; refused
rather than persisted as an empty concept, for the same reason as
`extractor_empty_pdf`
- `extractor_ocr_group_missing` — a PDF page was to be read with OCR but
the optional `ocr` group is not installed. DISTINCT from
`extractor_extra_missing`: the `[extract]` extra can be fully installed
and the document parsed, with only the OCR engine absent, and one code
for both would send an operator to reinstall what they already have
"""

280
src/llm_ingestion_okf/extract.py
View file

@ -23,10 +23,12 @@ registry's. No guard call and no model call anywhere in this module.
from __future__ import annotations
import collections
import csv
import functools
import io
import re
import statistics
import tempfile
import warnings
import zipfile
@ -237,6 +239,182 @@ def _extra_missing(suffix: str) -> ExtractionError:
)
def _ocr_group_missing() -> ExtractionError:
"""The one rejection for `--ocr` without the optional `ocr` group.
A DIFFERENT code from `extractor_extra_missing`, because it is a different
fact and a different remedy: the `[extract]` extra can be fully installed
-- the document parsed, the pages counted -- and the OCR engine still be
absent. One error naming both would send an operator to reinstall
something they already have.
"""
return ExtractionError(
"reading a PDF page with OCR requires the optional 'ocr' group "
"(pip install 'llm-ingestion-okf[extract,ocr]'), which ships rapidocr "
"on onnxruntime; it is not installed",
code="extractor_ocr_group_missing",
)
#: The literal placeholder `pdfminer.six` (behind `pdfplumber`) emits for a
#: glyph whose font carries no usable ToUnicode mapping. The text is present on
#: the page and unreadable in the extraction -- a failure that looks like
#: success, which is why it needs a measurement rather than an exception.
_CID_CODE = re.compile(r"\(cid:\d+\)")
#: The share of a page's extracted characters that must be `(cid:N)` codes
#: before `--ocr` reads the page as an image instead.
#:
#: MEASURED, not chosen: `docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`
#: reports the per-page distribution over the K2 corpus, and it is bimodal
#: with nothing in between -- one document's pages sit near 1.0 and every other
#: page in the corpus sits at 0.0. Any value in that gap selects the same
#: pages, which is what makes 0.10 defensible and also what makes it
#: uninformative about a corpus that has intermediate pages. Stated rather than
#: implied: this threshold is bounded by the corpus, not by a property of the
#: format.
OCR_CID_SHARE = 0.10
#: The resolution a page is rendered at before it is read as an image.
#: 200 dpi is what the round-4 measurement was taken at; the engine's own
#: preprocessing rescales from there, so this is a floor on how much of the
#: page's detail reaches it rather than a tuning knob. It is part of the output
#: contract in the same way the parser version is: OCR text is deterministic
#: within one resolution and one model version, and across neither.
OCR_DPI = 200
def cid_share(text: str) -> float:
"""The share of `text` made of `(cid:N)` placeholder codes, 0.0 for empty.
Module level and importable: `tools/okf_cid_measure.py` answers the same
question at DOCUMENT level, and two definitions of one metric drift.
"""
if not text:
return 0.0
return sum(len(match.group(0)) for match in _CID_CODE.finditer(text)) / len(text)
def needs_ocr(text: str) -> bool:
"""Whether a page's extracted text is unusable enough to read the image.
TWO conditions, because there are two ways a page's text never arrives and
they look nothing alike: a page with no text layer extracts as the empty
string, and a page whose fonts carry no ToUnicode mapping extracts as a
full page of `(cid:N)`. A trigger written for one of them would leave the
other exactly where it was.
"""
return not text.strip() or cid_share(text) >= OCR_CID_SHARE
def _ocr_reader() -> Callable[[object], list[str]]:
"""The OCR engine, or the typed refusal. The import IS the gate.
Same shape as `_extract_pdf`'s probe and for the same reason: membership in
a suffix set cannot tell whether a package is importable, and this group is
the one a consumer is most likely not to have.
"""
try:
import rapidocr
except ImportError as exc:
raise _ocr_group_missing() from exc
if rapidocr is None: # pragma: no cover - the sys.modules probe in tests
raise _ocr_group_missing()
engine = rapidocr.RapidOCR()
def read(image: object) -> list[str]:
result = engine(image)
# `txts` is None when the detector found nothing at all, which is a
# legitimate answer for a blank page and not an error.
return [str(line) for line in (getattr(result, "txts", None) or ())]
return read
#: Bold as a PDF says it: in the font's NAME (`Helvetica-Bold`,
#: `ABCDEF+Arial-BoldMT`). There is no weight attribute on a character, so the
#: name is the only place a text extractor can read it.
_PDF_BOLD_MARKER = "bold"
#: The deepest ATX level the emitted markdown may use. `_ATX` in `propose.py`
#: reads one to six hashes, and a document with seven distinct heading sizes
#: would otherwise emit a line the proposer reads as body.
_PDF_MAX_HEADING_LEVEL = 6
def _dominant(values: list[str]) -> str:
"""The most frequent value, ties broken by first occurrence.
`Counter.most_common(1)` reduces to `max` over the items in insertion
order, so the tie-break is document order and the result is deterministic
for identical bytes -- which is the property everything downstream is
pinned to.
"""
return collections.Counter(values).most_common(1)[0][0]
def _typography(line: dict[str, object]) -> tuple[float, str] | None:
"""One line's dominant font size and font name, or `None` if it is blank.
Blank characters are excluded from both: a space carries a size and a font
like any other character, and a heading padded with body-sized spaces would
read as body.
"""
chars = [char for char in line["chars"] if str(char["text"]).strip()] # type: ignore[attr-defined]
if not chars:
return None
sizes = [f"{float(char['size']):.1f}" for char in chars]
fonts = [str(char["fontname"]) for char in chars]
return float(_dominant(sizes)), _dominant(fonts)
def _heading_levels(lines: list[tuple[str, float, str]]) -> dict[float, int]:
"""Which font sizes are headings in this document, and at what ATX level.
The rule is the CONJUNCTION this repository already measured: larger than
the body AND bold. `docs/2026-09-07-k3-arm-d.md`'s predecessor measured
size-and-bold from poppler at recall 1.000 / precision 0.846, and measured
that adding weight as a DISJUNCT made precision worse (0.786 -> 0.524). A
disjunction here would mark every emphasised phrase in the body.
The body size is the CHARACTER-weighted median over the whole document, not
the page: a title page is 100 % heading by line count, and a per-page
median would compare it with itself and mark nothing. Weighted by
characters rather than lines for the same reason in miniature -- a document
front-loaded with short lines has a line median that no paragraph shares.
The ATX LEVEL is the size's rank among the heading sizes, largest first, so
a document's own typographic hierarchy survives into the markdown instead
of flattening to one level. Deeper than six is clamped, because `_ATX`
reads six.
"""
weighted: list[float] = []
for text, size, _ in lines:
weighted.extend([size] * len(text.replace(" ", "")))
if not weighted:
return {}
body = statistics.median(weighted)
sizes = {size for _, size, font in lines if size > body and _PDF_BOLD_MARKER in font.lower()}
return {
size: min(rank, _PDF_MAX_HEADING_LEVEL)
for rank, size in enumerate(sorted(sizes, reverse=True), start=1)
}
def _mark_headings(lines: list[tuple[str, float, str]], levels: dict[float, int]) -> str:
"""One page's lines as markdown, the heading sizes carrying their hashes.
BOLD is checked again here rather than folded into the size map: a document
can set a caption in the same size as a heading without setting it bold,
and a map keyed on size alone would promote it.
"""
out: list[str] = []
for text, size, font in lines:
level = levels.get(size) if _PDF_BOLD_MARKER in font.lower() else None
out.append(f"{'#' * level} {text}" if level is not None and text else text)
return "\n".join(out)
# How `_extract_pdf` joins its pages, named because the locator below has to
# reproduce the exact same arithmetic to turn a character offset back into a
# page number. Two constants that must agree, written once.
@ -244,7 +422,9 @@ _PDF_PAGE_SEPARATOR = "\n\n"
@functools.lru_cache(maxsize=1)
def _pdf_pages(data: bytes) -> tuple[tuple[int, str], ...]:
def _pdf_pages(
data: bytes, headings: bool = False, ocr: bool = False
) -> tuple[tuple[int, str], ...]:
"""Every page that produced text, as `(page number, text)`, in page order.
The page NUMBER is 1-based and comes from the document, so a page that
@ -253,20 +433,69 @@ def _pdf_pages(data: bytes) -> tuple[tuple[int, str], ...]:
produced text" and "page 3", and the whole reason a locator is worth
writing down.
Memoised on the bytes with room for exactly one document: extraction and
location are two calls about the same file, back to back, and parsing it
twice would double the PDF cost of every corpus run for nothing. Anything
larger would hold whole documents in memory for no gain, since the caller
never returns to an earlier file.
Memoised on the bytes AND on the two options, with room for exactly one
entry: extraction and location are two calls about the same file with the
same options, back to back, and parsing it twice would double the PDF cost
of every corpus run for nothing. The options are part of the key because
two renderings of one document are two different strings, and a locator
built against the wrong one points at the wrong place with full confidence.
`headings` and `ocr` are INDEPENDENT and compose. With both off this is the
path every byte-pinned golden was measured on, unchanged: the default
branch still calls `page.extract_text()` rather than reassembling the page
from its lines. Measured, the two agree on 11 of 11 pages of a real tender
PDF -- but "agree on the document I tried" is not a contract, so the
default does not depend on it.
"""
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
read = _ocr_reader() if ocr else None
try:
with pdfplumber.open(io.BytesIO(data)) as pdf:
pages = [(page.extract_text() or "").rstrip() for page in pdf.pages]
# PASS ONE. Nothing is emitted here, because the heading rule needs
# a fact about the WHOLE document -- the body's size -- and a page
# cannot supply it. A title page is 100 % heading, and a per-page
# median would compare it with itself and mark nothing.
recovered: list[str | list[tuple[str, float, str]]] = []
for page in pdf.pages:
flat = (page.extract_text() or "").rstrip()
if read is not None and needs_ocr(flat):
# The page's own text is unusable, so it is replaced
# WHOLESALE rather than merged with: a page of `(cid:N)`
# has nothing worth keeping, and interleaving two readings
# of one page would put a guess and a fact in one paragraph
# with no way to tell them apart. An OCR'd page carries no
# typography either -- the engine reports text, not fonts --
# so it is a finished string and never a heading candidate.
recovered.append("\n".join(read(page.to_image(resolution=OCR_DPI).original)))
elif not headings:
recovered.append(flat)
else:
recovered.append(
[
(str(line["text"]), *found)
for line in page.extract_text_lines()
if (found := _typography(line)) is not None
]
)
levels = _heading_levels(
[
line
for page_lines in recovered
if not isinstance(page_lines, str)
for line in page_lines
]
)
# PASS TWO.
pages = [
page_lines
if isinstance(page_lines, str)
else _mark_headings(page_lines, levels).rstrip()
for page_lines in recovered
]
except ExtractionError:
raise
except Exception as exc: # noqa: BLE001 - third-party parser, wrapped never leaked
@ -276,7 +505,7 @@ def _pdf_pages(data: bytes) -> tuple[tuple[int, str], ...]:
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
def _extract_pdf(data: bytes) -> str:
def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.
@ -292,12 +521,13 @@ def _extract_pdf(data: bytes) -> str:
and pymupdf each emit all labels then all values. Re-pairing those is
guesswork, and in a requirements document a wrong pairing looks right.
"""
pages = _pdf_pages(data)
pages = _pdf_pages(data, headings, ocr)
text = _PDF_PAGE_SEPARATOR.join(page for _, page in pages)
if not text:
raise ExtractionError(
"the PDF yielded no text on any page; a scanned or image-only "
"document needs OCR, which this registry does not do",
"document needs OCR, which this registry does only behind the "
"optional 'ocr' group and only when asked",
code="extractor_empty_pdf",
)
# After the parse, not before: a run that produced no text has nothing to
@ -563,11 +793,11 @@ def _line_units(text: str) -> SourceUnits:
return SourceUnits("lines", tuple(starts), tuple(range(1, len(starts) + 1)))
def _pdf_units(data: bytes) -> SourceUnits:
def _pdf_units(data: bytes, headings: bool, ocr: bool) -> SourceUnits:
starts: list[int] = []
numbers: list[int] = []
offset = 0
for number, page in _pdf_pages(data):
for number, page in _pdf_pages(data, headings, ocr):
starts.append(offset)
numbers.append(number)
offset += len(page) + len(_PDF_PAGE_SEPARATOR)
@ -614,7 +844,9 @@ def _spreadsheet_units(text: str) -> SourceUnits | None:
return SourceUnits("rows", tuple(starts), tuple(numbers), tuple(scopes))
def source_units(filename: str, data: bytes, text: str) -> SourceUnits | None:
def source_units(
filename: str, data: bytes, text: str, *, pdf_headings: bool = False, ocr: bool = False
) -> SourceUnits | None:
"""The unit table for one dropped file, or `None` when it has none.
`text` must be what `extract_text` returned for these exact bytes: the
@ -627,7 +859,7 @@ def source_units(filename: str, data: bytes, text: str) -> SourceUnits | None:
"""
suffix = Path(filename).suffix.lower()
if suffix == ".pdf":
return _pdf_units(data)
return _pdf_units(data, pdf_headings, ocr)
if suffix == ".xlsx":
return _spreadsheet_units(text)
if suffix in _CORE_EXTRACTORS or suffix in _PANDOC_FORMATS:
@ -636,7 +868,12 @@ def source_units(filename: str, data: bytes, text: str) -> SourceUnits | None:
def extract_text(
filename: str, data: bytes, *, renderer: Callable[[str], str] | None = None
filename: str,
data: bytes,
*,
renderer: Callable[[str], str] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
) -> str:
"""Convert one dropped file's bytes to OKF concept text, dispatched by type.
@ -657,11 +894,22 @@ def extract_text(
The default is identity, which is what keeps every existing byte-pinned
golden byte-pinned.
`pdf_headings` and `ocr` are PDF-only and both default to off. They are
branched on here rather than expressed as two more registry rows because
the registry's contract is `bytes -> str`: a row per option combination
would be four rows for one reader, and a reader chosen by a suffix lookup
that also has to consult two flags is not a lookup. A non-PDF caller
passing either argument gets today's behaviour, silently, which is correct
-- the options describe a reader, not a policy for the run.
"""
suffix = Path(filename).suffix.lower()
extractor = _CORE_EXTRACTORS.get(suffix) or _OPTIONAL_EXTRACTORS.get(suffix)
if extractor is not None:
text = extractor(data)
if suffix == ".pdf" and (pdf_headings or ocr):
text = _extract_pdf(data, headings=pdf_headings, ocr=ocr)
else:
text = extractor(data)
return renderer(text) if renderer is not None else text
if suffix in _UNPARSED_OPTIONAL_EXTENSIONS:
raise _extra_missing(suffix)

12
src/llm_ingestion_okf/inbox.py
View file

@ -742,6 +742,8 @@ def process_inbox(
root_frontmatter_values: Mapping[str, str] | None = None,
segmentation: SegmentationPlan | None = None,
segmentations: Mapping[str, SegmentationPlan] | None = None,
pdf_headings: bool = False,
ocr: bool = False,
) -> InboxResult:
"""Convert every file dropped in `inbox_dir` into an OKF concept.
@ -957,6 +959,8 @@ def process_inbox(
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
pdf_headings=pdf_headings,
ocr=ocr,
)
# Computed from the SAME text the plan's offsets index, so the
# locator and the offset can never disagree about which rendering
@ -964,7 +968,13 @@ def process_inbox(
# building a unit table nobody writes would re-parse every PDF for
# a key that is never emitted.
units = (
source_units(source_name(path), source_bytes, text)
source_units(
source_name(path),
source_bytes,
text,
pdf_headings=pdf_headings,
ocr=ocr,
)
if profile.provenance is not None
else None
)

10
src/llm_ingestion_okf/propose.py
View file

@ -1118,6 +1118,8 @@ def run(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
pdf_headings: bool = False,
ocr: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -1155,7 +1157,13 @@ def run(
except OSError as exc:
raise ProposerError(f"cannot read {source}: {exc}") from exc
try:
text = extract_text(source.name, source_bytes)
# The two READER options, not arms: they change what the extraction
# says, and every arm below reads whatever it says. Passed here as well
# as to the run path because the plan's `text_sha256` indexes this
# exact string -- a plan proposed against one rendering and replayed
# against another is refused by `assert_plan_applies`, which is the
# right outcome and a confusing one to debug.
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
except IngestError as exc:
raise ProposerError(f"cannot extract text from {source.name}: {exc}") from exc

39
tests/fixtures/font-heading-krav.pdf vendored Normal file
View file

@ -0,0 +1,39 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R /F2 6 0 R >> >> >>
endobj
4 0 obj
<< /Length 221 >>
stream
BT /F2 20 Tf 50 700 Td (Generelle tekniske krav) Tj ET
BT /F1 10 Tf 50 670 Td (Utkilingen skal ha helning 1:15.) Tj ET
BT /F2 14 Tf 50 640 Td (Merking) Tj ET
BT /F1 10 Tf 50 610 Td (Kravet gjelder alle veiklasser.) Tj ET
endstream
endobj
5 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
endobj
6 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding >>
endobj
xref
0 7
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000251 00000 n
0000000522 00000 n
0000000619 00000 n
trailer
<< /Size 7 /Root 1 0 R >>
startxref
721
%%EOF

54
tests/fixtures/make_fixtures.py vendored
View file

@ -51,6 +51,57 @@ PAGED_CONTENTS = (
)
# Two fonts and three sizes on one page: a 20pt bold title, a 14pt bold
# subheading, and 10pt regular body. The PDF format carries no notion of a
# heading at all -- a heading in a PDF is a typographic fact, which is why the
# font-aware reader has to infer one -- so a fixture for that reader must state
# the typography and nothing else. The body is the majority of the characters,
# which is what gives the reader a body size to compare against.
FONT_HEADING_CONTENT = (
b"BT /F2 20 Tf 50 700 Td (Generelle tekniske krav) Tj ET\n"
b"BT /F1 10 Tf 50 670 Td (Utkilingen skal ha helning 1:15.) Tj ET\n"
b"BT /F2 14 Tf 50 640 Td (Merking) Tj ET\n"
b"BT /F1 10 Tf 50 610 Td (Kravet gjelder alle veiklasser.) Tj ET\n"
)
def build_two_font_pdf(content: bytes) -> bytes:
"""A one-page PDF whose resources declare BOTH a regular and a bold font.
Separate from `build_paged_pdf` rather than a parameter on it: that builder
emits exactly one font object and every existing fixture's bytes depend on
its object numbering. A second font changes the numbering, so sharing the
code would mean regenerating files whose whole value is that they have not
moved.
The page is Letter-sized rather than the 200x200 the other fixtures use,
because a 20pt line of this length does not fit inside 200 points and a
character laid outside the page box is not one a reader has to see.
"""
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R "
b"/Resources << /Font << /F1 5 0 R /F2 6 0 R >> >> >>",
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding >>",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for number, body in enumerate(objects, start=1):
offsets.append(len(out))
out += str(number).encode() + b" 0 obj\n" + body + b"\nendobj\n"
xref_at = len(out)
size = str(len(objects) + 1).encode()
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
for offset in offsets:
out += ("%010d 00000 n \n" % offset).encode()
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
return bytes(out)
def build_pdf(content: bytes) -> bytes:
"""Assemble a one-page PDF around `content` as the page content stream."""
return build_paged_pdf((content,))
@ -365,6 +416,9 @@ if __name__ == "__main__":
(HERE / "three-page-krav.pdf").write_bytes(build_paged_pdf(PAGED_CONTENTS))
print("wrote three-page-krav.pdf")
(HERE / "font-heading-krav.pdf").write_bytes(build_two_font_pdf(FONT_HEADING_CONTENT))
print("wrote font-heading-krav.pdf")
for name, parts in (
("two-line-krav.docx", _DOCX_PARTS),
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),

13
tests/test_error_codes.py
View file

@ -392,6 +392,19 @@ def test_converter_code_is_registered_and_carried(code: str) -> None:
assert ExtractionError("x", code=code).code == code
def test_extractor_ocr_group_missing(monkeypatch: pytest.MonkeyPatch) -> None:
"""The OCR engine's own absence, which is not the extra's absence.
Reached through the import probe for the same reason as the code above:
`None` in `sys.modules` is what CPython treats as a failed import, so the
test states the same thing on a machine where the group IS installed.
"""
monkeypatch.setitem(sys.modules, "rapidocr", None)
with pytest.raises(ExtractionError) as excinfo:
extract_text("scan.pdf", (FIXTURES / "no-text-layer.pdf").read_bytes(), ocr=True)
assert code_of(excinfo) == "extractor_ocr_group_missing"
@requires_extract
def test_extractor_empty_pdf() -> None:
with pytest.raises(ExtractionError) as excinfo:

21
tests/test_packaging.py
View file

@ -57,6 +57,27 @@ def test_the_extract_extra_pins_exactly_what_it_ships() -> None:
]
def test_the_ocr_group_is_pinned_and_is_not_a_runtime_dependency() -> None:
"""An inference runtime is the last thing that may arrive by accident.
Two claims, and the second is the one worth a test: the group's contents
are pinned like the extra's, AND none of them appears in
`project.dependencies`. The single-dependency test above would already
catch that, but it reads the list and this reads the names -- so a future
entry named differently still fails here.
"""
tomllib = pytest.importorskip("tomllib")
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert pyproject["project"]["optional-dependencies"]["ocr"] == [
"rapidocr>=3.9,<4",
"onnxruntime>=1.20,<2",
"pypdfium2>=4,<6",
]
runtime = " ".join(pyproject["project"]["dependencies"])
for package in ("rapidocr", "onnxruntime", "pypdfium2"):
assert package not in runtime
def test_the_declared_version_agrees_with_the_packaged_one() -> None:
"""The two places a version is written must not drift apart.

327
tests/test_pdf_font_and_ocr.py Normal file
View file

@ -0,0 +1,327 @@
"""Round 4's two PDF readers: font-aware headings, and OCR behind an extra.
Both are OFF by default and both are about the same hole. A PDF carries no
notion of a heading -- a heading in a PDF is a typographic fact -- so the text
stream `pdfplumber` hands over has thrown away the only evidence there was, and
the segment proposer downstream sees a wall of prose. The `docx` path never had
that problem, because the converter emits ATX headings the proposer already
reads.
So the font reader's output is ATX in the SAME markdown the `docx` path
produces. `_ATX` applies unchanged and no new segmentation rule exists; the
tests below pin exactly that, because the alternative -- a PDF-only heading
grammar in `propose.py` -- would be a second definition of "heading" that can
drift from the one the office path already uses.
The OCR half is the other end: a page whose text never arrived at all, either
as nothing or as `(cid:N)` placeholder codes. Its engine is an OPTIONAL
dependency group and the tests here never require it -- the refusal is
exercised through the import probe (the module set to `None` in `sys.modules`,
which is what CPython treats as a failed import), and the success path through
an injected fake engine. That splits the two claims on purpose: the plumbing is
tested here, the reading quality is measured in
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md` against a real document.
"""
from __future__ import annotations
import sys
import types
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import extract as extract_module
from llm_ingestion_okf.errors import ExtractionError
from llm_ingestion_okf.extract import extract_text, source_units
from llm_ingestion_okf.propose import _ATX, find_candidates
FIXTURES = Path(__file__).parent / "fixtures"
FONT_PDF = FIXTURES / "font-heading-krav.pdf"
FLAT_PDF = FIXTURES / "two-line-krav.pdf"
NO_TEXT_PDF = FIXTURES / "no-text-layer.pdf"
def _read(path: Path) -> bytes:
return path.read_bytes()
def _extract(path: Path, **kwargs: object) -> str:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
return extract_text(path.name, _read(path), **kwargs) # type: ignore[arg-type]
# --- A: font-aware headings --------------------------------------------------
def test_the_font_reader_is_off_by_default() -> None:
"""The default extraction is what it was: no heading markers anywhere.
Stated as a test rather than trusted to the signature default, because
every byte-pinned golden in this suite depends on it.
"""
text = _extract(FONT_PDF)
assert text == (
"Generelle tekniske krav\n"
"Utkilingen skal ha helning 1:15.\n"
"Merking\n"
"Kravet gjelder alle veiklasser."
)
assert "#" not in text
def test_font_headings_emit_atx_the_proposer_already_reads() -> None:
"""Size class becomes ATX level, largest first, body untouched."""
text = _extract(FONT_PDF, pdf_headings=True)
assert text == (
"# Generelle tekniske krav\n"
"Utkilingen skal ha helning 1:15.\n"
"## Merking\n"
"Kravet gjelder alle veiklasser."
)
matched = [line for line in text.split("\n") if _ATX.match(line)]
assert matched == ["# Generelle tekniske krav", "## Merking"]
def test_the_proposer_segments_the_font_headings_with_no_new_rule() -> None:
"""The whole point of emitting ATX: `find_candidates` needs no argument.
Arm B -- every arm flag off -- is what runs here, so a candidate appearing
is the heading rule reading the extraction, not an arm reading a PDF.
"""
flat = find_candidates(_extract(FONT_PDF))
marked = find_candidates(_extract(FONT_PDF, pdf_headings=True))
assert [candidate.title for candidate in flat] == []
assert [candidate.title for candidate in marked] == ["Generelle tekniske krav", "Merking"]
assert {candidate.rule for candidate in marked} == {"rule:heading"}
def test_a_pdf_without_font_variation_is_byte_identical_with_the_flag_on() -> None:
"""The known-negative, and the reason the rule is a CONJUNCTION.
`two-line-krav.pdf` is one font at one size. Nothing is larger than the
body median and nothing is bold, so the reader must return the same bytes
it returns with the flag off -- a rule that fired here would mark the first
line of every flat document as a chapter.
"""
assert _extract(FLAT_PDF, pdf_headings=True) == _extract(FLAT_PDF)
def test_the_locator_indexes_the_text_the_flag_produced() -> None:
"""A unit table built against the other rendering points at the wrong place.
`source_units` re-derives page offsets from the parse, so it has to be told
which rendering it is describing. Without the argument the page start would
be computed from an unmarked page and land mid-heading in a marked one.
"""
data = _read(FONT_PDF)
text = _extract(FONT_PDF, pdf_headings=True)
units = source_units(FONT_PDF.name, data, text, pdf_headings=True)
assert units is not None
assert units.unit == "pages"
assert units.starts == (0,)
assert text[units.starts[0] :].startswith("# Generelle tekniske krav")
# --- B: OCR behind the optional group ---------------------------------------
def test_ocr_without_the_optional_group_refuses_with_a_code(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A missing engine is a typed refusal, never a traceback.
Reached through the import probe rather than through the absence of the
package, so the test states the same thing on a machine where the group IS
installed. `None` in `sys.modules` is what CPython treats as a failed
import.
"""
monkeypatch.setitem(sys.modules, "rapidocr", None)
with pytest.raises(ExtractionError) as excinfo:
_extract(NO_TEXT_PDF, ocr=True)
assert excinfo.value.code == "extractor_ocr_group_missing"
message = str(excinfo.value)
assert "ocr" in message
assert "rapidocr" in message
def _fake_engine(lines: tuple[str, ...]) -> types.ModuleType:
"""A `rapidocr` stand-in returning fixed lines, so the SEAM is testable.
The engine's reading quality is not this suite's claim and cannot be: it
would need the optional group, a real scan, and a judgement about what the
page says. What IS this suite's claim is that the trigger fires on the
right pages, that the lines land in the page's place, and that nothing else
moves -- all of which a fixed reader states better than a real one.
"""
class _Result:
txts = lines
module = types.ModuleType("rapidocr")
module.RapidOCR = lambda *args, **kwargs: lambda image: _Result() # type: ignore[attr-defined]
return module
def test_ocr_recovers_a_page_that_produced_no_text(monkeypatch: pytest.MonkeyPatch) -> None:
"""Without the flag this document is `extractor_empty_pdf`; with it, text."""
monkeypatch.setitem(sys.modules, "rapidocr", _fake_engine(("Krav til helning", "1:15")))
extract_module._pdf_pages.cache_clear()
with pytest.raises(ExtractionError) as excinfo:
_extract(NO_TEXT_PDF)
assert excinfo.value.code == "extractor_empty_pdf"
assert _extract(NO_TEXT_PDF, ocr=True) == "Krav til helning\n1:15"
def test_ocr_leaves_a_page_that_already_has_text_alone(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The trigger is a measurement about the page, not a mode for the file.
A readable page must come back byte-identical with `--ocr` on, or the flag
would silently replace a document's own text with a guess about a picture
of it.
"""
monkeypatch.setitem(sys.modules, "rapidocr", _fake_engine(("WRONG",)))
extract_module._pdf_pages.cache_clear()
assert _extract(FLAT_PDF, ocr=True) == _extract(FLAT_PDF)
def test_the_cid_trigger_is_a_share_of_the_page_and_has_a_measured_floor() -> None:
"""The threshold is a number with a measurement behind it, not a taste.
Pinned here so that moving it is a red test and a decision, and checked
against the two shapes it has to separate: a page of placeholder codes and
a page of prose.
"""
assert extract_module.OCR_CID_SHARE == 0.10
assert extract_module.cid_share("(cid:3)(cid:4)(cid:5)") == 1.0
assert extract_module.cid_share("Krav til helning på utkilingen") == 0.0
assert extract_module.cid_share("") == 0.0
assert extract_module.needs_ocr("") is True
assert extract_module.needs_ocr(" \n ") is True
assert extract_module.needs_ocr("Krav til helning") is False
# --- both flags, reached from `okf build` ------------------------------------
#: A second document that ALWAYS yields a plan, so the run below is a run.
#: `okf build --segments on` refuses a corpus where nothing was proposed -- a
#: build asked to replay plans and given none would write a flat bundle and
#: report success -- so an inbox holding only the unsegmentable PDF would exit
#: 2 for a reason that has nothing to do with either flag.
ANCHOR = "# 1 Anker\n\nDette avsnittet finnes for at kjoeringen skal ha en plan.\n"
def _inbox(root: Path) -> Path:
inbox = root / "inbox"
inbox.mkdir(parents=True, exist_ok=True)
(inbox / "anker.md").write_text(ANCHOR, encoding="utf-8", newline="")
(inbox / FONT_PDF.name).write_bytes(_read(FONT_PDF))
return inbox
def _plan_titles(plans: Path) -> list[str]:
import json
return [
entry["title"]
for path in sorted(plans.glob("*.json"))
for entry in json.loads(path.read_text(encoding="utf-8"))["entries"]
]
def test_pdf_headings_reaches_the_proposer_from_the_build_command(tmp_path: Path) -> None:
"""The red test for round 4's A half: the flag exists and changes the plan.
Asserted on the plan's TITLES rather than a count, for the reason every
round-3 reach test is: a reader that cut in the wrong places would satisfy
a count assertion exactly as well.
ONE title from the PDF, not two, and that is the shipped default doing its
job rather than the reader failing: Arm F folds a deeper heading into its
parent, and `## Merking` is deeper than `# Generelle tekniske krav`. The
reader's own output is both -- `test_font_headings_emit_atx...` above pins
that -- and this is what the default does with it.
"""
from llm_ingestion_okf import cli
plans = tmp_path / "plans"
assert (
cli.main(
[
"build",
str(_inbox(tmp_path)),
"--bundle",
str(tmp_path / "bundle"),
"--bundle-id",
"font-fixture",
"--okf-version",
"0.2",
"--plans-dir",
str(plans),
"--pdf-headings",
"font",
]
)
== 0
)
assert _plan_titles(plans) == ["1 Anker", "Generelle tekniske krav"]
def test_the_build_default_leaves_the_font_pdf_unsegmented(tmp_path: Path) -> None:
"""The control the test above rests on: same inbox, no flag, no plan.
Without it, a change that turned the reader on by default would leave the
assertion above green while moving every PDF bundle a consumer has built.
"""
from llm_ingestion_okf import cli
plans = tmp_path / "plans-plain"
assert (
cli.main(
[
"build",
str(_inbox(tmp_path)),
"--bundle",
str(tmp_path / "bundle-plain"),
"--bundle-id",
"font-fixture",
"--okf-version",
"0.2",
"--plans-dir",
str(plans),
]
)
== 0
)
assert _plan_titles(plans) == ["1 Anker"]
assert cli.DEFAULT_PDF_HEADINGS is False
assert cli.DEFAULT_OCR is False
def test_ocr_without_the_group_is_a_coded_rejection_not_a_crash(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A missing engine must not take the run down; it accounts for the file.
`merged + coded rejections == N` is the conservation identity every corpus
run reports, and an engine nobody installed is exactly the kind of failure
that would otherwise leave a document unaccounted for.
"""
from llm_ingestion_okf import cli
monkeypatch.setitem(sys.modules, "rapidocr", None)
extract_module._pdf_pages.cache_clear()
report = cli.build(
_inbox(tmp_path),
tmp_path / "bundle-ocr",
bundle_id="font-fixture",
okf_version="0.2",
ocr=True,
)
assert dict(report.codes) == {"extractor_ocr_group_missing": 1}
assert report.merged + report.rejected == report.n == 2