open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off

Browse source 
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.

`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.

It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).

`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.

Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.

Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 23:10:47 +02:00
commit 53d5c74c96
15 changed files with 1394 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

39
tests/fixtures/font-heading-krav.pdf vendored Normal file
View file

@ -0,0 +1,39 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R /F2 6 0 R >> >> >>
endobj
4 0 obj
<< /Length 221 >>
stream
BT /F2 20 Tf 50 700 Td (Generelle tekniske krav) Tj ET
BT /F1 10 Tf 50 670 Td (Utkilingen skal ha helning 1:15.) Tj ET
BT /F2 14 Tf 50 640 Td (Merking) Tj ET
BT /F1 10 Tf 50 610 Td (Kravet gjelder alle veiklasser.) Tj ET
endstream
endobj
5 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
endobj
6 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding >>
endobj
xref
0 7
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000251 00000 n
0000000522 00000 n
0000000619 00000 n
trailer
<< /Size 7 /Root 1 0 R >>
startxref
721
%%EOF

54
tests/fixtures/make_fixtures.py vendored
View file

@ -51,6 +51,57 @@ PAGED_CONTENTS = (
)
# Two fonts and three sizes on one page: a 20pt bold title, a 14pt bold
# subheading, and 10pt regular body. The PDF format carries no notion of a
# heading at all -- a heading in a PDF is a typographic fact, which is why the
# font-aware reader has to infer one -- so a fixture for that reader must state
# the typography and nothing else. The body is the majority of the characters,
# which is what gives the reader a body size to compare against.
FONT_HEADING_CONTENT = (
b"BT /F2 20 Tf 50 700 Td (Generelle tekniske krav) Tj ET\n"
b"BT /F1 10 Tf 50 670 Td (Utkilingen skal ha helning 1:15.) Tj ET\n"
b"BT /F2 14 Tf 50 640 Td (Merking) Tj ET\n"
b"BT /F1 10 Tf 50 610 Td (Kravet gjelder alle veiklasser.) Tj ET\n"
)
def build_two_font_pdf(content: bytes) -> bytes:
"""A one-page PDF whose resources declare BOTH a regular and a bold font.
Separate from `build_paged_pdf` rather than a parameter on it: that builder
emits exactly one font object and every existing fixture's bytes depend on
its object numbering. A second font changes the numbering, so sharing the
code would mean regenerating files whose whole value is that they have not
moved.
The page is Letter-sized rather than the 200x200 the other fixtures use,
because a 20pt line of this length does not fit inside 200 points and a
character laid outside the page box is not one a reader has to see.
"""
objects = [
b"<< /Type /Catalog /Pages 2 0 R >>",
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R "
b"/Resources << /Font << /F1 5 0 R /F2 6 0 R >> >> >>",
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding >>",
]
out = bytearray(b"%PDF-1.4\n")
offsets = []
for number, body in enumerate(objects, start=1):
offsets.append(len(out))
out += str(number).encode() + b" 0 obj\n" + body + b"\nendobj\n"
xref_at = len(out)
size = str(len(objects) + 1).encode()
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
for offset in offsets:
out += ("%010d 00000 n \n" % offset).encode()
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
return bytes(out)
def build_pdf(content: bytes) -> bytes:
"""Assemble a one-page PDF around `content` as the page content stream."""
return build_paged_pdf((content,))
@ -365,6 +416,9 @@ if __name__ == "__main__":
(HERE / "three-page-krav.pdf").write_bytes(build_paged_pdf(PAGED_CONTENTS))
print("wrote three-page-krav.pdf")
(HERE / "font-heading-krav.pdf").write_bytes(build_two_font_pdf(FONT_HEADING_CONTENT))
print("wrote font-heading-krav.pdf")
for name, parts in (
("two-line-krav.docx", _DOCX_PARTS),
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),

13
tests/test_error_codes.py
View file

@ -392,6 +392,19 @@ def test_converter_code_is_registered_and_carried(code: str) -> None:
assert ExtractionError("x", code=code).code == code
def test_extractor_ocr_group_missing(monkeypatch: pytest.MonkeyPatch) -> None:
"""The OCR engine's own absence, which is not the extra's absence.
Reached through the import probe for the same reason as the code above:
`None` in `sys.modules` is what CPython treats as a failed import, so the
test states the same thing on a machine where the group IS installed.
"""
monkeypatch.setitem(sys.modules, "rapidocr", None)
with pytest.raises(ExtractionError) as excinfo:
extract_text("scan.pdf", (FIXTURES / "no-text-layer.pdf").read_bytes(), ocr=True)
assert code_of(excinfo) == "extractor_ocr_group_missing"
@requires_extract
def test_extractor_empty_pdf() -> None:
with pytest.raises(ExtractionError) as excinfo:

21
tests/test_packaging.py
View file

@ -57,6 +57,27 @@ def test_the_extract_extra_pins_exactly_what_it_ships() -> None:
]
def test_the_ocr_group_is_pinned_and_is_not_a_runtime_dependency() -> None:
"""An inference runtime is the last thing that may arrive by accident.
Two claims, and the second is the one worth a test: the group's contents
are pinned like the extra's, AND none of them appears in
`project.dependencies`. The single-dependency test above would already
catch that, but it reads the list and this reads the names -- so a future
entry named differently still fails here.
"""
tomllib = pytest.importorskip("tomllib")
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
assert pyproject["project"]["optional-dependencies"]["ocr"] == [
"rapidocr>=3.9,<4",
"onnxruntime>=1.20,<2",
"pypdfium2>=4,<6",
]
runtime = " ".join(pyproject["project"]["dependencies"])
for package in ("rapidocr", "onnxruntime", "pypdfium2"):
assert package not in runtime
def test_the_declared_version_agrees_with_the_packaged_one() -> None:
"""The two places a version is written must not drift apart.

327
tests/test_pdf_font_and_ocr.py Normal file
View file

@ -0,0 +1,327 @@
"""Round 4's two PDF readers: font-aware headings, and OCR behind an extra.
Both are OFF by default and both are about the same hole. A PDF carries no
notion of a heading -- a heading in a PDF is a typographic fact -- so the text
stream `pdfplumber` hands over has thrown away the only evidence there was, and
the segment proposer downstream sees a wall of prose. The `docx` path never had
that problem, because the converter emits ATX headings the proposer already
reads.
So the font reader's output is ATX in the SAME markdown the `docx` path
produces. `_ATX` applies unchanged and no new segmentation rule exists; the
tests below pin exactly that, because the alternative -- a PDF-only heading
grammar in `propose.py` -- would be a second definition of "heading" that can
drift from the one the office path already uses.
The OCR half is the other end: a page whose text never arrived at all, either
as nothing or as `(cid:N)` placeholder codes. Its engine is an OPTIONAL
dependency group and the tests here never require it -- the refusal is
exercised through the import probe (the module set to `None` in `sys.modules`,
which is what CPython treats as a failed import), and the success path through
an injected fake engine. That splits the two claims on purpose: the plumbing is
tested here, the reading quality is measured in
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md` against a real document.
"""
from __future__ import annotations
import sys
import types
import warnings
from pathlib import Path
import pytest
from llm_ingestion_okf import extract as extract_module
from llm_ingestion_okf.errors import ExtractionError
from llm_ingestion_okf.extract import extract_text, source_units
from llm_ingestion_okf.propose import _ATX, find_candidates
FIXTURES = Path(__file__).parent / "fixtures"
FONT_PDF = FIXTURES / "font-heading-krav.pdf"
FLAT_PDF = FIXTURES / "two-line-krav.pdf"
NO_TEXT_PDF = FIXTURES / "no-text-layer.pdf"
def _read(path: Path) -> bytes:
return path.read_bytes()
def _extract(path: Path, **kwargs: object) -> str:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
return extract_text(path.name, _read(path), **kwargs) # type: ignore[arg-type]
# --- A: font-aware headings --------------------------------------------------
def test_the_font_reader_is_off_by_default() -> None:
"""The default extraction is what it was: no heading markers anywhere.
Stated as a test rather than trusted to the signature default, because
every byte-pinned golden in this suite depends on it.
"""
text = _extract(FONT_PDF)
assert text == (
"Generelle tekniske krav\n"
"Utkilingen skal ha helning 1:15.\n"
"Merking\n"
"Kravet gjelder alle veiklasser."
)
assert "#" not in text
def test_font_headings_emit_atx_the_proposer_already_reads() -> None:
"""Size class becomes ATX level, largest first, body untouched."""
text = _extract(FONT_PDF, pdf_headings=True)
assert text == (
"# Generelle tekniske krav\n"
"Utkilingen skal ha helning 1:15.\n"
"## Merking\n"
"Kravet gjelder alle veiklasser."
)
matched = [line for line in text.split("\n") if _ATX.match(line)]
assert matched == ["# Generelle tekniske krav", "## Merking"]
def test_the_proposer_segments_the_font_headings_with_no_new_rule() -> None:
"""The whole point of emitting ATX: `find_candidates` needs no argument.
Arm B -- every arm flag off -- is what runs here, so a candidate appearing
is the heading rule reading the extraction, not an arm reading a PDF.
"""
flat = find_candidates(_extract(FONT_PDF))
marked = find_candidates(_extract(FONT_PDF, pdf_headings=True))
assert [candidate.title for candidate in flat] == []
assert [candidate.title for candidate in marked] == ["Generelle tekniske krav", "Merking"]
assert {candidate.rule for candidate in marked} == {"rule:heading"}
def test_a_pdf_without_font_variation_is_byte_identical_with_the_flag_on() -> None:
"""The known-negative, and the reason the rule is a CONJUNCTION.
`two-line-krav.pdf` is one font at one size. Nothing is larger than the
body median and nothing is bold, so the reader must return the same bytes
it returns with the flag off -- a rule that fired here would mark the first
line of every flat document as a chapter.
"""
assert _extract(FLAT_PDF, pdf_headings=True) == _extract(FLAT_PDF)
def test_the_locator_indexes_the_text_the_flag_produced() -> None:
"""A unit table built against the other rendering points at the wrong place.
`source_units` re-derives page offsets from the parse, so it has to be told
which rendering it is describing. Without the argument the page start would
be computed from an unmarked page and land mid-heading in a marked one.
"""
data = _read(FONT_PDF)
text = _extract(FONT_PDF, pdf_headings=True)
units = source_units(FONT_PDF.name, data, text, pdf_headings=True)
assert units is not None
assert units.unit == "pages"
assert units.starts == (0,)
assert text[units.starts[0] :].startswith("# Generelle tekniske krav")
# --- B: OCR behind the optional group ---------------------------------------
def test_ocr_without_the_optional_group_refuses_with_a_code(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A missing engine is a typed refusal, never a traceback.
Reached through the import probe rather than through the absence of the
package, so the test states the same thing on a machine where the group IS
installed. `None` in `sys.modules` is what CPython treats as a failed
import.
"""
monkeypatch.setitem(sys.modules, "rapidocr", None)
with pytest.raises(ExtractionError) as excinfo:
_extract(NO_TEXT_PDF, ocr=True)
assert excinfo.value.code == "extractor_ocr_group_missing"
message = str(excinfo.value)
assert "ocr" in message
assert "rapidocr" in message
def _fake_engine(lines: tuple[str, ...]) -> types.ModuleType:
"""A `rapidocr` stand-in returning fixed lines, so the SEAM is testable.
The engine's reading quality is not this suite's claim and cannot be: it
would need the optional group, a real scan, and a judgement about what the
page says. What IS this suite's claim is that the trigger fires on the
right pages, that the lines land in the page's place, and that nothing else
moves -- all of which a fixed reader states better than a real one.
"""
class _Result:
txts = lines
module = types.ModuleType("rapidocr")
module.RapidOCR = lambda *args, **kwargs: lambda image: _Result() # type: ignore[attr-defined]
return module
def test_ocr_recovers_a_page_that_produced_no_text(monkeypatch: pytest.MonkeyPatch) -> None:
"""Without the flag this document is `extractor_empty_pdf`; with it, text."""
monkeypatch.setitem(sys.modules, "rapidocr", _fake_engine(("Krav til helning", "1:15")))
extract_module._pdf_pages.cache_clear()
with pytest.raises(ExtractionError) as excinfo:
_extract(NO_TEXT_PDF)
assert excinfo.value.code == "extractor_empty_pdf"
assert _extract(NO_TEXT_PDF, ocr=True) == "Krav til helning\n1:15"
def test_ocr_leaves_a_page_that_already_has_text_alone(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The trigger is a measurement about the page, not a mode for the file.
A readable page must come back byte-identical with `--ocr` on, or the flag
would silently replace a document's own text with a guess about a picture
of it.
"""
monkeypatch.setitem(sys.modules, "rapidocr", _fake_engine(("WRONG",)))
extract_module._pdf_pages.cache_clear()
assert _extract(FLAT_PDF, ocr=True) == _extract(FLAT_PDF)
def test_the_cid_trigger_is_a_share_of_the_page_and_has_a_measured_floor() -> None:
"""The threshold is a number with a measurement behind it, not a taste.
Pinned here so that moving it is a red test and a decision, and checked
against the two shapes it has to separate: a page of placeholder codes and
a page of prose.
"""
assert extract_module.OCR_CID_SHARE == 0.10
assert extract_module.cid_share("(cid:3)(cid:4)(cid:5)") == 1.0
assert extract_module.cid_share("Krav til helning på utkilingen") == 0.0
assert extract_module.cid_share("") == 0.0
assert extract_module.needs_ocr("") is True
assert extract_module.needs_ocr(" \n ") is True
assert extract_module.needs_ocr("Krav til helning") is False
# --- both flags, reached from `okf build` ------------------------------------
#: A second document that ALWAYS yields a plan, so the run below is a run.
#: `okf build --segments on` refuses a corpus where nothing was proposed -- a
#: build asked to replay plans and given none would write a flat bundle and
#: report success -- so an inbox holding only the unsegmentable PDF would exit
#: 2 for a reason that has nothing to do with either flag.
ANCHOR = "# 1 Anker\n\nDette avsnittet finnes for at kjoeringen skal ha en plan.\n"
def _inbox(root: Path) -> Path:
inbox = root / "inbox"
inbox.mkdir(parents=True, exist_ok=True)
(inbox / "anker.md").write_text(ANCHOR, encoding="utf-8", newline="")
(inbox / FONT_PDF.name).write_bytes(_read(FONT_PDF))
return inbox
def _plan_titles(plans: Path) -> list[str]:
import json
return [
entry["title"]
for path in sorted(plans.glob("*.json"))
for entry in json.loads(path.read_text(encoding="utf-8"))["entries"]
]
def test_pdf_headings_reaches_the_proposer_from_the_build_command(tmp_path: Path) -> None:
"""The red test for round 4's A half: the flag exists and changes the plan.
Asserted on the plan's TITLES rather than a count, for the reason every
round-3 reach test is: a reader that cut in the wrong places would satisfy
a count assertion exactly as well.
ONE title from the PDF, not two, and that is the shipped default doing its
job rather than the reader failing: Arm F folds a deeper heading into its
parent, and `## Merking` is deeper than `# Generelle tekniske krav`. The
reader's own output is both -- `test_font_headings_emit_atx...` above pins
that -- and this is what the default does with it.
"""
from llm_ingestion_okf import cli
plans = tmp_path / "plans"
assert (
cli.main(
[
"build",
str(_inbox(tmp_path)),
"--bundle",
str(tmp_path / "bundle"),
"--bundle-id",
"font-fixture",
"--okf-version",
"0.2",
"--plans-dir",
str(plans),
"--pdf-headings",
"font",
]
)
== 0
)
assert _plan_titles(plans) == ["1 Anker", "Generelle tekniske krav"]
def test_the_build_default_leaves_the_font_pdf_unsegmented(tmp_path: Path) -> None:
"""The control the test above rests on: same inbox, no flag, no plan.
Without it, a change that turned the reader on by default would leave the
assertion above green while moving every PDF bundle a consumer has built.
"""
from llm_ingestion_okf import cli
plans = tmp_path / "plans-plain"
assert (
cli.main(
[
"build",
str(_inbox(tmp_path)),
"--bundle",
str(tmp_path / "bundle-plain"),
"--bundle-id",
"font-fixture",
"--okf-version",
"0.2",
"--plans-dir",
str(plans),
]
)
== 0
)
assert _plan_titles(plans) == ["1 Anker"]
assert cli.DEFAULT_PDF_HEADINGS is False
assert cli.DEFAULT_OCR is False
def test_ocr_without_the_group_is_a_coded_rejection_not_a_crash(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A missing engine must not take the run down; it accounts for the file.
`merged + coded rejections == N` is the conservation identity every corpus
run reports, and an engine nobody installed is exactly the kind of failure
that would otherwise leave a document unaccounted for.
"""
from llm_ingestion_okf import cli
monkeypatch.setitem(sys.modules, "rapidocr", None)
extract_module._pdf_pages.cache_clear()
report = cli.build(
_inbox(tmp_path),
tmp_path / "bundle-ocr",
bundle_id="font-fixture",
okf_version="0.2",
ocr=True,
)
assert dict(report.codes) == {"extractor_ocr_group_missing": 1}
assert report.merged + report.rejected == report.n == 2