A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.
`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.
It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).
`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.
Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.
Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md
Co-Authored-By: Claude <claude-opus-5>
108 lines
4.8 KiB
Python
108 lines
4.8 KiB
Python
"""Packaging contract: a PEP 561 typed package with exactly one dependency.
|
|
|
|
Consumers run mypy --strict against the inline annotations; without the
|
|
py.typed marker mypy degrades every imported symbol to Any.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
import llm_ingestion_okf
|
|
|
|
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
|
|
|
|
|
def test_package_ships_py_typed_marker() -> None:
|
|
package_dir = Path(llm_ingestion_okf.__file__).parent
|
|
assert (package_dir / "py.typed").is_file()
|
|
|
|
|
|
def test_the_only_runtime_dependency_is_the_security_boundary() -> None:
|
|
"""The stdlib-only rule, enforced rather than asserted in prose.
|
|
|
|
One dependency is permitted — the guard — because security is the one
|
|
thing this library must not implement. Everything else stays stdlib, so
|
|
a consumer vendoring this package takes on no transitive surface. The
|
|
version RANGE is the pin: it resolves against a package index, and is
|
|
satisfied by the git+https tag install until that index exists.
|
|
"""
|
|
tomllib = pytest.importorskip("tomllib") # stdlib from 3.11; the pin holds on 3.10 too
|
|
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
|
|
assert pyproject["project"]["dependencies"] == ["llm-ingestion-guard>=1.2,<2.0"]
|
|
|
|
|
|
def test_the_extract_extra_pins_exactly_what_it_ships() -> None:
|
|
"""`project.dependencies` was pinned; the extra's contents were not.
|
|
|
|
The single-dependency test above reads `project.dependencies` only, so a
|
|
second package could be added to `[extract]` and no test would notice --
|
|
and an extra is exactly where an unexamined transitive tree arrives. The
|
|
extra is opt-in, but "opt-in" is a statement about who installs it, not
|
|
about whether its contents were chosen.
|
|
|
|
Both entries are pins with a stated reason, not conveniences:
|
|
`pdfplumber` for the pdf reader, `pypandoc-binary` because the converter
|
|
BINARY travels with the wheel. Vendoring the binary is what makes the
|
|
output reproducible -- see the resolver, which refuses any version but the
|
|
pinned one.
|
|
"""
|
|
tomllib = pytest.importorskip("tomllib")
|
|
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
|
|
assert pyproject["project"]["optional-dependencies"]["extract"] == [
|
|
"pdfplumber>=0.11.10,<0.12",
|
|
"pypandoc-binary==1.17",
|
|
]
|
|
|
|
|
|
def test_the_ocr_group_is_pinned_and_is_not_a_runtime_dependency() -> None:
|
|
"""An inference runtime is the last thing that may arrive by accident.
|
|
|
|
Two claims, and the second is the one worth a test: the group's contents
|
|
are pinned like the extra's, AND none of them appears in
|
|
`project.dependencies`. The single-dependency test above would already
|
|
catch that, but it reads the list and this reads the names -- so a future
|
|
entry named differently still fails here.
|
|
"""
|
|
tomllib = pytest.importorskip("tomllib")
|
|
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
|
|
assert pyproject["project"]["optional-dependencies"]["ocr"] == [
|
|
"rapidocr>=3.9,<4",
|
|
"onnxruntime>=1.20,<2",
|
|
"pypdfium2>=4,<6",
|
|
]
|
|
runtime = " ".join(pyproject["project"]["dependencies"])
|
|
for package in ("rapidocr", "onnxruntime", "pypdfium2"):
|
|
assert package not in runtime
|
|
|
|
|
|
def test_the_declared_version_agrees_with_the_packaged_one() -> None:
|
|
"""The two places a version is written must not drift apart.
|
|
|
|
The install channel is a direct git reference, so a consumer pins a TAG
|
|
while pip records `project.version`. Nothing in the run path reads
|
|
`__version__` — which is exactly why a stale one survives a green suite,
|
|
and why a consumer installing at a pre-release tag can end up with a
|
|
package that reports the previous release. This is the only machine check
|
|
on that; the tag name itself remains a human step.
|
|
"""
|
|
tomllib = pytest.importorskip("tomllib")
|
|
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
|
|
assert llm_ingestion_okf.__version__ == pyproject["project"]["version"]
|
|
|
|
|
|
def test_operational_tooling_stays_out_of_the_wheel() -> None:
|
|
"""`tools/` is ours, not the consumer's.
|
|
|
|
The upstream watch drives git and the coord mailbox — machinery that is
|
|
meaningful on this machine and meaningless in a consumer's site-packages.
|
|
It lives outside `src/` so it cannot ship, and this test is what makes
|
|
that a promise instead of an accident of the current build config.
|
|
"""
|
|
tomllib = pytest.importorskip("tomllib")
|
|
pyproject = tomllib.loads((PROJECT_ROOT / "pyproject.toml").read_text(encoding="utf-8"))
|
|
packages = pyproject["tool"]["hatch"]["build"]["targets"]["wheel"]["packages"]
|
|
assert packages == ["src/llm_ingestion_okf"]
|
|
assert (PROJECT_ROOT / "tools" / "okf_watch.py").is_file(), "the test must have a subject"
|