feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.
`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.
It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).
`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.
Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.
Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
f6fea13299
commit
53d5c74c96
15 changed files with 1394 additions and 28 deletions
|
|
@ -1118,6 +1118,8 @@ def run(
|
|||
keep_table_heading: bool = False,
|
||||
sheet_section_rows: bool = False,
|
||||
drop_wrapped_outline: bool = False,
|
||||
pdf_headings: bool = False,
|
||||
ocr: bool = False,
|
||||
) -> int:
|
||||
if max_segment_chars < 0:
|
||||
raise ProposerError(
|
||||
|
|
@ -1155,7 +1157,13 @@ def run(
|
|||
except OSError as exc:
|
||||
raise ProposerError(f"cannot read {source}: {exc}") from exc
|
||||
try:
|
||||
text = extract_text(source.name, source_bytes)
|
||||
# The two READER options, not arms: they change what the extraction
|
||||
# says, and every arm below reads whatever it says. Passed here as well
|
||||
# as to the run path because the plan's `text_sha256` indexes this
|
||||
# exact string -- a plan proposed against one rendering and replayed
|
||||
# against another is refused by `assert_plan_applies`, which is the
|
||||
# right outcome and a confusing one to debug.
|
||||
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
|
||||
except IngestError as exc:
|
||||
raise ProposerError(f"cannot extract text from {source.name}: {exc}") from exc
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue