feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.
`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.
It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).
`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.
Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.
Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md
Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
f6fea13299
commit
53d5c74c96
15 changed files with 1394 additions and 28 deletions
32
CLAUDE.md
32
CLAUDE.md
|
|
@ -278,8 +278,10 @@ and fixtures, never code.
|
|||
so the two layers disagree on purpose and `cli.DEFAULT_OUTLINE_RUN` /
|
||||
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
|
||||
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
|
||||
(`9cd74519...`, the delivered 2026-09-03 tree) to **517 / 969**
|
||||
(`42c8646d...`, default flags, default epoch stamp). **And the number the
|
||||
(`9cd74519...`, the delivered 2026-09-03 tree) to **492 / 944**
|
||||
(`bdefa679...`, default flags, default epoch stamp, measured 2026-09-08 on
|
||||
`f6fea13`). That digest SUPERSEDES `42c8646d...` (517 / 969), which named the
|
||||
round-3 default before Arm E joined it. **And the number the
|
||||
decision cites belongs to another configuration:** Arm F's 5 of 12 was
|
||||
measured with `--table-grid` ON; without it the same sample scores 2 of 12
|
||||
and `docx` 0 of 3, because the fold's table clause has no joined table to
|
||||
|
|
@ -307,6 +309,32 @@ and fixtures, never code.
|
|||
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
|
||||
`docs/2026-09-08-k3-runde2-per-filtype.md` and
|
||||
`docs/2026-09-08-k3-runde3-per-filtype.md`.
|
||||
- **Two PDF READER flags, both off, and they sit BEFORE every segmentation
|
||||
flag** -- an arm changes how the proposer cuts a text, these change what the
|
||||
text says. `--pdf-headings font` infers a heading from typography (dominant
|
||||
font size above the document's character-weighted body median AND a bold font
|
||||
name -- the CONJUNCTION measured at recall 1.000 / precision 0.846, where
|
||||
adding weight as a disjunct took precision 0.786 -> 0.524) and emits it as an
|
||||
ATX heading in the SAME markdown the office path produces, so `_ATX` applies
|
||||
unchanged and **no PDF-only heading grammar exists**. It is off **by
|
||||
measurement, not by caution**: against the operator's unit worksheet it takes
|
||||
`pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those
|
||||
documents the outline rule already recovers the document's own numbered
|
||||
chapters and a second heading source can only add. The cost of the ATX form is
|
||||
named rather than hidden: a font-inferred heading carries `rule:heading` and is
|
||||
indistinguishable in the artifact from one the document declared, which is why
|
||||
`RULE_POPPLER_SIZE_AND_BOLD` was deliberately NOT assigned to it -- that name
|
||||
records a poppler measurement on a path that cannot ship. `--ocr` reads a page
|
||||
as an IMAGE when its own text never arrived (empty, or `(cid:N)` codes at or
|
||||
above `OCR_CID_SHARE = 0.10`, a threshold READ OFF the measured per-page
|
||||
distribution: 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
|
||||
above, nothing between). Its engine is the optional `ocr` group
|
||||
(`rapidocr`/`onnxruntime`/`pypdfium2`) and **never** a runtime dependency; a
|
||||
packaging test pins both halves, and without it every affected file is a coded
|
||||
rejection (`extractor_ocr_group_missing`), never a crash. `--ocr` can never
|
||||
become a default -- an optional dependency in the default path would make an
|
||||
ordinary install fail on the first scanned page. Report:
|
||||
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`.
|
||||
- Consume a bundle: `okf consume <bundle> --question "<q>"
|
||||
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
|
||||
`docs/consumption-contract.md` § 1 defines, and the only reading direction
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue