llm-ingestion-okf/src/llm_ingestion_okf
Kjell Tore Guttormsen 53d5c74c96 feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.

`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.

It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).

`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.

Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.

Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md

Co-Authored-By: Claude <claude-opus-5>
2026-09-08 23:10:47 +02:00
..
__init__.py feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
_pandoc.py feat(extract): resolve the vendored converter and refuse an unpinned version 2026-09-02 14:07:20 +02:00
cli.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
connectors.py fix(errors): type the NUL-byte path failure as SourceError 2026-07-20 10:30:42 +02:00
consume.py feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
contract_check.py feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
corpus.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
errors.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
extract.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
guard_adapter.py feat(guard): bump the pin to >=0.3,<0.4 and pin Door C's allow_reserved=False 2026-08-02 21:08:53 +02:00
importer.py feat(index): one ordering helper, called by both doors 2026-09-01 19:54:45 +02:00
inbox.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
manifest.py feat(profiles): materialize_bundle takes a keyword-only profile (req 6) 2026-07-31 15:37:17 +02:00
materialize.py feat(materialize): sources takes a list and renders N flow mappings 2026-09-02 16:12:19 +02:00
profiles.py feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
project.py feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
propose.py feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
py.typed feat(typing): ship the PEP 561 py.typed marker 2026-07-17 03:46:51 +02:00
render.py feat(errors): stable machine-readable codes on IngestError 2026-07-17 04:01:04 +02:00
segmentation.py feat(segmentation): parse the adjudication state a plan already carries 2026-09-02 14:44:06 +02:00
skill.py feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
structure.py fix(frontmatter): a nested key must not substitute for a top-level one 2026-08-31 23:31:53 +02:00