llm-ingestion-okf/docs
Kjell Tore Guttormsen 53d5c74c96 feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off
A PDF carries no notion of a heading -- a heading in a PDF is a typographic
fact -- so the text stream `pdfplumber` hands the segment proposer has already
thrown away the only evidence there was. The `docx` path never had that problem:
the converter emits ATX headings and `_ATX` cuts on them. Two readers close the
gap, and both are OFF.

`--pdf-headings font` infers a heading from the conjunction this repository
already measured (size above the document's character-weighted body median AND
a bold font name, recall 1.000 / precision 0.846) and emits it as ATX in the
SAME markdown the office path produces, so `_ATX` applies unchanged and no
PDF-only heading grammar exists.

It stays off BY MEASUREMENT, and the measurement is the point of the round:
against the operator's unit worksheet it takes `pdf` from 2 of 8 to 0 of 8,
losing two exact matches. The mechanism of the loss is stated rather than
guessed -- on those documents the outline rule already recovers the document's
own numbered chapters, so a second heading source can only add. Whole-corpus
screen: 25 of 32 `pdf` change, 0 of 5 `docx`, 0 of 2 `xlsx`. The default bundle
is byte-identical before and after this commit (`diff -r`, exit 0).

`--ocr` reads a page as an image when its own text never arrived: empty, or
`(cid:N)` placeholder codes at or above a threshold READ OFF a measured
distribution -- 834 pages over 32 files, 818 at exactly 0.0 and 16 at 0.93 or
above, nothing in between. On the one corpus document with the failure: 95.07 %
cid to 0 %, 44 to 2561 words of four or more letters, 17 to 18 pages with text.
Its engine is an optional dependency group and never a runtime dependency; a
packaging test pins both halves, and without the group every affected file is a
coded rejection (`extractor_ocr_group_missing`) rather than a crash.

Also corrects two stale published facts found while measuring: the README still
said two segmentation rules were on by default after `f6fea13` made it three,
and CLAUDE.md's K2 digest named the round-3 default. The current default is
492 concepts / 944 files, `bdefa679...`.

Report: docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md

Co-Authored-By: Claude <claude-opus-5>
2026-09-08 23:10:47 +02:00
..
plan fix(corpus): the root index links the bundle's own log 2026-09-03 20:23:56 +02:00
2026-08-21-g2-pdf-extraction-measurement.md docs(extract): measure one Vegnormalene PDF page against the extraction registry 2026-08-21 19:16:15 +02:00
2026-08-29-konverteringsarmen-arm-a.md docs(measure): run the conversion arm on K2 - pandoc wins both office formats 2026-08-30 00:41:48 +02:00
2026-08-29-telling-offentlig-prosjektkorpus.md docs(measure): count the public project corpus - K2 measured, K1 blocked 2026-08-29 07:03:50 +02:00
2026-09-02-k3-k4-k5-baseline.md docs(measurement): K3, K4 and K5 numbers against the committed method 2026-09-02 15:55:19 +02:00
2026-09-02-k3-k4-k5-metode.md docs(measurement): K3, K4 and K5 method with n declared before any verdict 2026-09-02 15:46:03 +02:00
2026-09-02-korpuskjoring-k1-k2.md docs(measurement): K1 and K2 over the K2 corpus 2026-09-02 15:43:45 +02:00
2026-09-03-k2-bundle-rebuild.md test(corpus): a rebuild with log.md present reproduces the bundle exactly 2026-09-03 05:16:25 +02:00
2026-09-04-cid-andel-k2.md docs(measure): CID-glyph share across K2 -- Bilag 9.1 alone, not a K1 arm 2026-09-04 20:23:41 +02:00
2026-09-04-k2-pptx-odt-rtf.md docs(measure): K2 on pptx/odt/rtf -- denominator 0 in both named corpora 2026-09-04 21:12:41 +02:00
2026-09-04-k3-arm-c.md docs(measure): K3 with Arm C beside the baseline, two numbers, no threshold 2026-09-04 17:58:50 +02:00
2026-09-07-k2-pptx-odt-rtf-fixtures.md test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote 2026-09-07 05:22:12 +02:00
2026-09-07-k3-arm-d.md docs(measure): K3 Arm D beside a re-rated Arm B -- reach and no threshold 2026-09-07 03:04:01 +02:00
2026-09-07-k3-arm-e.md docs(measure): K3 Arm E beside Arm B and Arm D -- three rows, no threshold 2026-09-07 11:48:58 +02:00
2026-09-07-k3-enhetsark-metode.md docs(measure): K3 unit worksheet delivered for operator, no code changed 2026-09-07 16:36:16 +02:00
2026-09-07-okf-konsumskill-maaling.md docs(consume): refutation sweep — smoothed-IDF rarity weighting cannot zero both known-negatives 2026-09-07 09:56:36 +02:00
2026-09-08-blindsone-below-k-k2.md feat(consume): measure the below-k blind spot, add one flag-gated vocabulary bridge 2026-09-08 04:55:45 +02:00
2026-09-08-blindsone-laas2-budsjett-k2.md feat(consume): measure the budget lock, add one flag-gated top-rank reservation 2026-09-08 05:18:30 +02:00
2026-09-08-claude-code-skill-vilkaarlig-bundle.md docs(consume): the Claude Code recipe, measured end to end on two bundles 2026-09-08 15:32:32 +02:00
2026-09-08-eksakt-oppslag.md fix(consume): match an identifier by equality, deliver the concept a question names 2026-09-08 13:27:59 +02:00
2026-09-08-k3-arm-f-mot-enhetsarket.md feat(propose): Arm F, one unit fold behind a flag, measured against the operator's worksheet [skip-docs] 2026-09-08 16:28:41 +02:00
2026-09-08-k3-runde2-per-filtype.md feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms 2026-09-08 18:21:25 +02:00
2026-09-08-k3-runde3-per-filtype.md docs(cli): the new default's K2 digest, measured on the bundle it produces 2026-09-08 21:14:51 +02:00
2026-09-08-k3-runde4-pdf-skrift-og-ocr.md feat(extract,cli): typography as a PDF heading source and OCR behind an optional group, both off 2026-09-08 23:10:47 +02:00
2026-09-08-kravnummer-tokenisering.md feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved 2026-09-08 20:34:18 +02:00
2026-09-08-o5-okf-project.md feat(cli): okf project/consume/check/skill, and a generated skill with no path into a checkout 2026-09-08 21:47:38 +02:00
2026-09-08-prisform-og-loggen-k2.md fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
2026-09-08-proveniens-k2.md feat(inbox): point every concept at the document it came from, with a locator per format 2026-09-08 14:39:24 +02:00
2026-09-08-rangeringsbom-sammensatte-ord.md fix(consume): the compound-word miss is a degenerate signal's tie-break, behind a flag 2026-09-08 18:53:18 +02:00
2026-09-08-sjeldenhetsvekt.md feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved 2026-09-08 20:34:18 +02:00
consumption-contract.md feat(consume): carry every source_* key by prefix, and generate a skill per bundle 2026-09-08 15:13:22 +02:00
okf-nokkelinventar.md build(deps): pin llm-ingestion-guard v1.3.0 so the gate reads our own goldens 2026-09-03 20:41:03 +02:00
phase-3-split-table.md docs(phase-3): keep the index-entry grammar on one line 2026-08-04 12:37:02 +02:00
upstream-okf-upgrade-runbook.md feat(watch): weekly OKF upstream watch that can prove it found nothing 2026-08-23 20:38:37 +02:00