open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): the PDF shipped a structure index and the build discarded it unopened

Browse source 
`okf build` recovers a PDF's boundaries from the shape of its page text and
never opens the file's own `/Outlines` bookmark tree. On a 701-page process
code whose publisher also ships a NISO-STS structure for it, measured outside
this repository and reproduced here exactly: the shipped default finds 1967 of
2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their
heading text PRESENT in the extracted text. The line was read; the boundary
was never opened. The same file's bookmark tree matches 2761 of 2761 of those
titles exactly after normalisation.

`--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares.

  boundaries                 1967 of 2761  ->  2759 of 2761  (gate was 2700)
  depth 1                       0 of 28    ->     28 of 28
  titles identical to source        --     ->   2761 of 2761
  false positives             163 of 2182  ->      3 of 2762
  directories with two files  132 of 2050  ->      2 of 2738
  front-matter concepts        72 of 2182  ->      2 of 2762
  consumption fasit present       4 of 7   ->        7 of 7
  hit@1 / hit@8 / hit@50      1/6 2/6 4/6  ->   3/6 5/6 6/6

It is a SEGMENTATION arm, not a reader option: the extracted text is byte for
byte the same either way. A PDF with no tree builds byte-identically with the
flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm
on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a
boundary and never a refusal of the file.

The bridge from (page, y) to a line index is the whole risk, so both routes
are measured. `extract_text_lines` splits lines identically to `extract_text`
on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route
and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to
8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The
naive "nearest line" rule was wrong on 1840 of 2762, one line early every time.

The orphan check is not applied to a bookmark mark: it asks whether anything
stands under a candidate's first line, which is the right question for a
heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762
marks are container sections; applying it scores 2079 instead of 2759.

No new dependency and no second parse of the pages: `pdfminer.six` already
ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS
3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries
a usable tree at all.

`.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the
most measurement behind it and no entry in the table.

Report: docs/2026-09-10-k3-runde12-pdf-outlines.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 02:27:41 +02:00
commit e1f4faa098
12 changed files with 1261 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

42
CLAUDE.md
View file

@ -42,7 +42,9 @@ one boundary rule:
files**, character ratio **1.000000**. `_SKIP_TAGS` stays `{script, style}`.
Exposure elsewhere measured rather than argued: **0 of 86** K2 corpus files and
**0 of 5** smoke-folder files are HTML, and the smoke bundle is byte-identical
before and after. `_EVIDENCE` gains a `.html` row at `measured`, with the limit
before and after. `_EVIDENCE` gains a `.html` row at `measured` -- and a `.pdf` row at
`measured` since 2026-09-10, the row with the most measurement behind it and
no entry in the table at all -- with the limit
that travels with it -- one product, one format, one publisher, and a
generator's cut, not 828 documents anyone wrote. `pdf`/`docx`/`xlsx` only via
the optional `[extract]` extra; without it those types are rejected
@ -476,6 +478,44 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is
the only one here that does not read the extracted text at all: it cuts a PDF
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D
-- `--outline-run`/`--outline-gate` are a TEXT heuristic over numbered lines
in the extracted text, and this opens a structure index the file already
carries. Measured on ONE 701-page process code whose publisher also ships a
NISO-STS structure for it, so the fasit is the publisher's own 2 761 titled
sections: the shipped default finds **1 967 of 2 761**, **0 of its 28**
chapters, and **794 of 794** misses have their heading text PRESENT in the
extracted text -- the line is read, the boundary is never opened. With the
arm: **2 759 of 2 761 (99.9 %)**, chapter level **28 of 28**, concept titles
identical to the source after normalisation **2 761 of 2 761** (the bookmark
title is complete because it does not come from the page), false positives
**163 of 2 182 -> 3 of 2 762**, directories carrying two concept files
**132 -> 2** with the 65 contents-copy pairs at **0**, front-matter concepts
**72 -> 2**. Consumption: fasit present in the bundle **4 of 7 -> 7 of 7**,
hit@1/8/50 **1/6 - 2/6 - 4/6 -> 3/6 - 5/6 - 6/6**; the known-positive is a
real concept now and ranks 13, so it is delivered at k=50 and not at k=8 --
the segmentation half of that row is closed and the ranking half is not.
**The bridge from (page, `/XYZ` top) to a line index is the whole risk and
BOTH routes are measured:** `extract_text_lines` splits lines identically to
`extract_text` on **701 of 701** pages and that check SHIPS per page, the y
route and the title route disagree on **0 of 2 762**, flat from 0 to 8 pt and
collapsing at 12, so the rule carries **no tolerance constant**; the naive
nearest-line rule was wrong on **1 840 of 2 762**, one line early every time.
The orphan check is NOT applied to a bookmark mark -- it asks whether anything
stands under a candidate's first line, the right question for a guess and the
wrong one for a publisher's declaration; **683 of 2 762** marks are container
sections and applying it scores 2 079. An unresolvable `/Dest` is dropped and
COUNTED (R761 has 0 of 2 763; one of the eight reference PDFs has 2 of 2). NO
new dependency: `pdfminer.six` already ships under `pdfplumber` in
`[extract]`, so `uv.lock` is untouched and `pypdf` stays out. Cost 119.22 s ->
183.31 s wall, peak RSS 3 252 -> 3 251 MiB, pages parsed 1 -> 1. **The default
did not move**, and the reach is why: **1 of 8** reference PDFs carries a
usable tree, and a bookmark tree is the publisher's CLAIM about its own
structure. On the folder where no PDF has one, `diff -r` is empty against both
the arm off and the pre-change tree. Report:
`docs/2026-09-10-k3-runde12-pdf-outlines.md`.
- **Three PDF READER flags, all off, and they sit BEFORE every segmentation
flag** -- an arm changes how the proposer cuts a text, these change what the
text says. `--pdf-headings font` infers a heading from typography (dominant