feat(propose): the PDF shipped a structure index and the build discarded it unopened
`okf build` recovers a PDF's boundaries from the shape of its page text and never opens the file's own `/Outlines` bookmark tree. On a 701-page process code whose publisher also ships a NISO-STS structure for it, measured outside this repository and reproduced here exactly: the shipped default finds 1967 of 2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their heading text PRESENT in the extracted text. The line was read; the boundary was never opened. The same file's bookmark tree matches 2761 of 2761 of those titles exactly after normalisation. `--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares. boundaries 1967 of 2761 -> 2759 of 2761 (gate was 2700) depth 1 0 of 28 -> 28 of 28 titles identical to source -- -> 2761 of 2761 false positives 163 of 2182 -> 3 of 2762 directories with two files 132 of 2050 -> 2 of 2738 front-matter concepts 72 of 2182 -> 2 of 2762 consumption fasit present 4 of 7 -> 7 of 7 hit@1 / hit@8 / hit@50 1/6 2/6 4/6 -> 3/6 5/6 6/6 It is a SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way. A PDF with no tree builds byte-identically with the flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a boundary and never a refusal of the file. The bridge from (page, y) to a line index is the whole risk, so both routes are measured. `extract_text_lines` splits lines identically to `extract_text` on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to 8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The naive "nearest line" rule was wrong on 1840 of 2762, one line early every time. The orphan check is not applied to a bookmark mark: it asks whether anything stands under a candidate's first line, which is the right question for a heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762 marks are container sections; applying it scores 2079 instead of 2759. No new dependency and no second parse of the pages: `pdfminer.six` already ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS 3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries a usable tree at all. `.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the most measurement behind it and no entry in the table. Report: docs/2026-09-10-k3-runde12-pdf-outlines.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8e1917946a
commit
e1f4faa098
12 changed files with 1261 additions and 15 deletions
27
README.md
27
README.md
|
|
@ -146,6 +146,32 @@ pre-2026-09-10 bundle byte for byte — measured with `diff -rq`, 0 differences,
|
|||
not asserted. Each line below carries the number it was measured at, and
|
||||
nothing beyond it.
|
||||
|
||||
An **eleventh** flag, `--pdf-outline`, is **off** by default, and it is the one
|
||||
rule here that does not read the extracted text at all: it cuts a PDF at the
|
||||
boundaries the file's own `/Outlines` bookmark tree declares. It is not
|
||||
`--outline-run` under another name — that one is a text heuristic over numbered
|
||||
lines in the extracted text, while this one opens a structure index the PDF
|
||||
already carries and the build had never looked at.
|
||||
|
||||
It is a segmentation arm and not a reader option: the extracted text is byte
|
||||
for byte the same either way, and a PDF that carries no bookmark tree builds
|
||||
byte-identically with the flag on. Two consequences come free with it. The
|
||||
concept title comes from the BOOKMARK, so it is not cut short where the page
|
||||
wrapped the heading across two lines; and a page that lies before the first
|
||||
bookmark destination is the contents listing rather than a second copy of the
|
||||
body, so a contents entry and the section it lists stop landing as two concepts
|
||||
under one id.
|
||||
|
||||
The measurement is one 701-page process code whose publisher also ships a
|
||||
NISO-STS structure for it, so the fasit is the publisher's own. Under the
|
||||
shipped default that document gives 1967 of 2761 boundaries, none of its 28
|
||||
chapters, and 794 of 794 misses have their heading text present in the text the
|
||||
build read. With the arm it gives 2759 of 2761 and 28 of 28. The flag stays off
|
||||
because reach is the open question, not quality: **1 of the 8** reference PDFs
|
||||
in this repository's own sample carries a usable tree, and a bookmark tree is
|
||||
the publisher's *claim* about its own structure — a stale or wrongly pointing
|
||||
one carries that error straight into the segmentation.
|
||||
|
||||
A **tenth** flag, `--bold-title`, is **off** by default. It is the rule for the
|
||||
type whose container declares nothing: `rtf` has no heading style, so an
|
||||
author's title is bold text, and the row was measured at 0 of 0 declared
|
||||
|
|
@ -189,6 +215,7 @@ rank was measured in the same session and did NOT ship; see
|
|||
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
|
||||
| `--first-span-from-zero` (**on** by default since 2026-09-10; opt out with `--no-first-span-from-zero`) | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | Measured over the 39-document corpus, the default before this rule left **207 435 characters — 11.92 %** — in no segment at all: **163 804 above the first entry** (in **32 of the 32** documents that get a plan), 26 041 *between* entries and 17 590 after the last. This rule closes the first part entirely, 79 % of the whole, leaving **43 631 characters (2.51 %) over 8 of 32 documents** with two named mechanisms of their own. It adds no boundary and the K2 concept count is identical with and without it (**425 = 425**); on the 12-position reference it changes **not one cell**, and hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–** under both tie-breaks |
|
||||
| `--close-span-gaps` (**on** by default since 2026-09-11; opt out with `--no-close-span-gaps`) | close a concept's span against the next SURVIVING concept, and the last against the end of the text. Three steps remove a candidate AFTER its neighbour's span was already closed against it — the orphan check, and `fold_units` clause 1 both between entries and on the last run — and the removed mark's text then belongs to no segment. Adds no boundary and removes none; only spans' ends move | It closes the whole remainder the rule above left: **43 631 characters, 2.51 % of the corpus over 8 of the 32 documents with a plan, to 0** — both the 26 041 between entries and the 17 590 after the last. Decomposed: orphan check **18 527** over 15 of 39 documents, clause 1 **7 514** between entries, clause 1 on the last run **all 17 590** of the tail (with `unit_fold=False` the corpus tail gap is 0). The entry count is identical (**429 = 429** on the corpus, **436 = 436** concepts on K2, **52 = 52** md on a five-document folder); on the 12-position reference it changes **not one cell** (11 of 12 under `|F|`[3]=12, 10 of 12 under `|F|`[3]=11), and hit@8 holds **5 of 6 at ranks 1,1,1,1,1,–** on the new bundle, the previous default and Arm B alike |
|
||||
| `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one 701-page process code whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own: boundaries **1967 of 2761 → 2759 of 2761**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **2761 of 2761**, false positives **163 of 2182 → 3 of 2762**, directories carrying two concept files **132 of 2050 → 2 of 2738**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation |
|
||||
|
||||
They compose, and the order above is the order they apply in. Measured on a
|
||||
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue