feat(propose): the PDF shipped a structure index and the build discarded it unopened

`okf build` recovers a PDF's boundaries from the shape of its page text and
never opens the file's own `/Outlines` bookmark tree. On a 701-page process
code whose publisher also ships a NISO-STS structure for it, measured outside
this repository and reproduced here exactly: the shipped default finds 1967 of
2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their
heading text PRESENT in the extracted text. The line was read; the boundary
was never opened. The same file's bookmark tree matches 2761 of 2761 of those
titles exactly after normalisation.

`--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares.

  boundaries                 1967 of 2761  ->  2759 of 2761  (gate was 2700)
  depth 1                       0 of 28    ->     28 of 28
  titles identical to source        --     ->   2761 of 2761
  false positives             163 of 2182  ->      3 of 2762
  directories with two files  132 of 2050  ->      2 of 2738
  front-matter concepts        72 of 2182  ->      2 of 2762
  consumption fasit present       4 of 7   ->        7 of 7
  hit@1 / hit@8 / hit@50      1/6 2/6 4/6  ->   3/6 5/6 6/6

It is a SEGMENTATION arm, not a reader option: the extracted text is byte for
byte the same either way. A PDF with no tree builds byte-identically with the
flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm
on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a
boundary and never a refusal of the file.

The bridge from (page, y) to a line index is the whole risk, so both routes
are measured. `extract_text_lines` splits lines identically to `extract_text`
on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route
and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to
8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The
naive "nearest line" rule was wrong on 1840 of 2762, one line early every time.

The orphan check is not applied to a bookmark mark: it asks whether anything
stands under a candidate's first line, which is the right question for a
heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762
marks are container sections; applying it scores 2079 instead of 2759.

No new dependency and no second parse of the pages: `pdfminer.six` already
ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS
3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries
a usable tree at all.

`.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the
most measurement behind it and no entry in the table.

Report: docs/2026-09-10-k3-runde12-pdf-outlines.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 02:27:41 +02:00
commit e1f4faa098
12 changed files with 1261 additions and 15 deletions

View file

@ -234,6 +234,13 @@ DEFAULT_OCR = False
#: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`.
DEFAULT_PDF_HEADINGS_RESERVE = False
#: The PDF bookmark arm. OFF, and this round did not move it: the arm was
#: measured on ONE document, and a default that changes every PDF bundle a
#: consumer holds is an operator's call with the numbers in front of them.
#: It is BYTE-IDENTICAL on a PDF that carries no `/Outlines`, which is the
#: common case and the reason the flag is safe to hand out before that call.
DEFAULT_PDF_OUTLINE = False
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -262,6 +269,7 @@ def _propose_plans(
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -300,6 +308,7 @@ def _propose_plans(
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -337,6 +346,7 @@ def build(
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -409,6 +419,7 @@ def build(
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -784,6 +795,34 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"for two at once"
),
)
build_parser.add_argument(
"--pdf-outline",
action="store_true",
default=DEFAULT_PDF_OUTLINE,
help=(
"OFF. Cut a PDF at the boundaries its own /Outlines bookmark tree "
"declares, instead of at the ones the text rules recover. It is a "
"SEGMENTATION arm and not a reader option: the extracted text is "
"byte for byte the same either way, and a PDF that carries no "
"bookmark tree builds byte-identically with the flag on. Measured "
"on one 701-page process code whose publisher also ships a NISO-STS "
"structure for it: the text rules recover 1967 of 2761 titled "
"sections and 0 of its 28 chapters, while its bookmark tree matches "
"2761 of 2761 exactly. The title comes from the BOOKMARK, so it is "
"not cut short at the page's line break, and a page before the "
"first bookmark destination is the table of contents rather than a "
"second copy of the body. ONE document, ONE format, ONE publisher, "
"and a bookmark tree is the publisher's CLAIM about its own "
"structure -- a stale or wrong tree carries its error straight into "
"the segmentation"
),
)
build_parser.add_argument(
"--no-pdf-outline",
action="store_false",
dest="pdf_outline",
help="The arm's explicit opt-out",
)
build_parser.add_argument(
"--ocr",
action="store_true",
@ -836,6 +875,7 @@ def main(argv: list[str] | None = None) -> int:
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,
pdf_outline=args.pdf_outline,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)