open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): the PDF shipped a structure index and the build discarded it unopened

Browse source 
`okf build` recovers a PDF's boundaries from the shape of its page text and
never opens the file's own `/Outlines` bookmark tree. On a 701-page process
code whose publisher also ships a NISO-STS structure for it, measured outside
this repository and reproduced here exactly: the shipped default finds 1967 of
2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their
heading text PRESENT in the extracted text. The line was read; the boundary
was never opened. The same file's bookmark tree matches 2761 of 2761 of those
titles exactly after normalisation.

`--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares.

  boundaries                 1967 of 2761  ->  2759 of 2761  (gate was 2700)
  depth 1                       0 of 28    ->     28 of 28
  titles identical to source        --     ->   2761 of 2761
  false positives             163 of 2182  ->      3 of 2762
  directories with two files  132 of 2050  ->      2 of 2738
  front-matter concepts        72 of 2182  ->      2 of 2762
  consumption fasit present       4 of 7   ->        7 of 7
  hit@1 / hit@8 / hit@50      1/6 2/6 4/6  ->   3/6 5/6 6/6

It is a SEGMENTATION arm, not a reader option: the extracted text is byte for
byte the same either way. A PDF with no tree builds byte-identically with the
flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm
on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a
boundary and never a refusal of the file.

The bridge from (page, y) to a line index is the whole risk, so both routes
are measured. `extract_text_lines` splits lines identically to `extract_text`
on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route
and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to
8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The
naive "nearest line" rule was wrong on 1840 of 2762, one line early every time.

The orphan check is not applied to a bookmark mark: it asks whether anything
stands under a candidate's first line, which is the right question for a
heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762
marks are container sections; applying it scores 2079 instead of 2759.

No new dependency and no second parse of the pages: `pdfminer.six` already
ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS
3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries
a usable tree at all.

`.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the
most measurement behind it and no entry in the table.

Report: docs/2026-09-10-k3-runde12-pdf-outlines.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 02:27:41 +02:00
commit e1f4faa098
12 changed files with 1261 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

40
src/llm_ingestion_okf/cli.py
View file

@ -234,6 +234,13 @@ DEFAULT_OCR = False
#: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`.
DEFAULT_PDF_HEADINGS_RESERVE = False
#: The PDF bookmark arm. OFF, and this round did not move it: the arm was
#: measured on ONE document, and a default that changes every PDF bundle a
#: consumer holds is an operator's call with the numbers in front of them.
#: It is BYTE-IDENTICAL on a PDF that carries no `/Outlines`, which is the
#: common case and the reason the flag is safe to hand out before that call.
DEFAULT_PDF_OUTLINE = False
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -262,6 +269,7 @@ def _propose_plans(
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -300,6 +308,7 @@ def _propose_plans(
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -337,6 +346,7 @@ def build(
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -409,6 +419,7 @@ def build(
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -784,6 +795,34 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"for two at once"
),
)
build_parser.add_argument(
"--pdf-outline",
action="store_true",
default=DEFAULT_PDF_OUTLINE,
help=(
"OFF. Cut a PDF at the boundaries its own /Outlines bookmark tree "
"declares, instead of at the ones the text rules recover. It is a "
"SEGMENTATION arm and not a reader option: the extracted text is "
"byte for byte the same either way, and a PDF that carries no "
"bookmark tree builds byte-identically with the flag on. Measured "
"on one 701-page process code whose publisher also ships a NISO-STS "
"structure for it: the text rules recover 1967 of 2761 titled "
"sections and 0 of its 28 chapters, while its bookmark tree matches "
"2761 of 2761 exactly. The title comes from the BOOKMARK, so it is "
"not cut short at the page's line break, and a page before the "
"first bookmark destination is the table of contents rather than a "
"second copy of the body. ONE document, ONE format, ONE publisher, "
"and a bookmark tree is the publisher's CLAIM about its own "
"structure -- a stale or wrong tree carries its error straight into "
"the segmentation"
),
)
build_parser.add_argument(
"--no-pdf-outline",
action="store_false",
dest="pdf_outline",
help="The arm's explicit opt-out",
)
build_parser.add_argument(
"--ocr",
action="store_true",
@ -836,6 +875,7 @@ def main(argv: list[str] | None = None) -> int:
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,
pdf_outline=args.pdf_outline,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

207
src/llm_ingestion_okf/extract.py
View file

@ -106,7 +106,18 @@ _PANDOC_FORMATS: dict[str, str] = {
# the honesty limit that travels with it: the 828 files are ONE product in ONE
# format from ONE publisher, and the file boundaries and `<h1>`s are a
# generator's cut of that document, not 828 documents anyone wrote.
#
# `.pdf` JOINED THE TABLE 2026-09-10, as `measured`, and it enters on the
# strongest evidence of any row here: eight real corpus PDFs with a fasit the
# operator hand-counted document by document, plus a 701-page process code
# whose PUBLISHER also ships a NISO-STS structure for it -- 2 761 titled
# sections, written for their own purposes and long before any lookup of ours.
# The honesty limit that travels with it: those 2 761 rows are ONE product in
# ONE format from ONE publisher, its structure is a strict numbered hierarchy
# on 2 739 of 2 761 titles, and a running prose document would measure
# something else entirely.
_EVIDENCE: dict[str, str] = {
".pdf": "measured",
".docx": "measured",
".xlsx": "measured",
".pptx": "constructed",
@ -638,6 +649,202 @@ def _pdf_pages(
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
@dataclass(frozen=True)
class OutlineMark:
"""One `/Outlines` node, placed on a LINE of the string `extract_text` returns.
`level` is what the TREE declares, not a distance normalised against
anything: a document whose outline carries its own root node puts its
chapters at level 2, and rewriting that here would state a structure the
publisher did not. Measured on a 701-page process code -- the tree's levels
2..8 hold 28/118/500/1141/872/93/9 nodes against the publisher's own
NISO-STS depths 1..7 at 28/118/500/1141/868/97/9, so the mapping is level
minus one on five rows and the publisher disagrees with the publisher on
four nodes. That disagreement is data, and it survives only if the level is
reported rather than fixed up.
"""
line: int
level: int
title: str
@dataclass(frozen=True)
class PdfOutline:
"""The bookmark tree, bridged onto lines -- with what did not bridge counted.
`unresolved` is not decoration. A `/Dest` that names an object which is not
a page, or a page that produced no text, has to be DROPPED: fabricating a
boundary from it would put a heading somewhere the document never had one,
and raising would refuse a file over a defect in one of its bookmarks.
Dropping silently is the third option this library refuses everywhere else,
so the count is part of the return value.
"""
marks: tuple[OutlineMark, ...]
unresolved: int
def _outline_page_and_top(doc: object, dest: object, action: object) -> tuple[object, float | None]:
"""`(page reference, /XYZ top)` from a bookmark's destination, or `(None, None)`.
Four shapes reach here and all four are in the wild: an explicit array, a
NAMED destination resolved through the document's name tree, a `GoTo`
action carrying either, and an indirect reference to any of them.
"""
target = dest
if target is None and action is not None:
resolved = action.resolve() if hasattr(action, "resolve") else action
if isinstance(resolved, dict):
target = resolved.get("D")
if isinstance(target, (bytes, str)) or hasattr(target, "name"):
name = target.name if hasattr(target, "name") else target
try:
target = doc.get_dest(name) # type: ignore[attr-defined]
except Exception:
return (None, None)
if hasattr(target, "resolve"):
try:
target = target.resolve()
except Exception:
return (None, None)
if isinstance(target, dict):
target = target.get("D")
if not isinstance(target, list) or not target:
return (None, None)
top: float | None = None
if len(target) > 3 and getattr(target[1], "name", None) == "XYZ":
candidate = target[3]
if isinstance(candidate, (int, float)):
top = float(candidate)
return (target[0], top)
def pdf_outline(
name: str, data: bytes, *, pdf_headings: bool = False, ocr: bool = False
) -> PdfOutline:
"""`pdf`: the file's own `/Outlines` tree, as marks on the extracted text.
THE BRIDGE IS THE WHOLE PROBLEM, and both routes are measured rather than
argued. A bookmark states a PAGE and a y position; a candidate needs a LINE
index. On the 701-page document this was built against, 2 706 of 2 761
bookmarks share a destination page with another bookmark, so the page alone
is never a cut point.
Y ROUTE (primary). `page.extract_text_lines()` carries a `top` per line,
and the mark takes the FIRST line at or below the destination. It needs
the line splitting to be the one `page.extract_text()` produced -- an
assumption, so it is CHECKED per page and the route is used only where
the two strings are identical. Measured: 701 of 701 pages, and the
resulting index agrees with the title route on 2 762 of 2 762 nodes,
flat from a 0 pt tolerance to 8 pt and collapsing at 12 (the line
spacing). It therefore ships with NO tolerance constant at all.
TITLE ROUTE (fallback). The bookmark's title, normalised, searched in the
destination page's own lines. It resolved 2 762 of 2 763 on that
document, and its weakness is real: a title like `Armering` occurs nine
times in that structure, so it is scoped to the destination page and is
never asked a question the y route already answered.
`pdf_headings` and `ocr` are passed through so the line indices address the
SAME rendering the caller extracted. They are not options of this arm: a
plan indexes one exact string, and marks computed against another one point
at the right words in the wrong places.
"""
if Path(name).suffix.lower() != ".pdf":
return PdfOutline((), 0)
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
from pdfminer.pdfdocument import PDFNoOutlines
from pdfminer.pdfpage import PDFPage
rendered = _pdf_pages(data, pdf_headings, ocr)
starts: dict[int, int] = {}
page_lines: dict[int, list[str]] = {}
offset = 0
for number, page_text in rendered:
starts[number] = offset
page_lines[number] = page_text.split("\n")
offset += len(page_lines[number]) + 1
unresolved = 0
placed: dict[int, OutlineMark] = {}
with pdfplumber.open(io.BytesIO(data)) as pdf:
try:
nodes = list(pdf.doc.get_outlines())
except PDFNoOutlines:
# NOT an error, and not zero concepts either: this file simply
# carries no index, which is the common case and the one the
# byte-identical guarantee below rests on.
return PdfOutline((), 0)
except Exception as exc:
raise ExtractionError(
f"the PDF parser failed reading /Outlines: {exc}",
code="extractor_pdf_error",
) from exc
numbers = {
page.pageid: index + 1 for index, page in enumerate(PDFPage.create_pages(pdf.doc))
}
wanted: dict[int, list[tuple[int, str, float | None]]] = {}
for level, title, dest, action, _ in nodes:
reference, top = _outline_page_and_top(pdf.doc, dest, action)
page_number = numbers.get(getattr(reference, "objid", None))
if page_number is None or page_number not in starts:
unresolved += 1
continue
wanted.setdefault(page_number, []).append((int(level), str(title), top))
# Geometry is read only for the pages that carry a bookmark, because
# `extract_text_lines` costs a second render of every page it is asked
# about -- 78 s over 701 pages, and nothing at all over the pages no
# bookmark points at.
for page in pdf.pages:
number = page.page_number
group = wanted.get(number)
if not group:
continue
lines = page_lines[number]
tops: list[float] | None = None
geometry = page.extract_text_lines()
if [str(entry["text"]) for entry in geometry] == lines:
tops = [float(entry["top"]) for entry in geometry]
height = float(page.height)
for level, title, top in group:
index: int | None = None
if tops is not None and top is not None:
want = height - top
index = next(
(position for position, value in enumerate(tops) if value >= want),
len(tops) - 1,
)
else:
target = _normalise_outline(title)
joined = ""
bounds: list[int] = []
for rendered_line in lines:
bounds.append(len(joined))
joined += _normalise_outline(rendered_line)
found = joined.find(target)
if found >= 0:
index = max(position for position, at in enumerate(bounds) if at <= found)
if index is None:
unresolved += 1
continue
at = starts[number] + index
# FIRST in tree order wins a shared line. Two marks on one line
# would give the second an empty span, and the orphan check
# deletes an empty span silently -- the same trap the proposer
# documents for a second-pass candidate list.
placed.setdefault(at, OutlineMark(line=at, level=level, title=title))
return PdfOutline(tuple(placed[at] for at in sorted(placed)), unresolved)
def _normalise_outline(value: str) -> str:
"""Whitespace out, case folded -- the form the title route compares on."""
return re.sub(r"\s+", "", value).lower()
def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.

161
src/llm_ingestion_okf/propose.py
View file

@ -56,13 +56,14 @@ import json
import re
import sys
import unicodedata
from collections.abc import Iterable
from collections.abc import Iterable, Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any
from .errors import IngestError
from .extract import extract_text, strip_converter_attribute
from .extract import OutlineMark, extract_text, strip_converter_attribute
from .extract import pdf_outline as extract_pdf_outline
from .materialize import reduce_to_id_grammar
from .segmentation import observed_extractor_version
@ -135,6 +136,12 @@ RULE_SHEET_SECTION = "rule:sheet-section"
#: readable documents, that gate is what takes the false-positive count to 0
#: of the 31 that declare.
RULE_BOLD_TITLE = "rule:bold-title"
#: The PDF's own `/Outlines` tree. NOT `RULE_OUTLINE`: that one is Arm D, a
#: TEXT heuristic over numbered lines in the extracted text, and this one opens
#: a structure index the file already carries. A reader who cannot tell the two
#: apart in an artifact cannot tell a recovered heading from a declared one.
RULE_PDF_OUTLINE = "rule:pdf-outline"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -144,6 +151,7 @@ RULE_NAMES = (
RULE_TABLE_GRID,
RULE_SHEET_SECTION,
RULE_BOLD_TITLE,
RULE_PDF_OUTLINE,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -597,6 +605,34 @@ def _sheet_section_rows(lines: list[str]) -> dict[int, tuple[str, str]]:
return sections
#: Rules the orphan check is not asked about. D3 because a sheet row carries
#: its content in its own cells, and the bookmark arm because the check judges
#: whether a GUESS was a heading -- a question a publisher's own tree has
#: already answered, and one that deletes every container section if asked.
_ORPHAN_EXEMPT = (RULE_SHEET_SECTION, RULE_PDF_OUTLINE)
def _split_outline_title(title: str) -> tuple[str | None, str]:
"""`("14.121", "Langsg\u00e5ende sikring T1")` -- the number becomes the directory.
Two grammars, both already in this module and neither invented here:
`_NUMBERED` for a dotted section number, then `_OUTLINE`'s bare one- or
two-digit form. The second matters because a document's TOP level is where
the dot has not appeared yet -- on the corpus this arm was measured
against, 59 of 2 761 titled sections carry a dotless token and all 28 of
its chapter-level sections are among them. Without it the whole top level
would land with no section number at all, in a bundle whose every other
level has one.
"""
dotted = _NUMBERED.match(title)
if dotted is not None:
return (dotted.group("number"), dotted.group("title"))
bare = _OUTLINE.match(title)
if bare is not None:
return (bare.group("number"), bare.group("title"))
return (None, title.strip())
def find_candidates(
text: str,
*,
@ -611,6 +647,7 @@ def find_candidates(
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
outline_marks: Sequence[OutlineMark] | None = None,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -682,6 +719,22 @@ def find_candidates(
before spans are closed, so the text they opened is carried by the mark
above rather than lost.
`outline_marks` is the PDF bookmark arm, and it is the only input here that
REPLACES the rules rather than gating one of them. A non-empty list is the
publisher's own declaration of the document's structure, so nothing below
votes against it: the text heuristics, the two gates and Arm F's fold are
all skipped, and the orphan check is not applied to its marks. An EMPTY
list means "this file carries no index" and leaves every rule untouched --
the two are different answers and must not collapse into one.
THE ORPHAN EXEMPTION IS THE ONE JUDGEMENT CALL HERE, and it is the same
shape as D3's. The check asks whether anything stands UNDER a candidate's
first line, which is the right question for a heading a heuristic GUESSED
and the wrong one for a section a publisher DECLARED: a chapter followed
immediately by its first subsection is a container, not a false positive.
Measured on a 701-page process code: 683 of 2 762 marks are containers, and
applying the check scores 2 079 of 2 761 boundaries instead of 2 762.
`sheet_section_rows` is D3's gate and it is OFF at False, where the scan is
not run at all. On, a RUN of numbered rows inside an open table block cuts
it: each such row opens a candidate that reaches the next section row, or
@ -717,6 +770,41 @@ def find_candidates(
# ordering rather than a property of this corpus.
admitted = {index: title for index, _, title in runs[-1]}
# The bookmark arm, and it is computed here for the same reason `admitted`
# is: the marks must be known and SORTED before the loop. They arrive
# already deduplicated by line, so `marked` stays ordered by construction
# and no span can close before it opens.
if outline_marks:
declared: list[tuple[int, Candidate]] = [
(
mark.line,
Candidate(
title=title,
level=mark.level,
number=number,
rule=RULE_PDF_OUTLINE,
start=offsets[mark.line],
end=end_of_text,
),
)
for mark in outline_marks
if mark.line < len(offsets)
for number, title in (_split_outline_title(mark.title),)
]
return _close_candidates(
text,
declared,
offsets,
end_of_text,
joined=set(),
absorbed=set(),
contents_run=set(),
unit_fold=False,
contents_name=False,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
)
# D3's input, and the same whole-text reasoning as `admitted` above: a run
# is a property of the line list, not of a line.
sections = _sheet_section_rows(lines) if sheet_section_rows else {}
@ -887,6 +975,49 @@ def find_candidates(
if outline_gate:
marked, joined = _gate_outline(marked, joined, end_of_text, len(text))
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = (
_contents_run_positions(marked, contents_name=contents_name) if unit_fold else set()
)
return _close_candidates(
text,
marked,
offsets,
end_of_text,
joined=joined,
absorbed=absorbed,
contents_run=contents_run,
unit_fold=unit_fold,
contents_name=contents_name,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
)
def _close_candidates(
text: str,
marked: list[tuple[int, Candidate]],
offsets: list[int],
end_of_text: int,
*,
joined: set[int],
absorbed: set[int],
contents_run: set[int],
unit_fold: bool,
contents_name: bool,
first_span_from_zero: bool,
close_span_gaps: bool,
) -> list[Candidate]:
"""`marked` -> the candidate list: orphan check, fold, then span closing.
Carved out of `find_candidates` when the bookmark arm arrived, unchanged in
behaviour: the arm produces its `marked` from a structure index instead of
from the line grammar, and every step from here down is the same question
for both. Duplicating it would be two orphan checks to keep in agreement.
"""
candidates: list[Candidate] = []
# The name an orphaned heading leaves behind, and the ONE candidate allowed
# to pick it up.
@ -909,13 +1040,6 @@ def find_candidates(
# empty, so the orphan check below stops firing on it by itself and no
# branch is needed there. Computed only when the caller asked, so every
# other arm's `marked` -> `candidates` mapping is untouched code.
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = (
_contents_run_positions(marked, contents_name=contents_name) if unit_fold else set()
)
for position_in_list, (_, candidate) in enumerate(marked):
if position_in_list in absorbed:
continue
@ -936,7 +1060,7 @@ def find_candidates(
# the rule could not fire at all. It is the same shape as a contents
# list without dot leaders, and it is the reason that one needed
# `Candidate.contents`.
orphan = candidate.rule != RULE_SHEET_SECTION and (
orphan = candidate.rule not in _ORPHAN_EXEMPT and (
not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip()
)
if orphan:
@ -1329,6 +1453,7 @@ def build_plan(
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
outline_marks: Sequence[OutlineMark] | None = None,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1347,6 +1472,7 @@ def build_plan(
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
outline_marks=outline_marks,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -1428,6 +1554,7 @@ def run(
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
pdf_outline: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -1472,6 +1599,7 @@ def run(
# against another is refused by `assert_plan_applies`, which is the
# right outcome and a confusing one to debug.
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
reading_fonts = pdf_headings
# The reserve, and the reason it re-extracts rather than post-processes:
# the font reader works on the PDF's glyph geometry, which the joined
# text no longer carries. Skipped outright when the font reader is
@ -1480,6 +1608,18 @@ def run(
if pdf_headings_reserve and not pdf_headings:
if heading_reserve_applies(text, outline_run=outline_run):
text = extract_text(source.name, source_bytes, pdf_headings=True, ocr=ocr)
reading_fonts = True
# LAST, and against the text that is final: a plan indexes one exact
# string, so marks bridged onto the pre-reserve rendering would name
# the right words at the wrong offsets. `reading_fonts` is what the
# reserve may have changed, and the marks follow it.
marks = (
extract_pdf_outline(
source.name, source_bytes, pdf_headings=reading_fonts, ocr=ocr
).marks
if pdf_outline
else ()
)
except IngestError as exc:
raise ProposerError(f"cannot extract text from {source.name}: {exc}") from exc
@ -1502,6 +1642,7 @@ def run(
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
outline_marks=marks,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan