feat(propose): the PDF shipped a structure index and the build discarded it unopened
`okf build` recovers a PDF's boundaries from the shape of its page text and never opens the file's own `/Outlines` bookmark tree. On a 701-page process code whose publisher also ships a NISO-STS structure for it, measured outside this repository and reproduced here exactly: the shipped default finds 1967 of 2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their heading text PRESENT in the extracted text. The line was read; the boundary was never opened. The same file's bookmark tree matches 2761 of 2761 of those titles exactly after normalisation. `--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares. boundaries 1967 of 2761 -> 2759 of 2761 (gate was 2700) depth 1 0 of 28 -> 28 of 28 titles identical to source -- -> 2761 of 2761 false positives 163 of 2182 -> 3 of 2762 directories with two files 132 of 2050 -> 2 of 2738 front-matter concepts 72 of 2182 -> 2 of 2762 consumption fasit present 4 of 7 -> 7 of 7 hit@1 / hit@8 / hit@50 1/6 2/6 4/6 -> 3/6 5/6 6/6 It is a SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way. A PDF with no tree builds byte-identically with the flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a boundary and never a refusal of the file. The bridge from (page, y) to a line index is the whole risk, so both routes are measured. `extract_text_lines` splits lines identically to `extract_text` on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to 8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The naive "nearest line" rule was wrong on 1840 of 2762, one line early every time. The orphan check is not applied to a bookmark mark: it asks whether anything stands under a candidate's first line, which is the right question for a heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762 marks are container sections; applying it scores 2079 instead of 2759. No new dependency and no second parse of the pages: `pdfminer.six` already ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS 3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries a usable tree at all. `.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the most measurement behind it and no entry in the table. Report: docs/2026-09-10-k3-runde12-pdf-outlines.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8e1917946a
commit
e1f4faa098
12 changed files with 1261 additions and 15 deletions
174
tests/fixtures/make_fixtures.py
vendored
174
tests/fixtures/make_fixtures.py
vendored
|
|
@ -82,6 +82,172 @@ NUMBERED_FONT_CONTENT = (
|
|||
)
|
||||
|
||||
|
||||
# THREE pages and a THREE-LEVEL `/Outlines` tree, which is the shape the
|
||||
# bookmark arm has to be proved against. A two-level tree cannot tell "the
|
||||
# level the node declares" apart from "one below the root", and a
|
||||
# one-page fixture cannot tell a page-local line offset from a document-wide
|
||||
# one -- the arm's whole risk is the bridge from (page, y) to a line index.
|
||||
#
|
||||
# The third page carries FOUR lines and its second bookmark points at the
|
||||
# THIRD of them, so a bridge that resolved to the page and stopped would put
|
||||
# the mark two lines early and still look like it worked.
|
||||
OUTLINED_CONTENTS = (
|
||||
b"BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET\n",
|
||||
b"BT /F1 12 Tf 20 170 Td (1.1 Omfang) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Omfanget dekker hele arbeidet.) Tj ET\n",
|
||||
b"BT /F1 12 Tf 20 170 Td (1.2 Krav) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Kravet gjelder alle klasser.) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 130 Td (1.2.1 Materialer) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 110 Td (Materialene skal vaere godkjente.) Tj ET\n",
|
||||
)
|
||||
|
||||
#: `(title, level, page index, /XYZ top)`. The `top` values are the PDF's own
|
||||
#: bottom-up user space: 185 sits above the first line of a page and 150 above
|
||||
#: its third, which is what makes the third-line mark a statement rather than
|
||||
#: a coincidence.
|
||||
OUTLINED_TREE = (
|
||||
("1 Grunnlag", 1, 0, 185),
|
||||
("1.1 Omfang", 2, 1, 185),
|
||||
("1.2 Krav", 2, 2, 185),
|
||||
("1.2.1 Materialer", 3, 2, 150),
|
||||
)
|
||||
|
||||
#: One resolvable bookmark and one whose `/Dest` names an object that is not a
|
||||
#: page. A PDF in the wild carries these; R761 carries none of them, so
|
||||
#: without this fixture the "drop it, count it, do not fabricate a boundary"
|
||||
#: branch would ship having never run.
|
||||
BROKEN_DEST_CONTENT = (
|
||||
b"BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET\n",
|
||||
)
|
||||
|
||||
|
||||
def build_outlined_pdf(
|
||||
contents: tuple[bytes, ...],
|
||||
tree: tuple[tuple[str, int, int, int], ...],
|
||||
*,
|
||||
broken_dest: bool = False,
|
||||
) -> bytes:
|
||||
"""`build_paged_pdf` plus a hand-laid `/Outlines` tree in the catalog.
|
||||
|
||||
The tree is written from the flat `(title, level, page, top)` list rather
|
||||
than from a nested literal, because `/First`, `/Last`, `/Next`, `/Prev` and
|
||||
`/Parent` all have to agree with each other and with the level column --
|
||||
a hand-written nest gets one of them wrong silently and the reader then
|
||||
reports a level the document never declared.
|
||||
|
||||
`broken_dest` appends one extra item whose `/Dest` names the FONT object
|
||||
instead of a page. It is a valid indirect reference to a real object that
|
||||
is not a page, which is the failure a reader has to survive.
|
||||
"""
|
||||
count = len(contents)
|
||||
page_numbers = [3 + 2 * index for index in range(count)]
|
||||
font_number = 3 + 2 * count
|
||||
outlines_number = font_number + 1
|
||||
items = list(tree) + ([("Uoppl\u00f8selig", 1, -1, 185)] if broken_dest else [])
|
||||
item_numbers = [outlines_number + 1 + index for index in range(len(items))]
|
||||
|
||||
kids = b" ".join(str(number).encode() + b" 0 R" for number in page_numbers)
|
||||
objects = [
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Outlines " + str(outlines_number).encode() + b" 0 R >>",
|
||||
b"<< /Type /Pages /Kids [" + kids + b"] /Count " + str(count).encode() + b" >>",
|
||||
]
|
||||
for index, content in enumerate(contents):
|
||||
stream_number = page_numbers[index] + 1
|
||||
objects.append(
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents "
|
||||
+ str(stream_number).encode()
|
||||
+ b" 0 R /Resources << /Font << /F1 "
|
||||
+ str(font_number).encode()
|
||||
+ b" 0 R >> >> >>"
|
||||
)
|
||||
objects.append(
|
||||
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream"
|
||||
)
|
||||
objects.append(
|
||||
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"
|
||||
)
|
||||
|
||||
# The parent of an item is the last item seen at the level above it; its
|
||||
# previous sibling is the last item seen at its OWN level under that same
|
||||
# parent. Both are resolved in one forward pass so the links cannot drift.
|
||||
parents: list[int | None] = []
|
||||
last_at_level: dict[int, int] = {}
|
||||
siblings: list[int | None] = []
|
||||
children: dict[int, list[int]] = {}
|
||||
roots: list[int] = []
|
||||
for position, (_, level, _, _) in enumerate(items):
|
||||
parent = last_at_level.get(level - 1) if level > 1 else None
|
||||
parents.append(parent)
|
||||
previous = None
|
||||
group = children.setdefault(parent, []) if parent is not None else roots
|
||||
if group:
|
||||
previous = group[-1]
|
||||
siblings.append(previous)
|
||||
group.append(position)
|
||||
last_at_level[level] = position
|
||||
for deeper in [key for key in last_at_level if key > level]:
|
||||
del last_at_level[deeper]
|
||||
|
||||
def _ref(position: int | None) -> bytes:
|
||||
return b"" if position is None else str(item_numbers[position]).encode() + b" 0 R"
|
||||
|
||||
root_body = b"<< /Type /Outlines /Count " + str(len(items)).encode() + b" >>"
|
||||
if roots:
|
||||
root_body = (
|
||||
b"<< /Type /Outlines /First "
|
||||
+ _ref(roots[0])
|
||||
+ b" /Last "
|
||||
+ _ref(roots[-1])
|
||||
+ b" /Count "
|
||||
+ str(len(items)).encode()
|
||||
+ b" >>"
|
||||
)
|
||||
objects.append(root_body)
|
||||
|
||||
for position, (title, _, page, top) in enumerate(items):
|
||||
parent = parents[position]
|
||||
parent_ref = _ref(parent) if parent is not None else str(outlines_number).encode() + b" 0 R"
|
||||
group = children.get(parent, []) if parent is not None else roots
|
||||
index_in_group = group.index(position)
|
||||
body = b"<< /Title (" + _pdf_text(title) + b") /Parent " + parent_ref
|
||||
if index_in_group > 0:
|
||||
body += b" /Prev " + _ref(group[index_in_group - 1])
|
||||
if index_in_group + 1 < len(group):
|
||||
body += b" /Next " + _ref(group[index_in_group + 1])
|
||||
own = children.get(position, [])
|
||||
if own:
|
||||
body += b" /First " + _ref(own[0]) + b" /Last " + _ref(own[-1])
|
||||
body += b" /Count " + str(len(own)).encode()
|
||||
target = str(font_number).encode() if page < 0 else str(page_numbers[page]).encode()
|
||||
body += b" /Dest [" + target + b" 0 R /XYZ 20 " + str(top).encode() + b" 0] >>"
|
||||
objects.append(body)
|
||||
|
||||
out = bytearray(b"%PDF-1.4\n")
|
||||
offsets = []
|
||||
for number, item in enumerate(objects, start=1):
|
||||
offsets.append(len(out))
|
||||
out += str(number).encode() + b" 0 obj\n" + item + b"\nendobj\n"
|
||||
|
||||
xref_at = len(out)
|
||||
size = str(len(objects) + 1).encode()
|
||||
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
|
||||
for offset in offsets:
|
||||
out += ("%010d 00000 n \n" % offset).encode()
|
||||
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
|
||||
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def _pdf_text(value: str) -> bytes:
|
||||
"""A PDF literal string in WinAnsi, with the three delimiters escaped."""
|
||||
raw = value.encode("cp1252")
|
||||
for character in (b"\\", b"(", b")"):
|
||||
raw = raw.replace(character, b"\\" + character)
|
||||
return raw
|
||||
|
||||
|
||||
def build_two_font_pdf(content: bytes) -> bytes:
|
||||
"""A one-page PDF whose resources declare BOTH a regular and a bold font.
|
||||
|
||||
|
|
@ -439,6 +605,14 @@ if __name__ == "__main__":
|
|||
(HERE / "numbered-font-krav.pdf").write_bytes(build_two_font_pdf(NUMBERED_FONT_CONTENT))
|
||||
print("wrote numbered-font-krav.pdf")
|
||||
|
||||
(HERE / "outlined-krav.pdf").write_bytes(build_outlined_pdf(OUTLINED_CONTENTS, OUTLINED_TREE))
|
||||
print("wrote outlined-krav.pdf")
|
||||
|
||||
(HERE / "outline-broken-dest.pdf").write_bytes(
|
||||
build_outlined_pdf(BROKEN_DEST_CONTENT, (("1 Grunnlag", 1, 0, 185),), broken_dest=True)
|
||||
)
|
||||
print("wrote outline-broken-dest.pdf")
|
||||
|
||||
for name, parts in (
|
||||
("two-line-krav.docx", _DOCX_PARTS),
|
||||
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),
|
||||
|
|
|
|||
45
tests/fixtures/outline-broken-dest.pdf
vendored
Normal file
45
tests/fixtures/outline-broken-dest.pdf
vendored
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<< /Type /Catalog /Pages 2 0 R /Outlines 6 0 R >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Length 105 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
|
||||
endobj
|
||||
6 0 obj
|
||||
<< /Type /Outlines /First 7 0 R /Last 8 0 R /Count 2 >>
|
||||
endobj
|
||||
7 0 obj
|
||||
<< /Title (1 Grunnlag) /Parent 6 0 R /Next 8 0 R /Dest [3 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
8 0 obj
|
||||
<< /Title (Uoppløselig) /Parent 6 0 R /Prev 7 0 R /Dest [5 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
xref
|
||||
0 9
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000074 00000 n
|
||||
0000000131 00000 n
|
||||
0000000257 00000 n
|
||||
0000000412 00000 n
|
||||
0000000509 00000 n
|
||||
0000000580 00000 n
|
||||
0000000675 00000 n
|
||||
trailer
|
||||
<< /Size 9 /Root 1 0 R >>
|
||||
startxref
|
||||
771
|
||||
%%EOF
|
||||
79
tests/fixtures/outlined-krav.pdf
vendored
Normal file
79
tests/fixtures/outlined-krav.pdf
vendored
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<< /Type /Catalog /Pages 2 0 R /Outlines 10 0 R >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Type /Pages /Kids [3 0 R 5 0 R 7 0 R] /Count 3 >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 9 0 R >> >> >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Length 105 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 6 0 R /Resources << /Font << /F1 9 0 R >> >> >>
|
||||
endobj
|
||||
6 0 obj
|
||||
<< /Length 104 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1.1 Omfang) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Omfanget dekker hele arbeidet.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
7 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 8 0 R /Resources << /Font << /F1 9 0 R >> >> >>
|
||||
endobj
|
||||
8 0 obj
|
||||
<< /Length 213 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1.2 Krav) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Kravet gjelder alle klasser.) Tj ET
|
||||
BT /F1 12 Tf 20 130 Td (1.2.1 Materialer) Tj ET
|
||||
BT /F1 12 Tf 20 110 Td (Materialene skal vaere godkjente.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
9 0 obj
|
||||
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
|
||||
endobj
|
||||
10 0 obj
|
||||
<< /Type /Outlines /First 11 0 R /Last 11 0 R /Count 4 >>
|
||||
endobj
|
||||
11 0 obj
|
||||
<< /Title (1 Grunnlag) /Parent 10 0 R /First 12 0 R /Last 13 0 R /Count 2 /Dest [3 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
12 0 obj
|
||||
<< /Title (1.1 Omfang) /Parent 11 0 R /Next 13 0 R /Dest [5 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
13 0 obj
|
||||
<< /Title (1.2 Krav) /Parent 11 0 R /Prev 12 0 R /First 14 0 R /Last 14 0 R /Count 1 /Dest [7 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
14 0 obj
|
||||
<< /Title (1.2.1 Materialer) /Parent 13 0 R /Dest [7 0 R /XYZ 20 150 0] >>
|
||||
endobj
|
||||
xref
|
||||
0 15
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000075 00000 n
|
||||
0000000144 00000 n
|
||||
0000000270 00000 n
|
||||
0000000425 00000 n
|
||||
0000000551 00000 n
|
||||
0000000705 00000 n
|
||||
0000000831 00000 n
|
||||
0000001094 00000 n
|
||||
0000001191 00000 n
|
||||
0000001265 00000 n
|
||||
0000001386 00000 n
|
||||
0000001484 00000 n
|
||||
0000001616 00000 n
|
||||
trailer
|
||||
<< /Size 15 /Root 1 0 R >>
|
||||
startxref
|
||||
1707
|
||||
%%EOF
|
||||
Loading…
Add table
Add a link
Reference in a new issue