feat(propose): the PDF shipped a structure index and the build discarded it unopened
`okf build` recovers a PDF's boundaries from the shape of its page text and never opens the file's own `/Outlines` bookmark tree. On a 701-page process code whose publisher also ships a NISO-STS structure for it, measured outside this repository and reproduced here exactly: the shipped default finds 1967 of 2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their heading text PRESENT in the extracted text. The line was read; the boundary was never opened. The same file's bookmark tree matches 2761 of 2761 of those titles exactly after normalisation. `--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares. boundaries 1967 of 2761 -> 2759 of 2761 (gate was 2700) depth 1 0 of 28 -> 28 of 28 titles identical to source -- -> 2761 of 2761 false positives 163 of 2182 -> 3 of 2762 directories with two files 132 of 2050 -> 2 of 2738 front-matter concepts 72 of 2182 -> 2 of 2762 consumption fasit present 4 of 7 -> 7 of 7 hit@1 / hit@8 / hit@50 1/6 2/6 4/6 -> 3/6 5/6 6/6 It is a SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way. A PDF with no tree builds byte-identically with the flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a boundary and never a refusal of the file. The bridge from (page, y) to a line index is the whole risk, so both routes are measured. `extract_text_lines` splits lines identically to `extract_text` on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to 8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The naive "nearest line" rule was wrong on 1840 of 2762, one line early every time. The orphan check is not applied to a bookmark mark: it asks whether anything stands under a candidate's first line, which is the right question for a heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762 marks are container sections; applying it scores 2079 instead of 2759. No new dependency and no second parse of the pages: `pdfminer.six` already ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS 3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries a usable tree at all. `.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the most measurement behind it and no entry in the table. Report: docs/2026-09-10-k3-runde12-pdf-outlines.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8e1917946a
commit
e1f4faa098
12 changed files with 1261 additions and 15 deletions
174
tests/fixtures/make_fixtures.py
vendored
174
tests/fixtures/make_fixtures.py
vendored
|
|
@ -82,6 +82,172 @@ NUMBERED_FONT_CONTENT = (
|
|||
)
|
||||
|
||||
|
||||
# THREE pages and a THREE-LEVEL `/Outlines` tree, which is the shape the
|
||||
# bookmark arm has to be proved against. A two-level tree cannot tell "the
|
||||
# level the node declares" apart from "one below the root", and a
|
||||
# one-page fixture cannot tell a page-local line offset from a document-wide
|
||||
# one -- the arm's whole risk is the bridge from (page, y) to a line index.
|
||||
#
|
||||
# The third page carries FOUR lines and its second bookmark points at the
|
||||
# THIRD of them, so a bridge that resolved to the page and stopped would put
|
||||
# the mark two lines early and still look like it worked.
|
||||
OUTLINED_CONTENTS = (
|
||||
b"BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET\n",
|
||||
b"BT /F1 12 Tf 20 170 Td (1.1 Omfang) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Omfanget dekker hele arbeidet.) Tj ET\n",
|
||||
b"BT /F1 12 Tf 20 170 Td (1.2 Krav) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Kravet gjelder alle klasser.) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 130 Td (1.2.1 Materialer) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 110 Td (Materialene skal vaere godkjente.) Tj ET\n",
|
||||
)
|
||||
|
||||
#: `(title, level, page index, /XYZ top)`. The `top` values are the PDF's own
|
||||
#: bottom-up user space: 185 sits above the first line of a page and 150 above
|
||||
#: its third, which is what makes the third-line mark a statement rather than
|
||||
#: a coincidence.
|
||||
OUTLINED_TREE = (
|
||||
("1 Grunnlag", 1, 0, 185),
|
||||
("1.1 Omfang", 2, 1, 185),
|
||||
("1.2 Krav", 2, 2, 185),
|
||||
("1.2.1 Materialer", 3, 2, 150),
|
||||
)
|
||||
|
||||
#: One resolvable bookmark and one whose `/Dest` names an object that is not a
|
||||
#: page. A PDF in the wild carries these; R761 carries none of them, so
|
||||
#: without this fixture the "drop it, count it, do not fabricate a boundary"
|
||||
#: branch would ship having never run.
|
||||
BROKEN_DEST_CONTENT = (
|
||||
b"BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET\n"
|
||||
b"BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET\n",
|
||||
)
|
||||
|
||||
|
||||
def build_outlined_pdf(
|
||||
contents: tuple[bytes, ...],
|
||||
tree: tuple[tuple[str, int, int, int], ...],
|
||||
*,
|
||||
broken_dest: bool = False,
|
||||
) -> bytes:
|
||||
"""`build_paged_pdf` plus a hand-laid `/Outlines` tree in the catalog.
|
||||
|
||||
The tree is written from the flat `(title, level, page, top)` list rather
|
||||
than from a nested literal, because `/First`, `/Last`, `/Next`, `/Prev` and
|
||||
`/Parent` all have to agree with each other and with the level column --
|
||||
a hand-written nest gets one of them wrong silently and the reader then
|
||||
reports a level the document never declared.
|
||||
|
||||
`broken_dest` appends one extra item whose `/Dest` names the FONT object
|
||||
instead of a page. It is a valid indirect reference to a real object that
|
||||
is not a page, which is the failure a reader has to survive.
|
||||
"""
|
||||
count = len(contents)
|
||||
page_numbers = [3 + 2 * index for index in range(count)]
|
||||
font_number = 3 + 2 * count
|
||||
outlines_number = font_number + 1
|
||||
items = list(tree) + ([("Uoppl\u00f8selig", 1, -1, 185)] if broken_dest else [])
|
||||
item_numbers = [outlines_number + 1 + index for index in range(len(items))]
|
||||
|
||||
kids = b" ".join(str(number).encode() + b" 0 R" for number in page_numbers)
|
||||
objects = [
|
||||
b"<< /Type /Catalog /Pages 2 0 R /Outlines " + str(outlines_number).encode() + b" 0 R >>",
|
||||
b"<< /Type /Pages /Kids [" + kids + b"] /Count " + str(count).encode() + b" >>",
|
||||
]
|
||||
for index, content in enumerate(contents):
|
||||
stream_number = page_numbers[index] + 1
|
||||
objects.append(
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents "
|
||||
+ str(stream_number).encode()
|
||||
+ b" 0 R /Resources << /Font << /F1 "
|
||||
+ str(font_number).encode()
|
||||
+ b" 0 R >> >> >>"
|
||||
)
|
||||
objects.append(
|
||||
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream"
|
||||
)
|
||||
objects.append(
|
||||
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"
|
||||
)
|
||||
|
||||
# The parent of an item is the last item seen at the level above it; its
|
||||
# previous sibling is the last item seen at its OWN level under that same
|
||||
# parent. Both are resolved in one forward pass so the links cannot drift.
|
||||
parents: list[int | None] = []
|
||||
last_at_level: dict[int, int] = {}
|
||||
siblings: list[int | None] = []
|
||||
children: dict[int, list[int]] = {}
|
||||
roots: list[int] = []
|
||||
for position, (_, level, _, _) in enumerate(items):
|
||||
parent = last_at_level.get(level - 1) if level > 1 else None
|
||||
parents.append(parent)
|
||||
previous = None
|
||||
group = children.setdefault(parent, []) if parent is not None else roots
|
||||
if group:
|
||||
previous = group[-1]
|
||||
siblings.append(previous)
|
||||
group.append(position)
|
||||
last_at_level[level] = position
|
||||
for deeper in [key for key in last_at_level if key > level]:
|
||||
del last_at_level[deeper]
|
||||
|
||||
def _ref(position: int | None) -> bytes:
|
||||
return b"" if position is None else str(item_numbers[position]).encode() + b" 0 R"
|
||||
|
||||
root_body = b"<< /Type /Outlines /Count " + str(len(items)).encode() + b" >>"
|
||||
if roots:
|
||||
root_body = (
|
||||
b"<< /Type /Outlines /First "
|
||||
+ _ref(roots[0])
|
||||
+ b" /Last "
|
||||
+ _ref(roots[-1])
|
||||
+ b" /Count "
|
||||
+ str(len(items)).encode()
|
||||
+ b" >>"
|
||||
)
|
||||
objects.append(root_body)
|
||||
|
||||
for position, (title, _, page, top) in enumerate(items):
|
||||
parent = parents[position]
|
||||
parent_ref = _ref(parent) if parent is not None else str(outlines_number).encode() + b" 0 R"
|
||||
group = children.get(parent, []) if parent is not None else roots
|
||||
index_in_group = group.index(position)
|
||||
body = b"<< /Title (" + _pdf_text(title) + b") /Parent " + parent_ref
|
||||
if index_in_group > 0:
|
||||
body += b" /Prev " + _ref(group[index_in_group - 1])
|
||||
if index_in_group + 1 < len(group):
|
||||
body += b" /Next " + _ref(group[index_in_group + 1])
|
||||
own = children.get(position, [])
|
||||
if own:
|
||||
body += b" /First " + _ref(own[0]) + b" /Last " + _ref(own[-1])
|
||||
body += b" /Count " + str(len(own)).encode()
|
||||
target = str(font_number).encode() if page < 0 else str(page_numbers[page]).encode()
|
||||
body += b" /Dest [" + target + b" 0 R /XYZ 20 " + str(top).encode() + b" 0] >>"
|
||||
objects.append(body)
|
||||
|
||||
out = bytearray(b"%PDF-1.4\n")
|
||||
offsets = []
|
||||
for number, item in enumerate(objects, start=1):
|
||||
offsets.append(len(out))
|
||||
out += str(number).encode() + b" 0 obj\n" + item + b"\nendobj\n"
|
||||
|
||||
xref_at = len(out)
|
||||
size = str(len(objects) + 1).encode()
|
||||
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
|
||||
for offset in offsets:
|
||||
out += ("%010d 00000 n \n" % offset).encode()
|
||||
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
|
||||
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def _pdf_text(value: str) -> bytes:
|
||||
"""A PDF literal string in WinAnsi, with the three delimiters escaped."""
|
||||
raw = value.encode("cp1252")
|
||||
for character in (b"\\", b"(", b")"):
|
||||
raw = raw.replace(character, b"\\" + character)
|
||||
return raw
|
||||
|
||||
|
||||
def build_two_font_pdf(content: bytes) -> bytes:
|
||||
"""A one-page PDF whose resources declare BOTH a regular and a bold font.
|
||||
|
||||
|
|
@ -439,6 +605,14 @@ if __name__ == "__main__":
|
|||
(HERE / "numbered-font-krav.pdf").write_bytes(build_two_font_pdf(NUMBERED_FONT_CONTENT))
|
||||
print("wrote numbered-font-krav.pdf")
|
||||
|
||||
(HERE / "outlined-krav.pdf").write_bytes(build_outlined_pdf(OUTLINED_CONTENTS, OUTLINED_TREE))
|
||||
print("wrote outlined-krav.pdf")
|
||||
|
||||
(HERE / "outline-broken-dest.pdf").write_bytes(
|
||||
build_outlined_pdf(BROKEN_DEST_CONTENT, (("1 Grunnlag", 1, 0, 185),), broken_dest=True)
|
||||
)
|
||||
print("wrote outline-broken-dest.pdf")
|
||||
|
||||
for name, parts in (
|
||||
("two-line-krav.docx", _DOCX_PARTS),
|
||||
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),
|
||||
|
|
|
|||
45
tests/fixtures/outline-broken-dest.pdf
vendored
Normal file
45
tests/fixtures/outline-broken-dest.pdf
vendored
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<< /Type /Catalog /Pages 2 0 R /Outlines 6 0 R >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Length 105 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
|
||||
endobj
|
||||
6 0 obj
|
||||
<< /Type /Outlines /First 7 0 R /Last 8 0 R /Count 2 >>
|
||||
endobj
|
||||
7 0 obj
|
||||
<< /Title (1 Grunnlag) /Parent 6 0 R /Next 8 0 R /Dest [3 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
8 0 obj
|
||||
<< /Title (Uoppløselig) /Parent 6 0 R /Prev 7 0 R /Dest [5 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
xref
|
||||
0 9
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000074 00000 n
|
||||
0000000131 00000 n
|
||||
0000000257 00000 n
|
||||
0000000412 00000 n
|
||||
0000000509 00000 n
|
||||
0000000580 00000 n
|
||||
0000000675 00000 n
|
||||
trailer
|
||||
<< /Size 9 /Root 1 0 R >>
|
||||
startxref
|
||||
771
|
||||
%%EOF
|
||||
79
tests/fixtures/outlined-krav.pdf
vendored
Normal file
79
tests/fixtures/outlined-krav.pdf
vendored
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<< /Type /Catalog /Pages 2 0 R /Outlines 10 0 R >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Type /Pages /Kids [3 0 R 5 0 R 7 0 R] /Count 3 >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 9 0 R >> >> >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Length 105 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 6 0 R /Resources << /Font << /F1 9 0 R >> >> >>
|
||||
endobj
|
||||
6 0 obj
|
||||
<< /Length 104 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1.1 Omfang) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Omfanget dekker hele arbeidet.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
7 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 8 0 R /Resources << /Font << /F1 9 0 R >> >> >>
|
||||
endobj
|
||||
8 0 obj
|
||||
<< /Length 213 >>
|
||||
stream
|
||||
BT /F1 12 Tf 20 170 Td (1.2 Krav) Tj ET
|
||||
BT /F1 12 Tf 20 150 Td (Kravet gjelder alle klasser.) Tj ET
|
||||
BT /F1 12 Tf 20 130 Td (1.2.1 Materialer) Tj ET
|
||||
BT /F1 12 Tf 20 110 Td (Materialene skal vaere godkjente.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
9 0 obj
|
||||
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
|
||||
endobj
|
||||
10 0 obj
|
||||
<< /Type /Outlines /First 11 0 R /Last 11 0 R /Count 4 >>
|
||||
endobj
|
||||
11 0 obj
|
||||
<< /Title (1 Grunnlag) /Parent 10 0 R /First 12 0 R /Last 13 0 R /Count 2 /Dest [3 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
12 0 obj
|
||||
<< /Title (1.1 Omfang) /Parent 11 0 R /Next 13 0 R /Dest [5 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
13 0 obj
|
||||
<< /Title (1.2 Krav) /Parent 11 0 R /Prev 12 0 R /First 14 0 R /Last 14 0 R /Count 1 /Dest [7 0 R /XYZ 20 185 0] >>
|
||||
endobj
|
||||
14 0 obj
|
||||
<< /Title (1.2.1 Materialer) /Parent 13 0 R /Dest [7 0 R /XYZ 20 150 0] >>
|
||||
endobj
|
||||
xref
|
||||
0 15
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000075 00000 n
|
||||
0000000144 00000 n
|
||||
0000000270 00000 n
|
||||
0000000425 00000 n
|
||||
0000000551 00000 n
|
||||
0000000705 00000 n
|
||||
0000000831 00000 n
|
||||
0000001094 00000 n
|
||||
0000001191 00000 n
|
||||
0000001265 00000 n
|
||||
0000001386 00000 n
|
||||
0000001484 00000 n
|
||||
0000001616 00000 n
|
||||
trailer
|
||||
<< /Size 15 /Root 1 0 R >>
|
||||
startxref
|
||||
1707
|
||||
%%EOF
|
||||
|
|
@ -335,14 +335,19 @@ def test_evidence_class_is_asserted_not_commented() -> None:
|
|||
measured on. A comment saying so rots; an assertion that names them keeps
|
||||
an unmeasured row from quietly presenting as a supported one.
|
||||
|
||||
The table is NOT the converter's rows alone: `.html` is core-supported and
|
||||
was the one core row that had never met a real document, so it carries a
|
||||
class here too (2026-09-09, `measured`, 828 files).
|
||||
The table is NOT the converter's rows alone: `.html` and `.pdf` are
|
||||
core-supported and carry a class here too -- `.html` since 2026-09-09
|
||||
(`measured`, 828 files), `.pdf` since 2026-09-10 (`measured`, eight corpus
|
||||
documents with a hand-counted fasit plus a publisher's own 2 761-section
|
||||
structure for a 701-page one). `.pdf` was the row with the most measurement
|
||||
behind it and no row in the table at all, which is the one way a table like
|
||||
this can mislead while every entry in it is true.
|
||||
"""
|
||||
from llm_ingestion_okf.extract import _EVIDENCE, _PANDOC_FORMATS
|
||||
|
||||
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html"}, "every row needs a class"
|
||||
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html", ".pdf"}, "every row needs a class"
|
||||
assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {
|
||||
".pdf",
|
||||
".docx",
|
||||
".xlsx",
|
||||
".html",
|
||||
|
|
|
|||
131
tests/test_pdf_outline.py
Normal file
131
tests/test_pdf_outline.py
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
"""The PDF's own `/Outlines` tree as a segmentation source.
|
||||
|
||||
MEASURED, OUTSIDE THIS REPOSITORY, ON ONE DOCUMENT: a 701-page process code
|
||||
whose publisher also ships a NISO-STS structure for it. `okf build` on the
|
||||
default recovered 1 967 of its 2 761 titled sections, 0 of 28 at the top level,
|
||||
and **794 of 794 misses had their heading text present in the extracted text**
|
||||
-- the boundary was never opened, the line was read. The same file carries an
|
||||
`/Outlines` tree of 2 763 nodes which, after `re.sub(r"\\s+","",s).lower()`,
|
||||
matches 2 761 of 2 761 STS titles exactly. The index shipped inside the file
|
||||
and the build discarded it unopened.
|
||||
|
||||
WHAT THESE TESTS PIN, and why each one exists rather than "it found more":
|
||||
|
||||
- the LEVEL a node declares, not its distance from the root. A two-level tree
|
||||
cannot tell those apart, so the fixture has three.
|
||||
- the LINE a mark lands on. 2 706 of 2 761 bookmarks in that document share a
|
||||
destination page with another bookmark, so a bridge that resolved to the page
|
||||
and stopped would be wrong on almost every node while still looking like it
|
||||
worked. The fixture's last page carries four lines and its second bookmark
|
||||
points at the third.
|
||||
- a PDF with no `/Outlines` behaving IDENTICALLY with the arm on -- `pdfminer`
|
||||
raises `PDFNoOutlines` there, and "this file has no index" is not an error.
|
||||
- an unresolvable `/Dest` being dropped and COUNTED. That document has 0 of
|
||||
2 763; a PDF in the wild has them, so without this the branch would ship
|
||||
having never run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_okf import extract, propose
|
||||
|
||||
pdfplumber = pytest.importorskip("pdfplumber")
|
||||
|
||||
FIXTURES = Path(__file__).parent / "fixtures"
|
||||
OUTLINED = FIXTURES / "outlined-krav.pdf"
|
||||
BROKEN = FIXTURES / "outline-broken-dest.pdf"
|
||||
NO_OUTLINE = FIXTURES / "three-page-krav.pdf"
|
||||
|
||||
|
||||
def test_marks_carry_the_declared_level_and_the_line_they_land_on() -> None:
|
||||
data = OUTLINED.read_bytes()
|
||||
text = extract.extract_text(OUTLINED.name, data)
|
||||
lines = text.split("\n")
|
||||
outline = extract.pdf_outline(OUTLINED.name, data)
|
||||
|
||||
assert outline.unresolved == 0
|
||||
assert [(mark.line, mark.level, mark.title) for mark in outline.marks] == [
|
||||
(0, 1, "1 Grunnlag"),
|
||||
(3, 2, "1.1 Omfang"),
|
||||
(6, 2, "1.2 Krav"),
|
||||
(8, 3, "1.2.1 Materialer"),
|
||||
]
|
||||
# The offsets, stated against the text rather than against the tree: a mark
|
||||
# that named the right title on the wrong line would pass the list above if
|
||||
# the list were derived from the same walk.
|
||||
for mark in outline.marks:
|
||||
assert lines[mark.line] == mark.title
|
||||
|
||||
|
||||
def test_the_last_mark_is_on_the_third_line_of_its_page_not_the_first() -> None:
|
||||
"""Page resolution alone would put it two lines early, and still look right."""
|
||||
data = OUTLINED.read_bytes()
|
||||
marks = extract.pdf_outline(OUTLINED.name, data).marks
|
||||
page_three = [mark for mark in marks if mark.title.startswith("1.2")]
|
||||
assert [mark.line for mark in page_three] == [6, 8]
|
||||
|
||||
|
||||
def test_a_pdf_with_no_outlines_yields_no_marks_and_no_error() -> None:
|
||||
data = NO_OUTLINE.read_bytes()
|
||||
outline = extract.pdf_outline(NO_OUTLINE.name, data)
|
||||
assert outline.marks == ()
|
||||
assert outline.unresolved == 0
|
||||
|
||||
|
||||
def test_a_non_pdf_is_not_asked_the_question() -> None:
|
||||
outline = extract.pdf_outline("notat.md", b"# Overskrift\n\nBrodtekst.\n")
|
||||
assert outline.marks == ()
|
||||
assert outline.unresolved == 0
|
||||
|
||||
|
||||
def test_an_unresolvable_destination_is_dropped_and_counted() -> None:
|
||||
data = BROKEN.read_bytes()
|
||||
outline = extract.pdf_outline(BROKEN.name, data)
|
||||
assert [mark.title for mark in outline.marks] == ["1 Grunnlag"]
|
||||
assert outline.unresolved == 1
|
||||
|
||||
|
||||
def test_the_outline_replaces_the_text_heuristics_when_it_is_given() -> None:
|
||||
data = OUTLINED.read_bytes()
|
||||
text = extract.extract_text(OUTLINED.name, data)
|
||||
marks = extract.pdf_outline(OUTLINED.name, data).marks
|
||||
|
||||
candidates = propose.find_candidates(text, outline_marks=marks, close_span_gaps=True)
|
||||
assert [(c.number, c.title, c.level, c.rule) for c in candidates] == [
|
||||
("1", "Grunnlag", 1, propose.RULE_PDF_OUTLINE),
|
||||
("1.1", "Omfang", 2, propose.RULE_PDF_OUTLINE),
|
||||
("1.2", "Krav", 2, propose.RULE_PDF_OUTLINE),
|
||||
("1.2.1", "Materialer", 3, propose.RULE_PDF_OUTLINE),
|
||||
]
|
||||
# The spans partition the text: the arm cuts, it does not drop.
|
||||
assert candidates[0].start == 0
|
||||
assert candidates[-1].end == len(text)
|
||||
|
||||
|
||||
def test_a_declared_section_with_no_prose_of_its_own_survives() -> None:
|
||||
"""The orphan check asks the wrong question of a publisher's own tree.
|
||||
|
||||
Measured on that 701-page document: 683 of its 2 762 bookmark marks are
|
||||
followed immediately by their first subsection, with no prose between. They
|
||||
are container sections, not mis-detected headings -- the check exists to
|
||||
catch a heuristic's false positive, and there is no heuristic here. Left
|
||||
in, the arm scores 2 079 of 2 761 instead of 2 762 of 2 763.
|
||||
"""
|
||||
text = "1 Grunnlag\n1.1 Omfang\nOmfanget dekker alt.\n"
|
||||
marks = (
|
||||
extract.OutlineMark(line=0, level=1, title="1 Grunnlag"),
|
||||
extract.OutlineMark(line=1, level=2, title="1.1 Omfang"),
|
||||
)
|
||||
candidates = propose.find_candidates(text, outline_marks=marks)
|
||||
assert [c.title for c in candidates] == ["Grunnlag", "Omfang"]
|
||||
|
||||
|
||||
def test_an_empty_mark_list_leaves_every_other_rule_untouched() -> None:
|
||||
"""`()` is "this file has no index", never "propose nothing"."""
|
||||
data = OUTLINED.read_bytes()
|
||||
text = extract.extract_text(OUTLINED.name, data)
|
||||
assert propose.find_candidates(text, outline_marks=()) == propose.find_candidates(text)
|
||||
Loading…
Add table
Add a link
Reference in a new issue