open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): the PDF shipped a structure index and the build discarded it unopened

Browse source 
`okf build` recovers a PDF's boundaries from the shape of its page text and
never opens the file's own `/Outlines` bookmark tree. On a 701-page process
code whose publisher also ships a NISO-STS structure for it, measured outside
this repository and reproduced here exactly: the shipped default finds 1967 of
2761 titled sections, 0 of its 28 chapters, and 794 of 794 misses have their
heading text PRESENT in the extracted text. The line was read; the boundary
was never opened. The same file's bookmark tree matches 2761 of 2761 of those
titles exactly after normalisation.

`--pdf-outline`, OFF, cuts a PDF at the boundaries its tree declares.

  boundaries                 1967 of 2761  ->  2759 of 2761  (gate was 2700)
  depth 1                       0 of 28    ->     28 of 28
  titles identical to source        --     ->   2761 of 2761
  false positives             163 of 2182  ->      3 of 2762
  directories with two files  132 of 2050  ->      2 of 2738
  front-matter concepts        72 of 2182  ->      2 of 2762
  consumption fasit present       4 of 7   ->        7 of 7
  hit@1 / hit@8 / hit@50      1/6 2/6 4/6  ->   3/6 5/6 6/6

It is a SEGMENTATION arm, not a reader option: the extracted text is byte for
byte the same either way. A PDF with no tree builds byte-identically with the
flag on -- `diff -r` empty across the pre-change tree, the arm off and the arm
on. An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a
boundary and never a refusal of the file.

The bridge from (page, y) to a line index is the whole risk, so both routes
are measured. `extract_text_lines` splits lines identically to `extract_text`
on 701 of 701 pages, and is CHECKED per page rather than assumed. The y route
and the title route disagree on 0 of 2762 nodes, flat from a 0pt tolerance to
8pt and collapsing at 12pt, so the rule ships with no tolerance constant. The
naive "nearest line" rule was wrong on 1840 of 2762, one line early every time.

The orphan check is not applied to a bookmark mark: it asks whether anything
stands under a candidate's first line, which is the right question for a
heuristic's guess and the wrong one for a publisher's declaration. 683 of 2762
marks are container sections; applying it scores 2079 instead of 2759.

No new dependency and no second parse of the pages: `pdfminer.six` already
ships under `pdfplumber` in `[extract]`. 119.22s -> 183.31s wall, peak RSS
3252 -> 3251 MiB. The default does not move; 1 of the 8 reference PDFs carries
a usable tree at all.

`.pdf` also gains its `_EVIDENCE` row, as `measured` -- it was the row with the
most measurement behind it and no entry in the table.

Report: docs/2026-09-10-k3-runde12-pdf-outlines.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 02:27:41 +02:00
commit e1f4faa098
12 changed files with 1261 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

40
CHANGELOG.md
View file

@ -7,6 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **`--pdf-outline` (OFF): cut a PDF at the boundaries its own `/Outlines`
bookmark tree declares.** Measured outside this repository on one 701-page
process code whose publisher also ships a NISO-STS structure for it: the
shipped default recovers **1 967 of 2 761** titled sections, **0 of its 28**
chapters, and **794 of 794** misses have their heading text present in the
extracted text -- the line was read, the boundary was never opened. The same
file carries a 2 763-node bookmark tree that matches **2 761 of 2 761** STS
titles exactly after `re.sub(r"\s+","",s).lower()`. With the arm on:
**2 759 of 2 761 boundaries (99.9 %)**, depth 1 **28 of 28**, concept titles
identical to the publisher's own after that normalisation **2 761 of 2 761**,
false positives **3 of 2 762** (was 163 of 2 182), directories carrying two
concept files **2** (was 132, of which 65 were a contents copy and a body
section under one id), front-matter concepts **2 of 2 762** (was 72). Seven
of seven consumption fasit now exist in the bundle (was four); hit@1/8/50 is
**3/6 · 5/6 · 6/6** against **1/6 · 2/6 · 4/6**.
- It is a SEGMENTATION arm, not a reader option: the extracted text is byte
for byte the same either way, and `--pdf-headings`/`--ocr` stay the only
two things that change what a PDF says.
- A PDF with no bookmark tree is **byte-identical with the flag on**;
`pdfminer`'s `PDFNoOutlines` is "this file has no index", never an error.
Measured on the five-document smoke folder: `diff -r` empty against both
the arm off and the pre-change tree.
- No new dependency and no second parse of the file's pages: the tree is read
through `pdfminer.six`'s `PDFDocument.get_outlines()`, which
`pdfplumber` already ships under the existing `[extract]` extra. Cost on
the 701-page document: 119.22 s -> 183.31 s wall, peak RSS 3 252 -> 3 251
MiB.
- An unresolvable `/Dest` is dropped and COUNTED, never fabricated into a
boundary and never a refusal of the file. That document has 0 of 2 763;
one of the eight reference PDFs in this repository's own sample has 2 of 2.
- **The default does not move in this release.** Reach measured: 1 of the 8
reference PDFs carries a usable tree at all.
- **`.pdf` has a row in `extract._EVIDENCE`, as `measured`.** It was the row
with the most measurement behind it and no entry in the table, which is the
one way a table like that misleads while every entry in it is true.
## [0.7.0] — 2026-09-09
The first screen an agent reads, the three shapes of request the skill answers,

42
CLAUDE.md
View file

@ -42,7 +42,9 @@ one boundary rule:
files**, character ratio **1.000000**. `_SKIP_TAGS` stays `{script, style}`.
Exposure elsewhere measured rather than argued: **0 of 86** K2 corpus files and
**0 of 5** smoke-folder files are HTML, and the smoke bundle is byte-identical
before and after. `_EVIDENCE` gains a `.html` row at `measured`, with the limit
before and after. `_EVIDENCE` gains a `.html` row at `measured` -- and a `.pdf` row at
`measured` since 2026-09-10, the row with the most measurement behind it and
no entry in the table at all -- with the limit
that travels with it -- one product, one format, one publisher, and a
generator's cut, not 828 documents anyone wrote. `pdf`/`docx`/`xlsx` only via
the optional `[extract]` extra; without it those types are rejected
@ -476,6 +478,44 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **A TWELFTH flag, `--pdf-outline`, is OFF** (round 12, 2026-09-10) and it is
the only one here that does not read the extracted text at all: it cuts a PDF
at the boundaries its own `/Outlines` bookmark tree declares. It is NOT Arm D
-- `--outline-run`/`--outline-gate` are a TEXT heuristic over numbered lines
in the extracted text, and this opens a structure index the file already
carries. Measured on ONE 701-page process code whose publisher also ships a
NISO-STS structure for it, so the fasit is the publisher's own 2 761 titled
sections: the shipped default finds **1 967 of 2 761**, **0 of its 28**
chapters, and **794 of 794** misses have their heading text PRESENT in the
extracted text -- the line is read, the boundary is never opened. With the
arm: **2 759 of 2 761 (99.9 %)**, chapter level **28 of 28**, concept titles
identical to the source after normalisation **2 761 of 2 761** (the bookmark
title is complete because it does not come from the page), false positives
**163 of 2 182 -> 3 of 2 762**, directories carrying two concept files
**132 -> 2** with the 65 contents-copy pairs at **0**, front-matter concepts
**72 -> 2**. Consumption: fasit present in the bundle **4 of 7 -> 7 of 7**,
hit@1/8/50 **1/6 - 2/6 - 4/6 -> 3/6 - 5/6 - 6/6**; the known-positive is a
real concept now and ranks 13, so it is delivered at k=50 and not at k=8 --
the segmentation half of that row is closed and the ranking half is not.
**The bridge from (page, `/XYZ` top) to a line index is the whole risk and
BOTH routes are measured:** `extract_text_lines` splits lines identically to
`extract_text` on **701 of 701** pages and that check SHIPS per page, the y
route and the title route disagree on **0 of 2 762**, flat from 0 to 8 pt and
collapsing at 12, so the rule carries **no tolerance constant**; the naive
nearest-line rule was wrong on **1 840 of 2 762**, one line early every time.
The orphan check is NOT applied to a bookmark mark -- it asks whether anything
stands under a candidate's first line, the right question for a guess and the
wrong one for a publisher's declaration; **683 of 2 762** marks are container
sections and applying it scores 2 079. An unresolvable `/Dest` is dropped and
COUNTED (R761 has 0 of 2 763; one of the eight reference PDFs has 2 of 2). NO
new dependency: `pdfminer.six` already ships under `pdfplumber` in
`[extract]`, so `uv.lock` is untouched and `pypdf` stays out. Cost 119.22 s ->
183.31 s wall, peak RSS 3 252 -> 3 251 MiB, pages parsed 1 -> 1. **The default
did not move**, and the reach is why: **1 of 8** reference PDFs carries a
usable tree, and a bookmark tree is the publisher's CLAIM about its own
structure. On the folder where no PDF has one, `diff -r` is empty against both
the arm off and the pre-change tree. Report:
`docs/2026-09-10-k3-runde12-pdf-outlines.md`.
- **Three PDF READER flags, all off, and they sit BEFORE every segmentation
flag** -- an arm changes how the proposer cuts a text, these change what the
text says. `--pdf-headings font` infers a heading from typography (dominant

27
README.md
View file

@ -146,6 +146,32 @@ pre-2026-09-10 bundle byte for byte — measured with `diff -rq`, 0 differences,
not asserted. Each line below carries the number it was measured at, and
nothing beyond it.
An **eleventh** flag, `--pdf-outline`, is **off** by default, and it is the one
rule here that does not read the extracted text at all: it cuts a PDF at the
boundaries the file's own `/Outlines` bookmark tree declares. It is not
`--outline-run` under another name — that one is a text heuristic over numbered
lines in the extracted text, while this one opens a structure index the PDF
already carries and the build had never looked at.
It is a segmentation arm and not a reader option: the extracted text is byte
for byte the same either way, and a PDF that carries no bookmark tree builds
byte-identically with the flag on. Two consequences come free with it. The
concept title comes from the BOOKMARK, so it is not cut short where the page
wrapped the heading across two lines; and a page that lies before the first
bookmark destination is the contents listing rather than a second copy of the
body, so a contents entry and the section it lists stop landing as two concepts
under one id.
The measurement is one 701-page process code whose publisher also ships a
NISO-STS structure for it, so the fasit is the publisher's own. Under the
shipped default that document gives 1967 of 2761 boundaries, none of its 28
chapters, and 794 of 794 misses have their heading text present in the text the
build read. With the arm it gives 2759 of 2761 and 28 of 28. The flag stays off
because reach is the open question, not quality: **1 of the 8** reference PDFs
in this repository's own sample carries a usable tree, and a bookmark tree is
the publisher's *claim* about its own structure — a stale or wrongly pointing
one carries that error straight into the segmentation.
A **tenth** flag, `--bold-title`, is **off** by default. It is the rule for the
type whose container declares nothing: `rtf` has no heading style, so an
author's title is bold text, and the row was measured at 0 of 0 declared
@ -189,6 +215,7 @@ rank was measured in the same session and did NOT ship; see
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
| `--first-span-from-zero` (**on** by default since 2026-09-10; opt out with `--no-first-span-from-zero`) | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | Measured over the 39-document corpus, the default before this rule left **207 435 characters — 11.92 %** — in no segment at all: **163 804 above the first entry** (in **32 of the 32** documents that get a plan), 26 041 *between* entries and 17 590 after the last. This rule closes the first part entirely, 79 % of the whole, leaving **43 631 characters (2.51 %) over 8 of 32 documents** with two named mechanisms of their own. It adds no boundary and the K2 concept count is identical with and without it (**425 = 425**); on the 12-position reference it changes **not one cell**, and hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–** under both tie-breaks |
| `--close-span-gaps` (**on** by default since 2026-09-11; opt out with `--no-close-span-gaps`) | close a concept's span against the next SURVIVING concept, and the last against the end of the text. Three steps remove a candidate AFTER its neighbour's span was already closed against it — the orphan check, and `fold_units` clause 1 both between entries and on the last run — and the removed mark's text then belongs to no segment. Adds no boundary and removes none; only spans' ends move | It closes the whole remainder the rule above left: **43 631 characters, 2.51 % of the corpus over 8 of the 32 documents with a plan, to 0** — both the 26 041 between entries and the 17 590 after the last. Decomposed: orphan check **18 527** over 15 of 39 documents, clause 1 **7 514** between entries, clause 1 on the last run **all 17 590** of the tail (with `unit_fold=False` the corpus tail gap is 0). The entry count is identical (**429 = 429** on the corpus, **436 = 436** concepts on K2, **52 = 52** md on a five-document folder); on the 12-position reference it changes **not one cell** (11 of 12 under `|F|`[3]=12, 10 of 12 under `|F|`[3]=11), and hit@8 holds **5 of 6 at ranks 1,1,1,1,1,–** on the new bundle, the previous default and Arm B alike |
| `--pdf-outline` (**off**; opt out is the default, opt in with the flag) | cut a PDF at the boundaries its own `/Outlines` bookmark tree declares, instead of at the ones the text rules recover. A SEGMENTATION arm, not a reader option: the extracted text is byte for byte the same either way, and a PDF that carries no tree builds byte-identically with the flag on. The title comes from the BOOKMARK, so it is not cut short at the page's line break, and a page before the first bookmark destination is the contents listing rather than a second copy of the body | one 701-page process code whose publisher also ships a NISO-STS structure for it, so the fasit is the publisher's own: boundaries **1967 of 2761 → 2759 of 2761**, chapter level **0 of 28 → 28 of 28**, concept titles identical to the source title after normalisation **2761 of 2761**, false positives **163 of 2182 → 3 of 2762**, directories carrying two concept files **132 of 2050 → 2 of 2738**, contents-copy pairs **65 → 0**. Consumption on the same eight questions: fasit present in the bundle **4 of 7 → 7 of 7**, hit@1/8/50 **1/6 · 2/6 · 4/6 → 3/6 · 5/6 · 6/6**. Cost 119.22 s → 183.31 s wall, peak RSS 3252 → 3251 MiB, no new dependency and no second parse of the pages. **Off, and the reach is why:** **1 of the 8** reference PDFs carries a usable tree at all, and a bookmark tree is the publisher's CLAIM about its own structure — a stale or wrongly pointing one carries that error straight into the segmentation |
They compose, and the order above is the order they apply in. Measured on a
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per

317
docs/2026-09-10-k3-runde12-pdf-outlines.md Normal file
View file

@ -0,0 +1,317 @@
# K3 round 12: the PDF shipped a finished structure index and the build discarded it unopened
`okf build` reads a PDF's page text and recovers boundaries from the shape of
that text. It never opens the file's `/Outlines` bookmark tree. On a document
whose publisher wrote one, that is the difference between guessing the
structure and being told it.
This round adds `--pdf-outline`, **off by default**, which cuts a PDF at the
boundaries its own bookmark tree declares. It is a segmentation arm: the
extracted text is byte for byte the same either way.
Every number below carries its denominator. The two that matter most are that
the fasit is the **publisher's own** structure, and that it describes **one
document, one format, one publisher** (§ 7).
## 0. Three divergences from the numbers this round was handed
The order and the incoming FYI both rest on `vegnormal-okf`'s SK2 measurement.
Its headline reproduces here **exactly** -- 1 967 of 2 761, depth 1 `0 of 28`,
false positives 163 of 2 182, 794 of 794 misses present in the text -- both
against SK2's own bundle and against one rebuilt on this repository's HEAD
(§ 2). Three characterisations around it do not survive re-measurement.
**(a) "No okf concept directory is a bare number" is a property of that
BUNDLE, not of okf.** The claim is what makes 81 of 2 761 fasit rows
unreachable for SK2's pair key, and all 28 depth-1 rows sit in that group. It
is true of the arm-off bundle: **0 of its directories** is a bare integer.
It is false of okf: with the arm on, **57 of 2 738** directories are, because
`_split_outline_title` reads the same bare one-or-two-digit grammar Arm D
already carries. The rows were unreachable because the boundaries were never
opened, not because the id form forbids it.
**(b) The corrected instrument does NOT lift the arm-off number.** The
correction -- accept a hit when the concept's own normalised title equals the
fasit's, alongside the pair key -- was expected to show that "depth 1: 0 of 28"
was partly an instrument artifact. Measured on the arm-off bundle it changes
**1 967 to 1 967** and depth 1 stays **0 of 28 in both forms**. So on the
shipped default that row was a real segmentation hole measured with an
instrument that *could* have reached it, and not only an artifact. The
correction earns its keep on the arm-**on** bundle, where it moves 2 737 to
2 759 and depth 1 from 12 of 28 to 28 of 28 (§ 3).
**(c) The 145 truncated titles and the 65 contents duplicates reproduce as
described**, and both are closed as a side effect rather than as separate work
(§ 3). They are re-characterisations of SK2's own numbers, not new
measurements on new material.
## 1. Where the gap is
- 794 of 794 missed STS titles have their heading text **present** in the
extracted text; 0 of 794 are absent. The gap is segmentation, not
extraction.
- The file carries an `/Outlines` tree of **2 763 nodes**. All **2 763 of
2 763** resolve to a page index, and all **2 763 of 2 763** carry an `/XYZ`
destination with a non-null `top`. After `re.sub(r"\s+","",s).lower()`,
**2 761 of 2 761** STS titles match a node exactly; without the
normalisation, 1 of 2 761.
- **Three holes, not one**, and the tree closes all three: the boundary that is
never opened (794 of 2 761); the title cut short at the PDF's line break
(145 of the 163 "false positives" and 81 of the 794 "misses" are this -- a
bookmark title does not come from the page, so it is complete); and the table
of contents becoming a second set of concepts (65 directories carrying both a
contents copy and a body section under one id -- a bookmark's destination
page *is* the body).
## 2. Reproduced before anything was built on it
| row | SK2 (`de9564d`) | this HEAD, arm off |
|---|---|---|
| boundaries | 1 967 of 2 761 (71.2 %) | **1 967 of 2 761 (71.2 %)** |
| depth 1 … 7 | 0/28 · 59/118 · 410/500 · 913/1141 · 551/868 · 31/97 · 3/9 | **identical** |
| false positives | 163 of 2 182 | **163 of 2 182** |
| misses present in the text | 794 of 794 | **794 of 794** |
| directories with two concept files | 132 of 2 050 | **132 of 2 050** |
| bare-integer directories | — | **0** |
`git log de9564d..HEAD -- extract.py propose.py` names exactly one commit
(`732f84d`, the HTML extractor), so the PDF path was unchanged between the
bundle SK2 measured and this tree -- and the rebuild confirms it rather than
resting on it.
## 3. The arm, measured
Built with `okf build … --pdf-outline`, same input, same epoch stamp.
| row | arm off | arm on (SK2's instrument) | arm on (corrected) |
|---|---|---|---|
| **boundaries** | 1 967 of 2 761 | 2 737 of 2 761 (99.1 %) | **2 759 of 2 761 (99.9 %)** |
| depth 1 | 0 of 28 | 12 of 28 | **28 of 28** |
| depth 2 | 59 of 118 | 116 of 118 | **118 of 118** |
| depth 3 | 410 of 500 | 499 of 500 | **500 of 500** |
| depth 4 | 913 of 1 141 | 1 140 of 1 141 | **1 141 of 1 141** |
| depth 5 | 551 of 868 | 866 of 868 | **868 of 868** |
| depth 6 | 31 of 97 | 95 of 97 | **95 of 97** |
| depth 7 | 3 of 9 | 9 of 9 | **9 of 9** |
| **titles identical to the STS title after `NORM`** | — | — | **2 761 of 2 761** |
| **false positives** | 163 of 2 182 | 25 of 2 762 | **3 of 2 762** |
| **directories with two concept files** | 132 of 2 050 | **2 of 2 738** | (same) |
| concepts in the front matter (page < 68) | 72 of 2 182 | **2 of 2 762** | (same) |
| concepts | 2 182 | **2 762** | (same) |
| K3-9: ends in a digit | 16 of 99 | 99 of 99 | **99 of 99** |
| K3-9: opens with a numbering token | 1 967 of 2 739 | 2 737 of 2 739 | **2 737 of 2 739** |
| K3-9: unnumbered | 0 of 22 | 0 of 22 | **22 of 22** |
The gate was **2 700 of 2 761** with the corrected instrument; the result is
2 759, and the ceiling is 2 761.
**The distance from the ceiling, named row by row rather than left as a
remainder.** Two fasit rows are missed, and they are the *same two nodes* the
false-positive column counts from the other side: `36.7331 48V strømforsyning
PoE` and `36.7332 24V strømforsyning`. The publisher's bookmark for each omits
the numbering token the STS `<title>` glues on, so the concept lands with no
number directory and the pair key cannot reach it. Both sections exist as
concepts with the right body; only the address differs. The third false
positive is the tree's own **root node**, `R761 Prosesskoden` -- one candidate,
the document's own title.
**The two remaining directories with two concept files are not contents
duplicates.** They are `11` and `12`, and each holds two genuinely distinct
declared sections that share a number: `11 ARBEIDSSTIKNING, TEKNISK KONTROLL`
(page 183) beside `11 Oversikt over prosesser` (pages 99–181), and the same
shape at 12. The 65 contents-copy pairs are **0**. The collision that remains
is the publisher reusing a number in two chapters, which is a different
mechanism and is not closed by anything here.
**Depth 6 stays 95 of 97 in both instrument forms.** The publisher's own tree
and the publisher's own STS disagree about four nodes: the tree's levels 2..8
hold 28/118/500/1141/**872**/**93**/9 against the STS depths 1..7 at
28/118/500/1141/**868**/**97**/9. The mapping is level minus one on five rows
and off by four on two of them, so the level is read from the tree and reported
as declared rather than normalised (`OutlineMark.level`). Reporting a depth-1
row as `28 of 28` because a `+1` shift was *assumed* would be the same failure
`0 of 28` was.
### 3.1 The bridge, both routes, with denominators
A bookmark states a page and a y position; a candidate needs a line index.
**2 706 of 2 761 bookmarks share a destination page with another bookmark**, so
the page alone is never a cut point.
- **Y route (primary).** `page.extract_text_lines()` carries a `top` per line.
It requires the line splitting to be the one `page.extract_text()` produced,
which is an assumption and is therefore CHECKED per page: measured **701 of
701 pages identical**, and the check ships, so a page where it fails uses the
title route instead of guessing.
- **Title route (fallback).** The normalised bookmark title, searched in the
destination page's own lines: **2 762 of 2 763** resolved.
- **They disagree on 0 of 2 762.** Swept over tolerances 0, 1, 2, 3, 5, 8 and
12 pt, the agreement is **flat at 2 762 with 0 disagreements from 0 to 8 pt
and collapses to 924 at 12 pt** (the document's line spacing). Because 0 pt
is already exact, the rule ships with **no tolerance constant at all** -- the
first line at or below the destination.
- The first naive rule -- *nearest* line -- was wrong on **1 840 of 2 762**
nodes, one line early every time, because `/XYZ top` sits above the heading.
It is recorded because it produced a plausible-looking bundle.
**Provenance.** For **2 762 of 2 762** concepts, the first page of
`source_pages` equals the bookmark's own destination page. `source_units`
still indexes exactly the string `extract_text` returned: the arm moves
boundaries, never units. A PDF concept's address still names the page range
its span covers -- now the range the publisher's own bookmark opened.
### 3.2 Consumption
Fasit presence was validated against the bundle **before** any rank was read.
| | arm off | arm on |
|---|---|---|
| fasit present in the bundle | 4 of 7 | **7 of 7** |
| hit@1 · hit@8 · hit@50 (default `k`) | 1/6 · 2/6 · 4/6 (SK2) | **3/6 · 5/6 · 5/6** |
| hit@1 · hit@8 · hit@50 (`k` = 50) | — | **3/6 · 5/6 · 6/6** |
| payload slots taken by the front matter | 0 of 8 (SK2) | **0 of 8, all eight questions** |
S3, S4 and the known-positive existed in no bundle before, because the section
that answers them never became a boundary.
**The known-positive is not delivered at default `k`, and that is stated rather
than rounded.** `1 Bruksområder for Prosesskoden` is a real 1 827-byte concept
now, reading as the answer to the question asked of it, and it ranks **13** --
delivered at `k` = 50, not at `k` = 8. SK2 reported it as `null` because the
boundary did not exist. The segmentation half of that row is closed; the
ranking half is not, and no part of this round touches ranking.
`S5` (`84.3 Armering`) is the one row still missing at `k` = 8: rank 11 at
`k` = 50, with `Armering av betonghvelv` at rank 1. `Armering` occurs nine
times in this structure, and that collision is in the ranking, not the bridge.
### 3.3 Cost
| | arm off | arm on |
|---|---|---|
| wall (`/usr/bin/time -l real`) | 119.22 s | **183.31 s** (+53.8 %) |
| peak RSS | 3 410 038 784 B = **3 252 MiB** | 3 409 002 496 B = **3 251 MiB** |
| times the PDF's pages are parsed | 1 | 1 |
| times the file is opened | 1 | 2 |
`/usr/bin/time -l` on macOS reports maximum resident set size in **bytes**, not
KB; both figures above are the byte value converted. Neither is a 2×
increase. SK2 measured the arm-off build at 180.50 s on their machine against
119.22 s here, so the honest comparison is the pair measured on one machine.
The extra open reads the bookmark tree and the line geometry; the page text
comes from the memoised extraction pass and is not re-rendered.
**No new dependency.** `pdfminer.six` 20260107 already ships under
`pdfplumber` in the existing `[extract]` extra, and exposes
`PDFDocument.get_outlines()`. `uv.lock` and `[extract]` are untouched;
`pypdf` is not introduced.
## 4. Nothing else moved, and it is measured rather than argued
- **`~/okf-test/dokumenter`, three ways: identical.** A build from a frozen
`git archive` export of `8e19179`, a build on this tree with the arm off, and
a build on this tree with the arm **on** -- `diff -r` is empty between all
three. 52 md / 26 concepts, 0 of 5 rejected. Both PDFs there carry no
`/Outlines`, which is what makes the third comparison a real control.
- `okf project` is still byte-equal to `okf build` (`diff -r` empty), and the
new flag's argparse default and Python signature default agree -- the test
that compares every same-typed parameter covers it.
- `okf skill` + `okf consume` + `okf check --skill … --payload …`:
**conformant, 15 rules over 8 excerpts and 18 withheld entries, 0 findings.**
- The pinned K2 bundle's concept count, per-row hit@8 ranks and all three
known-negatives are unchanged: the arm is off by default and the arm-off code
path is byte-identical.
- **The 12-position reference is untouched by the shipped default** and stays
`pdf` 7 of 8, `docx` 3 of 3, in both readings of the `xlsx` fasit
(`|F|` = 11 → 10 of 12, `|F|` = 12 → 11 of 12). This is byte-identity, not a
re-score.
**Reach, and the number that decides whether the default should move.** Of the
eight `pdf` positions on the reference, **1 of 8** carries a usable bookmark
tree (position 9, 11 nodes). One more (position 0) carries a tree whose two
nodes both fail to resolve -- the drop-and-count branch firing on a real
document rather than only on a fixture. The other six carry none.
With the arm on, position 9 goes from **4 concepts to 11**, and the eleven are
the document's own chapter names (`Innledning`, `Organisering av prosjektet`,
`Risikoforhold og spesifikke tiltak`, `Fremdriftsplan`, `Vedlegg 1: …`) --
seven chapters the extracted text never showed as headings at all. The
operator's cross for that position says a unit is a top-level chapter and the
free text says "ville jeg hatt hvert kapittel". Whether 11 is nearer that than
4 depends on an `|F|` the sheet does not define, because the sheet's own
"document's disposition" section could only list the four the text showed.
**That is an operator question and this round does not answer it.**
## 5. What did NOT move
No HTML change, no OCR, no consumption change, no new fasit, no new corpus, no
`.xml`/NISO-STS extractor, no change to Arm D (`--outline-run`/`--outline-gate`
are a text heuristic over numbered lines and are a different mechanism
entirely), no re-rating of the reference positions, no `--bold-title` move, no
version bump, no tag, no push. `uv.lock` and `[extract]` untouched. The
default did not move.
## 6. The tests, and what each one is for
`tests/test_pdf_outline.py`, over two hand-laid fixtures built by
`tests/fixtures/make_fixtures.py` -- no library's output, objects laid out by
hand, xref offsets computed from the emitted bytes (`outlined-krav.pdf`
2 073 B, `outline-broken-dest.pdf` 1 014 B).
- **Three levels, not two.** A two-level tree cannot tell "the level the node
declares" from "one below the root".
- **The line, not the page.** The fixture's last page carries four lines and
its second bookmark points at the third, so a bridge that resolved to the
page and stopped would fail visibly instead of looking like it worked.
- **No `/Outlines` is not an error.** `PDFNoOutlines` yields no marks, no
count, and identical behaviour.
- **An unresolvable `/Dest` is dropped and counted**, never fabricated and
never a refusal.
- **An empty mark list leaves every other rule untouched** -- "this file has no
index" and "propose nothing" must not collapse into one answer.
- **A declared section with no prose of its own survives.** The orphan check
asks whether anything stands under a candidate's first line: the right
question for a heuristic's guess, the wrong one for a publisher's
declaration. Measured: **683 of 2 762** marks are container sections, and
applying the check scores **2 079** instead of 2 759.
## 7. Honesty limits
- **The instrument was run in two forms and the two answer different
questions.** SK2's pair key `(numbering token, normalised residual title)`
cannot reach **81 of 2 761** fasit rows -- 59 with a dotless token, 22
unnumbered -- and all 28 depth-1 rows are among them. Every percentage above
says which form it rests on, and the gate was set at **2 700**, not at the
ceiling 2 761, precisely because of those 81. The corrected form lives in a
scratch copy of SK2's script; `vegnormal-okf` was read and never written, and
the correction is offered to them, not applied on their behalf.
- **The fasit is the publisher's own STS structure.** "The right boundary" is
therefore defined by the publisher, not by what is useful to a reader. A
reader might well want 2 761 sections merged into far fewer.
- **The denominator 2 761 is CHOSEN.** The document has 7 715 `<sec>`; 4 954 of
them carry a letter-point label (`a)`, `c)`) inside a process description and
0 of those 4 954 is a process number. Measuring against 7 715 would answer a
different question.
- **Matching is exact after normalisation.** A boundary set one word away from
the source title counts as a miss. That is what makes 2 759 credible and also
what makes it a floor rather than a satisfaction score.
- **One document, one format, one publisher.** R761 is a guideline with 2 739
of 2 761 titles numbered in a strict hierarchy. A running prose document
would measure something else, and this round has no such document with a
fasit.
- **A bookmark tree is the publisher's CLAIM about its own structure**, not a
measurement of the document. A PDF with a stale, incomplete or wrongly
pointing tree carries that error straight into the segmentation, and R761
says nothing about how common that is. One of the eight reference PDFs here
already carries a tree whose destinations do not resolve.
- **SK2's own figures** (1 967, 163, 794, 180.50 s, 3 253 MiB, 1/6 · 2/6 · 4/6)
are `vegnormal-okf`'s documentation and stand as premises wherever they were
not re-measured here. The rows in § 2 were re-measured.
- **The 12-position reference is n = 12**, stratified 8 `pdf`, 3 `docx`,
1 `xlsx`, so every per-type fraction is a fraction of 8, 3 or 1. Both
readings of the `xlsx` fasit are reported, never one.
- **"Delivered" is not "answered."** hit@k asks whether the gold concept
reached the payload, and nothing here evaluates an answer.
- Position 0's `|F|`, the `xlsx` fasit's `|F|`, the `pptx` fasit, re-rating the
reference positions and routing between bundles on `--id` remain blocked on
an **operator decision**, not on code. This round moved none of them.

40
src/llm_ingestion_okf/cli.py
View file

@ -234,6 +234,13 @@ DEFAULT_OCR = False
#: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`.
DEFAULT_PDF_HEADINGS_RESERVE = False
#: The PDF bookmark arm. OFF, and this round did not move it: the arm was
#: measured on ONE document, and a default that changes every PDF bundle a
#: consumer holds is an operator's call with the numbers in front of them.
#: It is BYTE-IDENTICAL on a PDF that carries no `/Outlines`, which is the
#: common case and the reason the flag is safe to hand out before that call.
DEFAULT_PDF_OUTLINE = False
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -262,6 +269,7 @@ def _propose_plans(
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -300,6 +308,7 @@ def _propose_plans(
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -337,6 +346,7 @@ def build(
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
pdf_outline: bool = DEFAULT_PDF_OUTLINE,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -409,6 +419,7 @@ def build(
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
pdf_outline=pdf_outline,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -784,6 +795,34 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"for two at once"
),
)
build_parser.add_argument(
"--pdf-outline",
action="store_true",
default=DEFAULT_PDF_OUTLINE,
help=(
"OFF. Cut a PDF at the boundaries its own /Outlines bookmark tree "
"declares, instead of at the ones the text rules recover. It is a "
"SEGMENTATION arm and not a reader option: the extracted text is "
"byte for byte the same either way, and a PDF that carries no "
"bookmark tree builds byte-identically with the flag on. Measured "
"on one 701-page process code whose publisher also ships a NISO-STS "
"structure for it: the text rules recover 1967 of 2761 titled "
"sections and 0 of its 28 chapters, while its bookmark tree matches "
"2761 of 2761 exactly. The title comes from the BOOKMARK, so it is "
"not cut short at the page's line break, and a page before the "
"first bookmark destination is the table of contents rather than a "
"second copy of the body. ONE document, ONE format, ONE publisher, "
"and a bookmark tree is the publisher's CLAIM about its own "
"structure -- a stale or wrong tree carries its error straight into "
"the segmentation"
),
)
build_parser.add_argument(
"--no-pdf-outline",
action="store_false",
dest="pdf_outline",
help="The arm's explicit opt-out",
)
build_parser.add_argument(
"--ocr",
action="store_true",
@ -836,6 +875,7 @@ def main(argv: list[str] | None = None) -> int:
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,
pdf_outline=args.pdf_outline,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

207
src/llm_ingestion_okf/extract.py
View file

@ -106,7 +106,18 @@ _PANDOC_FORMATS: dict[str, str] = {
# the honesty limit that travels with it: the 828 files are ONE product in ONE
# format from ONE publisher, and the file boundaries and `<h1>`s are a
# generator's cut of that document, not 828 documents anyone wrote.
#
# `.pdf` JOINED THE TABLE 2026-09-10, as `measured`, and it enters on the
# strongest evidence of any row here: eight real corpus PDFs with a fasit the
# operator hand-counted document by document, plus a 701-page process code
# whose PUBLISHER also ships a NISO-STS structure for it -- 2 761 titled
# sections, written for their own purposes and long before any lookup of ours.
# The honesty limit that travels with it: those 2 761 rows are ONE product in
# ONE format from ONE publisher, its structure is a strict numbered hierarchy
# on 2 739 of 2 761 titles, and a running prose document would measure
# something else entirely.
_EVIDENCE: dict[str, str] = {
".pdf": "measured",
".docx": "measured",
".xlsx": "measured",
".pptx": "constructed",
@ -638,6 +649,202 @@ def _pdf_pages(
return tuple((number, page) for number, page in enumerate(pages, start=1) if page)
@dataclass(frozen=True)
class OutlineMark:
"""One `/Outlines` node, placed on a LINE of the string `extract_text` returns.
`level` is what the TREE declares, not a distance normalised against
anything: a document whose outline carries its own root node puts its
chapters at level 2, and rewriting that here would state a structure the
publisher did not. Measured on a 701-page process code -- the tree's levels
2..8 hold 28/118/500/1141/872/93/9 nodes against the publisher's own
NISO-STS depths 1..7 at 28/118/500/1141/868/97/9, so the mapping is level
minus one on five rows and the publisher disagrees with the publisher on
four nodes. That disagreement is data, and it survives only if the level is
reported rather than fixed up.
"""
line: int
level: int
title: str
@dataclass(frozen=True)
class PdfOutline:
"""The bookmark tree, bridged onto lines -- with what did not bridge counted.
`unresolved` is not decoration. A `/Dest` that names an object which is not
a page, or a page that produced no text, has to be DROPPED: fabricating a
boundary from it would put a heading somewhere the document never had one,
and raising would refuse a file over a defect in one of its bookmarks.
Dropping silently is the third option this library refuses everywhere else,
so the count is part of the return value.
"""
marks: tuple[OutlineMark, ...]
unresolved: int
def _outline_page_and_top(doc: object, dest: object, action: object) -> tuple[object, float | None]:
"""`(page reference, /XYZ top)` from a bookmark's destination, or `(None, None)`.
Four shapes reach here and all four are in the wild: an explicit array, a
NAMED destination resolved through the document's name tree, a `GoTo`
action carrying either, and an indirect reference to any of them.
"""
target = dest
if target is None and action is not None:
resolved = action.resolve() if hasattr(action, "resolve") else action
if isinstance(resolved, dict):
target = resolved.get("D")
if isinstance(target, (bytes, str)) or hasattr(target, "name"):
name = target.name if hasattr(target, "name") else target
try:
target = doc.get_dest(name) # type: ignore[attr-defined]
except Exception:
return (None, None)
if hasattr(target, "resolve"):
try:
target = target.resolve()
except Exception:
return (None, None)
if isinstance(target, dict):
target = target.get("D")
if not isinstance(target, list) or not target:
return (None, None)
top: float | None = None
if len(target) > 3 and getattr(target[1], "name", None) == "XYZ":
candidate = target[3]
if isinstance(candidate, (int, float)):
top = float(candidate)
return (target[0], top)
def pdf_outline(
name: str, data: bytes, *, pdf_headings: bool = False, ocr: bool = False
) -> PdfOutline:
"""`pdf`: the file's own `/Outlines` tree, as marks on the extracted text.
THE BRIDGE IS THE WHOLE PROBLEM, and both routes are measured rather than
argued. A bookmark states a PAGE and a y position; a candidate needs a LINE
index. On the 701-page document this was built against, 2 706 of 2 761
bookmarks share a destination page with another bookmark, so the page alone
is never a cut point.
Y ROUTE (primary). `page.extract_text_lines()` carries a `top` per line,
and the mark takes the FIRST line at or below the destination. It needs
the line splitting to be the one `page.extract_text()` produced -- an
assumption, so it is CHECKED per page and the route is used only where
the two strings are identical. Measured: 701 of 701 pages, and the
resulting index agrees with the title route on 2 762 of 2 762 nodes,
flat from a 0 pt tolerance to 8 pt and collapsing at 12 (the line
spacing). It therefore ships with NO tolerance constant at all.
TITLE ROUTE (fallback). The bookmark's title, normalised, searched in the
destination page's own lines. It resolved 2 762 of 2 763 on that
document, and its weakness is real: a title like `Armering` occurs nine
times in that structure, so it is scoped to the destination page and is
never asked a question the y route already answered.
`pdf_headings` and `ocr` are passed through so the line indices address the
SAME rendering the caller extracted. They are not options of this arm: a
plan indexes one exact string, and marks computed against another one point
at the right words in the wrong places.
"""
if Path(name).suffix.lower() != ".pdf":
return PdfOutline((), 0)
try:
import pdfplumber
except ImportError as exc:
raise _extra_missing(".pdf") from exc
from pdfminer.pdfdocument import PDFNoOutlines
from pdfminer.pdfpage import PDFPage
rendered = _pdf_pages(data, pdf_headings, ocr)
starts: dict[int, int] = {}
page_lines: dict[int, list[str]] = {}
offset = 0
for number, page_text in rendered:
starts[number] = offset
page_lines[number] = page_text.split("\n")
offset += len(page_lines[number]) + 1
unresolved = 0
placed: dict[int, OutlineMark] = {}
with pdfplumber.open(io.BytesIO(data)) as pdf:
try:
nodes = list(pdf.doc.get_outlines())
except PDFNoOutlines:
# NOT an error, and not zero concepts either: this file simply
# carries no index, which is the common case and the one the
# byte-identical guarantee below rests on.
return PdfOutline((), 0)
except Exception as exc:
raise ExtractionError(
f"the PDF parser failed reading /Outlines: {exc}",
code="extractor_pdf_error",
) from exc
numbers = {
page.pageid: index + 1 for index, page in enumerate(PDFPage.create_pages(pdf.doc))
}
wanted: dict[int, list[tuple[int, str, float | None]]] = {}
for level, title, dest, action, _ in nodes:
reference, top = _outline_page_and_top(pdf.doc, dest, action)
page_number = numbers.get(getattr(reference, "objid", None))
if page_number is None or page_number not in starts:
unresolved += 1
continue
wanted.setdefault(page_number, []).append((int(level), str(title), top))
# Geometry is read only for the pages that carry a bookmark, because
# `extract_text_lines` costs a second render of every page it is asked
# about -- 78 s over 701 pages, and nothing at all over the pages no
# bookmark points at.
for page in pdf.pages:
number = page.page_number
group = wanted.get(number)
if not group:
continue
lines = page_lines[number]
tops: list[float] | None = None
geometry = page.extract_text_lines()
if [str(entry["text"]) for entry in geometry] == lines:
tops = [float(entry["top"]) for entry in geometry]
height = float(page.height)
for level, title, top in group:
index: int | None = None
if tops is not None and top is not None:
want = height - top
index = next(
(position for position, value in enumerate(tops) if value >= want),
len(tops) - 1,
)
else:
target = _normalise_outline(title)
joined = ""
bounds: list[int] = []
for rendered_line in lines:
bounds.append(len(joined))
joined += _normalise_outline(rendered_line)
found = joined.find(target)
if found >= 0:
index = max(position for position, at in enumerate(bounds) if at <= found)
if index is None:
unresolved += 1
continue
at = starts[number] + index
# FIRST in tree order wins a shared line. Two marks on one line
# would give the second an empty span, and the orphan check
# deletes an empty span silently -- the same trap the proposer
# documents for a second-pass candidate list.
placed.setdefault(at, OutlineMark(line=at, level=level, title=title))
return PdfOutline(tuple(placed[at] for at in sorted(placed)), unresolved)
def _normalise_outline(value: str) -> str:
"""Whitespace out, case folded -- the form the title route compares on."""
return re.sub(r"\s+", "", value).lower()
def _extract_pdf(data: bytes, *, headings: bool = False, ocr: bool = False) -> str:
"""`pdf`: page text via `pdfplumber`, in page order, pages separated by a
blank line.

161
src/llm_ingestion_okf/propose.py
View file

@ -56,13 +56,14 @@ import json
import re
import sys
import unicodedata
from collections.abc import Iterable
from collections.abc import Iterable, Sequence
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any
from .errors import IngestError
from .extract import extract_text, strip_converter_attribute
from .extract import OutlineMark, extract_text, strip_converter_attribute
from .extract import pdf_outline as extract_pdf_outline
from .materialize import reduce_to_id_grammar
from .segmentation import observed_extractor_version
@ -135,6 +136,12 @@ RULE_SHEET_SECTION = "rule:sheet-section"
#: readable documents, that gate is what takes the false-positive count to 0
#: of the 31 that declare.
RULE_BOLD_TITLE = "rule:bold-title"
#: The PDF's own `/Outlines` tree. NOT `RULE_OUTLINE`: that one is Arm D, a
#: TEXT heuristic over numbered lines in the extracted text, and this one opens
#: a structure index the file already carries. A reader who cannot tell the two
#: apart in an artifact cannot tell a recovered heading from a declared one.
RULE_PDF_OUTLINE = "rule:pdf-outline"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -144,6 +151,7 @@ RULE_NAMES = (
RULE_TABLE_GRID,
RULE_SHEET_SECTION,
RULE_BOLD_TITLE,
RULE_PDF_OUTLINE,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -597,6 +605,34 @@ def _sheet_section_rows(lines: list[str]) -> dict[int, tuple[str, str]]:
return sections
#: Rules the orphan check is not asked about. D3 because a sheet row carries
#: its content in its own cells, and the bookmark arm because the check judges
#: whether a GUESS was a heading -- a question a publisher's own tree has
#: already answered, and one that deletes every container section if asked.
_ORPHAN_EXEMPT = (RULE_SHEET_SECTION, RULE_PDF_OUTLINE)
def _split_outline_title(title: str) -> tuple[str | None, str]:
"""`("14.121", "Langsg\u00e5ende sikring T1")` -- the number becomes the directory.
Two grammars, both already in this module and neither invented here:
`_NUMBERED` for a dotted section number, then `_OUTLINE`'s bare one- or
two-digit form. The second matters because a document's TOP level is where
the dot has not appeared yet -- on the corpus this arm was measured
against, 59 of 2 761 titled sections carry a dotless token and all 28 of
its chapter-level sections are among them. Without it the whole top level
would land with no section number at all, in a bundle whose every other
level has one.
"""
dotted = _NUMBERED.match(title)
if dotted is not None:
return (dotted.group("number"), dotted.group("title"))
bare = _OUTLINE.match(title)
if bare is not None:
return (bare.group("number"), bare.group("title"))
return (None, title.strip())
def find_candidates(
text: str,
*,
@ -611,6 +647,7 @@ def find_candidates(
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
outline_marks: Sequence[OutlineMark] | None = None,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -682,6 +719,22 @@ def find_candidates(
before spans are closed, so the text they opened is carried by the mark
above rather than lost.
`outline_marks` is the PDF bookmark arm, and it is the only input here that
REPLACES the rules rather than gating one of them. A non-empty list is the
publisher's own declaration of the document's structure, so nothing below
votes against it: the text heuristics, the two gates and Arm F's fold are
all skipped, and the orphan check is not applied to its marks. An EMPTY
list means "this file carries no index" and leaves every rule untouched --
the two are different answers and must not collapse into one.
THE ORPHAN EXEMPTION IS THE ONE JUDGEMENT CALL HERE, and it is the same
shape as D3's. The check asks whether anything stands UNDER a candidate's
first line, which is the right question for a heading a heuristic GUESSED
and the wrong one for a section a publisher DECLARED: a chapter followed
immediately by its first subsection is a container, not a false positive.
Measured on a 701-page process code: 683 of 2 762 marks are containers, and
applying the check scores 2 079 of 2 761 boundaries instead of 2 762.
`sheet_section_rows` is D3's gate and it is OFF at False, where the scan is
not run at all. On, a RUN of numbered rows inside an open table block cuts
it: each such row opens a candidate that reaches the next section row, or
@ -717,6 +770,41 @@ def find_candidates(
# ordering rather than a property of this corpus.
admitted = {index: title for index, _, title in runs[-1]}
# The bookmark arm, and it is computed here for the same reason `admitted`
# is: the marks must be known and SORTED before the loop. They arrive
# already deduplicated by line, so `marked` stays ordered by construction
# and no span can close before it opens.
if outline_marks:
declared: list[tuple[int, Candidate]] = [
(
mark.line,
Candidate(
title=title,
level=mark.level,
number=number,
rule=RULE_PDF_OUTLINE,
start=offsets[mark.line],
end=end_of_text,
),
)
for mark in outline_marks
if mark.line < len(offsets)
for number, title in (_split_outline_title(mark.title),)
]
return _close_candidates(
text,
declared,
offsets,
end_of_text,
joined=set(),
absorbed=set(),
contents_run=set(),
unit_fold=False,
contents_name=False,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
)
# D3's input, and the same whole-text reasoning as `admitted` above: a run
# is a property of the line list, not of a line.
sections = _sheet_section_rows(lines) if sheet_section_rows else {}
@ -887,6 +975,49 @@ def find_candidates(
if outline_gate:
marked, joined = _gate_outline(marked, joined, end_of_text, len(text))
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = (
_contents_run_positions(marked, contents_name=contents_name) if unit_fold else set()
)
return _close_candidates(
text,
marked,
offsets,
end_of_text,
joined=joined,
absorbed=absorbed,
contents_run=contents_run,
unit_fold=unit_fold,
contents_name=contents_name,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
)
def _close_candidates(
text: str,
marked: list[tuple[int, Candidate]],
offsets: list[int],
end_of_text: int,
*,
joined: set[int],
absorbed: set[int],
contents_run: set[int],
unit_fold: bool,
contents_name: bool,
first_span_from_zero: bool,
close_span_gaps: bool,
) -> list[Candidate]:
"""`marked` -> the candidate list: orphan check, fold, then span closing.
Carved out of `find_candidates` when the bookmark arm arrived, unchanged in
behaviour: the arm produces its `marked` from a structure index instead of
from the line grammar, and every step from here down is the same question
for both. Duplicating it would be two orphan checks to keep in agreement.
"""
candidates: list[Candidate] = []
# The name an orphaned heading leaves behind, and the ONE candidate allowed
# to pick it up.
@ -909,13 +1040,6 @@ def find_candidates(
# empty, so the orphan check below stops firing on it by itself and no
# branch is needed there. Computed only when the caller asked, so every
# other arm's `marked` -> `candidates` mapping is untouched code.
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = (
_contents_run_positions(marked, contents_name=contents_name) if unit_fold else set()
)
for position_in_list, (_, candidate) in enumerate(marked):
if position_in_list in absorbed:
continue
@ -936,7 +1060,7 @@ def find_candidates(
# the rule could not fire at all. It is the same shape as a contents
# list without dot leaders, and it is the reason that one needed
# `Candidate.contents`.
orphan = candidate.rule != RULE_SHEET_SECTION and (
orphan = candidate.rule not in _ORPHAN_EXEMPT and (
not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip()
)
if orphan:
@ -1329,6 +1453,7 @@ def build_plan(
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
outline_marks: Sequence[OutlineMark] | None = None,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1347,6 +1472,7 @@ def build_plan(
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
outline_marks=outline_marks,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -1428,6 +1554,7 @@ def run(
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
pdf_outline: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -1472,6 +1599,7 @@ def run(
# against another is refused by `assert_plan_applies`, which is the
# right outcome and a confusing one to debug.
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
reading_fonts = pdf_headings
# The reserve, and the reason it re-extracts rather than post-processes:
# the font reader works on the PDF's glyph geometry, which the joined
# text no longer carries. Skipped outright when the font reader is
@ -1480,6 +1608,18 @@ def run(
if pdf_headings_reserve and not pdf_headings:
if heading_reserve_applies(text, outline_run=outline_run):
text = extract_text(source.name, source_bytes, pdf_headings=True, ocr=ocr)
reading_fonts = True
# LAST, and against the text that is final: a plan indexes one exact
# string, so marks bridged onto the pre-reserve rendering would name
# the right words at the wrong offsets. `reading_fonts` is what the
# reserve may have changed, and the marks follow it.
marks = (
extract_pdf_outline(
source.name, source_bytes, pdf_headings=reading_fonts, ocr=ocr
).marks
if pdf_outline
else ()
)
except IngestError as exc:
raise ProposerError(f"cannot extract text from {source.name}: {exc}") from exc
@ -1502,6 +1642,7 @@ def run(
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
outline_marks=marks,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan

174
tests/fixtures/make_fixtures.py vendored
View file

@ -82,6 +82,172 @@ NUMBERED_FONT_CONTENT = (
)
# THREE pages and a THREE-LEVEL `/Outlines` tree, which is the shape the
# bookmark arm has to be proved against. A two-level tree cannot tell "the
# level the node declares" apart from "one below the root", and a
# one-page fixture cannot tell a page-local line offset from a document-wide
# one -- the arm's whole risk is the bridge from (page, y) to a line index.
#
# The third page carries FOUR lines and its second bookmark points at the
# THIRD of them, so a bridge that resolved to the page and stopped would put
# the mark two lines early and still look like it worked.
OUTLINED_CONTENTS = (
b"BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET\n"
b"BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET\n",
b"BT /F1 12 Tf 20 170 Td (1.1 Omfang) Tj ET\n"
b"BT /F1 12 Tf 20 150 Td (Omfanget dekker hele arbeidet.) Tj ET\n",
b"BT /F1 12 Tf 20 170 Td (1.2 Krav) Tj ET\n"
b"BT /F1 12 Tf 20 150 Td (Kravet gjelder alle klasser.) Tj ET\n"
b"BT /F1 12 Tf 20 130 Td (1.2.1 Materialer) Tj ET\n"
b"BT /F1 12 Tf 20 110 Td (Materialene skal vaere godkjente.) Tj ET\n",
)
#: `(title, level, page index, /XYZ top)`. The `top` values are the PDF's own
#: bottom-up user space: 185 sits above the first line of a page and 150 above
#: its third, which is what makes the third-line mark a statement rather than
#: a coincidence.
OUTLINED_TREE = (
("1 Grunnlag", 1, 0, 185),
("1.1 Omfang", 2, 1, 185),
("1.2 Krav", 2, 2, 185),
("1.2.1 Materialer", 3, 2, 150),
)
#: One resolvable bookmark and one whose `/Dest` names an object that is not a
#: page. A PDF in the wild carries these; R761 carries none of them, so
#: without this fixture the "drop it, count it, do not fabricate a boundary"
#: branch would ship having never run.
BROKEN_DEST_CONTENT = (
b"BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET\n"
b"BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET\n",
)
def build_outlined_pdf(
contents: tuple[bytes, ...],
tree: tuple[tuple[str, int, int, int], ...],
*,
broken_dest: bool = False,
) -> bytes:
"""`build_paged_pdf` plus a hand-laid `/Outlines` tree in the catalog.
The tree is written from the flat `(title, level, page, top)` list rather
than from a nested literal, because `/First`, `/Last`, `/Next`, `/Prev` and
`/Parent` all have to agree with each other and with the level column --
a hand-written nest gets one of them wrong silently and the reader then
reports a level the document never declared.
`broken_dest` appends one extra item whose `/Dest` names the FONT object
instead of a page. It is a valid indirect reference to a real object that
is not a page, which is the failure a reader has to survive.
"""
count = len(contents)
page_numbers = [3 + 2 * index for index in range(count)]
font_number = 3 + 2 * count
outlines_number = font_number + 1
items = list(tree) + ([("Uoppl\u00f8selig", 1, -1, 185)] if broken_dest else [])
item_numbers = [outlines_number + 1 + index for index in range(len(items))]
kids = b" ".join(str(number).encode() + b" 0 R" for number in page_numbers)
objects = [
b"<< /Type /Catalog /Pages 2 0 R /Outlines " + str(outlines_number).encode() + b" 0 R >>",
b"<< /Type /Pages /Kids [" + kids + b"] /Count " + str(count).encode() + b" >>",
]
for index, content in enumerate(contents):
stream_number = page_numbers[index] + 1
objects.append(
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents "
+ str(stream_number).encode()
+ b" 0 R /Resources << /Font << /F1 "
+ str(font_number).encode()
+ b" 0 R >> >> >>"
)
objects.append(
b"<< /Length " + str(len(content)).encode() + b" >>\nstream\n" + content + b"endstream"
)
objects.append(
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"
)
# The parent of an item is the last item seen at the level above it; its
# previous sibling is the last item seen at its OWN level under that same
# parent. Both are resolved in one forward pass so the links cannot drift.
parents: list[int | None] = []
last_at_level: dict[int, int] = {}
siblings: list[int | None] = []
children: dict[int, list[int]] = {}
roots: list[int] = []
for position, (_, level, _, _) in enumerate(items):
parent = last_at_level.get(level - 1) if level > 1 else None
parents.append(parent)
previous = None
group = children.setdefault(parent, []) if parent is not None else roots
if group:
previous = group[-1]
siblings.append(previous)
group.append(position)
last_at_level[level] = position
for deeper in [key for key in last_at_level if key > level]:
del last_at_level[deeper]
def _ref(position: int | None) -> bytes:
return b"" if position is None else str(item_numbers[position]).encode() + b" 0 R"
root_body = b"<< /Type /Outlines /Count " + str(len(items)).encode() + b" >>"
if roots:
root_body = (
b"<< /Type /Outlines /First "
+ _ref(roots[0])
+ b" /Last "
+ _ref(roots[-1])
+ b" /Count "
+ str(len(items)).encode()
+ b" >>"
)
objects.append(root_body)
for position, (title, _, page, top) in enumerate(items):
parent = parents[position]
parent_ref = _ref(parent) if parent is not None else str(outlines_number).encode() + b" 0 R"
group = children.get(parent, []) if parent is not None else roots
index_in_group = group.index(position)
body = b"<< /Title (" + _pdf_text(title) + b") /Parent " + parent_ref
if index_in_group > 0:
body += b" /Prev " + _ref(group[index_in_group - 1])
if index_in_group + 1 < len(group):
body += b" /Next " + _ref(group[index_in_group + 1])
own = children.get(position, [])
if own:
body += b" /First " + _ref(own[0]) + b" /Last " + _ref(own[-1])
body += b" /Count " + str(len(own)).encode()
target = str(font_number).encode() if page < 0 else str(page_numbers[page]).encode()
body += b" /Dest [" + target + b" 0 R /XYZ 20 " + str(top).encode() + b" 0] >>"
objects.append(body)
out = bytearray(b"%PDF-1.4\n")
offsets = []
for number, item in enumerate(objects, start=1):
offsets.append(len(out))
out += str(number).encode() + b" 0 obj\n" + item + b"\nendobj\n"
xref_at = len(out)
size = str(len(objects) + 1).encode()
out += b"xref\n0 " + size + b"\n0000000000 65535 f \n"
for offset in offsets:
out += ("%010d 00000 n \n" % offset).encode()
out += b"trailer\n<< /Size " + size + b" /Root 1 0 R >>\n"
out += b"startxref\n" + str(xref_at).encode() + b"\n%%EOF\n"
return bytes(out)
def _pdf_text(value: str) -> bytes:
"""A PDF literal string in WinAnsi, with the three delimiters escaped."""
raw = value.encode("cp1252")
for character in (b"\\", b"(", b")"):
raw = raw.replace(character, b"\\" + character)
return raw
def build_two_font_pdf(content: bytes) -> bytes:
"""A one-page PDF whose resources declare BOTH a regular and a bold font.
@ -439,6 +605,14 @@ if __name__ == "__main__":
(HERE / "numbered-font-krav.pdf").write_bytes(build_two_font_pdf(NUMBERED_FONT_CONTENT))
print("wrote numbered-font-krav.pdf")
(HERE / "outlined-krav.pdf").write_bytes(build_outlined_pdf(OUTLINED_CONTENTS, OUTLINED_TREE))
print("wrote outlined-krav.pdf")
(HERE / "outline-broken-dest.pdf").write_bytes(
build_outlined_pdf(BROKEN_DEST_CONTENT, (("1 Grunnlag", 1, 0, 185),), broken_dest=True)
)
print("wrote outline-broken-dest.pdf")
for name, parts in (
("two-line-krav.docx", _DOCX_PARTS),
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),

45
tests/fixtures/outline-broken-dest.pdf vendored Normal file
View file

@ -0,0 +1,45 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R /Outlines 6 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>
endobj
4 0 obj
<< /Length 105 >>
stream
BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET
BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET
endstream
endobj
5 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
endobj
6 0 obj
<< /Type /Outlines /First 7 0 R /Last 8 0 R /Count 2 >>
endobj
7 0 obj
<< /Title (1 Grunnlag) /Parent 6 0 R /Next 8 0 R /Dest [3 0 R /XYZ 20 185 0] >>
endobj
8 0 obj
<< /Title (Uoppløselig) /Parent 6 0 R /Prev 7 0 R /Dest [5 0 R /XYZ 20 185 0] >>
endobj
xref
0 9
0000000000 65535 f
0000000009 00000 n
0000000074 00000 n
0000000131 00000 n
0000000257 00000 n
0000000412 00000 n
0000000509 00000 n
0000000580 00000 n
0000000675 00000 n
trailer
<< /Size 9 /Root 1 0 R >>
startxref
771
%%EOF

79
tests/fixtures/outlined-krav.pdf vendored Normal file
View file

@ -0,0 +1,79 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R /Outlines 10 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R 5 0 R 7 0 R] /Count 3 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Resources << /Font << /F1 9 0 R >> >> >>
endobj
4 0 obj
<< /Length 105 >>
stream
BT /F1 12 Tf 20 170 Td (1 Grunnlag) Tj ET
BT /F1 12 Tf 20 150 Td (Innledende tekst om grunnlaget.) Tj ET
endstream
endobj
5 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 6 0 R /Resources << /Font << /F1 9 0 R >> >> >>
endobj
6 0 obj
<< /Length 104 >>
stream
BT /F1 12 Tf 20 170 Td (1.1 Omfang) Tj ET
BT /F1 12 Tf 20 150 Td (Omfanget dekker hele arbeidet.) Tj ET
endstream
endobj
7 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 8 0 R /Resources << /Font << /F1 9 0 R >> >> >>
endobj
8 0 obj
<< /Length 213 >>
stream
BT /F1 12 Tf 20 170 Td (1.2 Krav) Tj ET
BT /F1 12 Tf 20 150 Td (Kravet gjelder alle klasser.) Tj ET
BT /F1 12 Tf 20 130 Td (1.2.1 Materialer) Tj ET
BT /F1 12 Tf 20 110 Td (Materialene skal vaere godkjente.) Tj ET
endstream
endobj
9 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
endobj
10 0 obj
<< /Type /Outlines /First 11 0 R /Last 11 0 R /Count 4 >>
endobj
11 0 obj
<< /Title (1 Grunnlag) /Parent 10 0 R /First 12 0 R /Last 13 0 R /Count 2 /Dest [3 0 R /XYZ 20 185 0] >>
endobj
12 0 obj
<< /Title (1.1 Omfang) /Parent 11 0 R /Next 13 0 R /Dest [5 0 R /XYZ 20 185 0] >>
endobj
13 0 obj
<< /Title (1.2 Krav) /Parent 11 0 R /Prev 12 0 R /First 14 0 R /Last 14 0 R /Count 1 /Dest [7 0 R /XYZ 20 185 0] >>
endobj
14 0 obj
<< /Title (1.2.1 Materialer) /Parent 13 0 R /Dest [7 0 R /XYZ 20 150 0] >>
endobj
xref
0 15
0000000000 65535 f
0000000009 00000 n
0000000075 00000 n
0000000144 00000 n
0000000270 00000 n
0000000425 00000 n
0000000551 00000 n
0000000705 00000 n
0000000831 00000 n
0000001094 00000 n
0000001191 00000 n
0000001265 00000 n
0000001386 00000 n
0000001484 00000 n
0000001616 00000 n
trailer
<< /Size 15 /Root 1 0 R >>
startxref
1707
%%EOF

13
tests/test_extract.py
View file

@ -335,14 +335,19 @@ def test_evidence_class_is_asserted_not_commented() -> None:
measured on. A comment saying so rots; an assertion that names them keeps
an unmeasured row from quietly presenting as a supported one.
The table is NOT the converter's rows alone: `.html` is core-supported and
was the one core row that had never met a real document, so it carries a
class here too (2026-09-09, `measured`, 828 files).
The table is NOT the converter's rows alone: `.html` and `.pdf` are
core-supported and carry a class here too -- `.html` since 2026-09-09
(`measured`, 828 files), `.pdf` since 2026-09-10 (`measured`, eight corpus
documents with a hand-counted fasit plus a publisher's own 2 761-section
structure for a 701-page one). `.pdf` was the row with the most measurement
behind it and no row in the table at all, which is the one way a table like
this can mislead while every entry in it is true.
"""
from llm_ingestion_okf.extract import _EVIDENCE, _PANDOC_FORMATS
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html"}, "every row needs a class"
assert set(_EVIDENCE) == set(_PANDOC_FORMATS) | {".html", ".pdf"}, "every row needs a class"
assert {s for s, e in _EVIDENCE.items() if e == "measured"} == {
".pdf",
".docx",
".xlsx",
".html",

131
tests/test_pdf_outline.py Normal file
View file

@ -0,0 +1,131 @@
"""The PDF's own `/Outlines` tree as a segmentation source.
MEASURED, OUTSIDE THIS REPOSITORY, ON ONE DOCUMENT: a 701-page process code
whose publisher also ships a NISO-STS structure for it. `okf build` on the
default recovered 1 967 of its 2 761 titled sections, 0 of 28 at the top level,
and **794 of 794 misses had their heading text present in the extracted text**
-- the boundary was never opened, the line was read. The same file carries an
`/Outlines` tree of 2 763 nodes which, after `re.sub(r"\\s+","",s).lower()`,
matches 2 761 of 2 761 STS titles exactly. The index shipped inside the file
and the build discarded it unopened.
WHAT THESE TESTS PIN, and why each one exists rather than "it found more":
- the LEVEL a node declares, not its distance from the root. A two-level tree
cannot tell those apart, so the fixture has three.
- the LINE a mark lands on. 2 706 of 2 761 bookmarks in that document share a
destination page with another bookmark, so a bridge that resolved to the page
and stopped would be wrong on almost every node while still looking like it
worked. The fixture's last page carries four lines and its second bookmark
points at the third.
- a PDF with no `/Outlines` behaving IDENTICALLY with the arm on -- `pdfminer`
raises `PDFNoOutlines` there, and "this file has no index" is not an error.
- an unresolvable `/Dest` being dropped and COUNTED. That document has 0 of
2 763; a PDF in the wild has them, so without this the branch would ship
having never run.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from llm_ingestion_okf import extract, propose
pdfplumber = pytest.importorskip("pdfplumber")
FIXTURES = Path(__file__).parent / "fixtures"
OUTLINED = FIXTURES / "outlined-krav.pdf"
BROKEN = FIXTURES / "outline-broken-dest.pdf"
NO_OUTLINE = FIXTURES / "three-page-krav.pdf"
def test_marks_carry_the_declared_level_and_the_line_they_land_on() -> None:
data = OUTLINED.read_bytes()
text = extract.extract_text(OUTLINED.name, data)
lines = text.split("\n")
outline = extract.pdf_outline(OUTLINED.name, data)
assert outline.unresolved == 0
assert [(mark.line, mark.level, mark.title) for mark in outline.marks] == [
(0, 1, "1 Grunnlag"),
(3, 2, "1.1 Omfang"),
(6, 2, "1.2 Krav"),
(8, 3, "1.2.1 Materialer"),
]
# The offsets, stated against the text rather than against the tree: a mark
# that named the right title on the wrong line would pass the list above if
# the list were derived from the same walk.
for mark in outline.marks:
assert lines[mark.line] == mark.title
def test_the_last_mark_is_on_the_third_line_of_its_page_not_the_first() -> None:
"""Page resolution alone would put it two lines early, and still look right."""
data = OUTLINED.read_bytes()
marks = extract.pdf_outline(OUTLINED.name, data).marks
page_three = [mark for mark in marks if mark.title.startswith("1.2")]
assert [mark.line for mark in page_three] == [6, 8]
def test_a_pdf_with_no_outlines_yields_no_marks_and_no_error() -> None:
data = NO_OUTLINE.read_bytes()
outline = extract.pdf_outline(NO_OUTLINE.name, data)
assert outline.marks == ()
assert outline.unresolved == 0
def test_a_non_pdf_is_not_asked_the_question() -> None:
outline = extract.pdf_outline("notat.md", b"# Overskrift\n\nBrodtekst.\n")
assert outline.marks == ()
assert outline.unresolved == 0
def test_an_unresolvable_destination_is_dropped_and_counted() -> None:
data = BROKEN.read_bytes()
outline = extract.pdf_outline(BROKEN.name, data)
assert [mark.title for mark in outline.marks] == ["1 Grunnlag"]
assert outline.unresolved == 1
def test_the_outline_replaces_the_text_heuristics_when_it_is_given() -> None:
data = OUTLINED.read_bytes()
text = extract.extract_text(OUTLINED.name, data)
marks = extract.pdf_outline(OUTLINED.name, data).marks
candidates = propose.find_candidates(text, outline_marks=marks, close_span_gaps=True)
assert [(c.number, c.title, c.level, c.rule) for c in candidates] == [
("1", "Grunnlag", 1, propose.RULE_PDF_OUTLINE),
("1.1", "Omfang", 2, propose.RULE_PDF_OUTLINE),
("1.2", "Krav", 2, propose.RULE_PDF_OUTLINE),
("1.2.1", "Materialer", 3, propose.RULE_PDF_OUTLINE),
]
# The spans partition the text: the arm cuts, it does not drop.
assert candidates[0].start == 0
assert candidates[-1].end == len(text)
def test_a_declared_section_with_no_prose_of_its_own_survives() -> None:
"""The orphan check asks the wrong question of a publisher's own tree.
Measured on that 701-page document: 683 of its 2 762 bookmark marks are
followed immediately by their first subsection, with no prose between. They
are container sections, not mis-detected headings -- the check exists to
catch a heuristic's false positive, and there is no heuristic here. Left
in, the arm scores 2 079 of 2 761 instead of 2 762 of 2 763.
"""
text = "1 Grunnlag\n1.1 Omfang\nOmfanget dekker alt.\n"
marks = (
extract.OutlineMark(line=0, level=1, title="1 Grunnlag"),
extract.OutlineMark(line=1, level=2, title="1.1 Omfang"),
)
candidates = propose.find_candidates(text, outline_marks=marks)
assert [c.title for c in candidates] == ["Grunnlag", "Omfang"]
def test_an_empty_mark_list_leaves_every_other_rule_untouched() -> None:
"""`()` is "this file has no index", never "propose nothing"."""
data = OUTLINED.read_bytes()
text = extract.extract_text(OUTLINED.name, data)
assert propose.find_candidates(text, outline_marks=()) == propose.find_candidates(text)