open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it

Browse source 
K3 round 5. Three questions, three answers, and two of them correct a figure
this repository published.

RETRIEVAL FIRST, because it could have reversed a default. hit@8 over the six
questions on BOTH K2 bundles -- Arm B at 629 concepts and the shipped default
at 492 -- is 5 of 6 with ranks 1,1,1,1,1,- on each, so 0 of 6 rows lost. The
order's rule reverses `--unit-fold` at >= 2 of 6; it does not fire, and the
default stands. The gold sets shrink (49->26, 20->17, 43->36, 11->18) while
every rank holds at 1, which is the fold merging concepts rather than removing
a document from the top.

TWO PUBLISHED NUMBERS CORRECTED, both ours. The S7 candidate ranks 96 of 629
and 159 of 492 were measured with the cost vocabulary passed to
`concept_scores` and NOT to `document_scores`, while `build_payload` passes it
to both; scored the way the shipped payload scores it, the same concept is 10
of 629 and 19 of 492. And round 4 attributed its non-delivery to the default
move -- measured here, it is not delivered on the Arm B bundle either, for a
different reason (knapsack eviction at 68 046 bytes of a 120 000 budget, versus
`below_k`). That column had been inherited from round 3's own build, never
re-measured.

`--pdf-headings font-reserve`, OFF, and the hypothesis behind it is falsified
by its own condition rather than by a score: position 7, the one position the
flag exists for, has THREE outline runs, so the reserve is silent there at
every minimum. It changes 0 of 12 cells on the reference and reaches 4 of 39
corpus documents, none of them rated. Built anyway because it was authorised
and because the condition is now measured rather than assumed. The predicate
lives in one place (`propose.heading_reserve_applies`) and the door receives it
as a callable, like `gate`: a plan indexes the exact string it was proposed
against, so a reserve firing on one side only would make every document it
touches a coded rejection.

The `xlsx` re-reading is confirmed on the artifact -- 11 `rule:sheet-section`
units plus 1 `rule:table-block` ingress -- but the number alone makes the cell
worse (distance 1 -> 2), because the criterion counts that ingress as a table
that should have been merged. A hit needs both halves ratified, and the
reference is the operator's.

`--sheet-section-rows` as a default: three cells better and none worse on the
twelve positions, but the K2 control moves -- row 1's gold document splits 1 ->
12 concepts and its best concept ranks 2 instead of 1. Condition not met,
default not moved.

Default build byte-identical before and after (`diff -r`, 30 md files).
Suite 1441 -> 1449; three of the eight were red first.

Report: docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 00:25:51 +02:00
commit b01492b7f5
10 changed files with 743 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

27
CLAUDE.md
View file

@ -309,7 +309,7 @@ and fixtures, never code.
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- **Two PDF READER flags, both off, and they sit BEFORE every segmentation
- **Three PDF READER flags, all off, and they sit BEFORE every segmentation
flag** -- an arm changes how the proposer cuts a text, these change what the
text says. `--pdf-headings font` infers a heading from typography (dominant
font size above the document's character-weighted body median AND a bold font
@ -335,6 +335,31 @@ and fixtures, never code.
become a default -- an optional dependency in the default path would make an
ordinary install fail on the first scanned page. Report:
`docs/2026-09-08-k3-runde4-pdf-skrift-og-ocr.md`.
**`--pdf-headings font-reserve`** is the third value on that same option
(`none`, `font`, `font-reserve` — three answers to one question, so no caller
can ask for two at once): the same typographic rule applied ONLY where Arm D's
outline gate admits no run at all, typography as a second heading source where
there is no first one. The condition lives in ONE function
(`propose.heading_reserve_applies`) that the proposer and the door both
consult, the door receiving it as a callable the way it already receives
`gate` — a plan indexes the exact string it was proposed against, so a reserve
firing on one side only would turn every document it touches into a coded
rejection. It reads the gate AS CONFIGURED, so at `--outline-run 0` it is
unconditional and equals round 4's "font instead of Arm D". **Off, and the
measurement is that it changes nothing measurable:** on the twelve-position
reference it alters **not one cell** — the five positions where it fires are
one PDF whose glyphs carry no ToUnicode mapping and four office documents the
PDF reader never touches — and the position it was built for has **three**
outline runs, so the reserve is silent there by construction. Its reach is
real but unrated: **4 of 39** readable corpus documents, none in the sample.
Report: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`, which also
corrects two of this repository's own published figures — the S7 candidate
ranks (96 of 629 / 159 of 492 were measured with the cost vocabulary reaching
only half the ranker; consistently scored they are **10 of 629** and **19 of
492**) and round 4's attribution of that concept's non-delivery to the default
move (it is not delivered on the Arm B bundle either, by a different
mechanism). hit@8 over the six published questions holds at **5 of 6 on both
K2 bundles**, so the default move cost the retrieval side nothing.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction

3
README.md
View file

@ -132,11 +132,12 @@ here.
Separate from the six above, and they sit before every one of them: a
segmentation flag changes how the proposer cuts a text, these change what the
text says. **Both are off by default.**
text says. **All three are off by default.**
| flag | what it does | measured |
|---|---|---|
| `--pdf-headings font` | a PDF carries no heading markup, so one is inferred from typography — a line whose dominant font size is above the document's character-weighted median AND whose dominant font name says bold — and emitted as an ATX heading in the same markdown the office path produces, so the existing heading rule reads it | on a tender PDF: **9 of 9** numbered chapters found, plus 4 extra candidates. Whole corpus: **25 of 32 `pdf`** change, **0 of 5 `docx`**, **0 of 2 `xlsx`**. **Off by measurement:** against the operator's unit worksheet it takes `pdf` from **2 of 8 to 0 of 8**, losing two exact matches, because on those documents the outline rule already found the chapters and a second heading source can only add |
| `--pdf-headings font-reserve` | the same typographic rule, applied ONLY to a document whose own numbering the outline arm finds no run of — typography as a second heading source where there is no first one, never on top of one. Three values of one option (`none`, `font`, `font-reserve`), so no caller can ask for two at once | reaches **4 of 39** readable corpus documents (10 of 32 `pdf` admit no outline run; 4 of those render differently at all). **Off by measurement, and the measurement is that it changes nothing measurable:** on the operator's twelve-position unit worksheet it alters **not one cell** — the five positions where it fires are one PDF whose glyphs carry no ToUnicode mapping and four office documents the PDF reader never touches. The position it was built for numbers its own chapters, so the reserve is silent there by construction |
| `--ocr` | read a PDF page as an image when its own text never arrived: the page extracts empty, or as `(cid:N)` placeholder codes at or above 10 % of its characters. Needs the optional `ocr` group | on the one corpus document with the failure: **95.07 % → 0 %** cid, **44 → 2561** words of four or more letters, 17 → **18** pages with text, 3.6 s/page. Whole corpus: **16 of 834** pages qualify, in **1 of 32** files |
```

388
docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md Normal file
View file

@ -0,0 +1,388 @@
# K3 round 5: hit@8 on both K2 bundles, and typography as a reserve
**Date:** 2026-09-08 · **HEAD at start:** `53d5c74` · **Suite at start:** 1441
Round 4 shipped two PDF readers and left three questions behind it. This round
answers the retrieval one first, because it is the one that could have reversed
a default, and then measures the segmentation candidate the round-4 report
pointed at. Two of the three answers are negative, and the negative ones are the
informative ones.
## 0. What this round found, in one paragraph
The default move did **not** cost anything on the retrieval side: hit@8 is 5 of
6 on both K2 bundles, with the same five ranks at 1 and the same row missing on
both. The decision rule therefore does not fire and `--unit-fold` stays in the
default. The font **reserve** — typography read only where the outline arm finds
no run — was built, is off, and must stay off: it cannot reach the position it
was designed for, because that position numbers its own chapters. And the
`xlsx` re-reading works, but only if BOTH halves of it move together; the number
alone makes the cell worse.
Two published figures are corrected here rather than left standing. Both are
this repository's own.
---
## 1. Reproduction of the starting state
Every figure below was measured on `53d5c74` with a clean tree.
| what | expected | measured |
|---|---|---|
| `git rev-parse HEAD` | `53d5c74` | `53d5c74c96b4316dcb3380227d72be908e1be270` |
| `git status --short` | empty | empty |
| `pytest -q` | 1441 | **1441 passed** |
| `ruff check src tests tools` | clean | `All checks passed!` |
| `mypy src` | clean | `Success: no issues found in 21 source files` |
| `okf build <folder>`, no flags | 30 md | **30**, exit 0, 0/5 rejected |
| same, `--keep-table-heading --sheet-section-rows --drop-wrapped-outline` | 52 md | **52**, exit 0, 0/5 rejected |
| unpushed commits | 4 | **4** (`47f6619`, `efa92ce`, `f6fea13`, `53d5c74`) |
| tag `v0.6.0` | on `f6fea13` | **on `f6fea13`**, one commit behind HEAD |
Both cached bundles were counted before use rather than trusted: the Arm B tree
holds **1108** md files and round 4's default artifact **944**, so neither was
rebuilt.
---
## 2. hit@8 on both K2 bundles
One instrument (`tools/okf_consume_measure.py`), one gold set, one `--k 8`, run
against both bundles. The gold set is local-only and neither the questions nor
the gold documents appear here; the row number carries the identity, as in
`docs/2026-09-07-okf-konsumskill-maaling.md`.
**Bundle 1 (Arm B):** 629 concepts, 1108 files, every arm flag off.
**Bundle 2 (default):** 492 concepts, 944 files, digest `bdefa679957edd7d…`,
default flags and default epoch stamp.
Both bundles are the same 39-document readable set, so `documents` is 39 in
both and the chance baselines are directly comparable.
| row | gold concepts (B) | hit (B) | rank (B) | considered (B) | spent (B) | chance (B) | gold concepts (D) | hit (D) | rank (D) | considered (D) | spent (D) | chance (D) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | yes | **1** | 629 | 76 897 | 0.013 | 1 | yes | **1** | 492 | 43 288 | 0.016 |
| 2 | 49 | yes | **1** | 629 | 21 403 | 0.479 | 26 | yes | **1** | 492 | 18 458 | 0.354 |
| 3 | 20 | yes | **1** | 629 | 26 282 | 0.229 | 17 | yes | **1** | 492 | 28 347 | 0.247 |
| 4 | 43 | yes | **1** | 629 | 19 818 | 0.434 | 36 | yes | **1** | 492 | 21 125 | 0.458 |
| 5 | 11 | yes | **1** | 629 | 22 459 | 0.132 | 18 | yes | **1** | 492 | 17 866 | 0.259 |
| 6 | 5 | **no** | – | 629 | 20 282 | 0.062 | 5 | **no** | – | 492 | 33 113 | 0.079 |
| **total** | | **5 of 6** | | | | 1.35 expected by chance | | **5 of 6** | | | | 1.41 expected by chance |
**0 of 6 rows lost.** The decision rule in the order fires at `>= 2 of 6`, so it
does not fire: `--unit-fold` stays in the default, and no `DEFAULT_UNIT_FOLD`
was touched. This reproduces the PM measurement of 21:20Z exactly, including
the ranks.
**The gold sets change size while the rank does not.** 49→26, 20→17, 43→36 and
11→18 concepts between the two bundles, and every one of those rows still ranks
1. That is the fold merging concepts rather than removing a document from the
top — which is the property the fold was defended on, measured here on the
retrieval side for the first time.
Both known-negative controls behave identically on both bundles: 2 of 6 tokens
reaching nothing on the planned negative, 4 of 6 on the lexically disjoint one,
8 excerpts delivered in each case. The negatives are a control on the question,
not on the ranker, and they did not move.
### 2.1 The S7 control row, and a correction to how it was measured
A separate question with a separate answer, measured with
`--cost-vocabulary --k 12` on both bundles:
| | Arm B (629) | default (492) |
|---|---|---|
| priced concept present | yes | yes |
| **candidate rank (whole ranking)** | **10 of 629** | **19 of 492** |
| rank among concepts that survive the gate | 10 of 280 | 19 of 280 |
| delivered at `--k 12` | **no** | **no** |
| why not | `over_budget_after_knapsack` — inside the shortlist, evicted; the excerpt weighs **68 046** of a 120 000 budget | `below_k` — outside the shortlist at k = 12 |
| excerpts delivered | 11 | 12 |
| budget spent | 84 953 | 51 810 |
**Two published numbers are corrected by this row.**
**First, the ranks.** Round 3 and round 4 published this concept at *96 of 629*
and *159 of 492*. Both are reproducible and both name a mixed configuration:
the measuring script (`/tmp/k3r4/rank4.py`) passes `cost_vocabulary=True` to
`concept_scores` but calls `document_scores(root, question)` with the vocabulary
left off, while `build_payload` passes it to both. Scored the way the shipped
payload scores it, the same concept is **10 of 629** and **19 of 492**. The
published figures are not wrong about their own arithmetic; they are answers
about a ranking the library never emits.
**Second, the attribution.** Round 4 wrote that the non-delivery "is the default
move of `47f6619`/`f6fea13` showing up on the retrieval side for the first
time". Measured here on the Arm B bundle with today's code, the concept is **not
delivered there either**. The default move is not the cause; the two bundles
fail to deliver it for two *different* reasons, and only one of them is about
bundle size. The knapsack eviction on Arm B is the case `--reserve-top-rank`
was built for and cannot fix, because the reservation is for rank one and this
is rank ten — which is exactly what the flag's own documentation already says.
What round 4 could not have known is that the Arm B column was never measured;
it was carried over from round 3's own build. Re-running it was the whole
correction.
---
## 3. The font reserve
Round 4 measured the font reader two ways and shipped neither: on TOP of Arm D
it took `pdf` from 2 of 8 to 0 of 8, and INSTEAD of Arm D (`--outline-run 0`) it
scored 1 of 8 while landing position 7 on 30 = its reference count. The
hypothesis this round was asked to test: use typography only where Arm D's
outline gate admits no run, and position 7 is bought without paying positions 2
and 8.
**The hypothesis is falsified, and it is falsified by the condition itself
rather than by a score.**
### 3.1 Where the reserve can fire, on the 12-position reference
Measured before any code was written, on the cached round-3 rendering:
| position | type | outline runs at minimum 3 | reserve |
|---|---|---|---|
| 0 | pdf | 0 | **fires** |
| 1 | pdf | 2 (last run 7 long) | silent |
| 2 | pdf | 2 (last run 9) | silent |
| 3 | xlsx | 0 | fires (no effect: the font reader is PDF-only) |
| 4 | pdf | 1 (last run 5) | silent |
| 5 | docx | 0 | fires (no effect: PDF-only) |
| 6 | pdf | 1 (last run 3) | silent |
| **7** | **pdf** | **3 (last run 10)** | **silent** |
| 8 | pdf | 2 (last run 8) | silent |
| 9 | pdf | 3 (last run 6) | silent |
| 10 | docx | 0 | fires (no effect: PDF-only) |
| 11 | docx | 0 | fires (no effect: PDF-only) |
Position 7 — the one position the flag exists for — has **three** outline runs,
the last of them ten entries long. No reading of "no run at all" reaches it: it
is a run at minimum 3, at minimum 1, and at every minimum up to 10. The 1 of 8
that round 4 measured under `--outline-run 0` came from *removing* Arm D on that
document, not from adding a second source where there was none.
### 3.2 What it scores
| pos | type | ref \|F\| | default | **+ reserve** | + font everywhere |
|---|---|---|---|---|---|
| 0 | pdf | undefined | for grovt (U1) | for grovt (U1) | for grovt (U1) |
| 1 | pdf | 15 | for fint (U17, d2) | for fint (U17, d2) | for grovt (U5, d10) |
| 2 | pdf | 9 | **treffer** (d0) | **treffer** (d0) | for fint (U12, d3) |
| 3 | xlsx | 11 | for grovt (U0, S1, d12) | for grovt (U0, S1, d12) | for grovt (U0, S1, d12) |
| 4 | pdf | 1 | for fint (U4, d3) | for fint (U4, d3) | for fint (U11, d10) |
| 5 | docx | 3 | **treffer** (d0) | **treffer** (d0) | **treffer** (d0) |
| 6 | pdf | 6 | for fint (U7, d1) | for fint (U7, d1) | for fint (U11, d5) |
| 7 | pdf | 30 | for fint (U33, d3) | for fint (U33, d3) | for fint (U33, d3) |
| 8 | pdf | 8 | **treffer** (d0) | **treffer** (d0) | for fint (U16, d8) |
| 9 | pdf | 4 | for fint (U10, d6) | for fint (U10, d6) | for fint (U12, d8) |
| 10 | docx | 2 | **treffer** (d0) | **treffer** (d0) | **treffer** (d0) |
| 11 | docx | 1 | **treffer** (d0) | **treffer** (d0) | **treffer** (d0) |
| | | | **5 of 12** · pdf 2 of 8 · docx 3 of 3 · xlsx 0 of 1 | **5 of 12**, identical cell for cell | 4 of 12 · pdf 1 of 8 |
**The reserve changes not one cell of the twelve.** It fires on five positions
and alters none of them: position 0's glyphs carry no ToUnicode mapping, so its
font rendering is byte-identical to its plain one, and the other four are office
documents the PDF reader never touches. The default column reproduces round 4
exactly (pdf 2 of 8, docx 3 of 3, xlsx 0 of 1), which is what makes the
instrument calibrated rather than merely re-run.
The acceptance criterion the order set — *position 7 hits 30 without losing 2, 6
and 8* — is **unreachable by construction**, not merely unmet. And position 7's
`(c)` in the reference is `I`, so the criterion's literal gate forbids a hit
there in any case; both readings stood in round 4 § 3 and both stand here.
### 3.3 What the reserve reaches on the whole corpus
The twelve positions are a sample of 43. Over the full cached corpus:
| | count |
|---|---|
| readable documents | 39 |
| of which pdf | 32 |
| pdf admitting no outline run at minimum 3 | **10** |
| of those, whose font rendering differs at all | **4** |
| of those, whose plan changes | **4** (1→4, 1→2, 0→2 and 5→6 entries) |
So the flag is not inert: it acts on **4 of 39** readable documents. But **none
of those four is in the rated sample**, so there is no reference count for any
of them, and this round cannot say whether the four changes are improvements.
A flag with a reach and no measurable quality signal is exactly a flag that
stays off.
### 3.4 What was built anyway, and why
`--pdf-headings font-reserve`: a third value on the option round 4 introduced,
so `none`, `font` and `font-reserve` are three answers to one question and no
caller can ask for two at once.
The condition lives in one function, `propose.heading_reserve_applies(text, *,
outline_run)`, and the proposer and the door both call it. That is not tidiness:
a plan indexes the exact string it was proposed against (`text_sha256`), so a
reserve that fired on one side and not the other would turn every document it
touches into a coded rejection. The door receives it as a **predicate**
(`heading_reserve: Callable[[str], bool] | None`), the same shape `gate` already
has, because the condition is the proposer's outline grammar and Door B does not
own that grammar — a direct import would point the dependency against the
layering.
The reserve reads the gate **as configured**. At `--outline-run 0` the gate
admits nothing at all, so the reserve becomes unconditional; that combination is
round 4's "font instead of Arm D" at 1 of 8, and a caller reaching it does so
deliberately. This is stated in the code and pinned by a test.
**Control:** the default build of the operator's folder is **byte-identical**
before and after the change (`diff -r`, 30 md files either way), and the suite
went 1441 → **1449** with eight new tests, three of which were red first for
three different reasons (missing predicate, wrong expected segmentation, wrong
report field).
---
## 4. The `xlsx` reference count, re-read as 12
The operator's re-reading is that the priced worksheet holds **11 numbered cost
groups plus one leading block of real prose that has to live somewhere**, so
11 + 1 = 12. Measured on the extraction, under `--sheet-section-rows`:
| unit | rule | count |
|---|---|---|
| the leading block | `rule:table-block` | 1 |
| the cost groups | `rule:sheet-section` | 11 |
| **total** | | **12** |
The rule produces exactly 12 entries, and the split is 11 + 1 in the shape the
re-reading describes. **The re-reading is confirmed on the artifact.**
**But the number alone does not deliver the hit, and this is the finding.** The
criterion as written treats a table-derived concept under `b = M` as *S* — a
table that should have been merged into a heading — not as a unit. So:
| reading of the reference | U | T | S | distance | label |
|---|---|---|---|---|---|
| \|F\| = 11, leading block counted as S (rounds 3 and 4) | 11 | 0 | 1 | **1** | for fint |
| \|F\| = 12, leading block still counted as S | 11 | 0 | 1 | **2** | for grovt |
| \|F\| = 12, leading block counted as the 12th unit | **12** | 0 | 0 | **0** | **treffer** |
Changing the number without changing what the criterion counts makes the cell
**worse**, from distance 1 to distance 2. `xlsx` reaches **1 of 1** only when
both halves move together.
**This is the operator's reference, not mine.** What is measured here is that
the artifact contains 11 + 1 units in exactly the shape the re-reading names,
and that a hit requires ratifying the second half too. Rounds 3 and 4 both used
the reading "11" and both reported distance 1, so nothing published so far
depends on the outcome either way.
---
## 5. `--sheet-section-rows` as a default: what the 12 positions say
Because § 4 raised it, the D1/D3 configuration was scored against the shipped
default on all twelve positions:
| pos | default | + `--keep-table-heading --sheet-section-rows --drop-wrapped-outline` |
|---|---|---|
| 3 (xlsx) | for grovt, d12 | **for fint, d1** (treffer, d0 under the re-read reference) |
| 4 (pdf) | for fint, d3 | **treffer, d0** |
| 9 (pdf) | for fint, d6 | **for fint, d2** |
| all other nine | — | identical, cell for cell |
**Three positions improve, none regresses**, taking `pdf` from 2 of 8 to 3 of 8
and leaving `docx` at 3 of 3. That is the strongest segmentation result this
round has, and it is **not acted on here**: the order makes a default move
conditional on the K2 control as well, and it is a fourth change to every
consumer's bytes in two days. The K2 control was run for exactly that
reason and it does not hold (§ 7), so the default stays where it is.
---
## 6. What still stands, unchanged
- **The prefix over-matches the other way** (round 3 finding, untouched since):
a compound word matches a four-character prefix, df 159 of 270. It belongs to
the rarity weight and was not this round's work.
- **`pdf` position 1** (17 against a reference of 15) is now refused on four
axes: number, outline, typography, and the reserve — the reserve is silent
there, because that document has two outline runs.
- **vegnormal-okf FUNN 1 and FUNN 2** are still not investigated in this repo.
- **`tests/test_okf_consume.py:940` pins hit@8 against the Arm B bundle alone.**
The default bundle is pinned nowhere. That is a gap, and it is named in
STATE's next block rather than closed here: O4/O5 fence that file, and this
round only ran the instrument.
---
## 7. The K2 control for the D1/D3 configuration: it moves, so nothing moves
The whole 43-document corpus rebuilt with
`--keep-table-heading --sheet-section-rows --drop-wrapped-outline` on top of the
shipped default. Exit 0, 2305 s wall (773 s inside the door), 39/43 persisted,
4/43 coded rejections -- the same conservation identity as the default build.
| | default | + D1/D3 |
|---|---|---|
| concepts reachable from the index tree | 492 | **491** |
| md files | 944 | **942** |
| hit@8 over the six questions | **5 of 6** | **5 of 6** |
| expected by chance | 1.41 | 1.57 |
| row | gold concepts (default) | rank | gold concepts (D1/D3) | rank |
|---|---|---|---|---|
| 1 | 1 | **1** | **12** | **2** |
| 2 | 26 | 1 | 26 | 1 |
| 3 | 17 | 1 | 17 | 1 |
| 4 | 36 | 1 | 36 | 1 |
| 5 | 18 | 1 | 17 | 1 |
| 6 | 5 | -- | 5 | -- |
**The count holds and the control does not.** Row 1's gold document is split
from one concept into twelve by the sheet-section rule, and its best concept
then ranks **2 instead of 1**. It is still a hit at 8, and it is still a change
to a published number on a corpus the operator reads.
The order's condition for a default move is "no regression on the 43-document
corpus **or** the K2 control". The 12-position reference improves three cells
and regresses none (§ 5); the K2 control regresses one row's rank. **The
condition is therefore not met and the default is not moved.** The flags stay
reachable and off, exactly as they were.
That is worth stating plainly rather than as a near miss: the segmentation side
and the retrieval side disagreed about this configuration, and the disagreement
is real -- splitting a worksheet into its rows is what the reference asks for
and what costs the ranker a rank position, because twelve small concepts each
carry less of the question than one whole sheet did.
---
## 8. Honesty limits
- **n = 6 questions** on the retrieval side, and **n = 12 positions** on the
segmentation side, of which 8 pdf, 3 docx and 1 xlsx. Every per-type figure
in this report is a fraction of 8, 3 or 1. A "3 of 8" is three documents.
- **One rater, one reference.** The twelve-cell reference and the criterion that
scores it were authored by the same operator, and neither has an independent
second rating. Where the criterion and the reference disagree (§ 4), that is
visible only because the disagreement was arithmetic.
- **Both bundles are the same 43-document corpus**, so "hit@8 held on both" is a
statement about one corpus read two ways, not about two corpora.
- **"Delivered" is not "answered".** hit@8 says a gold document appeared among
eight excerpts. Nothing here measures whether a reader could answer the
question from those excerpts; the one measurement that ever tried
(`docs/2026-09-07-okf-konsumskill-maaling.md`) got the correct answer on 1 of
3 bundles.
- **The reserve's corpus reach (§ 3.3) was measured by simulation**, not by
running `okf build --pdf-headings font-reserve` over all 43 documents: the
two cached renderings were combined by the same predicate the shipped code
calls. The predicate is identical; the extraction path around it was exercised
only on the fixtures and on the 12-document sample.
- **The four documents the reserve changes have no reference count.** They are
outside the rated sample, so "4 of 39 change" is a reach, not a quality claim.
- **The corrected S7 ranks (§ 2.1) were measured on today's code.** Round 3's
own build is not available, so this report cannot separate "the instrument was
mixed" from "the code also moved" for the *delivery* outcome; it can and does
separate it for the *rank*, because the rank reproduces exactly under the
published instrument and changes only when the vocabulary is passed
consistently.

44
src/llm_ingestion_okf/cli.py
View file

@ -70,6 +70,7 @@ from __future__ import annotations
import argparse
import sys
import tempfile
from functools import partial
from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, load_plans, measure
@ -77,6 +78,7 @@ from .errors import IngestError
from .inbox import walk_inbox
from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
from .propose import ProposerError
from .propose import heading_reserve_applies
from .propose import run as propose_run
__all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
@ -150,6 +152,18 @@ DEFAULT_UNIT_FOLD = True
DEFAULT_PDF_HEADINGS = False
DEFAULT_OCR = False
#: Round 5's third value on the SAME axis, and off for a third reason: not
#: caution, not construction, but a reachable set with no reference in it.
#: The reserve reads typography only where the outline gate admits no run, so
#: over the 43-document corpus it can act on 4 of 39 readable documents -- and
#: on the twelve-position reference it changes NOTHING, because the only
#: positions where it fires are one PDF whose glyphs carry no ToUnicode
#: mapping and four office documents the PDF reader never touches. The
#: position it was built for numbers its own chapters, so the reserve is
#: silent there by construction. Measured in
#: `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`.
DEFAULT_PDF_HEADINGS_RESERVE = False
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -171,6 +185,7 @@ def _propose_plans(
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -203,6 +218,7 @@ def _propose_plans(
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
)
except ProposerError as exc:
@ -234,6 +250,7 @@ def build(
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -252,6 +269,13 @@ def build(
"""
if proposed_at is None:
proposed_at = ingested_at
# Bound to THIS run's outline minimum, once, so the proposer and the door
# cannot be handed two different thresholds for the same question.
reserve = (
partial(heading_reserve_applies, outline_run=outline_run)
if pdf_headings_reserve and not pdf_headings
else None
)
if not segments:
report = measure(
inbox,
@ -259,6 +283,7 @@ def build(
ingested_at=ingested_at,
profile=STRUCTURED_V1,
pdf_headings=pdf_headings,
heading_reserve=reserve,
ocr=ocr,
)
_write_log(bundle, report, profile=STRUCTURED_V1)
@ -292,6 +317,7 @@ def build(
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
)
print(
@ -308,6 +334,7 @@ def build(
profile=SEGMENTED_OKF_V0_2,
root_frontmatter_values={"okf_version": okf_version, "bundle_id": bundle_id},
pdf_headings=pdf_headings,
heading_reserve=reserve,
ocr=ocr,
)
_write_log(bundle, report, profile=SEGMENTED_OKF_V0_2)
@ -507,8 +534,14 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font"),
default="font" if DEFAULT_PDF_HEADINGS else "none",
choices=("none", "font", "font-reserve"),
default=(
"font"
if DEFAULT_PDF_HEADINGS
else "font-reserve"
if DEFAULT_PDF_HEADINGS_RESERVE
else "none"
),
help=(
"how a PDF's headings are recovered before any arm reads the text. "
"none (the default) is today's extraction: a PDF carries no heading "
@ -518,7 +551,11 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"heading in the same markdown the office path produces, so the "
"existing heading rule reads it and no PDF-only grammar exists. A "
"CONJUNCTION, measured: adding weight as a disjunct took precision "
"from 0.786 to 0.524"
"from 0.786 to 0.524. font-reserve is font applied ONLY to a "
"document whose own numbering the outline arm finds nothing of -- "
"typography as a second source where there is no first one, never "
"on top of one. Three values of one option, so no caller can ask "
"for two at once"
),
)
build_parser.add_argument(
@ -566,6 +603,7 @@ def main(argv: list[str] | None = None) -> int:
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,
)
except (IngestError, OSError, ValueError) as exc:

4
src/llm_ingestion_okf/corpus.py
View file

@ -36,7 +36,7 @@ import argparse
import json
import sys
import time
from collections.abc import Mapping
from collections.abc import Callable, Mapping
from dataclasses import dataclass, replace
from pathlib import Path
@ -281,6 +281,7 @@ def measure(
profile: BundleProfile = STRUCTURED_V1,
root_frontmatter_values: Mapping[str, str] | None = None,
pdf_headings: bool = False,
heading_reserve: Callable[[str], bool] | None = None,
ocr: bool = False,
) -> CorpusReport:
"""Run the corpus through the door and count what happened.
@ -304,6 +305,7 @@ def measure(
root_frontmatter_values=root_frontmatter_values,
segmentations=plans,
pdf_headings=pdf_headings,
heading_reserve=heading_reserve,
ocr=ocr,
)
elapsed = time.monotonic() - started

20
src/llm_ingestion_okf/inbox.py
View file

@ -743,6 +743,7 @@ def process_inbox(
segmentation: SegmentationPlan | None = None,
segmentations: Mapping[str, SegmentationPlan] | None = None,
pdf_headings: bool = False,
heading_reserve: Callable[[str], bool] | None = None,
ocr: bool = False,
) -> InboxResult:
"""Convert every file dropped in `inbox_dir` into an OKF concept.
@ -962,6 +963,23 @@ def process_inbox(
pdf_headings=pdf_headings,
ocr=ocr,
)
# The heading RESERVE, supplied as a predicate rather than decided
# here: the condition is the proposer's outline grammar, and the
# door does not own that grammar. A callable keeps the dependency
# pointing the way the layers do -- the same shape `gate` already
# has -- while guaranteeing the two sides ask ONE question. Both
# `text` and `units` move together, because a locator built from
# one rendering cannot address the other.
reading_fonts = pdf_headings
if heading_reserve is not None and not pdf_headings and heading_reserve(text):
reading_fonts = True
text = extract_text(
source_name(path),
source_bytes,
renderer=_resolve_renderer(profile, path.name),
pdf_headings=True,
ocr=ocr,
)
# Computed from the SAME text the plan's offsets index, so the
# locator and the offset can never disagree about which rendering
# they describe. `None` when the profile names no provenance:
@ -972,7 +990,7 @@ def process_inbox(
source_name(path),
source_bytes,
text,
pdf_headings=pdf_headings,
pdf_headings=reading_fonts,
ocr=ocr,
)
if profile.provenance is not None

29
src/llm_ingestion_okf/propose.py
View file

@ -310,6 +310,26 @@ def outline_lines(text: str) -> list[tuple[int, int, str]]:
return found
def heading_reserve_applies(text: str, *, outline_run: int) -> bool:
"""Whether this text needs a SECOND heading source, having no run of its own.
The font reader's reserve condition, and the only place it is decided. The
proposer and the door both call this, because a plan indexes the exact
string it was proposed against: a reserve that fired on one side and not
the other would make every document it touched a coded rejection.
It reads the gate AS CONFIGURED rather than a fixed minimum -- raising the
arm's threshold widens the reserve, which is the same document property
seen through the same threshold. At `outline_run` 0 the gate admits nothing
at all, so the reserve is unconditional; that combination is round 4's
"font instead of Arm D", measured at 1 of 8, and a caller reaching it gets
it deliberately.
"""
if outline_run <= 0:
return True
return not outline_runs(outline_lines(text), outline_run)
def outline_runs(
entries: list[tuple[int, int, str]], minimum: int
) -> list[list[tuple[int, int, str]]]:
@ -1119,6 +1139,7 @@ def run(
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
) -> int:
if max_segment_chars < 0:
@ -1164,6 +1185,14 @@ def run(
# against another is refused by `assert_plan_applies`, which is the
# right outcome and a confusing one to debug.
text = extract_text(source.name, source_bytes, pdf_headings=pdf_headings, ocr=ocr)
# The reserve, and the reason it re-extracts rather than post-processes:
# the font reader works on the PDF's glyph geometry, which the joined
# text no longer carries. Skipped outright when the font reader is
# already on -- `font` and `font-reserve` are two values of one option,
# never a pair to combine.
if pdf_headings_reserve and not pdf_headings:
if heading_reserve_applies(text, outline_run=outline_run):
text = extract_text(source.name, source_bytes, pdf_headings=True, ocr=ocr)
except IngestError as exc:
raise ProposerError(f"cannot extract text from {source.name}: {exc}") from exc

20
tests/fixtures/make_fixtures.py vendored
View file

@ -65,6 +65,23 @@ FONT_HEADING_CONTENT = (
)
# The SAME typography as `FONT_HEADING_CONTENT`, over a document that numbers
# its own chapters. It exists for the heading RESERVE and for nothing else: the
# reserve reads typography only where Arm D's outline gate admits no run, so
# proving it stays silent needs a document where both signals are present and
# only one of them may be used. Three ascending integers at line start, which
# is a run at the build default's minimum of 3.
NUMBERED_FONT_CONTENT = (
b"BT /F2 20 Tf 50 700 Td (Forord) Tj ET\n"
b"BT /F1 10 Tf 50 670 Td (1 Generelle krav) Tj ET\n"
b"BT /F1 10 Tf 50 640 Td (Utkilingen skal ha helning 1:15.) Tj ET\n"
b"BT /F1 10 Tf 50 610 Td (2 Merking) Tj ET\n"
b"BT /F1 10 Tf 50 580 Td (Kravet gjelder alle veiklasser.) Tj ET\n"
b"BT /F1 10 Tf 50 550 Td (3 Vedlegg) Tj ET\n"
b"BT /F1 10 Tf 50 520 Td (Vedlegget er eget oppslag.) Tj ET\n"
)
def build_two_font_pdf(content: bytes) -> bytes:
"""A one-page PDF whose resources declare BOTH a regular and a bold font.
@ -419,6 +436,9 @@ if __name__ == "__main__":
(HERE / "font-heading-krav.pdf").write_bytes(build_two_font_pdf(FONT_HEADING_CONTENT))
print("wrote font-heading-krav.pdf")
(HERE / "numbered-font-krav.pdf").write_bytes(build_two_font_pdf(NUMBERED_FONT_CONTENT))
print("wrote numbered-font-krav.pdf")
for name, parts in (
("two-line-krav.docx", _DOCX_PARTS),
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),

42
tests/fixtures/numbered-font-krav.pdf vendored Normal file
View file

@ -0,0 +1,42 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R /F2 6 0 R >> >> >>
endobj
4 0 obj
<< /Length 353 >>
stream
BT /F2 20 Tf 50 700 Td (Forord) Tj ET
BT /F1 10 Tf 50 670 Td (1 Generelle krav) Tj ET
BT /F1 10 Tf 50 640 Td (Utkilingen skal ha helning 1:15.) Tj ET
BT /F1 10 Tf 50 610 Td (2 Merking) Tj ET
BT /F1 10 Tf 50 580 Td (Kravet gjelder alle veiklasser.) Tj ET
BT /F1 10 Tf 50 550 Td (3 Vedlegg) Tj ET
BT /F1 10 Tf 50 520 Td (Vedlegget er eget oppslag.) Tj ET
endstream
endobj
5 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>
endobj
6 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding >>
endobj
xref
0 7
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000251 00000 n
0000000654 00000 n
0000000751 00000 n
trailer
<< /Size 7 /Root 1 0 R >>
startxref
853
%%EOF

173
tests/test_pdf_heading_reserve.py Normal file
View file

@ -0,0 +1,173 @@
"""The font reader as a RESERVE: typography only where the outline gate is empty.
Round 4 measured the font reader two ways and shipped neither as a default. On
TOP of Arm D it took `pdf` from 2 of 8 to 0 of 8, because the documents it broke
already numbered their own chapters and the second heading source could only
ADD a title page and a contents listing. INSTEAD of Arm D it scored 1 of 8 but
landed one position exactly on its reference count, which is the observation
this flag comes from: use typography only where the document does not number
itself.
The mechanism is one condition, and these tests pin the condition rather than
the outcome: the reserve reads a rendering it may not have asked for only when
`outline_runs` admits nothing at the configured minimum. Everything else --
which arm ships, whether the reserve helps -- is a measurement, and the
measurement is in `docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md`.
**The reserve is OFF by default and the measurement says it must stay off**: on
the twelve-position reference it changes not one cell, because the only
positions where it fires are a PDF whose glyphs carry no ToUnicode mapping and
four office documents the PDF reader never touches.
"""
from __future__ import annotations
import warnings
from pathlib import Path
from llm_ingestion_okf import cli
from llm_ingestion_okf.extract import extract_text
from llm_ingestion_okf.propose import build_plan, heading_reserve_applies
FIXTURES = Path(__file__).parent / "fixtures"
NUMBERED_PDF = FIXTURES / "numbered-font-krav.pdf"
FONT_PDF = FIXTURES / "font-heading-krav.pdf"
def _extract(path: Path, **kwargs: object) -> str:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
return extract_text(path.name, path.read_bytes(), **kwargs) # type: ignore[arg-type]
# --- the condition ----------------------------------------------------------
def test_the_reserve_is_silent_where_the_document_numbers_itself() -> None:
"""The whole point: a document with its own chapter run keeps its own run."""
assert heading_reserve_applies(_extract(NUMBERED_PDF), outline_run=3) is False
def test_the_reserve_fires_where_the_outline_gate_admits_nothing() -> None:
assert heading_reserve_applies(_extract(FONT_PDF), outline_run=3) is True
def test_a_run_shorter_than_the_minimum_is_not_a_run() -> None:
"""The condition reads the gate as CONFIGURED, not a fixed grammar.
The same three-chapter document is a run at 3 and not one at 4, so raising
the arm's minimum widens the reserve rather than leaving it behind.
"""
text = _extract(NUMBERED_PDF)
assert heading_reserve_applies(text, outline_run=4) is True
def test_with_arm_d_off_the_reserve_fires_everywhere() -> None:
"""`--outline-run 0` admits no run at all, so the reserve is unconditional.
Stated rather than left to the reader: that configuration IS round 4's
"font instead of Arm D", which measured 1 of 8. A caller combining the two
flags gets that, and gets it deliberately.
"""
assert heading_reserve_applies(_extract(NUMBERED_PDF), outline_run=0) is True
# --- the effect on a plan ---------------------------------------------------
def _entries(text: str) -> list[str]:
plan = build_plan(
Path("x.pdf"),
text,
b"",
okf_type="reference",
proposed_at="1970-01-01T00:00:00Z",
outline_run=3,
table_grid=True,
unit_fold=True,
)
return [str(entry["title"]) for entry in plan["entries"]]
def test_the_numbered_document_is_segmented_by_its_own_numbering() -> None:
"""The reserve buys nothing here, and that is the guarantee, not a defect.
Measured rather than assumed, and the measurement is stronger than the
guarantee: at the build default this fixture lands on the same three units
from BOTH renderings, because the outline arm's run wins the document and
the fold puts the font-inferred `Forord` inside the first unit rather than
beside it. So the reserve's silence here has two independent reasons, and
only one of them -- the condition -- is what this flag controls. The
condition itself is pinned above, on the extraction, where it is decided.
"""
assert _entries(_extract(NUMBERED_PDF)) == ["Generelle krav", "Merking", "Vedlegg"]
assert _entries(_extract(NUMBERED_PDF, pdf_headings=True)) == [
"Generelle krav",
"Merking",
"Vedlegg",
]
# --- the flag ---------------------------------------------------------------
def test_the_build_default_does_not_reach_the_reserve() -> None:
"""Every byte-pinned golden depends on this, so it is a test and not a note."""
assert cli.DEFAULT_PDF_HEADINGS is False
assert cli.DEFAULT_PDF_HEADINGS_RESERVE is False
def test_the_reserve_is_reachable_from_the_build_command(tmp_path: Path) -> None:
"""`--pdf-headings font-reserve` is a third value on the same axis.
One axis, three values, because `none`, `font` and `font-reserve` are three
answers to one question -- how a PDF's headings are recovered -- and a
separate boolean flag would let a caller ask for two of them at once.
"""
base = [
"build",
str(tmp_path),
"--bundle",
str(tmp_path / "b"),
"--bundle-id",
"x",
"--okf-version",
"0.2",
]
assert cli.parse_args([*base, "--pdf-headings", "font-reserve"]).pdf_headings == "font-reserve"
assert cli.parse_args([*base, "--pdf-headings", "font"]).pdf_headings == "font"
assert cli.parse_args(base).pdf_headings == "none"
def test_the_reserve_reaches_the_proposer_and_the_door_alike(tmp_path: Path) -> None:
"""Propose and apply must choose the SAME rendering, or nothing applies.
A plan indexes the exact string it was proposed against (`text_sha256`), so
a reserve that fired on one side and not the other would turn every
document it touches into a coded rejection. The build is the test: the
unnumbered PDF must come back segmented by its typography, and the numbered
one by its own chapters, in ONE run with ONE flag.
"""
inbox = tmp_path / "in"
inbox.mkdir()
(inbox / "numbered.pdf").write_bytes(NUMBERED_PDF.read_bytes())
(inbox / "unnumbered.pdf").write_bytes(FONT_PDF.read_bytes())
with warnings.catch_warnings():
warnings.simplefilter("ignore")
report = cli.build(
inbox,
tmp_path / "bundle",
bundle_id="reserve-fixture",
okf_version="0.2",
pdf_headings_reserve=True,
)
assert report.rejected == 0
names = sorted(path.name for path in (tmp_path / "bundle").rglob("*.md"))
# The unnumbered PDF: the default leaves it one flat concept (round 4 pins
# that), and the reserve gives it the unit its typography names.
assert "generelle-tekniske-krav.md" in names, names
# The numbered PDF: its own chapters, and no `Forord` the reserve added.
assert "generelle-krav.md" in names, names
assert "merking.md" in names, names
assert "vedlegg.md" in names, names
assert "forord.md" not in names, names