open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,cli): typography as a reserve, and the two of our own numbers it took to measure it

Browse source 
K3 round 5. Three questions, three answers, and two of them correct a figure
this repository published.

RETRIEVAL FIRST, because it could have reversed a default. hit@8 over the six
questions on BOTH K2 bundles -- Arm B at 629 concepts and the shipped default
at 492 -- is 5 of 6 with ranks 1,1,1,1,1,- on each, so 0 of 6 rows lost. The
order's rule reverses `--unit-fold` at >= 2 of 6; it does not fire, and the
default stands. The gold sets shrink (49->26, 20->17, 43->36, 11->18) while
every rank holds at 1, which is the fold merging concepts rather than removing
a document from the top.

TWO PUBLISHED NUMBERS CORRECTED, both ours. The S7 candidate ranks 96 of 629
and 159 of 492 were measured with the cost vocabulary passed to
`concept_scores` and NOT to `document_scores`, while `build_payload` passes it
to both; scored the way the shipped payload scores it, the same concept is 10
of 629 and 19 of 492. And round 4 attributed its non-delivery to the default
move -- measured here, it is not delivered on the Arm B bundle either, for a
different reason (knapsack eviction at 68 046 bytes of a 120 000 budget, versus
`below_k`). That column had been inherited from round 3's own build, never
re-measured.

`--pdf-headings font-reserve`, OFF, and the hypothesis behind it is falsified
by its own condition rather than by a score: position 7, the one position the
flag exists for, has THREE outline runs, so the reserve is silent there at
every minimum. It changes 0 of 12 cells on the reference and reaches 4 of 39
corpus documents, none of them rated. Built anyway because it was authorised
and because the condition is now measured rather than assumed. The predicate
lives in one place (`propose.heading_reserve_applies`) and the door receives it
as a callable, like `gate`: a plan indexes the exact string it was proposed
against, so a reserve firing on one side only would make every document it
touches a coded rejection.

The `xlsx` re-reading is confirmed on the artifact -- 11 `rule:sheet-section`
units plus 1 `rule:table-block` ingress -- but the number alone makes the cell
worse (distance 1 -> 2), because the criterion counts that ingress as a table
that should have been merged. A hit needs both halves ratified, and the
reference is the operator's.

`--sheet-section-rows` as a default: three cells better and none worse on the
twelve positions, but the K2 control moves -- row 1's gold document splits 1 ->
12 concepts and its best concept ranks 2 instead of 1. Condition not met,
default not moved.

Default build byte-identical before and after (`diff -r`, 30 md files).
Suite 1441 -> 1449; three of the eight were red first.

Report: docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 00:25:51 +02:00
commit b01492b7f5
10 changed files with 743 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

388
docs/2026-09-08-k3-runde5-hitat8-og-skriftakse.md Normal file
View file

@ -0,0 +1,388 @@
# K3 round 5: hit@8 on both K2 bundles, and typography as a reserve
**Date:** 2026-09-08 · **HEAD at start:** `53d5c74` · **Suite at start:** 1441
Round 4 shipped two PDF readers and left three questions behind it. This round
answers the retrieval one first, because it is the one that could have reversed
a default, and then measures the segmentation candidate the round-4 report
pointed at. Two of the three answers are negative, and the negative ones are the
informative ones.
## 0. What this round found, in one paragraph
The default move did **not** cost anything on the retrieval side: hit@8 is 5 of
6 on both K2 bundles, with the same five ranks at 1 and the same row missing on
both. The decision rule therefore does not fire and `--unit-fold` stays in the
default. The font **reserve** — typography read only where the outline arm finds
no run — was built, is off, and must stay off: it cannot reach the position it
was designed for, because that position numbers its own chapters. And the
`xlsx` re-reading works, but only if BOTH halves of it move together; the number
alone makes the cell worse.
Two published figures are corrected here rather than left standing. Both are
this repository's own.
---
## 1. Reproduction of the starting state
Every figure below was measured on `53d5c74` with a clean tree.
| what | expected | measured |
|---|---|---|
| `git rev-parse HEAD` | `53d5c74` | `53d5c74c96b4316dcb3380227d72be908e1be270` |
| `git status --short` | empty | empty |
| `pytest -q` | 1441 | **1441 passed** |
| `ruff check src tests tools` | clean | `All checks passed!` |
| `mypy src` | clean | `Success: no issues found in 21 source files` |
| `okf build <folder>`, no flags | 30 md | **30**, exit 0, 0/5 rejected |
| same, `--keep-table-heading --sheet-section-rows --drop-wrapped-outline` | 52 md | **52**, exit 0, 0/5 rejected |
| unpushed commits | 4 | **4** (`47f6619`, `efa92ce`, `f6fea13`, `53d5c74`) |
| tag `v0.6.0` | on `f6fea13` | **on `f6fea13`**, one commit behind HEAD |
Both cached bundles were counted before use rather than trusted: the Arm B tree
holds **1108** md files and round 4's default artifact **944**, so neither was
rebuilt.
---
## 2. hit@8 on both K2 bundles
One instrument (`tools/okf_consume_measure.py`), one gold set, one `--k 8`, run
against both bundles. The gold set is local-only and neither the questions nor
the gold documents appear here; the row number carries the identity, as in
`docs/2026-09-07-okf-konsumskill-maaling.md`.
**Bundle 1 (Arm B):** 629 concepts, 1108 files, every arm flag off.
**Bundle 2 (default):** 492 concepts, 944 files, digest `bdefa679957edd7d…`,
default flags and default epoch stamp.
Both bundles are the same 39-document readable set, so `documents` is 39 in
both and the chance baselines are directly comparable.
| row | gold concepts (B) | hit (B) | rank (B) | considered (B) | spent (B) | chance (B) | gold concepts (D) | hit (D) | rank (D) | considered (D) | spent (D) | chance (D) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | yes | **1** | 629 | 76 897 | 0.013 | 1 | yes | **1** | 492 | 43 288 | 0.016 |
| 2 | 49 | yes | **1** | 629 | 21 403 | 0.479 | 26 | yes | **1** | 492 | 18 458 | 0.354 |
| 3 | 20 | yes | **1** | 629 | 26 282 | 0.229 | 17 | yes | **1** | 492 | 28 347 | 0.247 |
| 4 | 43 | yes | **1** | 629 | 19 818 | 0.434 | 36 | yes | **1** | 492 | 21 125 | 0.458 |
| 5 | 11 | yes | **1** | 629 | 22 459 | 0.132 | 18 | yes | **1** | 492 | 17 866 | 0.259 |
| 6 | 5 | **no** | – | 629 | 20 282 | 0.062 | 5 | **no** | – | 492 | 33 113 | 0.079 |
| **total** | | **5 of 6** | | | | 1.35 expected by chance | | **5 of 6** | | | | 1.41 expected by chance |
**0 of 6 rows lost.** The decision rule in the order fires at `>= 2 of 6`, so it
does not fire: `--unit-fold` stays in the default, and no `DEFAULT_UNIT_FOLD`
was touched. This reproduces the PM measurement of 21:20Z exactly, including
the ranks.
**The gold sets change size while the rank does not.** 49→26, 20→17, 43→36 and
11→18 concepts between the two bundles, and every one of those rows still ranks
1. That is the fold merging concepts rather than removing a document from the
top — which is the property the fold was defended on, measured here on the
retrieval side for the first time.
Both known-negative controls behave identically on both bundles: 2 of 6 tokens
reaching nothing on the planned negative, 4 of 6 on the lexically disjoint one,
8 excerpts delivered in each case. The negatives are a control on the question,
not on the ranker, and they did not move.
### 2.1 The S7 control row, and a correction to how it was measured
A separate question with a separate answer, measured with
`--cost-vocabulary --k 12` on both bundles:
| | Arm B (629) | default (492) |
|---|---|---|
| priced concept present | yes | yes |
| **candidate rank (whole ranking)** | **10 of 629** | **19 of 492** |
| rank among concepts that survive the gate | 10 of 280 | 19 of 280 |
| delivered at `--k 12` | **no** | **no** |
| why not | `over_budget_after_knapsack` — inside the shortlist, evicted; the excerpt weighs **68 046** of a 120 000 budget | `below_k` — outside the shortlist at k = 12 |
| excerpts delivered | 11 | 12 |
| budget spent | 84 953 | 51 810 |
**Two published numbers are corrected by this row.**
**First, the ranks.** Round 3 and round 4 published this concept at *96 of 629*
and *159 of 492*. Both are reproducible and both name a mixed configuration:
the measuring script (`/tmp/k3r4/rank4.py`) passes `cost_vocabulary=True` to
`concept_scores` but calls `document_scores(root, question)` with the vocabulary
left off, while `build_payload` passes it to both. Scored the way the shipped
payload scores it, the same concept is **10 of 629** and **19 of 492**. The
published figures are not wrong about their own arithmetic; they are answers
about a ranking the library never emits.
**Second, the attribution.** Round 4 wrote that the non-delivery "is the default
move of `47f6619`/`f6fea13` showing up on the retrieval side for the first
time". Measured here on the Arm B bundle with today's code, the concept is **not
delivered there either**. The default move is not the cause; the two bundles
fail to deliver it for two *different* reasons, and only one of them is about
bundle size. The knapsack eviction on Arm B is the case `--reserve-top-rank`
was built for and cannot fix, because the reservation is for rank one and this
is rank ten — which is exactly what the flag's own documentation already says.
What round 4 could not have known is that the Arm B column was never measured;
it was carried over from round 3's own build. Re-running it was the whole
correction.
---
## 3. The font reserve
Round 4 measured the font reader two ways and shipped neither: on TOP of Arm D
it took `pdf` from 2 of 8 to 0 of 8, and INSTEAD of Arm D (`--outline-run 0`) it
scored 1 of 8 while landing position 7 on 30 = its reference count. The
hypothesis this round was asked to test: use typography only where Arm D's
outline gate admits no run, and position 7 is bought without paying positions 2
and 8.
**The hypothesis is falsified, and it is falsified by the condition itself
rather than by a score.**
### 3.1 Where the reserve can fire, on the 12-position reference
Measured before any code was written, on the cached round-3 rendering:
| position | type | outline runs at minimum 3 | reserve |
|---|---|---|---|
| 0 | pdf | 0 | **fires** |
| 1 | pdf | 2 (last run 7 long) | silent |
| 2 | pdf | 2 (last run 9) | silent |
| 3 | xlsx | 0 | fires (no effect: the font reader is PDF-only) |
| 4 | pdf | 1 (last run 5) | silent |
| 5 | docx | 0 | fires (no effect: PDF-only) |
| 6 | pdf | 1 (last run 3) | silent |
| **7** | **pdf** | **3 (last run 10)** | **silent** |
| 8 | pdf | 2 (last run 8) | silent |
| 9 | pdf | 3 (last run 6) | silent |
| 10 | docx | 0 | fires (no effect: PDF-only) |
| 11 | docx | 0 | fires (no effect: PDF-only) |
Position 7 — the one position the flag exists for — has **three** outline runs,
the last of them ten entries long. No reading of "no run at all" reaches it: it
is a run at minimum 3, at minimum 1, and at every minimum up to 10. The 1 of 8
that round 4 measured under `--outline-run 0` came from *removing* Arm D on that
document, not from adding a second source where there was none.
### 3.2 What it scores
| pos | type | ref \|F\| | default | **+ reserve** | + font everywhere |
|---|---|---|---|---|---|
| 0 | pdf | undefined | for grovt (U1) | for grovt (U1) | for grovt (U1) |
| 1 | pdf | 15 | for fint (U17, d2) | for fint (U17, d2) | for grovt (U5, d10) |
| 2 | pdf | 9 | **treffer** (d0) | **treffer** (d0) | for fint (U12, d3) |
| 3 | xlsx | 11 | for grovt (U0, S1, d12) | for grovt (U0, S1, d12) | for grovt (U0, S1, d12) |
| 4 | pdf | 1 | for fint (U4, d3) | for fint (U4, d3) | for fint (U11, d10) |
| 5 | docx | 3 | **treffer** (d0) | **treffer** (d0) | **treffer** (d0) |
| 6 | pdf | 6 | for fint (U7, d1) | for fint (U7, d1) | for fint (U11, d5) |
| 7 | pdf | 30 | for fint (U33, d3) | for fint (U33, d3) | for fint (U33, d3) |
| 8 | pdf | 8 | **treffer** (d0) | **treffer** (d0) | for fint (U16, d8) |
| 9 | pdf | 4 | for fint (U10, d6) | for fint (U10, d6) | for fint (U12, d8) |
| 10 | docx | 2 | **treffer** (d0) | **treffer** (d0) | **treffer** (d0) |
| 11 | docx | 1 | **treffer** (d0) | **treffer** (d0) | **treffer** (d0) |
| | | | **5 of 12** · pdf 2 of 8 · docx 3 of 3 · xlsx 0 of 1 | **5 of 12**, identical cell for cell | 4 of 12 · pdf 1 of 8 |
**The reserve changes not one cell of the twelve.** It fires on five positions
and alters none of them: position 0's glyphs carry no ToUnicode mapping, so its
font rendering is byte-identical to its plain one, and the other four are office
documents the PDF reader never touches. The default column reproduces round 4
exactly (pdf 2 of 8, docx 3 of 3, xlsx 0 of 1), which is what makes the
instrument calibrated rather than merely re-run.
The acceptance criterion the order set — *position 7 hits 30 without losing 2, 6
and 8* — is **unreachable by construction**, not merely unmet. And position 7's
`(c)` in the reference is `I`, so the criterion's literal gate forbids a hit
there in any case; both readings stood in round 4 § 3 and both stand here.
### 3.3 What the reserve reaches on the whole corpus
The twelve positions are a sample of 43. Over the full cached corpus:
| | count |
|---|---|
| readable documents | 39 |
| of which pdf | 32 |
| pdf admitting no outline run at minimum 3 | **10** |
| of those, whose font rendering differs at all | **4** |
| of those, whose plan changes | **4** (1→4, 1→2, 0→2 and 5→6 entries) |
So the flag is not inert: it acts on **4 of 39** readable documents. But **none
of those four is in the rated sample**, so there is no reference count for any
of them, and this round cannot say whether the four changes are improvements.
A flag with a reach and no measurable quality signal is exactly a flag that
stays off.
### 3.4 What was built anyway, and why
`--pdf-headings font-reserve`: a third value on the option round 4 introduced,
so `none`, `font` and `font-reserve` are three answers to one question and no
caller can ask for two at once.
The condition lives in one function, `propose.heading_reserve_applies(text, *,
outline_run)`, and the proposer and the door both call it. That is not tidiness:
a plan indexes the exact string it was proposed against (`text_sha256`), so a
reserve that fired on one side and not the other would turn every document it
touches into a coded rejection. The door receives it as a **predicate**
(`heading_reserve: Callable[[str], bool] | None`), the same shape `gate` already
has, because the condition is the proposer's outline grammar and Door B does not
own that grammar — a direct import would point the dependency against the
layering.
The reserve reads the gate **as configured**. At `--outline-run 0` the gate
admits nothing at all, so the reserve becomes unconditional; that combination is
round 4's "font instead of Arm D" at 1 of 8, and a caller reaching it does so
deliberately. This is stated in the code and pinned by a test.
**Control:** the default build of the operator's folder is **byte-identical**
before and after the change (`diff -r`, 30 md files either way), and the suite
went 1441 → **1449** with eight new tests, three of which were red first for
three different reasons (missing predicate, wrong expected segmentation, wrong
report field).
---
## 4. The `xlsx` reference count, re-read as 12
The operator's re-reading is that the priced worksheet holds **11 numbered cost
groups plus one leading block of real prose that has to live somewhere**, so
11 + 1 = 12. Measured on the extraction, under `--sheet-section-rows`:
| unit | rule | count |
|---|---|---|
| the leading block | `rule:table-block` | 1 |
| the cost groups | `rule:sheet-section` | 11 |
| **total** | | **12** |
The rule produces exactly 12 entries, and the split is 11 + 1 in the shape the
re-reading describes. **The re-reading is confirmed on the artifact.**
**But the number alone does not deliver the hit, and this is the finding.** The
criterion as written treats a table-derived concept under `b = M` as *S* — a
table that should have been merged into a heading — not as a unit. So:
| reading of the reference | U | T | S | distance | label |
|---|---|---|---|---|---|
| \|F\| = 11, leading block counted as S (rounds 3 and 4) | 11 | 0 | 1 | **1** | for fint |
| \|F\| = 12, leading block still counted as S | 11 | 0 | 1 | **2** | for grovt |
| \|F\| = 12, leading block counted as the 12th unit | **12** | 0 | 0 | **0** | **treffer** |
Changing the number without changing what the criterion counts makes the cell
**worse**, from distance 1 to distance 2. `xlsx` reaches **1 of 1** only when
both halves move together.
**This is the operator's reference, not mine.** What is measured here is that
the artifact contains 11 + 1 units in exactly the shape the re-reading names,
and that a hit requires ratifying the second half too. Rounds 3 and 4 both used
the reading "11" and both reported distance 1, so nothing published so far
depends on the outcome either way.
---
## 5. `--sheet-section-rows` as a default: what the 12 positions say
Because § 4 raised it, the D1/D3 configuration was scored against the shipped
default on all twelve positions:
| pos | default | + `--keep-table-heading --sheet-section-rows --drop-wrapped-outline` |
|---|---|---|
| 3 (xlsx) | for grovt, d12 | **for fint, d1** (treffer, d0 under the re-read reference) |
| 4 (pdf) | for fint, d3 | **treffer, d0** |
| 9 (pdf) | for fint, d6 | **for fint, d2** |
| all other nine | — | identical, cell for cell |
**Three positions improve, none regresses**, taking `pdf` from 2 of 8 to 3 of 8
and leaving `docx` at 3 of 3. That is the strongest segmentation result this
round has, and it is **not acted on here**: the order makes a default move
conditional on the K2 control as well, and it is a fourth change to every
consumer's bytes in two days. The K2 control was run for exactly that
reason and it does not hold (§ 7), so the default stays where it is.
---
## 6. What still stands, unchanged
- **The prefix over-matches the other way** (round 3 finding, untouched since):
a compound word matches a four-character prefix, df 159 of 270. It belongs to
the rarity weight and was not this round's work.
- **`pdf` position 1** (17 against a reference of 15) is now refused on four
axes: number, outline, typography, and the reserve — the reserve is silent
there, because that document has two outline runs.
- **vegnormal-okf FUNN 1 and FUNN 2** are still not investigated in this repo.
- **`tests/test_okf_consume.py:940` pins hit@8 against the Arm B bundle alone.**
The default bundle is pinned nowhere. That is a gap, and it is named in
STATE's next block rather than closed here: O4/O5 fence that file, and this
round only ran the instrument.
---
## 7. The K2 control for the D1/D3 configuration: it moves, so nothing moves
The whole 43-document corpus rebuilt with
`--keep-table-heading --sheet-section-rows --drop-wrapped-outline` on top of the
shipped default. Exit 0, 2305 s wall (773 s inside the door), 39/43 persisted,
4/43 coded rejections -- the same conservation identity as the default build.
| | default | + D1/D3 |
|---|---|---|
| concepts reachable from the index tree | 492 | **491** |
| md files | 944 | **942** |
| hit@8 over the six questions | **5 of 6** | **5 of 6** |
| expected by chance | 1.41 | 1.57 |
| row | gold concepts (default) | rank | gold concepts (D1/D3) | rank |
|---|---|---|---|---|
| 1 | 1 | **1** | **12** | **2** |
| 2 | 26 | 1 | 26 | 1 |
| 3 | 17 | 1 | 17 | 1 |
| 4 | 36 | 1 | 36 | 1 |
| 5 | 18 | 1 | 17 | 1 |
| 6 | 5 | -- | 5 | -- |
**The count holds and the control does not.** Row 1's gold document is split
from one concept into twelve by the sheet-section rule, and its best concept
then ranks **2 instead of 1**. It is still a hit at 8, and it is still a change
to a published number on a corpus the operator reads.
The order's condition for a default move is "no regression on the 43-document
corpus **or** the K2 control". The 12-position reference improves three cells
and regresses none (§ 5); the K2 control regresses one row's rank. **The
condition is therefore not met and the default is not moved.** The flags stay
reachable and off, exactly as they were.
That is worth stating plainly rather than as a near miss: the segmentation side
and the retrieval side disagreed about this configuration, and the disagreement
is real -- splitting a worksheet into its rows is what the reference asks for
and what costs the ranker a rank position, because twelve small concepts each
carry less of the question than one whole sheet did.
---
## 8. Honesty limits
- **n = 6 questions** on the retrieval side, and **n = 12 positions** on the
segmentation side, of which 8 pdf, 3 docx and 1 xlsx. Every per-type figure
in this report is a fraction of 8, 3 or 1. A "3 of 8" is three documents.
- **One rater, one reference.** The twelve-cell reference and the criterion that
scores it were authored by the same operator, and neither has an independent
second rating. Where the criterion and the reference disagree (§ 4), that is
visible only because the disagreement was arithmetic.
- **Both bundles are the same 43-document corpus**, so "hit@8 held on both" is a
statement about one corpus read two ways, not about two corpora.
- **"Delivered" is not "answered".** hit@8 says a gold document appeared among
eight excerpts. Nothing here measures whether a reader could answer the
question from those excerpts; the one measurement that ever tried
(`docs/2026-09-07-okf-konsumskill-maaling.md`) got the correct answer on 1 of
3 bundles.
- **The reserve's corpus reach (§ 3.3) was measured by simulation**, not by
running `okf build --pdf-headings font-reserve` over all 43 documents: the
two cached renderings were combined by the same predicate the shipped code
calls. The predicate is identical; the extraction path around it was exercised
only on the fixtures and on the 12-document sample.
- **The four documents the reserve changes have no reference count.** They are
outside the rated sample, so "4 of 39 change" is a reach, not a quality claim.
- **The corrected S7 ranks (§ 2.1) were measured on today's code.** Round 3's
own build is not available, so this report cannot separate "the instrument was
mixed" from "the code also moved" for the *delivery* outcome; it can and does
separate it for the *rank*, because the rank reproduces exactly under the
published instrument and changes only when the vocabulary is passed
consistently.