open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms

Browse source 
K3 round 2, per file type. Order 20260908T143513Z-6327528123-from-.claude, carrying two operator decisions taken beforehand: D1 the orphan-gate variant goes behind a flag, D2 the arms become reachable from `okf build`. No default moved. Report: docs/2026-09-08-k3-runde2-per-filtype.md.

THE REPRODUCTION HELD, all three numbers, before any edit: `okf build` on the five-document tender folder gives 31 markdown files with both PDFs flat and 5/5 merged; the tender PDF gives no boundary without a flag and 9 with `--outline-run 3` (reference 9); the price sheet gives 1 on HEAD. Both proposer runs had to go through `bash -c` -- zsh does not word-split an unquoted `$flags`, so a sweep hands `--outline-run 3` to argparse as one token and every row comes back exit 2.

D2 -- `cli.py:_propose_plans` called the proposer with no arm argument, so the build path ran Arm B while `tools/okf_propose_segments.py` could run D, E and F. It now passes `--outline-run`, `--table-grid`, `--unit-fold` and `--keep-table-heading` through unchanged. THE DEFAULT DOES NOT MOVE and that is measured, not asserted: same folder, no flags, before and after the change, digest 3af10770...8fbbe2 both times and `diff -rq` clean. The "before" bundle was built before the first edit, because the editable install reads src/ live. Red test on the PLANS and on titles rather than a count, with the same fixture and no flags as its control. Per-document table for B/D/E/F/F2 is in the README and the report; the tender PDF is 1 under the default and 9 under every arm above it, and the reference is 9.

D1 -- a sheet heading with a table opening under it has an empty body, so the orphan check drops it: the NAME survives (carried onto the table block), the LINE does not. `--keep-table-heading` lets the heading survive and absorb the table instead. Price sheet 1 -> 1 concepts, `source_offset` [34, 11048] -> [0, 11048], body now starting at the heading. ELEVEN IS NOT REACHABLE THIS WAY and the number says why: the sheet is one heading and one continuous pipe-table block, and the eleven cost groups are eleven ROWS inside it (lines 10-20 of 103). What is missing is a section-row rule inside a sheet -- the opposite of `--table-grid`. Corpus: the flag changes 2 of 39 documents, both `.xlsx`, under arms B, E and F alike; known-negative 0 of 32 `pdf` and 0 of 5 `docx`. With it off, Arm E over all 43 is byte-identical to session 109's tree (33 plans, 43 `.err`, 4 FAILED, diff exit 0, counts asserted first).

THE PDF REMAINDER, one at a time. Position 9: clause 1 read the list AFTER the orphan check, and a contents list without dot leaders is a run of bodiless headings, so all but the last entry were already gone and the run was one. The run is now measured on the pre-orphan list, predicate written once and read in both places. 11 -> 10. Position 7: the same clause required siblings, and a numbered report's contents list interleaves 1.1/1.1.1/2.1 -- its 34 entries are one block that the level condition cut into runs of 9, 1, 1, 1, 5, 2, 10, 2 and 3, so the short runs survived. The level condition is dropped; the run LENGTH, which is what the CONTENTS_RUN sweep bought, is unchanged. Measured outward: the relaxation changes 1 document of 39 and removes exactly the leftover line. 34 -> 33.

TWO REMAINDERS ARE DECLINED WITH NUMBERS RATHER THAN FIXED. Position 1: the three level-1 candidates are 3 of 3 `rule:outline`, same level, same grammar, and the operator keeps one of them by prose alone -- there is no property to read. Position 4: a title-length rule was measured on paper and falsified -- a real chapter is 56 characters and a real heading in a document the arms already score correct is 88, sitting between position 4's 86 and 91, so no threshold separates the classes. Position 0 stays an extraction failure.

ONE SHIPPED EXPECTATION MOVED and is stated rather than quietly updated: `Innhold 1` is now discarded with the contents list it heads. Its body is in no segment afterwards, which is a real cost on a fixture where that heading has one.

Nine new tests: five red before the implementation, four green by construction and named as such. Three mutations, three red, unmutated control green each time -- restoring the level condition, computing the run post-orphan, absorbing a table unconditionally. 1379 -> 1388 tests. ruff clean, mypy --strict clean on 17 files. K2 bundle untouched (1108 files, 9cd74519...). The K2 ranking control is NOT measured: no bundle was rebuilt with the flag, so the rank is a prediction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 18:21:25 +02:00
commit ff79cfa19b
7 changed files with 989 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

18
CLAUDE.md
View file

@ -258,11 +258,19 @@ and fixtures, never code.
recursively, and two same-named documents in different folders must not recursively, and two same-named documents in different folders must not
collide); `--ingested-at` and `--proposed-at` default to one shared epoch collide); `--ingested-at` and `--proposed-at` default to one shared epoch
constant rather than the clock, because a wall-clock default takes constant rather than the clock, because a wall-clock default takes
rebuild-equals-incremental away from anyone who omits them. Arm C, Arm D, rebuild-equals-incremental away from anyone who omits them.
Arm E and Arm F are off and not exposed here -- which is measured, not **Four segmentation rules are REACHABLE here and all four are OFF by
incidental: the build path calls the proposer with no arm flag at all, so a default** (2026-09-08): `--outline-run N` (Arm D), `--table-grid` (Arm E),
tender PDF that Arm D splits into nine concepts lands as one `--unit-fold` (Arm F) and `--keep-table-heading` (D1), each passed to the
(`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`, the `okf build` section). proposer unchanged. Until that day the build path called the proposer with no
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
as one -- a build path a full arm behind the proposer. Exposing them is not
the same decision as moving one: **which arm should ship as the default is
the operator's, and a no-flag build stays byte-identical** (measured by
digest before and after, and by Arm E over all 43 corpus documents). Arm C
(`--max-segment-chars`) stays unexposed: no reference has ever been measured
for its cap. Reports: `docs/2026-09-08-k3-arm-f-mot-enhetsarket.md` and
`docs/2026-09-08-k3-runde2-per-filtype.md`.
- Consume a bundle: `python3 tools/okf_consume.py <bundle> --question "<q>" - Consume a bundle: `python3 tools/okf_consume.py <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass** [--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction `docs/consumption-contract.md` § 1 defines, and the only reading direction

33
README.md
View file

@ -80,6 +80,39 @@ than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them. did not pass them.
### The segmentation flags
Four rules are reachable from `okf build` and **all four are off by default**;
a build with none of them is byte-identical to every bundle published here. Each
line below carries the number it was measured at, and nothing beyond it — none
of them is a recommendation, because which one should become the default has
not been decided.
| flag | what it does | measured |
|---|---|---|
| `--outline-run N` | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N` | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
| `--table-grid` | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
| `--unit-fold` | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match, against **2** for the default, no cell worse. Whole corpus: 679 → 485 proposed segments over 39 readable documents |
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
They compose, and the order above is the order they apply in. Measured on a
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
document:
| document | default | `--outline-run 3` | `+ --table-grid` | `+ --unit-fold` | `+ --keep-table-heading` |
|---|---|---|---|---|---|
| tender PDF, technical requirements | 1 (no boundary) | 9 | 9 | 9 | 9 |
| tender PDF, technical layout | 1 (no boundary) | 1 | 1 | 1 | 1 |
| price sheet `.xlsx` | 1 | 1 | 1 | 1 | 1 |
| experience list `.docx` | 21 | 21 | 6 | 3 | 3 |
| agreement `.docx` | 2 | 2 | 1 | 1 | 1 |
| markdown files in the bundle | 31 | 49 | 33 | 30 | 30 |
Every column merged 5 of 5 with 0 rejections. The reference for the first row
is 9, so the default is a full arm behind what the proposer can do on that
document — which is a statement about the default, not a licence to change it
here.
Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
three files no reader accepts), one three files no reader accepts), one
`okf build` invocation replacing the shell loop over `tools/` that produced the `okf build` invocation replacing the shell loop over `tools/` that produced the

388
docs/2026-09-08-k3-runde2-per-filtype.md Normal file
View file

@ -0,0 +1,388 @@
# K3 round 2, per file type: the build command, the spreadsheet, the PDF remainder
Second iteration of the per-file-type directive of 2026-09-08 13:05Z. Order
`20260908T143513Z-6327528123-from-.claude`, which carries two decisions already
taken by the operator and asks for neither to be widened: **D1** — the orphan
gate variant goes behind a flag, off by default; **D2** — the measurement arms
become reachable from `okf build`, with the default unchanged.
Everything below is measured on the same 43-document corpus and the same
12-document sample as `docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`, against the
same reference: the operator's filled unit worksheet, which is local-only and
gitignored. No document title or extracted text from it is reproduced here.
## Section 0: what was measured and what was not
**Measured.** The three reproduction numbers the order names, before any edit.
Four flags reaching the proposer from `okf build`, on the artifact. A
byte-identical no-flag bundle, by digest and by `diff -r`. Two rule adjustments,
each red-first, each with a known-negative and a mutation control. The whole
corpus under six configurations, from ONE cached extraction so every variant
reads the same bytes. The four PDF remainders, each decomposed to a named
mechanism.
**Not measured.** No default moved. No K2 bundle was rebuilt, so the K2 ranking
control (the priced concept at rank ≤ 12 with `--cost-vocabulary --k 12`) is
**not measured this round** — it is a property of a bundle built WITH the flag,
and no such bundle exists. The K2 bundle on disk is unchanged, which proves
nothing was overwritten and nothing more. Two of the four PDF remainders are
diagnosed and declined rather than fixed, with the numbers that decline them.
## 1. The reproduction, before any edit
| # | control | command | result |
|---|---|---|---|
| a | `okf build` on the five-document tender folder | `okf build ~/okf-test/dokumenter --bundle … --bundle-id … --okf-version 0.2` | **31** markdown files, both PDFs as one flat `inbox-*.md` each, merged **5/5**, 0 rejections |
| b | the tender PDF through the proposer | `tools/okf_propose_segments.py "<tender>.pdf" --out …` | **no boundary** (exit 1); with `--outline-run 3`: **9** entries |
| c | the price sheet through the proposer | same, on the `.xlsx` | **1** entry on HEAD, against 3 in the 2026-09-07 archive; the reference is **11** cost groups |
All three reproduce. (b) and (c) had to be run under `bash -c`: zsh does not
word-split an unquoted `$flags`, so a sweep loop hands `--outline-run 3` to
`argparse` as ONE token and every row comes back `exit 2`. That is the trap
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md` recorded, hit again.
## 2. D2: `okf build` reaches the arms, and the default does not move
`cli.py`'s `_propose_plans` called `propose.run` with no arm argument, so the
build path ran Arm B while `tools/okf_propose_segments.py` could run D, E and F.
It now passes `--outline-run`, `--table-grid`, `--unit-fold` and
`--keep-table-heading` through unchanged, same names and same semantics.
**The red test came first** and asserts on the PLANS, on titles rather than on a
count: a fold that kept the right number of concepts by discarding the wrong
ones passes a count assertion. Its control is the same fixture with no flags.
**The byte control.** Same folder, no flags, before and after the change:
```
BEFORE 3af107707b66090e7ab4e4372c5976ca2c448ac0d771c7e9a9d74746d08fbbe2
AFTER 3af107707b66090e7ab4e4372c5976ca2c448ac0d771c7e9a9d74746d08fbbe2
diff -rq: identical, 31 markdown files either way
```
The "before" bundle was built at the start of the session, from the editable
install, before the first edit — a fresh build afterwards would have read the
changed `src/` and compared a tree against itself.
### Concepts per document, per configuration
Five documents, 2 `pdf` / 2 `docx` / 1 `xlsx`. Every column merged 5 of 5 with
0 coded rejections.
| document | B (default) | D | E | F | F2 |
|---|---|---|---|---|---|
| tender PDF, technical requirements | no boundary → 1 | **9** | 9 | 9 | 9 |
| tender PDF, technical layout | no boundary → 1 | no boundary | no boundary | no boundary | no boundary |
| price sheet `.xlsx` | 1 | 1 | 1 | 1 | 1 |
| experience list `.docx` | 21 | 21 | 6 | **3** | 3 |
| agreement `.docx` | 2 | 2 | 1 | **1** | 1 |
| markdown files in the bundle | 31 | 49 | 33 | 30 | 30 |
`B` = no flags. `D` = `--outline-run 3`. `E` = `+ --table-grid`.
`F` = `+ --unit-fold`. `F2` = `+ --keep-table-heading`.
The reference for the first row is 9. **What the measurement says about moving
the default is therefore one sentence: on this folder the default is a full arm
behind the proposer on the one document with a reference.** It is five
documents, one folder, and the decision is the operator's.
## 3. D1: the spreadsheet, behind `--keep-table-heading`
**The mechanism, named.** `propose.py`'s orphan check drops a heading whose body
is empty, and a sheet heading with a pipe table opening two lines below it has
an empty body. The heading's NAME survives (it is carried onto the table block),
its LINE does not: the concept starts at the first table row.
**The variant.** When a heading is orphaned *only* because the very next mark is
a table block, the heading survives and the table is absorbed into its span
rather than emitted. The count does not move; the first byte does. Measured on
the price sheet:
| | default | `--keep-table-heading` |
|---|---|---|
| concepts | 1 | 1 |
| `source_offset` | `[34, 11048]` | `[0, 11048]` |
| first body line | `\| Prisskjema \| …` | `## Prissammenstilling {#sheet-1}` |
**The reference wants eleven, and eleven is not reachable this way.** The sheet
extracts to 103 lines: ONE heading (line 1) and one continuous pipe-table block
(lines 3–103). The eleven numbered cost groups are eleven ROWS inside that
block (`01`, `11+12`, `02` … `92`, lines 10–20). Reaching them needs a rule that
splits a table on its section rows — the exact opposite of `--table-grid`, which
exists to stop a table becoming one concept per row group. **What is missing is
a section-row rule inside a sheet, and no heading-and-table rule can supply it.**
### The corpus numbers for D1
From one cached extraction, so every configuration reads identical bytes.
39 of 43 documents are extractable (4 `FAILED`, unchanged).
| control | result |
|---|---|
| documents changed by `--keep-table-heading`, arm B | **2 of 39** — both `.xlsx` |
| documents changed, arm E | **2 of 39** — the same two |
| documents changed, arm F | **2 of 39** — the same two |
| **known-negative: `pdf` changed** | **0 of 32** |
| **known-negative: `docx` changed** | **0 of 5** |
| both spreadsheets in the corpus | **2 of 2** changed, `[20,1921] → [0,1921]` and `[34,11048] → [0,11048]` |
**With the flag off, nothing moves.** Arm E re-run over all 43 documents on the
round-2 code, against session 109's Arm E tree: 33 plans either way, 43 `.err`
either way, 4 `FAILED` either way, `diff -rq … -x '*.err' -x '_index.txt'` exit
`0`. Counts asserted before the diff, because a diff over two trees where every
document failed compares nothing and exits `0`.
**The K2 ranking control is not measured.** `56ae274` fixed a ranking regression
by restoring the concept's title, and this flag restores the title's LINE to the
body — which is the same direction, and a prediction, not a measurement. No K2
bundle was rebuilt with the flag, so nothing here says where the priced concept
ranks in one. It is round 3's.
## 4. The PDF remainder, one at a time
The reference row was `pdf` **2 of 8**. Each of the four is decomposed below.
Two get a rule adjustment behind the SAME flag as Arm F (`--unit-fold`); two are
declined, with the numbers that decline them.
### Position 9 — one leftover contents line. FIXED.
**Mechanism.** Clause 1 discards a RUN of at least `CONTENTS_RUN` page-numbered
headings. It reads the list `fold_units` is given, which is the list AFTER the
orphan check — and a contents list without dot leaders is precisely a run of
bodiless headings, so the orphan check deletes all of it but the last entry. The
run clause 1 looks for is gone before clause 1 runs; what is left is a run of
one, which is below the threshold by design.
**Adjustment (one).** The run is measured on the PRE-orphan list and carried
forward on the candidate. The predicate is written once and read in both places,
so there cannot be two definitions of a contents list.
**Result.** Position 9: 11 → **10**. Still `too fine` — the reference is 4, and
the remaining 6 are Arm D recovering a risk table's rows as an integer outline,
which the operator ruled out in free text ("radene er ikke overskrifter"). That
is Arm D's recovery, not the fold's, and `--unit-fold` adds no boundary it could
withdraw.
### Position 7 — a contents list that interleaves levels. FIXED.
**Mechanism.** The same clause also required the run's members to be SIBLINGS.
A numbered report's contents list interleaves `1.1`, `1.1.1`, `2.1`, so the
level condition cuts one block into many runs. Measured on the document: its 34
contents entries are one continuous block of page-numbered headings at levels 2
and 3, and the level condition split it into runs of 9, 1, 1, 1, 5, 2, 10, 2 and
3. The runs of one and two survived, and one of them
(`6.2.2 Tverrfaglig kontroll ....`) had a body, so it was emitted as a concept
while its neighbours were discarded.
**Adjustment (one).** The level condition is dropped. What still bounds the rule
is the run LENGTH — the number `CONTENTS_RUN`'s sweep bought — and that is
unchanged.
**Generalisation, measured outward.** Across all 39 readable documents the
relaxation changes **1 document**, and there it removes exactly that one line:
| | value |
|---|---|
| documents whose plan changes | **1 of 39** |
| entries removed | 1 (position 7: 34 → **33**) |
| other documents touched | 0 |
**Result.** Position 7: 34 → **33**. Still `too fine` against a reference of 30,
and the operator had already marked that document "none". The remaining 3 are
the heading rule proposing units the operator does not want; that is not the
fold's to withdraw either.
**One shipped expectation moved with it, and it is stated rather than quietly
updated.** In `test_a_contents_run_is_discarded_and_the_body_survives` the
level-1 heading `Innhold 1` used to survive, because it is not a sibling of the
three level-2 contents lines after it. It is now part of the run and is
discarded — which is what "innholdsfortegnelsen er ikke konsepter" says about a
contents heading. The cost is real and bounded: in that fixture the heading has
a body of its own, and a discarded candidate leaves its body in no segment.
### Position 1 — two recovered chapters. DECLINED, with numbers.
**Mechanism.** 17 concepts against a reference of 15. The three level-1
candidates are `Innledning` (§1), `Referanser` (§6) and `Vedlegg` (§7), and all
three are `rule:outline` — Arm D's recovery of the document's bare-integer
numbering. The operator's free text keeps ONE of them ("vedlegget er eget
oppslag") and the criterion's reference set keeps only that one.
**Why no rule.** **3 of 3** are the same rule, the same level, the same grammar
and the same shape; the worksheet showed the operator dotted headings only, so
the reference set contains `Vedlegg` because the operator named it in prose, not
because anything in the document distinguishes it. A rule that dropped two of
the three would have to read the title, and there is no property here to read.
One document is not a rule, and this one does not even offer a property.
### Position 4 — quoted regulation text. DECLINED, with numbers.
**Mechanism.** The reference is 1 (the whole note as one argument); Arm B gives
exactly that and every arm above it gives 4. The four are `rule:outline`
candidates recovered from numbered legal paragraphs — full sentences, truncated
at the line break, which is why the operator wrote "sitert forskriftstekst er
ikke overskrifter".
**The rule that was tried on paper and falsified.** A heading is a heading and a
sentence is not, so: discard an outline candidate whose TITLE is longer than
some threshold. Every outline title in the 12-document sample, sorted:
| class | lengths |
|---|---|
| real headings | 7, 7, 7, 7, 8, 8, 9, 9, 10, 10, 10, 10, 12, 12, 13, 14, 18, 18, 20, 24, 24, 24, 31, 45, **56**, **88** |
| position 4's sentences | 86, 91, 92, 100 |
**The classes overlap and no threshold separates them.** A real chapter at
position 9 is 56 characters (`Rutiner for behandling av endringer av betydning
for SHA`) and a real heading at position 2 — a document the arms already score
`correct` — is 88. A threshold at 90 leaves one of position 4's four and costs
nothing; a threshold low enough to take all four deletes a chapter at a document
the arms currently get right. Declined.
### Position 0 — an extraction failure. REPORTED, not repaired.
95.1 % `(cid:N)` glyphs. No segmentation rule can repair a document whose text
never arrived. Unchanged, and it is the one position where Arm F proposes
nothing at all because there is no candidate to fold.
## 5. Per file type, before and after
The 12-document sample, concepts per arm. `Arm F` is session 109's number;
`F2` is this round's code with all four flags on.
| pos | type | reference | Arm B | Arm D | Arm E | Arm F (109) | F2 |
|---|---|---|---|---|---|---|---|
| 0 | pdf | — | 0 | 0 | 0 | 0 | 0 |
| 1 | pdf | 15 | 20 | 23 | 23 | 17 | 17 |
| 2 | pdf | 9 | 0 | 9 | 9 | 9 | 9 |
| 3 | xlsx | 11 | 1 | 1 | 1 | 1 | 1 |
| 4 | pdf | 1 | 0 | 4 | 4 | 4 | 4 |
| 5 | docx | 3 | 21 | 21 | 6 | 3 | 3 |
| 6 | pdf | 6 | 6 | 7 | 7 | 7 | 7 |
| 7 | pdf | 30 | 45 | 48 | 48 | 34 | **33** |
| 8 | pdf | 8 | 0 | 8 | 8 | 8 | 8 |
| 9 | pdf | 4 | 5 | 11 | 11 | 11 | **10** |
| 10 | docx | 2 | 15 | 15 | 3 | 2 | 2 |
| 11 | docx | 1 | 2 | 2 | 1 | 1 | 1 |
**The label row does not move.** Using session 109's criterion under its
dominance gate — which is the reading that survived, because column (c) is
relative and (a)/(b) are absolute:
| type | n | Arm B | Arm D | Arm E | Arm F | **F2** |
|---|---|---|---|---|---|---|
| `pdf` | 8 | 2 | 2 | 2 | 2 | **2** |
| `docx` | 3 | 0 | 0 | 1 | 3 | **3** |
| `xlsx` | 1 | 0 | 0 | 0 | 0 | **0** |
**Two cells moved closer and neither crossed.** Position 7 loses a concept the
reference does not want and is still 3 too many; position 9 loses one and is
still 6 too many. **`pdf` is 2 of 8 for the third round running**, and this
round says why with a mechanism per position rather than with one number:
two of the four are fixed as far as the fold can reach, and two are declined
on measurements that are written down above.
**Nothing regressed.** No count moved away from its reference in any cell.
## 6. Whole corpus, 43 documents
| figure | Arm E | Arm F (109) | **F2** | denominator |
|---|---|---|---|---|
| plans written | 33 | 32 | **32** | 43 |
| proposed entries | 679 | 491 | **485** | — |
| `.err` files | 43 | 43 | **43** | 43 |
| `.err` recording `FAILED` | 4 | 4 | **4** | so extractable **39/43** |
| plans differing from Arm F | — | — | **6 of 32** | — |
The six: two `.xlsx` where only the span start moves (D1), three `.pdf` where a
leftover contents line goes (the pre-orphan run), and one `.pdf` where the
level relaxation removes one more (position 7). Cross-checked: the entry count
of every one of the 39 written plans equals the count computed from the cached
extraction, **39 of 39**.
`K2-bundle-20260903` is unchanged: 1108 files,
`9cd745194346cda0c70eab9c7136fa44506203bbe85bc17d7eff2766c6e9b4d1`. That is a
statement that nothing overwrote it, not a rebuild.
## 7. Tests
Nine new tests, plus one shipped expectation moved and stated. **Five were red
before the implementation** (the bodiless run, the level-crossing run, the two
halves of D1, and the arms reaching `okf build`). **Four were green by
construction and are named as such rather than counted as evidence**: two
known-negatives (a single page-numbered heading survives; a heading with its own
body absorbs nothing — the latter was red only as a `TypeError` before the
parameter existed, and its value is the mutation below), a default control, and
a byte control. `--keep-table-heading` taking no argument is green either way,
because `argparse` rejects an unknown flag with the same exit code; it is a
guard, not a measurement.
Three mutations, three red, with the unmutated control green each time:
| mutation | expected to kill | result |
|---|---|---|
| restore the same-level condition in the contents run | the level test | **red** (2 tests) |
| compute the contents run after the orphan check | the bodiless-run test | **red** (3 tests) |
| absorb a table unconditionally | the D1 known-negative | **red** (1 test) |
| (control) unmutated | — | green, 93/93 |
`1379 → 1388` tests. `ruff check src tests tools` clean, `ruff format --check`
clean, `mypy --strict src/` clean on 17 files.
## Reproducing
```
# 1. The three numbers, in bash -- zsh does not split an unquoted $flags.
okf build ~/okf-test/dokumenter --bundle /tmp/b --bundle-id r --okf-version 0.2
find /tmp/b -name '*.md' | wc -l # 31
python3 tools/okf_propose_segments.py "<tender>.pdf" --out /tmp/t.json
python3 tools/okf_propose_segments.py "<tender>.pdf" --out /tmp/t.json --outline-run 3
# 2. The corpus, once, cached -- every variant then reads identical bytes.
# Ascending chunks, each a foreground call under 600 s.
C=~/corpora/okf-telling-20260829
for lo_hi in "1 12" "13 18" "19 26" "27 43"; do
set -- $lo_hi
arm_run "$C/K2-plans-r2-armFoff-20260908" 2026-09-07T00:00:00Z "$1" "$2" \
--outline-run 3 --table-grid
arm_run "$C/K2-plans-r2-armF2-20260908" 2026-09-08T00:00:00Z "$1" "$2" \
--outline-run 3 --table-grid --unit-fold --keep-table-heading
done
# 3. Counts BEFORE the diff.
ls "$C"/K2-plans-r2-armFoff-20260908/*.json | wc -l # 33
ls "$C"/K2-plans-r2-armF2-20260908/*.json | wc -l # 32
grep -l FAILED "$C"/K2-plans-r2-armF2-20260908/*.err | wc -l # 4 -> 39/43
# 4. The flag-off identity, and the flag-on delta.
diff -rq "$C/K2-plans-armE-check-20260908" "$C/K2-plans-r2-armFoff-20260908" \
-x '*.err' -x '_index.txt' # exit 0
diff -rq "$C/K2-plans-armF-20260908" "$C/K2-plans-r2-armF2-20260908" \
-x '*.err' -x '_index.txt' # 6 files
# 5. The consumer bundle, untouched.
cd "$C" && LC_ALL=C find K2-bundle-20260903 -type f | LC_ALL=C sort \
| xargs shasum -a 256 | shasum -a 256
# 1108 files, 9cd745194346cda0c70eab9c7136fa44506203bbe85bc17d7eff2766c6e9b4d1
```
`arm_run` is the function published in `docs/2026-09-07-k3-arm-e.md`
§ Reproducing, unchanged.
## Honesty limits
- **n = 12, one rater, one reference.** Per file type the denominators are 8, 3
and 1. A `1/1` is not a rate, and the `xlsx` row is that.
- **Two of the four PDF remainders are declined, not solved.** The declines rest
on measurements written above; they are still declines.
- **The two adjustments were derived from the two documents they move**, and
screened outward on 39. The outward screen is what bounds the fitting; it does
not remove it.
- **The rules were measured on plans, never on a bundle.** No bundle was built
with `--unit-fold` or `--keep-table-heading` beyond the five-document folder,
so nothing here says what these concepts look like after materialisation at
corpus scale.
- **The K2 ranking control is not measured**, for the reason § 3 gives.
- **No default moved, and no threshold is set.** Which arm should ship as the
default is the operator's decision and the measurement above is one folder,
one corpus and one worksheet wide.

100
src/llm_ingestion_okf/cli.py
View file

@ -38,11 +38,21 @@ the replay dated differently passes both explicitly.
## What it does not decide ## What it does not decide
Arm C (`--max-segment-chars`), Arm D (`--outline-run`) and Arm E **Every arm is OFF unless the caller asks, and this command does not move a
(`--table-grid`) are OFF here and are not exposed: they are measurement arms, default.** Which arm should ship as the default is the operator's decision and
all off by default by operator decision, and a build command is not where an is not taken here.
unadjudicated segmentation heuristic should become one flag away. `tools/`
still reaches them. The arms are, however, REACHABLE from here, and that is a change of 2026-09-08.
Until then `_propose_plans` called the proposer with no arm flag at all, so
`okf build` ran Arm B while `tools/okf_propose_segments.py` could run Arm D, E
and F -- a build path a full arm behind the proposer, reachable only by
retyping the loop the packaging removed. Measured on a five-document folder:
one tender PDF lands as ONE concept from the build path and as NINE with
`--outline-run 3`, and nine is what the operator's unit worksheet asks for.
Arm C (`--max-segment-chars`) stays unexposed: it is a character cap whose
value nobody has measured against a reference, so it has no number to offer a
caller.
""" """
from __future__ import annotations from __future__ import annotations
@ -71,7 +81,16 @@ DEFAULT_STAMP = "1970-01-01T00:00:00Z"
def _propose_plans( def _propose_plans(
inbox: Path, bundle: Path, plans_dir: Path, *, proposed_at: str, okf_type: str inbox: Path,
bundle: Path,
plans_dir: Path,
*,
proposed_at: str,
okf_type: str,
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> tuple[int, int, int]: ) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed). """Propose a plan per dropped file. Returns (written, nothing, failed).
@ -96,6 +115,10 @@ def _propose_plans(
okf_type=okf_type, okf_type=okf_type,
proposed_at=proposed_at, proposed_at=proposed_at,
path_prefix=relative.with_suffix("").as_posix(), path_prefix=relative.with_suffix("").as_posix(),
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
) )
except ProposerError as exc: except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr) print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -119,6 +142,10 @@ def build(
segments: bool = True, segments: bool = True,
plans_dir: Path | None = None, plans_dir: Path | None = None,
okf_type: str = "reference", okf_type: str = "reference",
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> CorpusReport: ) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing. """Folder in, bundle out. The whole command, minus argument parsing.
@ -157,7 +184,15 @@ def build(
target = plans_dir if plans_dir is not None else Path(scratch) target = plans_dir if plans_dir is not None else Path(scratch)
target.mkdir(parents=True, exist_ok=True) target.mkdir(parents=True, exist_ok=True)
written, nothing, failed = _propose_plans( written, nothing, failed = _propose_plans(
inbox, bundle, target, proposed_at=proposed_at, okf_type=okf_type inbox,
bundle,
target,
proposed_at=proposed_at,
okf_type=okf_type,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
) )
print( print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; " f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -257,6 +292,53 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"the run replayed" "the run replayed"
), ),
) )
build_parser.add_argument(
"--outline-run",
type=int,
default=0,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. 0 (the default) is OFF and "
"leaves the bundle byte-identical. Measured on a tender PDF whose "
"headings are bare integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
"--table-grid",
action="store_true",
help=(
"Arm E, passed to the proposer unchanged: a pandoc grid-table rule "
"line no longer closes an open table block, so one grid table is one "
"concept instead of one per row group. Absent (the default) is OFF. "
"Measured on the K3 sample: it changes a `.docx` experience list from "
"21 concepts to 6"
),
)
build_parser.add_argument(
"--unit-fold",
action="store_true",
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. Absent (the default) is OFF. Measured on the K3 "
"sample: 5 of 12 documents match the operator's unit worksheet, "
"against 2 for the shipped default, with no cell worse"
),
)
build_parser.add_argument(
"--keep-table-heading",
action="store_true",
help=(
"D1, passed to the proposer unchanged: keep a heading whose body is "
"empty only because a table opens under it, and absorb that table "
"into its span. Absent (the default) is OFF. Measured on a tender "
"price sheet: the concept count does not move (1 -> 1) and the "
"concept gains the heading line it was missing"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report") build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv) return parser.parse_args(argv)
@ -277,6 +359,10 @@ def main(argv: list[str] | None = None) -> int:
segments=args.segments == "on", segments=args.segments == "on",
plans_dir=args.plans_dir, plans_dir=args.plans_dir,
okf_type=args.okf_type, okf_type=args.okf_type,
outline_run=args.outline_run,
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
) )
except (IngestError, OSError, ValueError) as exc: except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr) print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

139
src/llm_ingestion_okf/propose.py
View file

@ -239,6 +239,15 @@ class Candidate:
#: a rule line -- a single-row grid table joins nothing and stays #: a rule line -- a single-row grid table joins nothing and stays
#: byte-identical to Arm D. #: byte-identical to Arm D.
grid: bool = False grid: bool = False
#: True when this candidate belongs to a run of page-numbered siblings that
#: was a contents list BEFORE the orphan check thinned it. Set only when
#: Arm F is on, because it exists only for Arm F's clause 1 to read: a
#: contents list without dot leaders is a run of bodiless headings, so the
#: orphan check deletes all but the last and the run clause 1 looks for is
#: gone by the time `fold_units` sees the list. Computed where the whole
#: pre-orphan list is still in hand, and nowhere else -- no candidate
#: carrying it survives clause 1, so it never reaches an artifact.
contents: bool = False
def _is_stop_word_only(title: str) -> bool: def _is_stop_word_only(title: str) -> bool:
@ -312,6 +321,7 @@ def find_candidates(
outline_run: int = 0, outline_run: int = 0,
table_grid: bool = False, table_grid: bool = False,
unit_fold: bool = False, unit_fold: bool = False,
keep_table_heading: bool = False,
) -> list[Candidate]: ) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order. """Every boundary the mechanical rules propose, in document order.
@ -338,6 +348,14 @@ def find_candidates(
table block, so one grid table proposes one candidate instead of one per row table block, so one grid table proposes one candidate instead of one per row
group. It only ever REMOVES marks, which is what keeps every surviving group. It only ever REMOVES marks, which is what keeps every surviving
candidate's `start` fixed and the orphan check monotone. candidate's `start` fixed and the orphan check monotone.
`keep_table_heading` is D1's gate and it is OFF at False. On, a heading
whose body is empty ONLY because a table block opens under it keeps that
table instead of being dropped: the table is absorbed into the heading's
span rather than emitted, so the concept starts at the heading line. The
count does not move -- one candidate either way -- and the first byte does.
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
""" """
lines = text.splitlines(keepends=True) lines = text.splitlines(keepends=True)
offsets: list[int] = [] offsets: list[int] = []
@ -479,8 +497,24 @@ def find_candidates(
# is still carried by a live candidate, so copying its title onto the table # is still carried by a live candidate, so copying its title onto the table
# as well would put one name on two concepts and rescue none. # as well would put one name on two concepts and rescue none.
orphaned_name: tuple[str, str | None] | None = None orphaned_name: tuple[str, str | None] | None = None
# D1. Which table blocks a heading ABSORBS, decided before the pass that
# consumes it: once a table is absorbed the heading's body is no longer
# empty, so the orphan check below stops firing on it by itself and no
# branch is needed there. Computed only when the caller asked, so every
# other arm's `marked` -> `candidates` mapping is untouched code.
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = _contents_run_positions(marked) if unit_fold else set()
for position_in_list, (_, candidate) in enumerate(marked): for position_in_list, (_, candidate) in enumerate(marked):
following = marked[position_in_list + 1 :] if position_in_list in absorbed:
continue
following = [
entry
for position, entry in enumerate(marked)
if position > position_in_list and position not in absorbed
]
end = offsets[following[0][0]] if following else end_of_text end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end] body = text[candidate.start : end]
# The orphan check: everything after the heading line itself. # The orphan check: everything after the heading line itself.
@ -503,11 +537,77 @@ def find_candidates(
start=candidate.start, start=candidate.start,
end=end, end=end,
grid=position_in_list in joined, grid=position_in_list in joined,
contents=position_in_list in contents_run,
) )
) )
return fold_units(candidates) if unit_fold else candidates return fold_units(candidates) if unit_fold else candidates
def _absorbed_tables(
text: str,
marked: list[tuple[int, Candidate]],
offsets: list[int],
end_of_text: int,
) -> set[int]:
"""D1: positions in `marked` of table blocks a bodiless heading keeps.
The condition is the orphan check's own, evaluated on the UNABSORBED
neighbour distance -- a heading is eligible only when the very next mark is
a table block and there is nothing between them but the heading line. That
is what keeps the variant from merging a section into a table it merely
contains: a heading with a paragraph of its own is not orphaned, so it
absorbs nothing.
"""
absorbed: set[int] = set()
for position, (_, candidate) in enumerate(marked):
if candidate.rule == RULE_TABLE_BLOCK or position + 1 >= len(marked):
continue
line_index, following = marked[position + 1]
if following.rule != RULE_TABLE_BLOCK:
continue
body = text[candidate.start : offsets[line_index]]
if body.splitlines()[1:] and "".join(body.splitlines()[1:]).strip():
continue
absorbed.add(position + 1)
return absorbed
def _contents_run_positions(marked: list[tuple[int, Candidate]]) -> set[int]:
"""Arm F clause 1, measured on the PRE-orphan list. Same predicate as there.
A run of at least `CONTENTS_RUN` consecutive page-numbered headings, table
blocks excluded and never a single line. Written once here and read by
`fold_units` through `Candidate.contents`, so the two cannot drift into two
definitions of what a contents list is.
LEVEL IS NOT PART OF THE PREDICATE, and that is the second round-2 change.
A numbered report's contents list interleaves `1`, `1.1`, `1.1.1`, `2`, so
requiring the members to be siblings breaks the run at every level change:
on K3 position 7 the level-2 entries formed runs long enough to discard and
`6.2.2 Tverrfaglig kontroll` did not, so one contents line was emitted as a
concept while its neighbours were not. What still bounds the rule is the run
LENGTH, which is what the `CONTENTS_RUN` sweep bought and is unchanged.
"""
inside: set[int] = set()
index = 0
while index < len(marked):
candidate = marked[index][1]
if candidate.rule == RULE_TABLE_BLOCK or not _TRAILING_PAGE_NUMBER.search(candidate.title):
index += 1
continue
end = index
while (
end < len(marked)
and marked[end][1].rule != RULE_TABLE_BLOCK
and _TRAILING_PAGE_NUMBER.search(marked[end][1].title)
):
end += 1
if end - index >= CONTENTS_RUN:
inside.update(range(index, end))
index = end
return inside
#: Arm F clause 1. How many CONSECUTIVE same-level page-numbered headings make #: Arm F clause 1. How many CONSECUTIVE same-level page-numbered headings make
#: a contents list. Three, and it is swept rather than guessed: at 1 and 2 the #: a contents list. Three, and it is swept rather than guessed: at 1 and 2 the
#: rule deletes body chapters (a heading like `... i henhold til TEK 17` ends #: rule deletes body chapters (a heading like `... i henhold til TEK 17` ends
@ -527,6 +627,9 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
`CONTENTS_RUN` consecutive same-level headings each ending in a page `CONTENTS_RUN` consecutive same-level headings each ending in a page
number is a contents list and is discarded. A run of siblings, never a number is a contents list and is discarded. A run of siblings, never a
single line: one body heading ending in a number is not a contents list. single line: one body heading ending in a number is not a contents list.
The run is measured on this list AND on the list before the orphan check
(`Candidate.contents`), because a contents list whose entries carry no
dot leaders is bodiless and reaches here as one surviving line.
2. "hvert h2-kapittel med sine h3" -- the unit level is the SHALLOWEST 2. "hvert h2-kapittel med sine h3" -- the unit level is the SHALLOWEST
heading level occurring more than once; anything deeper folds into the heading level occurring more than once; anything deeper folds into the
preceding candidate at or above that level, which EXTENDS the parent's preceding candidate at or above that level, which EXTENDS the parent's
@ -543,8 +646,13 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
if not candidates: if not candidates:
return candidates return candidates
# Clause 1. # Clause 1. Two inputs, one predicate. `contents` carries the run measured
drop: set[int] = set() # on the list BEFORE the orphan check thinned it -- a contents list with no
# dot leaders is a run of bodiless headings, so only its last entry reaches
# here and a run of one is below `CONTENTS_RUN`. The scan below still has
# work to do: a contents list WITH dot leaders keeps every entry, and that
# run exists only in this list.
drop: set[int] = {position for position, c in enumerate(candidates) if c.contents}
index = 0 index = 0
while index < len(candidates): while index < len(candidates):
candidate = candidates[index] candidate = candidates[index]
@ -760,13 +868,18 @@ def build_plan(
outline_run: int = 0, outline_run: int = 0,
table_grid: bool = False, table_grid: bool = False,
unit_fold: bool = False, unit_fold: bool = False,
keep_table_heading: bool = False,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated.""" """The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set() taken: set[str] = set()
extractor_id = source.suffix.lower().lstrip(".") or "none" extractor_id = source.suffix.lower().lstrip(".") or "none"
entries: list[dict[str, Any]] = [] entries: list[dict[str, Any]] = []
candidates = find_candidates( candidates = find_candidates(
text, outline_run=outline_run, table_grid=table_grid, unit_fold=unit_fold text,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
) )
for candidate in subdivide(text, candidates, max_segment_chars): for candidate in subdivide(text, candidates, max_segment_chars):
entries.append( entries.append(
@ -837,6 +950,7 @@ def run(
outline_run: int = 0, outline_run: int = 0,
table_grid: bool = False, table_grid: bool = False,
unit_fold: bool = False, unit_fold: bool = False,
keep_table_heading: bool = False,
) -> int: ) -> int:
if max_segment_chars < 0: if max_segment_chars < 0:
raise ProposerError( raise ProposerError(
@ -889,6 +1003,7 @@ def run(
outline_run=outline_run, outline_run=outline_run,
table_grid=table_grid, table_grid=table_grid,
unit_fold=unit_fold, unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
) )
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty # Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan # plan cannot be replayed -- `process_inbox` refuses one, because a plan
@ -997,6 +1112,21 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"either" "either"
), ),
) )
parser.add_argument(
"--keep-table-heading",
action="store_true",
help=(
"D1: keep a heading whose body is empty ONLY because a table block "
"opens under it, and absorb that table into the heading's span "
"instead of emitting it. Without this the orphan check drops the "
"heading, the table inherits its NAME, and the concept starts at "
"the first table row -- so the heading line is in no concept's "
"body. Measured on a spreadsheet: the concept count does not move "
"(1 -> 1), the first byte does. Absent (the default) is OFF and "
"leaves every artifact byte-identical. Its own flag rather than "
"part of an arm: the orphan check is reached by every file type"
),
)
parser.add_argument( parser.add_argument(
"--proposed-at", "--proposed-at",
default="1970-01-01T00:00:00Z", default="1970-01-01T00:00:00Z",
@ -1018,6 +1148,7 @@ def main(argv: list[str] | None = None) -> int:
outline_run=args.outline_run, outline_run=args.outline_run,
table_grid=args.table_grid, table_grid=args.table_grid,
unit_fold=args.unit_fold, unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
) )
except ProposerError as exc: except ProposerError as exc:
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr) print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)

112
tests/test_cli_build.py
View file

@ -469,3 +469,115 @@ def test_the_root_index_does_not_link_the_run_log(tmp_path: Path) -> None:
} }
assert concepts, "the fixture must produce concepts for this control to mean anything" assert concepts, "the fixture must produce concepts for this control to mean anything"
assert len(okf_consume.enumerate_concepts(bundle)) == len(concepts) assert len(okf_consume.enumerate_concepts(bundle)) == len(concepts)
# --- the arms are reachable, and still off ---------------------------------
#
# Round 2, 2026-09-08. `_propose_plans` called the proposer with no arm flag at
# all, so `okf build` ran Arm B while `tools/okf_propose_segments.py` could run
# Arm D, E and F. Measured on a five-document folder: one tender PDF landed as
# ONE concept from the build path and as NINE from the proposer with
# `--outline-run 3`, and nine is what the operator's unit worksheet asks for.
# The default is unchanged -- that is the operator's decision, not this
# layer's -- but the flags now exist here.
# Two headings the mechanical rules match at two levels, plus a table opening
# directly under a heading: enough for Arm F's clause 2 and D1 to have
# something to do, so "the flag reached the proposer" is measured on the
# ARTIFACT and not on the parser.
ARM_DOCUMENTS = {
"delta.md": (
"## 4 Grunnforhold\n\nGrunnen er morene over berg i hele omraadet.\n\n"
"### 4.1 Loesmasser\n\nLoesmassene er telefarlige og maa skiftes ut.\n"
),
"epsilon.md": ("## 5 Prissammenstilling\n\n| Post | Sum |\n|------|-----|\n| 01 | 100 |\n"),
}
def inbox_for_arms(root: Path) -> Path:
inbox = root / "arm-inbox"
inbox.mkdir(parents=True, exist_ok=True)
for name, body in ARM_DOCUMENTS.items():
(inbox / name).write_text(body, encoding="utf-8", newline="")
return inbox
def plan_titles(plans: Path) -> list[str]:
import json
titles: list[str] = []
for path in sorted(plans.glob("*.json")):
payload = json.loads(path.read_text(encoding="utf-8"))
titles.extend(str(entry["title"]) for entry in payload["entries"])
return titles
def test_the_arms_reach_the_proposer_from_the_build_command(tmp_path: Path) -> None:
"""The red test for round 2: the flags exist here and they change the plan.
Asserted on the PLANS, which is where an arm's effect is visible, and on
titles rather than a count -- a fold that kept the right number of concepts
by discarding the wrong ones would pass a count assertion.
`--unit-fold` folds `4.1 Loesmasser` into `4 Grunnforhold`;
`--keep-table-heading` keeps `5 Prissammenstilling` as the heading it is
instead of letting the table block carry the name.
"""
inbox = inbox_for_arms(tmp_path)
plans = tmp_path / "plans-armed"
assert (
build(
inbox,
tmp_path / "bundle-armed",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"3",
"--table-grid",
"--unit-fold",
"--keep-table-heading",
)
== 0
)
assert plan_titles(plans) == ["4 Grunnforhold", "5 Prissammenstilling"]
def test_the_build_default_is_unchanged_by_the_flags_existing(tmp_path: Path) -> None:
"""The control the test above rests on: same inbox, no flags, Arm B.
Both halves over the same fixture. Without this, a change that turned an
arm ON by default would leave the assertion above green and move every
bundle every consumer builds.
"""
inbox = inbox_for_arms(tmp_path)
plans = tmp_path / "plans-plain"
assert (
build(
inbox,
tmp_path / "bundle-plain",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
)
== 0
)
assert plan_titles(plans) == ["4 Grunnforhold", "4.1 Loesmasser", "5 Prissammenstilling"]
def test_a_bundle_built_with_no_flags_is_byte_identical_to_the_shipped_one(
tmp_path: Path,
) -> None:
"""The byte control, on the tree the other build tests use.
The proposer gained a parameter and the CLI gained four. A default that
moved by a byte would break rebuild-equals-incremental for every consumer
who never passes a flag, and a title assertion cannot see that.
"""
inbox = inbox_with_subdirectories(tmp_path)
reference = two_script_bundle(inbox, tmp_path / "reference-2")
bundle = tmp_path / "cli-bundle-2"
assert build(inbox, bundle, "--ingested-at", INGESTED_AT, "--proposed-at", PROPOSED_AT) == 0
assert tree(bundle) == tree(reference)

216
tests/test_propose_segments.py
View file

@ -1709,6 +1709,17 @@ def test_a_contents_run_is_discarded_and_the_body_survives() -> None:
that `_TRAILING_PAGE_NUMBER` reads as a contents line, and discarding it that `_TRAILING_PAGE_NUMBER` reads as a contents line, and discarding it
would delete a chapter. Measured on the K3 sample: one such heading at would delete a chapter. Measured on the K3 sample: one such heading at
position 1. position 1.
ROUND 2 MOVED ONE EXPECTATION HERE, and it is stated rather than quietly
updated: `Innhold 1` used to survive because the run required its members
to share a LEVEL, and it is a level-1 heading in front of three level-2
ones. The level condition is gone (a numbered report's contents list
interleaves `1`, `1.1`, `1.1.1`), so the run is now four and the contents
HEADING goes with the contents list -- which is what "innholdsfortegnelsen
er ikke konsepter" says. The cost is real and bounded: this heading has a
body of its own in the fixture, and discarding the candidate leaves that
body in no segment. Measured on the corpus, the relaxation changes ONE
document of 39, and there it removes exactly the leftover contents line.
""" """
off = okf_propose_segments.find_candidates(CONTENTS_THEN_BODY) off = okf_propose_segments.find_candidates(CONTENTS_THEN_BODY)
assert [c.title for c in off] == [ assert [c.title for c in off] == [
@ -1721,7 +1732,7 @@ def test_a_contents_run_is_discarded_and_the_body_survives() -> None:
"Vurdering", "Vurdering",
] ]
on = okf_propose_segments.find_candidates(CONTENTS_THEN_BODY, unit_fold=True) on = okf_propose_segments.find_candidates(CONTENTS_THEN_BODY, unit_fold=True)
assert [c.title for c in on] == ["Innhold 1", "Innledning", "Grunnforhold", "Vurdering"] assert [c.title for c in on] == ["Innledning", "Grunnforhold", "Vurdering"]
def test_a_deeper_heading_folds_into_its_parent() -> None: def test_a_deeper_heading_folds_into_its_parent() -> None:
@ -1844,3 +1855,206 @@ def test_a_recovered_outline_level_does_not_decide_the_unit() -> None:
"Loesmasser", "Loesmasser",
"Berggrunn", "Berggrunn",
] ]
# ---------------------------------------------------------------------------
# Round 2, 2026-09-08: two mechanisms the K3 per-file-type row named.
#
# Both stay behind flags that are off by default. The first refines Arm F's
# clause 1 (`--unit-fold`); the second is its own gate, because it changes the
# orphan check, which every file type reaches.
# ---------------------------------------------------------------------------
# A contents list as a PDF converter usually leaves one: no dot leaders, so
# every entry but the last is bodiless and the orphan check removes it. What
# reaches Arm F is a run of ONE, which is below `CONTENTS_RUN`, so clause 1
# cannot fire on the line that survived. This is the K3 position 9 shape.
BODILESS_CONTENTS = """## Formaalet med planen 4
## Orientering om prosjektet 4
## Maalsetting for SHA 5
SHA-plan for Stange skole, revisjon A.
## Formaalet med planen
Planen skal sikre arbeidet.
## Orientering om prosjektet
Prosjektet gjelder en utvidelse.
## Maalsetting for SHA
Null skader er maalet.
"""
# The known-negative for the same refinement: ONE bodiless page-numbered
# heading, no run at all. A rule that discarded it would delete a chapter whose
# title happens to end in a number.
SINGLE_BODILESS_PAGE_NUMBER = """## Sikkerhet i henhold til TEK 17
Kravene staar i forskriften.
## Grunnforhold
Loesmasser og berg.
"""
# The D1 shape: a sheet heading with a table opening on the line below it, so
# the heading's own body is empty. This is what a spreadsheet extracts to.
HEADING_THEN_TABLE = """## Prissammenstilling
| Post | Sum |
|------|-----|
| 01 | 100 |
| 02 | 200 |
"""
# The known-negative for D1: the heading has a body of its own, so it is not
# orphaned and nothing may be absorbed.
HEADING_WITH_BODY_THEN_TABLE = """## Prissammenstilling
Tabellen under viser postene i skjemaet.
| Post | Sum |
|------|-----|
| 01 | 100 |
"""
def test_a_bodiless_contents_run_is_discarded_too() -> None:
"""Clause 1 counts the run BEFORE the orphan check, not after.
The mechanism, measured: the orphan check (`find_candidates`) drops every
heading with an empty body, and a contents list without dot leaders is
exactly a sequence of such headings. All but the LAST entry are gone before
`fold_units` sees the list, so the run clause 1 looks for has length one and
the surviving contents line is emitted as a concept. Measured on the K3
sample: position 9 emits `Informasjon om og oppdatering av SHA-planen 5`
and position 7 emits `Tverrfaglig kontroll ....`, one leftover each.
Both halves over the same text: the flag-on assertion alone would stay
green if the default moved with it.
"""
off = okf_propose_segments.find_candidates(BODILESS_CONTENTS)
assert [c.title for c in off] == [
"Maalsetting for SHA 5",
"Formaalet med planen",
"Orientering om prosjektet",
"Maalsetting for SHA",
]
on = okf_propose_segments.find_candidates(BODILESS_CONTENTS, unit_fold=True)
assert [c.title for c in on] == [
"Formaalet med planen",
"Orientering om prosjektet",
"Maalsetting for SHA",
]
def test_a_single_bodiless_page_numbered_heading_survives() -> None:
"""The known-negative for the refinement above, and it is load-bearing.
Counting the run before the orphan check makes MORE candidates eligible for
clause 1, so the guard the sweep bought -- a run of siblings, never a single
line -- has to be measured again on the pre-orphan list. `TEK 17` is the
real heading that guard exists for.
"""
off = okf_propose_segments.find_candidates(SINGLE_BODILESS_PAGE_NUMBER)
on = okf_propose_segments.find_candidates(SINGLE_BODILESS_PAGE_NUMBER, unit_fold=True)
assert [c.title for c in off] == ["Sikkerhet i henhold til TEK 17", "Grunnforhold"]
assert off == on
def test_a_heading_keeps_the_table_that_opens_under_it() -> None:
"""D1, behind `--keep-table-heading`, off by default.
Default: the heading is bodiless, so the orphan check drops it and the
table block inherits its NAME -- the concept is named right but starts at
the table, so the heading line itself is not in the body any consumer reads.
On: the heading survives and its span extends over the table, which is
absorbed rather than emitted. The count does not move; the concept's first
byte does.
"""
off = okf_propose_segments.find_candidates(HEADING_THEN_TABLE)
assert [(c.title, c.rule) for c in off] == [
("Prissammenstilling", okf_propose_segments.RULE_TABLE_BLOCK)
]
assert off[0].start == HEADING_THEN_TABLE.index("| Post")
on = okf_propose_segments.find_candidates(HEADING_THEN_TABLE, keep_table_heading=True)
assert [(c.title, c.rule) for c in on] == [
("Prissammenstilling", okf_propose_segments.RULE_HEADING)
]
assert on[0].start == 0
assert on[0].end == off[0].end
def test_a_heading_with_its_own_body_absorbs_nothing() -> None:
"""The known-negative for D1: identical objects, not merely an equal count.
The gate is CONDITIONED on the drop. A heading that keeps its own body is
already carried by a live candidate, so absorbing the table as well would
silently merge two concepts on every document that has a table under a
section -- a default-shaped change wearing a flag.
"""
off = okf_propose_segments.find_candidates(HEADING_WITH_BODY_THEN_TABLE)
on = okf_propose_segments.find_candidates(HEADING_WITH_BODY_THEN_TABLE, keep_table_heading=True)
assert len(off) == 2
assert off == on
def test_keep_table_heading_takes_no_argument(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A boolean at the CLI: the variant has no number to sweep."""
source = write(tmp_path, HEADING_THEN_TABLE, "noarg-d1.md")
out = tmp_path / "noarg-d1.json"
with pytest.raises(SystemExit) as exit_info:
okf_propose_segments.main([str(source), "--out", str(out), "--keep-table-heading", "3"])
assert exit_info.value.code == 2
assert "usage:" in capsys.readouterr().err
# A contents list as a numbered report leaves one: the dotted entries carry
# their own depth, so `1.1`, `1.1.1`, `2.1` INTERLEAVE levels and no three
# consecutive entries are siblings. This is the K3 position 7 shape, where the
# level-2 entries fell in same-level runs and `6.2.2` did not.
INTERLEAVED_CONTENTS = """1.1 Om dette dokumentet ......... 4
1.1.1 Bakgrunn ......... 4
2.1 Versjoner ......... 5
Revisjon 04, for bruk til entreprise.
## Innledning
Bakgrunnen for arbeidet.
## Programvare
Programvaren er angitt her.
"""
def test_a_contents_run_crosses_heading_levels() -> None:
"""Clause 1 counts consecutive page-numbered headings, not siblings.
The mechanism, measured on K3 position 7: a numbered report's contents list
interleaves `1.1`, `1.1.1`, `2.1`, so requiring the run's members to share
a LEVEL breaks it at every level change. There the 34 contents entries are
one continuous block of page-numbered headings at levels 2 and 3, and the
level condition cut it into runs of 9, 1, 1, 1, 5, 2, 10, 2 and 3 -- so the
two-long runs survived and `6.2.2 Tverrfaglig kontroll ....` was emitted as
a concept while its neighbours were discarded.
The guard the sweep bought is unchanged and is still what makes this safe:
a run of at least `CONTENTS_RUN`, never a single line. One body heading
ending in a number is not a contents list, and that case has its own test.
Measured on the corpus, dropping the level condition changes ONE document
of 39, and there it removes exactly that leftover line.
"""
off = okf_propose_segments.find_candidates(INTERLEAVED_CONTENTS)
assert [c.title for c in off] == ["Versjoner ......... 5", "Innledning", "Programvare"]
on = okf_propose_segments.find_candidates(INTERLEAVED_CONTENTS, unit_fold=True)
assert [c.title for c in on] == ["Innledning", "Programvare"]