open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved

Browse source 
K3 round 3, order 20260908T170037Z-3622420612-from-.claude, plus the PM
injection of 17:55Z carrying the operator's D3 answer.

TWO NEW RULES, both OFF by default.

`--sheet-section-rows` cuts an open table block at the rows that label its
sections: a RUN of at least three rows whose first cell is a bare numeric
label. It is the first rule here that opens a span INSIDE a table, and the
opposite direction from Arm E -- that arm decides how far a block extends,
this decides where it is cut inside. The spreadsheet is the one file type no
arm had ever moved: 1 concept -> 12 on the priced sheet, of which 11 are the
reference's 11 cost groups. The run guard buys exactly three candidates on the
corpus (the rows stating a computation basis), and each section carries its
own `source_sheet` + `source_rows`, verified on the artifact.

`--drop-wrapped-outline` declines an outline candidate whose line continues
onto the next one. Round 2 falsified the LENGTH axis on this case; this one is
not about size. Over the 12-document sample it separates 8 of 34 candidates --
the quoted regulation paragraphs and the risk-table rows -- and none of the 26
the operator kept. Position 4: 4 concepts -> 1, the reference. The cell is
`treffer` by count and lossy by bytes, and the report says so.

Whole corpus, one cached extraction: 1 of 39 documents changes under the first
rule (0 of 32 pdf, 0 of 5 docx), 5 of 39 under the second (all pdf). Arm B and
F2 are byte-identical against a frozen c3b645b, both halves by `diff -r`.

THE `okf build` DEFAULT MOVED, and this is the operator's decision executed,
not a side effect: no flag now means `--outline-run 3 --unit-fold`. Each arm
keeps an explicit opt-out (`--outline-run 0`, `--no-unit-fold`) and the pair
reproduces the pre-move bytes exactly. The proposer's own defaults do NOT
move: they are what the goldens and every published reproduction block are
pinned to, so the two layers now disagree on purpose and `cli.py` says where.
Two shipped expectations moved with it and both are stated in place. MEASURED
AND REPORTED BACK: the configuration made default scores 2 of 12 on the unit
worksheet, `docx` 0 of 3 -- Arm F's published 5 of 12 was measured with
`--table-grid` ON, and without it the fold has no joined table to fold.

Position 1 is declined for the second round on the axis the order named: 3 of
3 recovered chapters have body under them, so "heading-on-heading" separates 0
of 3.

The K2 ranking control round 2 could not measure: two bundles from frozen
c3b645b differing only in `--keep-table-heading`, 2 of 1108 files apart. The
priced concept ranks 96 of 629 in BOTH and is delivered at rank 10 in both, so
the flag buys 35 payload bytes and zero rank positions. Round 2's prediction
is falsified: the concept's NAME was already restored by the orphan-name
inheritance; the flag adds the heading LINE.

The stale S7 literal is restated where it stood (`c759a657...`, 171 614 B,
re-measured on this HEAD), so the next order cannot inherit the superseded one.

1397 -> 1414 tests. Report: docs/2026-09-08-k3-runde3-per-filtype.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 20:33:52 +02:00
commit 47f6619e6f
11 changed files with 1381 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

20
CLAUDE.md
View file

@ -259,18 +259,28 @@ and fixtures, never code.
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
constant rather than the clock, because a wall-clock default takes
rebuild-equals-incremental away from anyone who omits them.
**Four segmentation rules are REACHABLE here and all four are OFF by
**Six segmentation rules are REACHABLE here and all six are OFF by
default** (2026-09-08): `--outline-run N` (Arm D), `--table-grid` (Arm E),
`--unit-fold` (Arm F) and `--keep-table-heading` (D1), each passed to the
proposer unchanged. Until that day the build path called the proposer with no
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
and `--drop-wrapped-outline` (both D3), each passed to the proposer
unchanged. Until that day the build path called the proposer with no
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
as one -- a build path a full arm behind the proposer. Exposing them is not
the same decision as moving one: **which arm should ship as the default is
the operator's, and a no-flag build stays byte-identical** (measured by
digest before and after, and by Arm E over all 43 corpus documents). Arm C
(`--max-segment-chars`) stays unexposed: no reference has ever been measured
for its cap. Reports: `docs/2026-09-08-k3-arm-f-mot-enhetsarket.md` and
`docs/2026-09-08-k3-runde2-per-filtype.md`.
for its cap. The two D3 rules read grammars nothing else here reads: a table
row's FIRST CELL (a run of bare numeric labels cuts the block that holds
them, which is the only way to reach a sheet whose units are rows and the
opposite direction from Arm E), and whether a RECOVERED heading's line is a
wrapped sentence (a heading is a complete line; quoted regulation and a
recovered table row are not). Both remove or add nothing anywhere else: over
the 43-document corpus they change 1 and 5 of 39 readable documents, and
**0 of 5 `docx` either way**. Reports:
`docs/2026-09-08-k3-arm-f-mot-enhetsarket.md`,
`docs/2026-09-08-k3-runde2-per-filtype.md` and
`docs/2026-09-08-k3-runde3-per-filtype.md`.
- Consume a bundle: `python3 tools/okf_consume.py <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction

20
README.md
View file

@ -82,7 +82,7 @@ did not pass them.
### The segmentation flags
Four rules are reachable from `okf build` and **all four are off by default**;
Six rules are reachable from `okf build` and **all six are off by default**;
a build with none of them is byte-identical to every bundle published here. Each
line below carries the number it was measured at, and nothing beyond it — none
of them is a recommendation, because which one should become the default has
@ -94,19 +94,21 @@ not been decided.
| `--table-grid` | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
| `--unit-fold` | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match, against **2** for the default, no cell worse. Whole corpus: 679 → 485 proposed segments over 39 readable documents |
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
| `--drop-wrapped-outline` | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept |
They compose, and the order above is the order they apply in. Measured on a
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
document:
| document | default | `--outline-run 3` | `+ --table-grid` | `+ --unit-fold` | `+ --keep-table-heading` |
|---|---|---|---|---|---|
| tender PDF, technical requirements | 1 (no boundary) | 9 | 9 | 9 | 9 |
| tender PDF, technical layout | 1 (no boundary) | 1 | 1 | 1 | 1 |
| price sheet `.xlsx` | 1 | 1 | 1 | 1 | 1 |
| experience list `.docx` | 21 | 21 | 6 | 3 | 3 |
| agreement `.docx` | 2 | 2 | 1 | 1 | 1 |
| markdown files in the bundle | 31 | 49 | 33 | 30 | 30 |
| document | default | `--outline-run 3` | `+ --table-grid` | `+ --unit-fold` | `+ --keep-table-heading` | `+ --sheet-section-rows --drop-wrapped-outline` |
|---|---|---|---|---|---|---|
| tender PDF, technical requirements | 1 (no boundary) | 9 | 9 | 9 | 9 | 9 |
| tender PDF, technical layout | 1 (no boundary) | 1 | 1 | 1 | 1 | 1 |
| price sheet `.xlsx` | 1 | 1 | 1 | 1 | 1 | **12** |
| experience list `.docx` | 21 | 21 | 6 | 3 | 3 | 3 |
| agreement `.docx` | 2 | 2 | 1 | 1 | 1 | 1 |
| markdown files in the bundle | 31 | 49 | 33 | 30 | 30 | 52 |
Every column merged 5 of 5 with 0 rejections. The reference for the first row
is 9, so the default is a full arm behind what the proposer can do on that

436
docs/2026-09-08-k3-runde3-per-filtype.md Normal file
View file

@ -0,0 +1,436 @@
# K3 round 3, per file type: the spreadsheet's rows, the wrapped sentence, and a default that moved
Third iteration of the per-file-type directive of 2026-09-08 13:05Z. Order
`20260908T170037Z-3622420612-from-.claude`, plus a mid-session injection from
the PM at 17:55Z carrying the operator's answer to the standing default
question (**D3**, alternative **b**).
Everything below is measured on the same 43-document corpus and the same
12-document sample as `docs/2026-09-08-k3-runde2-per-filtype.md`, against the
same reference: the operator's filled unit worksheet, which is local-only and
gitignored. No document title or extracted text from it is reproduced here.
## Section 0: what was measured and what was not
**Measured.** The three reproduction numbers the order names, before any edit.
Two new rules, each red-first, each with a known-negative and a whole-corpus
screen. The 12-document sample re-scored with session 109's criterion, which
survived on disk and reproduces its published gates exactly. The whole corpus
under five configurations from ONE cached extraction. Two K2 bundles, built
from a frozen `c3b645b` and differing only in one flag, and the ranking control
that pair exists to settle. A third K2 bundle on the new default.
**Not measured.** Whether a live model answers better with any of this. No
o200k token counts (no encoder is installed and the single-dependency rule does
not admit one); every figure is bytes plus `sha256`. Whether the two new rules
help a corpus that is not this one: the `xlsx` denominator is 2 documents and
the reference is 1 document.
## 1. The reproduction, before any edit
HEAD `c3b645b`, `git status --short` empty.
| # | control | result |
|---|---|---|
| a | `pytest -q` | **1397 passed** |
| b | `okf build ~/okf-test/dokumenter`, no flags | **31** markdown files |
| c | the same folder with the four round-2 flags | **30** markdown files, tender PDF plan **9** entries |
| d | consumer bundle `K2-bundle-20260903` | **1108 files**, `9cd74519…`, unchanged |
| e | S7 control question, default command | **`c759a657dbb41d71`, 171 614 B** |
All five reproduce. (e) is the literal round 111 measured after finding the
order's own known-positive stale; it is now restated in the two documents that
still publish the superseded figure, so the next order cannot inherit it.
## 2. `xlsx`: a rule that cuts a table at the rows that label it
**The mechanism, from round 2 § 3.** The priced sheet extracts to 103 lines:
one heading and ONE continuous pipe-table block (lines 3–103). The reference's
eleven cost groups are eleven ROWS inside that block. No heading rule can reach
them, and `--keep-table-heading` moved the first byte without moving the count.
**The rule, behind `--sheet-section-rows`, off by default.** A section row is a
table row whose FIRST cell is a bare numeric label and which carries at least
one other non-empty cell; each opens a candidate reaching the next section row
or the end of the block. It is general in the two ways that matter: no word
list, and no knowledge of which numbers any sheet uses. The label grammar is
digits with optional separators (`11+12`), so a row opening with a word is not
a section.
**What bounds it is a RUN, and the run is the module's own device.** A single
numbered row is a quantity, not a section. `SHEET_SECTION_RUN = 3` is the same
number and the same reasoning as `CONTENTS_RUN`, and its effect is measured
rather than assumed:
| run length | documents changed | candidates added |
|---|---|---|
| `>= 1` | 1 of 39 | **+14** |
| `>= 2` | 1 of 39 | +11 |
| `>= 3` (shipped) | 1 of 39 | **+11** |
| `>= 4` … `>= 6` | 1 of 39 | +11 |
The guard buys exactly three: the three rows where the sheet states a
computation basis (`16000`, `500000`, `500000`) on a row of its own. Above 2 the
threshold is inert on this corpus — the only run that fires is 11 long — so the
number is bounded by a fixture, not by the corpus, and that is said rather than
implied.
**Result on the priced sheet: 1 concept → 12.** Eleven of them are the
reference's eleven cost groups; the twelfth is the sheet's preamble, which is
real text and has to live somewhere.
| | default | `--keep-table-heading` | `+ --sheet-section-rows` |
|---|---|---|---|
| concepts | 1 | 1 | **12** |
| first concept's span | `[34, 11048]` | `[0, 11048]` | `[0, 1241]` |
| cost-group concepts | 0 | 0 | **11** |
**The reference is 11 and this is 12, so the cell does not cross.** Under
session 109's criterion the `xlsx` position moves from `for grovt` at distance
**10** to `for fint` at distance **1**. Getting to 11 would mean discarding the
preamble or merging it into the first cost group, and neither is a rule — the
first is a silent loss and the second is false about the document.
**One half of the order's specification is not reachable, and the reason is
structural.** The order asks that the table's heading row be copied into every
concept so each reads alone. A concept's body is `text[start:end]` and its
anchor quote must equal that slice (`segmentation.py`, `slice_segments` and
`_resolve_entry`), so a plan cannot express a body assembled from two places. A
non-contiguous body would either break the anchor or make the frontmatter
disagree with the bytes. What IS reachable in that direction is
`--keep-table-heading`, which gives the FIRST concept the sheet heading, and
that is what the pair does together.
**The provenance is per section, and it was verified on the artifact rather
than assumed.** Each cost group carries its own workbook rows:
| concept | `source_sheet` | `source_rows` |
|---|---|---|
| the preamble | `Prissammenstilling` | `[1, 6]` |
| `01` | `Prissammenstilling` | `[7, 7]` |
| `11+12` | `Prissammenstilling` | `[8, 8]` |
| … the eight in between … | `Prissammenstilling` | `[9, 9]` … `[16, 16]` |
| `92` (the last) | `Prissammenstilling` | `[17, 100]` |
The last section carries the sheet's tail, because a rule that cuts and never
discards has to put the rows after the final label somewhere.
**How the two table rules relate, since both are now reachable.**
`--table-grid` (Arm E) stops a converter's grid-rule line from CLOSING a block,
so one grid table proposes one candidate instead of one per row group.
`--sheet-section-rows` CUTS an open block at its section rows. They read
different grammars — a `+---+` rule line against a pipe row's first cell — and
they compose in one order: Arm E decides how far a block extends, this rule
decides where it is cut inside. With both on, the section scan runs over the
larger block Arm E produced. Neither disables the other, which is why they are
two flags and not one.
**Known-negatives, whole corpus, from the cached extraction:**
| control | result |
|---|---|
| documents changed by `--sheet-section-rows` | **1 of 39** |
| `pdf` changed | **0 of 32** |
| `docx` changed | **0 of 5** |
| `xlsx` changed | **1 of 2** — the other sheet has no run of numbered rows |
| a table with no section rows (fixture) | identical objects, not merely an equal count |
| a lone numbered row (fixture) | identical objects |
## 3. `pdf` position 4: a heading is a complete line
**Round 2 declined this one on a measurement**, and the measurement was about
LENGTH: sorted, the real headings in the sample run 7…88 characters and the
quoted regulation sentences run 86…100, so no threshold separates them.
**The new axis is not about size.** A heading is a complete line; a wrapped
sentence continues on the next one. `--drop-wrapped-outline` does not admit an
outline candidate whose next line begins in lower case. Measured over the 12
sample documents, on the candidates Arm D recovers:
| class | count |
|---|---|
| outline candidates in the sample | **34** |
| followed by a lower-case line | **8** |
| — of those, quoted regulation (position 4) | 4 of 4 |
| — of those, risk-table rows (position 9) | 4 |
| headings the operator kept, dropped by the rule | **0 of 26** |
**It judges RECOVERED candidates only.** A dotted or ATX heading is the
document's own claim about itself; declining to invent a heading is a different
decision from overruling a declaration, and the known-negative fixture pins it.
**Result: position 4 goes 4 concepts → 1, which is the reference**, and the
criterion scores the cell `treffer`. Position 9 goes 10 → **6** against a
reference of 4: closer, not there, and the remaining 6 are Arm D recovering a
risk table's rows as an outline, which the operator ruled out in free text.
**And the count is not the whole truth, so here is the rest of it.** Position
4's surviving concept is the ONE numbered paragraph that does not wrap
(`2)`), and its span starts at line 18 of 302. Under Arm B the document has no
candidate at all and lands as one flat concept covering everything; under this
rule it lands as one concept covering everything **from line 18**, so 940
characters of the note's opening are in no segment. The cell is `treffer` by
count and lossy by bytes. That is an argument for the flag staying off, and it
is why it is written here rather than in a footnote.
**Whole-corpus screen:** `--drop-wrapped-outline` changes **5 of 39** readable
documents, all `pdf` (**0 of 5 `docx`, 0 of 2 `xlsx`**). What it removes in the
three documents beyond the sample is the same shape every time: a product or
requirement row recovered as a chapter, whose sentence continues on the line
below.
## 4. `pdf` position 1: declined again, on the axis the order named
The order asked for one new axis: do the two recovered chapters have body under
them in the outline structure, or are they heading-on-heading?
| candidate | span | non-empty lines under it |
|---|---|---|
| `Innledning` (§1) | 621 | 9 |
| `Referanser` (§6) | 1 945 | 28 |
| `Vedlegg` (§7) | 17 524 | 341 |
**3 of 3 have body, so the axis separates 0 of 3.** The reference keeps
`Vedlegg` and drops the other two because the operator named it in prose, and
the document offers no property that says so. Declined, with numbers, for the
second round.
Position 0 (95.1 % `(cid:N)` glyphs) is reported and not repaired, unchanged.
## 5. Per file type, after round 3
The 12-document sample, concepts per configuration, from the cached extraction
session 109 wrote. `F2` is round 2's code with its four flags; `F3` adds the two
new ones. **The instrument was verified before it was read**: re-run on the
archive plans the operator rated, session 109's criterion returns
**LITERAL 7/12, DOMINANCE 11/12** — its published gates — and the F2 column
reproduces round 2's published counts on **12 of 12** positions.
| pos | type | reference | Arm B | Arm D | Arm E | F2 | **F3** |
|---|---|---|---|---|---|---|---|
| 0 | pdf | — | 1 | 1 | 1 | 1 | 1 |
| 1 | pdf | 15 | 20 | 23 | 23 | 17 | 17 |
| 2 | pdf | 9 | 1 | 9 | 9 | 9 | 9 |
| 3 | xlsx | 11 | 1 | 1 | 1 | 1 | **12** |
| 4 | pdf | 1 | 1 | 4 | 4 | 4 | **1** |
| 5 | docx | 3 | 21 | 21 | 6 | 3 | 3 |
| 6 | pdf | 6 | 6 | 7 | 7 | 7 | 7 |
| 7 | pdf | 30 | 45 | 48 | 48 | 33 | 33 |
| 8 | pdf | 8 | 1 | 8 | 8 | 8 | 8 |
| 9 | pdf | 4 | 5 | 11 | 11 | 10 | **6** |
| 10 | docx | 2 | 15 | 15 | 3 | 2 | 2 |
| 11 | docx | 1 | 2 | 2 | 1 | 1 | 1 |
(Arm B is written here as "1" where the proposer finds no boundary and the
document lands as one flat concept. Round 2's table wrote "0" for the same
state; the two conventions describe one bundle.)
**The label row, under session 109's criterion:**
| type | n | Arm B | Arm D | Arm E | F2 | **F3** |
|---|---|---|---|---|---|---|
| `pdf` | 8 | 2 | 2 | 2 | 2 | **3** |
| `docx` | 3 | 0 | 0 | 1 | 3 | **3** |
| `xlsx` | 1 | 0 | 0 | 0 | 0 | **0** |
| whole sample | 12 | 2 | 2 | 3 | 5 | **6** |
**`pdf` crosses for the first time in four rounds: 2 of 8 → 3 of 8.** The
`xlsx` cell does not cross and its distance falls from 10 to 1.
**One limit on that `xlsx` label, and it is the criterion's rather than the
rule's.** Session 109's criterion was written before this rule existed and
classifies a concept as "table-derived" by its title (`Tabell linje N`) or by
`rule:table-block`/`rule:table-grid` in `derived`. A `rule:sheet-section`
concept is neither, so the criterion counts the eleven cost groups as ordinary
units. Under the opposite reading — sections counted as table concepts, which
the position's `b = M` would penalise — the same plan scores distance **11**
rather than 1. Both readings are stated because the criterion's author never
faced this case, and picking the flattering one silently is the failure mode
this repository has a rule about.
## 6. Whole corpus, 43 documents
One cached extraction, so every configuration reads identical bytes. 39 of 43
are extractable (4 `FAILED`, unchanged).
| figure | Arm B | F2 | **F3** | denominator |
|---|---|---|---|---|
| plans written | 28 | 32 | **32** | 43 |
| proposed entries | 618 | 485 | **484** | — |
| readable documents | 39 | 39 | 39 | 43 |
| plans differing from F2 | — | — | **6 of 32** | — |
The six: one `.xlsx` (+11 from the section rows) and five `.pdf` (−12 from the
wrapped rule). **The flag-off identity holds twice**: F2 built from a frozen
`c3b645b` and from this working tree is byte-identical over all 39 documents
(`diff -r`, exit 0), and so is Arm B.
## 7. The K2 ranking control, which round 2 could not measure
Round 2 predicted that `--keep-table-heading` would move the priced concept's
rank back, because it restores the heading's LINE to the body and the
`56ae274` regression was caused by losing the heading's NAME. **The prediction
is falsified.**
Two K2 bundles, both built from frozen `c3b645b`, same command, same stamps,
differing only in the flag. **2 of 1108 files differ**, both spreadsheets:
| | no flag | `--keep-table-heading` |
|---|---|---|
| files | 1108 | 1108 |
| content digest | `f09035d9…` | `432ca424…` |
| concepts | 629 | 629 |
| `merged + coded rejections` | `39 + 4 = 43 = N` | idem |
| wall time | 883 s | 836 s |
And the ranking, on the S7 control question:
| command | no flag | `--keep-table-heading` |
|---|---|---|
| default (no vocabulary), candidate rank | **489 of 629**, `below_k` | **489 of 629**, `below_k` |
| `--cost-vocabulary --k 12`, candidate rank | **96 of 629** | **96 of 629** |
| `--cost-vocabulary --k 12`, delivered | **rank 10** | **rank 10** |
| payload bytes, vocabulary on | 189 690 | 189 725 |
**The flag buys 35 bytes and zero rank positions.** The reason is visible in
the concept itself: its title is `Prissammenstilling {#sheet-1}` in BOTH
bundles, because the orphaned heading's NAME is already carried onto the table
block by the rule `56ae274` added. What the flag adds is the heading LINE in
the body, which is a readability property and not a retrieval one.
The absolute rank differs from the 19 published in
`docs/2026-09-08-prisform-og-loggen-k2.md` § 7, and the difference is not this
flag: that bundle predates the O3 provenance work, so every concept in these
two carries `sources` and a locator it did not. The comparison that decides the
question is the one above, where the two bundles differ in 2 files out of 1108.
## 8. The default moved (operator, D3 alternative b)
The PM injection at 17:55Z carries the operator's answer: `okf build` with no
flag is now **`--outline-run 3 --unit-fold`**. `--table-grid` and
`--keep-table-heading` stay flags.
**Implemented as instructed**, red test first, with an explicit opt-out per arm
because a default nobody can turn off is not a default:
| arm | on by default | opt-out |
|---|---|---|
| Arm D | `--outline-run 3` | `--outline-run 0` |
| Arm F | `--unit-fold` | `--no-unit-fold` |
`--outline-run 0 --no-unit-fold` reproduces the pre-move bytes exactly, and the
test that used to pin the flagless build to the two-script path now pins both
halves: the two-script path WITH the arms equals the new default, and WITHOUT
them equals the opt-out. Two shipped expectations moved with it and both are
stated in place rather than rewritten.
**The proposer's own default did NOT move.** `propose.py`'s rules stay off:
they are what the golden fixtures and every published reproduction block are
pinned to, and `tools/okf_propose_segments.py` is the measuring instrument. The
two layers now disagree on purpose, and the constant that carries the decision
(`cli.DEFAULT_OUTLINE_RUN`, `cli.DEFAULT_UNIT_FOLD`) says so where a reader
will find it.
### The number the decision was made on belongs to a different configuration
This is the one thing in this report the operator should read before anything
else. **Arm F's published result — 5 of 12, `docx` 3 of 3 — was measured with
`--table-grid` ON.** The configuration just made default is Arm D plus Arm F
*without* Arm E. Measured on the same 12 documents with the same criterion:
| configuration | whole sample | `pdf` | `docx` | `xlsx` |
|---|---|---|---|---|
| Arm B (the old default) | 2 of 12 | 2 of 8 | 0 of 3 | 0 of 1 |
| Arm D | 2 of 12 | 2 of 8 | 0 of 3 | 0 of 1 |
| Arm E (D + `--table-grid`) | 3 of 12 | 2 of 8 | 1 of 3 | 0 of 1 |
| **the new default (D + F)** | **2 of 12** | 2 of 8 | **0 of 3** | 0 of 1 |
| D + E + F (round 2's F2) | 5 of 12 | 2 of 8 | 3 of 3 | 0 of 1 |
| D + E + F + the two new rules | **6 of 12** | **3 of 8** | 3 of 3 | 0 of 1 |
**The mechanism, so the row is not just a number.** Arm F's clause 3 folds a
table into the heading that introduces it. Without Arm E a grid table is not
one table but one candidate per row group, so the fold has nothing to fold: at
position 5 the table-concept penalty goes 3 → 12 and at position 10, 1 → 12.
The two arms in the new default are the two that need each other least on
`pdf` and most on `docx`.
On the five-document test folder the same thing is visible directly: with the
new default the `.docx` experience list lands as **15** concepts, against 3
with `--table-grid` added. The injection's expectation that the folder would
still produce 30 markdown files does not hold — it produces **43** — and the
tender PDF is 9 either way, which was the other half of that check.
The default is the operator's decision and it is implemented as given. The
measurement above is the report back, not an argument for reversing it.
## 9. Tests
`1397 → 1414`. Seventeen new, and which of them were red first is stated rather
than counted:
- **Red before the implementation (9):** the section-row cut and its
orphan-check exemption; the per-section provenance; the flag reaching
`okf build`; the wrapped-outline drop and its default control; the two
default-move tests; the moved two-script byte control.
- **Green by construction, and named as such (5):** three known-negatives (a
table with no section rows, a lone numbered row, a declared heading that
wraps), the `derived` naming, and the sheet default control. Their value is
the mutations below.
- **Guards rather than measurements (3):** the two "takes no argument" CLI
checks and the attribution-chunk test, which argparse would satisfy either
way.
Two shipped expectations moved, both stated in place: the flagless build's plan
titles, and the two-script byte control.
## Reproducing
```
# 1. The five reproduction numbers.
pytest -q # 1397 before
okf build ~/okf-test/dokumenter --bundle /tmp/b --bundle-id t --okf-version 0.2
# 2. Cache the corpus ONCE, then read every variant off the cache.
python3 /tmp/k3r3/cache43.py # 39 texts, 4 FAILED
python3 /tmp/k3r3/corpus43.py <src> <outdir> '<json kwargs>' # one variant
# 3. The flag-off identity, both halves.
diff -rq /tmp/k3r3/c43-F2-head /tmp/k3r3/c43-F2-now ; echo $? # 0
diff -rq /tmp/k3r3/c43-B-head /tmp/k3r3/c43-B-now ; echo $? # 0
# 4. The 12-sample, scored with session 109's own criterion.
python3 /tmp/k3r3/plans12.py && python3 /tmp/k3r3/score3.py
# 5. The K2 pair. Each build is a foreground nohup with an rc file and a
# bounded poll -- the Bash tool caps at 600 s and a build takes ~850-1270 s.
okf build ~/corpora/.../K2/trinn1 --bundle /tmp/k3r3/k2-noflag \
--bundle-id k2-trinn1-20260903 --okf-version 0.2
okf build ~/corpora/.../K2/trinn1 --bundle /tmp/k3r3/k2-kth \
--bundle-id k2-trinn1-20260903 --okf-version 0.2 --keep-table-heading
diff -rq /tmp/k3r3/k2-noflag /tmp/k3r3/k2-kth | wc -l # 2
python3 /tmp/k3r3/rank.py
# 6. The consumer bundle, untouched.
cd ~/corpora/okf-telling-20260829 && LC_ALL=C find K2-bundle-20260903 -type f \
| LC_ALL=C sort | xargs shasum -a 256 | shasum -a 256
# 1108 files, 9cd745194346cda0c70eab9c7136fa44506203bbe85bc17d7eff2766c6e9b4d1
```
## Honesty limits
- **n = 12, one rater, one reference.** Per file type the denominators are 8, 3
and 1. The `xlsx` row is a `0/1`, which is not a rate.
- **The `xlsx` rule was derived from the one document it moves**, and screened
outward on 39. The outward screen bounds the fitting; it does not remove it.
The corpus holds two spreadsheets and the rule fires on one.
- **The wrapped rule's cell is `treffer` by count and lossy by bytes** (§ 3).
- **`SHEET_SECTION_RUN` is not swept against a reference**, because none exists
for it. What is published is its corpus sensitivity, and above 2 it is inert.
- **The criterion predates the new rule** and its `xlsx` label depends on a
classification its author never made (§ 5). Both readings are given.
- **The K2 ranking control is one question on one bundle.** It says the flag
does not move THIS rank; it does not say the flag is inert for retrieval in
general.
- **The new default was measured after it was set, not before.** The operator's
decision arrived mid-session and § 8 reports the configuration's score
against the alternatives; nothing here reverses it.

7
docs/2026-09-08-kravnummer-tokenisering.md
View file

@ -36,6 +36,13 @@ on this machine before any of the after-columns were read:
| 8 | priced concept candidate rank 10 → 19 | this repo, 2026-09-08 (O1) § 7 | 10 → 19, and 251 / 1 / 1 for the other three rows |
| 9 | K2 payload bytes, two commands | this repo, O1 § 7 | 169 573 B and 172 588 B |
> **Superseded literal (added 2026-09-08, round 3).** The S7 control figures in
> this document were correct when it was written and are not the ones to reuse:
> `a37d5ce`, `17c49fc` and `c95d189` each widened the excerpt form afterwards.
> The current known-positive for the S7 control question under the default
> command is **`c759a657dbb41d71`, 171 614 B**, measured on `c3b645b` against
> the same bundle. `docs/2026-09-08-k3-runde3-per-filtype.md` § 4.
**NOT measured.** Whether a live model answers *better* with the named
requirement in the payload. This document measures a ranking; the consumer
measures the answer. Also not measured: that either rule helps a corpus other

7
docs/2026-09-08-sjeldenhetsvekt.md
View file

@ -38,6 +38,13 @@ was read:
| 8 | K2 delivery costs, four commands | previous session § 7 | 169 583 / 183 798 / 190 438 / 108 877 B |
| 9 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6** |
> **Superseded literal (added 2026-09-08, round 3).** The S7 control figures in
> this document were correct when it was written and are not the ones to reuse:
> `a37d5ce`, `17c49fc` and `c95d189` each widened the excerpt form afterwards.
> The current known-positive for the S7 control question under the default
> command is **`c759a657dbb41d71`, 171 614 B**, measured on `c3b645b` against
> the same bundle. `docs/2026-09-08-k3-runde3-per-filtype.md` § 4.
**NOT measured.** Two things, both named rather than glossed:
- **`o200k_base` token counts.** No `o200k` encoder is installed on this

85
src/llm_ingestion_okf/cli.py
View file

@ -73,6 +73,25 @@ __all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
CLI_ID = "okf build"
#: What `okf build` runs when no flag is given. Moved 2026-09-08 by the
#: operator, from Arm B to Arm D plus Arm F, on the round-2 and round-3
#: measurements: the previous default was a full arm behind the proposer on
#: the one document with a reference, and the fold matches 5 of 12 documents
#: against the default's 2 with no cell worse.
#:
#: THE VALUE LIVES HERE AND NOT IN `propose.py`. That module's rules stay OFF
#: by default: they are what the golden fixtures and every published
#: reproduction block are pinned to, and the proposer's own CLI is the
#: measuring instrument. What moved is this COMMAND's default, and the two
#: layers disagreeing on purpose is why the disagreement is written down here
#: rather than discovered later.
#:
#: Each arm keeps an explicit opt-out: `--outline-run 0` (the number was
#: always its own switch) and `--no-unit-fold`. A default a caller cannot turn
#: off is not a default.
DEFAULT_OUTLINE_RUN = 3
DEFAULT_UNIT_FOLD = True
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -91,6 +110,8 @@ def _propose_plans(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -119,6 +140,8 @@ def _propose_plans(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -142,10 +165,12 @@ def build(
segments: bool = True,
plans_dir: Path | None = None,
okf_type: str = "reference",
outline_run: int = 0,
outline_run: int = DEFAULT_OUTLINE_RUN,
table_grid: bool = False,
unit_fold: bool = False,
unit_fold: bool = DEFAULT_UNIT_FOLD,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -193,6 +218,8 @@ def build(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -295,14 +322,15 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--outline-run",
type=int,
default=0,
default=DEFAULT_OUTLINE_RUN,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. 0 (the default) is OFF and "
"leaves the bundle byte-identical. Measured on a tender PDF whose "
"headings are bare integers: no boundary at 0, nine at 3"
"sustain an ascending run of at least N. The default is 3 (operator, "
"2026-09-08); pass 0 to turn the arm off, which is this arm's "
"explicit opt-out. Measured on a tender PDF whose headings are bare "
"integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
@ -319,13 +347,25 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--unit-fold",
action="store_true",
default=DEFAULT_UNIT_FOLD,
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. Absent (the default) is OFF. Measured on the K3 "
"sample: 5 of 12 documents match the operator's unit worksheet, "
"against 2 for the shipped default, with no cell worse"
"only reduce a plan. ON by default since 2026-09-08 (operator); the "
"flag is kept so a call site that passes it stays valid. Measured on "
"the K3 sample: 5 of 12 documents match the operator's unit "
"worksheet, against 2 for the arm below it, with no cell worse"
),
)
build_parser.add_argument(
"--no-unit-fold",
action="store_false",
dest="unit_fold",
help=(
"Arm F's explicit opt-out. With --outline-run 0 it reproduces the "
"pre-2026-09-08 default byte for byte, which is what keeps the move "
"reversible for a consumer who needs the old bytes"
),
)
build_parser.add_argument(
@ -339,6 +379,31 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"concept gains the heading line it was missing"
),
)
build_parser.add_argument(
"--sheet-section-rows",
action="store_true",
help=(
"D3, passed to the proposer unchanged: cut an open table block at "
"the rows that label its sections -- a RUN of rows whose first cell "
"is a bare numeric label. The opposite direction from Arm E, which "
"decides how far a block extends; this decides where it is cut "
"inside. Absent (the default) is OFF. Measured on a tender price "
"sheet whose whole body is one table block: 1 concept -> 12, "
"against a reference of 11 cost groups plus the sheet's preamble"
),
)
build_parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
help=(
"D3, passed to the proposer unchanged: do not admit an outline "
"candidate whose line continues onto the next one. Recovered "
"candidates only, never a heading the document declares. Absent "
"(the default) is OFF. Measured on the K3 sample: 8 of 34 outline "
"candidates wrap, and they are the quoted regulation paragraphs "
"and the risk-table rows the operator wrote off"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
@ -363,6 +428,8 @@ def main(argv: list[str] | None = None) -> int:
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

211
src/llm_ingestion_okf/propose.py
View file

@ -108,6 +108,20 @@ RULE_OUTLINE = "rule:outline"
#: joined -- never on one whose span merely happens to contain a rule line, so
#: a single-row grid table stays byte-identical to Arm D.
RULE_TABLE_GRID = "rule:table-grid"
#: D3 round 3 only, and it is the first rule in this module that opens a span
#: INSIDE a table rather than at one. Like Arm C, D and E it is NOT one of Topic
#: 2's ported rules and NOT defined upstream -- `docs/2026-09-02-k3-k4-k5-metode.md`
#: contains no occurrence of the word "arm" at all -- so this definition was
#: written for order 20260908T170037Z-3622420612-from-.claude and is reported as
#: the author's. Its axis is a fourth one. Arm C names SIZE, Arm D what the
#: DOCUMENT declared, Arm E what the CONVERTER emitted; this names what the
#: SHEET labelled: a row whose first cell is a bare numeric label, in a run of
#: such rows. It is the opposite DIRECTION from Arm E -- that one stops a rule
#: line from closing a block so a grid table proposes one candidate instead of
#: many, this one cuts an open block at the rows that label its sections -- and
#: the two compose in one order: Arm E decides how far a block extends, this
#: decides where it is cut inside.
RULE_SHEET_SECTION = "rule:sheet-section"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -115,6 +129,7 @@ RULE_NAMES = (
RULE_SIZE_SPLIT,
RULE_OUTLINE,
RULE_TABLE_GRID,
RULE_SHEET_SECTION,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -211,6 +226,16 @@ _OUTLINE = re.compile(r"^\s{0,4}(?P<number>\d{1,2})[.)]?\s+(?P<title>\S.*?)\s*$"
# through `_segment_path` -- an unstripped page number would become part of a
# filename.
_TRAILING_PAGE_NUMBER = re.compile(r"[\s.]+\d{1,4}\s*$")
# D3's grammar, and it reads a CELL rather than a line. A sheet's section label
# is a bare number, optionally joined to another by a separator where two groups
# were merged (`11+12`), and nothing else: no letters, so a row opening with a
# word is not a section, and no word list, so the rule knows nothing about which
# numbers any real sheet uses.
_SHEET_SECTION_LABEL = re.compile(r"^\d+(?:[+./-]\d+)*$")
# A pipe that pandoc did not escape. Splitting a row on a bare `|` would cut a
# cell containing a literal pipe in half and misread the FIRST cell of the row
# after it, which is the only cell this rule judges.
_UNESCAPED_PIPE = re.compile(r"(?<!\\)\|")
class ProposerError(Exception):
@ -315,6 +340,79 @@ def outline_runs(
return [run for run in runs if len(run) >= minimum]
#: D3. How many CONSECUTIVE numbered rows make a sectioned table. Three, and it
#: is the same bounding device -- and the same number -- `CONTENTS_RUN` uses,
#: for the same reason: a single numbered row is a stated quantity, not a
#: section, and a rule that read one would cut a sheet at every computation
#: basis. NOT swept on a reference, because no reference exists for its value;
#: the corpus SENSITIVITY of the choice is published instead, in
#: `docs/2026-09-08-k3-runde3-per-filtype.md`. Not a CLI knob for the reason
#: `CONTENTS_RUN` is not one.
SHEET_SECTION_RUN = 3
def _wraps_onto_next_line(lines: list[str], index: int) -> bool:
"""True when the line at `index` is a sentence that continues below it.
The test is the next line's first character being lower case. A heading is
a complete line -- the line under it opens a new sentence, is blank, or is
a bullet -- while a hard-wrapped paragraph carries its own continuation.
This is deliberately NOT a length test. Round 2 sorted every outline title
in the K3 sample and measured the two classes overlapping: a real chapter
heading of 88 characters against quoted sentences of 86, 91, 92 and 100, so
no threshold separates them. Measured on the same sample, this axis
separates 8 of 34 -- the four quoted regulation paragraphs and four risk
table rows, and none of the 26 headings the operator kept.
"""
if index + 1 >= len(lines):
return False
following = lines[index + 1].strip()
return bool(following) and following[0].islower()
def _sheet_section_rows(lines: list[str]) -> dict[int, tuple[str, str]]:
"""D3: line index -> (label, row text) for every section row, or empty.
A section row is a table row whose FIRST cell is a bare numeric label and
which carries at least one other non-empty cell -- the label alone names
nothing, and the first non-empty cell after it is what the section is
called. The rows must come in a run of at least :data:`SHEET_SECTION_RUN`
consecutive lines: that is what separates a labelled section column from a
quantity stated on its own row, and it is the module's existing guard
rather than a new one.
Computed over the whole line list before the marking loop, for the reason
the outline runs are: a run is a property of the text, and a forward scan
that decided one row at a time could not know whether the run it is inside
is long enough.
"""
labelled: dict[int, tuple[str, str]] = {}
for index, line in enumerate(lines):
if not _TABLE_ROW.match(line):
continue
cells = [cell.strip() for cell in _UNESCAPED_PIPE.split(line.strip().strip("|"))]
if not cells or not _SHEET_SECTION_LABEL.match(cells[0]):
continue
rest = [cell for cell in cells[1:] if cell]
if not rest:
continue
labelled[index] = (cells[0], rest[0])
sections: dict[int, tuple[str, str]] = {}
ordered = sorted(labelled)
start = 0
while start < len(ordered):
end = start + 1
while end < len(ordered) and ordered[end] == ordered[end - 1] + 1:
end += 1
if end - start >= SHEET_SECTION_RUN:
for position in range(start, end):
sections[ordered[position]] = labelled[ordered[position]]
start = end
return sections
def find_candidates(
text: str,
*,
@ -322,6 +420,8 @@ def find_candidates(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -356,6 +456,14 @@ def find_candidates(
count does not move -- one candidate either way -- and the first byte does.
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
`sheet_section_rows` is D3's gate and it is OFF at False, where the scan is
not run at all. On, a RUN of numbered rows inside an open table block cuts
it: each such row opens a candidate that reaches the next section row, or
the end of the block. It is the only rule here that opens a span inside a
table, and it is its own flag for the same reason D1 is -- a sheet is the
one file type whose units are rows, and every other type reaches this scan
too.
"""
lines = text.splitlines(keepends=True)
offsets: list[int] = []
@ -384,6 +492,10 @@ def find_candidates(
# ordering rather than a property of this corpus.
admitted = {index: title for index, _, title in runs[-1]}
# D3's input, and the same whole-text reasoning as `admitted` above: a run
# is a property of the line list, not of a line.
sections = _sheet_section_rows(lines) if sheet_section_rows else {}
marked: list[tuple[int, Candidate]] = []
in_table = False
# Arm E's state, and all three of these are cleared together on the
@ -398,6 +510,36 @@ def find_candidates(
joined: set[int] = set()
for index, line in enumerate(lines):
if _TABLE_ROW.match(line):
section = sections.get(index)
if section is not None:
label, name = section
# A section row never ALSO opens a table block, even when the
# run starts on the block's first line: two marks on one line
# would give the second an empty span, and the orphan check
# deletes an empty span silently. `open_block` is cleared with
# it, because a block cut here is no longer one span an Arm E
# join could extend.
in_table = True
open_block = None
rule_pending = False
marked.append(
(
index,
Candidate(
title=f"{label} {name}",
# The table sentinel, not a heading depth: a sheet
# row declares no level, and letting it vote in
# Arm F's clause 2 would make a sectioned sheet a
# one-level document.
level=9,
number=label,
rule=RULE_SHEET_SECTION,
start=offsets[index],
end=end_of_text,
),
)
)
continue
if not in_table:
in_table = True
open_block = len(marked)
@ -431,6 +573,13 @@ def find_candidates(
outline_title = admitted.get(index)
if outline_title is not None:
if drop_wrapped_outline and _wraps_onto_next_line(lines, index):
# Filtered at ADMISSION rather than at run selection: the run
# this document sustains is a property of its numbering, and
# re-selecting it from a thinned list would move boundaries on
# documents where nothing wraps. The rule declines candidates;
# it does not rewrite which run won.
continue
outline_match = _OUTLINE.match(line)
assert outline_match is not None, "an admitted index still matches the grammar"
marked.append(
@ -518,7 +667,19 @@ def find_candidates(
end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end]
# The orphan check: everything after the heading line itself.
if not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip():
#
# D3 IS EXEMPT, and the exception is stated rather than worked around:
# the check asks whether anything stands UNDER a candidate's first
# line, which is the right question for a heading and the wrong one for
# a row. A one-row section carries its content in its own cells, so
# every section but the last would be read as bodiless and deleted --
# the rule could not fire at all. It is the same shape as a contents
# list without dot leaders, and it is the reason that one needed
# `Candidate.contents`.
orphan = candidate.rule != RULE_SHEET_SECTION and (
not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip()
)
if orphan:
orphaned_name = (candidate.title, candidate.number)
continue
inherited, orphaned_name = orphaned_name, None
@ -869,6 +1030,8 @@ def build_plan(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -880,6 +1043,8 @@ def build_plan(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -951,6 +1116,8 @@ def run(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -1004,6 +1171,8 @@ def run(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
@ -1127,6 +1296,44 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"part of an arm: the orphan check is reached by every file type"
),
)
parser.add_argument(
"--sheet-section-rows",
action="store_true",
help=(
"D3: cut an open table block at the rows that label its sections. "
"A section row is one of a RUN of at least SHEET_SECTION_RUN "
"consecutive rows whose first cell is a bare numeric label and "
"which carry at least one other non-empty cell; each opens a "
"candidate reaching the next section row or the end of the block. "
"The opposite direction from Arm E, which stops a grid rule line "
"from CLOSING a block: that arm decides how far a block extends, "
"this rule where it is cut inside, and they compose in that order. "
"Written for a spreadsheet whose whole body is one table block and "
"whose units are rows. Absent (the default) is OFF and leaves every "
"artifact byte-identical. A boolean: the run length is a module "
"constant, not a knob a caller can turn. D3 is the author's "
"definition, written for order "
"20260908T170037Z-3622420612-from-.claude; it is not defined "
"upstream, and the K3 method file does not name it either"
),
)
parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
help=(
"D3: do not admit an outline candidate whose line continues onto "
"the next one, because a wrapped sentence is not a heading. Judges "
"RECOVERED candidates only, never a heading the document declares "
"for itself. Written for quoted regulation text, whose numbered "
"paragraphs match Arm D's grammar exactly; a TITLE LENGTH rule was "
"tried first and falsified, because a real 88-character heading "
"sits between the quoted sentences at 86 and 91. Absent (the "
"default) is OFF and leaves every artifact byte-identical. It is "
"the author's definition, written for order "
"20260908T170037Z-3622420612-from-.claude; it is not defined "
"upstream, and the K3 method file does not name it either"
),
)
parser.add_argument(
"--proposed-at",
default="1970-01-01T00:00:00Z",
@ -1149,6 +1356,8 @@ def main(argv: list[str] | None = None) -> int:
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
)
except ProposerError as exc:
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)

293
tests/test_cli_build.py
View file

@ -29,6 +29,7 @@ from __future__ import annotations
import shutil
import subprocess
import sys
from collections.abc import Sequence
from dataclasses import replace
from pathlib import Path
@ -82,12 +83,16 @@ def tree(root: Path) -> dict[str, bytes]:
}
def two_script_bundle(inbox: Path, out: Path) -> Path:
def two_script_bundle(inbox: Path, out: Path, proposer_flags: Sequence[str] = ()) -> Path:
"""The path this command replaces, run as it is documented, as subprocesses.
Re-implementing the loop here would compare the CLI against a copy of
itself. Driving the actual scripts is what makes the byte comparison mean
"the old path and the new path agree".
`proposer_flags` is passed through verbatim. It exists because the BUILD
default moved and the proposer's did not: the two paths still agree, and
saying which flags make them agree is the honest form of that claim.
"""
plans = out / "plans"
plans.mkdir(parents=True, exist_ok=True)
@ -109,6 +114,7 @@ def two_script_bundle(inbox: Path, out: Path) -> Path:
relative.with_suffix("").as_posix(),
"--proposed-at",
PROPOSED_AT,
*proposer_flags,
],
capture_output=True,
check=True,
@ -545,11 +551,18 @@ def test_the_arms_reach_the_proposer_from_the_build_command(tmp_path: Path) -> N
def test_the_build_default_is_unchanged_by_the_flags_existing(tmp_path: Path) -> None:
"""The control the test above rests on: same inbox, no flags, Arm B.
"""The control the test above rests on: same inbox, no flags, the DEFAULT.
Both halves over the same fixture. Without this, a change that turned an
arm ON by default would leave the assertion above green and move every
bundle every consumer builds.
**This expectation MOVED on 2026-09-08 and the move is stated rather than
quietly rewritten.** It was written in round 2 to pin the flagless build to
Arm B, so that a change turning an arm on by default could not hide behind
the assertion above. The operator then turned two arms on, deliberately,
and the titles below are Arm D plus Arm F on this fixture.
What the test still measures is unchanged and is the reason it stays: the
flagless build has ONE declared behaviour and a drifting default moves it.
`test_each_arm_in_the_default_has_an_explicit_opt_out` holds the other end,
where Arm B's titles are still asserted, from the opt-out.
"""
inbox = inbox_for_arms(tmp_path)
plans = tmp_path / "plans-plain"
@ -564,6 +577,236 @@ def test_the_build_default_is_unchanged_by_the_flags_existing(tmp_path: Path) ->
)
== 0
)
# Arm F folds `4.1 Loesmasser` into `4 Grunnforhold`; the sheet heading is
# still carried by the table block, because `--keep-table-heading` did NOT
# move and is still a flag.
assert plan_titles(plans) == ["4 Grunnforhold", "5 Prissammenstilling"]
# A sheet whose units are ROWS: one heading, one continuous table block, a
# preamble and a run of three numbered rows. Kept apart from ARM_DOCUMENTS so
# the round-2 assertions above keep measuring what they were written for --
# `epsilon.md`'s table has ONE numbered row, which is a quantity and not a run.
SHEET_DOCUMENT = {
"zeta.md": (
"## 6 Kostnadsoversikt\n\n"
"| Skjema | | |\n|----|----|----|\n"
"| Skjemaet fylles ut i sin helhet. | | |\n"
"| 01 | Felleskostnader | |\n"
"| 02 | Bygning | |\n"
"| 07 | Utendoers | |\n"
)
}
def inbox_for_sheet_sections(root: Path) -> Path:
inbox = root / "sheet-inbox"
inbox.mkdir(parents=True, exist_ok=True)
for name, body in SHEET_DOCUMENT.items():
(inbox / name).write_text(body, encoding="utf-8", newline="")
return inbox
def test_sheet_section_rows_reaches_the_proposer_from_the_build_command(tmp_path: Path) -> None:
"""The red test for round 3: D3 exists here and it changes the plan.
Asserted on the plan's TITLES, for the reason round 2's is: a rule that
produced the right number of concepts by cutting in the wrong places would
pass a count assertion. Each title carries the row's own label, which is
what makes the concept findable by the number a reader is holding.
"""
inbox = inbox_for_sheet_sections(tmp_path)
plans = tmp_path / "plans-sheet"
assert (
build(
inbox,
tmp_path / "bundle-sheet",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
"--sheet-section-rows",
)
== 0
)
assert plan_titles(plans) == [
"6 Kostnadsoversikt",
"01 Felleskostnader",
"02 Bygning",
"07 Utendoers",
]
def test_the_build_default_leaves_a_sheet_as_one_concept(tmp_path: Path) -> None:
"""The control the test above rests on: same inbox, no flag, one concept.
Without it, a change that turned D3 on by default would leave the assertion
above green while cutting every table a consumer has ever ingested.
"""
inbox = inbox_for_sheet_sections(tmp_path)
plans = tmp_path / "plans-sheet-plain"
assert (
build(
inbox,
tmp_path / "bundle-sheet-plain",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
)
== 0
)
assert plan_titles(plans) == ["6 Kostnadsoversikt"]
# A document whose numbering is RECOVERED and whose first numbered line is a
# wrapped sentence: the shape of quoted regulation text.
WRAPPED_DOCUMENT = {
"eta.md": (
"Innledning uten nummer.\n\n"
"1 Krav og kriterier etter denne bestemmelsen skal ha som maal aa\n"
"redusere anskaffelsens samlede klimaavtrykk.\n\n"
"2 Ordinaert kapittel\n\nDette kapittelet har sin egen kropp.\n\n"
"3 Tredje kapittel\n\nOgsaa dette er en kropp.\n"
)
}
def inbox_for_wrapped_outline(root: Path) -> Path:
inbox = root / "wrapped-inbox"
inbox.mkdir(parents=True, exist_ok=True)
for name, body in WRAPPED_DOCUMENT.items():
(inbox / name).write_text(body, encoding="utf-8", newline="")
return inbox
def test_drop_wrapped_outline_reaches_the_proposer_from_the_build_command(
tmp_path: Path,
) -> None:
"""The second red test for round 3, and its control is the pair below.
The build path had four flags and the proposer now has six. A rule that
only the proposer can run is a rule no bundle is ever built with, which is
the defect round 2 fixed for the arms.
"""
inbox = inbox_for_wrapped_outline(tmp_path)
plans = tmp_path / "plans-wrapped"
assert (
build(
inbox,
tmp_path / "bundle-wrapped",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"3",
"--drop-wrapped-outline",
)
== 0
)
assert plan_titles(plans) == ["Ordinaert kapittel", "Tredje kapittel"]
def test_the_build_default_keeps_a_wrapped_outline_candidate(tmp_path: Path) -> None:
"""The control: Arm D alone recovers all three, including the sentence."""
inbox = inbox_for_wrapped_outline(tmp_path)
plans = tmp_path / "plans-wrapped-plain"
assert (
build(
inbox,
tmp_path / "bundle-wrapped-plain",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"3",
)
== 0
)
assert plan_titles(plans) == [
"Krav og kriterier etter denne bestemmelsen skal ha som maal aa",
"Ordinaert kapittel",
"Tredje kapittel",
]
# --- the default moved (operator, 2026-09-08) ------------------------------
#
# Round 3. The operator answered the standing question with alternative (b):
# `okf build` with no flag is now Arm D plus Arm F, `--outline-run 3
# --unit-fold`. Two arms and not four -- `--table-grid` and
# `--keep-table-heading` stay flags until the K2 ranking control is measured.
#
# Every arm keeps an EXPLICIT opt-out, because a default nobody can turn off is
# not a default, it is a behaviour: `--outline-run 0` for Arm D (the number was
# always its own switch) and `--no-unit-fold` for Arm F.
def test_the_build_default_is_now_arm_d_plus_arm_f(tmp_path: Path) -> None:
"""The red test for the move: no flags must EQUAL the two flags, byte for byte.
Asserted on the bundle rather than on the parser, and byte for byte rather
than on a count: a default that reached the proposer as a different number
would still produce a plausible plan, and only the bytes can tell.
"""
inbox = inbox_for_arms(tmp_path)
default = tmp_path / "default-bundle"
explicit = tmp_path / "explicit-bundle"
assert build(inbox, default, "--proposed-at", PROPOSED_AT) == 0
assert (
build(
inbox,
explicit,
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"3",
"--unit-fold",
)
== 0
)
assert tree(default) == tree(explicit)
def test_each_arm_in_the_default_has_an_explicit_opt_out(tmp_path: Path) -> None:
"""The other half: a caller can still get the pre-move behaviour, and say so.
`--outline-run 0 --no-unit-fold` must reproduce Arm B exactly. Without
this, the move would be irreversible for every consumer who needs the old
bytes, and "off by default" would have become "unreachable".
"""
inbox = inbox_for_arms(tmp_path)
opted_out = tmp_path / "opt-out-bundle"
assert (
build(
inbox,
opted_out,
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"0",
"--no-unit-fold",
)
== 0
)
plans = tmp_path / "plans-opt-out"
assert (
build(
inbox,
tmp_path / "opt-out-plans-bundle",
"--plans-dir",
str(plans),
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"0",
"--no-unit-fold",
)
== 0
)
# Arm B's own titles, the ones round 2 pinned for the flagless build.
assert plan_titles(plans) == ["4 Grunnforhold", "4.1 Loesmasser", "5 Prissammenstilling"]
@ -572,12 +815,42 @@ def test_a_bundle_built_with_no_flags_is_byte_identical_to_the_shipped_one(
) -> None:
"""The byte control, on the tree the other build tests use.
The proposer gained a parameter and the CLI gained four. A default that
moved by a byte would break rebuild-equals-incremental for every consumer
who never passes a flag, and a title assertion cannot see that.
**This expectation MOVED on 2026-09-08 and the move is stated rather than
quietly rewritten.** Until then `okf build` with no flag was Arm B, so it
equalled the two-script path run with no flag either. The operator moved
the default to Arm D plus Arm F, so the equivalence now has to name the
arms on one side or the other, and both halves are asserted:
- the two-script path with the two arms equals the new default;
- the two-script path with no arm equals `--outline-run 0 --no-unit-fold`,
which is what makes the move reversible for a consumer who needs the old
bytes.
A single half would leave the other unmeasured, and the second is the one
that goes red if an opt-out stops opting out.
"""
inbox = inbox_with_subdirectories(tmp_path)
reference = two_script_bundle(inbox, tmp_path / "reference-2")
armed = two_script_bundle(
inbox, tmp_path / "reference-armed", proposer_flags=("--outline-run", "3", "--unit-fold")
)
bundle = tmp_path / "cli-bundle-2"
assert build(inbox, bundle, "--ingested-at", INGESTED_AT, "--proposed-at", PROPOSED_AT) == 0
assert tree(bundle) == tree(reference)
assert tree(bundle) == tree(armed)
plain = two_script_bundle(inbox, tmp_path / "reference-2")
opted_out = tmp_path / "cli-bundle-opt-out"
assert (
build(
inbox,
opted_out,
"--ingested-at",
INGESTED_AT,
"--proposed-at",
PROPOSED_AT,
"--outline-run",
"0",
"--no-unit-fold",
)
== 0
)
assert tree(opted_out) == tree(plain)

94
tests/test_propose_segments.py
View file

@ -1061,7 +1061,13 @@ def test_a_non_integer_outline_run_is_refused_by_argparse(
#: Arm C is False on purpose -- its help says "not defined in the K3 method
#: file" instead, and the point of the check is that each arm's attribution
#: sits in its OWN chunk.
ARM_ATTRIBUTION = {"outline-run": True, "table-grid": True, "max-segment-chars": False}
ARM_ATTRIBUTION = {
"outline-run": True,
"table-grid": True,
"sheet-section-rows": True,
"drop-wrapped-outline": True,
"max-segment-chars": False,
}
def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
@ -1100,6 +1106,92 @@ def test_each_arm_flag_carries_its_attribution_inside_its_own_option_chunk(
assert squeezed.count("not defined upstream") == expected
# --- D3 round 3: a recovered heading that is really a wrapped sentence -----
#
# K3 position 4 quotes a regulation. Its numbered paragraphs (`1)`, `3)`, `4)`,
# `5)`) match Arm D's grammar exactly, so the arm recovers four "chapters" from
# text the operator wrote off in one line: "sitert forskriftstekst er ikke
# overskrifter". Round 2 tried to separate them by TITLE LENGTH and falsified
# it: a real chapter heading in the same sample is 88 characters and the
# quoted sentences are 86, 91, 92 and 100.
#
# The axis below is a different one and it is not about size. A heading is a
# complete line; a wrapped sentence continues on the next one. Measured over
# the 12-document sample: 8 of 34 outline candidates are followed by a line
# beginning in lower case -- the four quoted paragraphs at position 4 and four
# table rows at position 9, where the operator wrote "radene er ikke
# overskrifter". 0 of the 26 the operator kept.
#
# It judges RECOVERED candidates only. A dotted or ATX heading is the
# document's own claim about itself, and overruling that is a different
# decision from declining to invent one.
WRAPPED_OUTLINE = """Innledning til dokumentet uten nummer.
1 Krav og kriterier etter denne bestemmelsen skal ha som maal aa redusere
anskaffelsens samlede klimaavtrykk.
2 Ordinaert kapittel
Dette kapittelet har en kropp som begynner med stor forbokstav.
3 Tredje kapittel
Ogsaa dette er en kropp.
"""
DECLARED_HEADING_THAT_WRAPS = """## 2.1 Et erklaert kapittel
som fortsetter paa neste linje i samme setning.
## 2.2 Et annet kapittel
Kropp under det andre kapittelet.
"""
def test_a_wrapped_outline_candidate_is_not_a_heading() -> None:
"""The red test: the recovered candidate whose line wraps is dropped.
Only that one. The two whose next line opens a new sentence stay, which is
what keeps the rule from being "discard Arm D" wearing a flag.
"""
off = okf_propose_segments.find_candidates(WRAPPED_OUTLINE, outline_run=3)
assert [c.title for c in off] == [
"Krav og kriterier etter denne bestemmelsen skal ha som maal aa redusere",
"Ordinaert kapittel",
"Tredje kapittel",
]
on = okf_propose_segments.find_candidates(
WRAPPED_OUTLINE, outline_run=3, drop_wrapped_outline=True
)
assert [c.title for c in on] == ["Ordinaert kapittel", "Tredje kapittel"]
def test_a_declared_heading_that_wraps_is_left_alone() -> None:
"""The known-negative, and it is the rule's whole scope statement.
A document that numbers its own sections has declared them. The rule
exists because Arm D RECOVERS a numbering the document never declared, so
a recovered candidate that turns out to be a sentence was never a heading;
a declared one is a claim this tool does not get to overrule.
"""
off = okf_propose_segments.find_candidates(DECLARED_HEADING_THAT_WRAPS)
on = okf_propose_segments.find_candidates(
DECLARED_HEADING_THAT_WRAPS, drop_wrapped_outline=True
)
assert [c.title for c in off] == ["2.1 Et erklaert kapittel", "2.2 Et annet kapittel"]
assert off == on
def test_the_wrapped_rule_is_off_by_default() -> None:
"""The default control: same text, no flag, Arm D's own output."""
default = okf_propose_segments.find_candidates(WRAPPED_OUTLINE, outline_run=3)
explicit = okf_propose_segments.find_candidates(
WRAPPED_OUTLINE, outline_run=3, drop_wrapped_outline=False
)
assert default == explicit
assert len(default) == 3
# --- Arm E: the grid-rule join --------------------------------------------
#
# Arm E is not defined in `docs/2026-09-02-k3-k4-k5-metode.md` -- that file

194
tests/test_propose_sheet_sections.py Normal file
View file

@ -0,0 +1,194 @@
"""D3 round 3: section rows INSIDE a sheet's one table block, behind a flag.
The spreadsheet is the one file type no arm has ever moved. Its whole body
extracts as ONE continuous pipe-table block -- one heading and 101 rows -- so
every rule this module has proposes exactly one boundary for it, and the
operator's worksheet asks for one concept per numbered cost group. Those groups
are ROWS, not headings, which is why no heading rule can reach them and why
`--keep-table-heading` moved the first byte without moving the count.
The rule tested here is the opposite direction from `--table-grid`: that one
stops a converter's rule line from CLOSING a block, so one grid table proposes
one candidate instead of many; this one CUTS an open block at the rows that
label its sections. They read different grammars (a `+---+` rule line against a
pipe row's first cell) and compose in one order -- grid decides how far a block
extends, section rows decide where it is cut inside.
Generality is the whole point, so the fixtures below carry no word list and no
knowledge of any real document's labels: what makes a row a section row is that
its first cell is a bare numeric label and that it is one of a RUN of such rows.
The run is the same bounding device `CONTENTS_RUN` already uses in this module,
for the same reason: a single numbered row is a quantity, not a section.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from llm_ingestion_okf import propose as okf_propose_segments
# One heading, one continuous table block, a preamble, a run of four numbered
# section rows, and a summing tail. The shape of a priced sheet, written from
# the grammar rather than copied from one.
SECTIONED_SHEET = """## Kostnadsoversikt {#sheet-1}
| Skjema | | |
|----|----|----|
| Skjemaet fylles ut i sin helhet. | | |
| Post | | SUM |
| 01 | Felleskostnader | |
| 11+12 | Rigging og drift | |
| 02 | Bygning | |
| 07 | Utendoers | |
| Tilbudt fastpris eksklusive avgift | | 100 |
"""
# The same block with the numbered run removed. Nothing here is a section row,
# so the flag must leave it exactly as it found it.
UNSECTIONED_SHEET = """## Kostnadsoversikt {#sheet-1}
| Skjema | | |
|----|----|----|
| Skjemaet fylles ut i sin helhet. | | |
| Post | | SUM |
| Felleskostnader | | 10 |
| Rigging og drift | | 20 |
| Tilbudt fastpris eksklusive avgift | | 100 |
"""
# A single row whose first cell is a number -- a computation basis, not a
# section. This is the known-negative the run threshold exists for, and it is
# the shape that actually occurs: a sheet states a quantity on its own row.
LONE_NUMERIC_ROW = """## Regningsarbeider {#sheet-2}
| Grunnlag | | |
|----|----|----|
| Beregningsgrunnlag massehaandtering | tonn | 16000 |
| Paaslag i prosent | | 10 |
"""
def test_a_run_of_numbered_rows_cuts_the_block_into_sections() -> None:
"""The rule, red first: one candidate per section row, named from the row.
Default: the whole sheet is ONE table block whose title is inherited from
the orphaned heading. On: the preamble keeps that block and each numbered
row opens its own candidate, running to the next section row or to the end
of the block -- so the LAST section carries the sheet's tail, which is the
honest consequence of a rule that cuts and never discards.
"""
off = okf_propose_segments.find_candidates(SECTIONED_SHEET)
assert [(c.title, c.rule) for c in off] == [
("Kostnadsoversikt {#sheet-1}", okf_propose_segments.RULE_TABLE_BLOCK)
]
on = okf_propose_segments.find_candidates(SECTIONED_SHEET, sheet_section_rows=True)
assert [(c.number, c.title, c.rule) for c in on] == [
(None, "Kostnadsoversikt {#sheet-1}", okf_propose_segments.RULE_TABLE_BLOCK),
("01", "01 Felleskostnader", okf_propose_segments.RULE_SHEET_SECTION),
("11+12", "11+12 Rigging og drift", okf_propose_segments.RULE_SHEET_SECTION),
("02", "02 Bygning", okf_propose_segments.RULE_SHEET_SECTION),
("07", "07 Utendoers", okf_propose_segments.RULE_SHEET_SECTION),
]
# The preamble block ends where the first section opens, and the last
# section reaches the end of the text.
assert on[0].end == on[1].start
assert on[1].end == on[2].start
assert on[-1].end == len(SECTIONED_SHEET)
# The tail row is inside the last section rather than in no concept at all.
assert "Tilbudt fastpris" in SECTIONED_SHEET[on[-1].start : on[-1].end]
def test_a_section_row_survives_the_orphan_check() -> None:
"""A stated exception, because without it the rule cannot fire at all.
The orphan check reads the lines UNDER a candidate's first line, which is
right for a heading -- a heading with nothing under it names nothing. A
section row carries its content in its own cells, so every one-row section
is bodiless by that test and all but the last would be dropped. The
exception is written once, on the rule, and it is what the count below
proves.
"""
on = okf_propose_segments.find_candidates(SECTIONED_SHEET, sheet_section_rows=True)
single_row = [c for c in on if c.rule == okf_propose_segments.RULE_SHEET_SECTION][:3]
assert len(single_row) == 3
for candidate in single_row:
assert len(SECTIONED_SHEET[candidate.start : candidate.end].splitlines()) == 1
def test_a_table_without_section_rows_is_untouched() -> None:
"""First known-negative: identical objects, not merely an equal count."""
off = okf_propose_segments.find_candidates(UNSECTIONED_SHEET)
on = okf_propose_segments.find_candidates(UNSECTIONED_SHEET, sheet_section_rows=True)
assert len(off) == 1
assert off == on
def test_a_lone_numbered_row_is_not_a_section() -> None:
"""Second known-negative, and the reason the rule counts a RUN.
`16000` is a quantity. A rule that read one numbered cell as a section
would cut a sheet at every stated basis, which is the ungated widening this
module has already refused once for outline candidates.
"""
off = okf_propose_segments.find_candidates(LONE_NUMERIC_ROW)
on = okf_propose_segments.find_candidates(LONE_NUMERIC_ROW, sheet_section_rows=True)
assert len(off) == 1
assert off == on
def test_the_flag_composes_with_keep_table_heading() -> None:
"""Both on: the heading keeps the preamble, the sections stay separate.
D1 decides where the FIRST concept starts; this rule decides where the
block is cut. They read different things and neither disables the other,
which is what makes the pair safe to expose as two flags rather than one.
"""
on = okf_propose_segments.find_candidates(
SECTIONED_SHEET, sheet_section_rows=True, keep_table_heading=True
)
assert [(c.number, c.rule) for c in on] == [
(None, okf_propose_segments.RULE_HEADING),
("01", okf_propose_segments.RULE_SHEET_SECTION),
("11+12", okf_propose_segments.RULE_SHEET_SECTION),
("02", okf_propose_segments.RULE_SHEET_SECTION),
("07", okf_propose_segments.RULE_SHEET_SECTION),
]
assert on[0].start == 0
assert on[0].end == on[1].start
def test_the_rule_names_itself_in_derived() -> None:
"""A proposal an operator disagrees with is traceable to the rule."""
plan = okf_propose_segments.build_plan(
Path("ark.xlsx"),
SECTIONED_SHEET,
b"ark",
okf_type="requirement",
proposed_at="1970-01-01T00:00:00Z",
sheet_section_rows=True,
)
sections = [e for e in plan["entries"] if e["title"].startswith("01")]
assert len(sections) == 1
assert sections[0]["derived"] == [
okf_propose_segments.PROPOSED_MARKER,
okf_propose_segments.RULE_SHEET_SECTION,
]
# The label becomes the directory and is stripped from the stem, the same
# way a section number does for a heading.
assert sections[0]["path"] == "01/felleskostnader.md"
def test_sheet_section_rows_takes_no_argument(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A boolean at the CLI. The rule's one number is a module constant."""
source = tmp_path / "ark.md"
source.write_text(SECTIONED_SHEET, encoding="utf-8")
out = tmp_path / "ark.json"
with pytest.raises(SystemExit) as exit_info:
okf_propose_segments.main([str(source), "--out", str(out), "--sheet-section-rows", "3"])
assert exit_info.value.code == 2
assert "usage:" in capsys.readouterr().err

48
tests/test_provenance.py
View file

@ -28,6 +28,7 @@ import pytest
from llm_ingestion_okf.extract import extract_text, source_units
from llm_ingestion_okf.inbox import render_inbox_concept
from llm_ingestion_okf.propose import RULE_SHEET_SECTION, find_candidates
from llm_ingestion_okf.profiles import (
DEFAULT,
SEGMENTED_OKF_V0_2,
@ -229,6 +230,53 @@ def test_a_spreadsheet_concept_names_the_sheet_and_its_rows() -> None:
assert "source_rows: [1, 1]\n" in document
@requires_extract
def test_each_sheet_section_names_its_own_rows() -> None:
"""D3: a section's locator is the section's rows, not the sheet's.
The rule cuts one table block into one candidate per numbered row, so the
provenance layer is what decides whether those concepts are addressable at
all. It reads the unit table built AT EXTRACTION and the candidate's own
span, and neither knows about the rule — which is exactly why this has to
be measured rather than assumed: a locator that reported the whole sheet
for every section would look right in the frontmatter and point at nothing.
"""
data, text = _extract("prisark.xlsx")
units = source_units("prisark.xlsx", data, text)
candidates = find_candidates(text, sheet_section_rows=True)
sections = [c for c in candidates if c.rule == RULE_SHEET_SECTION]
assert [c.number for c in sections] == ["01", "02", "03", "04"]
located = []
for candidate in sections:
document = render_inbox_concept(
text,
okf_type="reference",
title=candidate.title,
source_file="prisark.xlsx",
source_bytes=data,
ingested_at="2026-09-08T00:00:00Z",
profile=SEGMENTED_OKF_V0_2,
units=units,
span=(candidate.start, candidate.end),
)
assert "source_sheet: Prisark\n" in document
located.append(
next(line for line in document.splitlines() if line.startswith("source_rows:"))
)
# Four sections, four DIFFERENT row locators, ascending, and each one is
# the workbook's own row number rather than a line in the extracted text.
# The last section stops at the second sheet's heading, which is the next
# mark: a span crossing two sheets would get no sheet and no rows at all,
# and the fixture is what proves it does not.
assert located == [
"source_rows: [3, 3]",
"source_rows: [4, 4]",
"source_rows: [5, 5]",
"source_rows: [6, 6]",
]
@requires_extract
def test_a_range_spanning_two_sheets_names_no_sheet_and_no_rows() -> None:
# A row number is only a place in the original once a sheet is named. A