docs(cli): the new default's K2 digest, measured on the bundle it produces

The default move landed in the previous commit; this is the number it costs a
consumer, measured rather than predicted. `okf build` with no flag over the
43-document reference corpus now yields 517 concepts in 969 files, content
digest `42c8646d...`, `merged + coded rejections = 43 = N`, 817.6 s. The
delivered 2026-09-03 bundle (629 concepts, 1108 files, `9cd74519...`) is
unchanged on disk and reproducing it needs `--outline-run 0 --no-unit-fold`
AND the stamp it was built with -- "the opt-out flags" alone is true and
incomplete, so the report says both.

README and CLAUDE.md now state which two rules are on, what each opt-out is,
and that the proposer's own defaults did not move. One CLAUDE.md sentence was
false as of the move ("a no-flag build stays byte-identical") and is corrected
in place rather than deleted: adding a flag still leaves the default
byte-identical; moving it does not, which is why it took an operator decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 21:14:51 +02:00
commit efa92ce170
3 changed files with 63 additions and 13 deletions

View file

@ -259,16 +259,32 @@ and fixtures, never code.
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
constant rather than the clock, because a wall-clock default takes
rebuild-equals-incremental away from anyone who omits them.
**Six segmentation rules are REACHABLE here and all six are OFF by
default** (2026-09-08): `--outline-run N` (Arm D), `--table-grid` (Arm E),
**Six segmentation rules are REACHABLE here, and since 2026-09-08 TWO of
them are ON by default** -- `--outline-run 3` and `--unit-fold`, the
operator's D3 answer (alternative b), each with an explicit opt-out
(`--outline-run 0`, `--no-unit-fold`) that together reproduce the pre-move
bytes. **The proposer's own defaults did NOT move** (`propose.py`'s rules stay
off): the goldens and every published reproduction block are pinned to them,
so the two layers disagree on purpose and `cli.DEFAULT_OUTLINE_RUN` /
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
(`9cd74519...`, the delivered 2026-09-03 tree) to **517 / 969**
(`42c8646d...`, default flags, default epoch stamp). **And the number the
decision cites belongs to another configuration:** Arm F's 5 of 12 was
measured with `--table-grid` ON; without it the same sample scores 2 of 12
and `docx` 0 of 3, because the fold's table clause has no joined table to
fold. The six (2026-09-08): `--outline-run N` (Arm D), `--table-grid` (Arm E),
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
and `--drop-wrapped-outline` (both D3), each passed to the proposer
unchanged. Until that day the build path called the proposer with no
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
as one -- a build path a full arm behind the proposer. Exposing them is not
the same decision as moving one: **which arm should ship as the default is
the operator's, and a no-flag build stays byte-identical** (measured by
digest before and after, and by Arm E over all 43 corpus documents). Arm C
as one -- a build path a full arm behind the proposer. Exposing them was not
the same decision as moving one, and the two were taken a session apart:
**which arm ships as the default is the operator's**, answered 2026-09-08 as
above. Adding a flag still leaves the default byte-identical (measured by
digest before and after, and by Arm E over all 43 corpus documents); MOVING
the default is the one thing that does not, which is why it took an operator
decision and carries an opt-out. Arm C
(`--max-segment-chars`) stays unexposed: no reference has ever been measured
for its cap. The two D3 rules read grammars nothing else here reads: a table
row's FIRST CELL (a run of bare numeric labels cuts the block that holds

View file

@ -82,17 +82,24 @@ did not pass them.
### The segmentation flags
Six rules are reachable from `okf build` and **all six are off by default**;
a build with none of them is byte-identical to every bundle published here. Each
line below carries the number it was measured at, and nothing beyond it — none
of them is a recommendation, because which one should become the default has
not been decided.
Six rules are reachable from `okf build`. **Two of them are ON by default since
2026-09-08** — `--outline-run 3` and `--unit-fold`, an operator decision — and
each has an explicit opt-out, `--outline-run 0` and `--no-unit-fold`. Passing
both opt-outs reproduces the pre-2026-09-08 bytes exactly. The other four are
off. Each line below carries the number it was measured at, and nothing beyond
it.
**A re-run is what this costs a consumer, and it is not a small one:** on the
43-document reference corpus the default bundle goes from **629 concepts in
1108 files** to **517 in 969**. The proposer's own defaults
(`tools/okf_propose_segments.py`) did NOT move, so every published reproduction
block still runs as written.
| flag | what it does | measured |
|---|---|---|
| `--outline-run N` | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N` | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
| `--table-grid` | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
| `--unit-fold` | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match, against **2** for the default, no cell worse. Whole corpus: 679 → 485 proposed segments over 39 readable documents |
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
| `--drop-wrapped-outline` | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept |

View file

@ -363,6 +363,33 @@ tender PDF is 9 either way, which was the other half of that check.
The default is the operator's decision and it is implemented as given. The
measurement above is the report back, not an argument for reversing it.
### The K2 bundle on the new default, and what `9cd74519…` still names
The injection asks for the K2 digest re-measured on the new default and written
down as the current one. Three bundles were built this session, all from the
same 43-document corpus, all with `--bundle-id k2-trinn1-20260903 --okf-version
0.2` and the **default epoch stamp** (`1970-01-01T00:00:00Z`), which is why
none of them can be compared to the delivered bundle's digest directly:
| build | flags | concepts | files | content digest |
|---|---|---|---|---|
| the delivered bundle, 2026-09-03 | two-script path, `--ingested-at 2026-09-03T00:00:00Z` | 629 | 1108 | `9cd74519…` |
| Arm B control | `--outline-run 0 --no-unit-fold` equivalent | 629 | 1108 | `f09035d9…` |
| the D1 pair's other half | `--keep-table-heading` | 629 | 1108 | `432ca424…` |
| **the new default** | none | **517** | **969** | **`42c8646d…`** |
`merged + coded rejections = 43 = N` on all three, exit 0.
**`9cd74519…` names a bundle, not a configuration.** It is the delivered
2026-09-03 tree and it is unchanged on disk, which is the control this session
re-ran. Reproducing it needs the stamp it was built with as well as the
opt-out flags, so "it now holds only under `--outline-run 0 --no-unit-fold`" is
true and incomplete: it also needs `--ingested-at 2026-09-03T00:00:00Z`. The
digest a consumer should compare against from today is `42c8646d…`, for the
command in the table above, and the 629 → 517 concept move is what a re-run
costs them.
## 9. Tests
`1397 → 1414`. Seventeen new, and which of them were red first is stated rather