open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(cli): the new default's K2 digest, measured on the bundle it produces

Browse source 
The default move landed in the previous commit; this is the number it costs a
consumer, measured rather than predicted. `okf build` with no flag over the
43-document reference corpus now yields 517 concepts in 969 files, content
digest `42c8646d...`, `merged + coded rejections = 43 = N`, 817.6 s. The
delivered 2026-09-03 bundle (629 concepts, 1108 files, `9cd74519...`) is
unchanged on disk and reproducing it needs `--outline-run 0 --no-unit-fold`
AND the stamp it was built with -- "the opt-out flags" alone is true and
incomplete, so the report says both.

README and CLAUDE.md now state which two rules are on, what each opt-out is,
and that the proposer's own defaults did not move. One CLAUDE.md sentence was
false as of the move ("a no-flag build stays byte-identical") and is corrected
in place rather than deleted: adding a flag still leaves the default
byte-identical; moving it does not, which is why it took an operator decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 21:14:51 +02:00
commit efa92ce170
3 changed files with 63 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

21
README.md
View file

@ -82,17 +82,24 @@ did not pass them.
### The segmentation flags
Six rules are reachable from `okf build` and **all six are off by default**;
a build with none of them is byte-identical to every bundle published here. Each
line below carries the number it was measured at, and nothing beyond it — none
of them is a recommendation, because which one should become the default has
not been decided.
Six rules are reachable from `okf build`. **Two of them are ON by default since
2026-09-08** — `--outline-run 3` and `--unit-fold`, an operator decision — and
each has an explicit opt-out, `--outline-run 0` and `--no-unit-fold`. Passing
both opt-outs reproduces the pre-2026-09-08 bytes exactly. The other four are
off. Each line below carries the number it was measured at, and nothing beyond
it.
**A re-run is what this costs a consumer, and it is not a small one:** on the
43-document reference corpus the default bundle goes from **629 concepts in
1108 files** to **517 in 969**. The proposer's own defaults
(`tools/okf_propose_segments.py`) did NOT move, so every published reproduction
block still runs as written.
| flag | what it does | measured |
|---|---|---|
| `--outline-run N` | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N` | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
| `--table-grid` | a pandoc grid-table rule line no longer closes an open table block, so one grid table is one concept | a `.docx` experience list: **21 → 6** concepts |
| `--unit-fold` | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match, against **2** for the default, no cell worse. Whole corpus: 679 → 485 proposed segments over 39 readable documents |
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
| `--drop-wrapped-outline` | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept |