feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment

One rule explains every remaining `pdf` miss on the twelve-position reference:
where a document DECLARES headings, Arm D's RECOVERED headings are the whole of
the excess, and every declared one is a unit the reference wants. `--outline-gate`
admits recovery only where the document declares none of its own, plus any one
recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units`
clause 2's own principle moved from voting to admission, and it filters at
ADMISSION so the text a removed mark opened is carried by the mark above it --
the post-filter form scores identically on all twelve positions and loses that
text, which is why only one of them shipped.

`--outline-gate` and `--drop-wrapped-outline` become the package default, one
decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8
alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of
3. Each keeps an explicit opt-out. The bar the move had to clear was not the
reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-,
no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of
12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2.
Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810.

DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum
measures size and a density is diluted by every unit carrying none of the
question, so a document split 1 -> 12 lost its prior by 12. Swept over five
values on 18 rows it is at least as good as the delivered density everywhere and
strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row
on any of four bundles, so it did not solve the knot it was adopted for -- what
did is that the `pdf` gain never needed `--sheet-section-rows`.

`--first-span-from-zero` is off and repairs a measured loss found while chasing
one position's 940 characters: 32 of the 32 documents that get a plan leave the
text above their first concept in no segment -- 159 704 characters, 9.18 % of
the corpus, 45 841 from one document. It changes nothing on the reference. Off
because it moves the first span of essentially every bundle with no hit@8 number
behind it yet.

vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every
concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads`
carries it for the SEGMENTED profiles, read-only, after the emitted pattern
misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2).

vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1,
`bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`,
keyword-only, rendered before any disk mutation, written only when the index is
created -- Door B's mechanism and ordering.

Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md.
Suite 1478 passed (1449 before), ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 14:17:00 +02:00
commit 38104b7df5
16 changed files with 1301 additions and 42 deletions

View file

@ -88,19 +88,30 @@ did not pass them.
### The segmentation flags
Six rules are reachable from `okf build`. **Three of them are ON by default
since 2026-09-08** — `--outline-run 3`, `--table-grid` and `--unit-fold`, an
operator decision taken in two steps — and each has an explicit opt-out,
`--outline-run 0`, `--no-table-grid` and `--no-unit-fold`. Passing all three
opt-outs reproduces the pre-2026-09-08 bytes exactly. The other three are off.
Each line below carries the number it was measured at, and nothing beyond it.
Eight rules are reachable from `okf build`. **Five of them are ON by default**
`--outline-run 3`, `--table-grid` and `--unit-fold` since 2026-09-08, and
`--drop-wrapped-outline` and `--outline-gate` since 2026-09-09 — each an
operator decision, and each with an explicit opt-out: `--outline-run 0`,
`--no-table-grid`, `--no-unit-fold`, `--keep-wrapped-outline`,
`--no-outline-gate`. Passing all five reproduces the pre-2026-09-08 bytes
exactly. The other three are off. Each line below carries the number it was
measured at, and nothing beyond it.
**A re-run is what this costs a consumer, and it is not a small one:** on the
43-document reference corpus the default bundle goes from **629 concepts in
1108 files** to **517 in 969**. The proposer's own defaults
1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move
and to **425 in 810** after the 2026-09-09 one. The proposer's own defaults
(`tools/okf_propose_segments.py`) did NOT move, so every published reproduction
block still runs as written.
**What the 2026-09-09 move had to clear, stated because it is the bar every
later move is held to:** the twelve-position reference improves (`pdf` **2 of 8
→ 7 of 8**, the sheet **5 of 12 → 10 of 12**, `docx` unchanged at 3 of 3) AND
hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,**, with
no row losing rank 1. A configuration that improved the reference and cost a
rank was measured in the same session and did NOT ship; see
`docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md` § 9.1.
| flag | what it does | measured |
|---|---|---|
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
@ -108,7 +119,9 @@ block still runs as written.
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
| `--drop-wrapped-outline` | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept |
| `--drop-wrapped-outline` (**on** by default since 2026-09-09; opt out with `--keep-wrapped-outline`) | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept. On the reference it carries `pdf` from **5 of 8 to 6 of 8** together with the gate below, and neither reaches 7 of 8 without the other |
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
| `--first-span-from-zero` | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | **off by default.** Measured over the 39-document corpus: **32 of the 32** documents that get a plan lose their opening today — **159 704 characters, 9.18 %** of the corpus, 45 841 from one document. On the 12-position reference it changes nothing (10 of 12 either way). It is off because it moves the first span of essentially every bundle and no hit@8 number has been measured behind it yet |
They compose, and the order above is the order they apply in. Measured on a
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per