feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment
One rule explains every remaining `pdf` miss on the twelve-position reference: where a document DECLARES headings, Arm D's RECOVERED headings are the whole of the excess, and every declared one is a unit the reference wants. `--outline-gate` admits recovery only where the document declares none of its own, plus any one recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units` clause 2's own principle moved from voting to admission, and it filters at ADMISSION so the text a removed mark opened is carried by the mark above it -- the post-filter form scores identically on all twelve positions and loses that text, which is why only one of them shipped. `--outline-gate` and `--drop-wrapped-outline` become the package default, one decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of 3. Each keeps an explicit opt-out. The bar the move had to clear was not the reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-, no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of 12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2. Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810. DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum measures size and a density is diluted by every unit carrying none of the question, so a document split 1 -> 12 lost its prior by 12. Swept over five values on 18 rows it is at least as good as the delivered density everywhere and strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row on any of four bundles, so it did not solve the knot it was adopted for -- what did is that the `pdf` gain never needed `--sheet-section-rows`. `--first-span-from-zero` is off and repairs a measured loss found while chasing one position's 940 characters: 32 of the 32 documents that get a plan leave the text above their first concept in no segment -- 159 704 characters, 9.18 % of the corpus, 45 841 from one document. It changes nothing on the reference. Off because it moves the first span of essentially every bundle with no hit@8 number behind it yet. vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads` carries it for the SEGMENTED profiles, read-only, after the emitted pattern misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2). vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1, `bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`, keyword-only, rendered before any disk mutation, written only when the index is created -- Door B's mechanism and ordering. Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md. Suite 1478 passed (1449 before), ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
b01492b7f5
commit
38104b7df5
16 changed files with 1301 additions and 42 deletions
29
README.md
29
README.md
|
|
@ -88,19 +88,30 @@ did not pass them.
|
|||
|
||||
### The segmentation flags
|
||||
|
||||
Six rules are reachable from `okf build`. **Three of them are ON by default
|
||||
since 2026-09-08** — `--outline-run 3`, `--table-grid` and `--unit-fold`, an
|
||||
operator decision taken in two steps — and each has an explicit opt-out,
|
||||
`--outline-run 0`, `--no-table-grid` and `--no-unit-fold`. Passing all three
|
||||
opt-outs reproduces the pre-2026-09-08 bytes exactly. The other three are off.
|
||||
Each line below carries the number it was measured at, and nothing beyond it.
|
||||
Eight rules are reachable from `okf build`. **Five of them are ON by default**
|
||||
— `--outline-run 3`, `--table-grid` and `--unit-fold` since 2026-09-08, and
|
||||
`--drop-wrapped-outline` and `--outline-gate` since 2026-09-09 — each an
|
||||
operator decision, and each with an explicit opt-out: `--outline-run 0`,
|
||||
`--no-table-grid`, `--no-unit-fold`, `--keep-wrapped-outline`,
|
||||
`--no-outline-gate`. Passing all five reproduces the pre-2026-09-08 bytes
|
||||
exactly. The other three are off. Each line below carries the number it was
|
||||
measured at, and nothing beyond it.
|
||||
|
||||
**A re-run is what this costs a consumer, and it is not a small one:** on the
|
||||
43-document reference corpus the default bundle goes from **629 concepts in
|
||||
1108 files** to **517 in 969**. The proposer's own defaults
|
||||
1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move
|
||||
and to **425 in 810** after the 2026-09-09 one. The proposer's own defaults
|
||||
(`tools/okf_propose_segments.py`) did NOT move, so every published reproduction
|
||||
block still runs as written.
|
||||
|
||||
**What the 2026-09-09 move had to clear, stated because it is the bar every
|
||||
later move is held to:** the twelve-position reference improves (`pdf` **2 of 8
|
||||
→ 7 of 8**, the sheet **5 of 12 → 10 of 12**, `docx` unchanged at 3 of 3) AND
|
||||
hit@8 on a K2 bundle built with it holds **5 of 6 at ranks 1,1,1,1,1,–**, with
|
||||
no row losing rank 1. A configuration that improved the reference and cost a
|
||||
rank was measured in the same session and did NOT ship; see
|
||||
`docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md` § 9.1.
|
||||
|
||||
| flag | what it does | measured |
|
||||
|---|---|---|
|
||||
| `--outline-run N` (default **3**) | also propose a boundary where the document's own bare-integer numbering sustains an ascending run of at least `N`; `0` is this arm's opt-out | a tender PDF whose headings are bare integers: **no boundary** at `0`, **9 concepts** at `3`, against a reference of 9 |
|
||||
|
|
@ -108,7 +119,9 @@ block still runs as written.
|
|||
| `--unit-fold` (**on** by default; opt out with `--no-unit-fold`) | discard a contents-list run, fold a deeper heading into its parent, fold a table into the shorter heading that introduces it. Adds no boundary, so it can only reduce a plan | on a 12-document sample scored against an operator's unit worksheet: **5 of 12** match — but that figure was measured with `--table-grid` ON, and the shipped default does not include it. Measured without it the same sample scores **2 of 12**, `docx` **0 of 3**, because the fold's table clause has no joined table to fold |
|
||||
| `--keep-table-heading` | keep a heading whose body is empty only because a table opens under it, and absorb that table into its span | the two spreadsheets in that corpus, and **0 of 32 `pdf` and 0 of 5 `docx`**: the concept count does not move (1 → 1), its first byte does — the concept gains the heading line it was missing |
|
||||
| `--sheet-section-rows` | cut an open table block at the rows that label its sections — a run of at least three rows whose first cell is a bare numeric label. The opposite direction from `--table-grid`, which decides how far a block extends | a tender price sheet whose whole body is one table block: **1 → 12 concepts**, against a reference of 11 cost groups plus the sheet's preamble. Whole corpus: **1 of 39** readable documents changes, **0 of 32 `pdf`, 0 of 5 `docx`, 1 of 2 `xlsx`** |
|
||||
| `--drop-wrapped-outline` | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept |
|
||||
| `--drop-wrapped-outline` (**on** by default since 2026-09-09; opt out with `--keep-wrapped-outline`) | do not admit an `--outline-run` candidate whose line continues onto the next one. Judges recovered candidates only, never a heading the document declares | quoted regulation text, whose numbered paragraphs match the outline grammar exactly: **4 → 1 concepts**, the reference. Whole corpus: **5 of 39**, all `pdf`; on the 12-document sample **8 of 34** outline candidates wrap, and none of the 26 the operator kept. On the reference it carries `pdf` from **5 of 8 to 6 of 8** together with the gate below, and neither reaches 7 of 8 without the other |
|
||||
| `--outline-gate` (**on** by default since 2026-09-09; opt out with `--no-outline-gate`) | admit `--outline-run`'s RECOVERED headings only where the document declares none of its own, plus any one recovered heading whose span covers `OUTLINE_SHARE` (0.20) of the text. Applied at admission, before spans are closed, so the text a removed mark opened is carried by the mark above it rather than lost | on the 12-document sample: `pdf` **2 of 8 → 5 of 8** alone and **7 of 8** with the rule above, `docx` unchanged at **3 of 3**. Whole corpus: it fires on **25 of 39** readable documents, changes the plan in **15 of 39**, and removes **64 of 485** proposed entries. No plan disappears (32 → 32) |
|
||||
| `--first-span-from-zero` | start the first concept at character 0, so the text above it belongs to a segment instead of to none. Adds no boundary and removes none | **off by default.** Measured over the 39-document corpus: **32 of the 32** documents that get a plan lose their opening today — **159 704 characters, 9.18 %** of the corpus, 45 841 from one document. On the 12-position reference it changes nothing (10 of 12 either way). It is off because it moves the first span of essentially every bundle and no hit@8 number has been measured behind it yet |
|
||||
|
||||
They compose, and the order above is the order they apply in. Measured on a
|
||||
five-document tender folder (2 `pdf`, 2 `docx`, 1 `xlsx`), concepts per
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue