feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment

One rule explains every remaining `pdf` miss on the twelve-position reference:
where a document DECLARES headings, Arm D's RECOVERED headings are the whole of
the excess, and every declared one is a unit the reference wants. `--outline-gate`
admits recovery only where the document declares none of its own, plus any one
recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units`
clause 2's own principle moved from voting to admission, and it filters at
ADMISSION so the text a removed mark opened is carried by the mark above it --
the post-filter form scores identically on all twelve positions and loses that
text, which is why only one of them shipped.

`--outline-gate` and `--drop-wrapped-outline` become the package default, one
decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8
alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of
3. Each keeps an explicit opt-out. The bar the move had to clear was not the
reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-,
no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of
12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2.
Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810.

DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum
measures size and a density is diluted by every unit carrying none of the
question, so a document split 1 -> 12 lost its prior by 12. Swept over five
values on 18 rows it is at least as good as the delivered density everywhere and
strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row
on any of four bundles, so it did not solve the knot it was adopted for -- what
did is that the `pdf` gain never needed `--sheet-section-rows`.

`--first-span-from-zero` is off and repairs a measured loss found while chasing
one position's 940 characters: 32 of the 32 documents that get a plan leave the
text above their first concept in no segment -- 159 704 characters, 9.18 % of
the corpus, 45 841 from one document. It changes nothing on the reference. Off
because it moves the first span of essentially every bundle with no hit@8 number
behind it yet.

vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every
concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads`
carries it for the SEGMENTED profiles, read-only, after the emitted pattern
misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2).

vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1,
`bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`,
keyword-only, rendered before any disk mutation, written only when the index is
created -- Door B's mechanism and ordering.

Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md.
Suite 1478 passed (1449 before), ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 14:17:00 +02:00
commit 38104b7df5
16 changed files with 1301 additions and 42 deletions

View file

@ -266,11 +266,27 @@ and fixtures, never code.
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
constant rather than the clock, because a wall-clock default takes
rebuild-equals-incremental away from anyone who omits them.
**Six segmentation rules are REACHABLE here, and since 2026-09-08 THREE of
them are ON by default** -- `--outline-run 3`, `--table-grid` and
`--unit-fold`, each with an explicit opt-out (`--outline-run 0`,
`--no-table-grid`, `--no-unit-fold`) that together reproduce the pre-move
bytes. Arm E joined a session after the other two, on a number measured
**Eight segmentation rules are REACHABLE here, and FIVE of them are ON by
default** -- `--outline-run 3`, `--table-grid` and `--unit-fold` since
2026-09-08, `--drop-wrapped-outline` and `--outline-gate` since 2026-09-09,
each with an explicit opt-out (`--outline-run 0`, `--no-table-grid`,
`--no-unit-fold`, `--keep-wrapped-outline`, `--no-outline-gate`) that
together reproduce the pre-move bytes. **The 2026-09-09 pair is one
decision and cannot be split**: the gate takes `pdf` from 2 of 8 to 5 of 8
and the pair takes it to 7 of 8 (the sheet 5 of 12 -> 10 of 12, `docx`
unchanged at 3 of 3). **The gate is G1+G2:** Arm D's RECOVERED headings are
admitted only where the document DECLARES none of its own -- which is
`fold_units` clause 2's principle moved from voting to admission -- plus any
one recovered heading covering `propose.OUTLINE_SHARE` (0.20, swept flat
from 0.10 to 0.30 and collapsing at both ends) of the text. It filters at
ADMISSION, before spans close, so the text a removed mark opened is carried
by the mark above; the post-filter form scores identically and loses that
text, which is why only one of them shipped. **The bar it had to clear is
now the bar**: reference cells up AND hit@8 holding rank 1 on every row on
every bundle. `--sheet-section-rows --keep-table-heading` reaches 11 of 12
and does NOT ship, because on a K2 bundle built with it row 1 falls rank
1 -> 2 (the gold document goes 1 concept -> 12) -- measured under BOTH prior
exponents, so the sublinear prior does not buy it. Arm E joined a session after the other two, on a number measured
AFTER the first move: without it Arm F's table clause has no joined table to
fold, and the shipped D+F default scored 2 of 12 with `docx` 0 of 3 against
the 5 of 12 the fold was published with. **The proposer's own defaults did NOT move** (`propose.py`'s rules stay
@ -278,17 +294,30 @@ and fixtures, never code.
so the two layers disagree on purpose and `cli.DEFAULT_OUTLINE_RUN` /
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
(`9cd74519...`, the delivered 2026-09-03 tree) to **492 / 944**
(`bdefa679...`, default flags, default epoch stamp, measured 2026-09-08 on
`f6fea13`). That digest SUPERSEDES `42c8646d...` (517 / 969), which named the
round-3 default before Arm E joined it. **And the number the
(the delivered 2026-09-03 tree) to 492 / 944 after the 2026-09-08 move and to
**425 / 810** after the 2026-09-09 one (`bdf4977ca5a443c4...`, default flags,
default epoch stamp, measured on `b01492b` + this round). Digests published
before 2026-09-09 were computed with a path-DEPENDENT command and are not
comparable to this one; the reproducible form is `find . -type f | sort |
xargs shasum -a 256 | shasum -a 256` from inside the bundle, under which the
previous default is `862116da16e422f6...`. The pinned artifact lives at
`~/corpora/okf-telling-20260829/K2-bundle-default-20260909` and
`tests/test_default_bundle_pin.py` holds its concept count AND its per-row
hit@8 ranks -- the count alone survived a configuration that lost a rank,
which is how the previous round's regression hid. **And the number the
decision cites belongs to another configuration:** Arm F's 5 of 12 was
measured with `--table-grid` ON; without it the same sample scores 2 of 12
and `docx` 0 of 3, because the fold's table clause has no joined table to
fold. The six (2026-09-08): `--outline-run N` (Arm D), `--table-grid` (Arm E),
fold. The eight: `--outline-run N` (Arm D), `--table-grid` (Arm E),
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
and `--drop-wrapped-outline` (both D3), each passed to the proposer
unchanged. Until that day the build path called the proposer with no
and `--drop-wrapped-outline` (both D3), `--outline-gate` (G1+G2) and
`--first-span-from-zero`, each passed to the proposer unchanged. That last
one is OFF and is not a segmentation rule at all -- it adds no boundary. It
repairs a measured loss the round found while chasing one position's 940
characters: **32 of the 32** documents that get a plan leave the text above
their first concept in NO segment, 159 704 characters, **9.18 %** of the
corpus, 45 841 from one document. It is off only because moving it changes
every bundle's first span and no hit@8 number stands behind it yet. Until that day the build path called the proposer with no
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
as one -- a build path a full arm behind the proposer. Exposing them was not
the same decision as moving one, and the two were taken a session apart: