feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment
One rule explains every remaining `pdf` miss on the twelve-position reference: where a document DECLARES headings, Arm D's RECOVERED headings are the whole of the excess, and every declared one is a unit the reference wants. `--outline-gate` admits recovery only where the document declares none of its own, plus any one recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units` clause 2's own principle moved from voting to admission, and it filters at ADMISSION so the text a removed mark opened is carried by the mark above it -- the post-filter form scores identically on all twelve positions and loses that text, which is why only one of them shipped. `--outline-gate` and `--drop-wrapped-outline` become the package default, one decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of 3. Each keeps an explicit opt-out. The bar the move had to clear was not the reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-, no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of 12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2. Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810. DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum measures size and a density is diluted by every unit carrying none of the question, so a document split 1 -> 12 lost its prior by 12. Swept over five values on 18 rows it is at least as good as the delivered density everywhere and strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row on any of four bundles, so it did not solve the knot it was adopted for -- what did is that the `pdf` gain never needed `--sheet-section-rows`. `--first-span-from-zero` is off and repairs a measured loss found while chasing one position's 940 characters: 32 of the 32 documents that get a plan leave the text above their first concept in no segment -- 159 704 characters, 9.18 % of the corpus, 45 841 from one document. It changes nothing on the reference. Off because it moves the first span of essentially every bundle with no hit@8 number behind it yet. vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads` carries it for the SEGMENTED profiles, read-only, after the emitted pattern misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2). vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1, `bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`, keyword-only, rendered before any disk mutation, written only when the index is created -- Door B's mechanism and ordering. Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md. Suite 1478 passed (1449 before), ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
b01492b7f5
commit
38104b7df5
16 changed files with 1301 additions and 42 deletions
53
CLAUDE.md
53
CLAUDE.md
|
|
@ -266,11 +266,27 @@ and fixtures, never code.
|
|||
collide); `--ingested-at` and `--proposed-at` default to one shared epoch
|
||||
constant rather than the clock, because a wall-clock default takes
|
||||
rebuild-equals-incremental away from anyone who omits them.
|
||||
**Six segmentation rules are REACHABLE here, and since 2026-09-08 THREE of
|
||||
them are ON by default** -- `--outline-run 3`, `--table-grid` and
|
||||
`--unit-fold`, each with an explicit opt-out (`--outline-run 0`,
|
||||
`--no-table-grid`, `--no-unit-fold`) that together reproduce the pre-move
|
||||
bytes. Arm E joined a session after the other two, on a number measured
|
||||
**Eight segmentation rules are REACHABLE here, and FIVE of them are ON by
|
||||
default** -- `--outline-run 3`, `--table-grid` and `--unit-fold` since
|
||||
2026-09-08, `--drop-wrapped-outline` and `--outline-gate` since 2026-09-09,
|
||||
each with an explicit opt-out (`--outline-run 0`, `--no-table-grid`,
|
||||
`--no-unit-fold`, `--keep-wrapped-outline`, `--no-outline-gate`) that
|
||||
together reproduce the pre-move bytes. **The 2026-09-09 pair is one
|
||||
decision and cannot be split**: the gate takes `pdf` from 2 of 8 to 5 of 8
|
||||
and the pair takes it to 7 of 8 (the sheet 5 of 12 -> 10 of 12, `docx`
|
||||
unchanged at 3 of 3). **The gate is G1+G2:** Arm D's RECOVERED headings are
|
||||
admitted only where the document DECLARES none of its own -- which is
|
||||
`fold_units` clause 2's principle moved from voting to admission -- plus any
|
||||
one recovered heading covering `propose.OUTLINE_SHARE` (0.20, swept flat
|
||||
from 0.10 to 0.30 and collapsing at both ends) of the text. It filters at
|
||||
ADMISSION, before spans close, so the text a removed mark opened is carried
|
||||
by the mark above; the post-filter form scores identically and loses that
|
||||
text, which is why only one of them shipped. **The bar it had to clear is
|
||||
now the bar**: reference cells up AND hit@8 holding rank 1 on every row on
|
||||
every bundle. `--sheet-section-rows --keep-table-heading` reaches 11 of 12
|
||||
and does NOT ship, because on a K2 bundle built with it row 1 falls rank
|
||||
1 -> 2 (the gold document goes 1 concept -> 12) -- measured under BOTH prior
|
||||
exponents, so the sublinear prior does not buy it. Arm E joined a session after the other two, on a number measured
|
||||
AFTER the first move: without it Arm F's table clause has no joined table to
|
||||
fold, and the shipped D+F default scored 2 of 12 with `docx` 0 of 3 against
|
||||
the 5 of 12 the fold was published with. **The proposer's own defaults did NOT move** (`propose.py`'s rules stay
|
||||
|
|
@ -278,17 +294,30 @@ and fixtures, never code.
|
|||
so the two layers disagree on purpose and `cli.DEFAULT_OUTLINE_RUN` /
|
||||
`cli.DEFAULT_UNIT_FOLD` say where. The cost to a consumer is a re-run and it
|
||||
is not small: the 43-document reference corpus goes 629 concepts / 1108 files
|
||||
(`9cd74519...`, the delivered 2026-09-03 tree) to **492 / 944**
|
||||
(`bdefa679...`, default flags, default epoch stamp, measured 2026-09-08 on
|
||||
`f6fea13`). That digest SUPERSEDES `42c8646d...` (517 / 969), which named the
|
||||
round-3 default before Arm E joined it. **And the number the
|
||||
(the delivered 2026-09-03 tree) to 492 / 944 after the 2026-09-08 move and to
|
||||
**425 / 810** after the 2026-09-09 one (`bdf4977ca5a443c4...`, default flags,
|
||||
default epoch stamp, measured on `b01492b` + this round). Digests published
|
||||
before 2026-09-09 were computed with a path-DEPENDENT command and are not
|
||||
comparable to this one; the reproducible form is `find . -type f | sort |
|
||||
xargs shasum -a 256 | shasum -a 256` from inside the bundle, under which the
|
||||
previous default is `862116da16e422f6...`. The pinned artifact lives at
|
||||
`~/corpora/okf-telling-20260829/K2-bundle-default-20260909` and
|
||||
`tests/test_default_bundle_pin.py` holds its concept count AND its per-row
|
||||
hit@8 ranks -- the count alone survived a configuration that lost a rank,
|
||||
which is how the previous round's regression hid. **And the number the
|
||||
decision cites belongs to another configuration:** Arm F's 5 of 12 was
|
||||
measured with `--table-grid` ON; without it the same sample scores 2 of 12
|
||||
and `docx` 0 of 3, because the fold's table clause has no joined table to
|
||||
fold. The six (2026-09-08): `--outline-run N` (Arm D), `--table-grid` (Arm E),
|
||||
fold. The eight: `--outline-run N` (Arm D), `--table-grid` (Arm E),
|
||||
`--unit-fold` (Arm F), `--keep-table-heading` (D1), `--sheet-section-rows`
|
||||
and `--drop-wrapped-outline` (both D3), each passed to the proposer
|
||||
unchanged. Until that day the build path called the proposer with no
|
||||
and `--drop-wrapped-outline` (both D3), `--outline-gate` (G1+G2) and
|
||||
`--first-span-from-zero`, each passed to the proposer unchanged. That last
|
||||
one is OFF and is not a segmentation rule at all -- it adds no boundary. It
|
||||
repairs a measured loss the round found while chasing one position's 940
|
||||
characters: **32 of the 32** documents that get a plan leave the text above
|
||||
their first concept in NO segment, 159 704 characters, **9.18 %** of the
|
||||
corpus, 45 841 from one document. It is off only because moving it changes
|
||||
every bundle's first span and no hit@8 number stands behind it yet. Until that day the build path called the proposer with no
|
||||
arm flag at all, so a tender PDF that Arm D splits into nine concepts landed
|
||||
as one -- a build path a full arm behind the proposer. Exposing them was not
|
||||
the same decision as moving one, and the two were taken a session apart:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue