feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment

One rule explains every remaining `pdf` miss on the twelve-position reference:
where a document DECLARES headings, Arm D's RECOVERED headings are the whole of
the excess, and every declared one is a unit the reference wants. `--outline-gate`
admits recovery only where the document declares none of its own, plus any one
recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units`
clause 2's own principle moved from voting to admission, and it filters at
ADMISSION so the text a removed mark opened is carried by the mark above it --
the post-filter form scores identically on all twelve positions and loses that
text, which is why only one of them shipped.

`--outline-gate` and `--drop-wrapped-outline` become the package default, one
decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8
alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of
3. Each keeps an explicit opt-out. The bar the move had to clear was not the
reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-,
no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of
12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2.
Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810.

DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum
measures size and a density is diluted by every unit carrying none of the
question, so a document split 1 -> 12 lost its prior by 12. Swept over five
values on 18 rows it is at least as good as the delivered density everywhere and
strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row
on any of four bundles, so it did not solve the knot it was adopted for -- what
did is that the `pdf` gain never needed `--sheet-section-rows`.

`--first-span-from-zero` is off and repairs a measured loss found while chasing
one position's 940 characters: 32 of the 32 documents that get a plan leave the
text above their first concept in no segment -- 159 704 characters, 9.18 % of
the corpus, 45 841 from one document. It changes nothing on the reference. Off
because it moves the first span of essentially every bundle with no hit@8 number
behind it yet.

vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every
concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads`
carries it for the SEGMENTED profiles, read-only, after the emitted pattern
misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2).

vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1,
`bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`,
keyword-only, rendered before any disk mutation, written only when the index is
created -- Door B's mechanism and ordering.

Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md.
Suite 1478 passed (1449 before), ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 14:17:00 +02:00
commit 38104b7df5
16 changed files with 1301 additions and 42 deletions

View file

@ -56,6 +56,7 @@ import json
import re
import sys
import unicodedata
from collections.abc import Iterable
from dataclasses import dataclass, replace
from pathlib import Path
from typing import Any
@ -371,6 +372,75 @@ def outline_runs(
SHEET_SECTION_RUN = 3
#: G2. How much of a document a RECOVERED heading must cover to survive the
#: gate below. Chosen the way `OCR_CID_SHARE` was: bounded by an empty region
#: on one side and by a measured collapse on the other, rather than by a round
#: number. Over the four gated reference positions the eight outline
#: candidates split 0.004 / 0.006 / 0.011 / 0.025 / 0.035 / 0.039 / 0.065 /
#: 0.094 -- every one the reference REJECTS -- and 0.316, the one it keeps.
#: Swept end to end, `pdf` holds at 7 of 8 from 0.10 through 0.30 and falls to
#: 6 of 8 at 0.05 (a 0.094 candidate enters) and at 0.35 (the 0.316 candidate
#: leaves). 0.20 is the middle of that plateau and of the empty region alike.
#: HONESTY LIMIT: n = 8 candidates in 4 documents, one rater, one reference.
OUTLINE_SHARE = 0.20
def declares_headings(candidates: Iterable[Candidate]) -> bool:
"""Whether the document's OWN heading grammar proposed anything at all.
`RULE_HEADING` is the delivered grammar -- ATX and dotted-numbered -- and
`RULE_OUTLINE` is Arm D's RECOVERY of an integer numbering run. This
predicate separates the two, and it lives here as one function on purpose:
the proposer and the door both reach it through `build_plan`, so there is
exactly one call site and the two sides cannot disagree about which
documents the gate fired on.
It is NOT handed to the door as a callable the way `heading_reserve_applies`
is, and the difference is worth stating rather than glossing. The reserve
chooses which TEXT a plan is proposed against, so a reserve firing on one
side only would index a plan against a string the other side never saw --
a coded rejection on `text_sha256`. This gate chooses which CANDIDATES a
plan contains, from a text both sides already agree on, and the plan is
built once. Passing it as a predicate would create a second definition to
keep in step and buy nothing: measured, a text-only predicate DISAGREES
with the candidate list on 2 of 39 corpus documents, because
`keep_table_heading` rescues a heading the orphan check would have dropped.
"""
return any(candidate.rule == RULE_HEADING for candidate in candidates)
def _gate_outline(
marked: list[tuple[int, Candidate]], joined: set[int], end_of_text: int, length: int
) -> tuple[list[tuple[int, Candidate]], set[int]]:
"""G1 and G2: recovery yields to declaration, unless it carries the document.
Applied to `marked` -- BEFORE the orphan pass -- rather than to the
finished entries, and that is the correctness half of the rule rather than
a style choice. The second pass closes each span at the NEXT mark, so
removing a mark here lets the preceding span reach through the text that
mark used to open. Filtering the finished entries instead leaves that text
in no segment at all, which is the silent loss this library refuses
everywhere else. Measured, the two forms agree on every one of the twelve
reference positions and on the whole 39-document corpus reach; they differ
only in the spans, and only one of them conserves the text.
`joined` holds POSITIONS in `marked`, so it is remapped rather than copied.
"""
if not declares_headings(candidate for _, candidate in marked):
return marked, joined
kept: list[tuple[int, Candidate]] = []
remap: dict[int, int] = {}
for position, entry in enumerate(marked):
candidate = entry[1]
if candidate.rule == RULE_OUTLINE:
following = marked[position + 1][1].start if position + 1 < len(marked) else end_of_text
if length <= 0 or (following - candidate.start) / length < OUTLINE_SHARE:
continue
remap[position] = len(kept)
kept.append(entry)
return kept, {remap[p] for p in joined if p in remap}
def _wraps_onto_next_line(lines: list[str], index: int) -> bool:
"""True when the line at `index` is a sentence that continues below it.
@ -442,6 +512,8 @@ def find_candidates(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
outline_gate: bool = False,
first_span_from_zero: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -477,6 +549,17 @@ def find_candidates(
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
`first_span_from_zero` is OFF at False, where the text above the first
concept belongs to no segment. On, the first surviving concept starts at 0.
It adds no boundary and removes none; only the first span's `start` moves.
`outline_gate` is G1+G2 and it is OFF at False. On, Arm D's recovered
headings are admitted only where the document declares none of its own,
plus any single recovered heading whose span covers `OUTLINE_SHARE` of
the text. Like Arm E it only ever REMOVES marks, and it removes them
before spans are closed, so the text they opened is carried by the mark
above rather than lost.
`sheet_section_rows` is D3's gate and it is OFF at False, where the scan is
not run at all. On, a RUN of numbered rows inside an open table block cuts
it: each such row opens a candidate that reaches the next section row, or
@ -649,6 +732,9 @@ def find_candidates(
)
)
if outline_gate:
marked, joined = _gate_outline(marked, joined, end_of_text, len(text))
candidates: list[Candidate] = []
# The name an orphaned heading leaves behind, and the ONE candidate allowed
# to pick it up.
@ -721,7 +807,13 @@ def find_candidates(
contents=position_in_list in contents_run,
)
)
return fold_units(candidates) if unit_fold else candidates
resolved = fold_units(candidates) if unit_fold else candidates
if first_span_from_zero and resolved and resolved[0].start > 0:
# Applied AFTER the fold, so the concept that SURVIVES is the one that
# opens at 0: the fold can discard a contents run, and moving the start
# before it would put the preamble on an entry the plan never carries.
resolved = [replace(resolved[0], start=0), *resolved[1:]]
return resolved
def _absorbed_tables(
@ -1052,6 +1144,8 @@ def build_plan(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
outline_gate: bool = False,
first_span_from_zero: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1065,6 +1159,8 @@ def build_plan(
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -1138,6 +1234,8 @@ def run(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
outline_gate: bool = False,
first_span_from_zero: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -1210,6 +1308,8 @@ def run(
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan