feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment
One rule explains every remaining `pdf` miss on the twelve-position reference: where a document DECLARES headings, Arm D's RECOVERED headings are the whole of the excess, and every declared one is a unit the reference wants. `--outline-gate` admits recovery only where the document declares none of its own, plus any one recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units` clause 2's own principle moved from voting to admission, and it filters at ADMISSION so the text a removed mark opened is carried by the mark above it -- the post-filter form scores identically on all twelve positions and loses that text, which is why only one of them shipped. `--outline-gate` and `--drop-wrapped-outline` become the package default, one decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of 3. Each keeps an explicit opt-out. The bar the move had to clear was not the reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-, no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of 12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2. Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810. DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum measures size and a density is diluted by every unit carrying none of the question, so a document split 1 -> 12 lost its prior by 12. Swept over five values on 18 rows it is at least as good as the delivered density everywhere and strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row on any of four bundles, so it did not solve the knot it was adopted for -- what did is that the `pdf` gain never needed `--sheet-section-rows`. `--first-span-from-zero` is off and repairs a measured loss found while chasing one position's 940 characters: 32 of the 32 documents that get a plan leave the text above their first concept in no segment -- 159 704 characters, 9.18 % of the corpus, 45 841 from one document. It changes nothing on the reference. Off because it moves the first span of essentially every bundle with no hit@8 number behind it yet. vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads` carries it for the SEGMENTED profiles, read-only, after the emitted pattern misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2). vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1, `bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`, keyword-only, rendered before any disk mutation, written only when the index is created -- Door B's mechanism and ordering. Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md. Suite 1478 passed (1449 before), ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
b01492b7f5
commit
38104b7df5
16 changed files with 1301 additions and 42 deletions
|
|
@ -56,6 +56,7 @@ import json
|
|||
import re
|
||||
import sys
|
||||
import unicodedata
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
|
@ -371,6 +372,75 @@ def outline_runs(
|
|||
SHEET_SECTION_RUN = 3
|
||||
|
||||
|
||||
#: G2. How much of a document a RECOVERED heading must cover to survive the
|
||||
#: gate below. Chosen the way `OCR_CID_SHARE` was: bounded by an empty region
|
||||
#: on one side and by a measured collapse on the other, rather than by a round
|
||||
#: number. Over the four gated reference positions the eight outline
|
||||
#: candidates split 0.004 / 0.006 / 0.011 / 0.025 / 0.035 / 0.039 / 0.065 /
|
||||
#: 0.094 -- every one the reference REJECTS -- and 0.316, the one it keeps.
|
||||
#: Swept end to end, `pdf` holds at 7 of 8 from 0.10 through 0.30 and falls to
|
||||
#: 6 of 8 at 0.05 (a 0.094 candidate enters) and at 0.35 (the 0.316 candidate
|
||||
#: leaves). 0.20 is the middle of that plateau and of the empty region alike.
|
||||
#: HONESTY LIMIT: n = 8 candidates in 4 documents, one rater, one reference.
|
||||
OUTLINE_SHARE = 0.20
|
||||
|
||||
|
||||
def declares_headings(candidates: Iterable[Candidate]) -> bool:
|
||||
"""Whether the document's OWN heading grammar proposed anything at all.
|
||||
|
||||
`RULE_HEADING` is the delivered grammar -- ATX and dotted-numbered -- and
|
||||
`RULE_OUTLINE` is Arm D's RECOVERY of an integer numbering run. This
|
||||
predicate separates the two, and it lives here as one function on purpose:
|
||||
the proposer and the door both reach it through `build_plan`, so there is
|
||||
exactly one call site and the two sides cannot disagree about which
|
||||
documents the gate fired on.
|
||||
|
||||
It is NOT handed to the door as a callable the way `heading_reserve_applies`
|
||||
is, and the difference is worth stating rather than glossing. The reserve
|
||||
chooses which TEXT a plan is proposed against, so a reserve firing on one
|
||||
side only would index a plan against a string the other side never saw --
|
||||
a coded rejection on `text_sha256`. This gate chooses which CANDIDATES a
|
||||
plan contains, from a text both sides already agree on, and the plan is
|
||||
built once. Passing it as a predicate would create a second definition to
|
||||
keep in step and buy nothing: measured, a text-only predicate DISAGREES
|
||||
with the candidate list on 2 of 39 corpus documents, because
|
||||
`keep_table_heading` rescues a heading the orphan check would have dropped.
|
||||
"""
|
||||
return any(candidate.rule == RULE_HEADING for candidate in candidates)
|
||||
|
||||
|
||||
def _gate_outline(
|
||||
marked: list[tuple[int, Candidate]], joined: set[int], end_of_text: int, length: int
|
||||
) -> tuple[list[tuple[int, Candidate]], set[int]]:
|
||||
"""G1 and G2: recovery yields to declaration, unless it carries the document.
|
||||
|
||||
Applied to `marked` -- BEFORE the orphan pass -- rather than to the
|
||||
finished entries, and that is the correctness half of the rule rather than
|
||||
a style choice. The second pass closes each span at the NEXT mark, so
|
||||
removing a mark here lets the preceding span reach through the text that
|
||||
mark used to open. Filtering the finished entries instead leaves that text
|
||||
in no segment at all, which is the silent loss this library refuses
|
||||
everywhere else. Measured, the two forms agree on every one of the twelve
|
||||
reference positions and on the whole 39-document corpus reach; they differ
|
||||
only in the spans, and only one of them conserves the text.
|
||||
|
||||
`joined` holds POSITIONS in `marked`, so it is remapped rather than copied.
|
||||
"""
|
||||
if not declares_headings(candidate for _, candidate in marked):
|
||||
return marked, joined
|
||||
kept: list[tuple[int, Candidate]] = []
|
||||
remap: dict[int, int] = {}
|
||||
for position, entry in enumerate(marked):
|
||||
candidate = entry[1]
|
||||
if candidate.rule == RULE_OUTLINE:
|
||||
following = marked[position + 1][1].start if position + 1 < len(marked) else end_of_text
|
||||
if length <= 0 or (following - candidate.start) / length < OUTLINE_SHARE:
|
||||
continue
|
||||
remap[position] = len(kept)
|
||||
kept.append(entry)
|
||||
return kept, {remap[p] for p in joined if p in remap}
|
||||
|
||||
|
||||
def _wraps_onto_next_line(lines: list[str], index: int) -> bool:
|
||||
"""True when the line at `index` is a sentence that continues below it.
|
||||
|
||||
|
|
@ -442,6 +512,8 @@ def find_candidates(
|
|||
keep_table_heading: bool = False,
|
||||
sheet_section_rows: bool = False,
|
||||
drop_wrapped_outline: bool = False,
|
||||
outline_gate: bool = False,
|
||||
first_span_from_zero: bool = False,
|
||||
) -> list[Candidate]:
|
||||
"""Every boundary the mechanical rules propose, in document order.
|
||||
|
||||
|
|
@ -477,6 +549,17 @@ def find_candidates(
|
|||
It is its own flag and not part of an arm because the orphan check is
|
||||
reached by every file type, and moving it is a decision about all of them.
|
||||
|
||||
`first_span_from_zero` is OFF at False, where the text above the first
|
||||
concept belongs to no segment. On, the first surviving concept starts at 0.
|
||||
It adds no boundary and removes none; only the first span's `start` moves.
|
||||
|
||||
`outline_gate` is G1+G2 and it is OFF at False. On, Arm D's recovered
|
||||
headings are admitted only where the document declares none of its own,
|
||||
plus any single recovered heading whose span covers `OUTLINE_SHARE` of
|
||||
the text. Like Arm E it only ever REMOVES marks, and it removes them
|
||||
before spans are closed, so the text they opened is carried by the mark
|
||||
above rather than lost.
|
||||
|
||||
`sheet_section_rows` is D3's gate and it is OFF at False, where the scan is
|
||||
not run at all. On, a RUN of numbered rows inside an open table block cuts
|
||||
it: each such row opens a candidate that reaches the next section row, or
|
||||
|
|
@ -649,6 +732,9 @@ def find_candidates(
|
|||
)
|
||||
)
|
||||
|
||||
if outline_gate:
|
||||
marked, joined = _gate_outline(marked, joined, end_of_text, len(text))
|
||||
|
||||
candidates: list[Candidate] = []
|
||||
# The name an orphaned heading leaves behind, and the ONE candidate allowed
|
||||
# to pick it up.
|
||||
|
|
@ -721,7 +807,13 @@ def find_candidates(
|
|||
contents=position_in_list in contents_run,
|
||||
)
|
||||
)
|
||||
return fold_units(candidates) if unit_fold else candidates
|
||||
resolved = fold_units(candidates) if unit_fold else candidates
|
||||
if first_span_from_zero and resolved and resolved[0].start > 0:
|
||||
# Applied AFTER the fold, so the concept that SURVIVES is the one that
|
||||
# opens at 0: the fold can discard a contents run, and moving the start
|
||||
# before it would put the preamble on an entry the plan never carries.
|
||||
resolved = [replace(resolved[0], start=0), *resolved[1:]]
|
||||
return resolved
|
||||
|
||||
|
||||
def _absorbed_tables(
|
||||
|
|
@ -1052,6 +1144,8 @@ def build_plan(
|
|||
keep_table_heading: bool = False,
|
||||
sheet_section_rows: bool = False,
|
||||
drop_wrapped_outline: bool = False,
|
||||
outline_gate: bool = False,
|
||||
first_span_from_zero: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
|
||||
taken: set[str] = set()
|
||||
|
|
@ -1065,6 +1159,8 @@ def build_plan(
|
|||
keep_table_heading=keep_table_heading,
|
||||
sheet_section_rows=sheet_section_rows,
|
||||
drop_wrapped_outline=drop_wrapped_outline,
|
||||
outline_gate=outline_gate,
|
||||
first_span_from_zero=first_span_from_zero,
|
||||
)
|
||||
for candidate in subdivide(text, candidates, max_segment_chars):
|
||||
entries.append(
|
||||
|
|
@ -1138,6 +1234,8 @@ def run(
|
|||
keep_table_heading: bool = False,
|
||||
sheet_section_rows: bool = False,
|
||||
drop_wrapped_outline: bool = False,
|
||||
outline_gate: bool = False,
|
||||
first_span_from_zero: bool = False,
|
||||
pdf_headings: bool = False,
|
||||
pdf_headings_reserve: bool = False,
|
||||
ocr: bool = False,
|
||||
|
|
@ -1210,6 +1308,8 @@ def run(
|
|||
keep_table_heading=keep_table_heading,
|
||||
sheet_section_rows=sheet_section_rows,
|
||||
drop_wrapped_outline=drop_wrapped_outline,
|
||||
outline_gate=outline_gate,
|
||||
first_span_from_zero=first_span_from_zero,
|
||||
)
|
||||
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
|
||||
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue