Kjell Tore Guttormsen
38104b7df5
One rule explains every remaining `pdf` miss on the twelve-position reference: where a document DECLARES headings, Arm D's RECOVERED headings are the whole of the excess, and every declared one is a unit the reference wants. `--outline-gate` admits recovery only where the document declares none of its own, plus any one recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units` clause 2's own principle moved from voting to admission, and it filters at ADMISSION so the text a removed mark opened is carried by the mark above it -- the post-filter form scores identically on all twelve positions and loses that text, which is why only one of them shipped. `--outline-gate` and `--drop-wrapped-outline` become the package default, one decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of 3. Each keeps an explicit opt-out. The bar the move had to clear was not the reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-, no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of 12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2. Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810. DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum measures size and a density is diluted by every unit carrying none of the question, so a document split 1 -> 12 lost its prior by 12. Swept over five values on 18 rows it is at least as good as the delivered density everywhere and strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row on any of four bundles, so it did not solve the knot it was adopted for -- what did is that the `pdf` gain never needed `--sheet-section-rows`. `--first-span-from-zero` is off and repairs a measured loss found while chasing one position's 940 characters: 32 of the 32 documents that get a plan leave the text above their first concept in no segment -- 159 704 characters, 9.18 % of the corpus, 45 841 from one document. It changes nothing on the reference. Off because it moves the first span of essentially every bundle with no hit@8 number behind it yet. vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads` carries it for the SEGMENTED profiles, read-only, after the emitted pattern misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2). vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1, `bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`, keyword-only, rendered before any disk mutation, written only when the index is created -- Door B's mechanism and ordering. Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md. Suite 1478 passed (1449 before), ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
72 lines
3.1 KiB
Python
72 lines
3.1 KiB
Python
"""No character stands before the first concept.
|
|
|
|
Round 3 recorded it as a footnote about ONE position: under
|
|
`--drop-wrapped-outline` that document's surviving concept starts at line 18 of
|
|
302, so 940 characters of the note's opening sit in no segment. Measured across
|
|
the whole 39-document corpus it is not one position -- it is **32 of the 32
|
|
documents that get a plan at all, 159 704 characters, 9.18 % of the corpus**,
|
|
with 45 841 characters lost from a single document. A concept boundary is a
|
|
statement about where a unit BEGINS; it was never a statement that the text
|
|
above the first one belongs nowhere.
|
|
|
|
The rule: when nothing precedes the first concept, the first concept starts at
|
|
0. It adds no boundary and removes none -- only the first span's `start` moves
|
|
-- so a plan's entry count is identical either way.
|
|
|
|
OFF by default, and the reason is the round's own acceptance criterion rather
|
|
than doubt about the defect: it moves the first span of essentially every
|
|
segmented bundle, and the round budget held one K2 build, which measured the
|
|
outline gate. A default that changes every bundle's first concept without a
|
|
hit@8 measurement behind it would be exactly the move this round refused to
|
|
make elsewhere. The measured cost and the outstanding measurement are in
|
|
`docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md`.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from llm_ingestion_okf import cli
|
|
from llm_ingestion_okf.propose import find_candidates
|
|
|
|
ARMS = dict(outline_run=3, table_grid=True, unit_fold=True)
|
|
|
|
WITH_PREAMBLE = (
|
|
"En tittelside og et forord som ingen overskrift star foran.\n"
|
|
"Enda en linje av det samme.\n"
|
|
"\n"
|
|
"## Forste avsnitt\nInnhold under.\n"
|
|
"## Andre avsnitt\nInnhold under.\n"
|
|
)
|
|
|
|
NO_PREAMBLE = "## Forste avsnitt\nInnhold under.\n## Andre avsnitt\nInnhold under.\n"
|
|
|
|
|
|
def test_the_default_still_loses_the_preamble() -> None:
|
|
"""The defect, kept as a test so the flag's reason stays visible."""
|
|
first = find_candidates(WITH_PREAMBLE, **ARMS)[0]
|
|
assert first.start > 0
|
|
|
|
|
|
def test_the_first_concept_starts_at_zero() -> None:
|
|
candidates = find_candidates(WITH_PREAMBLE, **ARMS, first_span_from_zero=True)
|
|
assert candidates[0].start == 0
|
|
assert WITH_PREAMBLE[candidates[0].start : candidates[0].end].startswith("En tittelside")
|
|
|
|
|
|
def test_a_document_with_no_preamble_is_untouched() -> None:
|
|
"""The known-negative, as IDENTICAL objects rather than an equal count."""
|
|
before = find_candidates(NO_PREAMBLE, **ARMS)
|
|
after = find_candidates(NO_PREAMBLE, **ARMS, first_span_from_zero=True)
|
|
assert before == after
|
|
assert before[0].start == 0
|
|
|
|
|
|
def test_the_rule_adds_and_removes_no_boundary() -> None:
|
|
plain = find_candidates(WITH_PREAMBLE, **ARMS)
|
|
moved = find_candidates(WITH_PREAMBLE, **ARMS, first_span_from_zero=True)
|
|
assert [c.title for c in plain] == [c.title for c in moved]
|
|
assert [c.end for c in plain] == [c.end for c in moved]
|
|
assert [c.start for c in plain][1:] == [c.start for c in moved][1:]
|
|
|
|
|
|
def test_it_is_off_by_default_in_the_build_command() -> None:
|
|
assert cli.DEFAULT_FIRST_SPAN_FROM_ZERO is False
|