open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment

Browse source 
One rule explains every remaining `pdf` miss on the twelve-position reference:
where a document DECLARES headings, Arm D's RECOVERED headings are the whole of
the excess, and every declared one is a unit the reference wants. `--outline-gate`
admits recovery only where the document declares none of its own, plus any one
recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units`
clause 2's own principle moved from voting to admission, and it filters at
ADMISSION so the text a removed mark opened is carried by the mark above it --
the post-filter form scores identically on all twelve positions and loses that
text, which is why only one of them shipped.

`--outline-gate` and `--drop-wrapped-outline` become the package default, one
decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8
alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of
3. Each keeps an explicit opt-out. The bar the move had to clear was not the
reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-,
no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of
12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2.
Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810.

DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum
measures size and a density is diluted by every unit carrying none of the
question, so a document split 1 -> 12 lost its prior by 12. Swept over five
values on 18 rows it is at least as good as the delivered density everywhere and
strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row
on any of four bundles, so it did not solve the knot it was adopted for -- what
did is that the `pdf` gain never needed `--sheet-section-rows`.

`--first-span-from-zero` is off and repairs a measured loss found while chasing
one position's 940 characters: 32 of the 32 documents that get a plan leave the
text above their first concept in no segment -- 159 704 characters, 9.18 % of
the corpus, 45 841 from one document. It changes nothing on the reference. Off
because it moves the first span of essentially every bundle with no hit@8 number
behind it yet.

vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every
concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads`
carries it for the SEGMENTED profiles, read-only, after the emitted pattern
misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2).

vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1,
`bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`,
keyword-only, rendered before any disk mutation, written only when the index is
created -- Door B's mechanism and ordering.

Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md.
Suite 1478 passed (1449 before), ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 14:17:00 +02:00
commit 38104b7df5
16 changed files with 1301 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

77
src/llm_ingestion_okf/cli.py
View file

@ -138,6 +138,29 @@ DEFAULT_OUTLINE_RUN = 3
DEFAULT_TABLE_GRID = True
DEFAULT_UNIT_FOLD = True
#: Round 6's two, and they moved together on 2026-09-09 because neither carries
#: the reference on its own: G1+G2 takes `pdf` from 2 of 8 to 5 of 8, and
#: `--drop-wrapped-outline` -- measured on its own in round 3 as a 1-document
#: change and shipped off -- takes the pair to 7 of 8. Each keeps its explicit
#: opt-out (`--no-outline-gate`, `--keep-wrapped-outline`), and the two of them
#: off reproduce the 2026-09-08 default byte for byte. The acceptance the move
#: had to clear was not the reference alone: hit@8 holds rank 1 on every row it
#: held before, on every bundle measured. Both measurements are in
#: `docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md`.
DEFAULT_OUTLINE_GATE = True
DEFAULT_DROP_WRAPPED_OUTLINE = True
#: Round 6's third rule, and the only one whose default this round did NOT
#: decide. Measured over the 39-document corpus: every one of the 32 documents
#: that gets a plan loses the text above its first concept -- 159 704
#: characters, 9.18 % of the corpus, 45 841 from a single document. The repair
#: is one line and it adds no boundary. It is off because it moves the first
#: span of essentially every segmented bundle and this round's budget held one
#: K2 build, which measured the outline gate; a default that changed every
#: bundle's first concept with no hit@8 number behind it would be the move this
#: round refused to make elsewhere. The number it needs is named in STATE.
DEFAULT_FIRST_SPAN_FROM_ZERO = False
#: Round 4's two PDF READERS, both off. They are not arms and they are not in
#: the paragraph above: an arm changes how the proposer cuts a text, and these
#: two change what the text SAYS -- so they sit before every arm rather than
@ -184,6 +207,8 @@ def _propose_plans(
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
outline_gate: bool = False,
first_span_from_zero: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -217,6 +242,8 @@ def _propose_plans(
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -248,7 +275,9 @@ def build(
unit_fold: bool = DEFAULT_UNIT_FOLD,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
drop_wrapped_outline: bool = DEFAULT_DROP_WRAPPED_OUTLINE,
outline_gate: bool = DEFAULT_OUTLINE_GATE,
first_span_from_zero: bool = DEFAULT_FIRST_SPAN_FROM_ZERO,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
@ -316,6 +345,8 @@ def build(
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -523,6 +554,7 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
default=DEFAULT_DROP_WRAPPED_OUTLINE,
help=(
"D3, passed to the proposer unchanged: do not admit an outline "
"candidate whose line continues onto the next one. Recovered "
@ -532,6 +564,47 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"and the risk-table rows the operator wrote off"
),
)
build_parser.add_argument(
"--keep-wrapped-outline",
action="store_false",
dest="drop_wrapped_outline",
help=(
"The opt-out for the rule above. With --no-outline-gate it "
"reproduces the pre-2026-09-09 default byte for byte"
),
)
build_parser.add_argument(
"--outline-gate",
action="store_true",
default=DEFAULT_OUTLINE_GATE,
help=(
"G1+G2, passed to the proposer unchanged: admit Arm D's RECOVERED "
"headings only where the document declares none of its own, plus "
"any one recovered heading whose span covers OUTLINE_SHARE of the "
"text. It only removes marks, and it removes them before spans are "
"closed, so no character falls outside every segment. Measured on "
"the K3 sample: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 with "
"--drop-wrapped-outline, `docx` unchanged at 3 of 3"
),
)
build_parser.add_argument(
"--no-outline-gate",
action="store_false",
dest="outline_gate",
help="The gate's explicit opt-out. A default a caller cannot turn off is not a default",
)
build_parser.add_argument(
"--first-span-from-zero",
action="store_true",
default=DEFAULT_FIRST_SPAN_FROM_ZERO,
help=(
"Start the first concept at character 0, so the text above it is "
"carried by a segment instead of by none. Absent (the default) is "
"OFF. Measured over the 39-document corpus: 32 of the 32 documents "
"with a plan lose their opening today -- 159 704 characters, 9.18 "
"%% of the corpus. It adds no boundary and moves one span's start"
),
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font", "font-reserve"),
@ -602,6 +675,8 @@ def main(argv: list[str] | None = None) -> int:
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
outline_gate=args.outline_gate,
first_span_from_zero=args.first_span_from_zero,
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,