open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,cli): the coverage gap had one cause, and round 7's own decomposition did not reproduce

Browse source 
Round 7 named two open items: a table-block candidate displacing a declared
heading (26 041 characters between entries, `md` at 3 of 4 declared headings)
and 17 590 characters after the last entry, never examined. Measured on
`a364ef4`, the first premise does not reproduce and the second is not a
separate mechanism.

`md` recovers 4 of 4, not 3 of 4. D1 (`--keep-table-heading`) is the repair
for exactly the mechanism round 7 § 5 describes, and round 7 moved it into
the default in the same commit -- so § 5 is a pre-move measurement presented
as the post-move state. That is round 7's own trap, stated in its own report:
a number about a rule is a measurement of a configuration. Nothing held the
cell, which is why it could be wrong and stay wrong.

The remaining gap has ONE cause. Every rule closes a span against the NEXT
MARK; three steps then remove a mark after its neighbour's `end` was fixed
against it. The orphan check leaks 18 527 characters over 15 of 39 documents,
`fold_units` clause 1 leaks 7 514 between entries, and the same clause on the
last run leaks all 17 590 tail characters -- with `unit_fold=False` the corpus
tail gap is 0. Round 6 already established the principle (filter at admission,
let the mark above carry the text) and it was never applied post-filter.

`--close-span-gaps` states it once, after the fold, where every removal has
happened: a span runs to the next SURVIVOR, the last to the end of the text.
It adds no boundary. Measured: 43 631 characters (2.51 %) -> 0, entries
429 = 429, K2 concepts 436 = 436, the operator's folder 52 md = 52.

All four acceptance conditions hold at once. The 12-position reference is
label-identical -- 11 of 12 under |F|[3]=12 and 10 of 12 under |F|[3]=11,
both readings reported, not one position moved. hit@8 holds [1,1,1,1,1,-] on
the round 8 bundle (436), the round 7 default (436) and Arm B (629), and the
known-negative reproduces on the new bytes: `--no-tie-shared-rank` gives
[2,1,1,1,1,-].

The pin is rebuilt against the default that actually ships
(`K2-bundle-default-20260911`, 832 files, digest 8c93e5e3...); two independent
builds of it differ in nothing, `log.md` included.

Two measurement failures are recorded rather than hidden. The corpus identity
check first used `xargs` without `-print0`, every filename split on its
spaces, `shasum` read nothing, and both sides agreed on a digest of the
failure; a control against the digest of empty input separated them. And the
smoke build cannot prove the flag arrives -- the operator's folder has a
coverage gap of zero already, so `diff -rq` is 0 either way; the plumbing has
its own test on a document that has the defect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 17:43:56 +02:00
commit 956714594d
8 changed files with 592 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

36
src/llm_ingestion_okf/cli.py
View file

@ -163,6 +163,13 @@ DEFAULT_DROP_WRAPPED_OUTLINE = True
#: `docx`, 10 of 12) AND hit@8 holds rank 1 on every row it held, measured end
#: to end on a K2 bundle built with it. Opt-out `--no-first-span-from-zero`.
DEFAULT_FIRST_SPAN_FROM_ZERO = True
#: Round 8. The remaining coverage gap round 7 named and left open -- 43 631
#: characters, 2.51 % of the corpus over 8 of 32 documents with a plan -- has
#: ONE cause: three steps remove a candidate AFTER its neighbour's span was
#: already closed against it, so the removed mark's text is in no segment. The
#: rule closes each span against the next SURVIVOR. It adds no boundary, so the
#: concept count is unchanged and the cost to a consumer is a re-run.
DEFAULT_CLOSE_SPAN_GAPS = True
#: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and
#: 6 by a RETRIEVAL regression rather than by the reference: they take the
@ -228,6 +235,7 @@ def _propose_plans(
drop_wrapped_outline: bool = False,
outline_gate: bool = False,
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -263,6 +271,7 @@ def _propose_plans(
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -297,6 +306,7 @@ def build(
drop_wrapped_outline: bool = DEFAULT_DROP_WRAPPED_OUTLINE,
outline_gate: bool = DEFAULT_OUTLINE_GATE,
first_span_from_zero: bool = DEFAULT_FIRST_SPAN_FROM_ZERO,
close_span_gaps: bool = DEFAULT_CLOSE_SPAN_GAPS,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
@ -366,6 +376,7 @@ def build(
drop_wrapped_outline=drop_wrapped_outline,
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -649,6 +660,30 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"pre-2026-09-10 first span byte for byte"
),
)
build_parser.add_argument(
"--close-span-gaps",
action="store_true",
default=DEFAULT_CLOSE_SPAN_GAPS,
help=(
"Close a concept's span against the next SURVIVING concept, and "
"the last one against the end of the text, so a candidate removed "
"after its neighbour's span was fixed does not take its text out "
"of the plan. Measured over the 39-document corpus: it closes the "
"43 631 characters -- 2.51 %% of the corpus over 8 of the 32 "
"documents with a plan -- that round 7 left, both the 26 041 "
"between entries and the 17 590 after the last. It adds no "
"boundary and moves only spans' ends. ON since 2026-09-11"
),
)
build_parser.add_argument(
"--no-close-span-gaps",
action="store_false",
dest="close_span_gaps",
help=(
"The repair's explicit opt-out, for a consumer who needs the "
"pre-2026-09-11 spans byte for byte"
),
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font", "font-reserve"),
@ -721,6 +756,7 @@ def main(argv: list[str] | None = None) -> int:
drop_wrapped_outline=args.drop_wrapped_outline,
outline_gate=args.outline_gate,
first_span_from_zero=args.first_span_from_zero,
close_span_gaps=args.close_span_gaps,
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,