open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,consume): the first span, and the fusion that punished fine-graining

Browse source 
Round 6 left two things behind: a measured coverage hole with the repair built
but not the number to move it, and a retrieval regression that kept two
spreadsheet rules off the default. Both are measured; four defaults move.

THE HOLE IS BIGGER THAN THE RULE BUILT FOR IT. Measured over the 39-document
corpus, the pre-move default left 207 435 characters -- 11.92 % -- in no
segment: 163 804 above the first entry, 26 041 between entries, 17 590 after
the last. `--first-span-from-zero` closes the first part entirely, 79 % of the
whole, leaving 43 631 (2.51 %) over 8 of 32 documents. It adds no boundary and
the K2 concept count is identical with and without it (425 = 425); hit@8 holds
[1,1,1,1,1,-] under both tie-breaks and the 12-position reference does not move
one cell.

THE RANKER WAS PUNISHING FINE-GRAINING FOR BEING FINE-GRAINED. Round 6 held
`--sheet-section-rows --keep-table-heading` back because on a bundle built with
them row 1 fell rank 1 -> 2, and ruled out the document prior because its rank
over 39 DOCUMENTS was 1 on both bundles. RRF reads a rank over CONCEPTS:
splitting the gold document 1 -> 12 puts its own twelve concepts in that
signal's whole top tie group, so the one leading the body signal takes position
11, contributing 1/71 where the undivided concept contributed 1/61. A signal
that scored them all EQUALLY still emitted twelve ranks, ordered by concept_id,
and the fusion read alphabetical order as a measurement.

AND THE REPAIR WAS ALREADY IN THE TREE, OFF FOR A COST THAT HAD BEEN REMOVED.
`--tie-shared-rank` shipped off 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6.
Swept over 2 prior exponents x 3 bundles x 6 rows: that fall exists only at
DOCUMENT_PRIOR_EXPONENT 1.0. Round 6 moved the exponent to 0.5 for an unrelated
reason and correctly reported it moved no hit@8 row -- nobody measured the
pair. A flag's "off by measurement" is a measurement of a CONFIGURATION, not a
property of the flag.

Shipped, each with an explicit opt-out and both directions measured rather than
asserted: --no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading reproduces the previous bundle byte for byte (diff -rq,
0 differences), and the no-flag build equals the explicit-flag build.
--no-tie-shared-rank reproduces the previous excerpt order -- the one change
here that alters a payload with no bundle changing.

K2 default: 436 concepts / 832 files, digest 8dff8a8e6c15d2f7..., hit@8 5 of 6
ranks [1,1,1,1,1,-]. The pin now holds its own known-negative on those bytes:
read with --no-tie-shared-rank the same bundle gives [2,1,1,1,1,-], so the
green assertion names its cause. Consumer cost is a re-run: 425/810 -> 436/832
on the reference corpus, 15/30 -> 26/52 on a five-document folder.

pptx and md measured end to end for the first time, on two hand-built
documents: md recovers 3 of 4 declared headings, pptx segments per slide only
where the converter recognised a title placeholder. A converter attribute leaks
into titles ({#slide-N}, {#sheet-1}) on 2 of 810 K2 files and 1 of 30 on a
five-document folder -- not fixed, because a filename is reduced from its title
and the fix renames concept ids a consumer has cited.

Suite 1486 (1478 before), ruff and mypy --strict clean over 21 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 15:40:01 +02:00
commit a364ef4c7d
11 changed files with 768 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

84
src/llm_ingestion_okf/cli.py
View file

@ -150,16 +150,35 @@ DEFAULT_UNIT_FOLD = True
DEFAULT_OUTLINE_GATE = True
DEFAULT_DROP_WRAPPED_OUTLINE = True
#: Round 6's third rule, and the only one whose default this round did NOT
#: decide. Measured over the 39-document corpus: every one of the 32 documents
#: that gets a plan loses the text above its first concept -- 159 704
#: characters, 9.18 % of the corpus, 45 841 from a single document. The repair
#: is one line and it adds no boundary. It is off because it moves the first
#: span of essentially every segmented bundle and this round's budget held one
#: K2 build, which measured the outline gate; a default that changed every
#: bundle's first concept with no hit@8 number behind it would be the move this
#: round refused to make elsewhere. The number it needs is named in STATE.
DEFAULT_FIRST_SPAN_FROM_ZERO = False
#: Round 6 built it and left it off for want of one number; round 7 measured
#: that number and moved it (2026-09-10). Measured over the 39-document corpus:
#: every one of the 32 documents that gets a plan lost the text above its first
#: concept -- 159 704 characters, 9.18 % of the corpus, 45 841 from a single
#: document. It adds NO boundary; it moves one span's start, so it is a repair
#: to coverage rather than a segmentation rule, which is why it sits apart from
#: the arms above.
#:
#: The acceptance it had to clear was the same one round 6 set: the 12-position
#: reference does not move (it changes not one cell, 7 of 8 `pdf`, 3 of 3
#: `docx`, 10 of 12) AND hit@8 holds rank 1 on every row it held, measured end
#: to end on a K2 bundle built with it. Opt-out `--no-first-span-from-zero`.
DEFAULT_FIRST_SPAN_FROM_ZERO = True
#: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and
#: 6 by a RETRIEVAL regression rather than by the reference: they take the
#: sheet to 11 of 12 and `xlsx` to its fasit, and on a K2 bundle built with
#: them row 1 of the hit@8 set fell from rank 1 to rank 2.
#:
#: Round 7 decomposed that fall and it was never these rules' fault. Splitting
#: the gold document from 1 concept into 12 makes its own 12 concepts occupy
#: the document-prior signal's entire top tie group, so the concept leading the
#: body signal takes prior position 11 instead of 1 -- RRF reading a tie-break
#: by `concept_id` as if it were a measurement. `consume.DEFAULT_TIE_SHARED_RANK`
#: is the repair, and with it in place all three of round 7's acceptance
#: conditions hold at once. Opt-outs `--no-sheet-section-rows`,
#: `--no-keep-table-heading`.
DEFAULT_SHEET_SECTION_ROWS = True
DEFAULT_KEEP_TABLE_HEADING = True
#: Round 4's two PDF READERS, both off. They are not arms and they are not in
#: the paragraph above: an arm changes how the proposer cuts a text, and these
@ -530,27 +549,42 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--keep-table-heading",
action="store_true",
default=DEFAULT_KEEP_TABLE_HEADING,
help=(
"D1, passed to the proposer unchanged: keep a heading whose body is "
"empty only because a table opens under it, and absorb that table "
"into its span. Absent (the default) is OFF. Measured on a tender "
"price sheet: the concept count does not move (1 -> 1) and the "
"concept gains the heading line it was missing"
"into its span. ON since 2026-09-10. Measured on a tender price "
"sheet: the concept count does not move (1 -> 1) and the concept "
"gains the heading line it was missing"
),
)
build_parser.add_argument(
"--no-keep-table-heading",
action="store_false",
dest="keep_table_heading",
help="D1's explicit opt-out. A default a caller cannot turn off is not a default",
)
build_parser.add_argument(
"--sheet-section-rows",
action="store_true",
default=DEFAULT_SHEET_SECTION_ROWS,
help=(
"D3, passed to the proposer unchanged: cut an open table block at "
"the rows that label its sections -- a RUN of rows whose first cell "
"is a bare numeric label. The opposite direction from Arm E, which "
"decides how far a block extends; this decides where it is cut "
"inside. Absent (the default) is OFF. Measured on a tender price "
"sheet whose whole body is one table block: 1 concept -> 12, "
"against a reference of 11 cost groups plus the sheet's preamble"
"inside. ON since 2026-09-10, once the fusion stopped punishing the "
"split it produces. Measured on a tender price sheet whose whole "
"body is one table block: 1 concept -> 12, against a reference of "
"11 cost groups plus the sheet's preamble"
),
)
build_parser.add_argument(
"--no-sheet-section-rows",
action="store_false",
dest="sheet_section_rows",
help="D3's explicit opt-out. With --no-keep-table-heading it reproduces the pre-2026-09-10 bytes",
)
build_parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
@ -599,10 +633,20 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
default=DEFAULT_FIRST_SPAN_FROM_ZERO,
help=(
"Start the first concept at character 0, so the text above it is "
"carried by a segment instead of by none. Absent (the default) is "
"OFF. Measured over the 39-document corpus: 32 of the 32 documents "
"with a plan lose their opening today -- 159 704 characters, 9.18 "
"%% of the corpus. It adds no boundary and moves one span's start"
"carried by a segment instead of by none. Measured over the "
"39-document corpus: 32 of the 32 documents with a plan lost "
"their opening before this moved -- 159 704 characters, 9.18 "
"%% of the corpus. It adds no boundary and moves one span's start. "
"ON since 2026-09-10"
),
)
build_parser.add_argument(
"--no-first-span-from-zero",
action="store_false",
dest="first_span_from_zero",
help=(
"The repair's explicit opt-out, for a consumer who needs the "
"pre-2026-09-10 first span byte for byte"
),
)
build_parser.add_argument(

40
src/llm_ingestion_okf/consume.py
View file

@ -1063,6 +1063,29 @@ def document_scores(
#: fusion is invariant to any monotone transform of the individual signals.
RRF_K = 60
#: Whether concepts a signal scores EQUALLY share that group's first rank
#: instead of being ordered inside it by `concept_id`. ON since 2026-09-10.
#:
#: WHY IT MOVED. It shipped OFF on 2026-09-08 on a measurement -- hit@8 over
#: the six published questions fell 5 of 6 to 4 of 6 on the 629-concept bundle.
#: Round 7 re-measured that fall and it is CONDITIONAL on the document prior
#: being a sum: swept over `DOCUMENT_PRIOR_EXPONENT` x 3 bundles x 6 rows, the
#: lost row is lost at exponent 1.0 and held at 0.5. Round 6 moved that
#: exponent to 0.5 for an unrelated reason and nobody re-measured the pair, so
#: a rule was left off by a cost that had already been removed.
#:
#: WHAT IT BUYS. It is the repair for the defect that kept `--sheet-section-rows
#: --keep-table-heading` off the build default: a document split from 1 concept
#: into 12 puts its own 12 concepts in the prior signal's whole top tie group,
#: so the concept leading the body signal takes prior position 11 rather than
#: 1 and the document loses fused rank 1 to a single-concept competitor that
#: leads nothing. Fine-graining was being punished for being fine-grained.
#:
#: Measured at exponent 0.5 on three bundles, ranks per row: `[1, 1, 1, 1, 1,
#: None]` on all three with it on, against `[2, 1, 1, 1, 1, None]` on the split
#: bundle with it off. Opt-out `--no-tie-shared-rank`.
DEFAULT_TIE_SHARED_RANK = True
def concept_scores(
concepts: Sequence[Concept],
@ -1072,7 +1095,7 @@ def concept_scores(
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
lookup: bool = True,
tie_shared_rank: bool = False,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -1464,7 +1487,7 @@ def build_payload(
cost_vocabulary: bool = False,
reserve_top_rank: bool = False,
rarity_weight: bool = False,
tie_shared_rank: bool = False,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
withheld_titles: bool = False,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -1666,13 +1689,22 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser.add_argument(
"--tie-shared-rank",
action="store_true",
default=DEFAULT_TIE_SHARED_RANK,
help=(
"let concepts a signal scores EQUALLY share that score group's "
"first rank, so a signal that separates nothing contributes the "
"same constant to each of them instead of ordering them by id. OFF "
"by default. See docs/2026-09-08-rangeringsbom-sammensatte-ord.md"
"same constant to each of them instead of ordering them by id. ON "
"since 2026-09-10. See docs/2026-09-08-rangeringsbom-sammensatte-ord.md "
"for the rule and docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md "
"for why its published cost no longer holds"
),
)
parser.add_argument(
"--no-tie-shared-rank",
action="store_false",
dest="tie_shared_rank",
help="The rule's explicit opt-out, reproducing the pre-2026-09-10 order",
)
parser.add_argument(
"--withheld-titles",
action="store_true",