open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,consume): the first span, and the fusion that punished fine-graining

Browse source 
Round 6 left two things behind: a measured coverage hole with the repair built
but not the number to move it, and a retrieval regression that kept two
spreadsheet rules off the default. Both are measured; four defaults move.

THE HOLE IS BIGGER THAN THE RULE BUILT FOR IT. Measured over the 39-document
corpus, the pre-move default left 207 435 characters -- 11.92 % -- in no
segment: 163 804 above the first entry, 26 041 between entries, 17 590 after
the last. `--first-span-from-zero` closes the first part entirely, 79 % of the
whole, leaving 43 631 (2.51 %) over 8 of 32 documents. It adds no boundary and
the K2 concept count is identical with and without it (425 = 425); hit@8 holds
[1,1,1,1,1,-] under both tie-breaks and the 12-position reference does not move
one cell.

THE RANKER WAS PUNISHING FINE-GRAINING FOR BEING FINE-GRAINED. Round 6 held
`--sheet-section-rows --keep-table-heading` back because on a bundle built with
them row 1 fell rank 1 -> 2, and ruled out the document prior because its rank
over 39 DOCUMENTS was 1 on both bundles. RRF reads a rank over CONCEPTS:
splitting the gold document 1 -> 12 puts its own twelve concepts in that
signal's whole top tie group, so the one leading the body signal takes position
11, contributing 1/71 where the undivided concept contributed 1/61. A signal
that scored them all EQUALLY still emitted twelve ranks, ordered by concept_id,
and the fusion read alphabetical order as a measurement.

AND THE REPAIR WAS ALREADY IN THE TREE, OFF FOR A COST THAT HAD BEEN REMOVED.
`--tie-shared-rank` shipped off 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6.
Swept over 2 prior exponents x 3 bundles x 6 rows: that fall exists only at
DOCUMENT_PRIOR_EXPONENT 1.0. Round 6 moved the exponent to 0.5 for an unrelated
reason and correctly reported it moved no hit@8 row -- nobody measured the
pair. A flag's "off by measurement" is a measurement of a CONFIGURATION, not a
property of the flag.

Shipped, each with an explicit opt-out and both directions measured rather than
asserted: --no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading reproduces the previous bundle byte for byte (diff -rq,
0 differences), and the no-flag build equals the explicit-flag build.
--no-tie-shared-rank reproduces the previous excerpt order -- the one change
here that alters a payload with no bundle changing.

K2 default: 436 concepts / 832 files, digest 8dff8a8e6c15d2f7..., hit@8 5 of 6
ranks [1,1,1,1,1,-]. The pin now holds its own known-negative on those bytes:
read with --no-tie-shared-rank the same bundle gives [2,1,1,1,1,-], so the
green assertion names its cause. Consumer cost is a re-run: 425/810 -> 436/832
on the reference corpus, 15/30 -> 26/52 on a five-document folder.

pptx and md measured end to end for the first time, on two hand-built
documents: md recovers 3 of 4 declared headings, pptx segments per slide only
where the converter recognised a title placeholder. A converter attribute leaks
into titles ({#slide-N}, {#sheet-1}) on 2 of 810 K2 files and 1 of 30 on a
five-document folder -- not fixed, because a filename is reduced from its title
and the fix renames concept ids a consumer has cited.

Suite 1486 (1478 before), ruff and mypy --strict clean over 21 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 15:40:01 +02:00
commit a364ef4c7d
11 changed files with 768 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

47
tests/test_tie_shared_rank.py
View file

@ -137,31 +137,54 @@ def test_shared_rank_lifts_the_concept_the_measuring_signals_lead(tmp_path: Path
assert _gold_rank(root, tie_shared_rank=True) == 1
def test_the_payload_is_byte_identical_with_the_flag_off(tmp_path: Path) -> None:
def test_the_payload_is_byte_identical_with_the_flag_on(tmp_path: Path) -> None:
"""The rule became the default 2026-09-10; saying so explicitly changes nothing.
The assertion is unchanged in kind -- the implicit and the explicit value
must produce the same bytes -- only the value it names moved.
"""
root = _tie_bundle(tmp_path / "bundle")
without = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION))
explicit_off = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, tie_shared_rank=False)
explicit_on = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, tie_shared_rank=True)
)
assert without == explicit_off
assert without == explicit_on
def test_the_opt_out_reproduces_the_order_the_default_used_to_give(tmp_path: Path) -> None:
"""The other half: a consumer needing the pre-2026-09-10 order can have it.
Load-bearing rather than symmetric. `--no-tie-shared-rank` is the only
thing standing between a consumer pinned to the old excerpt order and a
silent reordering, so the opt-out needs a test that goes red if it stops
being a real alternative -- which it would be if it produced the same
bytes as the default on the very fixture built to separate them.
"""
root = _tie_bundle(tmp_path / "bundle")
default = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION, k=3))
opted_out = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=False)
)
assert default != opted_out
def test_the_flag_changes_the_payload_it_is_meant_to_change(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
off = okf_consume.build_payload(root, question=QUESTION, k=3)
on = okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=True)
off = okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=False)
on = okf_consume.build_payload(root, question=QUESTION, k=3)
delivered_off = [excerpt["concept_id"] for excerpt in off["excerpts"]] # type: ignore[index]
delivered_on = [excerpt["concept_id"] for excerpt in on["excerpts"]] # type: ignore[index]
assert not any(str(cid).endswith("zz-gull") for cid in delivered_off)
assert str(delivered_on[0]).endswith("zz-gull")
def test_the_cli_exposes_the_flag_and_defaults_it_off(tmp_path: Path) -> None:
def test_the_cli_exposes_the_flag_and_defaults_it_on(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
parsed = okf_consume.parse_args([str(root), "--question", QUESTION])
assert parsed.tie_shared_rank is False
parsed_on = okf_consume.parse_args([str(root), "--question", QUESTION, "--tie-shared-rank"])
assert parsed_on.tie_shared_rank is True
# The other flag this session added, asserted here so "both default off"
# is one measurement rather than two files' worth of trust.
assert parsed.tie_shared_rank is True
parsed_off = okf_consume.parse_args([str(root), "--question", QUESTION, "--no-tie-shared-rank"])
assert parsed_off.tie_shared_rank is False
# `--withheld-titles` did NOT move with it, asserted here so the two are
# one measurement rather than two files' worth of trust: it is off for a
# reason of BYTES, which nothing this round touched.
assert parsed.withheld_titles is False