open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,consume): the first span, and the fusion that punished fine-graining

Browse source 
Round 6 left two things behind: a measured coverage hole with the repair built
but not the number to move it, and a retrieval regression that kept two
spreadsheet rules off the default. Both are measured; four defaults move.

THE HOLE IS BIGGER THAN THE RULE BUILT FOR IT. Measured over the 39-document
corpus, the pre-move default left 207 435 characters -- 11.92 % -- in no
segment: 163 804 above the first entry, 26 041 between entries, 17 590 after
the last. `--first-span-from-zero` closes the first part entirely, 79 % of the
whole, leaving 43 631 (2.51 %) over 8 of 32 documents. It adds no boundary and
the K2 concept count is identical with and without it (425 = 425); hit@8 holds
[1,1,1,1,1,-] under both tie-breaks and the 12-position reference does not move
one cell.

THE RANKER WAS PUNISHING FINE-GRAINING FOR BEING FINE-GRAINED. Round 6 held
`--sheet-section-rows --keep-table-heading` back because on a bundle built with
them row 1 fell rank 1 -> 2, and ruled out the document prior because its rank
over 39 DOCUMENTS was 1 on both bundles. RRF reads a rank over CONCEPTS:
splitting the gold document 1 -> 12 puts its own twelve concepts in that
signal's whole top tie group, so the one leading the body signal takes position
11, contributing 1/71 where the undivided concept contributed 1/61. A signal
that scored them all EQUALLY still emitted twelve ranks, ordered by concept_id,
and the fusion read alphabetical order as a measurement.

AND THE REPAIR WAS ALREADY IN THE TREE, OFF FOR A COST THAT HAD BEEN REMOVED.
`--tie-shared-rank` shipped off 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6.
Swept over 2 prior exponents x 3 bundles x 6 rows: that fall exists only at
DOCUMENT_PRIOR_EXPONENT 1.0. Round 6 moved the exponent to 0.5 for an unrelated
reason and correctly reported it moved no hit@8 row -- nobody measured the
pair. A flag's "off by measurement" is a measurement of a CONFIGURATION, not a
property of the flag.

Shipped, each with an explicit opt-out and both directions measured rather than
asserted: --no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading reproduces the previous bundle byte for byte (diff -rq,
0 differences), and the no-flag build equals the explicit-flag build.
--no-tie-shared-rank reproduces the previous excerpt order -- the one change
here that alters a payload with no bundle changing.

K2 default: 436 concepts / 832 files, digest 8dff8a8e6c15d2f7..., hit@8 5 of 6
ranks [1,1,1,1,1,-]. The pin now holds its own known-negative on those bytes:
read with --no-tie-shared-rank the same bundle gives [2,1,1,1,1,-], so the
green assertion names its cause. Consumer cost is a re-run: 425/810 -> 436/832
on the reference corpus, 15/30 -> 26/52 on a five-document folder.

pptx and md measured end to end for the first time, on two hand-built
documents: md recovers 3 of 4 declared headings, pptx segments per slide only
where the converter recognised a title placeholder. A converter attribute leaks
into titles ({#slide-N}, {#sheet-1}) on 2 of 810 K2 files and 1 of 30 on a
five-document folder -- not fixed, because a filename is reduced from its title
and the fix renames concept ids a consumer has cited.

Suite 1486 (1478 before), ruff and mypy --strict clean over 21 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 15:40:01 +02:00
commit a364ef4c7d
11 changed files with 768 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

65
tests/test_okf_consume.py
View file

@ -522,10 +522,19 @@ def test_reversing_the_input_order_does_not_change_the_output_order() -> None:
def test_a_concept_in_a_high_scoring_document_outranks_an_equally_lexical_one() -> None:
# `tie_shared_rank=False` for the same reason `lookup=False` appears
# elsewhere in this file: the claim is about the DOCUMENT PRIOR, and the
# default tie-break (shared since 2026-09-10) puts this fixture's two
# concepts in the same prior tie group, which makes the assertion true
# in both directions and so measures nothing. Isolate the stage under test.
concepts = _fixture_concepts()
question = "Hvordan skal prisene fylles ut?"
lifted = okf_consume.concept_scores(concepts, question, {"krav": 10.0, "dyp": 0.0})
dropped = okf_consume.concept_scores(concepts, question, {"krav": 0.0, "dyp": 10.0})
lifted = okf_consume.concept_scores(
concepts, question, {"krav": 10.0, "dyp": 0.0}, tie_shared_rank=False
)
dropped = okf_consume.concept_scores(
concepts, question, {"krav": 0.0, "dyp": 10.0}, tie_shared_rank=False
)
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/pristabell")
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/pristabell")
assert krav_first < krav_later
@ -1771,7 +1780,9 @@ def test_counting_leaves_the_identifier_worth_no_more_than_the_common_word() ->
# The claim here is about the FUSION, so the fusion is what is read.
for identifier_in_body in (False, True):
concepts = _rarity_corpus(identifier_in_body=identifier_in_body)
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, lookup=False)
ranked = okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, lookup=False, tie_shared_rank=False
)
assert _rank_of(ranked, RARITY_GOLD) == 18
lexical = {concept.concept_id: value for concept, _, value in ranked}
assert lexical["krav/c-00"] == 6
@ -1786,7 +1797,9 @@ def test_weighting_a_hit_by_its_rarity_lifts_the_identifier_into_the_cut() -> No
weights = okf_consume.rarity_weights(
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
)
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
ranked = okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, weights=weights, tie_shared_rank=False
)
assert _rank_of(ranked, RARITY_GOLD) <= 8
@ -1817,9 +1830,28 @@ def test_a_rarity_weight_cannot_move_a_signal_the_gold_already_leads() -> None:
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1
ranked = okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False, tie_shared_rank=False
)
assert _rank_of(ranked, RARITY_GOLD) == 18
# AND the falsification under the 2026-09-10 tie-break, measured rather
# than assumed -- with the COST it carries on this fixture written down.
# The claim survives and is stronger (30 -> 30 with and without the
# weight), but the gold's absolute rank is WORSE than the 18 above: the id
# order that had put it at 18 was luck inside a tie group, and shared ranks
# take that luck away in both directions. This fixture is the adverse
# shape for the rule -- one signal separates, two do not, and the one that
# separates is the one the gold loses. It is recorded here because it is
# the honest limit of the end-to-end measurement that moved the default:
# ranks held at 1 on every row of three real bundles, and a synthetic
# corpus of 30 near-identical concepts is not one of them.
under_default = okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False
)
unweighted_under_default = okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, lookup=False
)
assert _rank_of(under_default, RARITY_GOLD) == 30
assert _rank_of(unweighted_under_default, RARITY_GOLD) == 30
def test_the_weight_leaves_the_lexical_count_a_count_so_the_cut_is_untouched() -> None:
@ -1836,7 +1868,9 @@ def test_the_weight_leaves_the_lexical_count_a_count_so_the_cut_is_untouched() -
}
weighted = {
c.concept_id: v
for c, _, v in okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
for c, _, v in okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, weights=weights, tie_shared_rank=False
)
}
assert plain == weighted
@ -1989,14 +2023,29 @@ def test_the_weight_reaches_the_title_signal_and_reorders_it() -> None:
assert _signal_rank(concepts, question, "krav/t-29", weights) == 1
# `lookup=False`: the gold's title bears the identifier, so the lookup
# partition answers this question at rank one. The fusion is the claim.
plain = okf_consume.concept_scores(concepts, question, {}, lookup=False)
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights, lookup=False)
plain = okf_consume.concept_scores(concepts, question, {}, lookup=False, tie_shared_rank=False)
weighted = okf_consume.concept_scores(
concepts, question, {}, weights=weights, lookup=False, tie_shared_rank=False
)
# Fused: 30 -> 18. The title signal is reordered from last to first and the
# fused rank moves by twelve, not to one -- the other two signals still
# cannot see the identifier. That gap IS the finding of this session, and
# the numbers are here so a change to either half shows up as a diff.
assert _rank_of(plain, "krav/t-29") == 30
assert _rank_of(weighted, "krav/t-29") == 18
# AND WHAT THE 2026-09-10 TIE-BREAK DID TO THAT GAP, measured rather than
# left implicit. This fixture is the shape shared ranks were built for:
# every body is identical and the prior is empty, so both of the signals
# that "cannot see the identifier" separate NOTHING, and under shared ranks
# they stop ordering by id and contribute one constant each. The title
# signal is then alone in deciding and the gold goes to rank 1 -- the gap
# this test named does not survive the fusion change, and a reader of
# `docs/2026-09-08-sjeldenhetsvekt.md` needs to know the figure there was
# measured under the older tie-break.
under_default = okf_consume.concept_scores(
concepts, question, {}, weights=weights, lookup=False
)
assert _rank_of(under_default, "krav/t-29") == 1
# --- Step 14: exact identifier matching and the lookup signal (O2c) -----------