Kjell Tore Guttormsen
a364ef4c7d
Round 6 left two things behind: a measured coverage hole with the repair built
but not the number to move it, and a retrieval regression that kept two
spreadsheet rules off the default. Both are measured; four defaults move.
THE HOLE IS BIGGER THAN THE RULE BUILT FOR IT. Measured over the 39-document
corpus, the pre-move default left 207 435 characters -- 11.92 % -- in no
segment: 163 804 above the first entry, 26 041 between entries, 17 590 after
the last. `--first-span-from-zero` closes the first part entirely, 79 % of the
whole, leaving 43 631 (2.51 %) over 8 of 32 documents. It adds no boundary and
the K2 concept count is identical with and without it (425 = 425); hit@8 holds
[1,1,1,1,1,-] under both tie-breaks and the 12-position reference does not move
one cell.
THE RANKER WAS PUNISHING FINE-GRAINING FOR BEING FINE-GRAINED. Round 6 held
`--sheet-section-rows --keep-table-heading` back because on a bundle built with
them row 1 fell rank 1 -> 2, and ruled out the document prior because its rank
over 39 DOCUMENTS was 1 on both bundles. RRF reads a rank over CONCEPTS:
splitting the gold document 1 -> 12 puts its own twelve concepts in that
signal's whole top tie group, so the one leading the body signal takes position
11, contributing 1/71 where the undivided concept contributed 1/61. A signal
that scored them all EQUALLY still emitted twelve ranks, ordered by concept_id,
and the fusion read alphabetical order as a measurement.
AND THE REPAIR WAS ALREADY IN THE TREE, OFF FOR A COST THAT HAD BEEN REMOVED.
`--tie-shared-rank` shipped off 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6.
Swept over 2 prior exponents x 3 bundles x 6 rows: that fall exists only at
DOCUMENT_PRIOR_EXPONENT 1.0. Round 6 moved the exponent to 0.5 for an unrelated
reason and correctly reported it moved no hit@8 row -- nobody measured the
pair. A flag's "off by measurement" is a measurement of a CONFIGURATION, not a
property of the flag.
Shipped, each with an explicit opt-out and both directions measured rather than
asserted: --no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading reproduces the previous bundle byte for byte (diff -rq,
0 differences), and the no-flag build equals the explicit-flag build.
--no-tie-shared-rank reproduces the previous excerpt order -- the one change
here that alters a payload with no bundle changing.
K2 default: 436 concepts / 832 files, digest 8dff8a8e6c15d2f7..., hit@8 5 of 6
ranks [1,1,1,1,1,-]. The pin now holds its own known-negative on those bytes:
read with --no-tie-shared-rank the same bundle gives [2,1,1,1,1,-], so the
green assertion names its cause. Consumer cost is a re-run: 425/810 -> 436/832
on the reference corpus, 15/30 -> 26/52 on a five-document folder.
pptx and md measured end to end for the first time, on two hand-built
documents: md recovers 3 of 4 declared headings, pptx segments per slide only
where the converter recognised a title placeholder. A converter attribute leaks
into titles ({#slide-N}, {#sheet-1}) on 2 of 810 K2 files and 1 of 30 on a
five-document folder -- not fixed, because a filename is reduced from its title
and the fix renames concept ids a consumer has cited.
Suite 1486 (1478 before), ruff and mypy --strict clean over 21 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
84 lines
3.1 KiB
Python
84 lines
3.1 KiB
Python
"""What round 7 moved to the default, and the opt-out that reproduces the old bytes.
|
|
|
|
Three moves, each with its own acceptance measurement in
|
|
`docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md`:
|
|
|
|
* `--first-span-from-zero`, because 32 of the 32 corpus documents with a plan
|
|
left their opening text in NO segment -- 9.18 % of the corpus.
|
|
* `--sheet-section-rows --keep-table-heading`, which were held back in round 6
|
|
by a retrieval regression the third move removes.
|
|
* `tie_shared_rank` on the reading side, which is what removed it.
|
|
|
|
Every move keeps an explicit opt-out: a default a caller cannot turn off is
|
|
not a default, and a consumer needing the previous bytes needs a way to say so.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
from llm_ingestion_okf import cli, consume
|
|
|
|
|
|
def test_the_build_defaults_carry_the_three_rules_round_seven_moved() -> None:
|
|
assert cli.DEFAULT_FIRST_SPAN_FROM_ZERO is True
|
|
assert cli.DEFAULT_SHEET_SECTION_ROWS is True
|
|
assert cli.DEFAULT_KEEP_TABLE_HEADING is True
|
|
|
|
|
|
def test_the_reading_side_shares_a_rank_a_signal_did_not_earn_by_default() -> None:
|
|
"""`tie_shared_rank` is what let the two build rules above ship.
|
|
|
|
Measured on the D1+D3 bundle: the gold document splits from 1 concept into
|
|
12, its own 12 concepts occupy the document-prior signal's whole top tie
|
|
group, and the concept leading the body signal lands at prior position 11
|
|
instead of 1. Row 1 falls from fused rank 1 to 2. Shared ranks put it back.
|
|
"""
|
|
assert consume.DEFAULT_TIE_SHARED_RANK is True
|
|
|
|
|
|
def test_every_moved_default_has_an_opt_out_that_the_parser_accepts() -> None:
|
|
args = cli.parse_args(
|
|
[
|
|
"build",
|
|
"src",
|
|
"--bundle",
|
|
"out",
|
|
"--bundle-id",
|
|
"x",
|
|
"--no-first-span-from-zero",
|
|
"--no-sheet-section-rows",
|
|
"--no-keep-table-heading",
|
|
]
|
|
)
|
|
assert args.first_span_from_zero is False
|
|
assert args.sheet_section_rows is False
|
|
assert args.keep_table_heading is False
|
|
|
|
|
|
def test_the_build_defaults_reach_the_parser_without_a_flag() -> None:
|
|
args = cli.parse_args(["build", "src", "--bundle", "out", "--bundle-id", "x"])
|
|
assert args.first_span_from_zero is True
|
|
assert args.sheet_section_rows is True
|
|
assert args.keep_table_heading is True
|
|
|
|
|
|
def test_the_consume_parser_offers_an_opt_out_for_the_shared_rank() -> None:
|
|
assert consume.parse_args(["b", "--question", "q"]).tie_shared_rank is True
|
|
assert (
|
|
consume.parse_args(["b", "--question", "q", "--no-tie-shared-rank"]).tie_shared_rank
|
|
is False
|
|
)
|
|
|
|
|
|
def test_build_payload_defaults_to_the_shared_rank(tmp_path: Path) -> None:
|
|
"""The default reaches the API, not only the CLI.
|
|
|
|
`okf skill` emits `okf consume`, but `tests/test_default_bundle_pin.py` and
|
|
every measurement harness call `build_payload` directly. A default living
|
|
in `argparse` alone would make the two disagree.
|
|
"""
|
|
import inspect
|
|
|
|
signature = inspect.signature(consume.build_payload)
|
|
assert signature.parameters["tie_shared_rank"].default is True
|