test(retrieval-gate): three fixtures make M06, M07 and M10 decide a delivery — row 7 14 of 14
Row 7 left three mutants standing after v1.1 (11 of 14, bar 13): the passage signal reading no body, the field signal weighing no title and no path, and a flattened fusion (bm25.RRF_K = 10 000). Each switches off a mechanism the default ranking runs, and each moved 0 ranks, because every synthetic concept was short and opened with its own title -- none of the three ever decided a delivery here. Three bundles, one per mechanism, and one pinned set (set-mechanisms.json): - PASSAGE: a long concept answered in one window of its body, against ten short concepts whose titles carry the question's words. Rank 1; with no body windows it falls out of k (class b). - PATH: a concept named by its path alone, against ten decoys denser in the body. Rank 4 at k = 6; with no title/path weight it falls out of k. - FUSION: gold 1st in the passage signal and 21st in the field signal, one decoy 10th and 11th (rank sum 19 < 20). At RRF_K = 60 the gold leads, and it keeps the lead through K = 180 (measured in steps of 10); flattened, the decoy wins k = 1. Chose separate fixtures over one combined one because each fixture's comment can then name the single mechanism it pins. Verified per question against all 14 mutants: no previously felled mutant is lost. src/ is untouched, MUTANT_BAR and the roster are unchanged, the corpus pin moved with the corpus. Gate: rows 1 and 6 go 10/10 -> 13/13, row 7 11/14 -> 14/14 GREEN, rows 2/3/4 unchanged, GATE RED: rows 5, 8. Suite: 2444 passed, 1 skipped (+3), measured with FORCE_COLOR unset -- with FORCE_COLOR=3 in the environment Python 3.14 colours argparse help and test_each_arm_flag_carries_its_attribution fails, independently of this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
09d7e6ee08
commit
e503f6abd7
5 changed files with 234 additions and 48 deletions
|
|
@ -632,14 +632,15 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
|
|||
by_number = {row.number: row for row in rows}
|
||||
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
|
||||
# Rows 2 and 3 are green again since the synthetic corpus was re-measured
|
||||
# for BM25 (2026-09-21). Row 7 is red on three survivors, each printed with
|
||||
# what it moved.
|
||||
assert [row.number for row in rows if row.fails] == [5, 7, 8]
|
||||
assert (by_number[1].k, by_number[1].m) == (10, 10)
|
||||
# for BM25 (2026-09-21). Row 7 fells all 14 since 2026-09-22, when the three
|
||||
# v1.1 survivors each got a fixture that makes their mechanism decide a
|
||||
# delivery (`set-mechanisms.json`, three more row-1 units).
|
||||
assert [row.number for row in rows if row.fails] == [5, 8]
|
||||
assert (by_number[1].k, by_number[1].m) == (13, 13)
|
||||
assert (by_number[2].k, by_number[2].m) == (7, 7)
|
||||
assert (by_number[3].k, by_number[3].m) == (5, 5)
|
||||
assert (by_number[6].k, by_number[6].m) == (10, 10)
|
||||
assert (by_number[7].k, by_number[7].m) == (11, 14)
|
||||
assert (by_number[6].k, by_number[6].m) == (13, 13)
|
||||
assert (by_number[7].k, by_number[7].m) == (14, 14)
|
||||
|
||||
|
||||
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
|
||||
|
|
@ -655,7 +656,7 @@ def test_the_command_exits_one_and_prints_every_row(
|
|||
printed = capsys.readouterr().out
|
||||
for number in range(1, 9):
|
||||
assert f"\n{number} " in f"\n{printed}"
|
||||
assert "GATE RED: rows 5, 7, 8" in printed
|
||||
assert "GATE RED: rows 5, 8" in printed
|
||||
|
||||
|
||||
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue