open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(retrieval-gate): row 7 must fell M06, M07 and M10 — red

Browse source 
The three mutants that survive v1.1 each switch off a mechanism the default
BM25 ranking runs: the passage signal's body windows, the field signal's
title and path weight, and the rank-fusion constant. A gate that cannot see
them removed cannot see them break. One test per survivor, driven through
the gate's own `deterministic_rows` on the pinned sets and corpus: it fails
today with an AssertionError (no row got worse), and the same test passes
for M04, which the gate already fells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-22 22:19:12 +02:00
commit 09d7e6ee08
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 28 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

28
tests/test_retrieval_gate.py
View file

@ -524,6 +524,34 @@ def test_a_mutant_is_felled_by_the_row_that_got_worse_and_never_by_one_that_did_
assert after[3] >= before[3]
@pytest.mark.parametrize(
"prefix",
[
# The three that survived v1.1: each is a mechanism the default ranking
# RUNS, so a gate that cannot see it removed cannot see it break.
"M06 ", # the passage signal reads no body
"M07 ", # the field signal weighs no title and no path
"M10 ", # the fusion is flattened
],
)
def test_row_seven_fells_every_mutant_of_a_mechanism_the_ranking_runs(
tmp_path: Path, prefix: str
) -> None:
"""A survivor of row 7 is a mechanism the synthetic corpus never makes
decide a delivery. Felled means a row got WORSE on the pinned sets and the
pinned corpus -- never a row that merely moved."""
(mutant,) = [m for m in gate.MUTANTS if m.label.startswith(prefix)]
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
before = {row.number: row.k - row.m for row in gate.deterministic_rows(cases)}
with mutant.patch():
rows = gate.deterministic_rows(
[gate.measure_case(case.question_set, case.bundles) for case in cases]
)
after = {row.number: row.k - row.m for row in rows}
worse = [number for number in sorted(before) if after[number] < before[number]]
assert worse, f"{mutant.label} survives: no row got worse ({before} -> {after})"
# --- row 8 --------------------------------------------------------------------