test(retrieval-gate): three fixtures make M06, M07 and M10 decide a delivery — row 7 14 of 14

Row 7 left three mutants standing after v1.1 (11 of 14, bar 13): the passage
signal reading no body, the field signal weighing no title and no path, and a
flattened fusion (bm25.RRF_K = 10 000). Each switches off a mechanism the
default ranking runs, and each moved 0 ranks, because every synthetic concept
was short and opened with its own title -- none of the three ever decided a
delivery here.

Three bundles, one per mechanism, and one pinned set (set-mechanisms.json):

- PASSAGE: a long concept answered in one window of its body, against ten
  short concepts whose titles carry the question's words. Rank 1; with no
  body windows it falls out of k (class b).
- PATH: a concept named by its path alone, against ten decoys denser in the
  body. Rank 4 at k = 6; with no title/path weight it falls out of k.
- FUSION: gold 1st in the passage signal and 21st in the field signal, one
  decoy 10th and 11th (rank sum 19 < 20). At RRF_K = 60 the gold leads, and
  it keeps the lead through K = 180 (measured in steps of 10); flattened, the
  decoy wins k = 1.

Chose separate fixtures over one combined one because each fixture's comment
can then name the single mechanism it pins. Verified per question against all
14 mutants: no previously felled mutant is lost. src/ is untouched, MUTANT_BAR
and the roster are unchanged, the corpus pin moved with the corpus.

Gate: rows 1 and 6 go 10/10 -> 13/13, row 7 11/14 -> 14/14 GREEN, rows 2/3/4
unchanged, GATE RED: rows 5, 8. Suite: 2444 passed, 1 skipped (+3), measured
with FORCE_COLOR unset -- with FORCE_COLOR=3 in the environment Python 3.14
colours argparse help and test_each_arm_flag_carries_its_attribution fails,
independently of this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-22 22:41:01 +02:00
commit e503f6abd7
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
5 changed files with 234 additions and 48 deletions

View file

@ -0,0 +1,28 @@
{
"set_id": "synthetic-mechanisms",
"bundle": "passage",
"note": "Three questions, one per mechanism the default BM25 ranking runs and no earlier fixture made decide a delivery: the passage signal's body windows (bundle `passage`), the field signal's title and path weight (bundle `path`), and the rank-fusion constant (bundle `fusion`). Each is delivered at the shipped defaults and falls out of k when its own mechanism is switched off, so row 1 is what fells M06, M07 and M10.",
"questions": [
{
"id": "PA1",
"question": "Hvem kontrollerer varmekablene i taket?",
"bundle": "passage",
"fasit": [{"by": "concept", "value": "aarbok/kapittel-tre", "quote": "kontrolleres av vaktmesteren"}]
},
{
"id": "PT1",
"question": "Hva gjelder for vester ved kanopadling?",
"bundle": "path",
"k": 6,
"fasit": [{"by": "concept", "value": "kanopadling/regel", "quote": "brukes paa vannet"}]
},
{
"id": "FU1",
"question": "Hva gjelder for isbading?",
"bundle": "fusion",
"k": 1,
"fasit": [{"by": "concept", "value": "badebok/kapittel", "quote": "isbading om morgenen"}]
}
],
"controls": []
}