Kjell Tore Guttormsen
e503f6abd7
Row 7 left three mutants standing after v1.1 (11 of 14, bar 13): the passage signal reading no body, the field signal weighing no title and no path, and a flattened fusion (bm25.RRF_K = 10 000). Each switches off a mechanism the default ranking runs, and each moved 0 ranks, because every synthetic concept was short and opened with its own title -- none of the three ever decided a delivery here. Three bundles, one per mechanism, and one pinned set (set-mechanisms.json): - PASSAGE: a long concept answered in one window of its body, against ten short concepts whose titles carry the question's words. Rank 1; with no body windows it falls out of k (class b). - PATH: a concept named by its path alone, against ten decoys denser in the body. Rank 4 at k = 6; with no title/path weight it falls out of k. - FUSION: gold 1st in the passage signal and 21st in the field signal, one decoy 10th and 11th (rank sum 19 < 20). At RRF_K = 60 the gold leads, and it keeps the lead through K = 180 (measured in steps of 10); flattened, the decoy wins k = 1. Chose separate fixtures over one combined one because each fixture's comment can then name the single mechanism it pins. Verified per question against all 14 mutants: no previously felled mutant is lost. src/ is untouched, MUTANT_BAR and the roster are unchanged, the corpus pin moved with the corpus. Gate: rows 1 and 6 go 10/10 -> 13/13, row 7 11/14 -> 14/14 GREEN, rows 2/3/4 unchanged, GATE RED: rows 5, 8. Suite: 2444 passed, 1 skipped (+3), measured with FORCE_COLOR unset -- with FORCE_COLOR=3 in the environment Python 3.14 colours argparse help and test_each_arm_flag_carries_its_attribution fails, independently of this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
28 lines
1.2 KiB
JSON
28 lines
1.2 KiB
JSON
{
|
|
"set_id": "synthetic-mechanisms",
|
|
"bundle": "passage",
|
|
"note": "Three questions, one per mechanism the default BM25 ranking runs and no earlier fixture made decide a delivery: the passage signal's body windows (bundle `passage`), the field signal's title and path weight (bundle `path`), and the rank-fusion constant (bundle `fusion`). Each is delivered at the shipped defaults and falls out of k when its own mechanism is switched off, so row 1 is what fells M06, M07 and M10.",
|
|
"questions": [
|
|
{
|
|
"id": "PA1",
|
|
"question": "Hvem kontrollerer varmekablene i taket?",
|
|
"bundle": "passage",
|
|
"fasit": [{"by": "concept", "value": "aarbok/kapittel-tre", "quote": "kontrolleres av vaktmesteren"}]
|
|
},
|
|
{
|
|
"id": "PT1",
|
|
"question": "Hva gjelder for vester ved kanopadling?",
|
|
"bundle": "path",
|
|
"k": 6,
|
|
"fasit": [{"by": "concept", "value": "kanopadling/regel", "quote": "brukes paa vannet"}]
|
|
},
|
|
{
|
|
"id": "FU1",
|
|
"question": "Hva gjelder for isbading?",
|
|
"bundle": "fusion",
|
|
"k": 1,
|
|
"fasit": [{"by": "concept", "value": "badebok/kapittel", "quote": "isbading om morgenen"}]
|
|
}
|
|
],
|
|
"controls": []
|
|
}
|