open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(retrieval-gate): three fixtures make M06, M07 and M10 decide a delivery — row 7 14 of 14

Browse source 
Row 7 left three mutants standing after v1.1 (11 of 14, bar 13): the passage
signal reading no body, the field signal weighing no title and no path, and a
flattened fusion (bm25.RRF_K = 10 000). Each switches off a mechanism the
default ranking runs, and each moved 0 ranks, because every synthetic concept
was short and opened with its own title -- none of the three ever decided a
delivery here.

Three bundles, one per mechanism, and one pinned set (set-mechanisms.json):

- PASSAGE: a long concept answered in one window of its body, against ten
  short concepts whose titles carry the question's words. Rank 1; with no
  body windows it falls out of k (class b).
- PATH: a concept named by its path alone, against ten decoys denser in the
  body. Rank 4 at k = 6; with no title/path weight it falls out of k.
- FUSION: gold 1st in the passage signal and 21st in the field signal, one
  decoy 10th and 11th (rank sum 19 < 20). At RRF_K = 60 the gold leads, and
  it keeps the lead through K = 180 (measured in steps of 10); flattened, the
  decoy wins k = 1.

Chose separate fixtures over one combined one because each fixture's comment
can then name the single mechanism it pins. Verified per question against all
14 mutants: no previously felled mutant is lost. src/ is untouched, MUTANT_BAR
and the roster are unchanged, the corpus pin moved with the corpus.

Gate: rows 1 and 6 go 10/10 -> 13/13, row 7 11/14 -> 14/14 GREEN, rows 2/3/4
unchanged, GATE RED: rows 5, 8. Suite: 2444 passed, 1 skipped (+3), measured
with FORCE_COLOR unset -- with FORCE_COLOR=3 in the environment Python 3.14
colours argparse help and test_each_arm_flag_carries_its_attribution fails,
independently of this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-22 22:41:01 +02:00
commit e503f6abd7
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
5 changed files with 234 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

181
tools/okf_retrieval_gate.py
View file

@ -434,6 +434,158 @@ BUDGET = BundleSpec(
),
)
# --- three fixtures, one per mechanism the default ranking runs ---------------
#
# Until 2026-09-22 row 7 left three mutants standing (M06, M07, M10): every
# synthetic concept was short and opened with its own title, so the passage
# signal, the title/path weight and the fusion constant never DECIDED a
# delivery here, and switching any of them off moved 0 ranks. Each bundle below
# is built so that exactly one of them does. None of them is a tuning of the
# ranker: the corpus is pinned (`SPECS_SHA256`) and `src/` is untouched.
_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi. "
#: THE PASSAGE SIGNAL. One long concept answers in ONE window of its body; ten
#: short concepts carry the question's words in their TITLES and none in their
#: bodies. The field signal reads ten title-weighted matches in short documents
#: above one long body, so only the passage signal -- the best WINDOW, not the
#: whole concept -- puts the gold first. Remove the body windows (M06) and the
#: passage signal reads titles alone, where the gold has none of the words.
PASSAGE = BundleSpec(
"retrieval-passage",
(
DocumentSpec(
"aarbok",
"aarbok.md",
(
ConceptSpec(
"kapittel-tre",
"Kapittel tre i aarboka",
_NOTE * 15
+ "Varmekablene i taket kontrolleres av vaktmesteren hver oktober. "
+ _NOTE * 15,
),
),
),
*(
DocumentSpec(
f"rundskriv-{number:02d}",
f"rundskriv-{number:02d}.md",
(
ConceptSpec(
f"skriv-{number:02d}",
f"Varmekablene i taket, skriv {number:02d}",
"Skrivet er sendt ut og arkivert.",
),
),
)
for number in range(1, 11)
),
),
)
#: THE TITLE/PATH WEIGHT. The gold is named by its PATH alone -- the question's
#: `kanopadling` is its directory and its source file and occurs in no body --
#: while ten decoys carry the question's other word more densely than the gold
#: does, one fewer time each. The path weight puts the gold first in the field
#: signal, which lifts it into k (rank 4 at `k` = 6). Weigh no title and no path
#: (M07) and both signals read bodies only, where the decoys outrank it and it
#: falls out of k.
PATH = BundleSpec(
"retrieval-path",
(
DocumentSpec(
"kanopadling",
"kanopadling.md",
(ConceptSpec("regel", "Regel om vester", "Vester brukes paa vannet."),),
),
*(
DocumentSpec(
f"turnotat-{number:02d}",
f"turnotat-{number:02d}.md",
(
ConceptSpec(
f"notat-{number:02d}",
f"Turnotat {number:02d}",
" ".join(["Vester henger i boden."] * (11 - number))
+ " Notatet er arkivert.",
),
),
)
for number in range(1, 11)
),
),
)
_FUSION_NOTE = "Styret gjennomgaar saken og sekretaeren arkiverer den. "
#: THE FUSION CONSTANT. Reciprocal-rank fusion trades a concept with one very
#: good rank against one with two middling ranks, and `RRF_K` sets the price.
#: The gold is FIRST in the passage signal and 21st in the field signal (a long
#: body, below ten short `isbading-*` path matches, nine longer ones and the
#: decoy); the one decoy is 10th and 11th, a smaller rank SUM (19 against 20).
#: A flattened fusion (M10, `RRF_K` = 10 000, where only the sum counts) puts
#: the decoy first and `k` = 1 no longer reaches the gold. At the shipped
#: `RRF_K` = 60 the gold leads, and measured over K in steps of 10 it keeps the
#: lead through K = 180 and loses it at 190. The eight long log pages hold the
#: decoy's passage rank at 10th and rank below the gold in both signals.
FUSION = BundleSpec(
"retrieval-fusion",
(
DocumentSpec(
"badebok",
"badebok.md",
(
ConceptSpec(
"kapittel",
"Kapittel i badeboka",
_FUSION_NOTE
* 12
+ "Isbading ved brygga, isbading om morgenen, isbading med vakt, "
"isbading hver dag. " + _FUSION_NOTE * 12,
),
),
),
*(
DocumentSpec(
f"logg-{number:02d}",
f"logg-{number:02d}.md",
(
ConceptSpec(
"side",
f"Loggside {number:02d}",
_FUSION_NOTE * 16
+ "Isbading ved brygga, isbading om kvelden, isbading med vakt. "
+ _FUSION_NOTE * 16,
),
),
)
for number in range(8)
),
DocumentSpec(
"isbading-lapp-00",
"isbading-lapp-00.md",
(ConceptSpec("lapp", "Lapp 00", _FUSION_NOTE * 5 + "Isbading er tillatt."),),
),
*(
DocumentSpec(
f"isbading-{number:02d}",
f"isbading-{number:02d}.md",
(ConceptSpec("oppslag", f"Oppslag {number:02d}", "Oppslaget er arkivert."),),
)
for number in range(10)
),
*(
DocumentSpec(
f"isbading-arkiv-{number:02d}",
f"isbading-arkiv-{number:02d}.md",
(ConceptSpec("mappe", f"Mappe {number:02d}", _FUSION_NOTE * 10),),
)
for number in range(9)
),
),
)
SPECS: Mapping[str, BundleSpec] = {
"positive": POSITIVE,
"miss": MISS,
@ -441,6 +593,9 @@ SPECS: Mapping[str, BundleSpec] = {
"budget": BUDGET,
"lookup": LOOKUP,
"quota": QUOTA,
"passage": PASSAGE,
"path": PATH,
"fusion": FUSION,
}
@ -483,7 +638,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
#: The synthetic corpus, pinned the way the sets are.
SPECS_SHA256 = "23c447add9a58ee2e454ef5d5b92d5c3c97d441cd883d203aaf2f275dd2f68f9"
SPECS_SHA256 = "abf24378c32b02753916c36eb502f9b27183a5d1c7e1914b40773bde393878b5"
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
@ -648,6 +803,7 @@ SYNTHETIC_SETS: dict[str, str] = {
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
"set-mechanisms.json": "435ce620c30c94b0d151335883583520490482be2ade6797524ae928a9f1eab7",
}
@ -1599,8 +1755,10 @@ class Mutant:
expected_row: int
patch: Callable[[], contextlib.AbstractContextManager[None]]
#: Why this one may be invisible to a gate that judges DELIVERY. Written
#: for the two that survive, so a survivor is a statement about the
#: mechanism and not a shrug.
#: for a mutant that survives, so a survivor is a statement about the
#: mechanism and not a shrug. None does since 2026-09-22: the three that
#: carried a note (M06, M07, M10) each got a fixture that makes their
#: mechanism decide a delivery (`PASSAGE`, `PATH`, `FUSION`).
note: str = ""
@ -1717,23 +1875,12 @@ MUTANTS: tuple[Mutant, ...] = (
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
Mutant(
"M06 the passage signal reads no body",
1,
lambda: _patched_bm25(windows=lambda body: []),
note=(
"the field signal still reads every body, and no synthetic concept is "
"long enough for its best window to decide a delivery"
),
"M06 the passage signal reads no body", 1, lambda: _patched_bm25(windows=lambda body: [])
),
Mutant(
"M07 the field signal weighs no title and no path",
1,
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
note=(
"every synthetic body opens with its own title as a heading "
"(`build_bundle`), so the weight only re-weights words the body "
"already carries"
),
),
Mutant("M08 every token matches every token", 4, _every_term_everything),
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
@ -1741,10 +1888,6 @@ MUTANTS: tuple[Mutant, ...] = (
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
1,
lambda: _patched_bm25(RRF_K=10_000),
note=(
"over two signals K reorders only a concept whose two ranks lie far "
"apart against one whose ranks are close, and no fixture here holds that"
),
),
Mutant("M11 the cut takes the LAST k", 1, _last_k),
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),