open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(retrieval-gate): three fixtures make M06, M07 and M10 decide a delivery — row 7 14 of 14

Browse source 
Row 7 left three mutants standing after v1.1 (11 of 14, bar 13): the passage
signal reading no body, the field signal weighing no title and no path, and a
flattened fusion (bm25.RRF_K = 10 000). Each switches off a mechanism the
default ranking runs, and each moved 0 ranks, because every synthetic concept
was short and opened with its own title -- none of the three ever decided a
delivery here.

Three bundles, one per mechanism, and one pinned set (set-mechanisms.json):

- PASSAGE: a long concept answered in one window of its body, against ten
  short concepts whose titles carry the question's words. Rank 1; with no
  body windows it falls out of k (class b).
- PATH: a concept named by its path alone, against ten decoys denser in the
  body. Rank 4 at k = 6; with no title/path weight it falls out of k.
- FUSION: gold 1st in the passage signal and 21st in the field signal, one
  decoy 10th and 11th (rank sum 19 < 20). At RRF_K = 60 the gold leads, and
  it keeps the lead through K = 180 (measured in steps of 10); flattened, the
  decoy wins k = 1.

Chose separate fixtures over one combined one because each fixture's comment
can then name the single mechanism it pins. Verified per question against all
14 mutants: no previously felled mutant is lost. src/ is untouched, MUTANT_BAR
and the roster are unchanged, the corpus pin moved with the corpus.

Gate: rows 1 and 6 go 10/10 -> 13/13, row 7 11/14 -> 14/14 GREEN, rows 2/3/4
unchanged, GATE RED: rows 5, 8. Suite: 2444 passed, 1 skipped (+3), measured
with FORCE_COLOR unset -- with FORCE_COLOR=3 in the environment Python 3.14
colours argparse help and test_each_arm_flag_carries_its_attribution fails,
independently of this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-22 22:41:01 +02:00
commit e503f6abd7
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
5 changed files with 234 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

21
CLAUDE.md
View file

@ -1318,15 +1318,24 @@ and fixtures, never code.
what a bundle looks like, this asks whether the payload for a question
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
run against a synthetic corpus this repo GENERATES and six sets it ships
run against a synthetic corpus this repo GENERATES and seven sets it ships
pinned by sha256 -- no network, no private corpus, no clock; a set whose
bytes moved is exit 2, never a quiet continuation, because **a gold set is
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
2026-09-21, v1.1 K): **1 (10/10) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
6 (10/10) - 7 (11/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
`GATE RED: rows 5, 7, 8`.
2026-09-22): **1 (13/13) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
6 (13/13) - 7 (14/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
`GATE RED: rows 5, 8`. **Row 7 fells 14 of 14 since 2026-09-22**: the three
that survived v1.1 (M06 passage body, M07 title/path weight, M10
`bm25.RRF_K`) each got a fixture that makes its mechanism decide a delivery
(`PASSAGE`, `PATH`, `FUSION`, set `set-mechanisms.json`; the fusion one keeps
its lead through K = 180 and loses it at 190), with `src/` untouched. **Row
8's judge predates v1.1's passage delivery**: it requires a delivered text to
be the whole concept's bytes, so an excerpt `as_passage` cut is class (e)
even when it carries the citation -- the dominant miss class when row 8 is
run on the real set. Not changed here (a closing round on rows 7 and 8
measured it and left it for its own order).
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
tied to the operator's employer is not re-measured and not frozen, and
public tests and gates run on invented material. Row 9 (K2) and the `r761`
@ -1335,8 +1344,8 @@ and fixtures, never code.
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
row 7's mutants patch `bm25` where the default runs there; three survive
(passage body, field title weight, `bm25.RRF_K`) with 0 ranks moved. What
row 7's mutants patch `bm25` where the default runs there; the three that
survived with 0 ranks moved are felled since 2026-09-22 (above). What
the text below says about rows 8/9, K2 and the three sets is history.
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could

37
README.md
View file

@ -84,13 +84,13 @@ here was measured; none of it is a plan.
`--no-assets` takes the image path out entirely.
- **Three of this repository's own gates are RED, and each red row is a stated
finding rather than a bug to be surprised by.** The retrieval gate is red on
rows 5, 7 and 8, the MCP gate on row 2, and the content accounting's judge
rows 5 and 8, the MCP gate on row 2, and the content accounting's judge
on rows 2, 3 and 6. For a user that means: no ranking change has yet been
registered against a held-out set (row 5), the real-corpus row is NOT RUN
without its private sets and so is red rather than green by omission
(row 8), three mechanical mutants of the BM25 ranking survive with 0 ranks
moved — the passage body, the title field weight and the fusion constant —
so the gate cannot yet tell those parts from their absence (row 7), MCP anchors and concept ids are different
without its private set and so is red rather than green by omission
(row 8) -- and when it IS run, its judge still requires a delivered excerpt
to be the whole concept's bytes, so an excerpt v1.1 cuts to its answering
passage counts as a miss even where it carries the citation. MCP anchors and concept ids are different
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
and the accounting still reports real losses on the reference corpus (rows 2,
3, 6). The rows and their numbers are under [Judge the
@ -1196,18 +1196,18 @@ set names a consumer's documents and this repository is public. **The corpus
is pinned the same way** (`SPECS_SHA256`): every row counts against those
documents, so moving them without moving the pin is exit 2.
**It is RED today, on rows 5, 7 and 8**, and each of those is a
**It is RED today, on rows 5 and 8**, and each of those is a
finding rather than a defect in the gate:
| row | what it asks | today |
|---|---|---|
| 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
| 1 | hit@payload, one fasit entry = one unit | 13 of 13 |
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 14 |
| 6 | every delivery confirmed against the bundle's own bytes | 13 of 13 |
| 7 | mechanical mutants of the ranking and the cut, felled | 14 of 14 |
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
**The public rows run on invented material only** (operator decision
@ -1230,13 +1230,18 @@ delivered excerpt answers — facts, no verdict, and the one bar is the gate's
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
and it reports three survivors with what they moved rather than with a shrug:
the passage signal reading no body, the field signal weighing no title and no
path, and a flattened fusion (`bm25.RRF_K`) each moved **0 ranks and 0
deliveries** on these fixtures. Each has a mechanism: the field signal still
reads every body, every synthetic body opens with its own title as a heading,
and over two signals `K` only reorders a concept whose two ranks lie far apart,
which no fixture holds.
and it fells all 14. Until 2026-09-22 three survived with **0 ranks and 0
deliveries** moved -- the passage signal reading no body, the field signal
weighing no title and no path, and a flattened fusion (`bm25.RRF_K`) -- because
every synthetic concept was short and opened with its own title, so none of the
three ever decided a delivery. Three fixtures now make each one decide one
(`set-mechanisms.json`): a long concept answered in one window of its body
against short concepts whose titles carry the question's words; a concept named
by its path alone against decoys denser in the body; and a concept first in one
signal and 21st in the other against a decoy 10th and 11th, which a flattened
fusion ranks first because only the rank sum then counts. Each fixture is
delivered at the shipped defaults and falls out of `k` with its own mechanism
switched off; `src/` did not change.
Row 8 is never green by leaving something out: it requires every named set and
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.

28
tests/fixtures/retrieval/set-mechanisms.json vendored Normal file
View file

@ -0,0 +1,28 @@
{
"set_id": "synthetic-mechanisms",
"bundle": "passage",
"note": "Three questions, one per mechanism the default BM25 ranking runs and no earlier fixture made decide a delivery: the passage signal's body windows (bundle `passage`), the field signal's title and path weight (bundle `path`), and the rank-fusion constant (bundle `fusion`). Each is delivered at the shipped defaults and falls out of k when its own mechanism is switched off, so row 1 is what fells M06, M07 and M10.",
"questions": [
{
"id": "PA1",
"question": "Hvem kontrollerer varmekablene i taket?",
"bundle": "passage",
"fasit": [{"by": "concept", "value": "aarbok/kapittel-tre", "quote": "kontrolleres av vaktmesteren"}]
},
{
"id": "PT1",
"question": "Hva gjelder for vester ved kanopadling?",
"bundle": "path",
"k": 6,
"fasit": [{"by": "concept", "value": "kanopadling/regel", "quote": "brukes paa vannet"}]
},
{
"id": "FU1",
"question": "Hva gjelder for isbading?",
"bundle": "fusion",
"k": 1,
"fasit": [{"by": "concept", "value": "badebok/kapittel", "quote": "isbading om morgenen"}]
}
],
"controls": []
}

15
tests/test_retrieval_gate.py
View file

@ -632,14 +632,15 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
by_number = {row.number: row for row in rows}
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8]
# Rows 2 and 3 are green again since the synthetic corpus was re-measured
# for BM25 (2026-09-21). Row 7 is red on three survivors, each printed with
# what it moved.
assert [row.number for row in rows if row.fails] == [5, 7, 8]
assert (by_number[1].k, by_number[1].m) == (10, 10)
# for BM25 (2026-09-21). Row 7 fells all 14 since 2026-09-22, when the three
# v1.1 survivors each got a fixture that makes their mechanism decide a
# delivery (`set-mechanisms.json`, three more row-1 units).
assert [row.number for row in rows if row.fails] == [5, 8]
assert (by_number[1].k, by_number[1].m) == (13, 13)
assert (by_number[2].k, by_number[2].m) == (7, 7)
assert (by_number[3].k, by_number[3].m) == (5, 5)
assert (by_number[6].k, by_number[6].m) == (10, 10)
assert (by_number[7].k, by_number[7].m) == (11, 14)
assert (by_number[6].k, by_number[6].m) == (13, 13)
assert (by_number[7].k, by_number[7].m) == (14, 14)
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
@ -655,7 +656,7 @@ def test_the_command_exits_one_and_prints_every_row(
printed = capsys.readouterr().out
for number in range(1, 9):
assert f"\n{number} " in f"\n{printed}"
assert "GATE RED: rows 5, 7, 8" in printed
assert "GATE RED: rows 5, 8" in printed
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:

181
tools/okf_retrieval_gate.py
View file

@ -434,6 +434,158 @@ BUDGET = BundleSpec(
),
)
# --- three fixtures, one per mechanism the default ranking runs ---------------
#
# Until 2026-09-22 row 7 left three mutants standing (M06, M07, M10): every
# synthetic concept was short and opened with its own title, so the passage
# signal, the title/path weight and the fusion constant never DECIDED a
# delivery here, and switching any of them off moved 0 ranks. Each bundle below
# is built so that exactly one of them does. None of them is a tuning of the
# ranker: the corpus is pinned (`SPECS_SHA256`) and `src/` is untouched.
_NOTE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi. "
#: THE PASSAGE SIGNAL. One long concept answers in ONE window of its body; ten
#: short concepts carry the question's words in their TITLES and none in their
#: bodies. The field signal reads ten title-weighted matches in short documents
#: above one long body, so only the passage signal -- the best WINDOW, not the
#: whole concept -- puts the gold first. Remove the body windows (M06) and the
#: passage signal reads titles alone, where the gold has none of the words.
PASSAGE = BundleSpec(
"retrieval-passage",
(
DocumentSpec(
"aarbok",
"aarbok.md",
(
ConceptSpec(
"kapittel-tre",
"Kapittel tre i aarboka",
_NOTE * 15
+ "Varmekablene i taket kontrolleres av vaktmesteren hver oktober. "
+ _NOTE * 15,
),
),
),
*(
DocumentSpec(
f"rundskriv-{number:02d}",
f"rundskriv-{number:02d}.md",
(
ConceptSpec(
f"skriv-{number:02d}",
f"Varmekablene i taket, skriv {number:02d}",
"Skrivet er sendt ut og arkivert.",
),
),
)
for number in range(1, 11)
),
),
)
#: THE TITLE/PATH WEIGHT. The gold is named by its PATH alone -- the question's
#: `kanopadling` is its directory and its source file and occurs in no body --
#: while ten decoys carry the question's other word more densely than the gold
#: does, one fewer time each. The path weight puts the gold first in the field
#: signal, which lifts it into k (rank 4 at `k` = 6). Weigh no title and no path
#: (M07) and both signals read bodies only, where the decoys outrank it and it
#: falls out of k.
PATH = BundleSpec(
"retrieval-path",
(
DocumentSpec(
"kanopadling",
"kanopadling.md",
(ConceptSpec("regel", "Regel om vester", "Vester brukes paa vannet."),),
),
*(
DocumentSpec(
f"turnotat-{number:02d}",
f"turnotat-{number:02d}.md",
(
ConceptSpec(
f"notat-{number:02d}",
f"Turnotat {number:02d}",
" ".join(["Vester henger i boden."] * (11 - number))
+ " Notatet er arkivert.",
),
),
)
for number in range(1, 11)
),
),
)
_FUSION_NOTE = "Styret gjennomgaar saken og sekretaeren arkiverer den. "
#: THE FUSION CONSTANT. Reciprocal-rank fusion trades a concept with one very
#: good rank against one with two middling ranks, and `RRF_K` sets the price.
#: The gold is FIRST in the passage signal and 21st in the field signal (a long
#: body, below ten short `isbading-*` path matches, nine longer ones and the
#: decoy); the one decoy is 10th and 11th, a smaller rank SUM (19 against 20).
#: A flattened fusion (M10, `RRF_K` = 10 000, where only the sum counts) puts
#: the decoy first and `k` = 1 no longer reaches the gold. At the shipped
#: `RRF_K` = 60 the gold leads, and measured over K in steps of 10 it keeps the
#: lead through K = 180 and loses it at 190. The eight long log pages hold the
#: decoy's passage rank at 10th and rank below the gold in both signals.
FUSION = BundleSpec(
"retrieval-fusion",
(
DocumentSpec(
"badebok",
"badebok.md",
(
ConceptSpec(
"kapittel",
"Kapittel i badeboka",
_FUSION_NOTE
* 12
+ "Isbading ved brygga, isbading om morgenen, isbading med vakt, "
"isbading hver dag. " + _FUSION_NOTE * 12,
),
),
),
*(
DocumentSpec(
f"logg-{number:02d}",
f"logg-{number:02d}.md",
(
ConceptSpec(
"side",
f"Loggside {number:02d}",
_FUSION_NOTE * 16
+ "Isbading ved brygga, isbading om kvelden, isbading med vakt. "
+ _FUSION_NOTE * 16,
),
),
)
for number in range(8)
),
DocumentSpec(
"isbading-lapp-00",
"isbading-lapp-00.md",
(ConceptSpec("lapp", "Lapp 00", _FUSION_NOTE * 5 + "Isbading er tillatt."),),
),
*(
DocumentSpec(
f"isbading-{number:02d}",
f"isbading-{number:02d}.md",
(ConceptSpec("oppslag", f"Oppslag {number:02d}", "Oppslaget er arkivert."),),
)
for number in range(10)
),
*(
DocumentSpec(
f"isbading-arkiv-{number:02d}",
f"isbading-arkiv-{number:02d}.md",
(ConceptSpec("mappe", f"Mappe {number:02d}", _FUSION_NOTE * 10),),
)
for number in range(9)
),
),
)
SPECS: Mapping[str, BundleSpec] = {
"positive": POSITIVE,
"miss": MISS,
@ -441,6 +593,9 @@ SPECS: Mapping[str, BundleSpec] = {
"budget": BUDGET,
"lookup": LOOKUP,
"quota": QUOTA,
"passage": PASSAGE,
"path": PATH,
"fusion": FUSION,
}
@ -483,7 +638,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
#: The synthetic corpus, pinned the way the sets are.
SPECS_SHA256 = "23c447add9a58ee2e454ef5d5b92d5c3c97d441cd883d203aaf2f275dd2f68f9"
SPECS_SHA256 = "abf24378c32b02753916c36eb502f9b27183a5d1c7e1914b40773bde393878b5"
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
@ -648,6 +803,7 @@ SYNTHETIC_SETS: dict[str, str] = {
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
"set-mechanisms.json": "435ce620c30c94b0d151335883583520490482be2ade6797524ae928a9f1eab7",
}
@ -1599,8 +1755,10 @@ class Mutant:
expected_row: int
patch: Callable[[], contextlib.AbstractContextManager[None]]
#: Why this one may be invisible to a gate that judges DELIVERY. Written
#: for the two that survive, so a survivor is a statement about the
#: mechanism and not a shrug.
#: for a mutant that survives, so a survivor is a statement about the
#: mechanism and not a shrug. None does since 2026-09-22: the three that
#: carried a note (M06, M07, M10) each got a fixture that makes their
#: mechanism decide a delivery (`PASSAGE`, `PATH`, `FUSION`).
note: str = ""
@ -1717,23 +1875,12 @@ MUTANTS: tuple[Mutant, ...] = (
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
Mutant(
"M06 the passage signal reads no body",
1,
lambda: _patched_bm25(windows=lambda body: []),
note=(
"the field signal still reads every body, and no synthetic concept is "
"long enough for its best window to decide a delivery"
),
"M06 the passage signal reads no body", 1, lambda: _patched_bm25(windows=lambda body: [])
),
Mutant(
"M07 the field signal weighs no title and no path",
1,
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
note=(
"every synthetic body opens with its own title as a heading "
"(`build_bundle`), so the weight only re-weights words the body "
"already carries"
),
),
Mutant("M08 every token matches every token", 4, _every_term_everything),
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
@ -1741,10 +1888,6 @@ MUTANTS: tuple[Mutant, ...] = (
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
1,
lambda: _patched_bm25(RRF_K=10_000),
note=(
"over two signals K reorders only a concept whose two ranks lie far "
"apart against one whose ranks are close, and no fixture here holds that"
),
),
Mutant("M11 the cut takes the LAST k", 1, _last_k),
Mutant("M12 the delivered text is truncated to 40 characters", 6, _truncate_delivered),