test(retrieval-gate): three fixtures make M06, M07 and M10 decide a delivery — row 7 14 of 14
Row 7 left three mutants standing after v1.1 (11 of 14, bar 13): the passage signal reading no body, the field signal weighing no title and no path, and a flattened fusion (bm25.RRF_K = 10 000). Each switches off a mechanism the default ranking runs, and each moved 0 ranks, because every synthetic concept was short and opened with its own title -- none of the three ever decided a delivery here. Three bundles, one per mechanism, and one pinned set (set-mechanisms.json): - PASSAGE: a long concept answered in one window of its body, against ten short concepts whose titles carry the question's words. Rank 1; with no body windows it falls out of k (class b). - PATH: a concept named by its path alone, against ten decoys denser in the body. Rank 4 at k = 6; with no title/path weight it falls out of k. - FUSION: gold 1st in the passage signal and 21st in the field signal, one decoy 10th and 11th (rank sum 19 < 20). At RRF_K = 60 the gold leads, and it keeps the lead through K = 180 (measured in steps of 10); flattened, the decoy wins k = 1. Chose separate fixtures over one combined one because each fixture's comment can then name the single mechanism it pins. Verified per question against all 14 mutants: no previously felled mutant is lost. src/ is untouched, MUTANT_BAR and the roster are unchanged, the corpus pin moved with the corpus. Gate: rows 1 and 6 go 10/10 -> 13/13, row 7 11/14 -> 14/14 GREEN, rows 2/3/4 unchanged, GATE RED: rows 5, 8. Suite: 2444 passed, 1 skipped (+3), measured with FORCE_COLOR unset -- with FORCE_COLOR=3 in the environment Python 3.14 colours argparse help and test_each_arm_flag_carries_its_attribution fails, independently of this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
09d7e6ee08
commit
e503f6abd7
5 changed files with 234 additions and 48 deletions
21
CLAUDE.md
21
CLAUDE.md
|
|
@ -1318,15 +1318,24 @@ and fixtures, never code.
|
|||
what a bundle looks like, this asks whether the payload for a question
|
||||
carries the fasit. **Written RED, no capability**: eight rows, exit 0 only
|
||||
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
||||
run against a synthetic corpus this repo GENERATES and six sets it ships
|
||||
run against a synthetic corpus this repo GENERATES and seven sets it ships
|
||||
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
||||
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
||||
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
||||
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
|
||||
2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
|
||||
2026-09-21, v1.1 K): **1 (10/10) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
|
||||
6 (10/10) - 7 (11/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
|
||||
`GATE RED: rows 5, 7, 8`.
|
||||
2026-09-22): **1 (13/13) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
|
||||
6 (13/13) - 7 (14/14) - 8 (0 of 1 set, NOT RUN without `--real`)**,
|
||||
`GATE RED: rows 5, 8`. **Row 7 fells 14 of 14 since 2026-09-22**: the three
|
||||
that survived v1.1 (M06 passage body, M07 title/path weight, M10
|
||||
`bm25.RRF_K`) each got a fixture that makes its mechanism decide a delivery
|
||||
(`PASSAGE`, `PATH`, `FUSION`, set `set-mechanisms.json`; the fusion one keeps
|
||||
its lead through K = 180 and loses it at 190), with `src/` untouched. **Row
|
||||
8's judge predates v1.1's passage delivery**: it requires a delivered text to
|
||||
be the whole concept's bytes, so an excerpt `as_passage` cut is class (e)
|
||||
even when it carries the citation -- the dominant miss class when row 8 is
|
||||
run on the real set. Not changed here (a closing round on rows 7 and 8
|
||||
measured it and left it for its own order).
|
||||
**THE K2 TEST TRACK IS RETIRED (operator 2026-09-21, v1.1 K)**: material
|
||||
tied to the operator's employer is not re-measured and not frozen, and
|
||||
public tests and gates run on invented material. Row 9 (K2) and the `r761`
|
||||
|
|
@ -1335,8 +1344,8 @@ and fixtures, never code.
|
|||
the R761/N101/N200 deliveries; `okf_accounting_gate.py` has no default real
|
||||
corpus. The synthetic corpus was re-measured for BM25 (`MISS`, `LOOKUP`,
|
||||
`QUOTA`), so rows 2 and 3 are green through their own fixtures again, and
|
||||
row 7's mutants patch `bm25` where the default runs there; three survive
|
||||
(passage body, field title weight, `bm25.RRF_K`) with 0 ranks moved. What
|
||||
row 7's mutants patch `bm25` where the default runs there; the three that
|
||||
survived with 0 ranks moved are felled since 2026-09-22 (above). What
|
||||
the text below says about rows 8/9, K2 and the three sets is history.
|
||||
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
|
||||
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue