open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(retrieval-gate): red -- row 7's bar is 90 % of a list in the file it guards

Browse source 
PM's J3: seven duplicate `M03 k = 1` mutants appended to `MUTANTS` took row 7
from `11 of 13 RED` to `18 of 20 GREEN` with the SAME two survivors. The bar
is `MUTANT_BAR` of `len(mutants)`, and `MUTANTS` is a module constant in the
file a capability session edits -- a gate whose pass mark is a list the
candidate writes.

Reproduced here at 18 of 20 exactly. 1 red on an assertion about behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 21:11:11 +02:00
commit 68ea05c17d
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 15 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

15
tests/test_retrieval_gate.py
View file

@ -961,3 +961,18 @@ def test_j1_a_registration_this_session_wrote_is_not_a_hold_out(tmp_path: Path)
# file before the ranking moved.
assert any("committed" in detail for detail in row.details), row.details
assert any("ranking" in detail for detail in row.details), row.details
def test_j3_row_seven_refuses_a_mutant_list_that_is_not_the_pinned_roster(
tmp_path: Path,
) -> None:
"""PM's J3: seven duplicate `k = 1` mutants appended to `MUTANTS` took the
row from 11 of 13 RED to 18 of 20 GREEN with the SAME two survivors. The
bar was 90 % of a list living in the file a capability session edits."""
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
baseline = gate.deterministic_rows(cases)
padded = (*gate.MUTANTS, *(gate.MUTANTS[2] for _ in range(7)))
row = gate.row_seven(cases, baseline, mutants=padded)
assert row.status != gate.GREEN, row
assert row.fails
assert "roster" in row.reason or "duplicate" in row.reason