open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(gates): retire the K2 track; re-measure the retrieval gate's premises for BM25

Browse source 
Operator decision 2026-09-21: the test track built on material tied to the
operator's employer (K2) is retired -- not re-measured, not frozen. Public
tests and gates run on invented material.

Retrieval gate:
- The four FUSION_PREMISE xfails are gone and pass through their INPUTS: the
  synthetic MISS, LOOKUP and QUOTA bundles were re-measured for BM25 (the
  miss fasit no longer shares the rare word `maa`; lookup and quota decoys
  carry the question's words so each partition and the quota decide their
  own fixture). SPECS_SHA256 moved with them. Rows 2 and 3 green again.
- Row 7's mutants M04, M06, M07, M08, M10 now patch `bm25`, the code the
  default runs. Three survive with 0 ranks moved (passage body, title
  weight, bm25.RRF_K), each with its mechanism printed. M07 was not forced:
  every synthetic body carries its title as a heading.
- Row 9 (K2) removed; row 8 requires `wiki-20` alone, the `r761` and
  `vegnormal` adapters are gone. Chose the broad reading of "K2" because the
  operator decision defines it as employer-tied material and the order's
  grep includes `vegnormal`.

Also removed: tests/test_default_bundle_pin.py, the K2 arms of
test_okf_consume, the four real-arm tests of test_quality, the R761 soft
hyphen test, the N101/N200 delivery tests and okf_accounting_gate's default
real corpus (and H5's guard, which only existed for those defaults). Two
fixtures carrying road-standard identifiers are rewritten with invented ones.

Gate after: 1 10/10, 2 7/7, 3 5/5, 4 6/6, 5 0/1, 6 10/10, 7 11/14,
8 NOT RUN -> GATE RED: rows 5, 7, 8. Suite 2423 passed, 1 skipped,
0 xfailed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 10:05:43 +02:00
commit 3d149f955a
12 changed files with 288 additions and 1096 deletions
Download patch file Download diff file Expand all files Collapse all files

94
tests/test_accounting_gate.py
View file

@ -378,10 +378,10 @@ def test_a_prefixed_tag_that_names_no_role_stays_uncounted() -> None:
assert witness._sts_role_json(witness._local(tag), "sec", "body") is None
#: THE ONE PLACE THIS NUMBER LIVES. The count itself is measured over the
#: delivery by `test_n101s_own_prefixed_tags_are_counted_here_and_name_no_role`
#: -- but that test is `skipif`-gated on a file only this machine has, so on a
#: fresh clone the four published sentences were unguarded again, which is how
#: THE ONE PLACE THIS NUMBER LIVES. The count itself was measured over the
#: delivery by a `skipif`-gated test that read a file only this machine has
#: (removed 2026-09-21 with the retired test track), so on a fresh clone the
#: four published sentences were unguarded again, which is how
#: 574 survived in four docstrings until PM counted it. The guard below needs
#: no delivery: it reads the published sentences and holds them to each other
#: and to this constant.
@ -408,14 +408,14 @@ _TBX_PUBLISHERS = (
def test_the_published_tbx_count_is_one_number_and_needs_no_delivery() -> None:
"""The published strings are held to each other, on any machine.
This is the half the measurement could not cover. `_tags_of` counts the
real delivery and is right to; it also cannot run where the delivery is
absent, and an assertion that skips guards nothing. Editing `CLAUDE.md` to
This is the half the measurement could not cover. The delivery count
cannot run where the delivery is absent, and an assertion that skips
guards nothing. Editing `CLAUDE.md` to
600 tomorrow is red here, on a fresh clone, with no corpus.
It proves nothing about the WORLD -- five files agreeing is agreement, not
a count -- which is why the delivery test keeps its own measurement and
this one only holds the sentences to the constant it asserts.
a count -- and this test only holds the sentences to the constant it
asserts.
"""
root = Path(__file__).resolve().parents[1]
for name in _TBX_PUBLISHERS:
@ -440,58 +440,6 @@ def test_the_published_tbx_count_is_one_number_and_needs_no_delivery() -> None:
assert _PUBLISHED_TBX.findall(wrong) == ["574"]
N101_DELIVERY = gate.N200_DEFAULT.parent / "N101-2025-860031.json"
def _tags_of(payload: bytes) -> dict[str, int]:
"""Every `tag` string in a delivery, counted by a walk written HERE.
The witness's own reader is what the known-negative below judges, so
counting through it would make the two agree by construction.
"""
names: dict[str, int] = {}
def walk(node: Any) -> None:
if isinstance(node, dict):
tag = node.get("tag")
if isinstance(tag, str):
names[tag] = names.get(tag, 0) + 1
for value in node.values():
walk(value)
elif isinstance(node, list):
for value in node:
walk(value)
walk(json.loads(payload.decode("utf-8")))
return names
@pytest.mark.skipif(not N101_DELIVERY.is_file(), reason="N101 is not on this machine")
def test_n101s_own_prefixed_tags_are_counted_here_and_name_no_role() -> None:
"""The published number for that known-negative was a measurement nothing
could falsify: it lived in four docstrings and in no assertion, and it was
wrong. The count is made HERE, over the delivery itself, so the sentence
four files publish is red when it stops being true."""
names = _tags_of(N101_DELIVERY.read_bytes())
assert sum(names.values()) > 0, "the walk found no tag at all"
tbx = {tag: n for tag, n in names.items() if tag.startswith("tbx:")}
assert sum(tbx.values()) == N101_TBX_TAGS
for tag in sorted(tbx):
assert witness._sts_role_json(witness._local(tag), "sec", "body") is None, tag
@pytest.mark.skipif(not gate.N200_DEFAULT.is_file(), reason="N200 is not on this machine")
def test_the_json_role_map_counts_n200s_own_formulas_and_figure_captions() -> None:
"""The same defect on the delivery it was found in, with PM's numbers.
Skipped where the corpus is absent, and then this file's own fixture is
the only thing holding the rule -- which is why both exist."""
counts = witness.count_sts_json(gate.N200_DEFAULT.read_bytes()).counts
assert counts["math"] == 74
assert counts["figure_caption"] == 49
assert counts["citation"] == 194
assert counts["figure"] == 49
def test_the_two_sts_role_maps_are_written_twice_and_not_shared() -> None:
"""M-2: both STS witnesses went through ONE `_sts_role`, so row 5 could
never see a hole in it. Two maps, each written for its own delivery."""
@ -1547,35 +1495,11 @@ def test_a_unit_clean_in_only_one_of_the_two_builds_is_not_clean() -> None:
assert gate.clean_in_every_run([[clean], [dirty]]) == 0
def test_a_row_skipped_while_the_default_source_exists_exits_one(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""H5: the guard asked whether the corpora named by the ARGUMENTS are
available -- and row 6 is SKIPPED precisely when none of them is, so the
branch could never fire and no test covered it. The question it meant to
ask is about the machine: a corpus that is HERE and was pointed away from
is a row that did not run, and `CI=1` then printed a qualified GREEN and
exited 0.
Measured against its own known-negative below, so a guard that fires on
everything would not pass either."""
present = tmp_path / "corpus.json"
present.write_text("{}", encoding="utf-8")
skipped = gate.Row(6, "real corpora", 0, 0, gate.SKIPPED, "not measured, source missing: x")
monkeypatch.setattr(gate, "N200_DEFAULT", present)
monkeypatch.setattr(gate, "evaluate", lambda **kwargs: [skipped])
code = gate.main(["--r761", str(tmp_path / "absent"), "--n200", str(tmp_path / "absent.json")])
assert code == 1
assert "row 6 was skipped while its source exists" in capsys.readouterr().err
def test_a_row_skipped_with_no_source_on_the_machine_exits_zero(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The known-negative: nothing to measure is not a row that did not run."""
skipped = gate.Row(6, "real corpora", 0, 0, gate.SKIPPED, "not measured, source missing: x")
monkeypatch.setattr(gate, "R761_DEFAULT", tmp_path / "absent")
monkeypatch.setattr(gate, "N200_DEFAULT", tmp_path / "absent.json")
monkeypatch.setattr(gate, "evaluate", lambda **kwargs: [skipped])
assert gate.main([]) == 0