open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(gates): retire the K2 track; re-measure the retrieval gate's premises for BM25

Browse source 
Operator decision 2026-09-21: the test track built on material tied to the
operator's employer (K2) is retired -- not re-measured, not frozen. Public
tests and gates run on invented material.

Retrieval gate:
- The four FUSION_PREMISE xfails are gone and pass through their INPUTS: the
  synthetic MISS, LOOKUP and QUOTA bundles were re-measured for BM25 (the
  miss fasit no longer shares the rare word `maa`; lookup and quota decoys
  carry the question's words so each partition and the quota decide their
  own fixture). SPECS_SHA256 moved with them. Rows 2 and 3 green again.
- Row 7's mutants M04, M06, M07, M08, M10 now patch `bm25`, the code the
  default runs. Three survive with 0 ranks moved (passage body, title
  weight, bm25.RRF_K), each with its mechanism printed. M07 was not forced:
  every synthetic body carries its title as a heading.
- Row 9 (K2) removed; row 8 requires `wiki-20` alone, the `r761` and
  `vegnormal` adapters are gone. Chose the broad reading of "K2" because the
  operator decision defines it as employer-tied material and the order's
  grep includes `vegnormal`.

Also removed: tests/test_default_bundle_pin.py, the K2 arms of
test_okf_consume, the four real-arm tests of test_quality, the R761 soft
hyphen test, the N101/N200 delivery tests and okf_accounting_gate's default
real corpus (and H5's guard, which only existed for those defaults). Two
fixtures carrying road-standard identifiers are rewritten with invented ones.

Gate after: 1 10/10, 2 7/7, 3 5/5, 4 6/6, 5 0/1, 6 10/10, 7 11/14,
8 NOT RUN -> GATE RED: rows 5, 7, 8. Suite 2423 passed, 1 skipped,
0 xfailed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 10:05:43 +02:00
commit 3d149f955a
12 changed files with 288 additions and 1096 deletions
Download patch file Download diff file Expand all files Collapse all files

52
tests/test_quality.py
View file

@ -19,8 +19,6 @@ from __future__ import annotations
import json
from pathlib import Path
import pytest
from llm_ingestion_okf import quality
_FRONTMATTER = """---
@ -248,11 +246,6 @@ def test_the_no_source_file_row_reports_no_document_count(tmp_path: Path) -> Non
# `<title>` over into the concept id. The second match form -- the
# (directory, residual title) pair -- is what takes it to 2 759 of 2 761.
_GOOD_ARM = Path.home() / "repos/vegnormal-okf/build/ferdig/r761-2025-generisk"
_BAD_ARM = Path.home() / "repos/vegnormal-okf/build/860019-mdb-100"
_OTHER_PRODUCT = Path.home() / "corpora/okf-telling-20260829/K2-bundle-default-20260912"
_FASIT = Path.home() / "repos/vegnormal-okf/build/sk2-fasit-2761.json"
def _fasit(path: Path, titles: list[str]) -> Path:
"""A fasit file in the shipped shape: a list of rows carrying title and norm."""
@ -449,48 +442,3 @@ def test_the_boundary_threshold_names_its_corpus_and_denominator() -> None:
assert bar.limit_declared >= quality.MIN_DECLARED_FOR_A_THRESHOLD
assert bar.corpora == 1
assert bar.source
@pytest.mark.skipif(not (_GOOD_ARM.is_dir() and _FASIT.is_file()), reason="local gold set absent")
def test_the_known_good_arm_reproduces_its_published_recall() -> None:
"""2 759 of 2 761, and the decomposition P1 asked for: 22 literal, 2 737 paired."""
report = quality.measure_bundle(_GOOD_ARM, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (2759, 2761)
assert (report.boundaries.literal, report.boundaries.paired) == (22, 2737)
assert report.boundaries.verdict == "PASS"
@pytest.mark.skipif(not (_BAD_ARM.is_dir() and _FASIT.is_file()), reason="local gold set absent")
def test_the_known_bad_arm_reproduces_its_published_recall() -> None:
"""1 148 of 2 761 -- the 41.6 % the whole gate could not previously see."""
report = quality.measure_bundle(_BAD_ARM, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert (report.boundaries.recovered, report.boundaries.declared) == (1148, 2761)
assert report.boundaries.verdict == "FAIL"
assert report.exit_code == 1
@pytest.mark.skipif(
not (_OTHER_PRODUCT.is_dir() and _FASIT.is_file()), reason="local gold set absent"
)
def test_a_bundle_of_another_product_scores_zero_and_that_is_the_assertion(tmp_path: Path) -> None:
"""The adverse case, measured rather than argued.
The fasit describes ONE product. The pinned 43-document K2 bundle is a
different corpus entirely and scores 0 of 2 761 -- a FAIL that says the
caller's assertion was wrong, not that the bundle is bad. `--fasit` is an
assertion by the caller, the way `okf consume --ref` is.
"""
report = quality.measure_bundle(_OTHER_PRODUCT, fasit=quality.load_fasit(_FASIT))
assert report.boundaries is not None
assert report.boundaries.recovered == 0
assert report.boundaries.verdict == "FAIL"
@pytest.mark.skipif(not _FASIT.is_file(), reason="local gold set absent")
def test_the_shipped_fasit_reproduces_its_own_key_under_this_normalisation() -> None:
"""2 761 of 2 761 rows: `norm` is exactly whitespace-stripped, lowercased title."""
rows = quality.load_fasit(_FASIT)
assert len(rows) == 2761
assert sum(1 for row in rows if quality.normalise_title(row.title) == row.norm) == 2761