test(gates): retire the K2 track; re-measure the retrieval gate's premises for BM25
Operator decision 2026-09-21: the test track built on material tied to the operator's employer (K2) is retired -- not re-measured, not frozen. Public tests and gates run on invented material. Retrieval gate: - The four FUSION_PREMISE xfails are gone and pass through their INPUTS: the synthetic MISS, LOOKUP and QUOTA bundles were re-measured for BM25 (the miss fasit no longer shares the rare word `maa`; lookup and quota decoys carry the question's words so each partition and the quota decide their own fixture). SPECS_SHA256 moved with them. Rows 2 and 3 green again. - Row 7's mutants M04, M06, M07, M08, M10 now patch `bm25`, the code the default runs. Three survive with 0 ranks moved (passage body, title weight, bm25.RRF_K), each with its mechanism printed. M07 was not forced: every synthetic body carries its title as a heading. - Row 9 (K2) removed; row 8 requires `wiki-20` alone, the `r761` and `vegnormal` adapters are gone. Chose the broad reading of "K2" because the operator decision defines it as employer-tied material and the order's grep includes `vegnormal`. Also removed: tests/test_default_bundle_pin.py, the K2 arms of test_okf_consume, the four real-arm tests of test_quality, the R761 soft hyphen test, the N101/N200 delivery tests and okf_accounting_gate's default real corpus (and H5's guard, which only existed for those defaults). Two fixtures carrying road-standard identifiers are rewritten with invented ones. Gate after: 1 10/10, 2 7/7, 3 5/5, 4 6/6, 5 0/1, 6 10/10, 7 11/14, 8 NOT RUN -> GATE RED: rows 5, 7, 8. Suite 2423 passed, 1 skipped, 0 xfailed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
977040f575
commit
3d149f955a
12 changed files with 288 additions and 1096 deletions
|
|
@ -102,8 +102,9 @@ STS_TWIN = FIXTURES / "witness" / "prosess-84-sts.twin.json"
|
|||
PDF_FIXTURE = CORPUS / "prosess-84-tabell.pdf"
|
||||
README = REPO / "README.md"
|
||||
|
||||
R761_DEFAULT = Path.home() / "repos" / "vegnormal-okf" / "data" / "raw" / "860019"
|
||||
N200_DEFAULT = R761_DEFAULT.parent / "N200-2024-860015.json"
|
||||
#: No DEFAULT real corpus since 2026-09-21: the two this row was built on are
|
||||
#: material tied to the operator's employer, and that test track is retired.
|
||||
#: Row 6 measures a real corpus only when one is named on the command line.
|
||||
R761_ZIP = "14ce59dc-2150-480b-b661-6ea605fe3b24.zip"
|
||||
R761_JSON = "R761-2025-860019.json"
|
||||
R761_PDF = "R761-prosesskoden-2025.pdf"
|
||||
|
|
@ -1408,14 +1409,14 @@ def main(argv: list[str] | None = None) -> int:
|
|||
parser.add_argument(
|
||||
"--r761",
|
||||
type=Path,
|
||||
default=R761_DEFAULT,
|
||||
help="directory holding the R761 zip, JSON and PDF (read only)",
|
||||
default=None,
|
||||
help="directory holding the R761 zip, JSON and PDF (read only; no default)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--n200",
|
||||
type=Path,
|
||||
default=N200_DEFAULT,
|
||||
help="the N200 JSON delivery, the second real corpus (read only)",
|
||||
default=None,
|
||||
help="the N200 JSON delivery, the second real corpus (read only; no default)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--consume", action="store_true", help="also run row 7 (diagnostic, never fails)"
|
||||
|
|
@ -1447,23 +1448,10 @@ def main(argv: list[str] | None = None) -> int:
|
|||
print(json.dumps(payload, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(render(rows), end="")
|
||||
if any(r.fails for r in rows):
|
||||
return 1
|
||||
# A row skipped while its source is on this machine did not run, and a
|
||||
# zero here would report that as a pass. Measured against the DEFAULT
|
||||
# sources, never against the arguments: a row is SKIPPED exactly when the
|
||||
# corpora the arguments name are absent, so asking the arguments made this
|
||||
# branch unreachable (H5). Pointing `--r761` at nothing on a machine that
|
||||
# holds R761 is the case it exists for.
|
||||
machine = real_corpora(R761_DEFAULT, N200_DEFAULT)
|
||||
for row in rows:
|
||||
if row.status == SKIPPED and any(c.available for c in machine):
|
||||
print(
|
||||
f"okf-accounting-gate: row {row.number} was skipped while its source exists",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
return 0
|
||||
# H5's guard -- a row skipped while a DEFAULT source sat on this machine
|
||||
# exits 1 -- went with the defaults on 2026-09-21: with no default source
|
||||
# the gate knows of no corpus it could have been pointed away from.
|
||||
return 1 if any(r.fails for r in rows) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue