open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(gates): retire the K2 track; re-measure the retrieval gate's premises for BM25

Browse source 
Operator decision 2026-09-21: the test track built on material tied to the
operator's employer (K2) is retired -- not re-measured, not frozen. Public
tests and gates run on invented material.

Retrieval gate:
- The four FUSION_PREMISE xfails are gone and pass through their INPUTS: the
  synthetic MISS, LOOKUP and QUOTA bundles were re-measured for BM25 (the
  miss fasit no longer shares the rare word `maa`; lookup and quota decoys
  carry the question's words so each partition and the quota decide their
  own fixture). SPECS_SHA256 moved with them. Rows 2 and 3 green again.
- Row 7's mutants M04, M06, M07, M08, M10 now patch `bm25`, the code the
  default runs. Three survive with 0 ranks moved (passage body, title
  weight, bm25.RRF_K), each with its mechanism printed. M07 was not forced:
  every synthetic body carries its title as a heading.
- Row 9 (K2) removed; row 8 requires `wiki-20` alone, the `r761` and
  `vegnormal` adapters are gone. Chose the broad reading of "K2" because the
  operator decision defines it as employer-tied material and the order's
  grep includes `vegnormal`.

Also removed: tests/test_default_bundle_pin.py, the K2 arms of
test_okf_consume, the four real-arm tests of test_quality, the R761 soft
hyphen test, the N101/N200 delivery tests and okf_accounting_gate's default
real corpus (and H5's guard, which only existed for those defaults). Two
fixtures carrying road-standard identifiers are rewritten with invented ones.

Gate after: 1 10/10, 2 7/7, 3 5/5, 4 6/6, 5 0/1, 6 10/10, 7 11/14,
8 NOT RUN -> GATE RED: rows 5, 7, 8. Suite 2423 passed, 1 skipped,
0 xfailed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 10:05:43 +02:00
commit 3d149f955a
12 changed files with 288 additions and 1096 deletions
Download patch file Download diff file Expand all files Collapse all files

34
tools/okf_accounting_gate.py
View file

@ -102,8 +102,9 @@ STS_TWIN = FIXTURES / "witness" / "prosess-84-sts.twin.json"
PDF_FIXTURE = CORPUS / "prosess-84-tabell.pdf"
README = REPO / "README.md"
R761_DEFAULT = Path.home() / "repos" / "vegnormal-okf" / "data" / "raw" / "860019"
N200_DEFAULT = R761_DEFAULT.parent / "N200-2024-860015.json"
#: No DEFAULT real corpus since 2026-09-21: the two this row was built on are
#: material tied to the operator's employer, and that test track is retired.
#: Row 6 measures a real corpus only when one is named on the command line.
R761_ZIP = "14ce59dc-2150-480b-b661-6ea605fe3b24.zip"
R761_JSON = "R761-2025-860019.json"
R761_PDF = "R761-prosesskoden-2025.pdf"
@ -1408,14 +1409,14 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument(
"--r761",
type=Path,
default=R761_DEFAULT,
help="directory holding the R761 zip, JSON and PDF (read only)",
default=None,
help="directory holding the R761 zip, JSON and PDF (read only; no default)",
)
parser.add_argument(
"--n200",
type=Path,
default=N200_DEFAULT,
help="the N200 JSON delivery, the second real corpus (read only)",
default=None,
help="the N200 JSON delivery, the second real corpus (read only; no default)",
)
parser.add_argument(
"--consume", action="store_true", help="also run row 7 (diagnostic, never fails)"
@ -1447,23 +1448,10 @@ def main(argv: list[str] | None = None) -> int:
print(json.dumps(payload, indent=2, ensure_ascii=False))
else:
print(render(rows), end="")
if any(r.fails for r in rows):
return 1
# A row skipped while its source is on this machine did not run, and a
# zero here would report that as a pass. Measured against the DEFAULT
# sources, never against the arguments: a row is SKIPPED exactly when the
# corpora the arguments name are absent, so asking the arguments made this
# branch unreachable (H5). Pointing `--r761` at nothing on a machine that
# holds R761 is the case it exists for.
machine = real_corpora(R761_DEFAULT, N200_DEFAULT)
for row in rows:
if row.status == SKIPPED and any(c.available for c in machine):
print(
f"okf-accounting-gate: row {row.number} was skipped while its source exists",
file=sys.stderr,
)
return 1
return 0
# H5's guard -- a row skipped while a DEFAULT source sat on this machine
# exits 1 -- went with the defaults on 2026-09-21: with no default source
# the gate knows of no corpus it could have been pointed away from.
return 1 if any(r.fails for r in rows) else 0
if __name__ == "__main__":

357
tools/okf_retrieval_gate.py
View file

@ -10,27 +10,31 @@ WRITTEN RED, before any capability. Nothing in this module changes the
ranking, the fusion, the tokenisation or the cut; it only measures them. The
capability order is PM's to place after the rows here have been read.
WHY A ROW CAN BE RED WITHOUT A DEFECT IN THIS FILE. Rows 3, 4, 5, 8 and 9 are
red on the shipped code as it stands: the withheld label names the quota where
the truth is the rank (13 of 25 misses, measured 2026-09-17), the payload
carries no reading a consumer can act on when the bundle does not cover the
question (1 of 5 controls), no hold-out set has been registered, the real sets
live outside this repository, and the K2 gold set does not exist anywhere.
WHY A ROW CAN BE RED WITHOUT A DEFECT IN THIS FILE. Rows 5 and 8 are red on
the shipped code as it stands: no hold-out set has been registered, and the
real set lives outside this repository. Row 7 is red where a mutant survives,
and each survivor is printed with what it moved.
THE PUBLIC ROWS RUN ON INVENTED MATERIAL ONLY (operator decision 2026-09-21).
The test track built on material tied to the operator's employer -- the K2
corpus and the two road-standard sets -- is retired: not re-measured, not
frozen. Its row (9) and its sets' adapters are gone, and row 8 reads one
local set.
THE FASIT IS AN INPUT, NEVER A CONSTANT HERE -- `tools/okf_consume_measure.py`
states the rule and this module inherits it. This repository is PUBLIC: a gold
set names documents in a consumer's corpus, so a real set arrives as a path
plus an expected sha256 and is never committed. What IS committed is the
synthetic corpus below and the four synthetic sets beside it, whose subject
synthetic corpus below and the synthetic sets beside it, whose subject
matter is invented for this gate and names no real document.
GRANULARITY, STATED BECAUSE TWO FORMS ARE IN CIRCULATION. A unit here is one
FASIT ENTRY: a (concept, citation) pair. A question carrying three fasit
entries is three units, and `k of N` over units is never summed with `k of N`
over questions -- both are reported, per set, and the difference is printed.
The rule is the strictest of the three real sets' own (the wiki set's
`hit_rule`, verbatim in its file: an excerpt whose source is the fasit's
document AND whose text carries the fasit's quote).
The rule is the real set's own `hit_rule`, verbatim in its file: an excerpt
whose source is the fasit's document AND whose text carries the fasit's
quote.
THE JUDGE OPENS THE BUNDLE (row 6). Every hit is confirmed against the
concept file on disk: the fasit's citation must be IN the bundle before a miss
@ -58,7 +62,7 @@ REPO = TOOLS.parent
if str(REPO / "src") not in sys.path:
sys.path.insert(0, str(REPO / "src"))
from llm_ingestion_okf import consume # noqa: E402
from llm_ingestion_okf import bm25, consume # noqa: E402
# The two title forms a fasit can be met in, imported rather than written a
# second time: `okf quality --fasit` decides a boundary with exactly these,
@ -93,10 +97,6 @@ CLASSES: tuple[tuple[str, str], ...] = (
#: this gate must fell.
MUTANT_BAR = 0.90
#: K2's denominator, from an internal measurement note. The bundles exist on
#: the machine this row was written against; the gold set is nowhere, by design.
K2_QUESTIONS = 6
class GateUsage(Exception):
"""Wrong input: exit 2, never a quiet row."""
@ -274,7 +274,7 @@ MISS = BundleSpec(
ConceptSpec(
"flertallskrav",
"Naar saken er avgjort",
"To tredjedeler av de fremmoette medlemmer maa si ja.",
"To tredjedeler av de fremmoette medlemmer sier ja.",
),
*tuple(
ConceptSpec(
@ -355,8 +355,9 @@ LOOKUP = BundleSpec(
slug=f"notat-{number:02d}",
title=f"Notat {number:02d} om ettersyn",
body=(
"Kontrollen av hytta foeres i skjema. Hvert punkt i "
"kontrollen kvitteres av den som gaar runden."
"Kontrollen av hytta foeres etter punkt 4.2 og etter "
"noekkelrutine i skjema. Hvert punkt i kontrollen av "
"hytta kvitteres av den som gaar runden."
),
)
for number in range(1, 13)
@ -379,7 +380,10 @@ QUOTA = BundleSpec(
ConceptSpec(
slug=f"oversikt-{number:02d}",
title=f"Oversikt {number:02d} over dugnaden",
body=("Oversikten viser dugnaden og kontrollen av broennen gjennom sesongen."),
body=(
"Oversikten viser naar broennen proevetas, hvem som arkiverer "
"analysen, og dugnaden og kontrollen av broennen gjennom sesongen."
),
)
for number in range(1, 11)
),
@ -479,7 +483,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
#: The synthetic corpus, pinned the way the sets are.
SPECS_SHA256 = "8d999838f72a4c151e12ff6ac511b253c3437dba6290d2a7ea97dc546747242d"
SPECS_SHA256 = "23c447add9a58ee2e454ef5d5b92d5c3c97d441cd883d203aaf2f275dd2f68f9"
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
@ -497,12 +501,10 @@ def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> di
# --- the sets -----------------------------------------------------------------
#: How a fasit entry names the concept that answers it. Four, because the
#: three real sets name three different things and a gate that could read only
#: one of them would report two of the three as zero: the wiki set names a
#: source document plus a quote, `vegnormal` names a requirement number,
#: `R761-sk2` names an STS section title. The synthetic sets here name the
#: concept directly, which is the strictest form and the only one with no
#: How a fasit entry names the concept that answers it. Four, because real
#: sets name different things and a gate that could read only one form would
#: report the others as zero: a source document plus a quote, a requirement
#: number, a section title. The synthetic sets here name the concept directly, which is the strictest form and the only one with no
#: resolution step between the set and the bundle.
MATCHERS = ("concept", "source_file", "req_number", "title")
@ -534,8 +536,8 @@ class Question:
expect_class: str | None = None
k: int = consume.DEFAULT_K
limit: int = consume.DEFAULT_LIMIT
#: Set-level bundle override: one set can span several bundles (the
#: `vegnormal` set names a road standard per question).
#: Set-level bundle override: one set can span several bundles
#: (`set-classes.json` forces one class per bundle).
bundle: str = ""
@ -971,9 +973,8 @@ def _row(number: int, name: str, k: int, m: int, reason: str, details: list[str]
class Case:
"""One set measured once, and read by several rows.
A set may span bundles: `set-classes.json` forces one class per bundle,
and the real `vegnormal` set names a road standard per question. The
controls belong to the set's own bundle.
A set may span bundles: `set-classes.json` forces one class per bundle.
The controls belong to the set's own bundle.
"""
question_set: QuestionSet
@ -1180,14 +1181,15 @@ def row_three(cases: Sequence[Case]) -> Row:
#: Row 4's bar, and the ONE threshold this gate applies to a payload.
#:
#: SWEPT over 81 questions on 2026-09-20 -- the 16 of the synthetic sets and
#: the 65 of the three real sets -- against `coverage.unanswered_in_bundle` as
#: the 65 of the three real sets row 8 read then, two of which are retired
#: since 2026-09-21 -- against `coverage.unanswered_in_bundle` as
#: a share of the question's own terms:
#:
#: - at **0.50** row 4 is 6 of 6 and ELEVEN real questions whose fasit is in
#: their bundle come back marked;
#: - at **2/3** row 4 is 6 of 6 and **0 of 65** real questions are marked; the
#: highest a real covered question reaches is 0.6087 (two of the vegnormal
#: set's, 14 of 23 terms), and the lowest an uncovered control reaches is
#: highest a real covered question reaches is 0.6087 (two questions of one
#: retired set, 14 of 23 terms), and the lowest an uncovered control reaches is
#: 0.6667 (N3, 4 of 6);
#: - at **0.70** N3 falls under the bar and row 4 is 5 of 6.
#:
@ -1195,10 +1197,10 @@ def row_three(cases: Sequence[Case]) -> Row:
#: distribution rather than a number picked to pass. The margin is THIN --
#: 0.6087 against 0.6667 -- and it is published that way.
#:
#: WHAT IT DOES NOT CATCH, measured on the same data: `r761-sk2`'s own
#: WHAT IT DOES NOT CATCH, measured on the same data: one retired set's own
#: known-negative, a question naming a section number the document does not
#: have, sits at 0.2857 -- every other word of it is in a 2 756-concept road
#: standard. One term absent among seven is not a share, and no bar over this
#: have, sits at 0.2857 -- every other word of it is in a 2 756-concept
#: bundle. One term absent among seven is not a share, and no bar over this
#: list reaches it. Row 4's denominator is the synthetic controls, so this
#: costs the row nothing and is stated rather than implied.
UNANSWERED_BAR = 2 / 3
@ -1614,6 +1616,21 @@ def _patched(**attributes: object) -> Iterator[None]:
setattr(consume, name, value)
@contextlib.contextmanager
def _patched_bm25(**attributes: object) -> Iterator[None]:
"""`_patched` for the module the DEFAULT ranking scores in. Since v1.1 C1
`consume` hands the ordering to `bm25.rank`, so a mutant of the fusion's
functions changes code the default no longer runs and can fell nothing."""
original = {name: getattr(bm25, name) for name in attributes}
try:
for name, value in attributes.items():
setattr(bm25, name, value)
yield
finally:
for name, value in original.items():
setattr(bm25, name, value)
def _wrap_cut(**overrides: object) -> contextlib.AbstractContextManager[None]:
original = consume.cut
@ -1624,12 +1641,20 @@ def _wrap_cut(**overrides: object) -> contextlib.AbstractContextManager[None]:
def _reverse_scores() -> contextlib.AbstractContextManager[None]:
original = consume.concept_scores
original = bm25.rank
def mutant(*args: Any, **kwargs: Any) -> Any:
return tuple(reversed(original(*args, **kwargs)))
result = original(*args, **kwargs)
return replace(result, ranked=list(reversed(result.ranked)))
return _patched(concept_scores=mutant)
return _patched_bm25(rank=mutant)
def _every_term_everything() -> contextlib.AbstractContextManager[None]:
def mutant(query: Sequence[str], vocabulary: frozenset[str]) -> list[frozenset[str]]:
return [frozenset(vocabulary) for _ in dict.fromkeys(query)]
return _patched_bm25(query_groups=mutant)
def _last_k() -> contextlib.AbstractContextManager[None]:
@ -1691,30 +1716,34 @@ MUTANTS: tuple[Mutant, ...] = (
Mutant("M03 k = 1", 1, lambda: _wrap_cut(k=1)),
Mutant("M04 the ranking is reversed", 1, _reverse_scores),
Mutant("M05 the source quota is removed", 3, lambda: _wrap_cut(source_quota=None)),
Mutant("M06 the body signal is dead", 1, lambda: _patched(_overlap=lambda *a, **k: 0)),
Mutant(
"M07 the document prior is dead",
"M06 the passage signal reads no body",
1,
lambda: _patched(document_scores=lambda *a, **k: {}),
lambda: _patched_bm25(windows=lambda body: []),
note=(
"a question that NAMES its document reaches it through the title-and-id "
"signal as well, since the prior reads the same id path; a question that "
"does not can be moved by at most 1/(RRF_K+1)"
"the field signal still reads every body, and no synthetic concept is "
"long enough for its best window to decide a delivery"
),
),
Mutant(
"M08 every token matches every token",
4,
lambda: _patched(tokens_match=lambda *a, **k: True),
"M07 the field signal weighs no title and no path",
1,
lambda: _patched_bm25(TITLE_WEIGHT=0, PATH_WEIGHT=0),
note=(
"every synthetic body opens with its own title as a heading "
"(`build_bundle`), so the weight only re-weights words the body "
"already carries"
),
),
Mutant("M08 every token matches every token", 4, _every_term_everything),
Mutant("M09 no stem, no prefix: equality only", 1, _equality_only),
Mutant(
"M10 the fusion is flattened (RRF_K = 10 000)",
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
1,
lambda: _patched(RRF_K=10_000),
lambda: _patched_bm25(RRF_K=10_000),
note=(
"1/(K+r) is strictly decreasing in r for every K, so a larger K "
"compresses the scores without reordering them on its own"
"over two signals K reorders only a concept whose two ranks lie far "
"apart against one whose ranks are close, and no fixture here holds that"
),
),
Mutant("M11 the cut takes the LAST k", 1, _last_k),
@ -1740,11 +1769,11 @@ MUTANT_ROSTER: tuple[str, ...] = (
"M03 k = 1",
"M04 the ranking is reversed",
"M05 the source quota is removed",
"M06 the body signal is dead",
"M07 the document prior is dead",
"M06 the passage signal reads no body",
"M07 the field signal weighs no title and no path",
"M08 every token matches every token",
"M09 no stem, no prefix: equality only",
"M10 the fusion is flattened (RRF_K = 10 000)",
"M10 the fusion is flattened (bm25.RRF_K = 10 000)",
"M11 the cut takes the LAST k",
"M12 the delivered text is truncated to 40 characters",
"M13 the excerpt carries no text",
@ -1876,7 +1905,7 @@ def row_seven(
)
# --- rows 8 and 9: the sets that are not in this repository --------------------
# --- row 8: the sets that are not in this repository ------------------------
#: What PM measured 2026-09-17 with okf 0.10.0 at the shipped defaults. Carried
#: so row 8 is not blank when it has not run -- and labelled on every line,
@ -1889,18 +1918,16 @@ def row_seven(
#: integrity check is not a disclosure.
RECORDED = {
"wiki-20": "measured by its owner; figure not restated here",
"r761-sk2": "7 of 7 positives, all at rank 1 (8 entries incl. KP and KN)",
"vegnormal-32": "32 of 43 citations = 21 of 32 questions",
"total": "not summed: one set's figure is not restated here",
}
def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
"""One of the three real sets, in ITS OWN shape, read never written.
"""The real set, in ITS OWN shape, read never written.
Each set names a fasit differently and each names it in its own file; the
adapters below are the whole of this gate's knowledge of them, and none of
the question text ever reaches a tracked file here.
The adapter below is the whole of this gate's knowledge of it, and none of
the question text ever reaches a tracked file here. Two more adapters, for
sets built on material tied to the operator's employer, were removed
2026-09-21 with the test track they belonged to.
"""
raw = path.read_bytes()
measured = hashlib.sha256(raw).hexdigest()
@ -1925,60 +1952,16 @@ def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
for entry in spec["questions"]
)
return QuestionSet("wiki-20", "wiki", path, measured, questions, ())
if name == "r761":
r761_questions: list[Question] = []
controls: list[Control] = []
for entry in spec["sporsmal"]:
if str(entry["id"]).startswith("KN"):
controls.append(
Control(
str(entry["id"]), str(entry["sporsmal"]), "the set's own known-negative"
)
)
continue
r761_questions.append(
Question(
id=str(entry["id"]),
question=str(entry["sporsmal"]),
# The fasit is a section TITLE, and the set carries no
# quote: this set is measured at concept granularity.
fasit=(Fasit(by="title", value=str(entry["fasit"])),),
)
)
return QuestionSet(
"r761-sk2", "r761", path, measured, tuple(r761_questions), tuple(controls)
)
if name == "vegnormal":
vegnormal_questions: list[Question] = []
for entry in spec["sporsmal"]:
by_normal: dict[str, list[Fasit]] = {}
for item in entry["must_cite"]:
by_normal.setdefault(str(item["normal"]), []).append(
Fasit(by="req_number", value=str(item["req_number"]))
)
# One question citing two standards is two Questions, one per
# bundle, because a payload is built against one bundle. The unit
# count is unchanged, which is what the denominator counts.
for normal, fasit in sorted(by_normal.items()):
suffix = f"/{normal}" if len(by_normal) > 1 else ""
vegnormal_questions.append(
Question(
id=f"{entry['id']}{suffix}",
question=str(entry["sporsmal"]),
fasit=tuple(fasit),
bundle=normal,
)
)
return QuestionSet("vegnormal-32", "", path, measured, tuple(vegnormal_questions), ())
raise GateUsage(f"unknown real set `{name}`; one of wiki, r761, vegnormal")
raise GateUsage(f"unknown real set `{name}`; the one real set is `wiki`")
#: The three sets row 8 is the measurement of. All three, by name: a run that
#: hands over one of them has measured one of them, and the row says so. Left
#: to `len(real)` the row came back `6 of 6 GREEN` on a single set (PM's J2,
#: 2026-09-19) -- the realistic route being the one set that is at 7 of 7,
#: with the two that miss omitted.
REQUIRED_REAL_SETS: tuple[str, ...] = ("wiki-20", "r761-sk2", "vegnormal-32")
#: The sets row 8 is the measurement of, by name: a run that hands over some
#: of them has measured some of them, and the row says so. Left to
#: `len(real)` the row came back `6 of 6 GREEN` on one set of three (PM's J2,
#: 2026-09-19). One set since 2026-09-21, when the two built on material tied
#: to the operator's employer were retired; the rule stays for the day a
#: second set joins.
REQUIRED_REAL_SETS: tuple[str, ...] = ("wiki-20",)
@dataclass(frozen=True)
@ -1989,9 +1972,8 @@ class RealSetPin:
caller, and `set_id` was decided by the adapter rather than by the file:
three one-question files written in the three shapes, against a
self-written bundle, read `wiki-20: 1 of 1 ... | 3 of 3 | GREEN`. Nothing
said how big `wiki-20` is. Row 9 has had the mechanism since 2026-09-19
(`K2_QUESTIONS` refuses a set of another size); this is that mechanism for
the three sets row 8 requires.
said how big `wiki-20` is. This is the mechanism that refuses a set of
another size, for every set row 8 requires.
THREE COUNTS, NOT ONE. The sha256 is the strongest and the least
informative: it says the bytes are the pinned bytes and nothing about what
@ -2012,10 +1994,8 @@ class RealSetPin:
sha256: str
#: Measured 2026-09-20 against the three sources, each read through its own
#: adapter. `questions` is the number of `Question` objects the adapter
#: produces, which is why `vegnormal-32` is 37: five of its 32 questions cite
#: two standards, and a payload is built against one bundle.
#: Measured 2026-09-20 against the source, read through its own adapter.
#: `questions` is the number of `Question` objects the adapter produces.
REAL_SET_PINS: Mapping[str, RealSetPin] = {
"wiki-20": RealSetPin(
questions=20,
@ -2023,18 +2003,6 @@ REAL_SET_PINS: Mapping[str, RealSetPin] = {
controls=0,
sha256="972d0f5715d1377b3d89b8ddf391612709b96cd0fe8b96dfe517fe1931a9e333",
),
"r761-sk2": RealSetPin(
questions=7,
fasit_entries=7,
controls=1,
sha256="c834a478e4888300845de9e166808a3942085cb73c6e9e5fd2a3e1a6e9c5e6fd",
),
"vegnormal-32": RealSetPin(
questions=37,
fasit_entries=43,
controls=0,
sha256="c3932fc9abd144989bdbc50c4e4627ac5cc59937c4204f92422b7fe10af87faa",
),
}
@ -2087,19 +2055,19 @@ def bundle_identity(bundle: Path) -> str:
def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
"""The three real sets. RED when they have not run -- always, in this
order -- and never green by leaving a set out.
"""The real sets. RED when they have not run -- always, in this order --
and never green by leaving a set out.
THE HEADLINE IS AT QUESTION GRANULARITY, and that is not a style choice:
the three sets do not share a unit. `wiki-20` names a citation, `r761-sk2`
and `vegnormal-32` name a concept and a requirement number, and adding a
citation hit to a concept hit produces a number that is neither. A
question is the one thing all three sets have, so the row counts questions
-- answered meaning at least one of the question's fasit entries arrived,
the same reading row 1 prints beside its own units -- and the two unit
totals are printed below it, each with its own denominator, never summed.
sets need not share a unit. One names a citation, another may name a
concept, and adding a citation hit to a concept hit produces a number that
is neither. A question is the one thing every set has, so the row counts
questions -- answered meaning at least one of the question's fasit entries
arrived, the same reading row 1 prints beside its own units -- and the two
unit totals are printed below it, each with its own denominator, never
summed.
"""
name = "the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256"
name = "the real sets (" + ", ".join(REQUIRED_REAL_SETS) + "), run from path + sha256"
reason_tail = (
"a question counts as answered when at least one of its fasit entries "
"arrived; the two unit granularities are printed apart and never summed"
@ -2170,7 +2138,7 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
if missing:
details.append(
" the numbers above are what DID run; the row is not a measurement "
"of the three sets until all three are given"
"of the required sets until every one is given"
)
return Row(
8,
@ -2192,68 +2160,6 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
)
def row_nine(k2: tuple[QuestionSet, Mapping[str, Path]] | None = None) -> Row:
"""K2: the bundles are on this machine and the gold set is nowhere.
IT TAKES AN INPUT, so it is a measurement and not a placeholder. Until
2026-09-19 this row was a hard-coded RED that could not have gone green on
the day somebody wrote the set; it now reads one through `--k2`, in this
gate's own set shape, and `K2_QUESTIONS` is the denominator whatever the
file carries -- a set of five would be a different set with this one's
name.
WITHOUT A SET IT STAYS RED rather than NOT RUN, and that is this row's own
published rule ("a set that cannot be measured is a red number, never an
absent row"): the denominator is KNOWN -- six questions, recorded -- so
the absence is measured. Row 8 says NOT RUN because ITS denominator is not
known until the sets arrive. Both fail the gate identically.
"""
if k2 is None:
return Row(
9,
"K2, the sixth set",
0,
K2_QUESTIONS,
RED,
f"not measured: 0 of {K2_QUESTIONS} questions have a gold set anywhere",
[
" the bundles exist on the machine this row was written "
"against; the answer key does not, anywhere",
" a set that cannot be measured is a red number, never an absent row",
" who can write it: whoever holds the K2 corpus -- it names documents "
"that may not be committed here, so it arrives as a path plus a sha256",
" the shape to write it in is this gate's own set shape, the one "
"`tests/fixtures/retrieval/set-*.json` is written in",
],
)
question_set, bundles = k2
units = [
unit
for question in question_set.questions
for unit in measure_units(bundles[question.bundle or question_set.bundle], question)
]
answered = len({unit.question_id for unit in units if unit.hit})
details = [
f" {question_set.set_id}: {answered} of {K2_QUESTIONS} questions | "
f"{sum(1 for unit in units if unit.hit)} of {len(units)} fasit entries "
f"({'citation' if question_set.quoted else 'concept'} granularity) | "
f"sha256 {question_set.sha256[:12]}"
]
details += [
f" miss {unit.question_id} {unit.named}: class {unit.klass or '-'} ({unit.detail})"
for unit in units
if not unit.hit
]
return _row(
9,
"K2, the sixth set",
answered,
K2_QUESTIONS,
f"the recorded denominator is {K2_QUESTIONS} questions, whatever the file carries",
details,
)
# --- the run ------------------------------------------------------------------
#: What this gate cannot check, whatever the rows say. Printed on every run,
@ -2315,7 +2221,6 @@ def evaluate(
real: Sequence[tuple[QuestionSet, Mapping[str, Path]]] = (),
mutants: Sequence[Mutant] = MUTANTS,
sets: Mapping[str, str] = SYNTHETIC_SETS,
k2: tuple[QuestionSet, Mapping[str, Path]] | None = None,
) -> list[Row]:
cases, _ = synthetic_cases(root, fixtures, sets)
rows = deterministic_rows(cases)
@ -2324,7 +2229,6 @@ def evaluate(
row_five(registration),
row_seven(cases, rows, mutants=mutants),
row_eight(real),
row_nine(k2),
]
@ -2374,25 +2278,6 @@ def _real_sets(
return real
def _k2_set(
argument: Sequence[str] | None,
) -> tuple[QuestionSet, Mapping[str, Path]] | None:
if not argument:
return None
path, sha, bundle = argument
question_set = load_set(Path(path).expanduser(), sha)
if len(question_set.questions) != K2_QUESTIONS:
raise GateUsage(
f"{path}: K2's denominator is {K2_QUESTIONS} questions and this set "
f"carries {len(question_set.questions)}; a set of another size is "
"another set wearing this one's name"
)
bundles = _bundle_map(bundle)
if list(bundles) == [""]:
bundles = {question_set.bundle: bundles[""]}
return question_set, bundles
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
parser.add_argument("--json", action="store_true", help="emit the rows as JSON")
@ -2403,19 +2288,10 @@ def main(argv: list[str] | None = None) -> int:
metavar=("NAME", "SET", "SHA256", "BUNDLE"),
default=[],
help=(
"run row 8 against one real set: NAME is wiki, r761 or vegnormal; "
"run row 8 against one real set: NAME is wiki; "
"BUNDLE is a path, or `key=path,key=path` for a set spanning bundles"
),
)
parser.add_argument(
"--k2",
nargs=3,
metavar=("SET", "SHA256", "BUNDLE"),
help=(
"run row 9 against a K2 gold set, written in this gate's own set "
"shape; the denominator stays the recorded six questions"
),
)
parser.add_argument(
"--holdout",
type=Path,
@ -2425,9 +2301,8 @@ def main(argv: list[str] | None = None) -> int:
args = parser.parse_args(argv)
try:
real = _real_sets(args.real)
k2 = _k2_set(args.k2)
with tempfile.TemporaryDirectory(prefix="okf-retrieval-gate-") as scratch:
rows = evaluate(Path(scratch), registration=args.holdout, real=real, k2=k2)
rows = evaluate(Path(scratch), registration=args.holdout, real=real)
except GateUsage as error:
print(f"okf-retrieval-gate: {error}", file=sys.stderr)
return 2