open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(retrieval-gate): the headline test could not see the sum it forbids

Browse source 
X2 of this round's own mutation run SURVIVED: put the headline back to
`quoted + concept` over `quoted_units + concept_units` and all 64 tests stayed
green. The reason is the fixture, not the row -- three sets of one fasit entry
each make "questions answered" and "units hit" the same number, so the test
agreed with the mutant by construction. The same shape as round 158's X5, and
the same lesson as `konstruert-rad-kan-vaere-fasitradens-speil`.

One set now carries TWO fasit entries, which is the only shape where the two
readings differ: the headline must be 3 of 3 questions where the sum would be
4 of 4. `_hitting_set` takes `entries` and says why in its docstring.

64 passed. Mutation run in a scratch copy of HEAD, twelve mutants plus a
control that must survive: killed 11 of 12 before this commit, and X2 is the
twelfth.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 22:01:57 +02:00
commit 149836c722
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
1 changed files with 26 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

34
tests/test_retrieval_gate.py
View file

@ -800,9 +800,23 @@ def test_every_class_is_documented_in_the_output() -> None:
# table is 8 of 8 rather than 4 of 4.
def _hitting_set(set_id: str, *, quote: str) -> gate.QuestionSet:
def _hitting_set(set_id: str, *, quote: str, entries: int = 1) -> gate.QuestionSet:
"""A set the positive bundle answers, named by concept: the shortest way to
drive row 8 green without an adapter between the set and the row."""
drive row 8 green without an adapter between the set and the row.
`entries` is load-bearing for the headline test: one question carrying TWO
fasit entries is the only shape in which "questions answered" and "units
hit" are different numbers, and a test where they agree cannot see a
headline that sums the units.
"""
fasit = [
gate.Fasit(by="concept", value="haandbok/vinterberedskap", quote=quote),
gate.Fasit(
by="concept",
value="haandbok/noekkelkvittering",
quote="kvitteres ut mot signatur" if quote else "",
),
][:entries]
return gate.QuestionSet(
set_id=set_id,
bundle="positive",
@ -811,8 +825,11 @@ def _hitting_set(set_id: str, *, quote: str) -> gate.QuestionSet:
questions=(
gate.Question(
id=f"{set_id}-1",
question="Naar kontrolleres vinterberedskapen paa hytta?",
fasit=(gate.Fasit(by="concept", value="haandbok/vinterberedskap", quote=quote),),
question=(
"Naar kontrolleres vinterberedskapen paa hytta, og hvordan "
"kvitteres noekkelen ut?"
),
fasit=tuple(fasit),
),
),
controls=(),
@ -849,17 +866,18 @@ def test_j2b_row_eight_never_sums_the_two_granularities_into_its_headline(
saying the two are not summed."""
bundles = {"positive": _bundles(tmp_path)["positive"]}
real = [
(_hitting_set("wiki-20", quote="innen 1. november"), bundles),
(_hitting_set("wiki-20", quote="innen 1. november", entries=2), bundles),
(_hitting_set("r761-sk2", quote=""), bundles),
(_hitting_set("vegnormal-32", quote=""), bundles),
]
row = gate.row_eight(real)
# Three questions, one per set: the headline is at QUESTION granularity and
# is never the sum of one citation unit and two concept units.
# Three questions, one per set: the headline is at QUESTION granularity.
# The sum it must NOT be is 4 of 4 -- two citation units plus two concept
# units -- which is why the first set carries two fasit entries.
assert (row.k, row.m) == (3, 3)
assert "question" in row.reason
assert any(
"1 of 1 at citation granularity, 2 of 2 at concept granularity" in detail
"2 of 2 at citation granularity, 2 of 2 at concept granularity" in detail
for detail in row.details
)