fix(retrieval-gate): row 8 is the three sets, and its headline is one granularity

Two of PM's eight attacks, closed where they were measured.

J2 -- a set left out. `REQUIRED_REAL_SETS` names the three (`wiki-20`,
`r761-sk2`, `vegnormal-32`) and the row is NOT RUN until all three are given,
whatever the ones that ran scored. The numbers the run DID measure are still
printed: a missing set must not cost the reader the set that was measured, and
"not run" is the row's status, never a reason to withhold a figure.

J2b -- the headline was `quoted_hits + concept_hits` over
`quoted_units + concept_units`, written one line above the detail that says
the two are not summed. The three sets do not share a unit: `wiki-20` names a
citation, `r761-sk2` a section title, `vegnormal-32` a requirement number, and
a citation hit plus a concept hit is a number that is neither. A QUESTION is
the one unit all three have, so the headline is questions answered of
questions asked -- answered meaning at least one of the question's fasit
entries arrived, the reading row 1 already prints beside its own units -- and
the two unit totals keep their own denominators below it, unsummed.

49 passed (was 46 passed, 3 failed). The gate is unchanged where it did not
run: still exit 1, still `0 of 3 NOT RUN` with PM's recorded figures carried.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 20:51:41 +02:00
commit b4f03e44cb
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
2 changed files with 53 additions and 14 deletions

View file

@ -760,11 +760,7 @@ def _hitting_set(set_id: str, *, quote: str) -> gate.QuestionSet:
gate.Question( gate.Question(
id=f"{set_id}-1", id=f"{set_id}-1",
question="Naar kontrolleres vinterberedskapen paa hytta?", question="Naar kontrolleres vinterberedskapen paa hytta?",
fasit=( fasit=(gate.Fasit(by="concept", value="haandbok/vinterberedskap", quote=quote),),
gate.Fasit(
by="concept", value="haandbok/vinterberedskap", quote=quote
),
),
), ),
), ),
controls=(), controls=(),

View file

@ -1481,25 +1481,49 @@ def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
raise GateUsage(f"unknown real set `{name}`; one of wiki, r761, vegnormal") raise GateUsage(f"unknown real set `{name}`; one of wiki, r761, vegnormal")
#: The three sets row 8 is the measurement of. All three, by name: a run that
#: hands over one of them has measured one of them, and the row says so. Left
#: to `len(real)` the row came back `6 of 6 GREEN` on a single set (PM's J2,
#: 2026-09-19) -- the realistic route being the one set that is at 7 of 7,
#: with the two that miss omitted.
REQUIRED_REAL_SETS: tuple[str, ...] = ("wiki-20", "r761-sk2", "vegnormal-32")
def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row: def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
"""The three real sets. RED when they have not run -- always, in this """The three real sets. RED when they have not run -- always, in this
order -- and never green by leaving a set out.""" order -- and never green by leaving a set out.
THE HEADLINE IS AT QUESTION GRANULARITY, and that is not a style choice:
the three sets do not share a unit. `wiki-20` names a citation, `r761-sk2`
and `vegnormal-32` name a concept and a requirement number, and adding a
citation hit to a concept hit produces a number that is neither. A
question is the one thing all three sets have, so the row counts questions
-- answered meaning at least one of the question's fasit entries arrived,
the same reading row 1 prints beside its own units -- and the two unit
totals are printed below it, each with its own denominator, never summed.
"""
name = "the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256"
reason_tail = (
"a question counts as answered when at least one of its fasit entries "
"arrived; the two unit granularities are printed apart and never summed"
)
if not real: if not real:
return Row( return Row(
8, 8,
"the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256", name,
0, 0,
3, len(REQUIRED_REAL_SETS),
NOT_RUN, NOT_RUN,
"not run: no --real argument. The sets live in other repositories and " "not run: no --real argument. The sets live in other repositories and "
"are never committed here", "are never committed here",
[ [
f" recorded 2026-09-17 by PM, NOT measured by this gate: {name}: {value}" f" recorded 2026-09-17 by PM, NOT measured by this gate: {key}: {value}"
for name, value in RECORDED.items() for key, value in RECORDED.items()
], ],
) )
details: list[str] = [] details: list[str] = []
quoted_hits = quoted_units = concept_hits = concept_units = 0 quoted_hits = quoted_units = concept_hits = concept_units = 0
answered_total = asked_total = 0
for question_set, bundles in real: for question_set, bundles in real:
cases = [] cases = []
for question in question_set.questions: for question in question_set.questions:
@ -1512,6 +1536,8 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
hits = sum(1 for unit in units if unit.hit) hits = sum(1 for unit in units if unit.hit)
answered = len({unit.question_id for unit in units if unit.hit}) answered = len({unit.question_id for unit in units if unit.hit})
asked = len({unit.question_id for unit in units}) asked = len({unit.question_id for unit in units})
answered_total += answered
asked_total += asked
if question_set.quoted: if question_set.quoted:
quoted_hits += hits quoted_hits += hits
quoted_units += len(units) quoted_units += len(units)
@ -1533,12 +1559,29 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
f" NOT SUMMED INTO ONE NUMBER: {quoted_hits} of {quoted_units} at citation " f" NOT SUMMED INTO ONE NUMBER: {quoted_hits} of {quoted_units} at citation "
f"granularity, {concept_hits} of {concept_units} at concept granularity" f"granularity, {concept_hits} of {concept_units} at concept granularity"
) )
measured = {question_set.set_id for question_set, _ in real}
missing = [required for required in REQUIRED_REAL_SETS if required not in measured]
if missing:
details.append(
" the numbers above are what DID run; the row is not a measurement "
"of the three sets until all three are given"
)
return Row(
8,
name,
answered_total,
asked_total,
NOT_RUN,
"not run: " + ", ".join(missing) + " was not given. " + reason_tail,
details,
)
return _row( return _row(
8, 8,
"the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256", name,
quoted_hits + concept_hits, answered_total,
quoted_units + concept_units, asked_total,
"local row: the sets are read from their own repositories, never committed here", "local row: the sets are read from their own repositories, never committed "
"here. " + reason_tail,
details, details,
) )