fix(retrieval-gate): row 8 is the three sets, and its headline is one granularity
Two of PM's eight attacks, closed where they were measured. J2 -- a set left out. `REQUIRED_REAL_SETS` names the three (`wiki-20`, `r761-sk2`, `vegnormal-32`) and the row is NOT RUN until all three are given, whatever the ones that ran scored. The numbers the run DID measure are still printed: a missing set must not cost the reader the set that was measured, and "not run" is the row's status, never a reason to withhold a figure. J2b -- the headline was `quoted_hits + concept_hits` over `quoted_units + concept_units`, written one line above the detail that says the two are not summed. The three sets do not share a unit: `wiki-20` names a citation, `r761-sk2` a section title, `vegnormal-32` a requirement number, and a citation hit plus a concept hit is a number that is neither. A QUESTION is the one unit all three have, so the headline is questions answered of questions asked -- answered meaning at least one of the question's fasit entries arrived, the reading row 1 already prints beside its own units -- and the two unit totals keep their own denominators below it, unsummed. 49 passed (was 46 passed, 3 failed). The gate is unchanged where it did not run: still exit 1, still `0 of 3 NOT RUN` with PM's recorded figures carried. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
3d496de2c6
commit
b4f03e44cb
2 changed files with 53 additions and 14 deletions
|
|
@ -760,11 +760,7 @@ def _hitting_set(set_id: str, *, quote: str) -> gate.QuestionSet:
|
||||||
gate.Question(
|
gate.Question(
|
||||||
id=f"{set_id}-1",
|
id=f"{set_id}-1",
|
||||||
question="Naar kontrolleres vinterberedskapen paa hytta?",
|
question="Naar kontrolleres vinterberedskapen paa hytta?",
|
||||||
fasit=(
|
fasit=(gate.Fasit(by="concept", value="haandbok/vinterberedskap", quote=quote),),
|
||||||
gate.Fasit(
|
|
||||||
by="concept", value="haandbok/vinterberedskap", quote=quote
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
controls=(),
|
controls=(),
|
||||||
|
|
|
||||||
|
|
@ -1481,25 +1481,49 @@ def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
|
||||||
raise GateUsage(f"unknown real set `{name}`; one of wiki, r761, vegnormal")
|
raise GateUsage(f"unknown real set `{name}`; one of wiki, r761, vegnormal")
|
||||||
|
|
||||||
|
|
||||||
|
#: The three sets row 8 is the measurement of. All three, by name: a run that
|
||||||
|
#: hands over one of them has measured one of them, and the row says so. Left
|
||||||
|
#: to `len(real)` the row came back `6 of 6 GREEN` on a single set (PM's J2,
|
||||||
|
#: 2026-09-19) -- the realistic route being the one set that is at 7 of 7,
|
||||||
|
#: with the two that miss omitted.
|
||||||
|
REQUIRED_REAL_SETS: tuple[str, ...] = ("wiki-20", "r761-sk2", "vegnormal-32")
|
||||||
|
|
||||||
|
|
||||||
def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
|
def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
|
||||||
"""The three real sets. RED when they have not run -- always, in this
|
"""The three real sets. RED when they have not run -- always, in this
|
||||||
order -- and never green by leaving a set out."""
|
order -- and never green by leaving a set out.
|
||||||
|
|
||||||
|
THE HEADLINE IS AT QUESTION GRANULARITY, and that is not a style choice:
|
||||||
|
the three sets do not share a unit. `wiki-20` names a citation, `r761-sk2`
|
||||||
|
and `vegnormal-32` name a concept and a requirement number, and adding a
|
||||||
|
citation hit to a concept hit produces a number that is neither. A
|
||||||
|
question is the one thing all three sets have, so the row counts questions
|
||||||
|
-- answered meaning at least one of the question's fasit entries arrived,
|
||||||
|
the same reading row 1 prints beside its own units -- and the two unit
|
||||||
|
totals are printed below it, each with its own denominator, never summed.
|
||||||
|
"""
|
||||||
|
name = "the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256"
|
||||||
|
reason_tail = (
|
||||||
|
"a question counts as answered when at least one of its fasit entries "
|
||||||
|
"arrived; the two unit granularities are printed apart and never summed"
|
||||||
|
)
|
||||||
if not real:
|
if not real:
|
||||||
return Row(
|
return Row(
|
||||||
8,
|
8,
|
||||||
"the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256",
|
name,
|
||||||
0,
|
0,
|
||||||
3,
|
len(REQUIRED_REAL_SETS),
|
||||||
NOT_RUN,
|
NOT_RUN,
|
||||||
"not run: no --real argument. The sets live in other repositories and "
|
"not run: no --real argument. The sets live in other repositories and "
|
||||||
"are never committed here",
|
"are never committed here",
|
||||||
[
|
[
|
||||||
f" recorded 2026-09-17 by PM, NOT measured by this gate: {name}: {value}"
|
f" recorded 2026-09-17 by PM, NOT measured by this gate: {key}: {value}"
|
||||||
for name, value in RECORDED.items()
|
for key, value in RECORDED.items()
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
details: list[str] = []
|
details: list[str] = []
|
||||||
quoted_hits = quoted_units = concept_hits = concept_units = 0
|
quoted_hits = quoted_units = concept_hits = concept_units = 0
|
||||||
|
answered_total = asked_total = 0
|
||||||
for question_set, bundles in real:
|
for question_set, bundles in real:
|
||||||
cases = []
|
cases = []
|
||||||
for question in question_set.questions:
|
for question in question_set.questions:
|
||||||
|
|
@ -1512,6 +1536,8 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
|
||||||
hits = sum(1 for unit in units if unit.hit)
|
hits = sum(1 for unit in units if unit.hit)
|
||||||
answered = len({unit.question_id for unit in units if unit.hit})
|
answered = len({unit.question_id for unit in units if unit.hit})
|
||||||
asked = len({unit.question_id for unit in units})
|
asked = len({unit.question_id for unit in units})
|
||||||
|
answered_total += answered
|
||||||
|
asked_total += asked
|
||||||
if question_set.quoted:
|
if question_set.quoted:
|
||||||
quoted_hits += hits
|
quoted_hits += hits
|
||||||
quoted_units += len(units)
|
quoted_units += len(units)
|
||||||
|
|
@ -1533,12 +1559,29 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
|
||||||
f" NOT SUMMED INTO ONE NUMBER: {quoted_hits} of {quoted_units} at citation "
|
f" NOT SUMMED INTO ONE NUMBER: {quoted_hits} of {quoted_units} at citation "
|
||||||
f"granularity, {concept_hits} of {concept_units} at concept granularity"
|
f"granularity, {concept_hits} of {concept_units} at concept granularity"
|
||||||
)
|
)
|
||||||
|
measured = {question_set.set_id for question_set, _ in real}
|
||||||
|
missing = [required for required in REQUIRED_REAL_SETS if required not in measured]
|
||||||
|
if missing:
|
||||||
|
details.append(
|
||||||
|
" the numbers above are what DID run; the row is not a measurement "
|
||||||
|
"of the three sets until all three are given"
|
||||||
|
)
|
||||||
|
return Row(
|
||||||
|
8,
|
||||||
|
name,
|
||||||
|
answered_total,
|
||||||
|
asked_total,
|
||||||
|
NOT_RUN,
|
||||||
|
"not run: " + ", ".join(missing) + " was not given. " + reason_tail,
|
||||||
|
details,
|
||||||
|
)
|
||||||
return _row(
|
return _row(
|
||||||
8,
|
8,
|
||||||
"the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256",
|
name,
|
||||||
quoted_hits + concept_hits,
|
answered_total,
|
||||||
quoted_units + concept_units,
|
asked_total,
|
||||||
"local row: the sets are read from their own repositories, never committed here",
|
"local row: the sets are read from their own repositories, never committed "
|
||||||
|
"here. " + reason_tail,
|
||||||
details,
|
details,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue