open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(retrieval-gate): row 8 is the three sets, and its headline is one granularity

Browse source 
Two of PM's eight attacks, closed where they were measured.

J2 -- a set left out. `REQUIRED_REAL_SETS` names the three (`wiki-20`,
`r761-sk2`, `vegnormal-32`) and the row is NOT RUN until all three are given,
whatever the ones that ran scored. The numbers the run DID measure are still
printed: a missing set must not cost the reader the set that was measured, and
"not run" is the row's status, never a reason to withhold a figure.

J2b -- the headline was `quoted_hits + concept_hits` over
`quoted_units + concept_units`, written one line above the detail that says
the two are not summed. The three sets do not share a unit: `wiki-20` names a
citation, `r761-sk2` a section title, `vegnormal-32` a requirement number, and
a citation hit plus a concept hit is a number that is neither. A QUESTION is
the one unit all three have, so the headline is questions answered of
questions asked -- answered meaning at least one of the question's fasit
entries arrived, the reading row 1 already prints beside its own units -- and
the two unit totals keep their own denominators below it, unsummed.

49 passed (was 46 passed, 3 failed). The gate is unchanged where it did not
run: still exit 1, still `0 of 3 NOT RUN` with PM's recorded figures carried.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 20:51:41 +02:00
commit b4f03e44cb
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
2 changed files with 53 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

6
tests/test_retrieval_gate.py
View file

@ -760,11 +760,7 @@ def _hitting_set(set_id: str, *, quote: str) -> gate.QuestionSet:
gate.Question(
id=f"{set_id}-1",
question="Naar kontrolleres vinterberedskapen paa hytta?",
fasit=(
gate.Fasit(
by="concept", value="haandbok/vinterberedskap", quote=quote
),
),
fasit=(gate.Fasit(by="concept", value="haandbok/vinterberedskap", quote=quote),),
),
),
controls=(),

61
tools/okf_retrieval_gate.py
View file

@ -1481,25 +1481,49 @@ def read_real_set(name: str, path: Path, expected_sha256: str) -> QuestionSet:
raise GateUsage(f"unknown real set `{name}`; one of wiki, r761, vegnormal")
#: The three sets row 8 is the measurement of. All three, by name: a run that
#: hands over one of them has measured one of them, and the row says so. Left
#: to `len(real)` the row came back `6 of 6 GREEN` on a single set (PM's J2,
#: 2026-09-19) -- the realistic route being the one set that is at 7 of 7,
#: with the two that miss omitted.
REQUIRED_REAL_SETS: tuple[str, ...] = ("wiki-20", "r761-sk2", "vegnormal-32")
def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
"""The three real sets. RED when they have not run -- always, in this
order -- and never green by leaving a set out."""
order -- and never green by leaving a set out.
THE HEADLINE IS AT QUESTION GRANULARITY, and that is not a style choice:
the three sets do not share a unit. `wiki-20` names a citation, `r761-sk2`
and `vegnormal-32` name a concept and a requirement number, and adding a
citation hit to a concept hit produces a number that is neither. A
question is the one thing all three sets have, so the row counts questions
-- answered meaning at least one of the question's fasit entries arrived,
the same reading row 1 prints beside its own units -- and the two unit
totals are printed below it, each with its own denominator, never summed.
"""
name = "the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256"
reason_tail = (
"a question counts as answered when at least one of its fasit entries "
"arrived; the two unit granularities are printed apart and never summed"
)
if not real:
return Row(
8,
"the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256",
name,
0,
3,
len(REQUIRED_REAL_SETS),
NOT_RUN,
"not run: no --real argument. The sets live in other repositories and "
"are never committed here",
[
f" recorded 2026-09-17 by PM, NOT measured by this gate: {name}: {value}"
for name, value in RECORDED.items()
f" recorded 2026-09-17 by PM, NOT measured by this gate: {key}: {value}"
for key, value in RECORDED.items()
],
)
details: list[str] = []
quoted_hits = quoted_units = concept_hits = concept_units = 0
answered_total = asked_total = 0
for question_set, bundles in real:
cases = []
for question in question_set.questions:
@ -1512,6 +1536,8 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
hits = sum(1 for unit in units if unit.hit)
answered = len({unit.question_id for unit in units if unit.hit})
asked = len({unit.question_id for unit in units})
answered_total += answered
asked_total += asked
if question_set.quoted:
quoted_hits += hits
quoted_units += len(units)
@ -1533,12 +1559,29 @@ def row_eight(real: Sequence[tuple[QuestionSet, Mapping[str, Path]]]) -> Row:
f" NOT SUMMED INTO ONE NUMBER: {quoted_hits} of {quoted_units} at citation "
f"granularity, {concept_hits} of {concept_units} at concept granularity"
)
measured = {question_set.set_id for question_set, _ in real}
missing = [required for required in REQUIRED_REAL_SETS if required not in measured]
if missing:
details.append(
" the numbers above are what DID run; the row is not a measurement "
"of the three sets until all three are given"
)
return Row(
8,
name,
answered_total,
asked_total,
NOT_RUN,
"not run: " + ", ".join(missing) + " was not given. " + reason_tail,
details,
)
return _row(
8,
"the real sets (wiki-20, r761-sk2, vegnormal-32), run from path + sha256",
quoted_hits + concept_hits,
quoted_units + concept_units,
"local row: the sets are read from their own repositories, never committed here",
name,
answered_total,
asked_total,
"local row: the sets are read from their own repositories, never committed "
"here. " + reason_tail,
details,
)