open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(retrieval-gate): rows 2 and 3 count against the pinned set, not the run

Browse source 
PM's J10 and J8. Two mechanisms, one per attack, and neither is a pin a
capability session can edit in the same breath as the code.

J10 -- THE DENOMINATOR IS THE SET'S. `Unit` now carries the class its question
DECLARES. Row 2's denominator is the misses plus every forced fixture that
came back a hit, and row 3's is every unit whose set declares a withheld class
(b, c, e -- (a) is not in the bundle and (d) was delivered, so neither can
carry a printed reason) plus whatever the run withheld besides. A fixture that
stops producing its declared class is a BROKEN PREMISE, printed as one, and it
counts against its row: at `k = 32` row 2 stays RED with its denominator held
and row 3 keeps 5 units where it had shrunk to 2 and called that green.

J8 -- THE ROW CARRIES A KNOWN-POSITIVE. With `--source-quota` off, every
printed reason is true; that reading is not a lie, it is an empty measurement,
and row 3 must say so rather than print `6 of 6 GREEN` beside row 1 falling to
8 of 9. A set may now declare `source_quota_in_force`, and the row is NOT RUN
for such a set when the default cut and the quota-off cut deliver the same
concepts everywhere.

THE CONTROL'S OWN PREMISE WAS MEASURED FIRST, and it was false where it was
first put: over the five existing sets the two cuts deliver the SAME concepts
(the quota is topped back up), so 52 labels move `source_quota_exceeded` ->
`below_k` without one delivery changing. `set-quota.json` is the set where the
quota genuinely decides -- measured, `oversikt-08` is delivered without the
quota and withheld with it, and the fasit `svar/broennproeve` is delivered
only with it -- so the requirement is declared there and nowhere else. It
survives the honest fix, which changes labels and not the cut.

Rows 1 and 6 go 9 of 9 to 10 of 10: one added fixture, one added hit, both
green before and after. Rows 2 (7 of 7), 3 (2 of 5), 4, 5, 7, 8, 9 and the
verdict `GATE RED: rows 3, 4, 5, 7, 8, 9` are unchanged. 52 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 21:03:55 +02:00
commit be899106c4
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 132 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

22
tests/fixtures/retrieval/set-quota.json vendored Normal file
View file

@ -0,0 +1,22 @@
{
"set_id": "synthetic-quota",
"bundle": "quota",
"requires": [
"source_quota_in_force"
],
"note": "Row 3's known-positive, and the only set that declares one. One document floods the question with ten near-identical concepts and another holds the answer: with the shipped quota the answer is delivered and `oversikt-08` is withheld as `source_quota_exceeded` TRUTHFULLY, because the quota-off run delivers it. Without the quota the answer is not delivered at all and no concept anywhere carries that label -- which is the run row 3 must refuse to judge, rather than call 6 of 6 green (PM's J8, 2026-09-19).",
"questions": [
{
"id": "Q1",
"question": "Naar proevetas broennen og hvem arkiverer analysen av dugnaden?",
"fasit": [
{
"by": "concept",
"value": "svar/broennproeve",
"quote": "Broennen proevetas i juni"
}
]
}
],
"controls": []
}

17
tests/test_retrieval_gate.py
View file

@ -578,9 +578,12 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
by_number = {row.number: row for row in rows}
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8, 9]
assert [row.number for row in rows if row.fails] == [3, 4, 5, 7, 8, 9]
assert (by_number[1].k, by_number[1].m) == (9, 9)
# 10, not 9: `set-quota.json` adds row 3's known-positive, one question the
# source quota genuinely decides, and it is a hit.
assert (by_number[1].k, by_number[1].m) == (10, 10)
assert (by_number[2].k, by_number[2].m) == (7, 7)
assert (by_number[6].k, by_number[6].m) == (9, 9)
assert (by_number[3].k, by_number[3].m) == (2, 5)
assert (by_number[6].k, by_number[6].m) == (10, 10)
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
@ -824,14 +827,18 @@ def test_j10_a_wider_cut_does_not_make_rows_two_and_three_green(tmp_path: Path)
"""PM's J10: `k = 32` took rows 1, 2, 3 and 6 green at once, and not one
label had become true -- the denominator of rows 2 and 3 IS the misses, so
delivering more broadly shrinks it to the cases that were already honest."""
baseline = _attack(tmp_path)
rows = _attack(tmp_path, k=32)
assert rows[2].status == gate.RED, rows[2]
assert rows[3].status == gate.RED, rows[3]
assert rows[3].status != gate.GREEN, rows[3]
assert rows[3].fails
# The three fixtures that declare class b are delivered under this cut, so
# their premise no longer holds -- and a broken premise counts against the
# row rather than leaving it.
# row rather than leaving it. NEITHER denominator shrinks: that is the whole
# of the defect.
assert any("premise" in detail for detail in rows[2].details), rows[2].details
assert rows[2].m >= 7
assert rows[2].m >= baseline[2].m
assert rows[3].m >= baseline[3].m
def test_j8_removing_the_quota_leaves_row_three_unable_to_say_anything(

107
tools/okf_retrieval_gate.py
View file

@ -507,6 +507,10 @@ class QuestionSet:
sha256: str
questions: tuple[Question, ...]
controls: tuple[Control, ...]
#: What the set says must be TRUE OF THE RUN for its rows to mean
#: anything. Today one name: `source_quota_in_force`, declared by the set
#: whose fixtures exist to catch a false `source_quota_exceeded` label.
requires: tuple[str, ...] = ()
@property
def units(self) -> int:
@ -560,6 +564,7 @@ def load_set(path: Path, expected_sha256: str) -> QuestionSet:
)
for entry in spec.get("controls", [])
),
requires=tuple(str(item) for item in spec.get("requires", [])),
)
except (KeyError, TypeError, ValueError) as error:
raise GateUsage(f"{path}: not a question set this gate can read: {error}") from error
@ -589,6 +594,7 @@ SYNTHETIC_SETS: dict[str, str] = {
"set-miss.json": "3a75da8fe7177ebfc555ee3c8d1af94adcd927925b4f5dfccd0cb76e3fa5ae8c",
"set-classes.json": "65c3eb272dcbb972abc2072c7a334490f016a8259b7b1a33d780aa497ec1393d",
"set-signals.json": "17d83f305a10af8dd2a45b72943c1285704d23ed074ecb549c05e5fac3e137e9",
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
}
@ -711,6 +717,13 @@ class Unit:
#: and never a row-1 unit: counting a miss built to miss as a miss would
#: make row 1 unable to be green whatever the ranker does.
forced: bool = False
#: The class the set DECLARES for this unit, carried so rows 2 and 3 can
#: take their denominator from the pinned bytes instead of from the run.
expect_class: str | None = None
#: Did the source quota move this question at all -- did the default cut
#: and the quota-off cut deliver different concepts? Row 3's known-positive:
#: with no quota in force, the label row 3 judges is never printed.
quota_moved: bool = False
def _withheld_rules(payload: Mapping[str, object]) -> dict[str, str]:
@ -755,6 +768,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
withheld = _withheld_rules(default)
truth_delivered = _delivered(truth_run)
truth_withheld = _withheld_rules(truth_run)
quota_moved = set(delivered) != set(truth_delivered)
units: list[Unit] = []
for fasit in question.fasit:
candidates = index.candidates(fasit)
@ -839,6 +853,8 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
confirmed=confirmed,
detail=detail,
forced=bool(question.expect_class),
expect_class=question.expect_class,
quota_moved=quota_moved,
)
)
return units
@ -958,6 +974,12 @@ def row_two(cases: Sequence[Case]) -> Row:
"""Every miss has exactly ONE class -- and a miss with none takes the whole
row to 0, because a classification with a hole is not a classification."""
misses = [unit for case in cases for unit in case.units if not unit.hit]
# A fixture that DECLARES the class it forces and then comes back a hit has
# not been classified: its premise broke. Counting it out of the
# denominator is what let `k = 32` take this row from 7 of 7 to 4 of 4 and
# call it green (PM's J10, 2026-09-19). The denominator is therefore the
# pinned set's own, not the run's.
broken = [unit for case in cases for unit in case.units if unit.hit and unit.expect_class]
unplaced = [unit for unit in misses if unit.klass is None]
wrong = [
(unit, expected)
@ -979,23 +1001,62 @@ def row_two(cases: Sequence[Case]) -> Row:
f" forced class {expected}, measured {unit.klass}: {unit.question_id}"
for unit, expected in wrong
]
details += [
f" premise broken: {unit.question_id} {unit.named} declares class "
f"{unit.expect_class} and came back a hit; the fixture must be re-measured"
for unit in broken
]
return _row(
2,
"every miss carries exactly one class",
k,
len(misses),
"each class forced by its own fixture; an unplaced miss makes this 0 of N",
len(misses) + len(broken),
"each class forced by its own fixture; an unplaced miss makes this 0 of N, "
"and a fixture that stops producing its declared class counts against it",
details,
)
#: The classes a set DECLARES that come back withheld with a reason printed.
#: (a) is not in the bundle at all and (d) was delivered, so neither can carry
#: one -- the other three must.
WITHHELD_CLASSES = ("b", "c", "e")
#: A set may say what must be true OF THE RUN before its rows mean anything.
REQUIRE_QUOTA = "source_quota_in_force"
def row_three(cases: Sequence[Case]) -> Row:
"""The `rule` the payload prints for a withheld fasit, against what the
quota-off run says was true of it."""
withheld = [unit for case in cases for unit in case.units if unit.label_default is not None]
quota-off run says was true of it.
THE DENOMINATOR IS THE PINNED SET'S, NOT THE RUN'S. Counting only what
this run happened to withhold is how `k = 32` took the row from 2 of 5 to
2 of 2 and called it green (PM's J10): the three fixtures that declare
class b were delivered, and a unit that leaves the denominator answers
nothing. Every unit whose set declares a withheld class is judged whether
or not this run withheld it, and a declared unit with no printed reason is
not an honest one.
AND THE ROW CARRIES ITS OWN KNOWN-POSITIVE. With `--source-quota` off
every printed reason is true -- there is no quota left to name falsely --
and the row read `6 of 6 GREEN` while row 1 fell to 8 of 9 (PM's J8). That
reading is not a lie, it is an empty measurement, so a set may declare
`source_quota_in_force` and the row is NOT RUN for it when the default cut
and the quota-off cut deliver the same concepts everywhere.
"""
judged: list[Unit] = []
for case in cases:
for unit in case.units:
declared = unit.expect_class in WITHHELD_CLASSES
if declared or unit.label_default is not None:
judged.append(unit)
agreeing = []
lying = []
for unit in withheld:
for unit in judged:
if unit.label_default is None:
lying.append(unit)
continue
honest = (
unit.label_default == "source_quota_exceeded"
if unit.truth == "delivered"
@ -1003,16 +1064,44 @@ def row_three(cases: Sequence[Case]) -> Row:
)
(agreeing if honest else lying).append(unit)
details = [
f" {unit.question_id} {unit.named}: payload says "
f"`{unit.label_default}`, the quota-off run says `{unit.truth}`"
(
f" {unit.question_id} {unit.named}: the set declares class "
f"{unit.expect_class} and the payload printed no reason "
f"({'delivered' if unit.hit else unit.truth})"
)
if unit.label_default is None
else (
f" {unit.question_id} {unit.named}: payload says "
f"`{unit.label_default}`, the quota-off run says `{unit.truth}`"
)
for unit in lying
]
silent = [
case
for case in cases
if REQUIRE_QUOTA in case.question_set.requires
and not any(unit.quota_moved for unit in case.units)
]
if silent:
return Row(
3,
"the reason the payload prints for a withheld fasit is true",
len(agreeing),
len(judged),
NOT_RUN,
"not run: no quota is in force for "
+ ", ".join(case.question_set.set_id for case in silent)
+ " -- the default cut and the quota-off cut deliver the same concepts, "
"so the label this row judges was never printed",
details,
)
return _row(
3,
"the reason the payload prints for a withheld fasit is true",
len(agreeing),
len(withheld),
"`rule` must say what the same cut without the quota says",
len(judged),
"`rule` must say what the same cut without the quota says; the denominator "
"is the set's declared withheld units, never the ones this run withheld",
details,
)