open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(retrieval-gate): rows 2 and 3 count against the pinned set, not the run

Browse source 
PM's J10 and J8. Two mechanisms, one per attack, and neither is a pin a
capability session can edit in the same breath as the code.

J10 -- THE DENOMINATOR IS THE SET'S. `Unit` now carries the class its question
DECLARES. Row 2's denominator is the misses plus every forced fixture that
came back a hit, and row 3's is every unit whose set declares a withheld class
(b, c, e -- (a) is not in the bundle and (d) was delivered, so neither can
carry a printed reason) plus whatever the run withheld besides. A fixture that
stops producing its declared class is a BROKEN PREMISE, printed as one, and it
counts against its row: at `k = 32` row 2 stays RED with its denominator held
and row 3 keeps 5 units where it had shrunk to 2 and called that green.

J8 -- THE ROW CARRIES A KNOWN-POSITIVE. With `--source-quota` off, every
printed reason is true; that reading is not a lie, it is an empty measurement,
and row 3 must say so rather than print `6 of 6 GREEN` beside row 1 falling to
8 of 9. A set may now declare `source_quota_in_force`, and the row is NOT RUN
for such a set when the default cut and the quota-off cut deliver the same
concepts everywhere.

THE CONTROL'S OWN PREMISE WAS MEASURED FIRST, and it was false where it was
first put: over the five existing sets the two cuts deliver the SAME concepts
(the quota is topped back up), so 52 labels move `source_quota_exceeded` ->
`below_k` without one delivery changing. `set-quota.json` is the set where the
quota genuinely decides -- measured, `oversikt-08` is delivered without the
quota and withheld with it, and the fasit `svar/broennproeve` is delivered
only with it -- so the requirement is declared there and nowhere else. It
survives the honest fix, which changes labels and not the cut.

Rows 1 and 6 go 9 of 9 to 10 of 10: one added fixture, one added hit, both
green before and after. Rows 2 (7 of 7), 3 (2 of 5), 4, 5, 7, 8, 9 and the
verdict `GATE RED: rows 3, 4, 5, 7, 8, 9` are unchanged. 52 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 21:03:55 +02:00
commit be899106c4
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
3 changed files with 132 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

22
tests/fixtures/retrieval/set-quota.json vendored Normal file
View file

@ -0,0 +1,22 @@
{
"set_id": "synthetic-quota",
"bundle": "quota",
"requires": [
"source_quota_in_force"
],
"note": "Row 3's known-positive, and the only set that declares one. One document floods the question with ten near-identical concepts and another holds the answer: with the shipped quota the answer is delivered and `oversikt-08` is withheld as `source_quota_exceeded` TRUTHFULLY, because the quota-off run delivers it. Without the quota the answer is not delivered at all and no concept anywhere carries that label -- which is the run row 3 must refuse to judge, rather than call 6 of 6 green (PM's J8, 2026-09-19).",
"questions": [
{
"id": "Q1",
"question": "Naar proevetas broennen og hvem arkiverer analysen av dugnaden?",
"fasit": [
{
"by": "concept",
"value": "svar/broennproeve",
"quote": "Broennen proevetas i juni"
}
]
}
],
"controls": []
}