Two of PM's eight attacks, closed where they were measured.
J2 -- a set left out. `REQUIRED_REAL_SETS` names the three (`wiki-20`,
`r761-sk2`, `vegnormal-32`) and the row is NOT RUN until all three are given,
whatever the ones that ran scored. The numbers the run DID measure are still
printed: a missing set must not cost the reader the set that was measured, and
"not run" is the row's status, never a reason to withhold a figure.
J2b -- the headline was `quoted_hits + concept_hits` over
`quoted_units + concept_units`, written one line above the detail that says
the two are not summed. The three sets do not share a unit: `wiki-20` names a
citation, `r761-sk2` a section title, `vegnormal-32` a requirement number, and
a citation hit plus a concept hit is a number that is neither. A QUESTION is
the one unit all three have, so the headline is questions answered of
questions asked -- answered meaning at least one of the question's fasit
entries arrived, the reading row 1 already prints beside its own units -- and
the two unit totals keep their own denominators below it, unsummed.
49 passed (was 46 passed, 3 failed). The gate is unchanged where it did not
run: still exit 1, still `0 of 3 NOT RUN` with PM's recorded figures carried.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PM's checkpoint on 2c8296b ran eight cheating attacks at this gate and four
went through: a row came back GREEN without one label becoming true or one
concept ranking better. This is the first of them, written as a test that must
refuse it.
J2: `row_eight` iterates over whatever `--real` handed it and counts. One set
of three came back `6 of 6 GREEN`, and the realistic route is `--real r761`
alone -- the set PM's own registration puts at 7 of 7, with the two that miss
left out. The docstring already said "never green by leaving a set out"; the
code did not, and THIS REPOSITORY'S OWN TEST asserted the opposite at
`tests/test_retrieval_gate.py:486`, `(1, 1, GREEN)` for a single wiki set.
That assertion is corrected here rather than worked around: a test that pins
the defect is the defect.
J2b: the headline `k of N` is `quoted_hits + concept_hits` over
`quoted_units + concept_units`, written on the line directly above the detail
that says the two granularities are NOT summed into one number.
3 red on an assertion about behaviour (GREEN where NOT RUN is required, and a
headline that is a sum), 46 passed. The green direction -- all three named
sets present -- is in the same commit and passes already, so the fix cannot
close the rows by making the row unreachable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One command, one exit code, and no capability: nothing here changes the
ranking, the fusion, the tokenisation or the cut. For a frozen question set
against a bundle it asks how many of N measurement units the payload carries
the fasit for, gives every miss exactly ONE class, and asks the thing the
payload does not do at all -- say so when it does not know.
Today, on the synthetic corpus this repository generates and the five sets it
ships pinned by sha256: rows 1 (9 of 9), 2 (7 of 7) and 6 (9 of 9) green;
3 (2 of 5), 4 (3 of 6), 5 (0 of 1), 7 (11 of 13), 8 (not run) and 9 (0 of 6)
red. GATE RED, exit 1.
- Row 3 is the finding to read first. In a bundle built from ONE source
document, every concept past the first two carries that document's
`source_file`, so a concept the RANK had already lost is withheld as
`source_quota_exceeded`. The truth is decided by the SAME cut run without
the quota -- PM measured the same lie on 13 of 25 real misses 2026-09-17.
- Row 4 is the second. No payload key says "this bundle does not answer
that", so the only honest reading today is `delivered == 0`; three of six
controls come out right and the three that do not get eight excerpts and no
statement. A control the bundle DOES answer must come back unmarked, or the
marking says nothing, so both directions sit in the same row.
- A miss has exactly one class -- (a) not in the bundle, (b) below k, (c) cut
by quota or budget, (d) delivered without the citation, (e) other -- and
each is FORCED by its own fixture. An unplaced miss takes the row to 0 of N.
- The judge opens the bundle. Every concept is read off disk before a payload
is built, a delivered excerpt whose text is not the concept file's bytes is
not a hit, and the two title forms come from `quality` rather than being
written a second time.
- FELLED means a row got WORSE, never merely that something changed: removing
the quota makes row 3 green, and counting that as a kill would credit this
gate with a check it does not have. The two survivors are reported with what
they measured -- the document prior dead and RRF_K flattened each moved 0
ranks and 0 deliveries on these fixtures -- and with a mechanism for each.
- Rows 8 and 9 are never green by leaving something out. The three real sets
are read from path + sha256 and never committed: a gold set names a
consumer's documents and this repository is public. K2 is 0 of 6 because the
gold set exists nowhere while its bundles do.
Denominators re-measured against the sources 2026-09-19, and all four hold:
wiki 20 questions / 29 fasit entries (sha 972d0f57...), R761 8 (S1-S6 + KP +
KN), vegnormal 32 questions / 43 must_cite, K2 6. PM's 45 of 70 is itself a
mixed sum -- wiki at question level, vegnormal at citation level -- so this
gate carries it labelled as recorded and never sums the two granularities.
46 tests, each failing row driven both ways through inputs rather than by
editing the row. No network (the socket guard is fired before its silence
counts), no private corpus, no clock.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
