test(consume): the retrieval gate, written red over nine rows
One command, one exit code, and no capability: nothing here changes the ranking, the fusion, the tokenisation or the cut. For a frozen question set against a bundle it asks how many of N measurement units the payload carries the fasit for, gives every miss exactly ONE class, and asks the thing the payload does not do at all -- say so when it does not know. Today, on the synthetic corpus this repository generates and the five sets it ships pinned by sha256: rows 1 (9 of 9), 2 (7 of 7) and 6 (9 of 9) green; 3 (2 of 5), 4 (3 of 6), 5 (0 of 1), 7 (11 of 13), 8 (not run) and 9 (0 of 6) red. GATE RED, exit 1. - Row 3 is the finding to read first. In a bundle built from ONE source document, every concept past the first two carries that document's `source_file`, so a concept the RANK had already lost is withheld as `source_quota_exceeded`. The truth is decided by the SAME cut run without the quota -- PM measured the same lie on 13 of 25 real misses 2026-09-17. - Row 4 is the second. No payload key says "this bundle does not answer that", so the only honest reading today is `delivered == 0`; three of six controls come out right and the three that do not get eight excerpts and no statement. A control the bundle DOES answer must come back unmarked, or the marking says nothing, so both directions sit in the same row. - A miss has exactly one class -- (a) not in the bundle, (b) below k, (c) cut by quota or budget, (d) delivered without the citation, (e) other -- and each is FORCED by its own fixture. An unplaced miss takes the row to 0 of N. - The judge opens the bundle. Every concept is read off disk before a payload is built, a delivered excerpt whose text is not the concept file's bytes is not a hit, and the two title forms come from `quality` rather than being written a second time. - FELLED means a row got WORSE, never merely that something changed: removing the quota makes row 3 green, and counting that as a kill would credit this gate with a check it does not have. The two survivors are reported with what they measured -- the document prior dead and RRF_K flattened each moved 0 ranks and 0 deliveries on these fixtures -- and with a mechanism for each. - Rows 8 and 9 are never green by leaving something out. The three real sets are read from path + sha256 and never committed: a gold set names a consumer's documents and this repository is public. K2 is 0 of 6 because the gold set exists nowhere while its bundles do. Denominators re-measured against the sources 2026-09-19, and all four hold: wiki 20 questions / 29 fasit entries (sha 972d0f57...), R761 8 (S1-S6 + KP + KN), vegnormal 32 questions / 43 must_cite, K2 6. PM's 45 of 70 is itself a mixed sum -- wiki at question level, vegnormal at citation level -- so this gate carries it labelled as recorded and never sums the two granularities. 46 tests, each failing row driven both ways through inputs rather than by editing the row. No network (the socket guard is fired before its silence counts), no private corpus, no clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ae441abec4
commit
2c8296b807
9 changed files with 2748 additions and 0 deletions
41
CLAUDE.md
41
CLAUDE.md
|
|
@ -1182,6 +1182,47 @@ and fixtures, never code.
|
|||
stays unaccounted, and therefore red.
|
||||
- Reports: `docs/2026-09-17-innholdsregnskapet-rod-gate.md`,
|
||||
`docs/2026-09-17-innholdsregnskapet-bygget.md`.
|
||||
- **Judge the RETRIEVAL: `python3 tools/okf_retrieval_gate.py` (2026-09-19).**
|
||||
A separate question from `okf quality` and a separate command: quality asks
|
||||
what a bundle looks like, this asks whether the payload for a question
|
||||
carries the fasit. **Written RED, no capability**: nine rows, exit 0 only
|
||||
when all are green, 1 otherwise, 2 on wrong input, `--json`. Rows 1-4, 6, 7
|
||||
run against a synthetic corpus this repo GENERATES and four sets it ships
|
||||
pinned by sha256 -- no network, no private corpus, no clock; a set whose
|
||||
bytes moved is exit 2, never a quiet continuation, because **a gold set is
|
||||
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
|
||||
this repo is public). Today: **1 (9/9) - 2 (7/7) - 3 (2/5) - 4 (3/6) -
|
||||
5 (0/1) - 6 (9/9) - 7 (11/13) - 8 not run - 9 (0/6)**, `GATE RED: rows 3,
|
||||
4, 5, 7, 8, 9`. **Row 3 is the finding to read first**: in a bundle built
|
||||
from ONE source document every concept past the first two carries that
|
||||
document's `source_file`, so a concept the RANK had already lost is withheld
|
||||
as `source_quota_exceeded` -- the truth is decided by the SAME cut run
|
||||
without the quota, and PM measured the same lie on **13 of 25** real misses
|
||||
17.09. **Row 4 is the second**: no payload key says "this bundle does not
|
||||
answer that", so the only honest case is `delivered == 0`; 3 of 6 controls
|
||||
come out right and the three that do not get eight excerpts and no
|
||||
statement. **A miss has exactly ONE class** (a fasit not in the bundle, b
|
||||
below k, c cut by quota/budget, d delivered without the citation, e other),
|
||||
each FORCED by its own fixture, and an unplaced miss takes the row to 0 of
|
||||
N. **The judge opens the bundle**: `BundleIndex` reads every concept off
|
||||
disk before a payload is built, a delivered excerpt whose text is not the
|
||||
concept file's bytes is not a hit, and the title rules are imported from
|
||||
`quality` rather than written a second time. **Felled means WORSE, never
|
||||
merely different** -- removing the quota makes row 3 green, and calling that
|
||||
a kill would credit the gate with a check it does not have; the two
|
||||
survivors (document prior dead, `RRF_K` flattened) are reported with what
|
||||
they measured: **0 ranks and 0 deliveries moved**, and a mechanism for each.
|
||||
Rows 8 and 9 are never green by leaving something out: the three real sets
|
||||
are read from path + sha256 (`--real wiki|r761|vegnormal`, one Question per
|
||||
standard where a question cites two), and K2 is **0 of 6 because the gold
|
||||
set exists nowhere** while the bundles do. Granularity is stated and the two
|
||||
forms are NEVER summed: a citation-level set and a concept-level set are two
|
||||
numbers. Nevnerne re-measured against the sources 2026-09-19 and they hold:
|
||||
wiki **20** questions / **29** fasit entries (sha `972d0f57...`), R761
|
||||
**8** (S1-S6 + KP + KN), vegnormal **32** questions / **43** `must_cite` /
|
||||
32 `must_contain`, K2 **6**. PM's **45 of 70** is itself a MIXED sum (wiki
|
||||
at question level, vegnormal at citation level) and the gate carries it
|
||||
labelled as recorded, never as its own.
|
||||
- **Judge a bundle: `okf quality <bundle>` (G37, 2026-09-12).** A per-file-type
|
||||
verdict, with the denominator on every line, and it is a SEPARATE command
|
||||
from `okf check` on purpose: `check` is the contract check, and a green one
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue