test(consume): the retrieval gate, written red over nine rows
One command, one exit code, and no capability: nothing here changes the ranking, the fusion, the tokenisation or the cut. For a frozen question set against a bundle it asks how many of N measurement units the payload carries the fasit for, gives every miss exactly ONE class, and asks the thing the payload does not do at all -- say so when it does not know. Today, on the synthetic corpus this repository generates and the five sets it ships pinned by sha256: rows 1 (9 of 9), 2 (7 of 7) and 6 (9 of 9) green; 3 (2 of 5), 4 (3 of 6), 5 (0 of 1), 7 (11 of 13), 8 (not run) and 9 (0 of 6) red. GATE RED, exit 1. - Row 3 is the finding to read first. In a bundle built from ONE source document, every concept past the first two carries that document's `source_file`, so a concept the RANK had already lost is withheld as `source_quota_exceeded`. The truth is decided by the SAME cut run without the quota -- PM measured the same lie on 13 of 25 real misses 2026-09-17. - Row 4 is the second. No payload key says "this bundle does not answer that", so the only honest reading today is `delivered == 0`; three of six controls come out right and the three that do not get eight excerpts and no statement. A control the bundle DOES answer must come back unmarked, or the marking says nothing, so both directions sit in the same row. - A miss has exactly one class -- (a) not in the bundle, (b) below k, (c) cut by quota or budget, (d) delivered without the citation, (e) other -- and each is FORCED by its own fixture. An unplaced miss takes the row to 0 of N. - The judge opens the bundle. Every concept is read off disk before a payload is built, a delivered excerpt whose text is not the concept file's bytes is not a hit, and the two title forms come from `quality` rather than being written a second time. - FELLED means a row got WORSE, never merely that something changed: removing the quota makes row 3 green, and counting that as a kill would credit this gate with a check it does not have. The two survivors are reported with what they measured -- the document prior dead and RRF_K flattened each moved 0 ranks and 0 deliveries on these fixtures -- and with a mechanism for each. - Rows 8 and 9 are never green by leaving something out. The three real sets are read from path + sha256 and never committed: a gold set names a consumer's documents and this repository is public. K2 is 0 of 6 because the gold set exists nowhere while its bundles do. Denominators re-measured against the sources 2026-09-19, and all four hold: wiki 20 questions / 29 fasit entries (sha 972d0f57...), R761 8 (S1-S6 + KP + KN), vegnormal 32 questions / 43 must_cite, K2 6. PM's 45 of 70 is itself a mixed sum -- wiki at question level, vegnormal at citation level -- so this gate carries it labelled as recorded and never sums the two granularities. 46 tests, each failing row driven both ways through inputs rather than by editing the row. No network (the socket guard is fired before its silence counts), no private corpus, no clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ae441abec4
commit
2c8296b807
9 changed files with 2748 additions and 0 deletions
57
README.md
57
README.md
|
|
@ -998,6 +998,63 @@ on every run. Both facts, the arm it separates (1 148 of 2 761 against 2 759 of
|
|||
2 761) and the interval any bar could sit in are in
|
||||
[`docs/2026-09-12-g37-terskler.md`](docs/2026-09-12-g37-terskler.md) § 7.
|
||||
|
||||
## Judge the retrieval: `python3 tools/okf_retrieval_gate.py`
|
||||
|
||||
A separate question from `okf quality`, and a separate command: quality asks
|
||||
what a bundle looks like, this asks whether the payload for a question carries
|
||||
the fasit — and whether the payload says so when it does not know.
|
||||
|
||||
```bash
|
||||
python3 tools/okf_retrieval_gate.py # nine rows, one exit code
|
||||
python3 tools/okf_retrieval_gate.py --json # the same rows as JSON
|
||||
```
|
||||
|
||||
Nine rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong
|
||||
input. Rows 1–4, 6 and 7 run against a synthetic corpus this repository
|
||||
generates and four question sets it ships, pinned by sha256: no network, no
|
||||
private corpus, no clock. A question set is always an input — `sha256` is
|
||||
checked before a byte is measured and a mismatch is exit 2 — because a gold
|
||||
set names a consumer's documents and this repository is public.
|
||||
|
||||
**It is RED today, on rows 3, 4, 5, 7, 8 and 9**, and each of those is a
|
||||
finding rather than a defect in the gate:
|
||||
|
||||
| row | what it asks | today |
|
||||
|---|---|---|
|
||||
| 1 | hit@payload, one fasit entry = one unit | 9 of 9 |
|
||||
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
|
||||
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 2 of 5 |
|
||||
| 4 | an uncovered question comes back marked, a covered one does not | 3 of 6 |
|
||||
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
|
||||
| 6 | every delivery confirmed against the bundle's own bytes | 9 of 9 |
|
||||
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 13 |
|
||||
| 8 | the three real sets, from path + sha256 | not run |
|
||||
| 9 | K2 | 0 of 6, no gold set exists |
|
||||
|
||||
Row 3 is the one to read first: in a bundle built from ONE source document,
|
||||
every concept past the first two carries that document's `source_file`, so a
|
||||
concept the RANK had already lost is withheld as `source_quota_exceeded`. The
|
||||
gate decides the truth with the same cut run without the quota, and a
|
||||
consumer reading `rule` today gets the wrong reason. Row 4 is the second: the
|
||||
payload has no key a consumer can read as "this bundle does not answer that",
|
||||
so an uncovered question comes back with excerpts and no statement — the only
|
||||
honest case today is the one where nothing matched at all and nothing was
|
||||
delivered.
|
||||
|
||||
Row 7 reports two survivors with what they moved rather than with a shrug:
|
||||
killing the document prior and flattening the fusion (`RRF_K`) each moved
|
||||
**0 ranks and 0 deliveries** on these fixtures. Both have a mechanism —
|
||||
a question that names its document reaches it through the title-and-id signal
|
||||
as well, and `1/(K+r)` is strictly decreasing in `r` for every `K`.
|
||||
|
||||
Rows 8 and 9 are never green by leaving something out. The three real sets
|
||||
live in other repositories and are read, never written:
|
||||
`--real wiki <set.json> <sha256> <bundle>` runs one, and
|
||||
`--real vegnormal <set.json> <sha256> "N100:2023=<bundle>,N200:2024=<bundle>"`
|
||||
runs one that spans bundles. Granularity is stated on every line and the two
|
||||
forms are never summed: a set naming a citation is measured at citation
|
||||
granularity, a set naming only a section is measured at concept granularity.
|
||||
|
||||
## Consume in Claude Code
|
||||
|
||||
A folder of documents to an answer a model can cite, in **three lines**. You do
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue