open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

docs(retrieval-gate): the round the gate had to resist the work it judges

Browse source 
README's row table and the `--real` paragraph, CLAUDE.md's gate entry,
CHANGELOG under 0.10.1 (no new version, no tag), and the full report.

The report carries the eight attacks before and after, the row table before
and after, row 8's three-set measurement with its two granularities apart, the
mutation table as properties rather than as code lines, and § 6 -- the three
things this round does NOT close: row 3 can still be made green by an honest
change that costs elsewhere (the gate is the unit, not the row), the judge's
parser is still `consume`'s and is bounded by measurement rather than removed,
and `SPECS_SHA256` digests the corpus's fields rather than the bundle's bytes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-19 22:13:25 +02:00
commit 23588e5ada
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
4 changed files with 335 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

38
README.md
View file

@ -1011,24 +1011,26 @@ python3 tools/okf_retrieval_gate.py --json # the same rows as JSON
Nine rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong
input. Rows 1–4, 6 and 7 run against a synthetic corpus this repository
generates and four question sets it ships, pinned by sha256: no network, no
generates and six question sets it ships, pinned by sha256: no network, no
private corpus, no clock. A question set is always an input — `sha256` is
checked before a byte is measured and a mismatch is exit 2 — because a gold
set names a consumer's documents and this repository is public.
set names a consumer's documents and this repository is public. **The corpus
is pinned the same way** (`SPECS_SHA256`): every row counts against those
documents, so moving them without moving the pin is exit 2.
**It is RED today, on rows 3, 4, 5, 7, 8 and 9**, and each of those is a
finding rather than a defect in the gate:
| row | what it asks | today |
|---|---|---|
| 1 | hit@payload, one fasit entry = one unit | 9 of 9 |
| 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 2 of 5 |
| 4 | an uncovered question comes back marked, a covered one does not | 3 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 9 of 9 |
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 13 |
| 8 | the three real sets, from path + sha256 | not run |
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
| 7 | mechanical mutants of the ranking and the cut, felled | 12 of 14 |
| 8 | the three real sets, from path + sha256 | 44 of 64 questions |
| 9 | K2 | 0 of 6, no gold set exists |
Row 3 is the one to read first: in a bundle built from ONE source document,
@ -1047,13 +1049,25 @@ killing the document prior and flattening the fusion (`RRF_K`) each moved
a question that names its document reaches it through the title-and-id signal
as well, and `1/(K+r)` is strictly decreasing in `r` for every `K`.
Rows 8 and 9 are never green by leaving something out. The three real sets
live in other repositories and are read, never written:
Rows 8 and 9 are never green by leaving something out, and since 2026-09-19
that is enforced rather than stated: row 8 requires **all three** named sets
(`wiki-20`, `r761-sk2`, `vegnormal-32`) and is NOT RUN until it has them,
whatever the ones that ran scored — one set of three used to read `6 of 6
GREEN`. The sets live in other repositories and are read, never written:
`--real wiki <set.json> <sha256> <bundle>` runs one, and
`--real vegnormal <set.json> <sha256> "N100:2023=<bundle>,N200:2024=<bundle>"`
runs one that spans bundles. Granularity is stated on every line and the two
forms are never summed: a set naming a citation is measured at citation
granularity, a set naming only a section is measured at concept granularity.
`--real vegnormal <set.json> <sha256> "N100=<bundle>,N200=<bundle>"` runs one
that spans bundles. Row 9 takes `--k2 <set.json> <sha256> <bundle>` in this
gate's own set shape; without one it stays RED against its recorded
denominator of six.
Granularity is stated on every line and the two forms are never summed: a set
naming a citation is measured at citation granularity, a set naming only a
section is measured at concept granularity. **Row 8's own headline is
therefore at QUESTION granularity** — the one unit all three sets share —
with the two unit totals printed below it, each with its own denominator.
Measured 2026-09-19 against the three real sets on one machine: **44 of 64
questions**, and below it *7 of 29 at citation granularity, 38 of 50 at
concept granularity*.
## Consume in Claude Code