open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(gates): retire the K2 track; re-measure the retrieval gate's premises for BM25

Browse source 
Operator decision 2026-09-21: the test track built on material tied to the
operator's employer (K2) is retired -- not re-measured, not frozen. Public
tests and gates run on invented material.

Retrieval gate:
- The four FUSION_PREMISE xfails are gone and pass through their INPUTS: the
  synthetic MISS, LOOKUP and QUOTA bundles were re-measured for BM25 (the
  miss fasit no longer shares the rare word `maa`; lookup and quota decoys
  carry the question's words so each partition and the quota decide their
  own fixture). SPECS_SHA256 moved with them. Rows 2 and 3 green again.
- Row 7's mutants M04, M06, M07, M08, M10 now patch `bm25`, the code the
  default runs. Three survive with 0 ranks moved (passage body, title
  weight, bm25.RRF_K), each with its mechanism printed. M07 was not forced:
  every synthetic body carries its title as a heading.
- Row 9 (K2) removed; row 8 requires `wiki-20` alone, the `r761` and
  `vegnormal` adapters are gone. Chose the broad reading of "K2" because the
  operator decision defines it as employer-tied material and the order's
  grep includes `vegnormal`.

Also removed: tests/test_default_bundle_pin.py, the K2 arms of
test_okf_consume, the four real-arm tests of test_quality, the R761 soft
hyphen test, the N101/N200 delivery tests and okf_accounting_gate's default
real corpus (and H5's guard, which only existed for those defaults). Two
fixtures carrying road-standard identifiers are rewritten with invented ones.

Gate after: 1 10/10, 2 7/7, 3 5/5, 4 6/6, 5 0/1, 6 10/10, 7 11/14,
8 NOT RUN -> GATE RED: rows 5, 7, 8. Suite 2423 passed, 1 skipped,
0 xfailed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 10:05:43 +02:00
commit 3d149f955a
12 changed files with 288 additions and 1096 deletions
Download patch file Download diff file Expand all files Collapse all files

61
README.md
View file

@ -83,11 +83,10 @@ here was measured; none of it is a plan.
`--no-assets` takes the image path out entirely.
- **Three of this repository's own gates are RED, and each red row is a stated
finding rather than a bug to be surprised by.** The retrieval gate is red on
rows 5, 7, 8 and 9, the MCP gate on row 2, and the content accounting's judge
rows 5, 7 and 8, the MCP gate on row 2, and the content accounting's judge
on rows 2, 3 and 6. For a user that means: retrieval quality is measured but
not yet green on a held-out set (rows 5, 8), two mechanical mutants of the
ranking survive with 0 ranks and 0 deliveries moved (row 7), no gold set
exists for the K2 corpus (row 9), MCP anchors and concept ids are different
not yet green on a held-out set (rows 5, 8), three mechanical mutants of the
ranking survive with 0 ranks and 0 deliveries moved (row 7), MCP anchors and concept ids are different
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
and the accounting still reports real losses on the reference corpus (rows 2,
3, 6). The rows and their numbers are under [Judge the
@ -1158,11 +1157,11 @@ what a bundle looks like, this asks whether the payload for a question carries
the fasit — and whether the payload says so when it does not know.
```bash
python3 tools/okf_retrieval_gate.py # nine rows, one exit code
python3 tools/okf_retrieval_gate.py # eight rows, one exit code
python3 tools/okf_retrieval_gate.py --json # the same rows as JSON
```
Nine rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong
Eight rows, exit 0 only when every one is green, 1 otherwise, 2 for wrong
input. Rows 1–4, 6 and 7 run against a synthetic corpus this repository
generates and six question sets it ships, pinned by sha256: no network, no
private corpus, no clock. A question set is always an input — `sha256` is
@ -1171,7 +1170,7 @@ set names a consumer's documents and this repository is public. **The corpus
is pinned the same way** (`SPECS_SHA256`): every row counts against those
documents, so moving them without moving the pin is exit 2.
**It is RED today, on rows 5, 7, 8 and 9**, and each of those is a
**It is RED today, on rows 5, 7 and 8**, and each of those is a
finding rather than a defect in the gate:
| row | what it asks | today |
@ -1182,9 +1181,14 @@ finding rather than a defect in the gate:
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
| 7 | mechanical mutants of the ranking and the cut, felled | 12 of 14 |
| 8 | the three real sets, from path + sha256 | 0 of 3 sets, NOT RUN without `--real` |
| 9 | K2 | 0 of 6, no gold set exists |
| 7 | mechanical mutants of the ranking and the cut, felled | 11 of 14 |
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
**The public rows run on invented material only** (operator decision
2026-09-21). The test track built on material tied to the operator's employer
is retired rather than re-measured or frozen: its row 9 is gone, row 8 no
longer reads the two sets built on it, and no test or gate here needs that
material to run.
Rows 3 and 4 were this gate's two findings and both are closed, which is what
a gate written before the capability is for. Row 3: in a bundle built from ONE
@ -1199,34 +1203,25 @@ the pre-pass read, the terms no concept in the bundle answers and the terms no
delivered excerpt answers — facts, no verdict, and the one bar is the gate's
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
Row 7 reports two survivors with what they moved rather than with a shrug:
killing the document prior and flattening the fusion (`RRF_K`) each moved
**0 ranks and 0 deliveries** on these fixtures. Both have a mechanism —
a question that names its document reaches it through the title-and-id signal
as well, and `1/(K+r)` is strictly decreasing in `r` for every `K`.
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
and it reports three survivors with what they moved rather than with a shrug:
the passage signal reading no body, the field signal weighing no title and no
path, and a flattened fusion (`bm25.RRF_K`) each moved **0 ranks and 0
deliveries** on these fixtures. Each has a mechanism: the field signal still
reads every body, every synthetic body opens with its own title as a heading,
and over two signals `K` only reorders a concept whose two ranks lie far apart,
which no fixture holds.
Rows 8 and 9 are never green by leaving something out, and since 2026-09-19
that is enforced rather than stated: row 8 requires **all three** named sets
(`wiki-20`, `r761-sk2`, `vegnormal-32`) and is NOT RUN until it has them,
whatever the ones that ran scored — one set of three used to read `6 of 6
GREEN`. The sets live in other repositories and are read, never written:
`--real wiki <set.json> <sha256> <bundle>` runs one, and
`--real vegnormal <set.json> <sha256> "N100=<bundle>,N200=<bundle>"` runs one
that spans bundles. Row 9 takes `--k2 <set.json> <sha256> <bundle>` in this
gate's own set shape; without one it stays RED against its recorded
denominator of six.
Row 8 is never green by leaving something out: it requires every named set and
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.
The set lives in another repository and is read, never written:
`--real wiki <set.json> <sha256> <bundle>` runs it.
Granularity is stated on every line and the two forms are never summed: a set
naming a citation is measured at citation granularity, a set naming only a
section is measured at concept granularity. **Row 8's own headline is
therefore at QUESTION granularity** — the one unit all three sets share —
with the two unit totals printed below it, each with its own denominator. The
table above reports the gate's DEFAULT run, where row 8 is `0 of 3` and NOT
RUN because the sets are not here; the last run that was given all three, on
one machine 2026-09-19, scored **44 of 64 questions**, and below it *7 of 29
at citation granularity, 38 of 50 at concept granularity*. That figure is not
reproducible from this repository alone, which is why it is labelled with the
day and the machine rather than printed as a row.
therefore at QUESTION granularity**, with the two unit totals printed below
it, each with its own denominator.
## Consume in Claude Code