open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(retrieval-gate): the judge reads a passage as its exact reconstruction — M15 felled, row 7 15 of 15

Browse source 
Row 8's judge required every delivered text to be the whole concept's bytes
(the row-6 identity), which is older than v1.1 C1: an excerpt `as_passage` cut
to its answering passage was class e even when its span carried the citation.
PM's re-measurement of e503f6a found that the dominant miss class of row 8 on
the real set.

`passage_span` accepts a passage only when the delivered text IS what the
bundle's bytes rebuild: the span is body[start:end] byte for byte in the
DELIVERED body (the offsets land a frontmatter's length off in the file), `of`
is that body's length, `[...]` stands exactly where text is left out, and the
one other line allowed is a heading line of the body above the span, or a
prefix of one. The citation is read in the span alone. A passage that is not
its reconstruction is class e with its own detail. A quote only in the heading
or the markers is class d.

Chose to check that the heading is one of the body's own heading lines above
the span, and not to re-derive WHICH heading `as_passage` picks, because the
guarantee is "the bundle's bytes"; the nearest-heading rule is the product's
presentation and a judge that copies it agrees by construction.

M15 (a passage carrying one sentence the file does not) is row 7's fifteenth
mutant. No synthetic concept was long enough to be cut, so a new fixture
delivers one (DELIVERY, ~6 600 characters, set set-passage.json); the corpus
pin moved with it. MUTANT_BAR unchanged, src/ untouched.

Gate, synthetic: rows 1 and 6 13/13 -> 14/14, row 7 14/14 -> 15 of 15 GREEN
(M15 -> row 1, 6), rows 2/3/4 unchanged, GATE RED: rows 5, 8. Row 8 rerun on
the real set: no class e miss remains (its numbers are kept out of this public
history, per the wiki directive). Suite after git add,
FORCE_COLOR unset: 2450 passed, 1 skipped (+6). ruff, format, mypy --strict
clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-22 23:53:09 +02:00
commit cb0c10b421
5 changed files with 199 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

24
README.md
View file

@ -88,9 +88,9 @@ here was measured; none of it is a plan.
on rows 2, 3 and 6. For a user that means: no ranking change has yet been
registered against a held-out set (row 5), the real-corpus row is NOT RUN
without its private set and so is red rather than green by omission
(row 8) -- and when it IS run, its judge still requires a delivered excerpt
to be the whole concept's bytes, so an excerpt v1.1 cuts to its answering
passage counts as a miss even where it carries the citation. MCP anchors and concept ids are different
(row 8) -- and since 2026-09-22 its judge reads an excerpt v1.1 cuts to its
answering passage as that passage, accepted only when it is the exact
reconstruction from the bundle's bytes. MCP anchors and concept ids are different
vocabularies so `okf_fetch` cannot be addressed with a set's anchor (row 2),
and the accounting still reports real losses on the reference corpus (rows 2,
3, 6). The rows and their numbers are under [Judge the
@ -1201,13 +1201,13 @@ finding rather than a defect in the gate:
| row | what it asks | today |
|---|---|---|
| 1 | hit@payload, one fasit entry = one unit | 13 of 13 |
| 1 | hit@payload, one fasit entry = one unit | 14 of 14 |
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
| 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 13 of 13 |
| 7 | mechanical mutants of the ranking and the cut, felled | 14 of 14 |
| 6 | every delivery confirmed against the bundle's own bytes | 14 of 14 |
| 7 | mechanical mutants of the ranking and the cut, felled | 15 of 15 |
| 8 | the real set, from path + sha256 | 0 of 1 set, NOT RUN without `--real` |
**The public rows run on invented material only** (operator decision
@ -1230,7 +1230,7 @@ delivered excerpt answers — facts, no verdict, and the one bar is the gate's
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
Row 7's mutants patch the code the DEFAULT ranking runs (BM25 since v1.1),
and it fells all 14. Until 2026-09-22 three survived with **0 ranks and 0
and it fells all 15. Until 2026-09-22 three survived with **0 ranks and 0
deliveries** moved -- the passage signal reading no body, the field signal
weighing no title and no path, and a flattened fusion (`bm25.RRF_K`) -- because
every synthetic concept was short and opened with its own title, so none of the
@ -1243,6 +1243,16 @@ fusion ranks first because only the rank sum then counts. Each fixture is
delivered at the shipped defaults and falls out of `k` with its own mechanism
switched off; `src/` did not change.
A delivery is judged against the bundle's own bytes (row 6), and since v1.1 a
concept over 4 000 characters is delivered as its answering passage: the
nearest heading above it, `[...]`, the span, `[...]`. The judge counts such a
passage only when it is EXACTLY that reconstruction -- the span byte for byte
the delivered body at the passage's own offsets, markers only where text is
left out, the heading a line of the body above the span -- and it reads the
citation in the span alone. A looser rule ("the span occurs somewhere in the
text") would accept an invented sentence beside the span; the fifteenth
mutant (M15) is exactly that, and a fixture that delivers a passage fells it.
Row 8 is never green by leaving something out: it requires every named set and
is NOT RUN until it has them — one set of three used to read `6 of 6 GREEN`.
The set lives in another repository and is read, never written: