open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

fix(retrieval-gate): the judge reads a passage as its exact reconstruction — M15 felled, row 7 15 of 15

Browse source 
Row 8's judge required every delivered text to be the whole concept's bytes
(the row-6 identity), which is older than v1.1 C1: an excerpt `as_passage` cut
to its answering passage was class e even when its span carried the citation.
PM's re-measurement of e503f6a found that the dominant miss class of row 8 on
the real set.

`passage_span` accepts a passage only when the delivered text IS what the
bundle's bytes rebuild: the span is body[start:end] byte for byte in the
DELIVERED body (the offsets land a frontmatter's length off in the file), `of`
is that body's length, `[...]` stands exactly where text is left out, and the
one other line allowed is a heading line of the body above the span, or a
prefix of one. The citation is read in the span alone. A passage that is not
its reconstruction is class e with its own detail. A quote only in the heading
or the markers is class d.

Chose to check that the heading is one of the body's own heading lines above
the span, and not to re-derive WHICH heading `as_passage` picks, because the
guarantee is "the bundle's bytes"; the nearest-heading rule is the product's
presentation and a judge that copies it agrees by construction.

M15 (a passage carrying one sentence the file does not) is row 7's fifteenth
mutant. No synthetic concept was long enough to be cut, so a new fixture
delivers one (DELIVERY, ~6 600 characters, set set-passage.json); the corpus
pin moved with it. MUTANT_BAR unchanged, src/ untouched.

Gate, synthetic: rows 1 and 6 13/13 -> 14/14, row 7 14/14 -> 15 of 15 GREEN
(M15 -> row 1, 6), rows 2/3/4 unchanged, GATE RED: rows 5, 8. Row 8 rerun on
the real set: no class e miss remains (its numbers are kept out of this public
history, per the wiki directive). Suite after git add,
FORCE_COLOR unset: 2450 passed, 1 skipped (+6). ruff, format, mypy --strict
clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-22 23:53:09 +02:00
commit cb0c10b421
5 changed files with 199 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

149
tools/okf_retrieval_gate.py
View file

@ -586,6 +586,38 @@ FUSION = BundleSpec(
),
)
_ARCHIVE_LINE = "Styret gjennomgaar notatet og foerer det inn i arkivet. Sekretaeren sender kopi."
#: THE PASSAGE DELIVERY. Every concept above is shorter than
#: `consume.PASSAGE_CHARS`, so no synthetic payload carried a passage and a
#: mutant of one (M15) could not be felled. One concept of about 6 600
#: characters: a heading ten lines in, the answer thirty lines below it, so the
#: delivery is heading + `[...]` + span + `[...]` and the span carries the
#: citation. Rows 1 and 6 read it as a hit only because the judge reads a
#: passage as its exact reconstruction; M15 adds one sentence and fells both.
DELIVERY = BundleSpec(
"retrieval-delivery",
(
DocumentSpec(
"husbok",
"husbok.md",
(
ConceptSpec(
"drift",
"Drift av huset",
"\n".join(
[_ARCHIVE_LINE] * 10
+ ["## Teknisk rom"]
+ [_ARCHIVE_LINE] * 30
+ ["Varmepumpa i kjelleren faar service av roerleggeren hvert aar."]
+ [_ARCHIVE_LINE] * 40
),
),
),
),
),
)
SPECS: Mapping[str, BundleSpec] = {
"positive": POSITIVE,
"miss": MISS,
@ -596,6 +628,7 @@ SPECS: Mapping[str, BundleSpec] = {
"passage": PASSAGE,
"path": PATH,
"fusion": FUSION,
"delivery": DELIVERY,
}
@ -638,7 +671,7 @@ def specs_digest(specs: Mapping[str, BundleSpec] = SPECS) -> str:
#: The synthetic corpus, pinned the way the sets are.
SPECS_SHA256 = "abf24378c32b02753916c36eb502f9b27183a5d1c7e1914b40773bde393878b5"
SPECS_SHA256 = "089a6a9770c03c59fff2da22430e5f9aeb2b2374c49aed419bae99409a2a5cb9"
def synthetic_bundles(root: Path, specs: Mapping[str, BundleSpec] = SPECS) -> dict[str, Path]:
@ -804,6 +837,7 @@ SYNTHETIC_SETS: dict[str, str] = {
"set-quota.json": "61fda652719d7403ddf9701d50e914c46572a4fda77dc0336838695e18c498cf",
"set-controls.json": "c2894656326e5a69ec7063fdc280e763910d20b4cbc2c0124f639283864ce506",
"set-mechanisms.json": "435ce620c30c94b0d151335883583520490482be2ade6797524ae928a9f1eab7",
"set-passage.json": "93b6de8ef7012c84516fe9aee560924d4b579669fb046e6eadc70dbf444eb627",
}
@ -967,6 +1001,79 @@ def _carries(text: str, quote: str) -> bool:
return not quote or _flat(quote) in _flat(text)
#: What class e says when a passage is not its own reconstruction.
PASSAGE_NOT_RECONSTRUCTED = (
"the delivered passage is not its reconstruction from the bundle's bytes"
)
def passage_span(text: str, passage: object, body: str) -> str | None:
"""The span a passage delivery carries, or None when `text` is not EXACTLY
what the bundle's bytes rebuild.
Since v1.1 C1 a long concept is delivered as `[heading]`, `[...]`, the
span, `[...]` (`consume.as_passage`), with `passage = {start, end, of}`.
Two traps PM measured 2026-09-22, both held here:
- **The offsets count in the DELIVERED body** (`ConceptView.body`), never
in the concept file: read in the file they land a frontmatter's length
off, and 0 of 12 real spans matched.
- **"The span occurs somewhere in the text" is not a check.** It accepts
an invented sentence beside the span. Every character of `text` must be
accounted for: the span is `body[start:end]` byte for byte, a marker
stands exactly where the span leaves text out and nowhere else, and the
one other line allowed is a heading line of the body ABOVE the span (or a
prefix of one -- `as_passage` cuts a long heading). WHICH heading is the
product's choice and is not re-derived here; that it is the bundle's
bytes is the guarantee.
The citation is then read in the span alone -- not in the heading, not in
the markers, and not across the seam between them, which is no sequence
the concept file holds.
"""
if not isinstance(passage, Mapping):
return None
start, end, of = passage.get("start"), passage.get("end"), passage.get("of")
if not all(
isinstance(value, int) and not isinstance(value, bool) for value in (start, end, of)
):
return None
assert isinstance(start, int) and isinstance(end, int)
if of != len(body) or not 0 <= start < end <= len(body):
return None
span = body[start:end]
tail = f"\n{consume.PASSAGE_ELISION}" if end < len(body) else ""
if not text.endswith(span + tail):
return None
head = text[: len(text) - len(span + tail)]
if start == 0:
return span if head == "" else None
marker = f"{consume.PASSAGE_ELISION}\n"
if head == marker:
return span
if not head.endswith(f"\n{marker}"):
return None
heading = head[: -len(f"\n{marker}")]
above = body[:start].split("\n")
if heading.startswith("#") and any(
line.startswith("#") and line.startswith(heading) for line in above
):
return span
return None
def _judged(excerpt: Mapping[str, object], body: str) -> tuple[str, bool, str]:
"""(the text a citation is looked for in, whether it is the bundle's
bytes, the detail when it is not) for one delivered excerpt."""
text = str(excerpt.get("text", ""))
if "passage" in excerpt:
span = passage_span(text, excerpt["passage"], body)
if span is None:
return text, False, PASSAGE_NOT_RECONSTRUCTED
return span, True, ""
return text, _flat(text) == _flat(body), "the delivered text is not the bundle's bytes"
def measure_units(bundle: Path, question: Question) -> list[Unit]:
"""One question, measured at the shipped defaults, plus the quota-off run
that says what the truth of a withheld concept is.
@ -1013,18 +1120,21 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
for concept_id in holding
if _carries(index.concepts[concept_id].body, fasit.quote)
)
hit_ids = [
concept_id
# A whole-body delivery is judged whole; a passage is judged as its
# exact reconstruction, and its citation is read in its span alone.
judged = {
concept_id: _judged(delivered[concept_id], index.concepts[concept_id].body)
for concept_id in holding
if concept_id in delivered
and _carries(str(delivered[concept_id].get("text", "")), fasit.quote)
}
hit_ids = [
concept_id for concept_id, (text, _, _) in judged.items() if _carries(text, fasit.quote)
]
# The payload SAYS it delivered this; the bundle says what it is.
confirmed: bool | None = None
not_bytes = ""
for concept_id in hit_ids:
confirmed = _flat(str(delivered[concept_id].get("text", ""))) == _flat(
index.concepts[concept_id].body
)
_, confirmed, not_bytes = judged[concept_id]
if confirmed:
break
# The concept this entry is really about: the one the bundle holds the
@ -1054,7 +1164,7 @@ def measure_units(bundle: Path, question: Question) -> list[Unit]:
else "no concept in the bundle answers to this name"
)
elif hit_ids and not confirmed:
klass, detail = "e", "the delivered text is not the bundle's bytes"
klass, detail = "e", not_bytes
elif target in delivered:
klass = "d"
detail = "delivered without the citation" + (
@ -1845,6 +1955,25 @@ def _extend_delivered() -> contextlib.AbstractContextManager[None]:
return _patched(delivered_text=mutant)
def _extend_passage() -> contextlib.AbstractContextManager[None]:
"""M14's shape for the delivery form v1.1 C1 added: the passage still
CARRIES the citation in its span, and one sentence at the span's end is not
the concept file's. A judge that asked only whether the span occurs
somewhere in the text would count it."""
original = consume.as_passage
def mutant(excerpt: dict[str, object], window: int) -> dict[str, object]:
out = original(excerpt, window)
if "passage" in out:
text = str(out["text"])
tail = f"\n{consume.PASSAGE_ELISION}"
cut = len(text) - len(tail) if text.endswith(tail) else len(text)
out["text"] = f"{text[:cut]} En setning som ikke staar i konseptfila.{text[cut:]}"
return out
return _patched(as_passage=mutant)
def _drop_text_key() -> contextlib.AbstractContextManager[None]:
original = consume.excerpt_for
@ -1895,6 +2024,7 @@ MUTANTS: tuple[Mutant, ...] = (
Mutant(
"M14 the delivered text carries a sentence the concept file does not", 6, _extend_delivered
),
Mutant("M15 the passage carries a sentence the concept file does not", 6, _extend_passage),
)
@ -1921,10 +2051,11 @@ MUTANT_ROSTER: tuple[str, ...] = (
"M12 the delivered text is truncated to 40 characters",
"M13 the excerpt carries no text",
"M14 the delivered text carries a sentence the concept file does not",
"M15 the passage carries a sentence the concept file does not",
)
#: The roster's length, written as a number so appending is not one edit.
MUTANT_COUNT = 14
MUTANT_COUNT = 15
def _score(rows: Sequence[Row]) -> dict[int, int]: