open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(consume): a short title the question happens to cover passes the section it names

Browse source 
K3-17's known-negative as a mechanism, on a hand-written fixture in an
invented setting. The question names a long section by four of its title
tokens and happens to contain the whole of a neighbour's one-token title.
The fusion reads the long section first; round 16's partition lifts the
covered one-token title over it, because the partition states the covered
title's precision and never compares it with what the title above answers.

Two tests are red on 7cca9e0 and are the defect:
  test_a_covered_short_title_does_not_pass_a_title_answering_more_of_the_question
  test_a_blocked_covered_title_leaves_the_payload_as_the_fusion_built_it
Five are green and guard the repair's shape: the fusion order without the
rule, the group of one, round 16's upside past a narrower neighbour, the
blocker count by EQUALITY (a prefix-only match must not block), and
invariance to words no title holds (why the repair is not a share of the
question).

pytest -q on 7cca9e0 + this file: 2 failed, 1598 passed, 1 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 01:23:14 +02:00
commit 8cc2c54bd0
1 changed files with 218 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

218
tests/test_title_covered_rise.py Normal file
View file

@ -0,0 +1,218 @@
"""A short title the question happens to cover must not pass a title that
answers MORE of the question.
THE DEFECT, in the mechanism rather than in a corpus. Round 16's partition
reads every concept whose WHOLE title the question accounts for before
everything the fusion ranked above it. That is a statement about the covered
title's PRECISION -- it says nothing the question did not ask -- and it
overrode the fusion even where the concept above held a title answering several
of the question's tokens while the covered one answered one. A one-token
generic title that a long question happens to contain took rank 1 from the
section the question names.
The repair keeps the partition and bounds it by the title's RECALL: a covered
concept rises through the fusion's order and stops beneath the first concept
whose title shares MORE question tokens, by equality, than the covered title
holds at all. Where nothing above it does, it rises to the top exactly as in
round 16 -- which is why the rule changes nothing where it never fires and
nothing where no such concept exists.
The fixtures below are that shape and nothing else, written by hand in an
invented setting. They carry no sentence from any corpus and name no real
document.
"""
from __future__ import annotations
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
import okf_consume # noqa: E402
#: The question NAMES the long section by four of its tokens, and happens to
#: contain the whole of the short one's one-token title.
NAMED_QUESTION = "What does the LIX answer for under QX77 through design and handover?"
#: A question whose subject IS a one-token title, beside a narrower section
#: titled with that subject plus a qualifier -- the shape round 16 was built for.
SUBJECT_QUESTION = "What does inspection of bridge structures cover?"
_FRONTMATTER = (
"---\ntype: reference\ntitle: {title}\nsource_file: {slug}.md\n"
"source_sha256: {digest}\ningested_at: 2026-09-01T00:00:00Z\n"
"adjudication: proposed\nbundle_id: rise-fixture\n"
"verified: [{{ by: process:okf-check, at: 2026-09-01T00:00:00Z }}]\n---\n\n"
)
def _bundle(root: Path, sections: list[tuple[str, str, str]]) -> Path:
"""One source document, so the document prior separates nothing."""
(root / "part").mkdir(parents=True)
(root / "index.md").write_text(
"---\nokf_version: 0.2\nbundle_id: rise-fixture\n---\n\n- [part (index)](part/index.md)\n",
encoding="utf-8",
)
entries: list[str] = []
for slug, title, body in sections:
entries.append(f"- [{title}]({slug}.md) — adjudication: proposed\n")
(root / "part" / f"{slug}.md").write_text(
_FRONTMATTER.format(title=title, slug=slug, digest="1" * 64) + f"## {title}\n\n" + body,
encoding="utf-8",
)
(root / "part" / "index.md").write_text("".join(entries), encoding="utf-8")
return root
def _named_bundle(root: Path) -> Path:
"""`aa-lead` is the section the question names; `bb-handover` is a
neighbour whose one-token title the question merely contains."""
sections = [
(
"aa-lead",
"Lead inspector (LIX) per QX77 in design, build and commissioning",
"The LIX answers for inspection under QX77 through design, build and handover.\n" * 3,
),
(
"bb-handover",
"Handover",
"Handover is led by the contractor together with the LIX.\n" * 3,
),
]
sections += [
(f"cc-{n:02d}", f"Site works {n:02d}", "The contractor keeps the site works tidy.\n" * 3)
for n in range(1, 11)
]
return _bundle(root, sections)
def _subject_bundle(root: Path) -> Path:
"""`aa-inspection` is the section the question is about. `bb-simple` is a
narrower one whose title adds a qualifier, and shares ONE more question
token only through the four-character prefix `bridge|work` -- which the
ranking's matcher accepts and the blocker count must not."""
sections = [
(
"aa-inspection",
"Inspection",
"Inspection of a bridge is described here.\n" * 3,
),
(
"bb-simple",
"Simple inspection of bridgework",
"Simple inspection of bridge structures is what this section covers.\n" * 3,
),
]
sections += [
(f"cc-{n:02d}", f"Bridge works {n:02d}", "Works on a bridge are described here.\n" * 3)
for n in range(1, 11)
]
return _bundle(root, sections)
def _concepts(root: Path) -> list[okf_consume.Concept]:
return [
okf_consume.read_concept(
root / f"{concept_id}.md", bundle_root=root, root_bundle_id="rise-fixture"
)
for concept_id in okf_consume.enumerate_concepts(root)
]
def _ranking(root: Path, question: str, **kwargs: object) -> list[str]:
ranked = okf_consume.concept_scores(
_concepts(root),
question,
okf_consume.document_scores(root, question),
**kwargs, # type: ignore[arg-type]
)
return [concept.concept_id for concept, _, _ in ranked]
def _position(order: list[str], slug: str) -> int:
for index, concept_id in enumerate(order, start=1):
if concept_id.endswith(slug):
return index
raise AssertionError(f"{slug} is not in the ranking at all")
def test_the_fusion_reads_the_named_section_first_without_the_rule(tmp_path: Path) -> None:
# CHARACTERISATION: the fusion had it right before any partition ran.
root = _named_bundle(tmp_path / "bundle")
order = _ranking(root, NAMED_QUESTION, title_covered=False)
assert _position(order, "aa-lead") == 1
assert _position(order, "bb-handover") == 2
def test_the_short_title_is_the_only_one_the_question_covers(tmp_path: Path) -> None:
# CHARACTERISATION of the shape: a group of ONE, a one-token title, and a
# title that accounts for one question token of twelve.
root = _named_bundle(tmp_path / "bundle")
hits = okf_consume.title_covered_hits(_concepts(root), NAMED_QUESTION)
assert [concept_id.rsplit("/", 1)[-1] for concept_id in hits] == ["bb-handover"]
assert len(okf_consume.normalise(NAMED_QUESTION)) == 12
def test_a_covered_short_title_does_not_pass_a_title_answering_more_of_the_question(
tmp_path: Path,
) -> None:
"""THE KNOWN-NEGATIVE, as a mechanism. `aa-lead`'s title shares four
question tokens (`lix`, `qx77`, `design`, `and`); `bb-handover`'s whole
title is one. Reading the short one first is wrong: the question names the
long one."""
root = _named_bundle(tmp_path / "bundle")
order = _ranking(root, NAMED_QUESTION)
assert _position(order, "aa-lead") == 1
assert _position(order, "bb-handover") == 2
def test_a_blocked_covered_title_leaves_the_payload_as_the_fusion_built_it(
tmp_path: Path,
) -> None:
root = _named_bundle(tmp_path / "bundle")
assert okf_consume.build_payload(root, question=NAMED_QUESTION) == okf_consume.build_payload(
root, question=NAMED_QUESTION, title_covered=False
)
def test_the_subject_section_still_rises_past_its_narrower_neighbour(tmp_path: Path) -> None:
"""Round 16's upside, restated on a fixture whose narrower neighbour the
fusion puts first -- so this is not green for want of a competitor."""
root = _subject_bundle(tmp_path / "bundle")
assert _position(_ranking(root, SUBJECT_QUESTION, title_covered=False), "bb-simple") == 1
order = _ranking(root, SUBJECT_QUESTION)
assert _position(order, "aa-inspection") == 1
def test_the_blocker_count_reads_the_title_by_EQUALITY_and_not_by_shared_prefix(
tmp_path: Path,
) -> None:
"""`bridgework` shares the four-character word `bridge` with the question,
so the ranking's matcher counts it. Counted that way `bb-simple` would
answer two question tokens to `aa-inspection`'s one and block it. The
control first: the prefix match IS there to be counted."""
root = _subject_bundle(tmp_path / "bundle")
concepts = _concepts(root)
stems = frozenset(
token
for text in okf_consume.searchable_text(concepts)
for token in okf_consume.normalise(text)
)
assert okf_consume.tokens_match("bridge", "bridgework", stems=stems)
assert _position(_ranking(root, SUBJECT_QUESTION, stems=stems), "aa-inspection") == 1
def test_words_no_title_holds_do_not_change_what_the_rule_does(tmp_path: Path) -> None:
"""Why the repair is not a share of the QUESTION. A threshold on
`title tokens / question tokens` admits a one-token title only while the
question has few enough words, so padding a question with words no title
holds would switch the rule off. The comparison between titles cannot see
such words at all."""
root = _subject_bundle(tmp_path / "bundle")
padded = SUBJECT_QUESTION.rstrip("?") + " in practice during every yearly round?"
# The premise, measured: padded, the one-token title is under an eighth of
# the question, below every question-share threshold that kept the upside.
assert 1 / len(okf_consume.normalise(padded)) < 0.125
assert _position(_ranking(root, padded), "aa-inspection") == 1