feat(consume): a table fragment is read under the heading it stands under
C3, the remaining half. The proposer names a table block with no heading of its own after the line it starts on (`Tabell linje <n>`). A position, not a name: searched, it matched no question, and shown, it told a reader nothing about what the table is. `consume.inherit_table_titles` reads such a concept under the title of the nearest concept ABOVE it in the same source document -- in reading order, the heading the table stands under -- ordered by `source_offset`, else `source_lines`, one key per document and never mixed. Applied where the bundle is loaded, so the ranking, the excerpt and the near misses all see the same title; the excerpt keeps the file's own as `own_title`, so the name shown is never mistaken for the one in the file. A table with nothing named above it, or a document whose concepts do not all carry one position key, is left as it is. Chose to do it in the reading and not in the build: no bundle byte moves, no collection has to be rebuilt, and the proposer's goldens stay pinned. `consume.MECHANICAL_TITLE` is held against the proposer's own output by a test. Suite on a clean tree after `git add`: 2418 passed, 2 skipped, 4 xfailed. ruff, ruff format, mypy --strict clean. The search gate's table for the previous commit is kept in local state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
80aac93b8f
commit
f7cd84c5e6
4 changed files with 235 additions and 9 deletions
|
|
@ -1551,6 +1551,11 @@ R761 **8** (S1-S6 + KP + KN), vegnormal **32** questions / **43**
|
|||
bundle once (`bm25.prepare`), cuts each sub-question as alone and
|
||||
interleaves the deliveries round-robin under the same `k`/`limit`; one
|
||||
question is `build_payload`'s bytes. The search gate's (e)/(f) go through it.
|
||||
**C3 title inheritance:** a concept titled `Tabell linje N` (the proposer's
|
||||
mechanical table-block name, `consume.MECHANICAL_TITLE`) is read under the
|
||||
nearest concept above it in its document (`inherit_table_titles`, ordered by
|
||||
`source_offset` else `source_lines`), in ranking, excerpt and near misses;
|
||||
the excerpt carries `own_title`. A reading only -- no bundle bytes move.
|
||||
What follows describes the fusion.
|
||||
- Consume a bundle: `okf consume <bundle> --question "<q>"
|
||||
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
|
||||
|
|
|
|||
|
|
@ -827,6 +827,12 @@ the passage that answers, under the nearest heading above it, marked with
|
|||
they widen a signal (`--cost-vocabulary`, `--rarity-weight`) belong to it and
|
||||
are refused without it. The rest of this section describes the fusion.
|
||||
|
||||
**A table fragment is read under the heading it stands under.** When the
|
||||
builder can only name a table block by the line it starts on (`Tabell linje
|
||||
N`), `okf consume` reads it under the title of the nearest concept above it in
|
||||
the same document — in what it ranks and in what the excerpt shows — and the
|
||||
excerpt keeps the file's own title as `own_title`. No bundle byte moves.
|
||||
|
||||
**The payload says when the bundle looks like it does not cover a question.**
|
||||
`coverage.absent_terms` lists the question's words the bundle holds in no form
|
||||
— not as written and not through a relative it uses — and `coverage.weak` is
|
||||
|
|
|
|||
|
|
@ -354,6 +354,73 @@ class Concept:
|
|||
#: `True` when the concept carries a `parent` key that resolves to no
|
||||
#: concept of its own document -- the third state, as for `sources`.
|
||||
parent_unresolved: bool = False
|
||||
#: The title the FILE carries, set only when `title` was inherited from
|
||||
#: the heading the concept stands under (:func:`inherit_table_titles`).
|
||||
own_title: str | None = None
|
||||
|
||||
|
||||
#: The name the proposer gives a table block that has no heading of its own
|
||||
#: (`propose`, `rule:table-block`): the line the block starts on. A position,
|
||||
#: not a name -- held against the proposer's output by a test, so the two
|
||||
#: cannot drift apart.
|
||||
MECHANICAL_TITLE = re.compile(r"Tabell linje \d+")
|
||||
|
||||
_FIRST_NUMBER = re.compile(r"\s*\[\s*(\d+)")
|
||||
|
||||
#: The locators a concept's place in its document is read off, in order of
|
||||
#: preference. One key per DOCUMENT, never mixed: an offset and a line number
|
||||
#: are not on one scale.
|
||||
_POSITION_KEYS = ("source_offset", "source_lines")
|
||||
|
||||
|
||||
def _position(concept: Concept, key: str) -> int | None:
|
||||
match = _FIRST_NUMBER.match(concept.locators.get(key, ""))
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def inherit_table_titles(concepts: Sequence[Concept]) -> list[Concept]:
|
||||
"""Every concept, a table fragment with no name of its own read under the
|
||||
heading it stands under (v1.1 C3).
|
||||
|
||||
`Tabell linje <n>` is a line number: searched, it matches no question, and
|
||||
shown, it tells a reader nothing about what the table is. So such a concept
|
||||
takes the title of the nearest concept ABOVE it in the same source
|
||||
document -- in reading order, the heading the table stands under -- and
|
||||
keeps its own in `own_title`, so the name shown is never mistaken for the
|
||||
one in the file. A table with nothing named above it keeps its own title,
|
||||
and a document whose concepts do not all carry one position key is left
|
||||
alone rather than ordered by a guess.
|
||||
|
||||
A READING, never a write: no bundle byte moves, and the ranking, the near
|
||||
misses and the excerpt all see the same title because they all read it
|
||||
from here.
|
||||
"""
|
||||
by_source: dict[str, list[int]] = {}
|
||||
for index, concept in enumerate(concepts):
|
||||
if concept.source_file:
|
||||
by_source.setdefault(concept.source_file, []).append(index)
|
||||
out = list(concepts)
|
||||
for indices in by_source.values():
|
||||
if not any(MECHANICAL_TITLE.fullmatch(concepts[index].title) for index in indices):
|
||||
continue
|
||||
key = next(
|
||||
(
|
||||
key
|
||||
for key in _POSITION_KEYS
|
||||
if all(_position(concepts[index], key) is not None for index in indices)
|
||||
),
|
||||
None,
|
||||
)
|
||||
if key is None:
|
||||
continue
|
||||
heading: str | None = None
|
||||
for index in sorted(indices, key=lambda each: (_position(concepts[each], key), each)):
|
||||
concept = concepts[index]
|
||||
if not MECHANICAL_TITLE.fullmatch(concept.title):
|
||||
heading = concept.title
|
||||
elif heading is not None:
|
||||
out[index] = replace(concept, title=heading, own_title=concept.title)
|
||||
return out
|
||||
|
||||
|
||||
#: The key a concept names its enclosing section under, and the key that
|
||||
|
|
@ -1828,6 +1895,8 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
|
|||
"bundle_id_inherited": concept.bundle_id_inherited,
|
||||
"title": concept.title,
|
||||
}
|
||||
if concept.own_title is not None:
|
||||
excerpt["own_title"] = concept.own_title
|
||||
if concept.req_number:
|
||||
excerpt["req_number"] = concept.req_number
|
||||
if concept.sources:
|
||||
|
|
@ -2429,15 +2498,17 @@ def _load(
|
|||
)
|
||||
root_bundle_id = root_bundle_id_of(bundle_root, profile=profile)
|
||||
concept_ids = enumerate_concepts(bundle_root, profile=profile)
|
||||
concepts = link_parents(
|
||||
[
|
||||
read_concept(
|
||||
read_path_in_bundle(bundle_root, f"{concept_id}{profile.paths.concept_suffix}"),
|
||||
bundle_root=bundle_root,
|
||||
root_bundle_id=root_bundle_id,
|
||||
)
|
||||
for concept_id in concept_ids
|
||||
]
|
||||
concepts = inherit_table_titles(
|
||||
link_parents(
|
||||
[
|
||||
read_concept(
|
||||
read_path_in_bundle(bundle_root, f"{concept_id}{profile.paths.concept_suffix}"),
|
||||
bundle_root=bundle_root,
|
||||
root_bundle_id=root_bundle_id,
|
||||
)
|
||||
for concept_id in concept_ids
|
||||
]
|
||||
)
|
||||
)
|
||||
# The text the two lexical signals read, tokenised ONCE: the stem
|
||||
# vocabulary, the rarity `df` and the coverage report below all count over
|
||||
|
|
|
|||
144
tests/test_table_title_inheritance.py
Normal file
144
tests/test_table_title_inheritance.py
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
"""A table fragment with no name of its own takes the heading above it (v1.1 C3).
|
||||
|
||||
The proposer names a table block that has no heading of its own after the
|
||||
line it starts on (`Tabell linje <n>`, `rule:table-block`). That is a
|
||||
position, not a name: searched, it matches no question, and shown, it tells a
|
||||
reader nothing about what the table is. `okf consume` reads such a concept
|
||||
under the heading it stands under -- the nearest concept ABOVE it in the same
|
||||
source document -- both in what is ranked and in what the excerpt shows, and
|
||||
the excerpt keeps the concept's own title beside it as `own_title`, so the
|
||||
name shown is never mistaken for the one in the file.
|
||||
|
||||
Nothing in a bundle moves: the inheritance is a reading, done in `consume`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from llm_ingestion_okf import consume, contract_check
|
||||
from llm_ingestion_okf import skill as okf_skill
|
||||
|
||||
TOOLS = Path(__file__).resolve().parent.parent / "tools"
|
||||
if str(TOOLS) not in sys.path:
|
||||
sys.path.insert(0, str(TOOLS))
|
||||
|
||||
import okf_retrieval_gate as retrieval # noqa: E402
|
||||
|
||||
TABLE = "| Room | Heater |\n| --- | --- |\n| Hall | Panel |\n| Loft | Stove |"
|
||||
|
||||
#: (slug, title, body, first line, last line), in document order.
|
||||
CABIN = (
|
||||
("water", "Water", "The well is drained in autumn.", 1, 10),
|
||||
("tabell-linje-11", "Tabell linje 11", TABLE, 11, 14),
|
||||
("heating", "Heating", "The cabin is kept warm through the winter.", 15, 20),
|
||||
("tabell-linje-21", "Tabell linje 21", TABLE, 21, 24),
|
||||
)
|
||||
#: A document whose FIRST concept is a table: nothing stands above it.
|
||||
LEDGER = (("tabell-linje-1", "Tabell linje 1", TABLE, 1, 4),)
|
||||
|
||||
|
||||
def _write(root: Path) -> Path:
|
||||
spec = retrieval.BundleSpec(
|
||||
"table-titles",
|
||||
tuple(
|
||||
retrieval.DocumentSpec(
|
||||
name,
|
||||
f"{name}.md",
|
||||
tuple(
|
||||
retrieval.ConceptSpec(slug=slug, title=title, body=body)
|
||||
for slug, title, body, _, _ in rows
|
||||
),
|
||||
)
|
||||
for name, rows in (("cabin", CABIN), ("ledger", LEDGER))
|
||||
),
|
||||
)
|
||||
bundle = retrieval.build_bundle(root, spec)
|
||||
for name, rows in (("cabin", CABIN), ("ledger", LEDGER)):
|
||||
for slug, title, body, first, last in rows:
|
||||
path = bundle / name / f"{slug}.md"
|
||||
text = path.read_text(encoding="utf-8")
|
||||
text = text.replace(
|
||||
f"source_file: {name}.md\n",
|
||||
f"source_file: {name}.md\nsource_lines: [{first}, {last}]\n",
|
||||
)
|
||||
if title.startswith("Tabell linje"):
|
||||
# A real table block carries no heading line of its own.
|
||||
text = text.replace(f"## {title}\n\n", "")
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return bundle
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def bundle(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
return _write(tmp_path_factory.mktemp("table-titles") / "bundle")
|
||||
|
||||
|
||||
def _excerpts(bundle: Path, question: str) -> dict[str, dict[str, object]]:
|
||||
payload = consume.build_payload(bundle, question=question, k=8, source_quota=None)
|
||||
excerpts = payload["excerpts"]
|
||||
assert isinstance(excerpts, list)
|
||||
return {str(excerpt["concept_id"]): excerpt for excerpt in excerpts}
|
||||
|
||||
|
||||
def test_a_table_fragment_shows_the_heading_it_stands_under(bundle: Path) -> None:
|
||||
excerpts = _excerpts(bundle, "room heater")
|
||||
heating_table = excerpts["cabin/tabell-linje-21"]
|
||||
water_table = excerpts["cabin/tabell-linje-11"]
|
||||
assert heating_table["title"] == "Heating"
|
||||
assert heating_table["own_title"] == "Tabell linje 21"
|
||||
assert water_table["title"] == "Water"
|
||||
assert water_table["own_title"] == "Tabell linje 11"
|
||||
|
||||
|
||||
def test_a_concept_with_a_name_of_its_own_is_untouched(bundle: Path) -> None:
|
||||
excerpt = _excerpts(bundle, "cabin warm winter")["cabin/heating"]
|
||||
assert excerpt["title"] == "Heating"
|
||||
assert "own_title" not in excerpt
|
||||
|
||||
|
||||
def test_a_table_with_nothing_above_it_keeps_its_own_title(bundle: Path) -> None:
|
||||
excerpt = _excerpts(bundle, "room heater")["ledger/tabell-linje-1"]
|
||||
assert excerpt["title"] == "Tabell linje 1"
|
||||
assert "own_title" not in excerpt
|
||||
|
||||
|
||||
def test_the_inherited_heading_is_searched(bundle: Path) -> None:
|
||||
"""Two identical tables; the question names the heading ONE stands under.
|
||||
|
||||
Without the inheritance they tie and the id decides, which puts the table
|
||||
under `Water` first (`tabell-linje-11` sorts before `tabell-linje-21`).
|
||||
"""
|
||||
payload = consume.build_payload(bundle, question="heating room", k=8, source_quota=None)
|
||||
order = [str(excerpt["concept_id"]) for excerpt in payload["excerpts"]] # type: ignore[union-attr]
|
||||
assert order.index("cabin/tabell-linje-21") < order.index("cabin/tabell-linje-11")
|
||||
|
||||
|
||||
def test_the_near_misses_name_the_inherited_heading(bundle: Path) -> None:
|
||||
payload = consume.build_payload(bundle, question="room heater", k=1, source_quota=None)
|
||||
withheld = payload["withheld"]
|
||||
assert isinstance(withheld, dict)
|
||||
titles = {entry["concept_id"]: entry["title"] for entry in withheld["nearest"]}
|
||||
assert titles.get("cabin/tabell-linje-11", "Water") == "Water"
|
||||
assert titles.get("cabin/tabell-linje-21", "Heating") == "Heating"
|
||||
assert "Tabell linje 11" not in titles.values()
|
||||
assert "Tabell linje 21" not in titles.values()
|
||||
|
||||
|
||||
def test_the_contract_checker_accepts_it(bundle: Path) -> None:
|
||||
payload = consume.build_payload(bundle, question="room heater")
|
||||
assert contract_check.check(okf_skill.render_generic(), payload).findings == ()
|
||||
|
||||
|
||||
def test_the_mechanical_name_is_the_proposers() -> None:
|
||||
"""The pattern read here is the one `propose` writes, held against its output."""
|
||||
from llm_ingestion_okf import propose
|
||||
|
||||
text = "Intro line.\n\n| a | b |\n| --- | --- |\n| 1 | 2 |\n"
|
||||
titles = [candidate.title for candidate in propose.find_candidates(text)]
|
||||
table_titles = [title for title in titles if "linje" in title]
|
||||
assert table_titles, "the premise: the proposer names a table block"
|
||||
assert all(consume.MECHANICAL_TITLE.fullmatch(title) for title in table_titles)
|
||||
Loading…
Add table
Add a link
Reference in a new issue