open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): a grid rule line does not close an open table block

Browse source 
Arm E, off by default. `find_candidates` gains a keyword-only `table_grid`
whose branch is not even evaluated when it is False, so the flag-off path is
byte-identical by construction rather than by test.

The defect it addresses is measured. The converter emits pandoc GRID tables,
whose rows are separated by `+---+---+` rule lines that `_TABLE_ROW` cannot
match, so `in_table` resets between every pair of rows and ONE table becomes
one concept per row group. On the K2 corpus that is 33 of Arm D's 709 entries,
on exactly 3 of the 33 documents that produce a plan -- and those three are the
K3 sample positions 5, 10 and 11, all three rated `too fine`.

Three points where this could have gone silently wrong, and what each cost:

- `Candidate` is frozen and `dataclasses.replace` is not imported, so the join
  is recorded as a `set[int]` over `marked` and applied at the orphan-check
  pass that already rebuilds every candidate. `subdivide` rebuilds them again
  from an explicit keyword list, so `grid` is copied there too -- exactly the
  trap `split` already has.
- `rule_pending` is cleared on the fall-through together with `in_table` and
  `open_block`. A grid table ends with a bottom rule, which sets it; without
  the clear, the NEXT table's first row would be recorded as a join although
  nothing was joined. `test_two_tables_separated_by_a_blank_line_stay_two_concepts`
  is built to catch precisely that: its second table has ONE row group, so it
  cannot be joined, and the test asserts `grid is False` on it. A test that
  checked only candidate counts would stay green through the defect.
- A rule line can never OPEN a block: it is reached only with `in_table` true.
  So no surviving candidate's `start` moves, bodies only grow, and the orphan
  check -- which is monotone in the line set -- cannot drop a candidate it
  previously kept. Asserted as an offset, not a length.

Measured on the real corpus with this code, reproducing a prediction written
down before it was built: 21 -> 6, 15 -> 3, 2 -> 1 entries, and identical at
--outline-run 0 and 3, so Arm D and Arm E do not interact here.

Tests first: 8 red, then green. 1235 -> 1243.
ruff check: exit 0. ruff format --check: exit 0. mypy --strict src/ tools/:
27 files, Success. pytest -q: exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 10:49:52 +02:00
commit 7dae0d1a3a
2 changed files with 158 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

43
src/llm_ingestion_okf/propose.py
View file

@ -228,6 +228,14 @@ class Candidate:
#: Kept on the candidate rather than recomputed at write time so the entry
#: and the reason it exists cannot drift apart.
split: bool = False
#: True when Arm E JOINED this table block across at least one grid-rule
#: line. Same reasoning as `split`, and the same trap: both reconstruction
#: sites below rebuild a `Candidate` from an explicit keyword list, so a
#: field not copied there is silently defaulted back and the entry loses
#: the only trace of why it exists. NOT set merely because a span contains
#: a rule line -- a single-row grid table joins nothing and stays
#: byte-identical to Arm D.
grid: bool = False
def _is_stop_word_only(title: str) -> bool:
@ -295,7 +303,9 @@ def outline_runs(
return [run for run in runs if len(run) >= minimum]
def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
def find_candidates(
text: str, *, outline_run: int = 0, table_grid: bool = False
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
Two gates from Topic 2 are applied here and both REMOVE candidates:
@ -310,6 +320,12 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
exactly as it did before the rule existed. At `N >= 1` the document's own
numbered outline contributes boundaries where the integers sustain an
ascending run of at least `N`.
`table_grid` is Arm E's gate and it is OFF at False, where the branch is not
even evaluated. On, a pandoc grid-table rule line no longer closes an open
table block, so one grid table proposes one candidate instead of one per row
group. It only ever REMOVES marks, which is what keeps every surviving
candidate's `start` fixed and the orphan check monotone.
"""
lines = text.splitlines(keepends=True)
offsets: list[int] = []
@ -340,10 +356,21 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
marked: list[tuple[int, Candidate]] = []
in_table = False
# Arm E's state, and all three of these are cleared together on the
# fall-through below. `rule_pending` is the one that matters: a grid table
# ends with a bottom rule, which sets it, and if the blank line after the
# table did not clear it the NEXT table's first row would be recorded as a
# join although nothing was joined. `joined` holds positions in `marked`
# rather than mutating a candidate, because `Candidate` is frozen and the
# orphan-check pass below already rebuilds every one of them.
rule_pending = False
open_block: int | None = None
joined: set[int] = set()
for index, line in enumerate(lines):
if _TABLE_ROW.match(line):
if not in_table:
in_table = True
open_block = len(marked)
marked.append(
(
index,
@ -357,8 +384,20 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
),
)
)
elif rule_pending and open_block is not None:
joined.add(open_block)
rule_pending = False
continue
if table_grid and in_table and _GRID_RULE.match(line):
# The whole of Arm E: do NOT close the block. Reaching here with
# `in_table` False is impossible by construction, so a rule line can
# never OPEN a block -- a table's top border is not a boundary, and
# every surviving candidate keeps the `start` it had under Arm D.
rule_pending = True
continue
in_table = False
rule_pending = False
open_block = None
outline_title = admitted.get(index)
if outline_title is not None:
@ -427,6 +466,7 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
rule=candidate.rule,
start=candidate.start,
end=end,
grid=position_in_list in joined,
)
)
return candidates
@ -523,6 +563,7 @@ def subdivide(text: str, candidates: list[Candidate], cap: int) -> list[Candidat
start=start,
end=end,
split=True,
grid=candidate.grid,
)
)
return out

116
tests/test_propose_segments.py
View file

@ -1278,3 +1278,119 @@ def test_the_grid_rule_grammar_is_the_class_the_corpus_declared() -> None:
assert grid.match(line) is not None, line
for line in ("+\n", "++\n", "|---|---|\n", "+--- +---+\n", "---+---\n", "+abc+\n"):
assert grid.match(line) is None, line
def test_a_grid_rule_does_not_close_an_open_table_block() -> None:
"""Both halves in ONE test, because either alone is worthless.
A flag-on assertion by itself stays green if the DEFAULT also moved; a
flag-off assertion by itself is true by absence before the rule exists.
Asserting `3` and `1` over the same text is what makes each half depend on
the rule actually running.
"""
off = okf_propose_segments.find_candidates(GRID_TABLE)
assert len(off) == 3
on = okf_propose_segments.find_candidates(GRID_TABLE, table_grid=True)
assert len(on) == 1
assert on[0].title == "Tabell linje 2"
assert on[0].start == 18
assert on[0].end == len(GRID_TABLE)
assert on[0].grid is True
def test_a_leading_grid_rule_does_not_open_a_block() -> None:
"""The rule line suppresses a CLOSE; it never opens.
Asserted as an offset rather than as a length: a span of the right size
starting at 0 would swallow the table's top border and would still have
"the right number of candidates". `18` is the offset of the first `|` row,
unchanged from Arm D, which is what keeps every surviving candidate's
`start` fixed and the orphan check monotone.
"""
on = okf_propose_segments.find_candidates(GRID_TABLE, table_grid=True)
assert on[0].start == 18
assert GRID_TABLE[on[0].start :].startswith("| Navn")
def test_two_tables_separated_by_a_blank_line_stay_two_concepts() -> None:
"""The test that catches a join whose pending state is never cleared.
The second table has exactly ONE row group, so it cannot be joined. If the
bottom rule of the FIRST table leaves "a rule line is pending" set, and the
blank line after it does not clear the flag, the second table's first row
is marked joined although nothing was joined -- and a test that checked
only the candidate COUNT would stay green through it.
"""
off = okf_propose_segments.find_candidates(TWO_TABLES_BLANK_SEPARATED)
assert len(off) == 3
on = okf_propose_segments.find_candidates(TWO_TABLES_BLANK_SEPARATED, table_grid=True)
assert len(on) == 2
assert on[0].grid is True
assert on[1].grid is False
def test_two_tables_separated_by_a_rule_line_alone_are_merged() -> None:
"""A declared limit, measured rather than argued away.
Pandoc puts a blank line between two adjacent tables, so it does not emit
this shape -- but that is a property of the WRITER, not of this code, and
`in_table` survives an arbitrary run of rule lines. The corpus diff is what
tests whether the shape occurs; this test states what happens if it does.
"""
off = okf_propose_segments.find_candidates(TWO_TABLES_RULE_SEPARATED)
assert len(off) == 2
on = okf_propose_segments.find_candidates(TWO_TABLES_RULE_SEPARATED, table_grid=True)
assert len(on) == 1
assert on[0].grid is True
def test_a_pipe_table_is_unchanged_by_the_flag() -> None:
"""A markdown pipe table has no rule line, so Arm E has nothing to say."""
off = okf_propose_segments.find_candidates(PIPE_TABLE)
on = okf_propose_segments.find_candidates(PIPE_TABLE, table_grid=True)
assert len(off) == 1
assert off == on
assert on[0].grid is False
def test_a_rule_line_outside_a_table_changes_nothing() -> None:
"""A `+---+` in prose is not inside a table, so no block is open to keep.
The `len == 1` is the denominator: an equality over two empty lists would
be true and would prove nothing.
"""
off = okf_propose_segments.find_candidates(PROSE_WITH_A_STRAY_RULE)
on = okf_propose_segments.find_candidates(PROSE_WITH_A_STRAY_RULE, table_grid=True)
assert len(off) == 1
assert off == on
def test_the_grid_rule_is_off_at_the_function_level_by_default() -> None:
"""Default and explicit-off agree, AND the count is the pre-change one.
The equality alone would stay green if both had moved together; the
`len == 3` is what makes it a guard. The known-positive control proves the
fixture is not inert.
"""
default = okf_propose_segments.find_candidates(GRID_TABLE)
explicit = okf_propose_segments.find_candidates(GRID_TABLE, table_grid=False)
assert default == explicit
assert len(default) == 3
assert len(okf_propose_segments.find_candidates(GRID_TABLE, table_grid=True)) == 1
def test_every_candidate_present_in_both_arms_keeps_its_start() -> None:
"""Monotonicity: Arm E only ever DELETES marks, so bodies only grow.
This is what makes the orphan check unable to drop a candidate it
previously kept -- the check is monotone in the line set, so a growing body
can only ever become non-empty. Asserted over all three table fixtures,
with the number of common starts printed as the denominator.
"""
for text in (GRID_TABLE, TWO_TABLES_BLANK_SEPARATED, TWO_TABLES_RULE_SEPARATED, PIPE_TABLE):
off = {c.start: c for c in okf_propose_segments.find_candidates(text)}
on = {c.start: c for c in okf_propose_segments.find_candidates(text, table_grid=True)}
common = sorted(set(off) & set(on))
assert common, "no common start offsets -- the comparison would be vacuous"
for start in common:
assert on[start].end >= off[start].end