open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): a grid rule line does not close an open table block

Browse source 
Arm E, off by default. `find_candidates` gains a keyword-only `table_grid`
whose branch is not even evaluated when it is False, so the flag-off path is
byte-identical by construction rather than by test.

The defect it addresses is measured. The converter emits pandoc GRID tables,
whose rows are separated by `+---+---+` rule lines that `_TABLE_ROW` cannot
match, so `in_table` resets between every pair of rows and ONE table becomes
one concept per row group. On the K2 corpus that is 33 of Arm D's 709 entries,
on exactly 3 of the 33 documents that produce a plan -- and those three are the
K3 sample positions 5, 10 and 11, all three rated `too fine`.

Three points where this could have gone silently wrong, and what each cost:

- `Candidate` is frozen and `dataclasses.replace` is not imported, so the join
  is recorded as a `set[int]` over `marked` and applied at the orphan-check
  pass that already rebuilds every candidate. `subdivide` rebuilds them again
  from an explicit keyword list, so `grid` is copied there too -- exactly the
  trap `split` already has.
- `rule_pending` is cleared on the fall-through together with `in_table` and
  `open_block`. A grid table ends with a bottom rule, which sets it; without
  the clear, the NEXT table's first row would be recorded as a join although
  nothing was joined. `test_two_tables_separated_by_a_blank_line_stay_two_concepts`
  is built to catch precisely that: its second table has ONE row group, so it
  cannot be joined, and the test asserts `grid is False` on it. A test that
  checked only candidate counts would stay green through the defect.
- A rule line can never OPEN a block: it is reached only with `in_table` true.
  So no surviving candidate's `start` moves, bodies only grow, and the orphan
  check -- which is monotone in the line set -- cannot drop a candidate it
  previously kept. Asserted as an offset, not a length.

Measured on the real corpus with this code, reproducing a prediction written
down before it was built: 21 -> 6, 15 -> 3, 2 -> 1 entries, and identical at
--outline-run 0 and 3, so Arm D and Arm E do not interact here.

Tests first: 8 red, then green. 1235 -> 1243.
ruff check: exit 0. ruff format --check: exit 0. mypy --strict src/ tools/:
27 files, Success. pytest -q: exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 10:49:52 +02:00
commit 7dae0d1a3a
2 changed files with 158 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

43
src/llm_ingestion_okf/propose.py
View file

@ -228,6 +228,14 @@ class Candidate:
#: Kept on the candidate rather than recomputed at write time so the entry
#: and the reason it exists cannot drift apart.
split: bool = False
#: True when Arm E JOINED this table block across at least one grid-rule
#: line. Same reasoning as `split`, and the same trap: both reconstruction
#: sites below rebuild a `Candidate` from an explicit keyword list, so a
#: field not copied there is silently defaulted back and the entry loses
#: the only trace of why it exists. NOT set merely because a span contains
#: a rule line -- a single-row grid table joins nothing and stays
#: byte-identical to Arm D.
grid: bool = False
def _is_stop_word_only(title: str) -> bool:
@ -295,7 +303,9 @@ def outline_runs(
return [run for run in runs if len(run) >= minimum]
def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
def find_candidates(
text: str, *, outline_run: int = 0, table_grid: bool = False
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
Two gates from Topic 2 are applied here and both REMOVE candidates:
@ -310,6 +320,12 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
exactly as it did before the rule existed. At `N >= 1` the document's own
numbered outline contributes boundaries where the integers sustain an
ascending run of at least `N`.
`table_grid` is Arm E's gate and it is OFF at False, where the branch is not
even evaluated. On, a pandoc grid-table rule line no longer closes an open
table block, so one grid table proposes one candidate instead of one per row
group. It only ever REMOVES marks, which is what keeps every surviving
candidate's `start` fixed and the orphan check monotone.
"""
lines = text.splitlines(keepends=True)
offsets: list[int] = []
@ -340,10 +356,21 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
marked: list[tuple[int, Candidate]] = []
in_table = False
# Arm E's state, and all three of these are cleared together on the
# fall-through below. `rule_pending` is the one that matters: a grid table
# ends with a bottom rule, which sets it, and if the blank line after the
# table did not clear it the NEXT table's first row would be recorded as a
# join although nothing was joined. `joined` holds positions in `marked`
# rather than mutating a candidate, because `Candidate` is frozen and the
# orphan-check pass below already rebuilds every one of them.
rule_pending = False
open_block: int | None = None
joined: set[int] = set()
for index, line in enumerate(lines):
if _TABLE_ROW.match(line):
if not in_table:
in_table = True
open_block = len(marked)
marked.append(
(
index,
@ -357,8 +384,20 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
),
)
)
elif rule_pending and open_block is not None:
joined.add(open_block)
rule_pending = False
continue
if table_grid and in_table and _GRID_RULE.match(line):
# The whole of Arm E: do NOT close the block. Reaching here with
# `in_table` False is impossible by construction, so a rule line can
# never OPEN a block -- a table's top border is not a boundary, and
# every surviving candidate keeps the `start` it had under Arm D.
rule_pending = True
continue
in_table = False
rule_pending = False
open_block = None
outline_title = admitted.get(index)
if outline_title is not None:
@ -427,6 +466,7 @@ def find_candidates(text: str, *, outline_run: int = 0) -> list[Candidate]:
rule=candidate.rule,
start=candidate.start,
end=end,
grid=position_in_list in joined,
)
)
return candidates
@ -523,6 +563,7 @@ def subdivide(text: str, candidates: list[Candidate], cap: int) -> list[Candidat
start=start,
end=end,
split=True,
grid=candidate.grid,
)
)
return out