feat(propose): a grid rule line does not close an open table block

Arm E, off by default. `find_candidates` gains a keyword-only `table_grid`
whose branch is not even evaluated when it is False, so the flag-off path is
byte-identical by construction rather than by test.

The defect it addresses is measured. The converter emits pandoc GRID tables,
whose rows are separated by `+---+---+` rule lines that `_TABLE_ROW` cannot
match, so `in_table` resets between every pair of rows and ONE table becomes
one concept per row group. On the K2 corpus that is 33 of Arm D's 709 entries,
on exactly 3 of the 33 documents that produce a plan -- and those three are the
K3 sample positions 5, 10 and 11, all three rated `too fine`.

Three points where this could have gone silently wrong, and what each cost:

- `Candidate` is frozen and `dataclasses.replace` is not imported, so the join
  is recorded as a `set[int]` over `marked` and applied at the orphan-check
  pass that already rebuilds every candidate. `subdivide` rebuilds them again
  from an explicit keyword list, so `grid` is copied there too -- exactly the
  trap `split` already has.
- `rule_pending` is cleared on the fall-through together with `in_table` and
  `open_block`. A grid table ends with a bottom rule, which sets it; without
  the clear, the NEXT table's first row would be recorded as a join although
  nothing was joined. `test_two_tables_separated_by_a_blank_line_stay_two_concepts`
  is built to catch precisely that: its second table has ONE row group, so it
  cannot be joined, and the test asserts `grid is False` on it. A test that
  checked only candidate counts would stay green through the defect.
- A rule line can never OPEN a block: it is reached only with `in_table` true.
  So no surviving candidate's `start` moves, bodies only grow, and the orphan
  check -- which is monotone in the line set -- cannot drop a candidate it
  previously kept. Asserted as an offset, not a length.

Measured on the real corpus with this code, reproducing a prediction written
down before it was built: 21 -> 6, 15 -> 3, 2 -> 1 entries, and identical at
--outline-run 0 and 3, so Arm D and Arm E do not interact here.

Tests first: 8 red, then green. 1235 -> 1243.
ruff check: exit 0. ruff format --check: exit 0. mypy --strict src/ tools/:
27 files, Success. pytest -q: exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 10:49:52 +02:00
commit 7dae0d1a3a
2 changed files with 158 additions and 1 deletions

View file

@ -1278,3 +1278,119 @@ def test_the_grid_rule_grammar_is_the_class_the_corpus_declared() -> None:
assert grid.match(line) is not None, line
for line in ("+\n", "++\n", "|---|---|\n", "+--- +---+\n", "---+---\n", "+abc+\n"):
assert grid.match(line) is None, line
def test_a_grid_rule_does_not_close_an_open_table_block() -> None:
"""Both halves in ONE test, because either alone is worthless.
A flag-on assertion by itself stays green if the DEFAULT also moved; a
flag-off assertion by itself is true by absence before the rule exists.
Asserting `3` and `1` over the same text is what makes each half depend on
the rule actually running.
"""
off = okf_propose_segments.find_candidates(GRID_TABLE)
assert len(off) == 3
on = okf_propose_segments.find_candidates(GRID_TABLE, table_grid=True)
assert len(on) == 1
assert on[0].title == "Tabell linje 2"
assert on[0].start == 18
assert on[0].end == len(GRID_TABLE)
assert on[0].grid is True
def test_a_leading_grid_rule_does_not_open_a_block() -> None:
"""The rule line suppresses a CLOSE; it never opens.
Asserted as an offset rather than as a length: a span of the right size
starting at 0 would swallow the table's top border and would still have
"the right number of candidates". `18` is the offset of the first `|` row,
unchanged from Arm D, which is what keeps every surviving candidate's
`start` fixed and the orphan check monotone.
"""
on = okf_propose_segments.find_candidates(GRID_TABLE, table_grid=True)
assert on[0].start == 18
assert GRID_TABLE[on[0].start :].startswith("| Navn")
def test_two_tables_separated_by_a_blank_line_stay_two_concepts() -> None:
"""The test that catches a join whose pending state is never cleared.
The second table has exactly ONE row group, so it cannot be joined. If the
bottom rule of the FIRST table leaves "a rule line is pending" set, and the
blank line after it does not clear the flag, the second table's first row
is marked joined although nothing was joined -- and a test that checked
only the candidate COUNT would stay green through it.
"""
off = okf_propose_segments.find_candidates(TWO_TABLES_BLANK_SEPARATED)
assert len(off) == 3
on = okf_propose_segments.find_candidates(TWO_TABLES_BLANK_SEPARATED, table_grid=True)
assert len(on) == 2
assert on[0].grid is True
assert on[1].grid is False
def test_two_tables_separated_by_a_rule_line_alone_are_merged() -> None:
"""A declared limit, measured rather than argued away.
Pandoc puts a blank line between two adjacent tables, so it does not emit
this shape -- but that is a property of the WRITER, not of this code, and
`in_table` survives an arbitrary run of rule lines. The corpus diff is what
tests whether the shape occurs; this test states what happens if it does.
"""
off = okf_propose_segments.find_candidates(TWO_TABLES_RULE_SEPARATED)
assert len(off) == 2
on = okf_propose_segments.find_candidates(TWO_TABLES_RULE_SEPARATED, table_grid=True)
assert len(on) == 1
assert on[0].grid is True
def test_a_pipe_table_is_unchanged_by_the_flag() -> None:
"""A markdown pipe table has no rule line, so Arm E has nothing to say."""
off = okf_propose_segments.find_candidates(PIPE_TABLE)
on = okf_propose_segments.find_candidates(PIPE_TABLE, table_grid=True)
assert len(off) == 1
assert off == on
assert on[0].grid is False
def test_a_rule_line_outside_a_table_changes_nothing() -> None:
"""A `+---+` in prose is not inside a table, so no block is open to keep.
The `len == 1` is the denominator: an equality over two empty lists would
be true and would prove nothing.
"""
off = okf_propose_segments.find_candidates(PROSE_WITH_A_STRAY_RULE)
on = okf_propose_segments.find_candidates(PROSE_WITH_A_STRAY_RULE, table_grid=True)
assert len(off) == 1
assert off == on
def test_the_grid_rule_is_off_at_the_function_level_by_default() -> None:
"""Default and explicit-off agree, AND the count is the pre-change one.
The equality alone would stay green if both had moved together; the
`len == 3` is what makes it a guard. The known-positive control proves the
fixture is not inert.
"""
default = okf_propose_segments.find_candidates(GRID_TABLE)
explicit = okf_propose_segments.find_candidates(GRID_TABLE, table_grid=False)
assert default == explicit
assert len(default) == 3
assert len(okf_propose_segments.find_candidates(GRID_TABLE, table_grid=True)) == 1
def test_every_candidate_present_in_both_arms_keeps_its_start() -> None:
"""Monotonicity: Arm E only ever DELETES marks, so bodies only grow.
This is what makes the orphan check unable to drop a candidate it
previously kept -- the check is monotone in the line set, so a growing body
can only ever become non-empty. Asserted over all three table fixtures,
with the number of common starts printed as the denominator.
"""
for text in (GRID_TABLE, TWO_TABLES_BLANK_SEPARATED, TWO_TABLES_RULE_SEPARATED, PIPE_TABLE):
off = {c.start: c for c in okf_propose_segments.find_candidates(text)}
on = {c.start: c for c in okf_propose_segments.find_candidates(text, table_grid=True)}
common = sorted(set(off) & set(on))
assert common, "no common start offsets -- the comparison would be vacuous"
for start in common:
assert on[start].end >= off[start].end