fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table

Two consumer-reported defects, one rebuild.

The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split
`10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question
naming a requirement reached the ranker carrying only the word every concept
in a standards bundle carries. Measured on three real bundles (446, 1133 and
270 concepts), the named requirement was withheld `below_k` in three of three.
Numeric groups joined by `.` or `-` are now held together, dash variants fold
to the ASCII hyphen, and the noise floor is unchanged. The gold moves from
160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is
still `below_k`, because `_overlap` is a count and an exact requirement number
is worth no more than a common verb. That weighting is a separate decision.

The rule was narrowed by a measurement: a version that joined alphanumeric
groups swallowed a document slug whole and cost a hit@8 row. An equality-only
variant was measured on all three bundles and falsified -- better on one,
worse on two.

The orphan gate destroyed a heading's name: a table opening directly below a
heading left that heading with an empty body, the orphan check dropped it, and
the surviving table block kept the mechanical `Tabell linje <n>`. A table that
orphans its heading now takes that heading's title and section number.
Conditioned on the drop, on adjacency, and carrying both members -- each of
the three measured or mutation-tested.

One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43
= N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03
bundle carried, 1106 of 1108 files identical to it. New ref
sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b
-- c26eed6a... is superseded. The regression the previous session measured is
closed: candidate rank 19 -> 10, and the delivering command is now
`--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against
65 912 before. The specific question is unmoved at rank 1.

The tokeniser alone leaves the K2 control question byte-identical, measured
with the bundle held fixed and both published byte counts reproduced.

Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first;
6 mutations, 6 red, one of them only after the survivor was read as code and
a missing fixture was added. Suite 1287 -> 1295.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 11:43:39 +02:00
commit 56c1205ec4
5 changed files with 550 additions and 5 deletions

View file

@ -1187,6 +1187,93 @@ Innledende avsnitt om romskjemaet.
"""
HEADING_ORPHANED_BY_ITS_TABLE = """## Prissammenstilling
| Post | Sum |
|-------|-------|
| A | 100 |
"""
HEADING_WITH_A_BODY_ABOVE_A_TABLE = """## Prissammenstilling
Innledende tekst om skjemaet.
| Post | Sum |
|-------|-------|
| A | 100 |
"""
ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE = """## Tomt kapittel
## Innledning
Tekst i innledningen.
| Post | Sum |
|-------|-------|
| A | 100 |
"""
NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE = """## 3.1 Prissammenstilling
| Post | Sum |
|-------|-------|
| A | 100 |
"""
def test_a_table_that_orphans_its_heading_takes_that_headings_name() -> None:
"""MEASURED, twice, before the rule was written.
A spreadsheet row now renders as pipe rows, so `rule:table-block` opens on
the line after the sheet heading; the heading's own body is then empty and
the orphan check drops it; and the only surviving candidate carries the
mechanical `Tabell linje <n>`. The sheet's NAME is destroyed -- measured on
a 629-concept bundle, the concept fell from candidate rank 10 to 19 for a
question naming its subject, and restoring the title alone put it back at
10. The title is not lost here: it moves to the segment that holds the
heading's content.
"""
candidates = okf_propose_segments.find_candidates(HEADING_ORPHANED_BY_ITS_TABLE)
assert len(candidates) == 1
assert candidates[0].title == "Prissammenstilling"
assert candidates[0].rule == okf_propose_segments.RULE_TABLE_BLOCK
def test_a_table_below_a_heading_that_keeps_its_body_is_not_renamed() -> None:
"""The known-negative, and the reason the rule is conditioned on the drop.
Here the heading survives, so its title is still carried by a live
candidate. Renaming the table to it as well would put the same name on two
concepts and take a name away from neither.
"""
candidates = okf_propose_segments.find_candidates(HEADING_WITH_A_BODY_ABOVE_A_TABLE)
assert [c.title for c in candidates] == ["Prissammenstilling", "Tabell linje 5"]
def test_an_orphaned_headings_section_number_moves_with_its_title() -> None:
"""`_segment_path` reads BOTH: the number becomes the directory and is
stripped from the stem. Inheriting the title alone would emit
`3-1-prissammenstilling.md` at the top level -- a name the heading never
had."""
candidates = okf_propose_segments.find_candidates(NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE)
assert len(candidates) == 1
assert candidates[0].title == "3.1 Prissammenstilling"
assert candidates[0].number == "3.1"
assert okf_propose_segments._segment_path(candidates[0], set()) == "3-1/prissammenstilling.md"
def test_an_orphaned_name_is_not_carried_past_the_candidate_that_follows_it() -> None:
"""The name goes to the ADJACENT segment or nowhere.
Written because a mutation that never cleared the carried name survived the
rest of this band: with the name held indefinitely, the table below would
take the title of a heading it does not contain, several candidates away.
That is an invention, not a rescue.
"""
candidates = okf_propose_segments.find_candidates(ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE)
assert [c.title for c in candidates] == ["Innledning", "Tabell linje 6"]
def test_todays_grid_table_cuts_one_table_into_one_concept_per_row_group() -> None:
"""Characterization. ONE table, three concepts -- the defect Arm E measures.