fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table
Two consumer-reported defects, one rebuild. The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split `10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question naming a requirement reached the ranker carrying only the word every concept in a standards bundle carries. Measured on three real bundles (446, 1133 and 270 concepts), the named requirement was withheld `below_k` in three of three. Numeric groups joined by `.` or `-` are now held together, dash variants fold to the ASCII hyphen, and the noise floor is unchanged. The gold moves from 160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is still `below_k`, because `_overlap` is a count and an exact requirement number is worth no more than a common verb. That weighting is a separate decision. The rule was narrowed by a measurement: a version that joined alphanumeric groups swallowed a document slug whole and cost a hit@8 row. An equality-only variant was measured on all three bundles and falsified -- better on one, worse on two. The orphan gate destroyed a heading's name: a table opening directly below a heading left that heading with an empty body, the orphan check dropped it, and the surviving table block kept the mechanical `Tabell linje <n>`. A table that orphans its heading now takes that heading's title and section number. Conditioned on the drop, on adjacency, and carrying both members -- each of the three measured or mutation-tested. One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43 = N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03 bundle carried, 1106 of 1108 files identical to it. New ref sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b -- c26eed6a... is superseded. The regression the previous session measured is closed: candidate rank 19 -> 10, and the delivering command is now `--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against 65 912 before. The specific question is unmoved at rank 1. The tokeniser alone leaves the K2 control question byte-identical, measured with the bundle held fixed and both published byte counts reproduced. Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first; 6 mutations, 6 red, one of them only after the survivor was read as code and a missing fixture was added. Suite 1287 -> 1295. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
56ae274246
commit
56c1205ec4
5 changed files with 550 additions and 5 deletions
|
|
@ -1187,6 +1187,93 @@ Innledende avsnitt om romskjemaet.
|
|||
"""
|
||||
|
||||
|
||||
HEADING_ORPHANED_BY_ITS_TABLE = """## Prissammenstilling
|
||||
|
||||
| Post | Sum |
|
||||
|-------|-------|
|
||||
| A | 100 |
|
||||
"""
|
||||
|
||||
HEADING_WITH_A_BODY_ABOVE_A_TABLE = """## Prissammenstilling
|
||||
|
||||
Innledende tekst om skjemaet.
|
||||
|
||||
| Post | Sum |
|
||||
|-------|-------|
|
||||
| A | 100 |
|
||||
"""
|
||||
|
||||
ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE = """## Tomt kapittel
|
||||
## Innledning
|
||||
|
||||
Tekst i innledningen.
|
||||
|
||||
| Post | Sum |
|
||||
|-------|-------|
|
||||
| A | 100 |
|
||||
"""
|
||||
|
||||
NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE = """## 3.1 Prissammenstilling
|
||||
|
||||
| Post | Sum |
|
||||
|-------|-------|
|
||||
| A | 100 |
|
||||
"""
|
||||
|
||||
|
||||
def test_a_table_that_orphans_its_heading_takes_that_headings_name() -> None:
|
||||
"""MEASURED, twice, before the rule was written.
|
||||
|
||||
A spreadsheet row now renders as pipe rows, so `rule:table-block` opens on
|
||||
the line after the sheet heading; the heading's own body is then empty and
|
||||
the orphan check drops it; and the only surviving candidate carries the
|
||||
mechanical `Tabell linje <n>`. The sheet's NAME is destroyed -- measured on
|
||||
a 629-concept bundle, the concept fell from candidate rank 10 to 19 for a
|
||||
question naming its subject, and restoring the title alone put it back at
|
||||
10. The title is not lost here: it moves to the segment that holds the
|
||||
heading's content.
|
||||
"""
|
||||
candidates = okf_propose_segments.find_candidates(HEADING_ORPHANED_BY_ITS_TABLE)
|
||||
assert len(candidates) == 1
|
||||
assert candidates[0].title == "Prissammenstilling"
|
||||
assert candidates[0].rule == okf_propose_segments.RULE_TABLE_BLOCK
|
||||
|
||||
|
||||
def test_a_table_below_a_heading_that_keeps_its_body_is_not_renamed() -> None:
|
||||
"""The known-negative, and the reason the rule is conditioned on the drop.
|
||||
|
||||
Here the heading survives, so its title is still carried by a live
|
||||
candidate. Renaming the table to it as well would put the same name on two
|
||||
concepts and take a name away from neither.
|
||||
"""
|
||||
candidates = okf_propose_segments.find_candidates(HEADING_WITH_A_BODY_ABOVE_A_TABLE)
|
||||
assert [c.title for c in candidates] == ["Prissammenstilling", "Tabell linje 5"]
|
||||
|
||||
|
||||
def test_an_orphaned_headings_section_number_moves_with_its_title() -> None:
|
||||
"""`_segment_path` reads BOTH: the number becomes the directory and is
|
||||
stripped from the stem. Inheriting the title alone would emit
|
||||
`3-1-prissammenstilling.md` at the top level -- a name the heading never
|
||||
had."""
|
||||
candidates = okf_propose_segments.find_candidates(NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE)
|
||||
assert len(candidates) == 1
|
||||
assert candidates[0].title == "3.1 Prissammenstilling"
|
||||
assert candidates[0].number == "3.1"
|
||||
assert okf_propose_segments._segment_path(candidates[0], set()) == "3-1/prissammenstilling.md"
|
||||
|
||||
|
||||
def test_an_orphaned_name_is_not_carried_past_the_candidate_that_follows_it() -> None:
|
||||
"""The name goes to the ADJACENT segment or nowhere.
|
||||
|
||||
Written because a mutation that never cleared the carried name survived the
|
||||
rest of this band: with the name held indefinitely, the table below would
|
||||
take the title of a heading it does not contain, several candidates away.
|
||||
That is an invention, not a rescue.
|
||||
"""
|
||||
candidates = okf_propose_segments.find_candidates(ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE)
|
||||
assert [c.title for c in candidates] == ["Innledning", "Tabell linje 6"]
|
||||
|
||||
|
||||
def test_todays_grid_table_cuts_one_table_into_one_concept_per_row_group() -> None:
|
||||
"""Characterization. ONE table, three concepts -- the defect Arm E measures.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue