fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table

Two consumer-reported defects, one rebuild.

The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split
`10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question
naming a requirement reached the ranker carrying only the word every concept
in a standards bundle carries. Measured on three real bundles (446, 1133 and
270 concepts), the named requirement was withheld `below_k` in three of three.
Numeric groups joined by `.` or `-` are now held together, dash variants fold
to the ASCII hyphen, and the noise floor is unchanged. The gold moves from
160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is
still `below_k`, because `_overlap` is a count and an exact requirement number
is worth no more than a common verb. That weighting is a separate decision.

The rule was narrowed by a measurement: a version that joined alphanumeric
groups swallowed a document slug whole and cost a hit@8 row. An equality-only
variant was measured on all three bundles and falsified -- better on one,
worse on two.

The orphan gate destroyed a heading's name: a table opening directly below a
heading left that heading with an empty body, the orphan check dropped it, and
the surviving table block kept the mechanical `Tabell linje <n>`. A table that
orphans its heading now takes that heading's title and section number.
Conditioned on the drop, on adjacency, and carrying both members -- each of
the three measured or mutation-tested.

One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43
= N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03
bundle carried, 1106 of 1108 files identical to it. New ref
sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b
-- c26eed6a... is superseded. The regression the previous session measured is
closed: candidate rank 19 -> 10, and the delivering command is now
`--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against
65 912 before. The specific question is unmoved at rank 1.

The tokeniser alone leaves the K2 control question byte-identical, measured
with the bundle held fixed and both published byte counts reproduced.

Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first;
6 mutations, 6 red, one of them only after the survivor was read as code and
a missing fixture was added. Suite 1287 -> 1295.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tests/test_okf_consume.py

@@ -378,6 +378,64 @@ def test_normalise_drops_tokens_under_three_characters() -> None:
     assert okf_consume.normalise("er en pris i et skjema") == ("pris", "skjema")
 
 
+def test_normalise_holds_an_identifier_number_as_one_token() -> None:
+    # MEASURED 2026-09-08 over three vegnormal bundles (446, 1133 and 270
+    # concepts): `_TOKEN_SPLIT_RE` shatters `10.2-2` into `10`, `2`, `2` and
+    # `MIN_TOKEN_LENGTH` then drops every piece, so a question naming a
+    # requirement number reaches the ranker carrying only the word `krav` --
+    # which every concept in such a bundle also carries. BOTH mechanisms
+    # participate: the split destroys the number, the floor removes the
+    # remains. The gold requirement was `below_k` in three of three.
+    assert "10.2-2" in okf_consume.normalise("Krav 10.2\u20142")
+    assert okf_consume.normalise("3.3.1\u201413") == ("3.3.1-13",)
+    assert okf_consume.normalise("2.9.2\u201412") == ("2.9.2-12",)
+    assert okf_consume.normalise("R610.4") == ("r610.4",)
+    assert okf_consume.normalise("4.2.1") == ("4.2.1",)
+
+
+def test_an_identifiers_three_spellings_of_its_separator_normalise_alike() -> None:
+    # One requirement number arrives as an em dash from the source viewer, an
+    # en dash from a converter and a plain hyphen from a person typing the
+    # question. NFC folds NONE of the three, so a rule that does not fold them
+    # finds the number only in the spelling it was asked with.
+    hyphen = okf_consume.normalise("10.2-2")
+    assert okf_consume.normalise("10.2\u20142") == hyphen
+    assert okf_consume.normalise("10.2\u20132") == hyphen
+    assert hyphen == ("10.2-2",)
+
+
+def test_the_identifier_rule_leaves_the_noise_floor_it_was_added_under() -> None:
+    # The known-negative, and the whole reason `MIN_TOKEN_LENGTH` exists: a
+    # bare short number matches every page number, row count and year in a
+    # corpus, and a matcher that scores them ranks every document equally.
+    assert okf_consume.normalise("10") == ()
+    assert okf_consume.normalise("2") == ()
+    assert okf_consume.normalise("er en pris i et skjema") == ("pris", "skjema")
+    # A hyphenated WORD is not an identifier -- no digit stands on either side
+    # of the separator -- so it splits exactly as it always did.
+    assert okf_consume.normalise("skole-anbudet") == ("skole", "anbudet")
+    # And a separator this rule does not claim leaves its token set untouched:
+    # the K2 corpus spells standards this way.
+    assert okf_consume.normalise("NS3935:2019") == ("ns3935", "2019")
+    assert okf_consume.normalise("TEK 17") == ("tek",)
+
+
+def test_an_identifier_inside_a_slug_does_not_swallow_the_words_around_it() -> None:
+    # MEASURED, and the reason the rule joins DIGIT groups rather than
+    # alphanumeric ones. A first version joined alphanumeric groups across a
+    # separator; a corpus document's slug then became ONE token, because a
+    # `3-6` sits inside it, and that document's score for a question naming its
+    # subject fell from 0.735 to 0.0 -- one hit@8 row lost, on a question
+    # carrying no identifier at all. The slug below has that shape and is not
+    # the corpus's (SS "consumer content stays at form level"). The identifier
+    # is ADDED here; nothing is taken away.
+    tokens = okf_consume.normalise("rapport-iv-vedlegg-3-6-grunnforhold-akustikk")
+    assert "akustikk" in tokens
+    assert "grunnforhold" in tokens
+    assert "vedlegg" in tokens
+    assert "3-6" in tokens
+
+
 def test_two_tokens_match_on_a_shared_prefix_of_four_and_not_of_three() -> None:
     # "Stem-substring" is not an implementable rule: neither `prisene` nor
     # `prissammenstilling` contains the other. Shared prefix does the work --
@@ -1057,6 +1115,7 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
         PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md",
         PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md",
         PROJECT_ROOT / "docs" / "2026-09-08-prisform-og-loggen-k2.md",
+        PROJECT_ROOT / "docs" / "2026-09-08-kravnummer-tokenisering.md",
         PROJECT_ROOT / "README.md",
         PROJECT_ROOT / "CLAUDE.md",
     ]

tests/test_propose_segments.py

@@ -1187,6 +1187,93 @@ Innledende avsnitt om romskjemaet.
 """
 
 
+HEADING_ORPHANED_BY_ITS_TABLE = """## Prissammenstilling
+
+| Post  | Sum   |
+|-------|-------|
+| A     | 100   |
+"""
+
+HEADING_WITH_A_BODY_ABOVE_A_TABLE = """## Prissammenstilling
+
+Innledende tekst om skjemaet.
+
+| Post  | Sum   |
+|-------|-------|
+| A     | 100   |
+"""
+
+ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE = """## Tomt kapittel
+## Innledning
+
+Tekst i innledningen.
+
+| Post  | Sum   |
+|-------|-------|
+| A     | 100   |
+"""
+
+NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE = """## 3.1 Prissammenstilling
+
+| Post  | Sum   |
+|-------|-------|
+| A     | 100   |
+"""
+
+
+def test_a_table_that_orphans_its_heading_takes_that_headings_name() -> None:
+    """MEASURED, twice, before the rule was written.
+
+    A spreadsheet row now renders as pipe rows, so `rule:table-block` opens on
+    the line after the sheet heading; the heading's own body is then empty and
+    the orphan check drops it; and the only surviving candidate carries the
+    mechanical `Tabell linje <n>`. The sheet's NAME is destroyed -- measured on
+    a 629-concept bundle, the concept fell from candidate rank 10 to 19 for a
+    question naming its subject, and restoring the title alone put it back at
+    10. The title is not lost here: it moves to the segment that holds the
+    heading's content.
+    """
+    candidates = okf_propose_segments.find_candidates(HEADING_ORPHANED_BY_ITS_TABLE)
+    assert len(candidates) == 1
+    assert candidates[0].title == "Prissammenstilling"
+    assert candidates[0].rule == okf_propose_segments.RULE_TABLE_BLOCK
+
+
+def test_a_table_below_a_heading_that_keeps_its_body_is_not_renamed() -> None:
+    """The known-negative, and the reason the rule is conditioned on the drop.
+
+    Here the heading survives, so its title is still carried by a live
+    candidate. Renaming the table to it as well would put the same name on two
+    concepts and take a name away from neither.
+    """
+    candidates = okf_propose_segments.find_candidates(HEADING_WITH_A_BODY_ABOVE_A_TABLE)
+    assert [c.title for c in candidates] == ["Prissammenstilling", "Tabell linje 5"]
+
+
+def test_an_orphaned_headings_section_number_moves_with_its_title() -> None:
+    """`_segment_path` reads BOTH: the number becomes the directory and is
+    stripped from the stem. Inheriting the title alone would emit
+    `3-1-prissammenstilling.md` at the top level -- a name the heading never
+    had."""
+    candidates = okf_propose_segments.find_candidates(NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE)
+    assert len(candidates) == 1
+    assert candidates[0].title == "3.1 Prissammenstilling"
+    assert candidates[0].number == "3.1"
+    assert okf_propose_segments._segment_path(candidates[0], set()) == "3-1/prissammenstilling.md"
+
+
+def test_an_orphaned_name_is_not_carried_past_the_candidate_that_follows_it() -> None:
+    """The name goes to the ADJACENT segment or nowhere.
+
+    Written because a mutation that never cleared the carried name survived the
+    rest of this band: with the name held indefinitely, the table below would
+    take the title of a heading it does not contain, several candidates away.
+    That is an invention, not a rescue.
+    """
+    candidates = okf_propose_segments.find_candidates(ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE)
+    assert [c.title for c in candidates] == ["Innledning", "Tabell linje 6"]
+
+
 def test_todays_grid_table_cuts_one_table_into_one_concept_per_row_group() -> None:
     """Characterization. ONE table, three concepts -- the defect Arm E measures.