fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table

Two consumer-reported defects, one rebuild.

The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split
`10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question
naming a requirement reached the ranker carrying only the word every concept
in a standards bundle carries. Measured on three real bundles (446, 1133 and
270 concepts), the named requirement was withheld `below_k` in three of three.
Numeric groups joined by `.` or `-` are now held together, dash variants fold
to the ASCII hyphen, and the noise floor is unchanged. The gold moves from
160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is
still `below_k`, because `_overlap` is a count and an exact requirement number
is worth no more than a common verb. That weighting is a separate decision.

The rule was narrowed by a measurement: a version that joined alphanumeric
groups swallowed a document slug whole and cost a hit@8 row. An equality-only
variant was measured on all three bundles and falsified -- better on one,
worse on two.

The orphan gate destroyed a heading's name: a table opening directly below a
heading left that heading with an empty body, the orphan check dropped it, and
the surviving table block kept the mechanical `Tabell linje <n>`. A table that
orphans its heading now takes that heading's title and section number.
Conditioned on the drop, on adjacency, and carrying both members -- each of
the three measured or mutation-tested.

One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43
= N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03
bundle carried, 1106 of 1108 files identical to it. New ref
sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b
-- c26eed6a... is superseded. The regression the previous session measured is
closed: candidate rank 19 -> 10, and the delivering command is now
`--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against
65 912 before. The specific question is unmoved at rank 1.

The tokeniser alone leaves the K2 control question byte-identical, measured
with the bundle held fixed and both published byte counts reproduced.

Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first;
6 mutations, 6 red, one of them only after the survivor was read as code and
a missing fixture was added. Suite 1287 -> 1295.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 11:43:39 +02:00
commit 56c1205ec4
5 changed files with 550 additions and 5 deletions

View file

@ -454,18 +454,42 @@ def find_candidates(
)
candidates: list[Candidate] = []
# The name an orphaned heading leaves behind, and the ONE candidate allowed
# to pick it up.
#
# WHY THIS EXISTS. A table opening on the line below a heading gives that
# heading an empty body, so the orphan check drops it, and the table block
# keeps its mechanical `Tabell linje <n>`: the section's NAME is destroyed
# even though its content survives. Measured on a 629-concept bundle after
# a spreadsheet began rendering as pipe rows -- the concept fell from
# candidate rank 10 to 19 for a question naming its subject, and restoring
# the title alone put it back at 10. The name is not invented here, it is
# carried across: it moves to the segment that holds the heading's content.
#
# CONDITIONED ON THE DROP, deliberately. A heading that keeps its own body
# is still carried by a live candidate, so copying its title onto the table
# as well would put one name on two concepts and rescue none.
orphaned_name: tuple[str, str | None] | None = None
for position_in_list, (_, candidate) in enumerate(marked):
following = marked[position_in_list + 1 :]
end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end]
# The orphan check: everything after the heading line itself.
if not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip():
orphaned_name = (candidate.title, candidate.number)
continue
inherited, orphaned_name = orphaned_name, None
title, number = candidate.title, candidate.number
if inherited is not None and candidate.rule == RULE_TABLE_BLOCK:
# Both members, because `_segment_path` reads both: the number
# becomes the directory AND is stripped from the stem, so carrying
# the title alone would emit a name the heading never had.
title, number = inherited
candidates.append(
Candidate(
title=candidate.title,
title=title,
level=candidate.level,
number=candidate.number,
number=number,
rule=candidate.rule,
start=candidate.start,
end=end,