fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table
Two consumer-reported defects, one rebuild. The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split `10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question naming a requirement reached the ranker carrying only the word every concept in a standards bundle carries. Measured on three real bundles (446, 1133 and 270 concepts), the named requirement was withheld `below_k` in three of three. Numeric groups joined by `.` or `-` are now held together, dash variants fold to the ASCII hyphen, and the noise floor is unchanged. The gold moves from 160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is still `below_k`, because `_overlap` is a count and an exact requirement number is worth no more than a common verb. That weighting is a separate decision. The rule was narrowed by a measurement: a version that joined alphanumeric groups swallowed a document slug whole and cost a hit@8 row. An equality-only variant was measured on all three bundles and falsified -- better on one, worse on two. The orphan gate destroyed a heading's name: a table opening directly below a heading left that heading with an empty body, the orphan check dropped it, and the surviving table block kept the mechanical `Tabell linje <n>`. A table that orphans its heading now takes that heading's title and section number. Conditioned on the drop, on adjacency, and carrying both members -- each of the three measured or mutation-tested. One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43 = N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03 bundle carried, 1106 of 1108 files identical to it. New ref sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b -- c26eed6a... is superseded. The regression the previous session measured is closed: candidate rank 19 -> 10, and the delivering command is now `--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against 65 912 before. The specific question is unmoved at rank 1. The tokeniser alone leaves the K2 control question byte-identical, measured with the bundle held fixed and both published byte counts reproduced. Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first; 6 mutations, 6 red, one of them only after the survivor was read as code and a missing fixture was added. Suite 1287 -> 1295. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
56ae274246
commit
56c1205ec4
5 changed files with 550 additions and 5 deletions
|
|
@ -454,18 +454,42 @@ def find_candidates(
|
|||
)
|
||||
|
||||
candidates: list[Candidate] = []
|
||||
# The name an orphaned heading leaves behind, and the ONE candidate allowed
|
||||
# to pick it up.
|
||||
#
|
||||
# WHY THIS EXISTS. A table opening on the line below a heading gives that
|
||||
# heading an empty body, so the orphan check drops it, and the table block
|
||||
# keeps its mechanical `Tabell linje <n>`: the section's NAME is destroyed
|
||||
# even though its content survives. Measured on a 629-concept bundle after
|
||||
# a spreadsheet began rendering as pipe rows -- the concept fell from
|
||||
# candidate rank 10 to 19 for a question naming its subject, and restoring
|
||||
# the title alone put it back at 10. The name is not invented here, it is
|
||||
# carried across: it moves to the segment that holds the heading's content.
|
||||
#
|
||||
# CONDITIONED ON THE DROP, deliberately. A heading that keeps its own body
|
||||
# is still carried by a live candidate, so copying its title onto the table
|
||||
# as well would put one name on two concepts and rescue none.
|
||||
orphaned_name: tuple[str, str | None] | None = None
|
||||
for position_in_list, (_, candidate) in enumerate(marked):
|
||||
following = marked[position_in_list + 1 :]
|
||||
end = offsets[following[0][0]] if following else end_of_text
|
||||
body = text[candidate.start : end]
|
||||
# The orphan check: everything after the heading line itself.
|
||||
if not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip():
|
||||
orphaned_name = (candidate.title, candidate.number)
|
||||
continue
|
||||
inherited, orphaned_name = orphaned_name, None
|
||||
title, number = candidate.title, candidate.number
|
||||
if inherited is not None and candidate.rule == RULE_TABLE_BLOCK:
|
||||
# Both members, because `_segment_path` reads both: the number
|
||||
# becomes the directory AND is stripped from the stem, so carrying
|
||||
# the title alone would emit a name the heading never had.
|
||||
title, number = inherited
|
||||
candidates.append(
|
||||
Candidate(
|
||||
title=candidate.title,
|
||||
title=title,
|
||||
level=candidate.level,
|
||||
number=candidate.number,
|
||||
number=number,
|
||||
rule=candidate.rule,
|
||||
start=candidate.start,
|
||||
end=end,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue