open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table

Browse source 
Two consumer-reported defects, one rebuild.

The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split
`10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question
naming a requirement reached the ranker carrying only the word every concept
in a standards bundle carries. Measured on three real bundles (446, 1133 and
270 concepts), the named requirement was withheld `below_k` in three of three.
Numeric groups joined by `.` or `-` are now held together, dash variants fold
to the ASCII hyphen, and the noise floor is unchanged. The gold moves from
160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is
still `below_k`, because `_overlap` is a count and an exact requirement number
is worth no more than a common verb. That weighting is a separate decision.

The rule was narrowed by a measurement: a version that joined alphanumeric
groups swallowed a document slug whole and cost a hit@8 row. An equality-only
variant was measured on all three bundles and falsified -- better on one,
worse on two.

The orphan gate destroyed a heading's name: a table opening directly below a
heading left that heading with an empty body, the orphan check dropped it, and
the surviving table block kept the mechanical `Tabell linje <n>`. A table that
orphans its heading now takes that heading's title and section number.
Conditioned on the drop, on adjacency, and carrying both members -- each of
the three measured or mutation-tested.

One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43
= N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03
bundle carried, 1106 of 1108 files identical to it. New ref
sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b
-- c26eed6a... is superseded. The regression the previous session measured is
closed: candidate rank 19 -> 10, and the delivering command is now
`--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against
65 912 before. The specific question is unmoved at rank 1.

The tokeniser alone leaves the K2 control question byte-identical, measured
with the bundle held fixed and both published byte counts reproduced.

Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first;
6 mutations, 6 red, one of them only after the survivor was read as code and
a missing fixture was added. Suite 1287 -> 1295.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 11:43:39 +02:00
commit 56c1205ec4
5 changed files with 550 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

336
docs/2026-09-08-kravnummer-tokenisering.md Normal file
View file

@ -0,0 +1,336 @@
# The requirement number the pre-pass could not see, and the title the orphan gate destroyed, 2026-09-08
Two consumer-reported defects, one session, one rebuild. The first is a
consumption-side defect reported by `vegnormal-okf`: `okf_consume`'s tokeniser
shatters a requirement number, so the one concept an engineer names is withheld.
The second is a production-side defect this repository measured on itself in the
previous session: a table that opens directly under a heading destroys that
heading's name.
---
## 0. What IS measured, and what is NOT
**Measured.** The five tokeniser strings the consumer published, reproduced
here before anything was changed; where the number is lost, decomposed to both
mechanisms; the gold concept's rank on three real bundles, before and after, with
denominators; one alternative rule falsified against the same three bundles; the
K2 control question as a payload digest with the tokeniser as the only variable;
the title axis reproduced from the previous session's number before it was
touched; a full K2 rebuild with the conservation identity, the concept count, the
number of ids that moved against a denominator of 629, and the new content
identity; and six mutations against the new tests.
**Known-positive, run first.** Nine figures published elsewhere were reproduced
on this machine before any of the after-columns were read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | `normalise('Krav 10.2—2')` → `['krav']` | consumer § 3.1 | `('krav',)` |
| 2 | `normalise('10.2-2')` → `[]` | consumer § 3.1 | `()` |
| 3 | `normalise('3.3.1—13')` → `[]` | consumer § 3.1 | `()` |
| 4 | `'N500'` → `['n500']`, `'abc123'` → `['abc123']` | consumer § 3.1 | identical |
| 5 | considered / withheld / delivered per bundle | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
| 6 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
| 7 | payload tokens (o200k) per bundle | consumer § 3 | 22 138, 53 444, 14 694 |
| 8 | priced concept candidate rank 10 → 19 | this repo, 2026-09-08 (O1) § 7 | 10 → 19, and 251 / 1 / 1 for the other three rows |
| 9 | K2 payload bytes, two commands | this repo, O1 § 7 | 169 573 B and 172 588 B |
**NOT measured.** Whether a live model answers *better* with the named
requirement in the payload. This document measures a ranking; the consumer
measures the answer. Also not measured: that either rule helps a corpus other
than the four read here.
**Not closed.** The gold requirement is still `below_k` on all three bundles
after the fix. § 8 states why, with the number.
---
## 1. The five strings, and one correction to how they are read
```
PYTHONPATH=… python3 -c "from okf_consume import normalise; …"
'Krav 10.2—2' -> ('krav',) 'N500' -> ('n500',)
'10.2-2' -> () 'abc123' -> ('abc123',)
'3.3.1—13' -> () '10' -> ()
'2.9.2—12' -> () '2' -> ()
```
All five reproduce, and so do the two known-positives. So does the gold's place
in the `withheld` list: 119 of 438, 204 of 1125, 66 of 262.
**That place is a list position and not a rank, and the difference matters.**
The `withheld` array is emitted in `concept_id` order — measured, not assumed
(`sorted(ids) == ids` is `True` on all three payloads) — so it cannot move when
the ranking moves. It is a correct number that answers a different question. The
instrument that does move is the fused rank, and § 4 uses it.
One more figure the consumer's mechanism claim predicts, confirmed here: of the
262 withheld entries on the smallest bundle, **262 are `below_k` and none is
`no_lexical_match`**. Every concept matches, because every concept carries the
word the question carries.
---
## 2. Where the number is lost: both mechanisms, not one
The order asked which of the two is responsible. **Both**, and neither alone
would lose it:
```
'10.2-2' split-> ['10', '2', '2'] kept-> []
'3.3.1—13' split-> ['3', '3', '1', '13'] kept-> []
'R610.4' split-> ['r610', '4'] kept-> ['r610']
```
`_TOKEN_SPLIT_RE` (`tools/okf_consume.py:523` before this change) splits on
every non-alphanumeric character, which destroys the identifier; the
`MIN_TOKEN_LENGTH` filter in `normalise` (line 537) then removes the digit runs
that are left. Lowering the floor alone would not recover the number — it would
return `10`, `2` and `2`, which match every page number in the corpus. The
identifier has to survive the split.
---
## 3. The rule, and the alternative that was falsified
**The rule.** Numeric groups joined by `.` or `-` are held together as one
token, with a leading letter run allowed only where it touches its digits
without a separator. Dash variants fold to the ASCII hyphen first, so the same
number written with an em dash, an en dash or a hyphen is one token.
`MIN_TOKEN_LENGTH` is unchanged; the identifier is exempt from it by
construction rather than by lowering it for everyone.
**It was narrowed by a measurement, not by taste.** The first version joined
*alphanumeric* groups across a separator. That version swallowed a whole
document slug — a slug of the form `…-3-6-…` became ONE token — and that
document's stage-one score for a question naming its subject fell from **0.735
to 0.0**, costing one row of the repository's own hit@8 measurement. The
narrowed rule is purely additive on such a slug: the identifier is added, the
words stay.
**The falsified alternative.** Because `tokens_match` compares a four-character
prefix, `10.2-2` matches `10.2-5`: on the smallest bundle exactly **5 of 270**
concept titles match the question's identifier, and all five are the same
section. Making identifier tokens match only on *equality* is the obvious
narrowing, and it was measured before being adopted or rejected:
| | prefix (shipped) | equality (probe) |
|---|---|---|
| N100:2023 | 96 | **53** |
| N200:2024 | **9** | 26 |
| N500:2024 | **35** | 37 |
It improves one bundle and makes two worse. **Falsified; not built.**
---
## 4. The three bundles, before and after
Command, one per bundle, the questions quoted verbatim from the consumer's § 3:
```
python3 tools/okf_consume.py <bundle> \
--question "Hva krever Krav 10.2—2 i N500? Gjengi det sentrale vilkåret."
```
| | N100:2023 | N200:2024 | N500:2024 |
|---|---|---|---|
| concepts | 446 | 1 133 | 270 |
| gold fused rank, **before** | 160 | 143 | 100 |
| gold fused rank, **after** | **96** | **9** | **35** |
| gold lexical overlap, before → after | 3 → 4 | 4 → 6 | 3 → 4 |
| considered / withheld / delivered | 446/438/8 | 1133/1125/8 | 270/262/8 |
| budget spent, before → after | 8 977 → 12 095 | 17 818 → 17 818 | 10 517 → 10 517 |
| payload o200k, before → after | 22 138 → 23 583 | 53 444 → 53 444 | 14 694 → 14 694 |
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
| **gold delivered?** | **no, `below_k`** | **no, `below_k`** | **no, `below_k`** |
Two of the three payloads are byte-identical before and after: the gold moved a
long way without reaching the top eight, so the delivered set did not change.
**The bonus measurement the order asked for.** Three spellings of the same
requirement number, on the same bundle:
| question spelling | identifier token | gold rank |
|---|---|---|
| `Krav 10.2—2` (em dash) | `10.2-2` | 35 |
| `krav 10.2-2` (hyphen, lower case) | `10.2-2` | 35 |
| `Krav 10.2–2` (en dash) | `10.2-2` | 35 |
All three normalise alike and rank alike.
---
## 5. The K2 control: the tokeniser alone changes nothing
The order required the S7 control question to be compared byte for byte with
the tokeniser as the only variable, so it was run against ONE unchanged bundle
with the committed tool and the new tool:
| command | before | after |
|---|---|---|
| default | `sha256 2ae46f683a3b17fb…`, 169 573 B | **identical** |
| `--cost-vocabulary --k 12 --limit 160000` | `sha256 8aaf8558369b3d6b…`, 172 588 B | **identical** |
Byte for byte, both commands. The two byte counts are the ones published in the
previous session's report, which is what makes this a control and not a
coincidence. The corpus does carry identifiers of other shapes (`NS3935:2019`,
`TEK 17`); measured, the rule leaves both untouched, because a colon is not a
separator it claims and a space was never one.
---
## 6. The title axis in the orphan gate
Reproduced first, from the previous session's numbers, before the rule was
touched — all four rows to the integer:
| question | vocabulary | O1 measured | reproduced here |
|---|---|---|---|
| mandate-shaped | off | 251 of 271 | 251 |
| mandate-shaped | on | **19** of 280 | **19** |
| specific | off | 1 of 233 | 1 |
| specific | on | 1 of 248 | 1 |
**The rule.** When a heading is dropped by the orphan check and the candidate
immediately after it is a table block, the table block takes that heading's
title *and* its section number instead of the mechanical `Tabell linje <n>`.
Three properties of that sentence are each a decision:
1. **Conditioned on the drop.** A heading that keeps its own body is still
carried by a live candidate; copying its title onto the table as well would
put one name on two concepts and rescue none.
2. **Immediately after.** The name goes to the adjacent segment or nowhere.
Carried further, a table several candidates away would take the name of a
heading it does not contain — an invention rather than a rescue. This is the
one property no test caught until a surviving mutation exposed it (§ 9).
3. **Number as well as title.** `_segment_path` reads both — the number becomes
the directory and is stripped from the stem — so inheriting the title alone
would emit `3-1-prissammenstilling.md` at the top level, a name the heading
never had.
It is format-agnostic by construction: the orphan check and the table rule are,
so a sheet name in a spreadsheet and a heading in a PDF go the same way.
---
## 7. The rebuild: 629 concepts, two ids, a new identity
One rebuild for both fixes, `okf build` over the same 43-document folder, run
from a frozen copy of `src/` so no later edit could reach a measurement already
in flight.
| | value |
|---|---|
| `merged + coded rejections == N` | `39 + 4 = 43 = N`, exit 0 |
| concepts | **629**, unchanged |
| concept ids that moved | **2 of 629** (0.32 %) |
| new content identity | `sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b` |
**The two ids move BACK.** `…/tabell-linje-3` becomes `…/prissammenstilling-sheet-1`
and `…/ark1-sheet-1` again — the names they carried in the bundle delivered on
2026-09-03. Against that delivered bundle, **1 106 of 1 108 files are now
identical**, and the two that differ are the two spreadsheet bodies the previous
session deliberately changed. A consumer holding the 2026-09-03 tree gets its
file names back; the ref, however, is new, and `c26eed6a…` from the previous
session is superseded and stale.
**The ranking, on the rebuilt bundle:**
| question | vocabulary | O1 bundle | rebuilt |
|---|---|---|---|
| mandate-shaped | off | 251 of 271 | 251 |
| mandate-shaped | on | 19 of 280 | **10** |
| specific | off | 1 of 233 | 1 |
| specific | on | 1 of 248 | **1** |
Rank 10 is exactly where the concept sat before the previous session's form fix.
The regression is closed, and the question this repository's own hit@8
measurement uses is untouched.
**What it now costs to deliver that concept**, which is the figure a consumer
budgets against:
| command | bytes | o200k | priced sheet |
|---|---|---|---|
| default | 169 583 | 53 420 | `below_k` |
| `--cost-vocabulary --k 12` (default budget) | 183 798 | **58 907** | **delivered, rank 10** |
| `--cost-vocabulary --k 20` (default budget) | 190 438 | 61 955 | delivered, rank 10 |
| specific question, default | 108 877 | 37 189 | delivered, rank 1 |
The previous session's delivering command was `--cost-vocabulary --k 20` at
**65 912** o200k. The delivering command is now `--cost-vocabulary --k 12` at
**58 907** — 10.6 % cheaper, and it fits the default budget. `okf_contract_check`
returns exit 0 with 14 rules and 0 findings on all three payloads, and the
identity closes: `8 + 621 = 629`, `12 + 617 = 629`.
---
## 8. What did NOT close, and why — stated rather than glossed
**The requirement question is still `below_k` on all three bundles.** The fix
moved the gold from 160 to 96, 143 to 9 and 100 to 35, which is a large move and
not a delivery. The remaining blocker was decomposed rather than guessed:
`_overlap` returns a COUNT. On the smallest bundle the gold's overlap is 4 out
of a possible 7, and **70 of 270 concepts also score 4** while 12 score 5.
Matching the exact requirement number is worth exactly as much as matching the
verb in `Hva **krever** …` or the noun in `… det sentrale **vilkåret**`. The
identifier is now visible to the ranker; it is not yet *worth* more than a
common word.
Making it worth more is a weighting decision — a rarity weight, or a declared
class of high-specificity tokens — and it is a different change from this one:
it moves every ranking in every corpus, and it has no measurement behind it yet.
It is not in this order and is not decided here.
**And the honesty limit that governs all of the above.** This document measures
a *ranking*: which concepts a deterministic pre-pass puts in a payload. Whether
a live model then answers the engineer's question correctly is the consumer's
measurement, not this one. "Delivered" is not "read".
---
## 9. Tests, and the mutation that survived
Eight new tests, each written red first (1 287 -> 1 295). Six
mutations, six red, with the unmutated control green after every restore:
| mutation | what it breaks | result |
|---|---|---|
| widen the identifier back to alphanumeric groups | the slug regression | 3 red |
| drop the dash fold | one spelling finds, two do not | 2 red |
| never emit the identifier token | the whole rule | 3 red |
| inherit the title but not the number | the emitted path | 1 red |
| inherit onto any rule, not only a table | one name on two concepts | 1 red |
| never clear the carried name | a distant table takes the name | **survived** |
The survivor was read as code before it was called equivalent, per this
repository's own trap note: it is not equivalent — it lets a heading's name
travel arbitrarily far — and the band simply had no fixture where an orphan is
followed by something other than a table. One was added, and the mutation now
dies.
Suite: **1 295 passing** (was 1 287), `ruff` clean, `mypy` clean on 17 source
files. Both byte-pinned proposer goldens are unchanged, which is what says the
title rule does not fire where no heading is being destroyed.
---
## 10. Verification log
| # | claim | how it was checked |
|---|---|---|
| 1 | the five tokeniser strings | run against the committed tool before any edit; all five identical to the consumer's |
| 2 | the gold's `withheld` position is not a rank | `sorted(ids) == ids` on all three payloads → `True` |
| 3 | both mechanisms lose the number | split output printed separately from the length filter |
| 4 | the wide rule costs a hit@8 row | 5 of 6 → 4 of 6, and the losing row's document score 0.735 → 0.0 |
| 5 | equality matching is worse | measured on all three bundles; 1 better, 2 worse |
| 6 | the tokeniser alone is byte-stable on K2 | same bundle, two tools, two commands, identical sha256 |
| 7 | rank 10 → 19 (O1's regression) | reproduced on O1's own bundles before the rule was touched |
| 8 | 2 of 629 ids moved | set difference of the two concept-id sets |
| 9 | 1 106 of 1 108 files identical to the delivered bundle | `diff -rq`, and `find -type f | wc -l` for the denominator |
| 10 | the rebuild reads frozen source | `import llm_ingestion_okf; __file__` under the build's own `PYTHONPATH` |
| 11 | contract conformance | `okf_contract_check.py` exit 0, 14 rules, 0 findings, on six payloads |

28
src/llm_ingestion_okf/propose.py
View file

@ -454,18 +454,42 @@ def find_candidates(
) )
candidates: list[Candidate] = [] candidates: list[Candidate] = []
# The name an orphaned heading leaves behind, and the ONE candidate allowed
# to pick it up.
#
# WHY THIS EXISTS. A table opening on the line below a heading gives that
# heading an empty body, so the orphan check drops it, and the table block
# keeps its mechanical `Tabell linje <n>`: the section's NAME is destroyed
# even though its content survives. Measured on a 629-concept bundle after
# a spreadsheet began rendering as pipe rows -- the concept fell from
# candidate rank 10 to 19 for a question naming its subject, and restoring
# the title alone put it back at 10. The name is not invented here, it is
# carried across: it moves to the segment that holds the heading's content.
#
# CONDITIONED ON THE DROP, deliberately. A heading that keeps its own body
# is still carried by a live candidate, so copying its title onto the table
# as well would put one name on two concepts and rescue none.
orphaned_name: tuple[str, str | None] | None = None
for position_in_list, (_, candidate) in enumerate(marked): for position_in_list, (_, candidate) in enumerate(marked):
following = marked[position_in_list + 1 :] following = marked[position_in_list + 1 :]
end = offsets[following[0][0]] if following else end_of_text end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end] body = text[candidate.start : end]
# The orphan check: everything after the heading line itself. # The orphan check: everything after the heading line itself.
if not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip(): if not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip():
orphaned_name = (candidate.title, candidate.number)
continue continue
inherited, orphaned_name = orphaned_name, None
title, number = candidate.title, candidate.number
if inherited is not None and candidate.rule == RULE_TABLE_BLOCK:
# Both members, because `_segment_path` reads both: the number
# becomes the directory AND is stripped from the stem, so carrying
# the title alone would emit a name the heading never had.
title, number = inherited
candidates.append( candidates.append(
Candidate( Candidate(
title=candidate.title, title=title,
level=candidate.level, level=candidate.level,
number=candidate.number, number=number,
rule=candidate.rule, rule=candidate.rule,
start=candidate.start, start=candidate.start,
end=end, end=end,

59
tests/test_okf_consume.py
View file

@ -378,6 +378,64 @@ def test_normalise_drops_tokens_under_three_characters() -> None:
assert okf_consume.normalise("er en pris i et skjema") == ("pris", "skjema") assert okf_consume.normalise("er en pris i et skjema") == ("pris", "skjema")
def test_normalise_holds_an_identifier_number_as_one_token() -> None:
# MEASURED 2026-09-08 over three vegnormal bundles (446, 1133 and 270
# concepts): `_TOKEN_SPLIT_RE` shatters `10.2-2` into `10`, `2`, `2` and
# `MIN_TOKEN_LENGTH` then drops every piece, so a question naming a
# requirement number reaches the ranker carrying only the word `krav` --
# which every concept in such a bundle also carries. BOTH mechanisms
# participate: the split destroys the number, the floor removes the
# remains. The gold requirement was `below_k` in three of three.
assert "10.2-2" in okf_consume.normalise("Krav 10.2\u20142")
assert okf_consume.normalise("3.3.1\u201413") == ("3.3.1-13",)
assert okf_consume.normalise("2.9.2\u201412") == ("2.9.2-12",)
assert okf_consume.normalise("R610.4") == ("r610.4",)
assert okf_consume.normalise("4.2.1") == ("4.2.1",)
def test_an_identifiers_three_spellings_of_its_separator_normalise_alike() -> None:
# One requirement number arrives as an em dash from the source viewer, an
# en dash from a converter and a plain hyphen from a person typing the
# question. NFC folds NONE of the three, so a rule that does not fold them
# finds the number only in the spelling it was asked with.
hyphen = okf_consume.normalise("10.2-2")
assert okf_consume.normalise("10.2\u20142") == hyphen
assert okf_consume.normalise("10.2\u20132") == hyphen
assert hyphen == ("10.2-2",)
def test_the_identifier_rule_leaves_the_noise_floor_it_was_added_under() -> None:
# The known-negative, and the whole reason `MIN_TOKEN_LENGTH` exists: a
# bare short number matches every page number, row count and year in a
# corpus, and a matcher that scores them ranks every document equally.
assert okf_consume.normalise("10") == ()
assert okf_consume.normalise("2") == ()
assert okf_consume.normalise("er en pris i et skjema") == ("pris", "skjema")
# A hyphenated WORD is not an identifier -- no digit stands on either side
# of the separator -- so it splits exactly as it always did.
assert okf_consume.normalise("skole-anbudet") == ("skole", "anbudet")
# And a separator this rule does not claim leaves its token set untouched:
# the K2 corpus spells standards this way.
assert okf_consume.normalise("NS3935:2019") == ("ns3935", "2019")
assert okf_consume.normalise("TEK 17") == ("tek",)
def test_an_identifier_inside_a_slug_does_not_swallow_the_words_around_it() -> None:
# MEASURED, and the reason the rule joins DIGIT groups rather than
# alphanumeric ones. A first version joined alphanumeric groups across a
# separator; a corpus document's slug then became ONE token, because a
# `3-6` sits inside it, and that document's score for a question naming its
# subject fell from 0.735 to 0.0 -- one hit@8 row lost, on a question
# carrying no identifier at all. The slug below has that shape and is not
# the corpus's (SS "consumer content stays at form level"). The identifier
# is ADDED here; nothing is taken away.
tokens = okf_consume.normalise("rapport-iv-vedlegg-3-6-grunnforhold-akustikk")
assert "akustikk" in tokens
assert "grunnforhold" in tokens
assert "vedlegg" in tokens
assert "3-6" in tokens
def test_two_tokens_match_on_a_shared_prefix_of_four_and_not_of_three() -> None: def test_two_tokens_match_on_a_shared_prefix_of_four_and_not_of_three() -> None:
# "Stem-substring" is not an implementable rule: neither `prisene` nor # "Stem-substring" is not an implementable rule: neither `prisene` nor
# `prissammenstilling` contains the other. Shared prefix does the work -- # `prissammenstilling` contains the other. Shared prefix does the work --
@ -1057,6 +1115,7 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-prisform-og-loggen-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-prisform-og-loggen-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-kravnummer-tokenisering.md",
PROJECT_ROOT / "README.md", PROJECT_ROOT / "README.md",
PROJECT_ROOT / "CLAUDE.md", PROJECT_ROOT / "CLAUDE.md",
] ]

87
tests/test_propose_segments.py
View file

@ -1187,6 +1187,93 @@ Innledende avsnitt om romskjemaet.
""" """
HEADING_ORPHANED_BY_ITS_TABLE = """## Prissammenstilling
| Post | Sum |
|-------|-------|
| A | 100 |
"""
HEADING_WITH_A_BODY_ABOVE_A_TABLE = """## Prissammenstilling
Innledende tekst om skjemaet.
| Post | Sum |
|-------|-------|
| A | 100 |
"""
ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE = """## Tomt kapittel
## Innledning
Tekst i innledningen.
| Post | Sum |
|-------|-------|
| A | 100 |
"""
NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE = """## 3.1 Prissammenstilling
| Post | Sum |
|-------|-------|
| A | 100 |
"""
def test_a_table_that_orphans_its_heading_takes_that_headings_name() -> None:
"""MEASURED, twice, before the rule was written.
A spreadsheet row now renders as pipe rows, so `rule:table-block` opens on
the line after the sheet heading; the heading's own body is then empty and
the orphan check drops it; and the only surviving candidate carries the
mechanical `Tabell linje <n>`. The sheet's NAME is destroyed -- measured on
a 629-concept bundle, the concept fell from candidate rank 10 to 19 for a
question naming its subject, and restoring the title alone put it back at
10. The title is not lost here: it moves to the segment that holds the
heading's content.
"""
candidates = okf_propose_segments.find_candidates(HEADING_ORPHANED_BY_ITS_TABLE)
assert len(candidates) == 1
assert candidates[0].title == "Prissammenstilling"
assert candidates[0].rule == okf_propose_segments.RULE_TABLE_BLOCK
def test_a_table_below_a_heading_that_keeps_its_body_is_not_renamed() -> None:
"""The known-negative, and the reason the rule is conditioned on the drop.
Here the heading survives, so its title is still carried by a live
candidate. Renaming the table to it as well would put the same name on two
concepts and take a name away from neither.
"""
candidates = okf_propose_segments.find_candidates(HEADING_WITH_A_BODY_ABOVE_A_TABLE)
assert [c.title for c in candidates] == ["Prissammenstilling", "Tabell linje 5"]
def test_an_orphaned_headings_section_number_moves_with_its_title() -> None:
"""`_segment_path` reads BOTH: the number becomes the directory and is
stripped from the stem. Inheriting the title alone would emit
`3-1-prissammenstilling.md` at the top level -- a name the heading never
had."""
candidates = okf_propose_segments.find_candidates(NUMBERED_HEADING_ORPHANED_BY_ITS_TABLE)
assert len(candidates) == 1
assert candidates[0].title == "3.1 Prissammenstilling"
assert candidates[0].number == "3.1"
assert okf_propose_segments._segment_path(candidates[0], set()) == "3-1/prissammenstilling.md"
def test_an_orphaned_name_is_not_carried_past_the_candidate_that_follows_it() -> None:
"""The name goes to the ADJACENT segment or nowhere.
Written because a mutation that never cleared the carried name survived the
rest of this band: with the name held indefinitely, the table below would
take the title of a heading it does not contain, several candidates away.
That is an invention, not a rescue.
"""
candidates = okf_propose_segments.find_candidates(ORPHAN_THEN_A_LIVE_HEADING_THEN_A_TABLE)
assert [c.title for c in candidates] == ["Innledning", "Tabell linje 6"]
def test_todays_grid_table_cuts_one_table_into_one_concept_per_row_group() -> None: def test_todays_grid_table_cuts_one_table_into_one_concept_per_row_group() -> None:
"""Characterization. ONE table, three concepts -- the defect Arm E measures. """Characterization. ONE table, three concepts -- the defect Arm E measures.

45
tools/okf_consume.py
View file

@ -522,9 +522,27 @@ MIN_SHARED_PREFIX = 4
_TOKEN_SPLIT_RE = re.compile(r"[^0-9a-zà-öø-ÿ]+") _TOKEN_SPLIT_RE = re.compile(r"[^0-9a-zà-öø-ÿ]+")
#: Every dash a source spells an identifier's separator with, folded to the
#: ASCII hyphen. NFC folds NONE of them, so `10.2—2` from a document viewer and
#: `10.2-2` from a person typing the question are two different tokens until
#: this table runs. The set is the Unicode dash block plus the minus sign.
_DASH_TO_HYPHEN = str.maketrans(dict.fromkeys("‐‑‒–—―−", "-"))
#: An identifier: NUMERIC groups joined by `.` or `-`, with an optional letter
#: prefix that touches its digits without a separator (`R610.4`).
#:
#: THE LETTERS ARE THE POINT, and this pattern was narrowed by a measurement
#: rather than written this way. A rule that joined alphanumeric groups across
#: a separator swallowed a whole document slug -- `...bilag-3-6-premissrapport-
#: akustikk` became ONE token because `3-6` sits inside it -- and that
#: document's score for a question naming its subject fell from 0.735 to 0.0,
#: taking a hit@8 row with it. Only digits may stand on either side of a
#: separator, so a hyphenated word keeps its words.
_IDENTIFIER_RE = re.compile(r"[a-zà-öø-ÿ]*[0-9]+(?:[.-][0-9]+)+")
def normalise(text: str) -> tuple[str, ...]: def normalise(text: str) -> tuple[str, ...]:
"""Text as comparable tokens: NFC first, then casefold, then split. """Text as comparable tokens: NFC, casefold, dash-fold, then split.
NFC FIRST is load-bearing and not tidiness. macOS hands filenames over NFC FIRST is load-bearing and not tidiness. macOS hands filenames over
decomposed, so `å` arrives as `a` + U+030A; the combining ring is not a word decomposed, so `å` arrives as `a` + U+030A; the combining ring is not a word
@ -532,9 +550,30 @@ def normalise(text: str) -> tuple[str, ...]:
the term is silently lost. `æ` and `ø` have no canonical decomposition, so a the term is silently lost. `æ` and `ø` have no canonical decomposition, so a
test built on either passes while the bug is live -- which is why the test built on either passes while the bug is live -- which is why the
known-positive for this function uses `å`. known-positive for this function uses `å`.
IDENTIFIERS SURVIVE THE SPLIT. Splitting on every non-alphanumeric turns a
requirement number into digit runs, and `MIN_TOKEN_LENGTH` then removes
them: `Krav 10.2—2` reached the ranker as `krav` alone, a word every concept
in a standards bundle carries, so the ranking became a corpus-wide tie and
the named requirement was withheld `below_k` (measured 2026-09-08 on three
bundles). The floor stays -- a bare `10` matches every page number in a
corpus -- and the identifier is exempted from it rather than the floor
lowered for everyone.
""" """
folded = unicodedata.normalize("NFC", text).casefold() folded = unicodedata.normalize("NFC", text).casefold().translate(_DASH_TO_HYPHEN)
return tuple(token for token in _TOKEN_SPLIT_RE.split(folded) if len(token) >= MIN_TOKEN_LENGTH) tokens: list[str] = []
position = 0
for match in _IDENTIFIER_RE.finditer(folded):
tokens.extend(_split(folded[position : match.start()]))
tokens.append(match.group())
position = match.end()
tokens.extend(_split(folded[position:]))
return tuple(tokens)
def _split(folded: str) -> list[str]:
"""The generic split, on text already folded by `normalise`."""
return [token for token in _TOKEN_SPLIT_RE.split(folded) if len(token) >= MIN_TOKEN_LENGTH]
def tokens_match(left: str, right: str) -> bool: def tokens_match(left: str, right: str) -> bool: