llm-ingestion-okf/docs/2026-09-08-kravnummer-tokenisering.md

The requirement number the pre-pass could not see, and the title the orphan gate destroyed, 2026-09-08

Two consumer-reported defects, one session, one rebuild. The first is a consumption-side defect reported by vegnormal-okf: okf_consume's tokeniser shatters a requirement number, so the one concept an engineer names is withheld. The second is a production-side defect this repository measured on itself in the previous session: a table that opens directly under a heading destroys that heading's name.

---

0. What IS measured, and what is NOT

Measured. The five tokeniser strings the consumer published, reproduced here before anything was changed; where the number is lost, decomposed to both mechanisms; the gold concept's rank on three real bundles, before and after, with denominators; one alternative rule falsified against the same three bundles; the K2 control question as a payload digest with the tokeniser as the only variable; the title axis reproduced from the previous session's number before it was touched; a full K2 rebuild with the conservation identity, the concept count, the number of ids that moved against a denominator of 629, and the new content identity; and six mutations against the new tests.

Known-positive, run first. Nine figures published elsewhere were reproduced on this machine before any of the after-columns were read:

#	known-positive	source	measured here
1	normalise('Krav 10.2—2') → ['krav']	consumer § 3.1	('krav',)
2	normalise('10.2-2') → []	consumer § 3.1	()
3	normalise('3.3.1—13') → []	consumer § 3.1	()
4	'N500' → ['n500'], 'abc123' → ['abc123']	consumer § 3.1	identical
5	considered / withheld / delivered per bundle	consumer § 3	446/438/8, 1133/1125/8, 270/262/8
6	gold position in withheld	consumer § 3.1	119 of 438, 204 of 1125, 66 of 262
7	payload tokens (o200k) per bundle	consumer § 3	22 138, 53 444, 14 694
8	priced concept candidate rank 10 → 19	this repo, 2026-09-08 (O1) § 7	10 → 19, and 251 / 1 / 1 for the other three rows
9	K2 payload bytes, two commands	this repo, O1 § 7	169 573 B and 172 588 B

NOT measured. Whether a live model answers better with the named requirement in the payload. This document measures a ranking; the consumer measures the answer. Also not measured: that either rule helps a corpus other than the four read here.

Not closed. The gold requirement is still below_k on all three bundles after the fix. § 8 states why, with the number.

---

1. The five strings, and one correction to how they are read

PYTHONPATH=… python3 -c "from okf_consume import normalise; …"
  'Krav 10.2—2' -> ('krav',)          'N500'         -> ('n500',)
  '10.2-2'      -> ()                 'abc123'       -> ('abc123',)
  '3.3.1—13'    -> ()                 '10'           -> ()
  '2.9.2—12'    -> ()                 '2'            -> ()

All five reproduce, and so do the two known-positives. So does the gold's place in the withheld list: 119 of 438, 204 of 1125, 66 of 262.

That place is a list position and not a rank, and the difference matters. The withheld array is emitted in concept_id order — measured, not assumed (sorted(ids) == ids is True on all three payloads) — so it cannot move when the ranking moves. It is a correct number that answers a different question. The instrument that does move is the fused rank, and § 4 uses it.

One more figure the consumer's mechanism claim predicts, confirmed here: of the 262 withheld entries on the smallest bundle, 262 are below_k and none is no_lexical_match. Every concept matches, because every concept carries the word the question carries.

---

2. Where the number is lost: both mechanisms, not one

The order asked which of the two is responsible. Both, and neither alone would lose it:

'10.2-2'    split-> ['10', '2', '2']        kept-> []
'3.3.1—13'  split-> ['3', '3', '1', '13']   kept-> []
'R610.4'    split-> ['r610', '4']           kept-> ['r610']

_TOKEN_SPLIT_RE (tools/okf_consume.py:523 before this change) splits on every non-alphanumeric character, which destroys the identifier; the MIN_TOKEN_LENGTH filter in normalise (line 537) then removes the digit runs that are left. Lowering the floor alone would not recover the number — it would return 10, 2 and 2, which match every page number in the corpus. The identifier has to survive the split.

---

3. The rule, and the alternative that was falsified

The rule. Numeric groups joined by . or - are held together as one token, with a leading letter run allowed only where it touches its digits without a separator. Dash variants fold to the ASCII hyphen first, so the same number written with an em dash, an en dash or a hyphen is one token. MIN_TOKEN_LENGTH is unchanged; the identifier is exempt from it by construction rather than by lowering it for everyone.

It was narrowed by a measurement, not by taste. The first version joined alphanumeric groups across a separator. That version swallowed a whole document slug — a slug of the form …-3-6-… became ONE token — and that document's stage-one score for a question naming its subject fell from 0.735 to 0.0, costing one row of the repository's own hit@8 measurement. The narrowed rule is purely additive on such a slug: the identifier is added, the words stay.

The falsified alternative. Because tokens_match compares a four-character prefix, 10.2-2 matches 10.2-5: on the smallest bundle exactly 5 of 270 concept titles match the question's identifier, and all five are the same section. Making identifier tokens match only on equality is the obvious narrowing, and it was measured before being adopted or rejected:

	prefix (shipped)	equality (probe)
N100:2023	96	53
N200:2024	9	26
N500:2024	35	37

It improves one bundle and makes two worse. Falsified; not built.

---

4. The three bundles, before and after

Command, one per bundle, the questions quoted verbatim from the consumer's § 3:

python3 tools/okf_consume.py <bundle> \
  --question "Hva krever Krav 10.2—2 i N500? Gjengi det sentrale vilkåret."

	N100:2023	N200:2024	N500:2024
concepts	446	1 133	270
gold fused rank, before	160	143	100
gold fused rank, after	96	9	35
gold lexical overlap, before → after	3 → 4	4 → 6	3 → 4
considered / withheld / delivered	446/438/8	1133/1125/8	270/262/8
budget spent, before → after	8 977 → 12 095	17 818 → 17 818	10 517 → 10 517
payload o200k, before → after	22 138 → 23 583	53 444 → 53 444	14 694 → 14 694
okf_contract_check	exit 0, 14 rules, 0 findings	idem	idem
gold delivered?	no, below_k	no, below_k	no, below_k

Two of the three payloads are byte-identical before and after: the gold moved a long way without reaching the top eight, so the delivered set did not change.

The bonus measurement the order asked for. Three spellings of the same requirement number, on the same bundle:

question spelling	identifier token	gold rank
Krav 10.2—2 (em dash)	10.2-2	35
krav 10.2-2 (hyphen, lower case)	10.2-2	35
Krav 10.2–2 (en dash)	10.2-2	35

All three normalise alike and rank alike.

---

5. The K2 control: the tokeniser alone changes nothing

The order required the S7 control question to be compared byte for byte with the tokeniser as the only variable, so it was run against ONE unchanged bundle with the committed tool and the new tool:

command	before	after
default	sha256 2ae46f683a3b17fb…, 169 573 B	identical
--cost-vocabulary --k 12 --limit 160000	sha256 8aaf8558369b3d6b…, 172 588 B	identical

Byte for byte, both commands. The two byte counts are the ones published in the previous session's report, which is what makes this a control and not a coincidence. The corpus does carry identifiers of other shapes (NS3935:2019, TEK 17); measured, the rule leaves both untouched, because a colon is not a separator it claims and a space was never one.

---

6. The title axis in the orphan gate

Reproduced first, from the previous session's numbers, before the rule was touched — all four rows to the integer:

question	vocabulary	O1 measured	reproduced here
mandate-shaped	off	251 of 271	251
mandate-shaped	on	19 of 280	19
specific	off	1 of 233	1
specific	on	1 of 248	1

The rule. When a heading is dropped by the orphan check and the candidate immediately after it is a table block, the table block takes that heading's title and its section number instead of the mechanical Tabell linje <n>.

Three properties of that sentence are each a decision:

1. Conditioned on the drop. A heading that keeps its own body is still carried by a live candidate; copying its title onto the table as well would put one name on two concepts and rescue none.
2. Immediately after. The name goes to the adjacent segment or nowhere. Carried further, a table several candidates away would take the name of a heading it does not contain — an invention rather than a rescue. This is the one property no test caught until a surviving mutation exposed it (§ 9).
3. Number as well as title. _segment_path reads both — the number becomes the directory and is stripped from the stem — so inheriting the title alone would emit 3-1-prissammenstilling.md at the top level, a name the heading never had.

It is format-agnostic by construction: the orphan check and the table rule are, so a sheet name in a spreadsheet and a heading in a PDF go the same way.

---

7. The rebuild: 629 concepts, two ids, a new identity

One rebuild for both fixes, okf build over the same 43-document folder, run from a frozen copy of src/ so no later edit could reach a measurement already in flight.

	value
merged + coded rejections == N	39 + 4 = 43 = N, exit 0
concepts	629, unchanged
concept ids that moved	2 of 629 (0.32 %)
new content identity	sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b

The two ids move BACK. …/tabell-linje-3 becomes …/prissammenstilling-sheet-1 and …/ark1-sheet-1 again — the names they carried in the bundle delivered on 2026-09-03. Against that delivered bundle, 1 106 of 1 108 files are now identical, and the two that differ are the two spreadsheet bodies the previous session deliberately changed. A consumer holding the 2026-09-03 tree gets its file names back; the ref, however, is new, and c26eed6a… from the previous session is superseded and stale.

The ranking, on the rebuilt bundle:

question	vocabulary	O1 bundle	rebuilt
mandate-shaped	off	251 of 271	251
mandate-shaped	on	19 of 280	10
specific	off	1 of 233	1
specific	on	1 of 248	1

Rank 10 is exactly where the concept sat before the previous session's form fix. The regression is closed, and the question this repository's own hit@8 measurement uses is untouched.

What it now costs to deliver that concept, which is the figure a consumer budgets against:

command	bytes	o200k	priced sheet
default	169 583	53 420	below_k
--cost-vocabulary --k 12 (default budget)	183 798	58 907	delivered, rank 10
--cost-vocabulary --k 20 (default budget)	190 438	61 955	delivered, rank 10
specific question, default	108 877	37 189	delivered, rank 1

The previous session's delivering command was --cost-vocabulary --k 20 at 65 912 o200k. The delivering command is now --cost-vocabulary --k 12 at 58 907 — 10.6 % cheaper, and it fits the default budget. okf_contract_check returns exit 0 with 14 rules and 0 findings on all three payloads, and the identity closes: 8 + 621 = 629, 12 + 617 = 629.

---

8. What did NOT close, and why — stated rather than glossed

The requirement question is still below_k on all three bundles. The fix moved the gold from 160 to 96, 143 to 9 and 100 to 35, which is a large move and not a delivery. The remaining blocker was decomposed rather than guessed:

_overlap returns a COUNT. On the smallest bundle the gold's overlap is 4 out of a possible 7, and 70 of 270 concepts also score 4 while 12 score 5. Matching the exact requirement number is worth exactly as much as matching the verb in Hva **krever** … or the noun in … det sentrale **vilkåret**. The identifier is now visible to the ranker; it is not yet worth more than a common word.

Making it worth more is a weighting decision — a rarity weight, or a declared class of high-specificity tokens — and it is a different change from this one: it moves every ranking in every corpus, and it has no measurement behind it yet. It is not in this order and is not decided here.

And the honesty limit that governs all of the above. This document measures a ranking: which concepts a deterministic pre-pass puts in a payload. Whether a live model then answers the engineer's question correctly is the consumer's measurement, not this one. "Delivered" is not "read".

---

9. Tests, and the mutation that survived

Eight new tests, each written red first (1 287 -> 1 295). Six mutations, six red, with the unmutated control green after every restore:

mutation	what it breaks	result
widen the identifier back to alphanumeric groups	the slug regression	3 red
drop the dash fold	one spelling finds, two do not	2 red
never emit the identifier token	the whole rule	3 red
inherit the title but not the number	the emitted path	1 red
inherit onto any rule, not only a table	one name on two concepts	1 red
never clear the carried name	a distant table takes the name	survived

The survivor was read as code before it was called equivalent, per this repository's own trap note: it is not equivalent — it lets a heading's name travel arbitrarily far — and the band simply had no fixture where an orphan is followed by something other than a table. One was added, and the mutation now dies.

Suite: 1 295 passing (was 1 287), ruff clean, mypy clean on 17 source files. Both byte-pinned proposer goldens are unchanged, which is what says the title rule does not fire where no heading is being destroyed.

---

10. Verification log

#	claim	how it was checked
1	the five tokeniser strings	run against the committed tool before any edit; all five identical to the consumer's
2	the gold's withheld position is not a rank	sorted(ids) == ids on all three payloads → True
3	both mechanisms lose the number	split output printed separately from the length filter
4	the wide rule costs a hit@8 row	5 of 6 → 4 of 6, and the losing row's document score 0.735 → 0.0
5	equality matching is worse	measured on all three bundles; 1 better, 2 worse
6	the tokeniser alone is byte-stable on K2	same bundle, two tools, two commands, identical sha256
7	rank 10 → 19 (O1's regression)	reproduced on O1's own bundles before the rule was touched
8	2 of 629 ids moved	set difference of the two concept-id sets
9	1 106 of 1 108 files identical to the delivered bundle	diff -rq, and `find -type f
10	the rebuild reads frozen source	import llm_ingestion_okf; __file__ under the build's own PYTHONPATH
11	contract conformance	okf_contract_check.py exit 0, 14 rules, 0 findings, on six payloads