open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table

Browse source 
Two consumer-reported defects, one rebuild.

The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split
`10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question
naming a requirement reached the ranker carrying only the word every concept
in a standards bundle carries. Measured on three real bundles (446, 1133 and
270 concepts), the named requirement was withheld `below_k` in three of three.
Numeric groups joined by `.` or `-` are now held together, dash variants fold
to the ASCII hyphen, and the noise floor is unchanged. The gold moves from
160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is
still `below_k`, because `_overlap` is a count and an exact requirement number
is worth no more than a common verb. That weighting is a separate decision.

The rule was narrowed by a measurement: a version that joined alphanumeric
groups swallowed a document slug whole and cost a hit@8 row. An equality-only
variant was measured on all three bundles and falsified -- better on one,
worse on two.

The orphan gate destroyed a heading's name: a table opening directly below a
heading left that heading with an empty body, the orphan check dropped it, and
the surviving table block kept the mechanical `Tabell linje <n>`. A table that
orphans its heading now takes that heading's title and section number.
Conditioned on the drop, on adjacency, and carrying both members -- each of
the three measured or mutation-tested.

One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43
= N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03
bundle carried, 1106 of 1108 files identical to it. New ref
sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b
-- c26eed6a... is superseded. The regression the previous session measured is
closed: candidate rank 19 -> 10, and the delivering command is now
`--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against
65 912 before. The specific question is unmoved at rank 1.

The tokeniser alone leaves the K2 control question byte-identical, measured
with the bundle held fixed and both published byte counts reproduced.

Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first;
6 mutations, 6 red, one of them only after the survivor was read as code and
a missing fixture was added. Suite 1287 -> 1295.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 11:43:39 +02:00
commit 56c1205ec4
5 changed files with 550 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

336
docs/2026-09-08-kravnummer-tokenisering.md Normal file
View file

@ -0,0 +1,336 @@
# The requirement number the pre-pass could not see, and the title the orphan gate destroyed, 2026-09-08
Two consumer-reported defects, one session, one rebuild. The first is a
consumption-side defect reported by `vegnormal-okf`: `okf_consume`'s tokeniser
shatters a requirement number, so the one concept an engineer names is withheld.
The second is a production-side defect this repository measured on itself in the
previous session: a table that opens directly under a heading destroys that
heading's name.
---
## 0. What IS measured, and what is NOT
**Measured.** The five tokeniser strings the consumer published, reproduced
here before anything was changed; where the number is lost, decomposed to both
mechanisms; the gold concept's rank on three real bundles, before and after, with
denominators; one alternative rule falsified against the same three bundles; the
K2 control question as a payload digest with the tokeniser as the only variable;
the title axis reproduced from the previous session's number before it was
touched; a full K2 rebuild with the conservation identity, the concept count, the
number of ids that moved against a denominator of 629, and the new content
identity; and six mutations against the new tests.
**Known-positive, run first.** Nine figures published elsewhere were reproduced
on this machine before any of the after-columns were read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | `normalise('Krav 10.2—2')` → `['krav']` | consumer § 3.1 | `('krav',)` |
| 2 | `normalise('10.2-2')` → `[]` | consumer § 3.1 | `()` |
| 3 | `normalise('3.3.1—13')` → `[]` | consumer § 3.1 | `()` |
| 4 | `'N500'` → `['n500']`, `'abc123'` → `['abc123']` | consumer § 3.1 | identical |
| 5 | considered / withheld / delivered per bundle | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
| 6 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
| 7 | payload tokens (o200k) per bundle | consumer § 3 | 22 138, 53 444, 14 694 |
| 8 | priced concept candidate rank 10 → 19 | this repo, 2026-09-08 (O1) § 7 | 10 → 19, and 251 / 1 / 1 for the other three rows |
| 9 | K2 payload bytes, two commands | this repo, O1 § 7 | 169 573 B and 172 588 B |
**NOT measured.** Whether a live model answers *better* with the named
requirement in the payload. This document measures a ranking; the consumer
measures the answer. Also not measured: that either rule helps a corpus other
than the four read here.
**Not closed.** The gold requirement is still `below_k` on all three bundles
after the fix. § 8 states why, with the number.
---
## 1. The five strings, and one correction to how they are read
```
PYTHONPATH=… python3 -c "from okf_consume import normalise; …"
'Krav 10.2—2' -> ('krav',) 'N500' -> ('n500',)
'10.2-2' -> () 'abc123' -> ('abc123',)
'3.3.1—13' -> () '10' -> ()
'2.9.2—12' -> () '2' -> ()
```
All five reproduce, and so do the two known-positives. So does the gold's place
in the `withheld` list: 119 of 438, 204 of 1125, 66 of 262.
**That place is a list position and not a rank, and the difference matters.**
The `withheld` array is emitted in `concept_id` order — measured, not assumed
(`sorted(ids) == ids` is `True` on all three payloads) — so it cannot move when
the ranking moves. It is a correct number that answers a different question. The
instrument that does move is the fused rank, and § 4 uses it.
One more figure the consumer's mechanism claim predicts, confirmed here: of the
262 withheld entries on the smallest bundle, **262 are `below_k` and none is
`no_lexical_match`**. Every concept matches, because every concept carries the
word the question carries.
---
## 2. Where the number is lost: both mechanisms, not one
The order asked which of the two is responsible. **Both**, and neither alone
would lose it:
```
'10.2-2' split-> ['10', '2', '2'] kept-> []
'3.3.1—13' split-> ['3', '3', '1', '13'] kept-> []
'R610.4' split-> ['r610', '4'] kept-> ['r610']
```
`_TOKEN_SPLIT_RE` (`tools/okf_consume.py:523` before this change) splits on
every non-alphanumeric character, which destroys the identifier; the
`MIN_TOKEN_LENGTH` filter in `normalise` (line 537) then removes the digit runs
that are left. Lowering the floor alone would not recover the number — it would
return `10`, `2` and `2`, which match every page number in the corpus. The
identifier has to survive the split.
---
## 3. The rule, and the alternative that was falsified
**The rule.** Numeric groups joined by `.` or `-` are held together as one
token, with a leading letter run allowed only where it touches its digits
without a separator. Dash variants fold to the ASCII hyphen first, so the same
number written with an em dash, an en dash or a hyphen is one token.
`MIN_TOKEN_LENGTH` is unchanged; the identifier is exempt from it by
construction rather than by lowering it for everyone.
**It was narrowed by a measurement, not by taste.** The first version joined
*alphanumeric* groups across a separator. That version swallowed a whole
document slug — a slug of the form `…-3-6-…` became ONE token — and that
document's stage-one score for a question naming its subject fell from **0.735
to 0.0**, costing one row of the repository's own hit@8 measurement. The
narrowed rule is purely additive on such a slug: the identifier is added, the
words stay.
**The falsified alternative.** Because `tokens_match` compares a four-character
prefix, `10.2-2` matches `10.2-5`: on the smallest bundle exactly **5 of 270**
concept titles match the question's identifier, and all five are the same
section. Making identifier tokens match only on *equality* is the obvious
narrowing, and it was measured before being adopted or rejected:
| | prefix (shipped) | equality (probe) |
|---|---|---|
| N100:2023 | 96 | **53** |
| N200:2024 | **9** | 26 |
| N500:2024 | **35** | 37 |
It improves one bundle and makes two worse. **Falsified; not built.**
---
## 4. The three bundles, before and after
Command, one per bundle, the questions quoted verbatim from the consumer's § 3:
```
python3 tools/okf_consume.py <bundle> \
--question "Hva krever Krav 10.2—2 i N500? Gjengi det sentrale vilkåret."
```
| | N100:2023 | N200:2024 | N500:2024 |
|---|---|---|---|
| concepts | 446 | 1 133 | 270 |
| gold fused rank, **before** | 160 | 143 | 100 |
| gold fused rank, **after** | **96** | **9** | **35** |
| gold lexical overlap, before → after | 3 → 4 | 4 → 6 | 3 → 4 |
| considered / withheld / delivered | 446/438/8 | 1133/1125/8 | 270/262/8 |
| budget spent, before → after | 8 977 → 12 095 | 17 818 → 17 818 | 10 517 → 10 517 |
| payload o200k, before → after | 22 138 → 23 583 | 53 444 → 53 444 | 14 694 → 14 694 |
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
| **gold delivered?** | **no, `below_k`** | **no, `below_k`** | **no, `below_k`** |
Two of the three payloads are byte-identical before and after: the gold moved a
long way without reaching the top eight, so the delivered set did not change.
**The bonus measurement the order asked for.** Three spellings of the same
requirement number, on the same bundle:
| question spelling | identifier token | gold rank |
|---|---|---|
| `Krav 10.2—2` (em dash) | `10.2-2` | 35 |
| `krav 10.2-2` (hyphen, lower case) | `10.2-2` | 35 |
| `Krav 10.2–2` (en dash) | `10.2-2` | 35 |
All three normalise alike and rank alike.
---
## 5. The K2 control: the tokeniser alone changes nothing
The order required the S7 control question to be compared byte for byte with
the tokeniser as the only variable, so it was run against ONE unchanged bundle
with the committed tool and the new tool:
| command | before | after |
|---|---|---|
| default | `sha256 2ae46f683a3b17fb…`, 169 573 B | **identical** |
| `--cost-vocabulary --k 12 --limit 160000` | `sha256 8aaf8558369b3d6b…`, 172 588 B | **identical** |
Byte for byte, both commands. The two byte counts are the ones published in the
previous session's report, which is what makes this a control and not a
coincidence. The corpus does carry identifiers of other shapes (`NS3935:2019`,
`TEK 17`); measured, the rule leaves both untouched, because a colon is not a
separator it claims and a space was never one.
---
## 6. The title axis in the orphan gate
Reproduced first, from the previous session's numbers, before the rule was
touched — all four rows to the integer:
| question | vocabulary | O1 measured | reproduced here |
|---|---|---|---|
| mandate-shaped | off | 251 of 271 | 251 |
| mandate-shaped | on | **19** of 280 | **19** |
| specific | off | 1 of 233 | 1 |
| specific | on | 1 of 248 | 1 |
**The rule.** When a heading is dropped by the orphan check and the candidate
immediately after it is a table block, the table block takes that heading's
title *and* its section number instead of the mechanical `Tabell linje <n>`.
Three properties of that sentence are each a decision:
1. **Conditioned on the drop.** A heading that keeps its own body is still
carried by a live candidate; copying its title onto the table as well would
put one name on two concepts and rescue none.
2. **Immediately after.** The name goes to the adjacent segment or nowhere.
Carried further, a table several candidates away would take the name of a
heading it does not contain — an invention rather than a rescue. This is the
one property no test caught until a surviving mutation exposed it (§ 9).
3. **Number as well as title.** `_segment_path` reads both — the number becomes
the directory and is stripped from the stem — so inheriting the title alone
would emit `3-1-prissammenstilling.md` at the top level, a name the heading
never had.
It is format-agnostic by construction: the orphan check and the table rule are,
so a sheet name in a spreadsheet and a heading in a PDF go the same way.
---
## 7. The rebuild: 629 concepts, two ids, a new identity
One rebuild for both fixes, `okf build` over the same 43-document folder, run
from a frozen copy of `src/` so no later edit could reach a measurement already
in flight.
| | value |
|---|---|
| `merged + coded rejections == N` | `39 + 4 = 43 = N`, exit 0 |
| concepts | **629**, unchanged |
| concept ids that moved | **2 of 629** (0.32 %) |
| new content identity | `sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b` |
**The two ids move BACK.** `…/tabell-linje-3` becomes `…/prissammenstilling-sheet-1`
and `…/ark1-sheet-1` again — the names they carried in the bundle delivered on
2026-09-03. Against that delivered bundle, **1 106 of 1 108 files are now
identical**, and the two that differ are the two spreadsheet bodies the previous
session deliberately changed. A consumer holding the 2026-09-03 tree gets its
file names back; the ref, however, is new, and `c26eed6a…` from the previous
session is superseded and stale.
**The ranking, on the rebuilt bundle:**
| question | vocabulary | O1 bundle | rebuilt |
|---|---|---|---|
| mandate-shaped | off | 251 of 271 | 251 |
| mandate-shaped | on | 19 of 280 | **10** |
| specific | off | 1 of 233 | 1 |
| specific | on | 1 of 248 | **1** |
Rank 10 is exactly where the concept sat before the previous session's form fix.
The regression is closed, and the question this repository's own hit@8
measurement uses is untouched.
**What it now costs to deliver that concept**, which is the figure a consumer
budgets against:
| command | bytes | o200k | priced sheet |
|---|---|---|---|
| default | 169 583 | 53 420 | `below_k` |
| `--cost-vocabulary --k 12` (default budget) | 183 798 | **58 907** | **delivered, rank 10** |
| `--cost-vocabulary --k 20` (default budget) | 190 438 | 61 955 | delivered, rank 10 |
| specific question, default | 108 877 | 37 189 | delivered, rank 1 |
The previous session's delivering command was `--cost-vocabulary --k 20` at
**65 912** o200k. The delivering command is now `--cost-vocabulary --k 12` at
**58 907** — 10.6 % cheaper, and it fits the default budget. `okf_contract_check`
returns exit 0 with 14 rules and 0 findings on all three payloads, and the
identity closes: `8 + 621 = 629`, `12 + 617 = 629`.
---
## 8. What did NOT close, and why — stated rather than glossed
**The requirement question is still `below_k` on all three bundles.** The fix
moved the gold from 160 to 96, 143 to 9 and 100 to 35, which is a large move and
not a delivery. The remaining blocker was decomposed rather than guessed:
`_overlap` returns a COUNT. On the smallest bundle the gold's overlap is 4 out
of a possible 7, and **70 of 270 concepts also score 4** while 12 score 5.
Matching the exact requirement number is worth exactly as much as matching the
verb in `Hva **krever** …` or the noun in `… det sentrale **vilkåret**`. The
identifier is now visible to the ranker; it is not yet *worth* more than a
common word.
Making it worth more is a weighting decision — a rarity weight, or a declared
class of high-specificity tokens — and it is a different change from this one:
it moves every ranking in every corpus, and it has no measurement behind it yet.
It is not in this order and is not decided here.
**And the honesty limit that governs all of the above.** This document measures
a *ranking*: which concepts a deterministic pre-pass puts in a payload. Whether
a live model then answers the engineer's question correctly is the consumer's
measurement, not this one. "Delivered" is not "read".
---
## 9. Tests, and the mutation that survived
Eight new tests, each written red first (1 287 -> 1 295). Six
mutations, six red, with the unmutated control green after every restore:
| mutation | what it breaks | result |
|---|---|---|
| widen the identifier back to alphanumeric groups | the slug regression | 3 red |
| drop the dash fold | one spelling finds, two do not | 2 red |
| never emit the identifier token | the whole rule | 3 red |
| inherit the title but not the number | the emitted path | 1 red |
| inherit onto any rule, not only a table | one name on two concepts | 1 red |
| never clear the carried name | a distant table takes the name | **survived** |
The survivor was read as code before it was called equivalent, per this
repository's own trap note: it is not equivalent — it lets a heading's name
travel arbitrarily far — and the band simply had no fixture where an orphan is
followed by something other than a table. One was added, and the mutation now
dies.
Suite: **1 295 passing** (was 1 287), `ruff` clean, `mypy` clean on 17 source
files. Both byte-pinned proposer goldens are unchanged, which is what says the
title rule does not fire where no heading is being destroyed.
---
## 10. Verification log
| # | claim | how it was checked |
|---|---|---|
| 1 | the five tokeniser strings | run against the committed tool before any edit; all five identical to the consumer's |
| 2 | the gold's `withheld` position is not a rank | `sorted(ids) == ids` on all three payloads → `True` |
| 3 | both mechanisms lose the number | split output printed separately from the length filter |
| 4 | the wide rule costs a hit@8 row | 5 of 6 → 4 of 6, and the losing row's document score 0.735 → 0.0 |
| 5 | equality matching is worse | measured on all three bundles; 1 better, 2 worse |
| 6 | the tokeniser alone is byte-stable on K2 | same bundle, two tools, two commands, identical sha256 |
| 7 | rank 10 → 19 (O1's regression) | reproduced on O1's own bundles before the rule was touched |
| 8 | 2 of 629 ids moved | set difference of the two concept-id sets |
| 9 | 1 106 of 1 108 files identical to the delivered bundle | `diff -rq`, and `find -type f | wc -l` for the denominator |
| 10 | the rebuild reads frozen source | `import llm_ingestion_okf; __file__` under the build's own `PYTHONPATH` |
| 11 | contract conformance | `okf_contract_check.py` exit 0, 14 rules, 0 findings, on six payloads |