Kjell Tore Guttormsen
56c1205ec4
Two consumer-reported defects, one rebuild. The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split `10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question naming a requirement reached the ranker carrying only the word every concept in a standards bundle carries. Measured on three real bundles (446, 1133 and 270 concepts), the named requirement was withheld `below_k` in three of three. Numeric groups joined by `.` or `-` are now held together, dash variants fold to the ASCII hyphen, and the noise floor is unchanged. The gold moves from 160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is still `below_k`, because `_overlap` is a count and an exact requirement number is worth no more than a common verb. That weighting is a separate decision. The rule was narrowed by a measurement: a version that joined alphanumeric groups swallowed a document slug whole and cost a hit@8 row. An equality-only variant was measured on all three bundles and falsified -- better on one, worse on two. The orphan gate destroyed a heading's name: a table opening directly below a heading left that heading with an empty body, the orphan check dropped it, and the surviving table block kept the mechanical `Tabell linje <n>`. A table that orphans its heading now takes that heading's title and section number. Conditioned on the drop, on adjacency, and carrying both members -- each of the three measured or mutation-tested. One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43 = N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03 bundle carried, 1106 of 1108 files identical to it. New ref sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b -- c26eed6a... is superseded. The regression the previous session measured is closed: candidate rank 19 -> 10, and the delivering command is now `--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against 65 912 before. The specific question is unmoved at rank 1. The tokeniser alone leaves the K2 control question byte-identical, measured with the bundle held fixed and both published byte counts reproduced. Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first; 6 mutations, 6 red, one of them only after the survivor was read as code and a missing fixture was added. Suite 1287 -> 1295. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
336 lines
16 KiB
Markdown
336 lines
16 KiB
Markdown
# The requirement number the pre-pass could not see, and the title the orphan gate destroyed, 2026-09-08
|
||
|
||
Two consumer-reported defects, one session, one rebuild. The first is a
|
||
consumption-side defect reported by `vegnormal-okf`: `okf_consume`'s tokeniser
|
||
shatters a requirement number, so the one concept an engineer names is withheld.
|
||
The second is a production-side defect this repository measured on itself in the
|
||
previous session: a table that opens directly under a heading destroys that
|
||
heading's name.
|
||
|
||
---
|
||
|
||
## 0. What IS measured, and what is NOT
|
||
|
||
**Measured.** The five tokeniser strings the consumer published, reproduced
|
||
here before anything was changed; where the number is lost, decomposed to both
|
||
mechanisms; the gold concept's rank on three real bundles, before and after, with
|
||
denominators; one alternative rule falsified against the same three bundles; the
|
||
K2 control question as a payload digest with the tokeniser as the only variable;
|
||
the title axis reproduced from the previous session's number before it was
|
||
touched; a full K2 rebuild with the conservation identity, the concept count, the
|
||
number of ids that moved against a denominator of 629, and the new content
|
||
identity; and six mutations against the new tests.
|
||
|
||
**Known-positive, run first.** Nine figures published elsewhere were reproduced
|
||
on this machine before any of the after-columns were read:
|
||
|
||
| # | known-positive | source | measured here |
|
||
|---|---|---|---|
|
||
| 1 | `normalise('Krav 10.2—2')` → `['krav']` | consumer § 3.1 | `('krav',)` |
|
||
| 2 | `normalise('10.2-2')` → `[]` | consumer § 3.1 | `()` |
|
||
| 3 | `normalise('3.3.1—13')` → `[]` | consumer § 3.1 | `()` |
|
||
| 4 | `'N500'` → `['n500']`, `'abc123'` → `['abc123']` | consumer § 3.1 | identical |
|
||
| 5 | considered / withheld / delivered per bundle | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
|
||
| 6 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
|
||
| 7 | payload tokens (o200k) per bundle | consumer § 3 | 22 138, 53 444, 14 694 |
|
||
| 8 | priced concept candidate rank 10 → 19 | this repo, 2026-09-08 (O1) § 7 | 10 → 19, and 251 / 1 / 1 for the other three rows |
|
||
| 9 | K2 payload bytes, two commands | this repo, O1 § 7 | 169 573 B and 172 588 B |
|
||
|
||
**NOT measured.** Whether a live model answers *better* with the named
|
||
requirement in the payload. This document measures a ranking; the consumer
|
||
measures the answer. Also not measured: that either rule helps a corpus other
|
||
than the four read here.
|
||
|
||
**Not closed.** The gold requirement is still `below_k` on all three bundles
|
||
after the fix. § 8 states why, with the number.
|
||
|
||
---
|
||
|
||
## 1. The five strings, and one correction to how they are read
|
||
|
||
```
|
||
PYTHONPATH=… python3 -c "from okf_consume import normalise; …"
|
||
'Krav 10.2—2' -> ('krav',) 'N500' -> ('n500',)
|
||
'10.2-2' -> () 'abc123' -> ('abc123',)
|
||
'3.3.1—13' -> () '10' -> ()
|
||
'2.9.2—12' -> () '2' -> ()
|
||
```
|
||
|
||
All five reproduce, and so do the two known-positives. So does the gold's place
|
||
in the `withheld` list: 119 of 438, 204 of 1125, 66 of 262.
|
||
|
||
**That place is a list position and not a rank, and the difference matters.**
|
||
The `withheld` array is emitted in `concept_id` order — measured, not assumed
|
||
(`sorted(ids) == ids` is `True` on all three payloads) — so it cannot move when
|
||
the ranking moves. It is a correct number that answers a different question. The
|
||
instrument that does move is the fused rank, and § 4 uses it.
|
||
|
||
One more figure the consumer's mechanism claim predicts, confirmed here: of the
|
||
262 withheld entries on the smallest bundle, **262 are `below_k` and none is
|
||
`no_lexical_match`**. Every concept matches, because every concept carries the
|
||
word the question carries.
|
||
|
||
---
|
||
|
||
## 2. Where the number is lost: both mechanisms, not one
|
||
|
||
The order asked which of the two is responsible. **Both**, and neither alone
|
||
would lose it:
|
||
|
||
```
|
||
'10.2-2' split-> ['10', '2', '2'] kept-> []
|
||
'3.3.1—13' split-> ['3', '3', '1', '13'] kept-> []
|
||
'R610.4' split-> ['r610', '4'] kept-> ['r610']
|
||
```
|
||
|
||
`_TOKEN_SPLIT_RE` (`tools/okf_consume.py:523` before this change) splits on
|
||
every non-alphanumeric character, which destroys the identifier; the
|
||
`MIN_TOKEN_LENGTH` filter in `normalise` (line 537) then removes the digit runs
|
||
that are left. Lowering the floor alone would not recover the number — it would
|
||
return `10`, `2` and `2`, which match every page number in the corpus. The
|
||
identifier has to survive the split.
|
||
|
||
---
|
||
|
||
## 3. The rule, and the alternative that was falsified
|
||
|
||
**The rule.** Numeric groups joined by `.` or `-` are held together as one
|
||
token, with a leading letter run allowed only where it touches its digits
|
||
without a separator. Dash variants fold to the ASCII hyphen first, so the same
|
||
number written with an em dash, an en dash or a hyphen is one token.
|
||
`MIN_TOKEN_LENGTH` is unchanged; the identifier is exempt from it by
|
||
construction rather than by lowering it for everyone.
|
||
|
||
**It was narrowed by a measurement, not by taste.** The first version joined
|
||
*alphanumeric* groups across a separator. That version swallowed a whole
|
||
document slug — a slug of the form `…-3-6-…` became ONE token — and that
|
||
document's stage-one score for a question naming its subject fell from **0.735
|
||
to 0.0**, costing one row of the repository's own hit@8 measurement. The
|
||
narrowed rule is purely additive on such a slug: the identifier is added, the
|
||
words stay.
|
||
|
||
**The falsified alternative.** Because `tokens_match` compares a four-character
|
||
prefix, `10.2-2` matches `10.2-5`: on the smallest bundle exactly **5 of 270**
|
||
concept titles match the question's identifier, and all five are the same
|
||
section. Making identifier tokens match only on *equality* is the obvious
|
||
narrowing, and it was measured before being adopted or rejected:
|
||
|
||
| | prefix (shipped) | equality (probe) |
|
||
|---|---|---|
|
||
| N100:2023 | 96 | **53** |
|
||
| N200:2024 | **9** | 26 |
|
||
| N500:2024 | **35** | 37 |
|
||
|
||
It improves one bundle and makes two worse. **Falsified; not built.**
|
||
|
||
---
|
||
|
||
## 4. The three bundles, before and after
|
||
|
||
Command, one per bundle, the questions quoted verbatim from the consumer's § 3:
|
||
|
||
```
|
||
python3 tools/okf_consume.py <bundle> \
|
||
--question "Hva krever Krav 10.2—2 i N500? Gjengi det sentrale vilkåret."
|
||
```
|
||
|
||
| | N100:2023 | N200:2024 | N500:2024 |
|
||
|---|---|---|---|
|
||
| concepts | 446 | 1 133 | 270 |
|
||
| gold fused rank, **before** | 160 | 143 | 100 |
|
||
| gold fused rank, **after** | **96** | **9** | **35** |
|
||
| gold lexical overlap, before → after | 3 → 4 | 4 → 6 | 3 → 4 |
|
||
| considered / withheld / delivered | 446/438/8 | 1133/1125/8 | 270/262/8 |
|
||
| budget spent, before → after | 8 977 → 12 095 | 17 818 → 17 818 | 10 517 → 10 517 |
|
||
| payload o200k, before → after | 22 138 → 23 583 | 53 444 → 53 444 | 14 694 → 14 694 |
|
||
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
|
||
| **gold delivered?** | **no, `below_k`** | **no, `below_k`** | **no, `below_k`** |
|
||
|
||
Two of the three payloads are byte-identical before and after: the gold moved a
|
||
long way without reaching the top eight, so the delivered set did not change.
|
||
|
||
**The bonus measurement the order asked for.** Three spellings of the same
|
||
requirement number, on the same bundle:
|
||
|
||
| question spelling | identifier token | gold rank |
|
||
|---|---|---|
|
||
| `Krav 10.2—2` (em dash) | `10.2-2` | 35 |
|
||
| `krav 10.2-2` (hyphen, lower case) | `10.2-2` | 35 |
|
||
| `Krav 10.2–2` (en dash) | `10.2-2` | 35 |
|
||
|
||
All three normalise alike and rank alike.
|
||
|
||
---
|
||
|
||
## 5. The K2 control: the tokeniser alone changes nothing
|
||
|
||
The order required the S7 control question to be compared byte for byte with
|
||
the tokeniser as the only variable, so it was run against ONE unchanged bundle
|
||
with the committed tool and the new tool:
|
||
|
||
| command | before | after |
|
||
|---|---|---|
|
||
| default | `sha256 2ae46f683a3b17fb…`, 169 573 B | **identical** |
|
||
| `--cost-vocabulary --k 12 --limit 160000` | `sha256 8aaf8558369b3d6b…`, 172 588 B | **identical** |
|
||
|
||
Byte for byte, both commands. The two byte counts are the ones published in the
|
||
previous session's report, which is what makes this a control and not a
|
||
coincidence. The corpus does carry identifiers of other shapes (`NS3935:2019`,
|
||
`TEK 17`); measured, the rule leaves both untouched, because a colon is not a
|
||
separator it claims and a space was never one.
|
||
|
||
---
|
||
|
||
## 6. The title axis in the orphan gate
|
||
|
||
Reproduced first, from the previous session's numbers, before the rule was
|
||
touched — all four rows to the integer:
|
||
|
||
| question | vocabulary | O1 measured | reproduced here |
|
||
|---|---|---|---|
|
||
| mandate-shaped | off | 251 of 271 | 251 |
|
||
| mandate-shaped | on | **19** of 280 | **19** |
|
||
| specific | off | 1 of 233 | 1 |
|
||
| specific | on | 1 of 248 | 1 |
|
||
|
||
**The rule.** When a heading is dropped by the orphan check and the candidate
|
||
immediately after it is a table block, the table block takes that heading's
|
||
title *and* its section number instead of the mechanical `Tabell linje <n>`.
|
||
|
||
Three properties of that sentence are each a decision:
|
||
|
||
1. **Conditioned on the drop.** A heading that keeps its own body is still
|
||
carried by a live candidate; copying its title onto the table as well would
|
||
put one name on two concepts and rescue none.
|
||
2. **Immediately after.** The name goes to the adjacent segment or nowhere.
|
||
Carried further, a table several candidates away would take the name of a
|
||
heading it does not contain — an invention rather than a rescue. This is the
|
||
one property no test caught until a surviving mutation exposed it (§ 9).
|
||
3. **Number as well as title.** `_segment_path` reads both — the number becomes
|
||
the directory and is stripped from the stem — so inheriting the title alone
|
||
would emit `3-1-prissammenstilling.md` at the top level, a name the heading
|
||
never had.
|
||
|
||
It is format-agnostic by construction: the orphan check and the table rule are,
|
||
so a sheet name in a spreadsheet and a heading in a PDF go the same way.
|
||
|
||
---
|
||
|
||
## 7. The rebuild: 629 concepts, two ids, a new identity
|
||
|
||
One rebuild for both fixes, `okf build` over the same 43-document folder, run
|
||
from a frozen copy of `src/` so no later edit could reach a measurement already
|
||
in flight.
|
||
|
||
| | value |
|
||
|---|---|
|
||
| `merged + coded rejections == N` | `39 + 4 = 43 = N`, exit 0 |
|
||
| concepts | **629**, unchanged |
|
||
| concept ids that moved | **2 of 629** (0.32 %) |
|
||
| new content identity | `sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b` |
|
||
|
||
**The two ids move BACK.** `…/tabell-linje-3` becomes `…/prissammenstilling-sheet-1`
|
||
and `…/ark1-sheet-1` again — the names they carried in the bundle delivered on
|
||
2026-09-03. Against that delivered bundle, **1 106 of 1 108 files are now
|
||
identical**, and the two that differ are the two spreadsheet bodies the previous
|
||
session deliberately changed. A consumer holding the 2026-09-03 tree gets its
|
||
file names back; the ref, however, is new, and `c26eed6a…` from the previous
|
||
session is superseded and stale.
|
||
|
||
**The ranking, on the rebuilt bundle:**
|
||
|
||
| question | vocabulary | O1 bundle | rebuilt |
|
||
|---|---|---|---|
|
||
| mandate-shaped | off | 251 of 271 | 251 |
|
||
| mandate-shaped | on | 19 of 280 | **10** |
|
||
| specific | off | 1 of 233 | 1 |
|
||
| specific | on | 1 of 248 | **1** |
|
||
|
||
Rank 10 is exactly where the concept sat before the previous session's form fix.
|
||
The regression is closed, and the question this repository's own hit@8
|
||
measurement uses is untouched.
|
||
|
||
**What it now costs to deliver that concept**, which is the figure a consumer
|
||
budgets against:
|
||
|
||
| command | bytes | o200k | priced sheet |
|
||
|---|---|---|---|
|
||
| default | 169 583 | 53 420 | `below_k` |
|
||
| `--cost-vocabulary --k 12` (default budget) | 183 798 | **58 907** | **delivered, rank 10** |
|
||
| `--cost-vocabulary --k 20` (default budget) | 190 438 | 61 955 | delivered, rank 10 |
|
||
| specific question, default | 108 877 | 37 189 | delivered, rank 1 |
|
||
|
||
The previous session's delivering command was `--cost-vocabulary --k 20` at
|
||
**65 912** o200k. The delivering command is now `--cost-vocabulary --k 12` at
|
||
**58 907** — 10.6 % cheaper, and it fits the default budget. `okf_contract_check`
|
||
returns exit 0 with 14 rules and 0 findings on all three payloads, and the
|
||
identity closes: `8 + 621 = 629`, `12 + 617 = 629`.
|
||
|
||
---
|
||
|
||
## 8. What did NOT close, and why — stated rather than glossed
|
||
|
||
**The requirement question is still `below_k` on all three bundles.** The fix
|
||
moved the gold from 160 to 96, 143 to 9 and 100 to 35, which is a large move and
|
||
not a delivery. The remaining blocker was decomposed rather than guessed:
|
||
|
||
`_overlap` returns a COUNT. On the smallest bundle the gold's overlap is 4 out
|
||
of a possible 7, and **70 of 270 concepts also score 4** while 12 score 5.
|
||
Matching the exact requirement number is worth exactly as much as matching the
|
||
verb in `Hva **krever** …` or the noun in `… det sentrale **vilkåret**`. The
|
||
identifier is now visible to the ranker; it is not yet *worth* more than a
|
||
common word.
|
||
|
||
Making it worth more is a weighting decision — a rarity weight, or a declared
|
||
class of high-specificity tokens — and it is a different change from this one:
|
||
it moves every ranking in every corpus, and it has no measurement behind it yet.
|
||
It is not in this order and is not decided here.
|
||
|
||
**And the honesty limit that governs all of the above.** This document measures
|
||
a *ranking*: which concepts a deterministic pre-pass puts in a payload. Whether
|
||
a live model then answers the engineer's question correctly is the consumer's
|
||
measurement, not this one. "Delivered" is not "read".
|
||
|
||
---
|
||
|
||
## 9. Tests, and the mutation that survived
|
||
|
||
Eight new tests, each written red first (1 287 -> 1 295). Six
|
||
mutations, six red, with the unmutated control green after every restore:
|
||
|
||
| mutation | what it breaks | result |
|
||
|---|---|---|
|
||
| widen the identifier back to alphanumeric groups | the slug regression | 3 red |
|
||
| drop the dash fold | one spelling finds, two do not | 2 red |
|
||
| never emit the identifier token | the whole rule | 3 red |
|
||
| inherit the title but not the number | the emitted path | 1 red |
|
||
| inherit onto any rule, not only a table | one name on two concepts | 1 red |
|
||
| never clear the carried name | a distant table takes the name | **survived** |
|
||
|
||
The survivor was read as code before it was called equivalent, per this
|
||
repository's own trap note: it is not equivalent — it lets a heading's name
|
||
travel arbitrarily far — and the band simply had no fixture where an orphan is
|
||
followed by something other than a table. One was added, and the mutation now
|
||
dies.
|
||
|
||
Suite: **1 295 passing** (was 1 287), `ruff` clean, `mypy` clean on 17 source
|
||
files. Both byte-pinned proposer goldens are unchanged, which is what says the
|
||
title rule does not fire where no heading is being destroyed.
|
||
|
||
---
|
||
|
||
## 10. Verification log
|
||
|
||
| # | claim | how it was checked |
|
||
|---|---|---|
|
||
| 1 | the five tokeniser strings | run against the committed tool before any edit; all five identical to the consumer's |
|
||
| 2 | the gold's `withheld` position is not a rank | `sorted(ids) == ids` on all three payloads → `True` |
|
||
| 3 | both mechanisms lose the number | split output printed separately from the length filter |
|
||
| 4 | the wide rule costs a hit@8 row | 5 of 6 → 4 of 6, and the losing row's document score 0.735 → 0.0 |
|
||
| 5 | equality matching is worse | measured on all three bundles; 1 better, 2 worse |
|
||
| 6 | the tokeniser alone is byte-stable on K2 | same bundle, two tools, two commands, identical sha256 |
|
||
| 7 | rank 10 → 19 (O1's regression) | reproduced on O1's own bundles before the rule was touched |
|
||
| 8 | 2 of 629 ids moved | set difference of the two concept-id sets |
|
||
| 9 | 1 106 of 1 108 files identical to the delivered bundle | `diff -rq`, and `find -type f | wc -l` for the denominator |
|
||
| 10 | the rebuild reads frozen source | `import llm_ingestion_okf; __file__` under the build's own `PYTHONPATH` |
|
||
| 11 | contract conformance | `okf_contract_check.py` exit 0, 14 rules, 0 findings, on six payloads |
|