Kjell Tore Guttormsen
47f6619e6f
K3 round 3, order 20260908T170037Z-3622420612-from-.claude, plus the PM injection of 17:55Z carrying the operator's D3 answer. TWO NEW RULES, both OFF by default. `--sheet-section-rows` cuts an open table block at the rows that label its sections: a RUN of at least three rows whose first cell is a bare numeric label. It is the first rule here that opens a span INSIDE a table, and the opposite direction from Arm E -- that arm decides how far a block extends, this decides where it is cut inside. The spreadsheet is the one file type no arm had ever moved: 1 concept -> 12 on the priced sheet, of which 11 are the reference's 11 cost groups. The run guard buys exactly three candidates on the corpus (the rows stating a computation basis), and each section carries its own `source_sheet` + `source_rows`, verified on the artifact. `--drop-wrapped-outline` declines an outline candidate whose line continues onto the next one. Round 2 falsified the LENGTH axis on this case; this one is not about size. Over the 12-document sample it separates 8 of 34 candidates -- the quoted regulation paragraphs and the risk-table rows -- and none of the 26 the operator kept. Position 4: 4 concepts -> 1, the reference. The cell is `treffer` by count and lossy by bytes, and the report says so. Whole corpus, one cached extraction: 1 of 39 documents changes under the first rule (0 of 32 pdf, 0 of 5 docx), 5 of 39 under the second (all pdf). Arm B and F2 are byte-identical against a frozenc3b645b, both halves by `diff -r`. THE `okf build` DEFAULT MOVED, and this is the operator's decision executed, not a side effect: no flag now means `--outline-run 3 --unit-fold`. Each arm keeps an explicit opt-out (`--outline-run 0`, `--no-unit-fold`) and the pair reproduces the pre-move bytes exactly. The proposer's own defaults do NOT move: they are what the goldens and every published reproduction block are pinned to, so the two layers now disagree on purpose and `cli.py` says where. Two shipped expectations moved with it and both are stated in place. MEASURED AND REPORTED BACK: the configuration made default scores 2 of 12 on the unit worksheet, `docx` 0 of 3 -- Arm F's published 5 of 12 was measured with `--table-grid` ON, and without it the fold has no joined table to fold. Position 1 is declined for the second round on the axis the order named: 3 of 3 recovered chapters have body under them, so "heading-on-heading" separates 0 of 3. The K2 ranking control round 2 could not measure: two bundles from frozenc3b645bdiffering only in `--keep-table-heading`, 2 of 1108 files apart. The priced concept ranks 96 of 629 in BOTH and is delivered at rank 10 in both, so the flag buys 35 payload bytes and zero rank positions. Round 2's prediction is falsified: the concept's NAME was already restored by the orphan-name inheritance; the flag adds the heading LINE. The stale S7 literal is restated where it stood (`c759a657...`, 171 614 B, re-measured on this HEAD), so the next order cannot inherit the superseded one. 1397 -> 1414 tests. Report: docs/2026-09-08-k3-runde3-per-filtype.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
343 lines
16 KiB
Markdown
343 lines
16 KiB
Markdown
# The requirement number the pre-pass could not see, and the title the orphan gate destroyed, 2026-09-08
|
||
|
||
Two consumer-reported defects, one session, one rebuild. The first is a
|
||
consumption-side defect reported by `vegnormal-okf`: `okf_consume`'s tokeniser
|
||
shatters a requirement number, so the one concept an engineer names is withheld.
|
||
The second is a production-side defect this repository measured on itself in the
|
||
previous session: a table that opens directly under a heading destroys that
|
||
heading's name.
|
||
|
||
---
|
||
|
||
## 0. What IS measured, and what is NOT
|
||
|
||
**Measured.** The five tokeniser strings the consumer published, reproduced
|
||
here before anything was changed; where the number is lost, decomposed to both
|
||
mechanisms; the gold concept's rank on three real bundles, before and after, with
|
||
denominators; one alternative rule falsified against the same three bundles; the
|
||
K2 control question as a payload digest with the tokeniser as the only variable;
|
||
the title axis reproduced from the previous session's number before it was
|
||
touched; a full K2 rebuild with the conservation identity, the concept count, the
|
||
number of ids that moved against a denominator of 629, and the new content
|
||
identity; and six mutations against the new tests.
|
||
|
||
**Known-positive, run first.** Nine figures published elsewhere were reproduced
|
||
on this machine before any of the after-columns were read:
|
||
|
||
| # | known-positive | source | measured here |
|
||
|---|---|---|---|
|
||
| 1 | `normalise('Krav 10.2—2')` → `['krav']` | consumer § 3.1 | `('krav',)` |
|
||
| 2 | `normalise('10.2-2')` → `[]` | consumer § 3.1 | `()` |
|
||
| 3 | `normalise('3.3.1—13')` → `[]` | consumer § 3.1 | `()` |
|
||
| 4 | `'N500'` → `['n500']`, `'abc123'` → `['abc123']` | consumer § 3.1 | identical |
|
||
| 5 | considered / withheld / delivered per bundle | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
|
||
| 6 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
|
||
| 7 | payload tokens (o200k) per bundle | consumer § 3 | 22 138, 53 444, 14 694 |
|
||
| 8 | priced concept candidate rank 10 → 19 | this repo, 2026-09-08 (O1) § 7 | 10 → 19, and 251 / 1 / 1 for the other three rows |
|
||
| 9 | K2 payload bytes, two commands | this repo, O1 § 7 | 169 573 B and 172 588 B |
|
||
|
||
> **Superseded literal (added 2026-09-08, round 3).** The S7 control figures in
|
||
> this document were correct when it was written and are not the ones to reuse:
|
||
> `a37d5ce`, `17c49fc` and `c95d189` each widened the excerpt form afterwards.
|
||
> The current known-positive for the S7 control question under the default
|
||
> command is **`c759a657dbb41d71`, 171 614 B**, measured on `c3b645b` against
|
||
> the same bundle. `docs/2026-09-08-k3-runde3-per-filtype.md` § 4.
|
||
|
||
**NOT measured.** Whether a live model answers *better* with the named
|
||
requirement in the payload. This document measures a ranking; the consumer
|
||
measures the answer. Also not measured: that either rule helps a corpus other
|
||
than the four read here.
|
||
|
||
**Not closed.** The gold requirement is still `below_k` on all three bundles
|
||
after the fix. § 8 states why, with the number.
|
||
|
||
---
|
||
|
||
## 1. The five strings, and one correction to how they are read
|
||
|
||
```
|
||
PYTHONPATH=… python3 -c "from okf_consume import normalise; …"
|
||
'Krav 10.2—2' -> ('krav',) 'N500' -> ('n500',)
|
||
'10.2-2' -> () 'abc123' -> ('abc123',)
|
||
'3.3.1—13' -> () '10' -> ()
|
||
'2.9.2—12' -> () '2' -> ()
|
||
```
|
||
|
||
All five reproduce, and so do the two known-positives. So does the gold's place
|
||
in the `withheld` list: 119 of 438, 204 of 1125, 66 of 262.
|
||
|
||
**That place is a list position and not a rank, and the difference matters.**
|
||
The `withheld` array is emitted in `concept_id` order — measured, not assumed
|
||
(`sorted(ids) == ids` is `True` on all three payloads) — so it cannot move when
|
||
the ranking moves. It is a correct number that answers a different question. The
|
||
instrument that does move is the fused rank, and § 4 uses it.
|
||
|
||
One more figure the consumer's mechanism claim predicts, confirmed here: of the
|
||
262 withheld entries on the smallest bundle, **262 are `below_k` and none is
|
||
`no_lexical_match`**. Every concept matches, because every concept carries the
|
||
word the question carries.
|
||
|
||
---
|
||
|
||
## 2. Where the number is lost: both mechanisms, not one
|
||
|
||
The order asked which of the two is responsible. **Both**, and neither alone
|
||
would lose it:
|
||
|
||
```
|
||
'10.2-2' split-> ['10', '2', '2'] kept-> []
|
||
'3.3.1—13' split-> ['3', '3', '1', '13'] kept-> []
|
||
'R610.4' split-> ['r610', '4'] kept-> ['r610']
|
||
```
|
||
|
||
`_TOKEN_SPLIT_RE` (`tools/okf_consume.py:523` before this change) splits on
|
||
every non-alphanumeric character, which destroys the identifier; the
|
||
`MIN_TOKEN_LENGTH` filter in `normalise` (line 537) then removes the digit runs
|
||
that are left. Lowering the floor alone would not recover the number — it would
|
||
return `10`, `2` and `2`, which match every page number in the corpus. The
|
||
identifier has to survive the split.
|
||
|
||
---
|
||
|
||
## 3. The rule, and the alternative that was falsified
|
||
|
||
**The rule.** Numeric groups joined by `.` or `-` are held together as one
|
||
token, with a leading letter run allowed only where it touches its digits
|
||
without a separator. Dash variants fold to the ASCII hyphen first, so the same
|
||
number written with an em dash, an en dash or a hyphen is one token.
|
||
`MIN_TOKEN_LENGTH` is unchanged; the identifier is exempt from it by
|
||
construction rather than by lowering it for everyone.
|
||
|
||
**It was narrowed by a measurement, not by taste.** The first version joined
|
||
*alphanumeric* groups across a separator. That version swallowed a whole
|
||
document slug — a slug of the form `…-3-6-…` became ONE token — and that
|
||
document's stage-one score for a question naming its subject fell from **0.735
|
||
to 0.0**, costing one row of the repository's own hit@8 measurement. The
|
||
narrowed rule is purely additive on such a slug: the identifier is added, the
|
||
words stay.
|
||
|
||
**The falsified alternative.** Because `tokens_match` compares a four-character
|
||
prefix, `10.2-2` matches `10.2-5`: on the smallest bundle exactly **5 of 270**
|
||
concept titles match the question's identifier, and all five are the same
|
||
section. Making identifier tokens match only on *equality* is the obvious
|
||
narrowing, and it was measured before being adopted or rejected:
|
||
|
||
| | prefix (shipped) | equality (probe) |
|
||
|---|---|---|
|
||
| N100:2023 | 96 | **53** |
|
||
| N200:2024 | **9** | 26 |
|
||
| N500:2024 | **35** | 37 |
|
||
|
||
It improves one bundle and makes two worse. **Falsified; not built.**
|
||
|
||
---
|
||
|
||
## 4. The three bundles, before and after
|
||
|
||
Command, one per bundle, the questions quoted verbatim from the consumer's § 3:
|
||
|
||
```
|
||
python3 tools/okf_consume.py <bundle> \
|
||
--question "Hva krever Krav 10.2—2 i N500? Gjengi det sentrale vilkåret."
|
||
```
|
||
|
||
| | N100:2023 | N200:2024 | N500:2024 |
|
||
|---|---|---|---|
|
||
| concepts | 446 | 1 133 | 270 |
|
||
| gold fused rank, **before** | 160 | 143 | 100 |
|
||
| gold fused rank, **after** | **96** | **9** | **35** |
|
||
| gold lexical overlap, before → after | 3 → 4 | 4 → 6 | 3 → 4 |
|
||
| considered / withheld / delivered | 446/438/8 | 1133/1125/8 | 270/262/8 |
|
||
| budget spent, before → after | 8 977 → 12 095 | 17 818 → 17 818 | 10 517 → 10 517 |
|
||
| payload o200k, before → after | 22 138 → 23 583 | 53 444 → 53 444 | 14 694 → 14 694 |
|
||
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
|
||
| **gold delivered?** | **no, `below_k`** | **no, `below_k`** | **no, `below_k`** |
|
||
|
||
Two of the three payloads are byte-identical before and after: the gold moved a
|
||
long way without reaching the top eight, so the delivered set did not change.
|
||
|
||
**The bonus measurement the order asked for.** Three spellings of the same
|
||
requirement number, on the same bundle:
|
||
|
||
| question spelling | identifier token | gold rank |
|
||
|---|---|---|
|
||
| `Krav 10.2—2` (em dash) | `10.2-2` | 35 |
|
||
| `krav 10.2-2` (hyphen, lower case) | `10.2-2` | 35 |
|
||
| `Krav 10.2–2` (en dash) | `10.2-2` | 35 |
|
||
|
||
All three normalise alike and rank alike.
|
||
|
||
---
|
||
|
||
## 5. The K2 control: the tokeniser alone changes nothing
|
||
|
||
The order required the S7 control question to be compared byte for byte with
|
||
the tokeniser as the only variable, so it was run against ONE unchanged bundle
|
||
with the committed tool and the new tool:
|
||
|
||
| command | before | after |
|
||
|---|---|---|
|
||
| default | `sha256 2ae46f683a3b17fb…`, 169 573 B | **identical** |
|
||
| `--cost-vocabulary --k 12 --limit 160000` | `sha256 8aaf8558369b3d6b…`, 172 588 B | **identical** |
|
||
|
||
Byte for byte, both commands. The two byte counts are the ones published in the
|
||
previous session's report, which is what makes this a control and not a
|
||
coincidence. The corpus does carry identifiers of other shapes (`NS3935:2019`,
|
||
`TEK 17`); measured, the rule leaves both untouched, because a colon is not a
|
||
separator it claims and a space was never one.
|
||
|
||
---
|
||
|
||
## 6. The title axis in the orphan gate
|
||
|
||
Reproduced first, from the previous session's numbers, before the rule was
|
||
touched — all four rows to the integer:
|
||
|
||
| question | vocabulary | O1 measured | reproduced here |
|
||
|---|---|---|---|
|
||
| mandate-shaped | off | 251 of 271 | 251 |
|
||
| mandate-shaped | on | **19** of 280 | **19** |
|
||
| specific | off | 1 of 233 | 1 |
|
||
| specific | on | 1 of 248 | 1 |
|
||
|
||
**The rule.** When a heading is dropped by the orphan check and the candidate
|
||
immediately after it is a table block, the table block takes that heading's
|
||
title *and* its section number instead of the mechanical `Tabell linje <n>`.
|
||
|
||
Three properties of that sentence are each a decision:
|
||
|
||
1. **Conditioned on the drop.** A heading that keeps its own body is still
|
||
carried by a live candidate; copying its title onto the table as well would
|
||
put one name on two concepts and rescue none.
|
||
2. **Immediately after.** The name goes to the adjacent segment or nowhere.
|
||
Carried further, a table several candidates away would take the name of a
|
||
heading it does not contain — an invention rather than a rescue. This is the
|
||
one property no test caught until a surviving mutation exposed it (§ 9).
|
||
3. **Number as well as title.** `_segment_path` reads both — the number becomes
|
||
the directory and is stripped from the stem — so inheriting the title alone
|
||
would emit `3-1-prissammenstilling.md` at the top level, a name the heading
|
||
never had.
|
||
|
||
It is format-agnostic by construction: the orphan check and the table rule are,
|
||
so a sheet name in a spreadsheet and a heading in a PDF go the same way.
|
||
|
||
---
|
||
|
||
## 7. The rebuild: 629 concepts, two ids, a new identity
|
||
|
||
One rebuild for both fixes, `okf build` over the same 43-document folder, run
|
||
from a frozen copy of `src/` so no later edit could reach a measurement already
|
||
in flight.
|
||
|
||
| | value |
|
||
|---|---|
|
||
| `merged + coded rejections == N` | `39 + 4 = 43 = N`, exit 0 |
|
||
| concepts | **629**, unchanged |
|
||
| concept ids that moved | **2 of 629** (0.32 %) |
|
||
| new content identity | `sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b` |
|
||
|
||
**The two ids move BACK.** `…/tabell-linje-3` becomes `…/prissammenstilling-sheet-1`
|
||
and `…/ark1-sheet-1` again — the names they carried in the bundle delivered on
|
||
2026-09-03. Against that delivered bundle, **1 106 of 1 108 files are now
|
||
identical**, and the two that differ are the two spreadsheet bodies the previous
|
||
session deliberately changed. A consumer holding the 2026-09-03 tree gets its
|
||
file names back; the ref, however, is new, and `c26eed6a…` from the previous
|
||
session is superseded and stale.
|
||
|
||
**The ranking, on the rebuilt bundle:**
|
||
|
||
| question | vocabulary | O1 bundle | rebuilt |
|
||
|---|---|---|---|
|
||
| mandate-shaped | off | 251 of 271 | 251 |
|
||
| mandate-shaped | on | 19 of 280 | **10** |
|
||
| specific | off | 1 of 233 | 1 |
|
||
| specific | on | 1 of 248 | **1** |
|
||
|
||
Rank 10 is exactly where the concept sat before the previous session's form fix.
|
||
The regression is closed, and the question this repository's own hit@8
|
||
measurement uses is untouched.
|
||
|
||
**What it now costs to deliver that concept**, which is the figure a consumer
|
||
budgets against:
|
||
|
||
| command | bytes | o200k | priced sheet |
|
||
|---|---|---|---|
|
||
| default | 169 583 | 53 420 | `below_k` |
|
||
| `--cost-vocabulary --k 12` (default budget) | 183 798 | **58 907** | **delivered, rank 10** |
|
||
| `--cost-vocabulary --k 20` (default budget) | 190 438 | 61 955 | delivered, rank 10 |
|
||
| specific question, default | 108 877 | 37 189 | delivered, rank 1 |
|
||
|
||
The previous session's delivering command was `--cost-vocabulary --k 20` at
|
||
**65 912** o200k. The delivering command is now `--cost-vocabulary --k 12` at
|
||
**58 907** — 10.6 % cheaper, and it fits the default budget. `okf_contract_check`
|
||
returns exit 0 with 14 rules and 0 findings on all three payloads, and the
|
||
identity closes: `8 + 621 = 629`, `12 + 617 = 629`.
|
||
|
||
---
|
||
|
||
## 8. What did NOT close, and why — stated rather than glossed
|
||
|
||
**The requirement question is still `below_k` on all three bundles.** The fix
|
||
moved the gold from 160 to 96, 143 to 9 and 100 to 35, which is a large move and
|
||
not a delivery. The remaining blocker was decomposed rather than guessed:
|
||
|
||
`_overlap` returns a COUNT. On the smallest bundle the gold's overlap is 4 out
|
||
of a possible 7, and **70 of 270 concepts also score 4** while 12 score 5.
|
||
Matching the exact requirement number is worth exactly as much as matching the
|
||
verb in `Hva **krever** …` or the noun in `… det sentrale **vilkåret**`. The
|
||
identifier is now visible to the ranker; it is not yet *worth* more than a
|
||
common word.
|
||
|
||
Making it worth more is a weighting decision — a rarity weight, or a declared
|
||
class of high-specificity tokens — and it is a different change from this one:
|
||
it moves every ranking in every corpus, and it has no measurement behind it yet.
|
||
It is not in this order and is not decided here.
|
||
|
||
**And the honesty limit that governs all of the above.** This document measures
|
||
a *ranking*: which concepts a deterministic pre-pass puts in a payload. Whether
|
||
a live model then answers the engineer's question correctly is the consumer's
|
||
measurement, not this one. "Delivered" is not "read".
|
||
|
||
---
|
||
|
||
## 9. Tests, and the mutation that survived
|
||
|
||
Eight new tests, each written red first (1 287 -> 1 295). Six
|
||
mutations, six red, with the unmutated control green after every restore:
|
||
|
||
| mutation | what it breaks | result |
|
||
|---|---|---|
|
||
| widen the identifier back to alphanumeric groups | the slug regression | 3 red |
|
||
| drop the dash fold | one spelling finds, two do not | 2 red |
|
||
| never emit the identifier token | the whole rule | 3 red |
|
||
| inherit the title but not the number | the emitted path | 1 red |
|
||
| inherit onto any rule, not only a table | one name on two concepts | 1 red |
|
||
| never clear the carried name | a distant table takes the name | **survived** |
|
||
|
||
The survivor was read as code before it was called equivalent, per this
|
||
repository's own trap note: it is not equivalent — it lets a heading's name
|
||
travel arbitrarily far — and the band simply had no fixture where an orphan is
|
||
followed by something other than a table. One was added, and the mutation now
|
||
dies.
|
||
|
||
Suite: **1 295 passing** (was 1 287), `ruff` clean, `mypy` clean on 17 source
|
||
files. Both byte-pinned proposer goldens are unchanged, which is what says the
|
||
title rule does not fire where no heading is being destroyed.
|
||
|
||
---
|
||
|
||
## 10. Verification log
|
||
|
||
| # | claim | how it was checked |
|
||
|---|---|---|
|
||
| 1 | the five tokeniser strings | run against the committed tool before any edit; all five identical to the consumer's |
|
||
| 2 | the gold's `withheld` position is not a rank | `sorted(ids) == ids` on all three payloads → `True` |
|
||
| 3 | both mechanisms lose the number | split output printed separately from the length filter |
|
||
| 4 | the wide rule costs a hit@8 row | 5 of 6 → 4 of 6, and the losing row's document score 0.735 → 0.0 |
|
||
| 5 | equality matching is worse | measured on all three bundles; 1 better, 2 worse |
|
||
| 6 | the tokeniser alone is byte-stable on K2 | same bundle, two tools, two commands, identical sha256 |
|
||
| 7 | rank 10 → 19 (O1's regression) | reproduced on O1's own bundles before the rule was touched |
|
||
| 8 | 2 of 629 ids moved | set difference of the two concept-id sets |
|
||
| 9 | 1 106 of 1 108 files identical to the delivered bundle | `diff -rq`, and `find -type f | wc -l` for the denominator |
|
||
| 10 | the rebuild reads frozen source | `import llm_ingestion_okf; __file__` under the build's own `PYTHONPATH` |
|
||
| 11 | contract conformance | `okf_contract_check.py` exit 0, 14 rules, 0 findings, on six payloads |
|