llm-ingestion-okf/docs
Kjell Tore Guttormsen 56c1205ec4 fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table
Two consumer-reported defects, one rebuild.

The pre-pass could not see a requirement number: `_TOKEN_SPLIT_RE` split
`10.2-2` into digit runs and `MIN_TOKEN_LENGTH` removed them, so a question
naming a requirement reached the ranker carrying only the word every concept
in a standards bundle carries. Measured on three real bundles (446, 1133 and
270 concepts), the named requirement was withheld `below_k` in three of three.
Numeric groups joined by `.` or `-` are now held together, dash variants fold
to the ASCII hyphen, and the noise floor is unchanged. The gold moves from
160 to 96, 143 to 9 and 100 to 35 -- a large move, and NOT a delivery: it is
still `below_k`, because `_overlap` is a count and an exact requirement number
is worth no more than a common verb. That weighting is a separate decision.

The rule was narrowed by a measurement: a version that joined alphanumeric
groups swallowed a document slug whole and cost a hit@8 row. An equality-only
variant was measured on all three bundles and falsified -- better on one,
worse on two.

The orphan gate destroyed a heading's name: a table opening directly below a
heading left that heading with an empty body, the orphan check dropped it, and
the surviving table block kept the mechanical `Tabell linje <n>`. A table that
orphans its heading now takes that heading's title and section number.
Conditioned on the drop, on adjacency, and carrying both members -- each of
the three measured or mutation-tested.

One K2 rebuild for both, from a frozen source tree: 629 concepts, `39 + 4 = 43
= N`, 2 of 629 ids moved and both moved BACK to the names the 2026-09-03
bundle carried, 1106 of 1108 files identical to it. New ref
sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b
-- c26eed6a... is superseded. The regression the previous session measured is
closed: candidate rank 19 -> 10, and the delivering command is now
`--cost-vocabulary --k 12` inside the default budget at 58 907 o200k against
65 912 before. The specific question is unmoved at rank 1.

The tokeniser alone leaves the K2 control question byte-identical, measured
with the bundle held fixed and both published byte counts reproduced.

Report: docs/2026-09-08-kravnummer-tokenisering.md. 8 new tests, red first;
6 mutations, 6 red, one of them only after the survivor was read as code and
a missing fixture was added. Suite 1287 -> 1295.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-08 11:43:39 +02:00
..
plan fix(corpus): the root index links the bundle's own log 2026-09-03 20:23:56 +02:00
2026-08-21-g2-pdf-extraction-measurement.md docs(extract): measure one Vegnormalene PDF page against the extraction registry 2026-08-21 19:16:15 +02:00
2026-08-29-konverteringsarmen-arm-a.md docs(measure): run the conversion arm on K2 - pandoc wins both office formats 2026-08-30 00:41:48 +02:00
2026-08-29-telling-offentlig-prosjektkorpus.md docs(measure): count the public project corpus - K2 measured, K1 blocked 2026-08-29 07:03:50 +02:00
2026-09-02-k3-k4-k5-baseline.md docs(measurement): K3, K4 and K5 numbers against the committed method 2026-09-02 15:55:19 +02:00
2026-09-02-k3-k4-k5-metode.md docs(measurement): K3, K4 and K5 method with n declared before any verdict 2026-09-02 15:46:03 +02:00
2026-09-02-korpuskjoring-k1-k2.md docs(measurement): K1 and K2 over the K2 corpus 2026-09-02 15:43:45 +02:00
2026-09-03-k2-bundle-rebuild.md test(corpus): a rebuild with log.md present reproduces the bundle exactly 2026-09-03 05:16:25 +02:00
2026-09-04-cid-andel-k2.md docs(measure): CID-glyph share across K2 -- Bilag 9.1 alone, not a K1 arm 2026-09-04 20:23:41 +02:00
2026-09-04-k2-pptx-odt-rtf.md docs(measure): K2 on pptx/odt/rtf -- denominator 0 in both named corpora 2026-09-04 21:12:41 +02:00
2026-09-04-k3-arm-c.md docs(measure): K3 with Arm C beside the baseline, two numbers, no threshold 2026-09-04 17:58:50 +02:00
2026-09-07-k2-pptx-odt-rtf-fixtures.md test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote 2026-09-07 05:22:12 +02:00
2026-09-07-k3-arm-d.md docs(measure): K3 Arm D beside a re-rated Arm B -- reach and no threshold 2026-09-07 03:04:01 +02:00
2026-09-07-k3-arm-e.md docs(measure): K3 Arm E beside Arm B and Arm D -- three rows, no threshold 2026-09-07 11:48:58 +02:00
2026-09-07-k3-enhetsark-metode.md docs(measure): K3 unit worksheet delivered for operator, no code changed 2026-09-07 16:36:16 +02:00
2026-09-07-okf-konsumskill-maaling.md docs(consume): refutation sweep — smoothed-IDF rarity weighting cannot zero both known-negatives 2026-09-07 09:56:36 +02:00
2026-09-08-blindsone-below-k-k2.md feat(consume): measure the below-k blind spot, add one flag-gated vocabulary bridge 2026-09-08 04:55:45 +02:00
2026-09-08-blindsone-laas2-budsjett-k2.md feat(consume): measure the budget lock, add one flag-gated top-rank reservation 2026-09-08 05:18:30 +02:00
2026-09-08-kravnummer-tokenisering.md fix(consume,propose): hold an identifier number as one token, give an orphaned heading's name to its table 2026-09-08 11:43:39 +02:00
2026-09-08-prisform-og-loggen-k2.md fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index 2026-09-08 10:06:58 +02:00
consumption-contract.md docs(contract): the generic consumption contract, stated normatively 2026-09-02 16:05:11 +02:00
okf-nokkelinventar.md build(deps): pin llm-ingestion-guard v1.3.0 so the gate reads our own goldens 2026-09-03 20:41:03 +02:00
phase-3-split-table.md docs(phase-3): keep the index-entry grammar on one line 2026-08-04 12:37:02 +02:00
upstream-okf-upgrade-runbook.md feat(watch): weekly OKF upstream watch that can prove it found nothing 2026-08-23 20:38:37 +02:00