Kjell Tore Guttormsen
a37d5ced38
A question naming a requirement number now delivers that requirement at rank 1
on all three vegnormal bundles (was 96, 9, 35 of 446, 1 133, 270). Two
mechanisms, both measured, both default because no published figure moves.
The matcher: `tokens_match` compared four leading characters, so the unique
identifier `3.3.1-13` read as 135 of 446 common and the rarity weight ranked a
common adjective above the number naming the document. An identifier now
matches by equality alone; df falls to 1/1/1. Words keep the prefix rule, which
was measured for Norwegian compounds. Equality has no floor either, so a
three-character identifier stops matching nothing at all -- measured, `9.2`
reached 0 concepts while sitting verbatim in one title.
The lookup: a question carrying an identifier that sits verbatim in a concept's
title or id is answered by a partition over the fusion's output, not by a
fourth signal. The form was chosen by measurement -- a fourth RRF signal was
simulated first and put the gold at rank 26 / 15 / 19, none of them delivered,
because RRF consumes ranks only and one signal contributes at most 1/(RRF_K+1).
No frontmatter key list is declared: of 1 846 concepts carrying `req_number`,
1 846 also carry that identifier in the title.
The matcher alone is NOT a monotone win (N200 9 -> 26, because that gold's body
cross-references a neighbouring number that the prefix rule counted as a hit on
the question's). Only the partition delivers; the table is in the record.
Consumer corpus: every named control byte-identical against a frozen
`git archive` copy of 116d3e1 -- four payload digests, eight candidate ranks,
six hit@8 rows, both known-negatives. One document that was withheld at
position 621 of 621 is now delivered at rank 1, on a corpus with no
requirement-number grammar at all.
13 tests (12 red before the rules existed), 7 mutations, 7 red. 1 320 passed.
Consumption-side only; no bundle ref moves.
Record: docs/2026-09-08-eksakt-oppslag.md
Co-Authored-By: Claude <claude-opus-5>
344 lines
18 KiB
Markdown
344 lines
18 KiB
Markdown
# The exact lookup: naming a document is not searching for it, 2026-09-08
|
|
|
|
Two sessions made a requirement number visible to the consumption pre-pass and
|
|
then made it *worth* more than a common word, and neither delivered the
|
|
document the question named. The first left the gold `below_k` on all three
|
|
bundles (`docs/2026-09-08-kravnummer-tokenisering.md` § 8); the second
|
|
delivered one of three and decomposed the other two into a **matcher**
|
|
decision and a **fusion** decision, with no measurement behind either
|
|
(`docs/2026-09-08-sjeldenhetsvekt.md` § 3, § 4).
|
|
|
|
This session takes both decisions with numbers. The result is that a question
|
|
naming a requirement number now delivers that requirement at **rank 1 on 3 of
|
|
3 bundles**, and that every published figure on the consumer corpus is
|
|
**byte-identical**.
|
|
|
|
---
|
|
|
|
## 0. What IS measured, and what is NOT
|
|
|
|
**Measured.** Eleven known-positive figures reproduced before anything was
|
|
changed; document frequency under the matcher and under equality on three
|
|
bundles; the gold's fused rank, verdict, payload digest and byte count before,
|
|
after the matcher alone, and after both mechanisms; the three spellings of one
|
|
identifier; every named control on the consumer corpus against a frozen
|
|
`git archive` copy of `116d3e1`; a control question on that corpus that *does*
|
|
carry an identifier, before and after; hit@8 over the six published questions
|
|
and both known-negatives; the cost of each new pass; seven mutations.
|
|
|
|
**Known-positive, run first.** Every figure this session builds on was
|
|
reproduced on this machine, with the committed tool, on a clean tree at
|
|
`116d3e1`, before any after-column was read:
|
|
|
|
| # | known-positive | source | measured here |
|
|
|---|---|---|---|
|
|
| 1 | gold fused rank per bundle, weight off | previous session § 2 | **96, 9, 35** |
|
|
| 2 | gold fused rank per bundle, weight on | previous session § 2 | **103, 8, 35** |
|
|
| 3 | `df` under the matcher vs under equality | previous session § 3 | **135 / 17 / 5** vs **1 / 1 / 1** |
|
|
| 4 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
|
|
| 5 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
|
|
| 6 | payload `sha256`, weight off | previous session § 2 | `49e9c718…`, `adc2eac7…`, `21c5bbfe…` |
|
|
| 7 | payload `sha256`, weight on | previous session § 2 | `ba09a56a…`, `403ee989…`, `1758f6ca…` |
|
|
| 8 | consumer corpus, the good question's candidate rank, four rows | previous session § 5 | **1 / 1 / 1 / 1** |
|
|
| 9 | the priced sheet's candidate rank, four rows | previous session § 5, § 7 | **251 / 10 / 78 / 2** |
|
|
| 10 | the mandate question's default payload | previous session § 7 | **169 583 B** |
|
|
| 11 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6**, five of them at rank 1 |
|
|
|
|
**NOT measured**, named rather than glossed:
|
|
|
|
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
|
|
machine and this repository's single-runtime-dependency rule does not admit
|
|
one to produce a caption. Figures below are in `BUDGET_UNIT` — utf-8 bytes of
|
|
emitted JSON — plus the payload `sha256`, which is exact where a byte count
|
|
is only indicative.
|
|
- **The rebuilt consumer bundle of `56c1205` is not on disk.** Its content
|
|
identity (`sha256-tree:2f82fcfe…`) names a tree this machine no longer holds,
|
|
and neither does the bundle the two S7 byte figures in the order come from
|
|
(`2ae46f68…` 169 573 B and `8aaf8558…` 172 588 B). **Those two figures are
|
|
therefore not reproduced here.** The control corpus used instead is the
|
|
bundle delivered 2026-09-03 (`sha256-tree:9a4e5561…`), on which rows 8, 9 and
|
|
10 of the table above reproduce the published values exactly. This costs the
|
|
decision nothing: what the decision needs is byte-*identity* before and
|
|
after, and that is measured on one and the same bundle, against a frozen copy
|
|
of the committed code rather than against a remembered number.
|
|
- **Whether a live model answers better.** This measures a *ranking*.
|
|
"Delivered" is not "read", and that measurement belongs to the consumer.
|
|
|
|
---
|
|
|
|
## 1. Mechanism (a): an identifier matches by equality, and by nothing else
|
|
|
|
`tokens_match` compared four leading characters, for a reason that was measured
|
|
and still holds: Norwegian compounds do not match token-exactly, and
|
|
`vare|ne` and `vare|mottak` share a stem. A requirement number has no stem.
|
|
Four leading characters of `3.3.1-13` are four leading characters of every
|
|
requirement in section 3.3, so a **unique** identifier read as **135 of 446**
|
|
common — and the rarity weight then faithfully ranked a common adjective as
|
|
more informative than the number naming the document.
|
|
|
|
The rule, whole:
|
|
|
|
```python
|
|
if is_identifier(left) or is_identifier(right):
|
|
return left == right
|
|
```
|
|
|
|
`is_identifier` is `_IDENTIFIER_RE.fullmatch`, the pattern the tokeniser
|
|
already uses to keep an identifier whole. No new constant, no token class
|
|
declared anywhere, and words are untouched.
|
|
|
|
**Red first.** `tokens_match('3.3.1-13', '3.3.1-14')` was `True` and is now
|
|
`False`; `('3.3.1-13', '3.3.1-13')` was and is `True`. The known-positive ran
|
|
in the same file and passed before and after: `brannsikring`/`brannvern`,
|
|
`prisene`/`prissammenstilling`, `varene`/`varemottak` all still match, and
|
|
`brann`/`bygg` still does not.
|
|
|
|
**`df` after, same instrument as before:**
|
|
|
|
| bundle | identifier | `df` before | `df` after | `df` exact | concepts |
|
|
|---|---|---|---|---|---|
|
|
| N100 | `3.3.1-13` | **135** | **1** | 1 | 446 |
|
|
| N200 | `2.9.2-12` | 17 | **1** | 1 | 1 133 |
|
|
| N500 | `10.2-2` | 5 | **1** | 1 | 270 |
|
|
|
|
**A short identifier stops being invisible.** `MIN_SHARED_PREFIX` made a
|
|
three-character identifier match *nothing, not even itself*: measured on the
|
|
629-concept consumer bundle, `9.2` reached **0** concepts under the matcher
|
|
while sitting verbatim in one title. Equality has no floor, so it now reaches
|
|
that one. The floor stays where it was measured — for words.
|
|
|
|
**What the matcher alone bought, and what it cost.** This is the honest column,
|
|
and it is not all in one direction:
|
|
|
|
| bundle | gold fused rank, before | after (a) alone | with `--rarity-weight`, before → after |
|
|
|---|---|---|---|
|
|
| N100 | 96 | **53** | 103 → **45** |
|
|
| N200 | 9 | **26** | 8 → **23** |
|
|
| N500 | 35 | **37** | 35 → **36** |
|
|
|
|
**N200 got worse, and the reason is measured rather than guessed.** That gold's
|
|
body carries **88** identifiers, three of them occurrences of `2.9.2-1` — a
|
|
cross-reference to the neighbouring requirement. Under the prefix rule that
|
|
cross-reference answered the question's `2.9.2-12`, so the gold was scoring a
|
|
hit on its own body that it had not earned. Removing a false positive that
|
|
happened to help is still removing a false positive; the number is here so that
|
|
nobody has to take that on faith. On its own, mechanism (a) delivers **0 of 3**.
|
|
|
|
---
|
|
|
|
## 2. Mechanism (b): the form was chosen by a measurement, not by a preference
|
|
|
|
A question carrying an identifier that sits verbatim in a concept's title or id
|
|
is a **lookup**: the reader already knows which document they want and is
|
|
spelling its number. The order named two candidate forms and invited a third.
|
|
|
|
**Form (i), a fourth signal inside the RRF fusion, was simulated first** on the
|
|
same three bundles, on top of mechanism (a), with the lookup as a 0/1 signal:
|
|
|
|
| bundle | gold rank with a fourth RRF signal | of |
|
|
|---|---|---|
|
|
| N100 | **26** | 446 |
|
|
| N200 | **15** | 1 133 |
|
|
| N500 | **19** | 270 |
|
|
|
|
**None of them delivered**, and the reason is the one the previous session
|
|
already wrote down: RRF consumes **ranks only**, so any single signal
|
|
contributes at most `1/(RRF_K + 1)` however certain it is. A concept the
|
|
question *names* cannot outbid three signals that merely describe it. Form (i)
|
|
is refuted by its own numbers, not by taste.
|
|
|
|
**Form (ii) ships:** a partition applied to the fusion's output. Concepts whose
|
|
title or id bears the question's identifier are moved to the front, keeping the
|
|
order the fusion gave them; everything else keeps its order too. Nothing is
|
|
dropped, nothing is scored twice, and the fused ranks themselves are untouched.
|
|
|
|
```python
|
|
named = set(lookup_hits(concepts, question))
|
|
if named:
|
|
ranked_ids = [k for k in ranked_ids if k in named] + [k for k in ranked_ids if k not in named]
|
|
```
|
|
|
|
**It reads the text the title-and-id signal reads, and declares no key list.**
|
|
The order allowed a frontmatter key set (`req_number`, `identifier`, …). It is
|
|
not declared, because a measurement decided instead: of the **1 846** concepts
|
|
across the three bundles that carry a `req_number`, the identifier in that key
|
|
is **also in the title on 1 846** of them, and on **0** does the key carry an
|
|
identifier the title lacks. A key list would have bought nothing and would have
|
|
been a constant no measurement asked for. The consumer corpus has no such key
|
|
at all, and is served anyway (§ 4).
|
|
|
|
**The known-negative is structural, not lucky.** `lookup_hits` returns the
|
|
empty tuple whenever the question carries no identifier, and **every one of the
|
|
ten published control questions on the consumer corpus carries zero
|
|
identifiers** under `_IDENTIFIER_RE` — the six hit@8 questions, both
|
|
known-negatives, the good question and the mandate-shaped one. Verified with
|
|
`normalise` on each question before any payload was built.
|
|
|
|
**Red first,** on a synthetic 300-concept corpus where every concept carries
|
|
the word a standards corpus repeats on every page and the neighbours are
|
|
numbered so that they share the gold's first four characters: the named concept
|
|
comes back at rank 1; two concepts bearing the same number come back as the top
|
|
two; a question with no identifier leaves the order untouched; an identifier no
|
|
concept bears changes nothing and empties nothing.
|
|
|
|
---
|
|
|
|
## 3. After: three bundles, one instrument, before and after
|
|
|
|
Instrument, unchanged from the previous session and from § 0 above: the gold's
|
|
position in the fused `ranked` list, plus the payload's own verdict from
|
|
`build_payload`. Command:
|
|
|
|
```
|
|
python3 tools/okf_consume.py <bundle> \
|
|
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
|
|
[--rarity-weight]
|
|
```
|
|
|
|
| | N100:2023 | N200:2024 | N500:2024 |
|
|
|---|---|---|---|
|
|
| concepts | 446 | 1 133 | 270 |
|
|
| gold fused rank, **before** | 96 | 9 | 35 |
|
|
| gold fused rank, **after** | **1** | **1** | **1** |
|
|
| gold verdict, before | `below_k` | `below_k` | `below_k` |
|
|
| gold verdict, after | **delivered, rank 1** | **delivered, rank 1** | **delivered, rank 1** |
|
|
| the same with `--rarity-weight` | **1**, delivered | **1**, delivered | **1**, delivered |
|
|
| payload bytes, before → after | 61 566 → 58 409 | 142 861 → 146 145 | 40 626 → 39 193 |
|
|
| payload `sha256`, after | `5bf6ab78…` | `af5829cd…` | `9a9e7860…` |
|
|
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
|
|
| lookup hits found | 1 | 1 | 1 |
|
|
|
|
**3 of 3, at rank 1.** The order's target was rank ≤ 8 on 3 of 3.
|
|
|
|
**The three spellings are one lookup.** For each bundle, the question was asked
|
|
with an ASCII hyphen, an en dash and an em dash. All three deliver the **same
|
|
eight concepts in the same order**, with the gold at rank 1 — the payload
|
|
digests differ only because the question string is echoed in the payload, so
|
|
the excerpt list is compared instead.
|
|
|
|
**Cost.** The lookup is one pass over the concepts already in memory:
|
|
**0.011 s** over 446, **0.024 s** over 1 133, **0.006 s** over 270. The `df`
|
|
pass the rarity weight uses, re-measured under the new matcher: 0.055 s,
|
|
**0.227 s**, 0.064 s. Both are inside the order's one-second bound on the
|
|
largest bundle by a factor of four or better, and a question carrying no
|
|
identifier does not tokenise the corpus at all.
|
|
|
|
---
|
|
|
|
## 4. The consumer corpus: nothing moves, and one unreachable thing becomes reachable
|
|
|
|
Every named control was measured twice — once from a `git archive` copy of
|
|
`116d3e1` extracted to a scratch tree, and once from this working tree — on the
|
|
same bundle, with `PYTHONPATH` and `__file__` both checked so the frozen run
|
|
could not import the working copy.
|
|
|
|
| control | before | after |
|
|
|---|---|---|
|
|
| good question, candidate rank | 1 of 233 | **1 of 233** |
|
|
| good question with the cost vocabulary | 1 of 248 | **1 of 248** |
|
|
| the same two with `--rarity-weight` | 1 / 1 | **1 / 1** |
|
|
| priced sheet, mandate question, vocabulary off | 251 of 271 | **251 of 271** |
|
|
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **10 of 280** |
|
|
| the same two with `--rarity-weight` | 78 / 2 | **78 / 2** |
|
|
| mandate question, default payload | `78fec2ef…`, 169 583 B | **byte-identical** |
|
|
| mandate question, `--cost-vocabulary --k 12 --limit 160000` | `1f11e56b…`, 240 021 B | **byte-identical** |
|
|
| mandate question, `--cost-vocabulary --k 12` | `777986a5…`, 172 246 B | **byte-identical** |
|
|
| good question, default payload | `f7ff803e…`, 165 109 B | **byte-identical** |
|
|
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank, byte count and spend identical |
|
|
| both known-negatives | 8 delivered each | **byte-identical** |
|
|
|
|
**Every figure in that table is identical** — the four payload digests and
|
|
their byte counts, the eight candidate ranks, all six hit@8 rows with their
|
|
spends, and both known-negatives. No consumer's bytes move because these two
|
|
changes exist — which is what makes them safe to ship on by default rather than
|
|
behind a flag.
|
|
|
|
**Generality, on a corpus with no requirement numbers at all.** The consumer
|
|
bundle has no `req_number` key and no requirement grammar; its documents are
|
|
numbered in their titles instead (`… 9.2 …`). One control question was written
|
|
that names such a number. Before, that document was withheld `below_k` at
|
|
position **621 of 621** — dead last, because a three-character identifier
|
|
matched nothing under `MIN_SHARED_PREFIX`, not even itself. After, it is
|
|
**delivered at rank 1**. The mechanism is not a vegnormal mechanism.
|
|
|
|
---
|
|
|
|
## 5. The decision, made by the number
|
|
|
|
The order's rule: (a) is a defect fix and ships as default if every control on
|
|
the consumer corpus stands byte-identical; (b) ships as default if the
|
|
known-negative holds.
|
|
|
|
| mechanism | condition | measured | decision |
|
|
|---|---|---|---|
|
|
| (a) identifier matches by equality | all consumer controls byte-identical | **every control identical** | **default, no flag** |
|
|
| (b) the lookup partition | byte-identical for a question with no identifier | **structural: 0 identifiers in 10 of 10 control questions; every payload identical** | **default, no flag** |
|
|
|
|
Nothing fell, so no flag is added and nothing has to be named as a loss.
|
|
`--rarity-weight` is untouched and stays **off**; with it on, the three lookups
|
|
also deliver at rank 1, so the two mechanisms do not fight.
|
|
|
|
---
|
|
|
|
## 6. What did NOT close
|
|
|
|
- **`.` and `-` are not interchangeable in a lookup.** `_DASH_TO_HYPHEN` folds
|
|
every dash spelling to one, so em dash, en dash and hyphen are one lookup —
|
|
but a question spelling `1.10` does not find a document whose id spells it
|
|
`1-10`, and this library's own id grammar produces exactly that hyphenated
|
|
form from a dotted source. Measured and left open: the fix is a
|
|
canonicalisation inside the lookup, and it needs its own before/after.
|
|
- **A bundle whose identifiers live only in frontmatter is not served.** Today
|
|
that set is empty (0 of 1 846), which is why no key list was declared. If a
|
|
consumer arrives with one, this is the rule that has to grow.
|
|
- **An identifier that appears only in a concept's BODY is not a lookup.** It
|
|
is a search, and it ranks like one.
|
|
- **The matcher alone made two of three lookups worse** (§ 1). Only the
|
|
partition delivers. Anyone lifting one half without the other should read
|
|
that table first.
|
|
- **The two S7 byte figures from the O1 bundle are not reproduced**, because
|
|
neither that bundle nor the rebuilt one is on this machine (§ 0).
|
|
- **"Delivered" is not "read".** All three golds are now in the payload at rank
|
|
1. Whether a model then answers the engineer's question is the consumer's
|
|
measurement, and this document does not claim it.
|
|
|
|
---
|
|
|
|
## 7. Tests, mutations, gates
|
|
|
|
Thirteen new tests. **Twelve were written before either rule existed and
|
|
eleven of them failed**; the twelfth is the known-positive that had to pass
|
|
from the start (the Norwegian compounds), so the red was the rules' absence and
|
|
not a broken fixture. The thirteenth was written afterwards, against a mutation
|
|
that survived, and is red against it. One fixture defect was found by the red
|
|
run itself: the neighbours were numbered so that one of them *was* the gold's
|
|
number, which made the `df` 2 rather than 1.
|
|
|
|
**Seven mutations, seven red.** Three survived the first pass and none was
|
|
banded blind:
|
|
|
|
- **M2** (`is_identifier` matches a substring instead of the whole token)
|
|
survived because no token `normalise` emits can tell the two apart — the
|
|
generic split keeps neither `.` nor `-`, so a split token can never contain
|
|
an identifier. The commitment is real even where no output distinguishes it,
|
|
so it is now asserted on the function.
|
|
- **M4** (the early return for a question with no identifier) survived because
|
|
an empty identifier set intersects to nothing anyway. The guard is a **cost**
|
|
commitment, not a semantic one, so it is now measured as one: a question
|
|
without an identifier must not tokenise the corpus at all.
|
|
- **M6** (the lifted concepts re-sorted by id instead of keeping the fused
|
|
order) survived because the fixture's two holders tied. The fixture was
|
|
corrected so byte order and fused order disagree on purpose.
|
|
|
|
| gate | result |
|
|
|---|---|
|
|
| `pytest -q` | **1 320 passed**, 1 skipped (1 307 on this branch before, +13) |
|
|
| `ruff check src tests tools` | clean |
|
|
| `ruff format --check .` | clean, 88 files |
|
|
| `mypy --strict src/` | clean, 17 files |
|
|
| `okf_contract_check` on the three lookup payloads | exit 0, 14 rules, 0 findings |
|
|
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
|
|
|
|
**No rebuild.** The change is consumption-side only, confirmed by
|
|
`git diff --stat`: no production module under `src/` is touched, so every
|
|
bundle ref in flight is unchanged.
|