open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume): match an identifier by equality, deliver the concept a question names

Browse source 
A question naming a requirement number now delivers that requirement at rank 1
on all three vegnormal bundles (was 96, 9, 35 of 446, 1 133, 270). Two
mechanisms, both measured, both default because no published figure moves.

The matcher: `tokens_match` compared four leading characters, so the unique
identifier `3.3.1-13` read as 135 of 446 common and the rarity weight ranked a
common adjective above the number naming the document. An identifier now
matches by equality alone; df falls to 1/1/1. Words keep the prefix rule, which
was measured for Norwegian compounds. Equality has no floor either, so a
three-character identifier stops matching nothing at all -- measured, `9.2`
reached 0 concepts while sitting verbatim in one title.

The lookup: a question carrying an identifier that sits verbatim in a concept's
title or id is answered by a partition over the fusion's output, not by a
fourth signal. The form was chosen by measurement -- a fourth RRF signal was
simulated first and put the gold at rank 26 / 15 / 19, none of them delivered,
because RRF consumes ranks only and one signal contributes at most 1/(RRF_K+1).
No frontmatter key list is declared: of 1 846 concepts carrying `req_number`,
1 846 also carry that identifier in the title.

The matcher alone is NOT a monotone win (N200 9 -> 26, because that gold's body
cross-references a neighbouring number that the prefix rule counted as a hit on
the question's). Only the partition delivers; the table is in the record.

Consumer corpus: every named control byte-identical against a frozen
`git archive` copy of 116d3e1 -- four payload digests, eight candidate ranks,
six hit@8 rows, both known-negatives. One document that was withheld at
position 621 of 621 is now delivered at rank 1, on a corpus with no
requirement-number grammar at all.

13 tests (12 red before the rules existed), 7 mutations, 7 red. 1 320 passed.
Consumption-side only; no bundle ref moves.
Record: docs/2026-09-08-eksakt-oppslag.md

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 13:27:59 +02:00
commit a37d5ced38
3 changed files with 638 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

344
docs/2026-09-08-eksakt-oppslag.md Normal file
View file

@ -0,0 +1,344 @@
# The exact lookup: naming a document is not searching for it, 2026-09-08
Two sessions made a requirement number visible to the consumption pre-pass and
then made it *worth* more than a common word, and neither delivered the
document the question named. The first left the gold `below_k` on all three
bundles (`docs/2026-09-08-kravnummer-tokenisering.md` § 8); the second
delivered one of three and decomposed the other two into a **matcher**
decision and a **fusion** decision, with no measurement behind either
(`docs/2026-09-08-sjeldenhetsvekt.md` § 3, § 4).
This session takes both decisions with numbers. The result is that a question
naming a requirement number now delivers that requirement at **rank 1 on 3 of
3 bundles**, and that every published figure on the consumer corpus is
**byte-identical**.
---
## 0. What IS measured, and what is NOT
**Measured.** Eleven known-positive figures reproduced before anything was
changed; document frequency under the matcher and under equality on three
bundles; the gold's fused rank, verdict, payload digest and byte count before,
after the matcher alone, and after both mechanisms; the three spellings of one
identifier; every named control on the consumer corpus against a frozen
`git archive` copy of `116d3e1`; a control question on that corpus that *does*
carry an identifier, before and after; hit@8 over the six published questions
and both known-negatives; the cost of each new pass; seven mutations.
**Known-positive, run first.** Every figure this session builds on was
reproduced on this machine, with the committed tool, on a clean tree at
`116d3e1`, before any after-column was read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | gold fused rank per bundle, weight off | previous session § 2 | **96, 9, 35** |
| 2 | gold fused rank per bundle, weight on | previous session § 2 | **103, 8, 35** |
| 3 | `df` under the matcher vs under equality | previous session § 3 | **135 / 17 / 5** vs **1 / 1 / 1** |
| 4 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
| 5 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
| 6 | payload `sha256`, weight off | previous session § 2 | `49e9c718…`, `adc2eac7…`, `21c5bbfe…` |
| 7 | payload `sha256`, weight on | previous session § 2 | `ba09a56a…`, `403ee989…`, `1758f6ca…` |
| 8 | consumer corpus, the good question's candidate rank, four rows | previous session § 5 | **1 / 1 / 1 / 1** |
| 9 | the priced sheet's candidate rank, four rows | previous session § 5, § 7 | **251 / 10 / 78 / 2** |
| 10 | the mandate question's default payload | previous session § 7 | **169 583 B** |
| 11 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6**, five of them at rank 1 |
**NOT measured**, named rather than glossed:
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
machine and this repository's single-runtime-dependency rule does not admit
one to produce a caption. Figures below are in `BUDGET_UNIT` — utf-8 bytes of
emitted JSON — plus the payload `sha256`, which is exact where a byte count
is only indicative.
- **The rebuilt consumer bundle of `56c1205` is not on disk.** Its content
identity (`sha256-tree:2f82fcfe…`) names a tree this machine no longer holds,
and neither does the bundle the two S7 byte figures in the order come from
(`2ae46f68…` 169 573 B and `8aaf8558…` 172 588 B). **Those two figures are
therefore not reproduced here.** The control corpus used instead is the
bundle delivered 2026-09-03 (`sha256-tree:9a4e5561…`), on which rows 8, 9 and
10 of the table above reproduce the published values exactly. This costs the
decision nothing: what the decision needs is byte-*identity* before and
after, and that is measured on one and the same bundle, against a frozen copy
of the committed code rather than against a remembered number.
- **Whether a live model answers better.** This measures a *ranking*.
"Delivered" is not "read", and that measurement belongs to the consumer.
---
## 1. Mechanism (a): an identifier matches by equality, and by nothing else
`tokens_match` compared four leading characters, for a reason that was measured
and still holds: Norwegian compounds do not match token-exactly, and
`vare|ne` and `vare|mottak` share a stem. A requirement number has no stem.
Four leading characters of `3.3.1-13` are four leading characters of every
requirement in section 3.3, so a **unique** identifier read as **135 of 446**
common — and the rarity weight then faithfully ranked a common adjective as
more informative than the number naming the document.
The rule, whole:
```python
if is_identifier(left) or is_identifier(right):
return left == right
```
`is_identifier` is `_IDENTIFIER_RE.fullmatch`, the pattern the tokeniser
already uses to keep an identifier whole. No new constant, no token class
declared anywhere, and words are untouched.
**Red first.** `tokens_match('3.3.1-13', '3.3.1-14')` was `True` and is now
`False`; `('3.3.1-13', '3.3.1-13')` was and is `True`. The known-positive ran
in the same file and passed before and after: `brannsikring`/`brannvern`,
`prisene`/`prissammenstilling`, `varene`/`varemottak` all still match, and
`brann`/`bygg` still does not.
**`df` after, same instrument as before:**
| bundle | identifier | `df` before | `df` after | `df` exact | concepts |
|---|---|---|---|---|---|
| N100 | `3.3.1-13` | **135** | **1** | 1 | 446 |
| N200 | `2.9.2-12` | 17 | **1** | 1 | 1 133 |
| N500 | `10.2-2` | 5 | **1** | 1 | 270 |
**A short identifier stops being invisible.** `MIN_SHARED_PREFIX` made a
three-character identifier match *nothing, not even itself*: measured on the
629-concept consumer bundle, `9.2` reached **0** concepts under the matcher
while sitting verbatim in one title. Equality has no floor, so it now reaches
that one. The floor stays where it was measured — for words.
**What the matcher alone bought, and what it cost.** This is the honest column,
and it is not all in one direction:
| bundle | gold fused rank, before | after (a) alone | with `--rarity-weight`, before → after |
|---|---|---|---|
| N100 | 96 | **53** | 103 → **45** |
| N200 | 9 | **26** | 8 → **23** |
| N500 | 35 | **37** | 35 → **36** |
**N200 got worse, and the reason is measured rather than guessed.** That gold's
body carries **88** identifiers, three of them occurrences of `2.9.2-1` — a
cross-reference to the neighbouring requirement. Under the prefix rule that
cross-reference answered the question's `2.9.2-12`, so the gold was scoring a
hit on its own body that it had not earned. Removing a false positive that
happened to help is still removing a false positive; the number is here so that
nobody has to take that on faith. On its own, mechanism (a) delivers **0 of 3**.
---
## 2. Mechanism (b): the form was chosen by a measurement, not by a preference
A question carrying an identifier that sits verbatim in a concept's title or id
is a **lookup**: the reader already knows which document they want and is
spelling its number. The order named two candidate forms and invited a third.
**Form (i), a fourth signal inside the RRF fusion, was simulated first** on the
same three bundles, on top of mechanism (a), with the lookup as a 0/1 signal:
| bundle | gold rank with a fourth RRF signal | of |
|---|---|---|
| N100 | **26** | 446 |
| N200 | **15** | 1 133 |
| N500 | **19** | 270 |
**None of them delivered**, and the reason is the one the previous session
already wrote down: RRF consumes **ranks only**, so any single signal
contributes at most `1/(RRF_K + 1)` however certain it is. A concept the
question *names* cannot outbid three signals that merely describe it. Form (i)
is refuted by its own numbers, not by taste.
**Form (ii) ships:** a partition applied to the fusion's output. Concepts whose
title or id bears the question's identifier are moved to the front, keeping the
order the fusion gave them; everything else keeps its order too. Nothing is
dropped, nothing is scored twice, and the fused ranks themselves are untouched.
```python
named = set(lookup_hits(concepts, question))
if named:
ranked_ids = [k for k in ranked_ids if k in named] + [k for k in ranked_ids if k not in named]
```
**It reads the text the title-and-id signal reads, and declares no key list.**
The order allowed a frontmatter key set (`req_number`, `identifier`, …). It is
not declared, because a measurement decided instead: of the **1 846** concepts
across the three bundles that carry a `req_number`, the identifier in that key
is **also in the title on 1 846** of them, and on **0** does the key carry an
identifier the title lacks. A key list would have bought nothing and would have
been a constant no measurement asked for. The consumer corpus has no such key
at all, and is served anyway (§ 4).
**The known-negative is structural, not lucky.** `lookup_hits` returns the
empty tuple whenever the question carries no identifier, and **every one of the
ten published control questions on the consumer corpus carries zero
identifiers** under `_IDENTIFIER_RE` — the six hit@8 questions, both
known-negatives, the good question and the mandate-shaped one. Verified with
`normalise` on each question before any payload was built.
**Red first,** on a synthetic 300-concept corpus where every concept carries
the word a standards corpus repeats on every page and the neighbours are
numbered so that they share the gold's first four characters: the named concept
comes back at rank 1; two concepts bearing the same number come back as the top
two; a question with no identifier leaves the order untouched; an identifier no
concept bears changes nothing and empties nothing.
---
## 3. After: three bundles, one instrument, before and after
Instrument, unchanged from the previous session and from § 0 above: the gold's
position in the fused `ranked` list, plus the payload's own verdict from
`build_payload`. Command:
```
python3 tools/okf_consume.py <bundle> \
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
[--rarity-weight]
```
| | N100:2023 | N200:2024 | N500:2024 |
|---|---|---|---|
| concepts | 446 | 1 133 | 270 |
| gold fused rank, **before** | 96 | 9 | 35 |
| gold fused rank, **after** | **1** | **1** | **1** |
| gold verdict, before | `below_k` | `below_k` | `below_k` |
| gold verdict, after | **delivered, rank 1** | **delivered, rank 1** | **delivered, rank 1** |
| the same with `--rarity-weight` | **1**, delivered | **1**, delivered | **1**, delivered |
| payload bytes, before → after | 61 566 → 58 409 | 142 861 → 146 145 | 40 626 → 39 193 |
| payload `sha256`, after | `5bf6ab78…` | `af5829cd…` | `9a9e7860…` |
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
| lookup hits found | 1 | 1 | 1 |
**3 of 3, at rank 1.** The order's target was rank ≤ 8 on 3 of 3.
**The three spellings are one lookup.** For each bundle, the question was asked
with an ASCII hyphen, an en dash and an em dash. All three deliver the **same
eight concepts in the same order**, with the gold at rank 1 — the payload
digests differ only because the question string is echoed in the payload, so
the excerpt list is compared instead.
**Cost.** The lookup is one pass over the concepts already in memory:
**0.011 s** over 446, **0.024 s** over 1 133, **0.006 s** over 270. The `df`
pass the rarity weight uses, re-measured under the new matcher: 0.055 s,
**0.227 s**, 0.064 s. Both are inside the order's one-second bound on the
largest bundle by a factor of four or better, and a question carrying no
identifier does not tokenise the corpus at all.
---
## 4. The consumer corpus: nothing moves, and one unreachable thing becomes reachable
Every named control was measured twice — once from a `git archive` copy of
`116d3e1` extracted to a scratch tree, and once from this working tree — on the
same bundle, with `PYTHONPATH` and `__file__` both checked so the frozen run
could not import the working copy.
| control | before | after |
|---|---|---|
| good question, candidate rank | 1 of 233 | **1 of 233** |
| good question with the cost vocabulary | 1 of 248 | **1 of 248** |
| the same two with `--rarity-weight` | 1 / 1 | **1 / 1** |
| priced sheet, mandate question, vocabulary off | 251 of 271 | **251 of 271** |
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **10 of 280** |
| the same two with `--rarity-weight` | 78 / 2 | **78 / 2** |
| mandate question, default payload | `78fec2ef…`, 169 583 B | **byte-identical** |
| mandate question, `--cost-vocabulary --k 12 --limit 160000` | `1f11e56b…`, 240 021 B | **byte-identical** |
| mandate question, `--cost-vocabulary --k 12` | `777986a5…`, 172 246 B | **byte-identical** |
| good question, default payload | `f7ff803e…`, 165 109 B | **byte-identical** |
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank, byte count and spend identical |
| both known-negatives | 8 delivered each | **byte-identical** |
**Every figure in that table is identical** — the four payload digests and
their byte counts, the eight candidate ranks, all six hit@8 rows with their
spends, and both known-negatives. No consumer's bytes move because these two
changes exist — which is what makes them safe to ship on by default rather than
behind a flag.
**Generality, on a corpus with no requirement numbers at all.** The consumer
bundle has no `req_number` key and no requirement grammar; its documents are
numbered in their titles instead (`… 9.2 …`). One control question was written
that names such a number. Before, that document was withheld `below_k` at
position **621 of 621** — dead last, because a three-character identifier
matched nothing under `MIN_SHARED_PREFIX`, not even itself. After, it is
**delivered at rank 1**. The mechanism is not a vegnormal mechanism.
---
## 5. The decision, made by the number
The order's rule: (a) is a defect fix and ships as default if every control on
the consumer corpus stands byte-identical; (b) ships as default if the
known-negative holds.
| mechanism | condition | measured | decision |
|---|---|---|---|
| (a) identifier matches by equality | all consumer controls byte-identical | **every control identical** | **default, no flag** |
| (b) the lookup partition | byte-identical for a question with no identifier | **structural: 0 identifiers in 10 of 10 control questions; every payload identical** | **default, no flag** |
Nothing fell, so no flag is added and nothing has to be named as a loss.
`--rarity-weight` is untouched and stays **off**; with it on, the three lookups
also deliver at rank 1, so the two mechanisms do not fight.
---
## 6. What did NOT close
- **`.` and `-` are not interchangeable in a lookup.** `_DASH_TO_HYPHEN` folds
every dash spelling to one, so em dash, en dash and hyphen are one lookup —
but a question spelling `1.10` does not find a document whose id spells it
`1-10`, and this library's own id grammar produces exactly that hyphenated
form from a dotted source. Measured and left open: the fix is a
canonicalisation inside the lookup, and it needs its own before/after.
- **A bundle whose identifiers live only in frontmatter is not served.** Today
that set is empty (0 of 1 846), which is why no key list was declared. If a
consumer arrives with one, this is the rule that has to grow.
- **An identifier that appears only in a concept's BODY is not a lookup.** It
is a search, and it ranks like one.
- **The matcher alone made two of three lookups worse** (§ 1). Only the
partition delivers. Anyone lifting one half without the other should read
that table first.
- **The two S7 byte figures from the O1 bundle are not reproduced**, because
neither that bundle nor the rebuilt one is on this machine (§ 0).
- **"Delivered" is not "read".** All three golds are now in the payload at rank
1. Whether a model then answers the engineer's question is the consumer's
measurement, and this document does not claim it.
---
## 7. Tests, mutations, gates
Thirteen new tests. **Twelve were written before either rule existed and
eleven of them failed**; the twelfth is the known-positive that had to pass
from the start (the Norwegian compounds), so the red was the rules' absence and
not a broken fixture. The thirteenth was written afterwards, against a mutation
that survived, and is red against it. One fixture defect was found by the red
run itself: the neighbours were numbered so that one of them *was* the gold's
number, which made the `df` 2 rather than 1.
**Seven mutations, seven red.** Three survived the first pass and none was
banded blind:
- **M2** (`is_identifier` matches a substring instead of the whole token)
survived because no token `normalise` emits can tell the two apart — the
generic split keeps neither `.` nor `-`, so a split token can never contain
an identifier. The commitment is real even where no output distinguishes it,
so it is now asserted on the function.
- **M4** (the early return for a question with no identifier) survived because
an empty identifier set intersects to nothing anyway. The guard is a **cost**
commitment, not a semantic one, so it is now measured as one: a question
without an identifier must not tokenise the corpus at all.
- **M6** (the lifted concepts re-sorted by id instead of keeping the fused
order) survived because the fixture's two holders tied. The fixture was
corrected so byte order and fused order disagree on purpose.
| gate | result |
|---|---|
| `pytest -q` | **1 320 passed**, 1 skipped (1 307 on this branch before, +13) |
| `ruff check src tests tools` | clean |
| `ruff format --check .` | clean, 88 files |
| `mypy --strict src/` | clean, 17 files |
| `okf_contract_check` on the three lookup payloads | exit 0, 14 rules, 0 findings |
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
**No rebuild.** The change is consumption-side only, confirmed by
`git diff --stat`: no production module under `src/` is touched, so every
bundle ref in flight is unchanged.