open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

merge: o2c-oppslag - identifier equality matching and exact lookup delivery (a37d5ce)

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 13:31:15 +02:00
commit d3bfe92acd
3 changed files with 638 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

344
docs/2026-09-08-eksakt-oppslag.md Normal file
View file

@ -0,0 +1,344 @@
# The exact lookup: naming a document is not searching for it, 2026-09-08
Two sessions made a requirement number visible to the consumption pre-pass and
then made it *worth* more than a common word, and neither delivered the
document the question named. The first left the gold `below_k` on all three
bundles (`docs/2026-09-08-kravnummer-tokenisering.md` § 8); the second
delivered one of three and decomposed the other two into a **matcher**
decision and a **fusion** decision, with no measurement behind either
(`docs/2026-09-08-sjeldenhetsvekt.md` § 3, § 4).
This session takes both decisions with numbers. The result is that a question
naming a requirement number now delivers that requirement at **rank 1 on 3 of
3 bundles**, and that every published figure on the consumer corpus is
**byte-identical**.
---
## 0. What IS measured, and what is NOT
**Measured.** Eleven known-positive figures reproduced before anything was
changed; document frequency under the matcher and under equality on three
bundles; the gold's fused rank, verdict, payload digest and byte count before,
after the matcher alone, and after both mechanisms; the three spellings of one
identifier; every named control on the consumer corpus against a frozen
`git archive` copy of `116d3e1`; a control question on that corpus that *does*
carry an identifier, before and after; hit@8 over the six published questions
and both known-negatives; the cost of each new pass; seven mutations.
**Known-positive, run first.** Every figure this session builds on was
reproduced on this machine, with the committed tool, on a clean tree at
`116d3e1`, before any after-column was read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | gold fused rank per bundle, weight off | previous session § 2 | **96, 9, 35** |
| 2 | gold fused rank per bundle, weight on | previous session § 2 | **103, 8, 35** |
| 3 | `df` under the matcher vs under equality | previous session § 3 | **135 / 17 / 5** vs **1 / 1 / 1** |
| 4 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
| 5 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
| 6 | payload `sha256`, weight off | previous session § 2 | `49e9c718…`, `adc2eac7…`, `21c5bbfe…` |
| 7 | payload `sha256`, weight on | previous session § 2 | `ba09a56a…`, `403ee989…`, `1758f6ca…` |
| 8 | consumer corpus, the good question's candidate rank, four rows | previous session § 5 | **1 / 1 / 1 / 1** |
| 9 | the priced sheet's candidate rank, four rows | previous session § 5, § 7 | **251 / 10 / 78 / 2** |
| 10 | the mandate question's default payload | previous session § 7 | **169 583 B** |
| 11 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6**, five of them at rank 1 |
**NOT measured**, named rather than glossed:
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
machine and this repository's single-runtime-dependency rule does not admit
one to produce a caption. Figures below are in `BUDGET_UNIT` — utf-8 bytes of
emitted JSON — plus the payload `sha256`, which is exact where a byte count
is only indicative.
- **The rebuilt consumer bundle of `56c1205` is not on disk.** Its content
identity (`sha256-tree:2f82fcfe…`) names a tree this machine no longer holds,
and neither does the bundle the two S7 byte figures in the order come from
(`2ae46f68…` 169 573 B and `8aaf8558…` 172 588 B). **Those two figures are
therefore not reproduced here.** The control corpus used instead is the
bundle delivered 2026-09-03 (`sha256-tree:9a4e5561…`), on which rows 8, 9 and
10 of the table above reproduce the published values exactly. This costs the
decision nothing: what the decision needs is byte-*identity* before and
after, and that is measured on one and the same bundle, against a frozen copy
of the committed code rather than against a remembered number.
- **Whether a live model answers better.** This measures a *ranking*.
"Delivered" is not "read", and that measurement belongs to the consumer.
---
## 1. Mechanism (a): an identifier matches by equality, and by nothing else
`tokens_match` compared four leading characters, for a reason that was measured
and still holds: Norwegian compounds do not match token-exactly, and
`vare|ne` and `vare|mottak` share a stem. A requirement number has no stem.
Four leading characters of `3.3.1-13` are four leading characters of every
requirement in section 3.3, so a **unique** identifier read as **135 of 446**
common — and the rarity weight then faithfully ranked a common adjective as
more informative than the number naming the document.
The rule, whole:
```python
if is_identifier(left) or is_identifier(right):
return left == right
```
`is_identifier` is `_IDENTIFIER_RE.fullmatch`, the pattern the tokeniser
already uses to keep an identifier whole. No new constant, no token class
declared anywhere, and words are untouched.
**Red first.** `tokens_match('3.3.1-13', '3.3.1-14')` was `True` and is now
`False`; `('3.3.1-13', '3.3.1-13')` was and is `True`. The known-positive ran
in the same file and passed before and after: `brannsikring`/`brannvern`,
`prisene`/`prissammenstilling`, `varene`/`varemottak` all still match, and
`brann`/`bygg` still does not.
**`df` after, same instrument as before:**
| bundle | identifier | `df` before | `df` after | `df` exact | concepts |
|---|---|---|---|---|---|
| N100 | `3.3.1-13` | **135** | **1** | 1 | 446 |
| N200 | `2.9.2-12` | 17 | **1** | 1 | 1 133 |
| N500 | `10.2-2` | 5 | **1** | 1 | 270 |
**A short identifier stops being invisible.** `MIN_SHARED_PREFIX` made a
three-character identifier match *nothing, not even itself*: measured on the
629-concept consumer bundle, `9.2` reached **0** concepts under the matcher
while sitting verbatim in one title. Equality has no floor, so it now reaches
that one. The floor stays where it was measured — for words.
**What the matcher alone bought, and what it cost.** This is the honest column,
and it is not all in one direction:
| bundle | gold fused rank, before | after (a) alone | with `--rarity-weight`, before → after |
|---|---|---|---|
| N100 | 96 | **53** | 103 → **45** |
| N200 | 9 | **26** | 8 → **23** |
| N500 | 35 | **37** | 35 → **36** |
**N200 got worse, and the reason is measured rather than guessed.** That gold's
body carries **88** identifiers, three of them occurrences of `2.9.2-1` — a
cross-reference to the neighbouring requirement. Under the prefix rule that
cross-reference answered the question's `2.9.2-12`, so the gold was scoring a
hit on its own body that it had not earned. Removing a false positive that
happened to help is still removing a false positive; the number is here so that
nobody has to take that on faith. On its own, mechanism (a) delivers **0 of 3**.
---
## 2. Mechanism (b): the form was chosen by a measurement, not by a preference
A question carrying an identifier that sits verbatim in a concept's title or id
is a **lookup**: the reader already knows which document they want and is
spelling its number. The order named two candidate forms and invited a third.
**Form (i), a fourth signal inside the RRF fusion, was simulated first** on the
same three bundles, on top of mechanism (a), with the lookup as a 0/1 signal:
| bundle | gold rank with a fourth RRF signal | of |
|---|---|---|
| N100 | **26** | 446 |
| N200 | **15** | 1 133 |
| N500 | **19** | 270 |
**None of them delivered**, and the reason is the one the previous session
already wrote down: RRF consumes **ranks only**, so any single signal
contributes at most `1/(RRF_K + 1)` however certain it is. A concept the
question *names* cannot outbid three signals that merely describe it. Form (i)
is refuted by its own numbers, not by taste.
**Form (ii) ships:** a partition applied to the fusion's output. Concepts whose
title or id bears the question's identifier are moved to the front, keeping the
order the fusion gave them; everything else keeps its order too. Nothing is
dropped, nothing is scored twice, and the fused ranks themselves are untouched.
```python
named = set(lookup_hits(concepts, question))
if named:
ranked_ids = [k for k in ranked_ids if k in named] + [k for k in ranked_ids if k not in named]
```
**It reads the text the title-and-id signal reads, and declares no key list.**
The order allowed a frontmatter key set (`req_number`, `identifier`, …). It is
not declared, because a measurement decided instead: of the **1 846** concepts
across the three bundles that carry a `req_number`, the identifier in that key
is **also in the title on 1 846** of them, and on **0** does the key carry an
identifier the title lacks. A key list would have bought nothing and would have
been a constant no measurement asked for. The consumer corpus has no such key
at all, and is served anyway (§ 4).
**The known-negative is structural, not lucky.** `lookup_hits` returns the
empty tuple whenever the question carries no identifier, and **every one of the
ten published control questions on the consumer corpus carries zero
identifiers** under `_IDENTIFIER_RE` — the six hit@8 questions, both
known-negatives, the good question and the mandate-shaped one. Verified with
`normalise` on each question before any payload was built.
**Red first,** on a synthetic 300-concept corpus where every concept carries
the word a standards corpus repeats on every page and the neighbours are
numbered so that they share the gold's first four characters: the named concept
comes back at rank 1; two concepts bearing the same number come back as the top
two; a question with no identifier leaves the order untouched; an identifier no
concept bears changes nothing and empties nothing.
---
## 3. After: three bundles, one instrument, before and after
Instrument, unchanged from the previous session and from § 0 above: the gold's
position in the fused `ranked` list, plus the payload's own verdict from
`build_payload`. Command:
```
python3 tools/okf_consume.py <bundle> \
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
[--rarity-weight]
```
| | N100:2023 | N200:2024 | N500:2024 |
|---|---|---|---|
| concepts | 446 | 1 133 | 270 |
| gold fused rank, **before** | 96 | 9 | 35 |
| gold fused rank, **after** | **1** | **1** | **1** |
| gold verdict, before | `below_k` | `below_k` | `below_k` |
| gold verdict, after | **delivered, rank 1** | **delivered, rank 1** | **delivered, rank 1** |
| the same with `--rarity-weight` | **1**, delivered | **1**, delivered | **1**, delivered |
| payload bytes, before → after | 61 566 → 58 409 | 142 861 → 146 145 | 40 626 → 39 193 |
| payload `sha256`, after | `5bf6ab78…` | `af5829cd…` | `9a9e7860…` |
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
| lookup hits found | 1 | 1 | 1 |
**3 of 3, at rank 1.** The order's target was rank ≤ 8 on 3 of 3.
**The three spellings are one lookup.** For each bundle, the question was asked
with an ASCII hyphen, an en dash and an em dash. All three deliver the **same
eight concepts in the same order**, with the gold at rank 1 — the payload
digests differ only because the question string is echoed in the payload, so
the excerpt list is compared instead.
**Cost.** The lookup is one pass over the concepts already in memory:
**0.011 s** over 446, **0.024 s** over 1 133, **0.006 s** over 270. The `df`
pass the rarity weight uses, re-measured under the new matcher: 0.055 s,
**0.227 s**, 0.064 s. Both are inside the order's one-second bound on the
largest bundle by a factor of four or better, and a question carrying no
identifier does not tokenise the corpus at all.
---
## 4. The consumer corpus: nothing moves, and one unreachable thing becomes reachable
Every named control was measured twice — once from a `git archive` copy of
`116d3e1` extracted to a scratch tree, and once from this working tree — on the
same bundle, with `PYTHONPATH` and `__file__` both checked so the frozen run
could not import the working copy.
| control | before | after |
|---|---|---|
| good question, candidate rank | 1 of 233 | **1 of 233** |
| good question with the cost vocabulary | 1 of 248 | **1 of 248** |
| the same two with `--rarity-weight` | 1 / 1 | **1 / 1** |
| priced sheet, mandate question, vocabulary off | 251 of 271 | **251 of 271** |
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **10 of 280** |
| the same two with `--rarity-weight` | 78 / 2 | **78 / 2** |
| mandate question, default payload | `78fec2ef…`, 169 583 B | **byte-identical** |
| mandate question, `--cost-vocabulary --k 12 --limit 160000` | `1f11e56b…`, 240 021 B | **byte-identical** |
| mandate question, `--cost-vocabulary --k 12` | `777986a5…`, 172 246 B | **byte-identical** |
| good question, default payload | `f7ff803e…`, 165 109 B | **byte-identical** |
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank, byte count and spend identical |
| both known-negatives | 8 delivered each | **byte-identical** |
**Every figure in that table is identical** — the four payload digests and
their byte counts, the eight candidate ranks, all six hit@8 rows with their
spends, and both known-negatives. No consumer's bytes move because these two
changes exist — which is what makes them safe to ship on by default rather than
behind a flag.
**Generality, on a corpus with no requirement numbers at all.** The consumer
bundle has no `req_number` key and no requirement grammar; its documents are
numbered in their titles instead (`… 9.2 …`). One control question was written
that names such a number. Before, that document was withheld `below_k` at
position **621 of 621** — dead last, because a three-character identifier
matched nothing under `MIN_SHARED_PREFIX`, not even itself. After, it is
**delivered at rank 1**. The mechanism is not a vegnormal mechanism.
---
## 5. The decision, made by the number
The order's rule: (a) is a defect fix and ships as default if every control on
the consumer corpus stands byte-identical; (b) ships as default if the
known-negative holds.
| mechanism | condition | measured | decision |
|---|---|---|---|
| (a) identifier matches by equality | all consumer controls byte-identical | **every control identical** | **default, no flag** |
| (b) the lookup partition | byte-identical for a question with no identifier | **structural: 0 identifiers in 10 of 10 control questions; every payload identical** | **default, no flag** |
Nothing fell, so no flag is added and nothing has to be named as a loss.
`--rarity-weight` is untouched and stays **off**; with it on, the three lookups
also deliver at rank 1, so the two mechanisms do not fight.
---
## 6. What did NOT close
- **`.` and `-` are not interchangeable in a lookup.** `_DASH_TO_HYPHEN` folds
every dash spelling to one, so em dash, en dash and hyphen are one lookup —
but a question spelling `1.10` does not find a document whose id spells it
`1-10`, and this library's own id grammar produces exactly that hyphenated
form from a dotted source. Measured and left open: the fix is a
canonicalisation inside the lookup, and it needs its own before/after.
- **A bundle whose identifiers live only in frontmatter is not served.** Today
that set is empty (0 of 1 846), which is why no key list was declared. If a
consumer arrives with one, this is the rule that has to grow.
- **An identifier that appears only in a concept's BODY is not a lookup.** It
is a search, and it ranks like one.
- **The matcher alone made two of three lookups worse** (§ 1). Only the
partition delivers. Anyone lifting one half without the other should read
that table first.
- **The two S7 byte figures from the O1 bundle are not reproduced**, because
neither that bundle nor the rebuilt one is on this machine (§ 0).
- **"Delivered" is not "read".** All three golds are now in the payload at rank
1. Whether a model then answers the engineer's question is the consumer's
measurement, and this document does not claim it.
---
## 7. Tests, mutations, gates
Thirteen new tests. **Twelve were written before either rule existed and
eleven of them failed**; the twelfth is the known-positive that had to pass
from the start (the Norwegian compounds), so the red was the rules' absence and
not a broken fixture. The thirteenth was written afterwards, against a mutation
that survived, and is red against it. One fixture defect was found by the red
run itself: the neighbours were numbered so that one of them *was* the gold's
number, which made the `df` 2 rather than 1.
**Seven mutations, seven red.** Three survived the first pass and none was
banded blind:
- **M2** (`is_identifier` matches a substring instead of the whole token)
survived because no token `normalise` emits can tell the two apart — the
generic split keeps neither `.` nor `-`, so a split token can never contain
an identifier. The commitment is real even where no output distinguishes it,
so it is now asserted on the function.
- **M4** (the early return for a question with no identifier) survived because
an empty identifier set intersects to nothing anyway. The guard is a **cost**
commitment, not a semantic one, so it is now measured as one: a question
without an identifier must not tokenise the corpus at all.
- **M6** (the lifted concepts re-sorted by id instead of keeping the fused
order) survived because the fixture's two holders tied. The fixture was
corrected so byte order and fused order disagree on purpose.
| gate | result |
|---|---|
| `pytest -q` | **1 320 passed**, 1 skipped (1 307 on this branch before, +13) |
| `ruff check src tests tools` | clean |
| `ruff format --check .` | clean, 88 files |
| `mypy --strict src/` | clean, 17 files |
| `okf_contract_check` on the three lookup payloads | exit 0, 14 rules, 0 findings |
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
**No rebuild.** The change is consumption-side only, confirmed by
`git diff --stat`: no production module under `src/` is touched, so every
bundle ref in flight is unchanged.

205
tests/test_okf_consume.py
View file

@ -1669,9 +1669,12 @@ def test_counting_leaves_the_identifier_worth_no_more_than_the_common_word() ->
# The known-positive for the weight: without it, a rank of 1 with the # The known-positive for the weight: without it, a rank of 1 with the
# weight on would prove nothing, because nothing would have been shown to # weight on would prove nothing, because nothing would have been shown to
# hold the gold down in the first place. # hold the gold down in the first place.
# `lookup=False` throughout: this fixture puts the identifier in the
# gold's TITLE, which the lookup partition (Step 14) answers at rank one.
# The claim here is about the FUSION, so the fusion is what is read.
for identifier_in_body in (False, True): for identifier_in_body in (False, True):
concepts = _rarity_corpus(identifier_in_body=identifier_in_body) concepts = _rarity_corpus(identifier_in_body=identifier_in_body)
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}) ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, lookup=False)
assert _rank_of(ranked, RARITY_GOLD) == 18 assert _rank_of(ranked, RARITY_GOLD) == 18
lexical = {concept.concept_id: value for concept, _, value in ranked} lexical = {concept.concept_id: value for concept, _, value in ranked}
assert lexical["krav/c-00"] == 6 assert lexical["krav/c-00"] == 6
@ -1716,7 +1719,9 @@ def test_a_rarity_weight_cannot_move_a_signal_the_gold_already_leads() -> None:
# And the rank it buys is the rank it already had, so the fusion sees none of it. # And the rank it buys is the rank it already had, so the fusion sees none of it.
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1 assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1 assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights) ranked = okf_consume.concept_scores(
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False
)
assert _rank_of(ranked, RARITY_GOLD) == 18 assert _rank_of(ranked, RARITY_GOLD) == 18
@ -1885,11 +1890,203 @@ def test_the_weight_reaches_the_title_signal_and_reorders_it() -> None:
) )
assert _signal_rank(concepts, question, "krav/t-29", None) == 30 assert _signal_rank(concepts, question, "krav/t-29", None) == 30
assert _signal_rank(concepts, question, "krav/t-29", weights) == 1 assert _signal_rank(concepts, question, "krav/t-29", weights) == 1
plain = okf_consume.concept_scores(concepts, question, {}) # `lookup=False`: the gold's title bears the identifier, so the lookup
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights) # partition answers this question at rank one. The fusion is the claim.
plain = okf_consume.concept_scores(concepts, question, {}, lookup=False)
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights, lookup=False)
# Fused: 30 -> 18. The title signal is reordered from last to first and the # Fused: 30 -> 18. The title signal is reordered from last to first and the
# fused rank moves by twelve, not to one -- the other two signals still # fused rank moves by twelve, not to one -- the other two signals still
# cannot see the identifier. That gap IS the finding of this session, and # cannot see the identifier. That gap IS the finding of this session, and
# the numbers are here so a change to either half shows up as a diff. # the numbers are here so a change to either half shows up as a diff.
assert _rank_of(plain, "krav/t-29") == 30 assert _rank_of(plain, "krav/t-29") == 30
assert _rank_of(weighted, "krav/t-29") == 18 assert _rank_of(weighted, "krav/t-29") == 18
# --- Step 14: exact identifier matching and the lookup signal (O2c) -----------
#: The lookup fixture: 300 concepts that all carry the word a standards corpus
#: repeats on every page, one of which also carries the identifier. Synthetic
#: rather than borrowed, and 300 rather than 30 so a rank of one is a claim
#: about the rule and not about a small corpus.
LOOKUP_QUESTION = "Hva krever Krav 3.3.1-13? Gjengi det sentrale vilkåret."
LOOKUP_GOLD = "krav/c-299"
LOOKUP_SECOND = "krav/z-000"
def _lookup_corpus(*, second_holder: bool = False) -> list[okf_consume.Concept]:
"""299 neighbours numbered `3.3.1-<n>`, plus the concept the question names.
The neighbours share the identifier's first four characters, which is
exactly what made a unique requirement number read as 135-of-446 common on
a real bundle. The gold's `concept_id` sorts LAST, so nothing but the rule
can lift it.
"""
body = "Kravet gjengir det sentrale vilkåret for anlegget."
concepts = [
_synthetic(f"krav/c-{index:03d}", f"Krav 3.3.1-{index + 20} Alminnelig krav", body)
for index in range(299)
]
concepts.append(_synthetic(LOOKUP_GOLD, "Krav 3.3.1-13 Tekniske bygg", body))
if second_holder:
# The second holder's id sorts AFTER the gold's and its title answers
# four question tokens more, so the fusion ranks it FIRST while byte
# order ranks it second. The two orders disagree on purpose: a lookup
# that re-sorted its hits by id instead of keeping the fused order
# would otherwise be indistinguishable from one that keeps it.
concepts.append(
_synthetic(LOOKUP_SECOND, "Krav 3.3.1-13 Gjengi det sentrale vilkåret", body)
)
return concepts
def test_an_identifier_matches_its_own_spelling_and_no_neighbouring_number() -> None:
# The defect, at the case it costs most: `3.3.1-13` and `3.3.1-14` share
# four leading characters, so the prefix rule called them a match and every
# requirement number beginning `3.3.` counted as a hit.
assert okf_consume.tokens_match("3.3.1-13", "3.3.1-13") is True
assert okf_consume.tokens_match("3.3.1-13", "3.3.1-14") is False
assert okf_consume.tokens_match("3.3.1-13", "3.3.2-13") is False
# Symmetric, like the rule it replaces.
assert okf_consume.tokens_match("3.3.1-14", "3.3.1-13") is False
def test_a_norwegian_compound_still_matches_on_the_shared_prefix() -> None:
# The known-positive. `MIN_SHARED_PREFIX` exists because Norwegian
# compounds do not match token-exactly, and an identifier rule that also
# narrowed words would buy one lookup by losing every compound.
assert okf_consume.tokens_match("brannsikring", "brannvern") is True
assert okf_consume.tokens_match("prisene", "prissammenstilling") is True
assert okf_consume.tokens_match("varene", "varemottak") is True
assert okf_consume.tokens_match("brann", "bygg") is False
def test_a_short_identifier_becomes_reachable_because_equality_has_no_floor() -> None:
# `MIN_SHARED_PREFIX` made a three-character identifier match NOTHING, not
# even itself: measured on a 629-concept bundle, `9.2` reached 0 concepts
# under the matcher while sitting verbatim in one title.
assert okf_consume.tokens_match("9.2", "9.2") is True
assert okf_consume.tokens_match("9.2", "9.3") is False
# The floor still stands for words, which is what it was measured for.
assert okf_consume.tokens_match("veg", "veg") is False
def test_the_identifier_rule_makes_a_unique_number_unique_in_the_document_frequency() -> None:
# The df is what the rarity weight reads, and 135-of-446 was the reason a
# unique requirement number weighed less than a common adjective.
concepts = _lookup_corpus()
corpus = okf_consume.searchable_text(concepts)
identifier = "3.3.1-13"
under_matcher = sum(
1
for text in corpus
if any(okf_consume.tokens_match(identifier, other) for other in okf_consume.normalise(text))
)
assert under_matcher == 1
def test_is_identifier_names_numbers_and_not_words() -> None:
assert okf_consume.is_identifier("3.3.1-13") is True
assert okf_consume.is_identifier("10.2-2") is True
assert okf_consume.is_identifier("r610.4") is True
assert okf_consume.is_identifier("brannsikring") is False
assert okf_consume.is_identifier("krav") is False
# A bare number is not an identifier: it has no separator, and every page
# number in a corpus would become one.
assert okf_consume.is_identifier("2023") is False
# THE WHOLE TOKEN, never a part of one. No token `normalise` emits can tell
# a whole-token rule from a substring rule -- the generic split keeps
# neither `.` nor `-`, so a split token can never contain an identifier --
# which is why the commitment is asserted on the function rather than on an
# output it happens to leave unchanged.
assert okf_consume.is_identifier("bilag3.3-1x") is False
def test_a_question_naming_an_identifier_delivers_the_concept_that_bears_it() -> None:
# The lookup: 300 concepts all answer `krav`, one is NAMED. Rank one, not
# rank eight -- a question that names a concept is not a search.
concepts = _lookup_corpus()
ranked = okf_consume.concept_scores(concepts, LOOKUP_QUESTION, {})
assert _rank_of(ranked, LOOKUP_GOLD) == 1
def test_two_concepts_bearing_the_same_identifier_both_reach_the_top() -> None:
concepts = _lookup_corpus(second_holder=True)
ranked = okf_consume.concept_scores(concepts, LOOKUP_QUESTION, {})
top = [concept.concept_id for concept, _, _ in ranked[:2]]
assert sorted(top) == sorted([LOOKUP_GOLD, LOOKUP_SECOND])
def test_the_lookup_is_invisible_to_a_question_that_names_no_identifier() -> None:
# The KNOWN-NEGATIVE. Every published control question on the consumer
# corpus carries zero identifiers, so this is the property that lets the
# rule ship on by default.
concepts = _lookup_corpus()
question = "Hvordan skal det sentrale vilkåret oppfylles?"
assert okf_consume.lookup_hits(concepts, question) == ()
ranked = okf_consume.concept_scores(concepts, question, {})
order = [concept.concept_id for concept, _, _ in ranked]
assert order == sorted(order)
def test_a_question_without_an_identifier_never_reads_the_corpus(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# The early return is a COST commitment, not a semantic one: an empty
# identifier set intersects to nothing anyway, so no ranking distinguishes
# the guard from its absence. What it does buy is the corpus never being
# tokenised for a question that cannot be a lookup -- measured, the lookup
# pass costs 0.024 s over 1 133 concepts when it does run.
concepts = _lookup_corpus()
calls: list[str] = []
original = okf_consume.normalise
monkeypatch.setattr(
okf_consume, "normalise", lambda text: (calls.append(text), original(text))[1]
)
assert okf_consume.lookup_hits(concepts, "Hvordan oppfylles vilkåret?") == ()
assert calls == ["Hvordan oppfylles vilkåret?"]
def test_an_identifier_no_concept_bears_changes_nothing_and_empties_nothing() -> None:
concepts = _lookup_corpus()
question = "Hva krever Krav 9.9.9-99? Gjengi det sentrale vilkåret."
assert okf_consume.lookup_hits(concepts, question) == ()
ranked = okf_consume.concept_scores(concepts, question, {})
assert len(ranked) == len(concepts)
order = [concept.concept_id for concept, _, _ in ranked]
assert order == sorted(order)
def test_the_three_spellings_of_one_identifier_are_one_lookup() -> None:
concepts = _lookup_corpus()
for dash in ("-", "–", "—"):
question = f"Hva krever Krav 3.3.1{dash}13? Gjengi det sentrale vilkåret."
assert okf_consume.lookup_hits(concepts, question) == (LOOKUP_GOLD,)
def test_the_lookup_reads_the_text_the_title_signal_reads() -> None:
# Measured on three real bundles: `req_number` carries an identifier that
# is ALSO in the title on 1 846 of 1 846 concepts that have the key, so a
# frontmatter key list buys nothing and is not declared.
identified = _synthetic("krav/only-in-id", "Alminnelig krav", "Kravet gjelder anlegget.")
concepts = [*_lookup_corpus()[:10], identified]
assert okf_consume.lookup_hits(concepts, "Hva krever Krav 3.3.1-13?") == ()
in_id = _synthetic("krav/3.3.1-13", "Alminnelig krav", "Kravet gjelder anlegget.")
assert okf_consume.lookup_hits([*concepts, in_id], "Hva krever Krav 3.3.1-13?") == (
"krav/3.3.1-13",
)
def test_the_lookup_keeps_the_fused_order_among_the_concepts_it_lifts() -> None:
# Determinism with several hits: the lifted concepts keep the order the
# fusion gave them, which is itself declared down to the id tie-break.
concepts = _lookup_corpus(second_holder=True)
ranked = okf_consume.concept_scores(concepts, LOOKUP_QUESTION, {})
hits = okf_consume.lookup_hits(concepts, LOOKUP_QUESTION)
# `lookup_hits` reports in BYTE order, and the two orders disagree here on
# purpose: the second holder's id sorts second and its fusion rank is better.
assert hits == (LOOKUP_GOLD, LOOKUP_SECOND)
lifted = [concept.concept_id for concept, _, _ in ranked[: len(hits)]]
assert lifted == [LOOKUP_SECOND, LOOKUP_GOLD]
assert [concept.concept_id for concept, _, _ in ranked[len(hits) :]] == sorted(
concept.concept_id for concept in concepts if concept.concept_id not in hits
)

93
tools/okf_consume.py
View file

@ -577,12 +577,44 @@ def _split(folded: str) -> list[str]:
return [token for token in _TOKEN_SPLIT_RE.split(folded) if len(token) >= MIN_TOKEN_LENGTH] return [token for token in _TOKEN_SPLIT_RE.split(folded) if len(token) >= MIN_TOKEN_LENGTH]
def is_identifier(token: str) -> bool:
"""Whether a token is a NUMBER a document is known by, rather than a word.
The whole token, never a part of one: `normalise` emits an identifier as
one token, so a full match is what "this token is an identifier" means. A
bare number is not one -- `2023` has no separator, and every page number in
a corpus would become an identifier if it were.
"""
return _IDENTIFIER_RE.fullmatch(token) is not None
def tokens_match(left: str, right: str) -> bool: def tokens_match(left: str, right: str) -> bool:
"""Whether two tokens share a leading prefix of at least `MIN_SHARED_PREFIX`. """Whether two tokens share a leading prefix of at least `MIN_SHARED_PREFIX`.
Symmetric, and it degrades to equality for short tokens: two 4-character Symmetric, and it degrades to equality for short tokens: two 4-character
tokens match only if they are the same word. tokens match only if they are the same word.
**AN IDENTIFIER MATCHES BY EQUALITY ALONE**, and that is a defect fix
measured on the case it costs most rather than a preference. The prefix
rule was measured for Norwegian compounds, where `vare|ne` and
`vare|mottak` share a stem; a requirement number has no stem, and four
leading characters of `3.3.1-13` are four leading characters of every
requirement in section 3.3. MEASURED 2026-09-08 on a 446-concept bundle:
the unique identifier `3.3.1-13` reached **135** concepts under the prefix
rule and **1** under equality, which made the rarity weight rank a common
adjective as more informative than the number naming the document
(`docs/2026-09-08-sjeldenhetsvekt.md` SS 3). `54a0bc2` SS 1 named this
class -- "`df` measured over the colliding matcher measures collision
breadth, not rarity" -- and this is that sentence applied to the identifier
itself.
""" """
if is_identifier(left) or is_identifier(right):
# No floor, either: `MIN_SHARED_PREFIX` made a three-character
# identifier match NOTHING, not even itself. Measured on a 629-concept
# bundle, `9.2` reached 0 concepts under the matcher while sitting
# verbatim in one title, so a document known by a short number was
# unreachable by that number.
return left == right
limit = min(len(left), len(right)) limit = min(len(left), len(right))
if limit < MIN_SHARED_PREFIX: if limit < MIN_SHARED_PREFIX:
return False return False
@ -699,6 +731,43 @@ def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dic
} }
def lookup_hits(concepts: Sequence["Concept"], question: str) -> tuple[str, ...]:
"""The concepts a question NAMES, rather than the ones it describes.
A question carrying an identifier that sits VERBATIM in a concept's title
or id is a lookup, not a search: the reader already knows which document
they want and is spelling its number. Returns those concepts' ids, byte
sorted so several holders of one number arrive in a declared order, and the
EMPTY tuple whenever the question carries no identifier -- which is what
makes this rule invisible to every question that is not a lookup.
**It reads the text the title-and-id signal reads, and no frontmatter key
list is declared.** Measured 2026-09-08 on three real bundles: of the 1 846
concepts carrying a `req_number`, the identifier in that key is ALSO in the
title on **1 846** of them, and on **0** does the key carry an identifier
the title lacks. A key list would therefore have bought nothing here and
would have been a constant no measurement asked for. A bundle whose
identifiers live only in frontmatter is not served by this rule, and that
is stated rather than guessed at.
**Verbatim after `normalise`, so the three spellings of one identifier are
one lookup** (`_DASH_TO_HYPHEN`) -- but `.` and `-` are NOT interchangeable,
so a question spelling `1.10` does not find a document whose id spells it
`1-10`. Measured and left open.
"""
identifiers = {token for token in normalise(question) if is_identifier(token)}
if not identifiers:
return ()
return tuple(
sorted(
concept.concept_id
for concept in concepts
if identifiers
& set(normalise(f"{concept.title} {concept.concept_id.replace('/', ' ')}"))
)
)
def _overlap( def _overlap(
question_tokens: Sequence[str], question_tokens: Sequence[str],
candidate: str, candidate: str,
@ -814,6 +883,7 @@ def concept_scores(
*, *,
cost_vocabulary: bool = False, cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None, weights: Mapping[str, float] | None = None,
lookup: bool = True,
) -> list[tuple[Concept, float, int]]: ) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF. """Every concept, ordered best first, fused from three signals by RRF.
@ -829,6 +899,12 @@ def concept_scores(
**No float reaches the payload.** These scores order the cut; only ranks and **No float reaches the payload.** These scores order the cut; only ranks and
whole byte counts are emitted. whole byte counts are emitted.
`lookup=False` isolates the FUSION from the lookup partition below it, and
exists because two stages sharing one output cannot otherwise be measured
apart: the tests that state what the rarity weight does to the fusion, and
the harness that measures a lookup's effect, both need the fusion's own
order. No caller in the run path sets it and the CLI does not expose it.
The third element of each tuple is the concept's OWN lexical overlap -- The third element of each tuple is the concept's OWN lexical overlap --
signals 1 and 2 only, with the document prior excluded. The cut needs it signals 1 and 2 only, with the document prior excluded. The cut needs it
separately: a concept that answers nothing in the question, sitting in a separately: a concept that answers nothing in the question, sitting in a
@ -893,6 +969,23 @@ def concept_scores(
) )
by_id = {concept.concept_id: concept for concept in concepts} by_id = {concept.concept_id: concept for concept in concepts}
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key)) ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
named = set(lookup_hits(concepts, question)) if lookup else set()
if named:
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
# partition rather than a fourth signal. The form was chosen by
# measurement, not by preference: a fourth RRF signal was simulated on
# the same three bundles first and put the named concept at rank
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
# consumes RANKS ONLY, so any single signal contributes at most
# `1/(RRF_K + 1)` however certain it is, and a concept the question
# NAMES cannot outbid three signals that merely describe it
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
#
# STABLE: the named concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
ranked_ids = [key for key in ranked_ids if key in named] + [
key for key in ranked_ids if key not in named
]
return [ return [
(by_id[concept_id], fused[concept_id], lexical[concept_id]) for concept_id in ranked_ids (by_id[concept_id], fused[concept_id], lexical[concept_id]) for concept_id in ranked_ids
] ]