merge: o2c-oppslag - identifier equality matching and exact lookup delivery (a37d5ce)
This commit is contained in:
commit
d3bfe92acd
3 changed files with 638 additions and 4 deletions
344
docs/2026-09-08-eksakt-oppslag.md
Normal file
344
docs/2026-09-08-eksakt-oppslag.md
Normal file
|
|
@ -0,0 +1,344 @@
|
|||
# The exact lookup: naming a document is not searching for it, 2026-09-08
|
||||
|
||||
Two sessions made a requirement number visible to the consumption pre-pass and
|
||||
then made it *worth* more than a common word, and neither delivered the
|
||||
document the question named. The first left the gold `below_k` on all three
|
||||
bundles (`docs/2026-09-08-kravnummer-tokenisering.md` § 8); the second
|
||||
delivered one of three and decomposed the other two into a **matcher**
|
||||
decision and a **fusion** decision, with no measurement behind either
|
||||
(`docs/2026-09-08-sjeldenhetsvekt.md` § 3, § 4).
|
||||
|
||||
This session takes both decisions with numbers. The result is that a question
|
||||
naming a requirement number now delivers that requirement at **rank 1 on 3 of
|
||||
3 bundles**, and that every published figure on the consumer corpus is
|
||||
**byte-identical**.
|
||||
|
||||
---
|
||||
|
||||
## 0. What IS measured, and what is NOT
|
||||
|
||||
**Measured.** Eleven known-positive figures reproduced before anything was
|
||||
changed; document frequency under the matcher and under equality on three
|
||||
bundles; the gold's fused rank, verdict, payload digest and byte count before,
|
||||
after the matcher alone, and after both mechanisms; the three spellings of one
|
||||
identifier; every named control on the consumer corpus against a frozen
|
||||
`git archive` copy of `116d3e1`; a control question on that corpus that *does*
|
||||
carry an identifier, before and after; hit@8 over the six published questions
|
||||
and both known-negatives; the cost of each new pass; seven mutations.
|
||||
|
||||
**Known-positive, run first.** Every figure this session builds on was
|
||||
reproduced on this machine, with the committed tool, on a clean tree at
|
||||
`116d3e1`, before any after-column was read:
|
||||
|
||||
| # | known-positive | source | measured here |
|
||||
|---|---|---|---|
|
||||
| 1 | gold fused rank per bundle, weight off | previous session § 2 | **96, 9, 35** |
|
||||
| 2 | gold fused rank per bundle, weight on | previous session § 2 | **103, 8, 35** |
|
||||
| 3 | `df` under the matcher vs under equality | previous session § 3 | **135 / 17 / 5** vs **1 / 1 / 1** |
|
||||
| 4 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
|
||||
| 5 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
|
||||
| 6 | payload `sha256`, weight off | previous session § 2 | `49e9c718…`, `adc2eac7…`, `21c5bbfe…` |
|
||||
| 7 | payload `sha256`, weight on | previous session § 2 | `ba09a56a…`, `403ee989…`, `1758f6ca…` |
|
||||
| 8 | consumer corpus, the good question's candidate rank, four rows | previous session § 5 | **1 / 1 / 1 / 1** |
|
||||
| 9 | the priced sheet's candidate rank, four rows | previous session § 5, § 7 | **251 / 10 / 78 / 2** |
|
||||
| 10 | the mandate question's default payload | previous session § 7 | **169 583 B** |
|
||||
| 11 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6**, five of them at rank 1 |
|
||||
|
||||
**NOT measured**, named rather than glossed:
|
||||
|
||||
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
|
||||
machine and this repository's single-runtime-dependency rule does not admit
|
||||
one to produce a caption. Figures below are in `BUDGET_UNIT` — utf-8 bytes of
|
||||
emitted JSON — plus the payload `sha256`, which is exact where a byte count
|
||||
is only indicative.
|
||||
- **The rebuilt consumer bundle of `56c1205` is not on disk.** Its content
|
||||
identity (`sha256-tree:2f82fcfe…`) names a tree this machine no longer holds,
|
||||
and neither does the bundle the two S7 byte figures in the order come from
|
||||
(`2ae46f68…` 169 573 B and `8aaf8558…` 172 588 B). **Those two figures are
|
||||
therefore not reproduced here.** The control corpus used instead is the
|
||||
bundle delivered 2026-09-03 (`sha256-tree:9a4e5561…`), on which rows 8, 9 and
|
||||
10 of the table above reproduce the published values exactly. This costs the
|
||||
decision nothing: what the decision needs is byte-*identity* before and
|
||||
after, and that is measured on one and the same bundle, against a frozen copy
|
||||
of the committed code rather than against a remembered number.
|
||||
- **Whether a live model answers better.** This measures a *ranking*.
|
||||
"Delivered" is not "read", and that measurement belongs to the consumer.
|
||||
|
||||
---
|
||||
|
||||
## 1. Mechanism (a): an identifier matches by equality, and by nothing else
|
||||
|
||||
`tokens_match` compared four leading characters, for a reason that was measured
|
||||
and still holds: Norwegian compounds do not match token-exactly, and
|
||||
`vare|ne` and `vare|mottak` share a stem. A requirement number has no stem.
|
||||
Four leading characters of `3.3.1-13` are four leading characters of every
|
||||
requirement in section 3.3, so a **unique** identifier read as **135 of 446**
|
||||
common — and the rarity weight then faithfully ranked a common adjective as
|
||||
more informative than the number naming the document.
|
||||
|
||||
The rule, whole:
|
||||
|
||||
```python
|
||||
if is_identifier(left) or is_identifier(right):
|
||||
return left == right
|
||||
```
|
||||
|
||||
`is_identifier` is `_IDENTIFIER_RE.fullmatch`, the pattern the tokeniser
|
||||
already uses to keep an identifier whole. No new constant, no token class
|
||||
declared anywhere, and words are untouched.
|
||||
|
||||
**Red first.** `tokens_match('3.3.1-13', '3.3.1-14')` was `True` and is now
|
||||
`False`; `('3.3.1-13', '3.3.1-13')` was and is `True`. The known-positive ran
|
||||
in the same file and passed before and after: `brannsikring`/`brannvern`,
|
||||
`prisene`/`prissammenstilling`, `varene`/`varemottak` all still match, and
|
||||
`brann`/`bygg` still does not.
|
||||
|
||||
**`df` after, same instrument as before:**
|
||||
|
||||
| bundle | identifier | `df` before | `df` after | `df` exact | concepts |
|
||||
|---|---|---|---|---|---|
|
||||
| N100 | `3.3.1-13` | **135** | **1** | 1 | 446 |
|
||||
| N200 | `2.9.2-12` | 17 | **1** | 1 | 1 133 |
|
||||
| N500 | `10.2-2` | 5 | **1** | 1 | 270 |
|
||||
|
||||
**A short identifier stops being invisible.** `MIN_SHARED_PREFIX` made a
|
||||
three-character identifier match *nothing, not even itself*: measured on the
|
||||
629-concept consumer bundle, `9.2` reached **0** concepts under the matcher
|
||||
while sitting verbatim in one title. Equality has no floor, so it now reaches
|
||||
that one. The floor stays where it was measured — for words.
|
||||
|
||||
**What the matcher alone bought, and what it cost.** This is the honest column,
|
||||
and it is not all in one direction:
|
||||
|
||||
| bundle | gold fused rank, before | after (a) alone | with `--rarity-weight`, before → after |
|
||||
|---|---|---|---|
|
||||
| N100 | 96 | **53** | 103 → **45** |
|
||||
| N200 | 9 | **26** | 8 → **23** |
|
||||
| N500 | 35 | **37** | 35 → **36** |
|
||||
|
||||
**N200 got worse, and the reason is measured rather than guessed.** That gold's
|
||||
body carries **88** identifiers, three of them occurrences of `2.9.2-1` — a
|
||||
cross-reference to the neighbouring requirement. Under the prefix rule that
|
||||
cross-reference answered the question's `2.9.2-12`, so the gold was scoring a
|
||||
hit on its own body that it had not earned. Removing a false positive that
|
||||
happened to help is still removing a false positive; the number is here so that
|
||||
nobody has to take that on faith. On its own, mechanism (a) delivers **0 of 3**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Mechanism (b): the form was chosen by a measurement, not by a preference
|
||||
|
||||
A question carrying an identifier that sits verbatim in a concept's title or id
|
||||
is a **lookup**: the reader already knows which document they want and is
|
||||
spelling its number. The order named two candidate forms and invited a third.
|
||||
|
||||
**Form (i), a fourth signal inside the RRF fusion, was simulated first** on the
|
||||
same three bundles, on top of mechanism (a), with the lookup as a 0/1 signal:
|
||||
|
||||
| bundle | gold rank with a fourth RRF signal | of |
|
||||
|---|---|---|
|
||||
| N100 | **26** | 446 |
|
||||
| N200 | **15** | 1 133 |
|
||||
| N500 | **19** | 270 |
|
||||
|
||||
**None of them delivered**, and the reason is the one the previous session
|
||||
already wrote down: RRF consumes **ranks only**, so any single signal
|
||||
contributes at most `1/(RRF_K + 1)` however certain it is. A concept the
|
||||
question *names* cannot outbid three signals that merely describe it. Form (i)
|
||||
is refuted by its own numbers, not by taste.
|
||||
|
||||
**Form (ii) ships:** a partition applied to the fusion's output. Concepts whose
|
||||
title or id bears the question's identifier are moved to the front, keeping the
|
||||
order the fusion gave them; everything else keeps its order too. Nothing is
|
||||
dropped, nothing is scored twice, and the fused ranks themselves are untouched.
|
||||
|
||||
```python
|
||||
named = set(lookup_hits(concepts, question))
|
||||
if named:
|
||||
ranked_ids = [k for k in ranked_ids if k in named] + [k for k in ranked_ids if k not in named]
|
||||
```
|
||||
|
||||
**It reads the text the title-and-id signal reads, and declares no key list.**
|
||||
The order allowed a frontmatter key set (`req_number`, `identifier`, …). It is
|
||||
not declared, because a measurement decided instead: of the **1 846** concepts
|
||||
across the three bundles that carry a `req_number`, the identifier in that key
|
||||
is **also in the title on 1 846** of them, and on **0** does the key carry an
|
||||
identifier the title lacks. A key list would have bought nothing and would have
|
||||
been a constant no measurement asked for. The consumer corpus has no such key
|
||||
at all, and is served anyway (§ 4).
|
||||
|
||||
**The known-negative is structural, not lucky.** `lookup_hits` returns the
|
||||
empty tuple whenever the question carries no identifier, and **every one of the
|
||||
ten published control questions on the consumer corpus carries zero
|
||||
identifiers** under `_IDENTIFIER_RE` — the six hit@8 questions, both
|
||||
known-negatives, the good question and the mandate-shaped one. Verified with
|
||||
`normalise` on each question before any payload was built.
|
||||
|
||||
**Red first,** on a synthetic 300-concept corpus where every concept carries
|
||||
the word a standards corpus repeats on every page and the neighbours are
|
||||
numbered so that they share the gold's first four characters: the named concept
|
||||
comes back at rank 1; two concepts bearing the same number come back as the top
|
||||
two; a question with no identifier leaves the order untouched; an identifier no
|
||||
concept bears changes nothing and empties nothing.
|
||||
|
||||
---
|
||||
|
||||
## 3. After: three bundles, one instrument, before and after
|
||||
|
||||
Instrument, unchanged from the previous session and from § 0 above: the gold's
|
||||
position in the fused `ranked` list, plus the payload's own verdict from
|
||||
`build_payload`. Command:
|
||||
|
||||
```
|
||||
python3 tools/okf_consume.py <bundle> \
|
||||
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
|
||||
[--rarity-weight]
|
||||
```
|
||||
|
||||
| | N100:2023 | N200:2024 | N500:2024 |
|
||||
|---|---|---|---|
|
||||
| concepts | 446 | 1 133 | 270 |
|
||||
| gold fused rank, **before** | 96 | 9 | 35 |
|
||||
| gold fused rank, **after** | **1** | **1** | **1** |
|
||||
| gold verdict, before | `below_k` | `below_k` | `below_k` |
|
||||
| gold verdict, after | **delivered, rank 1** | **delivered, rank 1** | **delivered, rank 1** |
|
||||
| the same with `--rarity-weight` | **1**, delivered | **1**, delivered | **1**, delivered |
|
||||
| payload bytes, before → after | 61 566 → 58 409 | 142 861 → 146 145 | 40 626 → 39 193 |
|
||||
| payload `sha256`, after | `5bf6ab78…` | `af5829cd…` | `9a9e7860…` |
|
||||
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
|
||||
| lookup hits found | 1 | 1 | 1 |
|
||||
|
||||
**3 of 3, at rank 1.** The order's target was rank ≤ 8 on 3 of 3.
|
||||
|
||||
**The three spellings are one lookup.** For each bundle, the question was asked
|
||||
with an ASCII hyphen, an en dash and an em dash. All three deliver the **same
|
||||
eight concepts in the same order**, with the gold at rank 1 — the payload
|
||||
digests differ only because the question string is echoed in the payload, so
|
||||
the excerpt list is compared instead.
|
||||
|
||||
**Cost.** The lookup is one pass over the concepts already in memory:
|
||||
**0.011 s** over 446, **0.024 s** over 1 133, **0.006 s** over 270. The `df`
|
||||
pass the rarity weight uses, re-measured under the new matcher: 0.055 s,
|
||||
**0.227 s**, 0.064 s. Both are inside the order's one-second bound on the
|
||||
largest bundle by a factor of four or better, and a question carrying no
|
||||
identifier does not tokenise the corpus at all.
|
||||
|
||||
---
|
||||
|
||||
## 4. The consumer corpus: nothing moves, and one unreachable thing becomes reachable
|
||||
|
||||
Every named control was measured twice — once from a `git archive` copy of
|
||||
`116d3e1` extracted to a scratch tree, and once from this working tree — on the
|
||||
same bundle, with `PYTHONPATH` and `__file__` both checked so the frozen run
|
||||
could not import the working copy.
|
||||
|
||||
| control | before | after |
|
||||
|---|---|---|
|
||||
| good question, candidate rank | 1 of 233 | **1 of 233** |
|
||||
| good question with the cost vocabulary | 1 of 248 | **1 of 248** |
|
||||
| the same two with `--rarity-weight` | 1 / 1 | **1 / 1** |
|
||||
| priced sheet, mandate question, vocabulary off | 251 of 271 | **251 of 271** |
|
||||
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **10 of 280** |
|
||||
| the same two with `--rarity-weight` | 78 / 2 | **78 / 2** |
|
||||
| mandate question, default payload | `78fec2ef…`, 169 583 B | **byte-identical** |
|
||||
| mandate question, `--cost-vocabulary --k 12 --limit 160000` | `1f11e56b…`, 240 021 B | **byte-identical** |
|
||||
| mandate question, `--cost-vocabulary --k 12` | `777986a5…`, 172 246 B | **byte-identical** |
|
||||
| good question, default payload | `f7ff803e…`, 165 109 B | **byte-identical** |
|
||||
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank, byte count and spend identical |
|
||||
| both known-negatives | 8 delivered each | **byte-identical** |
|
||||
|
||||
**Every figure in that table is identical** — the four payload digests and
|
||||
their byte counts, the eight candidate ranks, all six hit@8 rows with their
|
||||
spends, and both known-negatives. No consumer's bytes move because these two
|
||||
changes exist — which is what makes them safe to ship on by default rather than
|
||||
behind a flag.
|
||||
|
||||
**Generality, on a corpus with no requirement numbers at all.** The consumer
|
||||
bundle has no `req_number` key and no requirement grammar; its documents are
|
||||
numbered in their titles instead (`… 9.2 …`). One control question was written
|
||||
that names such a number. Before, that document was withheld `below_k` at
|
||||
position **621 of 621** — dead last, because a three-character identifier
|
||||
matched nothing under `MIN_SHARED_PREFIX`, not even itself. After, it is
|
||||
**delivered at rank 1**. The mechanism is not a vegnormal mechanism.
|
||||
|
||||
---
|
||||
|
||||
## 5. The decision, made by the number
|
||||
|
||||
The order's rule: (a) is a defect fix and ships as default if every control on
|
||||
the consumer corpus stands byte-identical; (b) ships as default if the
|
||||
known-negative holds.
|
||||
|
||||
| mechanism | condition | measured | decision |
|
||||
|---|---|---|---|
|
||||
| (a) identifier matches by equality | all consumer controls byte-identical | **every control identical** | **default, no flag** |
|
||||
| (b) the lookup partition | byte-identical for a question with no identifier | **structural: 0 identifiers in 10 of 10 control questions; every payload identical** | **default, no flag** |
|
||||
|
||||
Nothing fell, so no flag is added and nothing has to be named as a loss.
|
||||
`--rarity-weight` is untouched and stays **off**; with it on, the three lookups
|
||||
also deliver at rank 1, so the two mechanisms do not fight.
|
||||
|
||||
---
|
||||
|
||||
## 6. What did NOT close
|
||||
|
||||
- **`.` and `-` are not interchangeable in a lookup.** `_DASH_TO_HYPHEN` folds
|
||||
every dash spelling to one, so em dash, en dash and hyphen are one lookup —
|
||||
but a question spelling `1.10` does not find a document whose id spells it
|
||||
`1-10`, and this library's own id grammar produces exactly that hyphenated
|
||||
form from a dotted source. Measured and left open: the fix is a
|
||||
canonicalisation inside the lookup, and it needs its own before/after.
|
||||
- **A bundle whose identifiers live only in frontmatter is not served.** Today
|
||||
that set is empty (0 of 1 846), which is why no key list was declared. If a
|
||||
consumer arrives with one, this is the rule that has to grow.
|
||||
- **An identifier that appears only in a concept's BODY is not a lookup.** It
|
||||
is a search, and it ranks like one.
|
||||
- **The matcher alone made two of three lookups worse** (§ 1). Only the
|
||||
partition delivers. Anyone lifting one half without the other should read
|
||||
that table first.
|
||||
- **The two S7 byte figures from the O1 bundle are not reproduced**, because
|
||||
neither that bundle nor the rebuilt one is on this machine (§ 0).
|
||||
- **"Delivered" is not "read".** All three golds are now in the payload at rank
|
||||
1. Whether a model then answers the engineer's question is the consumer's
|
||||
measurement, and this document does not claim it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Tests, mutations, gates
|
||||
|
||||
Thirteen new tests. **Twelve were written before either rule existed and
|
||||
eleven of them failed**; the twelfth is the known-positive that had to pass
|
||||
from the start (the Norwegian compounds), so the red was the rules' absence and
|
||||
not a broken fixture. The thirteenth was written afterwards, against a mutation
|
||||
that survived, and is red against it. One fixture defect was found by the red
|
||||
run itself: the neighbours were numbered so that one of them *was* the gold's
|
||||
number, which made the `df` 2 rather than 1.
|
||||
|
||||
**Seven mutations, seven red.** Three survived the first pass and none was
|
||||
banded blind:
|
||||
|
||||
- **M2** (`is_identifier` matches a substring instead of the whole token)
|
||||
survived because no token `normalise` emits can tell the two apart — the
|
||||
generic split keeps neither `.` nor `-`, so a split token can never contain
|
||||
an identifier. The commitment is real even where no output distinguishes it,
|
||||
so it is now asserted on the function.
|
||||
- **M4** (the early return for a question with no identifier) survived because
|
||||
an empty identifier set intersects to nothing anyway. The guard is a **cost**
|
||||
commitment, not a semantic one, so it is now measured as one: a question
|
||||
without an identifier must not tokenise the corpus at all.
|
||||
- **M6** (the lifted concepts re-sorted by id instead of keeping the fused
|
||||
order) survived because the fixture's two holders tied. The fixture was
|
||||
corrected so byte order and fused order disagree on purpose.
|
||||
|
||||
| gate | result |
|
||||
|---|---|
|
||||
| `pytest -q` | **1 320 passed**, 1 skipped (1 307 on this branch before, +13) |
|
||||
| `ruff check src tests tools` | clean |
|
||||
| `ruff format --check .` | clean, 88 files |
|
||||
| `mypy --strict src/` | clean, 17 files |
|
||||
| `okf_contract_check` on the three lookup payloads | exit 0, 14 rules, 0 findings |
|
||||
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
|
||||
|
||||
**No rebuild.** The change is consumption-side only, confirmed by
|
||||
`git diff --stat`: no production module under `src/` is touched, so every
|
||||
bundle ref in flight is unchanged.
|
||||
|
|
@ -1669,9 +1669,12 @@ def test_counting_leaves_the_identifier_worth_no_more_than_the_common_word() ->
|
|||
# The known-positive for the weight: without it, a rank of 1 with the
|
||||
# weight on would prove nothing, because nothing would have been shown to
|
||||
# hold the gold down in the first place.
|
||||
# `lookup=False` throughout: this fixture puts the identifier in the
|
||||
# gold's TITLE, which the lookup partition (Step 14) answers at rank one.
|
||||
# The claim here is about the FUSION, so the fusion is what is read.
|
||||
for identifier_in_body in (False, True):
|
||||
concepts = _rarity_corpus(identifier_in_body=identifier_in_body)
|
||||
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {})
|
||||
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, lookup=False)
|
||||
assert _rank_of(ranked, RARITY_GOLD) == 18
|
||||
lexical = {concept.concept_id: value for concept, _, value in ranked}
|
||||
assert lexical["krav/c-00"] == 6
|
||||
|
|
@ -1716,7 +1719,9 @@ def test_a_rarity_weight_cannot_move_a_signal_the_gold_already_leads() -> None:
|
|||
# And the rank it buys is the rank it already had, so the fusion sees none of it.
|
||||
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1
|
||||
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1
|
||||
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
|
||||
ranked = okf_consume.concept_scores(
|
||||
concepts, RARITY_QUESTION, {}, weights=weights, lookup=False
|
||||
)
|
||||
assert _rank_of(ranked, RARITY_GOLD) == 18
|
||||
|
||||
|
||||
|
|
@ -1885,11 +1890,203 @@ def test_the_weight_reaches_the_title_signal_and_reorders_it() -> None:
|
|||
)
|
||||
assert _signal_rank(concepts, question, "krav/t-29", None) == 30
|
||||
assert _signal_rank(concepts, question, "krav/t-29", weights) == 1
|
||||
plain = okf_consume.concept_scores(concepts, question, {})
|
||||
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights)
|
||||
# `lookup=False`: the gold's title bears the identifier, so the lookup
|
||||
# partition answers this question at rank one. The fusion is the claim.
|
||||
plain = okf_consume.concept_scores(concepts, question, {}, lookup=False)
|
||||
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights, lookup=False)
|
||||
# Fused: 30 -> 18. The title signal is reordered from last to first and the
|
||||
# fused rank moves by twelve, not to one -- the other two signals still
|
||||
# cannot see the identifier. That gap IS the finding of this session, and
|
||||
# the numbers are here so a change to either half shows up as a diff.
|
||||
assert _rank_of(plain, "krav/t-29") == 30
|
||||
assert _rank_of(weighted, "krav/t-29") == 18
|
||||
|
||||
|
||||
# --- Step 14: exact identifier matching and the lookup signal (O2c) -----------
|
||||
|
||||
#: The lookup fixture: 300 concepts that all carry the word a standards corpus
|
||||
#: repeats on every page, one of which also carries the identifier. Synthetic
|
||||
#: rather than borrowed, and 300 rather than 30 so a rank of one is a claim
|
||||
#: about the rule and not about a small corpus.
|
||||
LOOKUP_QUESTION = "Hva krever Krav 3.3.1-13? Gjengi det sentrale vilkåret."
|
||||
LOOKUP_GOLD = "krav/c-299"
|
||||
LOOKUP_SECOND = "krav/z-000"
|
||||
|
||||
|
||||
def _lookup_corpus(*, second_holder: bool = False) -> list[okf_consume.Concept]:
|
||||
"""299 neighbours numbered `3.3.1-<n>`, plus the concept the question names.
|
||||
|
||||
The neighbours share the identifier's first four characters, which is
|
||||
exactly what made a unique requirement number read as 135-of-446 common on
|
||||
a real bundle. The gold's `concept_id` sorts LAST, so nothing but the rule
|
||||
can lift it.
|
||||
"""
|
||||
body = "Kravet gjengir det sentrale vilkåret for anlegget."
|
||||
concepts = [
|
||||
_synthetic(f"krav/c-{index:03d}", f"Krav 3.3.1-{index + 20} Alminnelig krav", body)
|
||||
for index in range(299)
|
||||
]
|
||||
concepts.append(_synthetic(LOOKUP_GOLD, "Krav 3.3.1-13 Tekniske bygg", body))
|
||||
if second_holder:
|
||||
# The second holder's id sorts AFTER the gold's and its title answers
|
||||
# four question tokens more, so the fusion ranks it FIRST while byte
|
||||
# order ranks it second. The two orders disagree on purpose: a lookup
|
||||
# that re-sorted its hits by id instead of keeping the fused order
|
||||
# would otherwise be indistinguishable from one that keeps it.
|
||||
concepts.append(
|
||||
_synthetic(LOOKUP_SECOND, "Krav 3.3.1-13 Gjengi det sentrale vilkåret", body)
|
||||
)
|
||||
return concepts
|
||||
|
||||
|
||||
def test_an_identifier_matches_its_own_spelling_and_no_neighbouring_number() -> None:
|
||||
# The defect, at the case it costs most: `3.3.1-13` and `3.3.1-14` share
|
||||
# four leading characters, so the prefix rule called them a match and every
|
||||
# requirement number beginning `3.3.` counted as a hit.
|
||||
assert okf_consume.tokens_match("3.3.1-13", "3.3.1-13") is True
|
||||
assert okf_consume.tokens_match("3.3.1-13", "3.3.1-14") is False
|
||||
assert okf_consume.tokens_match("3.3.1-13", "3.3.2-13") is False
|
||||
# Symmetric, like the rule it replaces.
|
||||
assert okf_consume.tokens_match("3.3.1-14", "3.3.1-13") is False
|
||||
|
||||
|
||||
def test_a_norwegian_compound_still_matches_on_the_shared_prefix() -> None:
|
||||
# The known-positive. `MIN_SHARED_PREFIX` exists because Norwegian
|
||||
# compounds do not match token-exactly, and an identifier rule that also
|
||||
# narrowed words would buy one lookup by losing every compound.
|
||||
assert okf_consume.tokens_match("brannsikring", "brannvern") is True
|
||||
assert okf_consume.tokens_match("prisene", "prissammenstilling") is True
|
||||
assert okf_consume.tokens_match("varene", "varemottak") is True
|
||||
assert okf_consume.tokens_match("brann", "bygg") is False
|
||||
|
||||
|
||||
def test_a_short_identifier_becomes_reachable_because_equality_has_no_floor() -> None:
|
||||
# `MIN_SHARED_PREFIX` made a three-character identifier match NOTHING, not
|
||||
# even itself: measured on a 629-concept bundle, `9.2` reached 0 concepts
|
||||
# under the matcher while sitting verbatim in one title.
|
||||
assert okf_consume.tokens_match("9.2", "9.2") is True
|
||||
assert okf_consume.tokens_match("9.2", "9.3") is False
|
||||
# The floor still stands for words, which is what it was measured for.
|
||||
assert okf_consume.tokens_match("veg", "veg") is False
|
||||
|
||||
|
||||
def test_the_identifier_rule_makes_a_unique_number_unique_in_the_document_frequency() -> None:
|
||||
# The df is what the rarity weight reads, and 135-of-446 was the reason a
|
||||
# unique requirement number weighed less than a common adjective.
|
||||
concepts = _lookup_corpus()
|
||||
corpus = okf_consume.searchable_text(concepts)
|
||||
identifier = "3.3.1-13"
|
||||
under_matcher = sum(
|
||||
1
|
||||
for text in corpus
|
||||
if any(okf_consume.tokens_match(identifier, other) for other in okf_consume.normalise(text))
|
||||
)
|
||||
assert under_matcher == 1
|
||||
|
||||
|
||||
def test_is_identifier_names_numbers_and_not_words() -> None:
|
||||
assert okf_consume.is_identifier("3.3.1-13") is True
|
||||
assert okf_consume.is_identifier("10.2-2") is True
|
||||
assert okf_consume.is_identifier("r610.4") is True
|
||||
assert okf_consume.is_identifier("brannsikring") is False
|
||||
assert okf_consume.is_identifier("krav") is False
|
||||
# A bare number is not an identifier: it has no separator, and every page
|
||||
# number in a corpus would become one.
|
||||
assert okf_consume.is_identifier("2023") is False
|
||||
# THE WHOLE TOKEN, never a part of one. No token `normalise` emits can tell
|
||||
# a whole-token rule from a substring rule -- the generic split keeps
|
||||
# neither `.` nor `-`, so a split token can never contain an identifier --
|
||||
# which is why the commitment is asserted on the function rather than on an
|
||||
# output it happens to leave unchanged.
|
||||
assert okf_consume.is_identifier("bilag3.3-1x") is False
|
||||
|
||||
|
||||
def test_a_question_naming_an_identifier_delivers_the_concept_that_bears_it() -> None:
|
||||
# The lookup: 300 concepts all answer `krav`, one is NAMED. Rank one, not
|
||||
# rank eight -- a question that names a concept is not a search.
|
||||
concepts = _lookup_corpus()
|
||||
ranked = okf_consume.concept_scores(concepts, LOOKUP_QUESTION, {})
|
||||
assert _rank_of(ranked, LOOKUP_GOLD) == 1
|
||||
|
||||
|
||||
def test_two_concepts_bearing_the_same_identifier_both_reach_the_top() -> None:
|
||||
concepts = _lookup_corpus(second_holder=True)
|
||||
ranked = okf_consume.concept_scores(concepts, LOOKUP_QUESTION, {})
|
||||
top = [concept.concept_id for concept, _, _ in ranked[:2]]
|
||||
assert sorted(top) == sorted([LOOKUP_GOLD, LOOKUP_SECOND])
|
||||
|
||||
|
||||
def test_the_lookup_is_invisible_to_a_question_that_names_no_identifier() -> None:
|
||||
# The KNOWN-NEGATIVE. Every published control question on the consumer
|
||||
# corpus carries zero identifiers, so this is the property that lets the
|
||||
# rule ship on by default.
|
||||
concepts = _lookup_corpus()
|
||||
question = "Hvordan skal det sentrale vilkåret oppfylles?"
|
||||
assert okf_consume.lookup_hits(concepts, question) == ()
|
||||
ranked = okf_consume.concept_scores(concepts, question, {})
|
||||
order = [concept.concept_id for concept, _, _ in ranked]
|
||||
assert order == sorted(order)
|
||||
|
||||
|
||||
def test_a_question_without_an_identifier_never_reads_the_corpus(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# The early return is a COST commitment, not a semantic one: an empty
|
||||
# identifier set intersects to nothing anyway, so no ranking distinguishes
|
||||
# the guard from its absence. What it does buy is the corpus never being
|
||||
# tokenised for a question that cannot be a lookup -- measured, the lookup
|
||||
# pass costs 0.024 s over 1 133 concepts when it does run.
|
||||
concepts = _lookup_corpus()
|
||||
calls: list[str] = []
|
||||
original = okf_consume.normalise
|
||||
monkeypatch.setattr(
|
||||
okf_consume, "normalise", lambda text: (calls.append(text), original(text))[1]
|
||||
)
|
||||
assert okf_consume.lookup_hits(concepts, "Hvordan oppfylles vilkåret?") == ()
|
||||
assert calls == ["Hvordan oppfylles vilkåret?"]
|
||||
|
||||
|
||||
def test_an_identifier_no_concept_bears_changes_nothing_and_empties_nothing() -> None:
|
||||
concepts = _lookup_corpus()
|
||||
question = "Hva krever Krav 9.9.9-99? Gjengi det sentrale vilkåret."
|
||||
assert okf_consume.lookup_hits(concepts, question) == ()
|
||||
ranked = okf_consume.concept_scores(concepts, question, {})
|
||||
assert len(ranked) == len(concepts)
|
||||
order = [concept.concept_id for concept, _, _ in ranked]
|
||||
assert order == sorted(order)
|
||||
|
||||
|
||||
def test_the_three_spellings_of_one_identifier_are_one_lookup() -> None:
|
||||
concepts = _lookup_corpus()
|
||||
for dash in ("-", "–", "—"):
|
||||
question = f"Hva krever Krav 3.3.1{dash}13? Gjengi det sentrale vilkåret."
|
||||
assert okf_consume.lookup_hits(concepts, question) == (LOOKUP_GOLD,)
|
||||
|
||||
|
||||
def test_the_lookup_reads_the_text_the_title_signal_reads() -> None:
|
||||
# Measured on three real bundles: `req_number` carries an identifier that
|
||||
# is ALSO in the title on 1 846 of 1 846 concepts that have the key, so a
|
||||
# frontmatter key list buys nothing and is not declared.
|
||||
identified = _synthetic("krav/only-in-id", "Alminnelig krav", "Kravet gjelder anlegget.")
|
||||
concepts = [*_lookup_corpus()[:10], identified]
|
||||
assert okf_consume.lookup_hits(concepts, "Hva krever Krav 3.3.1-13?") == ()
|
||||
in_id = _synthetic("krav/3.3.1-13", "Alminnelig krav", "Kravet gjelder anlegget.")
|
||||
assert okf_consume.lookup_hits([*concepts, in_id], "Hva krever Krav 3.3.1-13?") == (
|
||||
"krav/3.3.1-13",
|
||||
)
|
||||
|
||||
|
||||
def test_the_lookup_keeps_the_fused_order_among_the_concepts_it_lifts() -> None:
|
||||
# Determinism with several hits: the lifted concepts keep the order the
|
||||
# fusion gave them, which is itself declared down to the id tie-break.
|
||||
concepts = _lookup_corpus(second_holder=True)
|
||||
ranked = okf_consume.concept_scores(concepts, LOOKUP_QUESTION, {})
|
||||
hits = okf_consume.lookup_hits(concepts, LOOKUP_QUESTION)
|
||||
# `lookup_hits` reports in BYTE order, and the two orders disagree here on
|
||||
# purpose: the second holder's id sorts second and its fusion rank is better.
|
||||
assert hits == (LOOKUP_GOLD, LOOKUP_SECOND)
|
||||
lifted = [concept.concept_id for concept, _, _ in ranked[: len(hits)]]
|
||||
assert lifted == [LOOKUP_SECOND, LOOKUP_GOLD]
|
||||
assert [concept.concept_id for concept, _, _ in ranked[len(hits) :]] == sorted(
|
||||
concept.concept_id for concept in concepts if concept.concept_id not in hits
|
||||
)
|
||||
|
|
|
|||
|
|
@ -577,12 +577,44 @@ def _split(folded: str) -> list[str]:
|
|||
return [token for token in _TOKEN_SPLIT_RE.split(folded) if len(token) >= MIN_TOKEN_LENGTH]
|
||||
|
||||
|
||||
def is_identifier(token: str) -> bool:
|
||||
"""Whether a token is a NUMBER a document is known by, rather than a word.
|
||||
|
||||
The whole token, never a part of one: `normalise` emits an identifier as
|
||||
one token, so a full match is what "this token is an identifier" means. A
|
||||
bare number is not one -- `2023` has no separator, and every page number in
|
||||
a corpus would become an identifier if it were.
|
||||
"""
|
||||
return _IDENTIFIER_RE.fullmatch(token) is not None
|
||||
|
||||
|
||||
def tokens_match(left: str, right: str) -> bool:
|
||||
"""Whether two tokens share a leading prefix of at least `MIN_SHARED_PREFIX`.
|
||||
|
||||
Symmetric, and it degrades to equality for short tokens: two 4-character
|
||||
tokens match only if they are the same word.
|
||||
|
||||
**AN IDENTIFIER MATCHES BY EQUALITY ALONE**, and that is a defect fix
|
||||
measured on the case it costs most rather than a preference. The prefix
|
||||
rule was measured for Norwegian compounds, where `vare|ne` and
|
||||
`vare|mottak` share a stem; a requirement number has no stem, and four
|
||||
leading characters of `3.3.1-13` are four leading characters of every
|
||||
requirement in section 3.3. MEASURED 2026-09-08 on a 446-concept bundle:
|
||||
the unique identifier `3.3.1-13` reached **135** concepts under the prefix
|
||||
rule and **1** under equality, which made the rarity weight rank a common
|
||||
adjective as more informative than the number naming the document
|
||||
(`docs/2026-09-08-sjeldenhetsvekt.md` SS 3). `54a0bc2` SS 1 named this
|
||||
class -- "`df` measured over the colliding matcher measures collision
|
||||
breadth, not rarity" -- and this is that sentence applied to the identifier
|
||||
itself.
|
||||
"""
|
||||
if is_identifier(left) or is_identifier(right):
|
||||
# No floor, either: `MIN_SHARED_PREFIX` made a three-character
|
||||
# identifier match NOTHING, not even itself. Measured on a 629-concept
|
||||
# bundle, `9.2` reached 0 concepts under the matcher while sitting
|
||||
# verbatim in one title, so a document known by a short number was
|
||||
# unreachable by that number.
|
||||
return left == right
|
||||
limit = min(len(left), len(right))
|
||||
if limit < MIN_SHARED_PREFIX:
|
||||
return False
|
||||
|
|
@ -699,6 +731,43 @@ def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dic
|
|||
}
|
||||
|
||||
|
||||
def lookup_hits(concepts: Sequence["Concept"], question: str) -> tuple[str, ...]:
|
||||
"""The concepts a question NAMES, rather than the ones it describes.
|
||||
|
||||
A question carrying an identifier that sits VERBATIM in a concept's title
|
||||
or id is a lookup, not a search: the reader already knows which document
|
||||
they want and is spelling its number. Returns those concepts' ids, byte
|
||||
sorted so several holders of one number arrive in a declared order, and the
|
||||
EMPTY tuple whenever the question carries no identifier -- which is what
|
||||
makes this rule invisible to every question that is not a lookup.
|
||||
|
||||
**It reads the text the title-and-id signal reads, and no frontmatter key
|
||||
list is declared.** Measured 2026-09-08 on three real bundles: of the 1 846
|
||||
concepts carrying a `req_number`, the identifier in that key is ALSO in the
|
||||
title on **1 846** of them, and on **0** does the key carry an identifier
|
||||
the title lacks. A key list would therefore have bought nothing here and
|
||||
would have been a constant no measurement asked for. A bundle whose
|
||||
identifiers live only in frontmatter is not served by this rule, and that
|
||||
is stated rather than guessed at.
|
||||
|
||||
**Verbatim after `normalise`, so the three spellings of one identifier are
|
||||
one lookup** (`_DASH_TO_HYPHEN`) -- but `.` and `-` are NOT interchangeable,
|
||||
so a question spelling `1.10` does not find a document whose id spells it
|
||||
`1-10`. Measured and left open.
|
||||
"""
|
||||
identifiers = {token for token in normalise(question) if is_identifier(token)}
|
||||
if not identifiers:
|
||||
return ()
|
||||
return tuple(
|
||||
sorted(
|
||||
concept.concept_id
|
||||
for concept in concepts
|
||||
if identifiers
|
||||
& set(normalise(f"{concept.title} {concept.concept_id.replace('/', ' ')}"))
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _overlap(
|
||||
question_tokens: Sequence[str],
|
||||
candidate: str,
|
||||
|
|
@ -814,6 +883,7 @@ def concept_scores(
|
|||
*,
|
||||
cost_vocabulary: bool = False,
|
||||
weights: Mapping[str, float] | None = None,
|
||||
lookup: bool = True,
|
||||
) -> list[tuple[Concept, float, int]]:
|
||||
"""Every concept, ordered best first, fused from three signals by RRF.
|
||||
|
||||
|
|
@ -829,6 +899,12 @@ def concept_scores(
|
|||
**No float reaches the payload.** These scores order the cut; only ranks and
|
||||
whole byte counts are emitted.
|
||||
|
||||
`lookup=False` isolates the FUSION from the lookup partition below it, and
|
||||
exists because two stages sharing one output cannot otherwise be measured
|
||||
apart: the tests that state what the rarity weight does to the fusion, and
|
||||
the harness that measures a lookup's effect, both need the fusion's own
|
||||
order. No caller in the run path sets it and the CLI does not expose it.
|
||||
|
||||
The third element of each tuple is the concept's OWN lexical overlap --
|
||||
signals 1 and 2 only, with the document prior excluded. The cut needs it
|
||||
separately: a concept that answers nothing in the question, sitting in a
|
||||
|
|
@ -893,6 +969,23 @@ def concept_scores(
|
|||
)
|
||||
by_id = {concept.concept_id: concept for concept in concepts}
|
||||
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
|
||||
named = set(lookup_hits(concepts, question)) if lookup else set()
|
||||
if named:
|
||||
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
|
||||
# partition rather than a fourth signal. The form was chosen by
|
||||
# measurement, not by preference: a fourth RRF signal was simulated on
|
||||
# the same three bundles first and put the named concept at rank
|
||||
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
|
||||
# consumes RANKS ONLY, so any single signal contributes at most
|
||||
# `1/(RRF_K + 1)` however certain it is, and a concept the question
|
||||
# NAMES cannot outbid three signals that merely describe it
|
||||
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
|
||||
#
|
||||
# STABLE: the named concepts keep the order the fusion gave them, and
|
||||
# so does everything else, so nothing here depends on dict order.
|
||||
ranked_ids = [key for key in ranked_ids if key in named] + [
|
||||
key for key in ranked_ids if key not in named
|
||||
]
|
||||
return [
|
||||
(by_id[concept_id], fused[concept_id], lexical[concept_id]) for concept_id in ranked_ids
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue