open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume): match an identifier by equality, deliver the concept a question names

Browse source 
A question naming a requirement number now delivers that requirement at rank 1
on all three vegnormal bundles (was 96, 9, 35 of 446, 1 133, 270). Two
mechanisms, both measured, both default because no published figure moves.

The matcher: `tokens_match` compared four leading characters, so the unique
identifier `3.3.1-13` read as 135 of 446 common and the rarity weight ranked a
common adjective above the number naming the document. An identifier now
matches by equality alone; df falls to 1/1/1. Words keep the prefix rule, which
was measured for Norwegian compounds. Equality has no floor either, so a
three-character identifier stops matching nothing at all -- measured, `9.2`
reached 0 concepts while sitting verbatim in one title.

The lookup: a question carrying an identifier that sits verbatim in a concept's
title or id is answered by a partition over the fusion's output, not by a
fourth signal. The form was chosen by measurement -- a fourth RRF signal was
simulated first and put the gold at rank 26 / 15 / 19, none of them delivered,
because RRF consumes ranks only and one signal contributes at most 1/(RRF_K+1).
No frontmatter key list is declared: of 1 846 concepts carrying `req_number`,
1 846 also carry that identifier in the title.

The matcher alone is NOT a monotone win (N200 9 -> 26, because that gold's body
cross-references a neighbouring number that the prefix rule counted as a hit on
the question's). Only the partition delivers; the table is in the record.

Consumer corpus: every named control byte-identical against a frozen
`git archive` copy of 116d3e1 -- four payload digests, eight candidate ranks,
six hit@8 rows, both known-negatives. One document that was withheld at
position 621 of 621 is now delivered at rank 1, on a corpus with no
requirement-number grammar at all.

13 tests (12 red before the rules existed), 7 mutations, 7 red. 1 320 passed.
Consumption-side only; no bundle ref moves.
Record: docs/2026-09-08-eksakt-oppslag.md

Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 13:27:59 +02:00
commit a37d5ced38
3 changed files with 638 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

93
tools/okf_consume.py
View file

@ -577,12 +577,44 @@ def _split(folded: str) -> list[str]:
return [token for token in _TOKEN_SPLIT_RE.split(folded) if len(token) >= MIN_TOKEN_LENGTH]
def is_identifier(token: str) -> bool:
"""Whether a token is a NUMBER a document is known by, rather than a word.
The whole token, never a part of one: `normalise` emits an identifier as
one token, so a full match is what "this token is an identifier" means. A
bare number is not one -- `2023` has no separator, and every page number in
a corpus would become an identifier if it were.
"""
return _IDENTIFIER_RE.fullmatch(token) is not None
def tokens_match(left: str, right: str) -> bool:
"""Whether two tokens share a leading prefix of at least `MIN_SHARED_PREFIX`.
Symmetric, and it degrades to equality for short tokens: two 4-character
tokens match only if they are the same word.
**AN IDENTIFIER MATCHES BY EQUALITY ALONE**, and that is a defect fix
measured on the case it costs most rather than a preference. The prefix
rule was measured for Norwegian compounds, where `vare|ne` and
`vare|mottak` share a stem; a requirement number has no stem, and four
leading characters of `3.3.1-13` are four leading characters of every
requirement in section 3.3. MEASURED 2026-09-08 on a 446-concept bundle:
the unique identifier `3.3.1-13` reached **135** concepts under the prefix
rule and **1** under equality, which made the rarity weight rank a common
adjective as more informative than the number naming the document
(`docs/2026-09-08-sjeldenhetsvekt.md` SS 3). `54a0bc2` SS 1 named this
class -- "`df` measured over the colliding matcher measures collision
breadth, not rarity" -- and this is that sentence applied to the identifier
itself.
"""
if is_identifier(left) or is_identifier(right):
# No floor, either: `MIN_SHARED_PREFIX` made a three-character
# identifier match NOTHING, not even itself. Measured on a 629-concept
# bundle, `9.2` reached 0 concepts under the matcher while sitting
# verbatim in one title, so a document known by a short number was
# unreachable by that number.
return left == right
limit = min(len(left), len(right))
if limit < MIN_SHARED_PREFIX:
return False
@ -699,6 +731,43 @@ def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dic
}
def lookup_hits(concepts: Sequence["Concept"], question: str) -> tuple[str, ...]:
"""The concepts a question NAMES, rather than the ones it describes.
A question carrying an identifier that sits VERBATIM in a concept's title
or id is a lookup, not a search: the reader already knows which document
they want and is spelling its number. Returns those concepts' ids, byte
sorted so several holders of one number arrive in a declared order, and the
EMPTY tuple whenever the question carries no identifier -- which is what
makes this rule invisible to every question that is not a lookup.
**It reads the text the title-and-id signal reads, and no frontmatter key
list is declared.** Measured 2026-09-08 on three real bundles: of the 1 846
concepts carrying a `req_number`, the identifier in that key is ALSO in the
title on **1 846** of them, and on **0** does the key carry an identifier
the title lacks. A key list would therefore have bought nothing here and
would have been a constant no measurement asked for. A bundle whose
identifiers live only in frontmatter is not served by this rule, and that
is stated rather than guessed at.
**Verbatim after `normalise`, so the three spellings of one identifier are
one lookup** (`_DASH_TO_HYPHEN`) -- but `.` and `-` are NOT interchangeable,
so a question spelling `1.10` does not find a document whose id spells it
`1-10`. Measured and left open.
"""
identifiers = {token for token in normalise(question) if is_identifier(token)}
if not identifiers:
return ()
return tuple(
sorted(
concept.concept_id
for concept in concepts
if identifiers
& set(normalise(f"{concept.title} {concept.concept_id.replace('/', ' ')}"))
)
)
def _overlap(
question_tokens: Sequence[str],
candidate: str,
@ -814,6 +883,7 @@ def concept_scores(
*,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
lookup: bool = True,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -829,6 +899,12 @@ def concept_scores(
**No float reaches the payload.** These scores order the cut; only ranks and
whole byte counts are emitted.
`lookup=False` isolates the FUSION from the lookup partition below it, and
exists because two stages sharing one output cannot otherwise be measured
apart: the tests that state what the rarity weight does to the fusion, and
the harness that measures a lookup's effect, both need the fusion's own
order. No caller in the run path sets it and the CLI does not expose it.
The third element of each tuple is the concept's OWN lexical overlap --
signals 1 and 2 only, with the document prior excluded. The cut needs it
separately: a concept that answers nothing in the question, sitting in a
@ -893,6 +969,23 @@ def concept_scores(
)
by_id = {concept.concept_id: concept for concept in concepts}
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
named = set(lookup_hits(concepts, question)) if lookup else set()
if named:
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
# partition rather than a fourth signal. The form was chosen by
# measurement, not by preference: a fourth RRF signal was simulated on
# the same three bundles first and put the named concept at rank
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
# consumes RANKS ONLY, so any single signal contributes at most
# `1/(RRF_K + 1)` however certain it is, and a concept the question
# NAMES cannot outbid three signals that merely describe it
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
#
# STABLE: the named concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
ranked_ids = [key for key in ranked_ids if key in named] + [
key for key in ranked_ids if key not in named
]
return [
(by_id[concept_id], fused[concept_id], lexical[concept_id]) for concept_id in ranked_ids
]