open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): weight a lexical hit by its rarity, off by default and measured

Browse source 
O2b asked whether a requirement number can be made worth more than a common
word by weighting each hit with the token's rarity in the bundle, with no
hand-set constant and no declared token class. It can, on one of the three
bundles, and the two it cannot are decomposed rather than guessed.

The rule is log(N/df) over the concepts' own tokens, counted with the same
four-character prefix rule a hit is scored with. It enters the RANKING and
never the GATE: `lexical` stays a count, because `krav` weighs exactly 0 on
all three bundles and a weighted gate would drop every concept matching only
that word -- which is the gate 54a0bc2 falsified for other reasons. One df
table per bundle reaches every stage that scores the question against text,
including the document prior. One pass, 0.241 s over 1 133 concepts.

Measured on four corpora, before and after, with every published figure
reproduced first: gold fused rank 96 -> 103, 9 -> 8 (withheld -> DELIVERED at
rank 8) and 35 -> 35; K2's priced sheet candidate rank 10 -> 2 with the cost
vocabulary and 251 -> 78 without; Q-good unmoved at rank 1; hit@8 5 of 6 with
every rank identical; the S7 control payload byte-identical on the default
command.

DEFAULT OFF, decided by the number and not by taste: it does not win on all
four, because N100's gold loses seven rank positions. Off means the bytes that
were already published, and that is measured -- 8 of 8 payload digests
identical against a frozen copy of 56c1205 built with git archive.

Two limits, both someone else's mechanism and both named: MIN_SHARED_PREFIX=4
makes a unique identifier read as 135-of-446 common on N100, so the weight
correctly ranks a common adjective above the exact requirement number; and RRF
consumes RANKS only, so on N500 -- where the gold already leads the one signal
that can see the identifier, and the other two cannot see it at all -- no
weighting inside a signal can move anything.

Consumption-side only, so no rebuild: the K2 bundle ref 2f82fcfe... stands.

Report: docs/2026-09-08-sjeldenhetsvekt.md. 13 new tests, red first; 8
mutations, 8 red, two of them only after the survivors were read as code -- one
exposed a fixture that put the identifier where the real corpus does not, and
the corrected fixture is what found the RRF limit. Suite 1295 -> 1308.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 12:36:57 +02:00
commit 116d3e1007
5 changed files with 755 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

153
tools/okf_consume.py
View file

@ -34,6 +34,7 @@ from __future__ import annotations
import argparse
import hashlib
import json
import math
import re
import sys
import unicodedata
@ -642,14 +643,80 @@ def question_uses_cost_vocabulary(question: str) -> bool:
return any(in_cost_vocabulary(token) for token in normalise(question))
def searchable_text(concepts: Sequence["Concept"]) -> list[str]:
"""The text a concept is scored against, one string per concept.
The same two fields the ranker's two lexical signals read -- title plus
id, and body -- joined, so a `df` counted here is a `df` over exactly what
a hit can be scored on. Counting rarity over one field and matching on
another would weight a token by how rare it is somewhere it is not read.
"""
return [
f"{concept.title} {concept.concept_id.replace('/', ' ')} {concept.body}"
for concept in concepts
]
def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dict[str, float]:
"""What one hit on each question token is worth, from the bundle alone.
`log(N / df)`: `N` concepts, and `df` the number of them bearing the token
under the SAME prefix rule a hit is scored with. No constant is set by
hand and no class of token is declared anywhere -- a word every concept
carries weighs exactly `log(1) == 0` of itself, and an identifier one
concept carries takes the corpus's maximum of itself.
**What this is NOT.** `54a0bc2` swept smoothed IDF as a GATE -- a threshold
below which a concept is withheld -- and falsified it: no threshold zeroed
both known-negatives while any positive question still reached its gold
document. That result stands and is not re-litigated here. This is the
other use: an ordering inside the candidate set, with the gate untouched
and `lexical` still a count. A ranking cannot withhold anything, so the
failure mode that refuted the gate has no counterpart here.
**`df` is measured over the colliding matcher, and so measures collision
breadth as well as rarity** (`54a0bc2` § 1: every `brann*` compound shares
four leading characters). Inherited deliberately rather than fixed here:
the weight must agree with the matcher it weights, and changing the matcher
is a different change with its own measurement.
One pass over the corpus. A token borne by no concept takes the weight of
a token borne by one -- it is never consumed, because a token that matches
nothing is never a hit.
"""
total = len(corpus)
if total == 0:
return {token: 0.0 for token in question_tokens}
counts = {token: 0 for token in question_tokens}
for text in corpus:
candidate_tokens = normalise(text)
for token in counts:
if any(tokens_match(token, other) for other in candidate_tokens):
counts[token] += 1
return {
token: math.log(total / count) if count else math.log(total)
for token, count in counts.items()
}
def _overlap(
question_tokens: Sequence[str], candidate: str, *, cost_vocabulary: bool = False
) -> int:
"""How many of the question's tokens the candidate text answers to."""
question_tokens: Sequence[str],
candidate: str,
*,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
) -> float:
"""What the candidate text answers of the question.
A COUNT when `weights` is None -- one per question token the candidate
answers to, which is what every caller got before rarity weighting existed
and what the cut still reads. With `weights`, the sum of those tokens'
rarity weights instead.
"""
candidate_tokens = normalise(candidate)
bridged = cost_vocabulary and any(in_cost_vocabulary(token) for token in candidate_tokens)
return sum(
1
1 if weights is None else weights.get(token, 1.0)
for token in question_tokens
if any(tokens_match(token, other) for other in candidate_tokens)
or (bridged and in_cost_vocabulary(token))
@ -662,6 +729,7 @@ def document_scores(
*,
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
) -> dict[str, float]:
"""One score per top-level document, from the indexes and the paths alone.
@ -696,14 +764,19 @@ def document_scores(
totals: dict[str, float] = {}
units: dict[str, int] = {}
def record(document: str, overlap: int) -> None:
def record(document: str, overlap: float) -> None:
totals[document] = totals.get(document, 0.0) + float(overlap)
units[document] = units.get(document, 0) + 1
for concept_id in concepts:
record(
concept_id.split("/", 1)[0],
_overlap(question_tokens, concept_id.replace("/", " "), cost_vocabulary=bridge),
_overlap(
question_tokens,
concept_id.replace("/", " "),
cost_vocabulary=bridge,
weights=weights,
),
)
for relative in indexes:
document = relative.split("/", 1)[0]
@ -713,7 +786,10 @@ def document_scores(
entry = profile.index.parse_entry(line)
if entry is None:
continue
record(document, _overlap(question_tokens, entry.label, cost_vocabulary=bridge))
record(
document,
_overlap(question_tokens, entry.label, cost_vocabulary=bridge, weights=weights),
)
return {document: totals[document] / units[document] for document in totals}
@ -737,6 +813,7 @@ def concept_scores(
document_score: Mapping[str, float],
*,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -760,20 +837,25 @@ def concept_scores(
"""
question_tokens = normalise(question)
bridge = cost_vocabulary and question_uses_cost_vocabulary(question)
titles = {
concept.concept_id: f"{concept.title} {concept.concept_id.replace('/', ' ')}"
for concept in concepts
}
signals: list[dict[str, float]] = [
{
concept.concept_id: float(
_overlap(
question_tokens,
f"{concept.title} {concept.concept_id.replace('/', ' ')}",
titles[concept.concept_id],
cost_vocabulary=bridge,
weights=weights,
)
)
for concept in concepts
},
{
concept.concept_id: float(
_overlap(question_tokens, concept.body, cost_vocabulary=bridge)
_overlap(question_tokens, concept.body, cost_vocabulary=bridge, weights=weights)
)
for concept in concepts
},
@ -789,10 +871,26 @@ def concept_scores(
order = sorted(signal, key=lambda key: (-signal[key], key))
for position, concept_id in enumerate(order, start=1):
fused[concept_id] += 1.0 / (RRF_K + position)
lexical = {
concept.concept_id: int(signals[0][concept.concept_id] + signals[1][concept.concept_id])
for concept in concepts
}
lexical = (
{
concept.concept_id: int(signals[0][concept.concept_id] + signals[1][concept.concept_id])
for concept in concepts
}
if weights is None
# A COUNT even when the signals are weighted. The cut reads this, and a
# word every concept carries weighs zero: were `lexical` the weighted
# sum, a concept matching only that word would fall to
# `no_lexical_match` -- turning a ranking change into the GATE
# `54a0bc2` falsified. The gate is a different axis and stays where it
# was, at the price of one more pass over the same two fields.
else {
concept.concept_id: int(
_overlap(question_tokens, titles[concept.concept_id], cost_vocabulary=bridge)
+ _overlap(question_tokens, concept.body, cost_vocabulary=bridge)
)
for concept in concepts
}
)
by_id = {concept.concept_id: concept for concept in concepts}
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
return [
@ -1003,12 +1101,13 @@ def build_payload(
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
reserve_top_rank: bool = False,
rarity_weight: bool = False,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank)` at
the same bytes returns the same object, every time.
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank,
rarity_weight)` at the same bytes returns the same object, every time.
"""
case, expected, measured = known_positive()
if expected != measured:
@ -1046,11 +1145,21 @@ def build_payload(
)
for concept_id in concept_ids
]
weights = (
rarity_weights(normalise(question), searchable_text(concepts)) if rarity_weight else None
)
ranked = concept_scores(
concepts,
question,
document_scores(bundle_root, question, profile=profile, cost_vocabulary=cost_vocabulary),
document_scores(
bundle_root,
question,
profile=profile,
cost_vocabulary=cost_vocabulary,
weights=weights,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
)
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
@ -1163,6 +1272,17 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"still refused"
),
)
parser.add_argument(
"--rarity-weight",
action="store_true",
help=(
"weight each lexical hit by log(N/df) over the bundle's own "
"concepts instead of counting it as one. OFF by default, and the "
"default is a MEASUREMENT rather than a preference: measured on "
"four corpora it moved one gold rank 9->8, left one at 35 and made "
"one 96->103 worse. See docs/2026-09-08-sjeldenhetsvekt.md"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument(
"--ref",
@ -1196,6 +1316,7 @@ def main(argv: list[str] | None = None) -> int:
limit=args.limit,
cost_vocabulary=args.cost_vocabulary,
reserve_top_rank=args.reserve_top_rank,
rarity_weight=args.rarity_weight,
)
except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr)