open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage

Browse source 
C1. `okf consume` and MCP's `okf_ask` now rank with BM25 (`bm25.py`) instead
of the three-signal fusion. Two signals, fused by reciprocal rank:

- passage: every body cut into 500-character windows every 250, a concept
  scored by its BEST window -- a narrow question is answered in one place;
- field: title three times, the id path and source name twice, then the body
  -- a broad question is answered by what a section is called.

The document prior and the rarity weight are gone from the default: the first
favoured big documents full of common words, the second gave its largest
weight to a word the collection does not hold. Under BM25 such a word weighs
exactly zero. A signal that scores a concept zero adds nothing to it, and ties
share a rank, so alphabetical order lifts nothing either.

Three rules carried over from the fusion, each with its own test, because the
suite showed what BM25 alone lost:
- a directory every concept shares is not read (K3-20's defect, one signal on);
- a number a section is known by (`4.2`, `10.2-2`) is kept as one token, or
  a question naming a section by its number matches nothing in it;
- a question word the collection does NOT hold is read as the collection's
  words it shares a leading word with (`consume.tokens_match`) -- Norwegian
  inflection and compounding -- at that word's idf, never at its own.

The lookup and title-covered partitions are shared with the fusion
(`_partitioned`). `ranking="fusion"` / `--ranking fusion` keeps the old order
reachable; `--cost-vocabulary` and `--rarity-weight` widen only the fusion and
are refused with the default (`ranking_flag_conflict`) rather than ignored.

C3. A concept longer than `PASSAGE_CHARS` (4 000) is delivered as the span
around its best window, snapped to whole lines, under the nearest heading
above it, with `[...]` where text was left out. `passage: {start, end, of}`
says so, `text_sha256` covers what was delivered, and `sha256` stays the
file's, so the whole can be fetched by `concept_id`. 4 000 because eight
excerpts of it stay far under a tool response's limit even with several
sub-questions merged, while a 500-character window keeps 3 500 characters of
surroundings. The budget pays for the passage, not the file.

Tests moved with the default, each stated rather than silenced:
- fusion-mechanism tests (cost vocabulary, rarity weight, reservation, shared
  rank, the reference-bundle pins) ask for `ranking="fusion"`, the order they
  were measured on; the BM25 reading of the reference bundle is a separate
  measurement, kept in local state;
- the retrieval gate still measures the shipped default. Row 1 holds. Four of
  its premises were built against the fusion (a concept forced below k that
  BM25 now delivers, a quota that no longer decides, mutants patching fusion
  code) and are `xfail(strict=True)` until the fixtures are re-measured;
- the shipped example payload is regenerated; the shipped skill is unchanged.

README's Consume section and CLAUDE.md state the new default and that the
flags described after it belong to the fusion.

The search gate's table for this commit is kept in local state: the question
sets belong to a consumer whose content does not go on a public mirror.

Suite on a clean tree after `git add`: 2390 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 06:22:56 +02:00
commit 735468f600
11 changed files with 988 additions and 118 deletions
Download patch file Download diff file Expand all files Collapse all files

329
src/llm_ingestion_okf/bm25.py Normal file
View file

@ -0,0 +1,329 @@
"""BM25 over a bundle's concepts: the default ranking of `okf consume` (v1.1 C1).
WHY IT REPLACED THE FUSION. The earlier ranking fused three signals -- token
overlap with the title, token overlap with the body, and a document prior --
and two of them rewarded the wrong thing on a large collection: the document
prior favoured big documents full of common words, and the rarity weight gave
its MAXIMUM weight to a word that occurs nowhere in the collection. BM25 has
neither property, needs no new dependency, and ranks in milliseconds. The
measurement that chose it lives with the measurement; this module states the
mechanism.
TWO SIGNALS, FUSED BY RANK.
- **Passage** -- every body is cut into windows of `WINDOW_CHARS` characters
every `WINDOW_STEP`, each window is a BM25 document, and a concept scores its
BEST window (pure max). A narrow factual question is answered by one place
in a concept, and a sum over windows -- even a damped one -- rewards a long
concept for mentioning a word often.
- **Field** -- one BM25 document per concept: its title three times, its
directory path and source file name twice, then its body. A broad question
is answered by what a section is CALLED, and a passage never sees the title.
Fused by reciprocal rank (`RRF_K`, the constant the old fusion used). A signal
that scored a concept zero contributes nothing to it, and concepts that tie
within a signal share the group's first rank -- so neither a word the
collection lacks nor alphabetical order can lift a concept.
**A word the collection does not hold weighs exactly zero** (`idf` of a term
with `df == 0`), which is the property the old rarity weight had backwards.
Deterministic: every sort breaks ties by `concept_id`, and no float leaves this
module except as an ordering key.
"""
from __future__ import annotations
import functools
import math
import re
import unicodedata
from collections import Counter
from collections.abc import Sequence
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from llm_ingestion_okf.consume import Concept
#: BM25's two parameters, at the textbook values. Not tuned: the measurement
#: that chose this ranking used them as they are.
K1 = 1.2
B = 0.75
#: The passage window, in characters, and the step between window starts --
#: half a window, so every sentence sits whole inside at least one window.
WINDOW_CHARS = 500
WINDOW_STEP = 250
#: How many times the field signal repeats a concept's title, and its path.
TITLE_WEIGHT = 3
PATH_WEIGHT = 2
#: The rank-fusion constant, the same one the older fusion used.
RRF_K = 60
_TOKEN = re.compile(r"[0-9a-zà-öø-ÿ]+(?:[-_][0-9a-zà-öø-ÿ]+)*")
#: English and Norwegian function words. Norwegian because an operator asks in
#: Norwegian against a collection that may be English; a Norwegian function
#: word is then noise that could only ever match by accident.
STOPWORDS = frozenset(
"""a an and are as at be but by for if in into is it its of on or such that the
their then there these they this to was will with what which how when where who why
do does did can could should would i you your my me we our us not no yes from over
under about after before more most other some only own same so than too very s t just
don now am been being have has had having he she him her his hers them up out off
again further once here all any both each few nor
og i jeg det at en et den til er som på de med han av ikke der så var meg seg
men ett har om vi min mitt ha hadde hun nå over da ved fra du ut sin dem oss opp man
kan hans hvor eller hva skal selv her alle vil bli ble blitt kunne inn når være kom
noen noe ville dere hvordan gjør gjøre lar la flere""".split()
)
_SUFFIXES = ("ingly", "edly", "ing", "ies", "ied", "ed", "es", "s", "ly", "er", "est")
@functools.lru_cache(maxsize=None)
def _stem(token: str) -> str:
"""A light English suffix strip, applied alike to question and text.
Memoised because it is pure and a collection repeats its words: every
question re-tokenises every window, so the same words are stemmed again
and again.
"""
if len(token) <= 3:
return token
for suffix in _SUFFIXES:
if token.endswith(suffix) and len(token) - len(suffix) >= 3:
return token[: -len(suffix)]
return token
#: A number a document is known by -- `4.2`, `10.2-2`, `2.1.219` -- kept as ONE
#: token beside its pieces. Split on the dot, `4.2` is two single characters,
#: which are dropped, so a question naming a section by its number would match
#: nothing in the section it names. The shape is `consume`'s identifier rule.
_IDENTIFIER = re.compile(r"[0-9a-zà-öø-ÿ]*[0-9]+(?:[.-][0-9]+)+")
_DASH_TO_HYPHEN = str.maketrans(dict.fromkeys("‐‑‒–—―−", "-"))
def tokens(text: str) -> list[str]:
"""Casefolded, NFC, stopwords and single characters out, lightly stemmed,
plus every identifier whole and unstemmed."""
folded = unicodedata.normalize("NFC", text).casefold().translate(_DASH_TO_HYPHEN)
words = [
_stem(token)
for token in _TOKEN.findall(folded)
if len(token) > 1 and token not in STOPWORDS
]
return words + [token for token in _IDENTIFIER.findall(folded) if token not in words]
class Index:
"""Okapi BM25 over a list of token lists."""
def __init__(self, documents: Sequence[Sequence[str]]) -> None:
self.size = len(documents)
self.lengths = [len(document) for document in documents]
self.average = sum(self.lengths) / self.size if self.size else 0.0
self.postings: dict[str, list[tuple[int, int]]] = {}
for position, document in enumerate(documents):
for term, frequency in sorted(Counter(document).items()):
self.postings.setdefault(term, []).append((position, frequency))
def idf(self, terms: frozenset[str]) -> float:
"""Zero for terms no document holds: absence lifts nothing."""
df = len(self.holders(terms))
if df == 0:
return 0.0
return math.log(1 + (self.size - df + 0.5) / (df + 0.5))
def holders(self, terms: frozenset[str]) -> set[int]:
"""Every document position holding at least one of `terms`."""
return {position for term in terms for position, _ in self.postings.get(term, ())}
def scores(self, query: Sequence[frozenset[str]]) -> dict[int, float]:
"""Positive scores only, keyed by document position.
Each query element is a GROUP of terms read as one: its frequency in a
document is the sum over the group, its `df` the documents holding any.
A group of one is plain BM25.
"""
out: dict[int, float] = {}
for group in query:
weight = self.idf(group)
if weight == 0.0:
continue
frequencies: Counter[int] = Counter()
for term in sorted(group):
for position, frequency in self.postings.get(term, ()):
frequencies[position] += frequency
for position, frequency in sorted(frequencies.items()):
norm = K1 * (1 - B + B * self.lengths[position] / self.average)
out[position] = out.get(position, 0.0) + weight * (
frequency * (K1 + 1) / (frequency + norm)
)
return out
def windows(body: str) -> list[tuple[int, str]]:
"""`(start, text)` for every window of a body; one empty-start window if blank."""
if not body.strip():
return []
out = []
for start in range(0, len(body), WINDOW_STEP):
chunk = body[start : start + WINDOW_CHARS]
if chunk.strip():
out.append((start, chunk))
if start + WINDOW_CHARS >= len(body):
break
return out
def field_text(concept: Concept, body: str, *, shared: int = 0, own_source: bool = True) -> str:
"""Title and path weighted up, then the body: the field signal's document.
`shared` is how many leading id segments EVERY concept carries, and
`own_source` whether the source file name separates this concept from any
other. What every concept carries separates nothing, and a question naming
it would match them all -- the defect `consume.shared_id_prefix` closed for
the older ranking, and the reason both are dropped here.
"""
path = " ".join(concept.concept_id.split("/")[shared:]).replace("-", " ")
source = concept.source_file.removesuffix(".md").replace("-", " ") if own_source else ""
return f"{concept.title} " * TITLE_WEIGHT + f"{path} {source} " * PATH_WEIGHT + body
def _shared_segments(concept_ids: Sequence[str]) -> int:
"""How many leading DIRECTORY segments every id shares (never the leaf)."""
if not concept_ids:
return 0
split = [concept_id.split("/")[:-1] for concept_id in concept_ids]
count = 0
for segments in zip(*split):
if len(set(segments)) != 1:
break
count += 1
return count
def query_groups(query: Sequence[str], vocabulary: frozenset[str]) -> list[frozenset[str]]:
"""Each question term as the group of collection terms it is read as.
A term the collection holds is read as itself and nothing else. A term it
does NOT hold -- an inflection, a compound, a word in another language --
is read as the collection's words it shares a leading WORD with, by
`consume.tokens_match`, the rule the older ranking measured for Norwegian
inflection and compounding. So an absent word lifts nothing by itself: it
reaches the collection only through a relative the collection uses, at that
relative's `idf` and never at a weight of its own.
"""
from llm_ingestion_okf.consume import MIN_SHARED_PREFIX, tokens_match
by_prefix: dict[str, list[str]] = {}
for term in sorted(vocabulary):
by_prefix.setdefault(term[:MIN_SHARED_PREFIX], []).append(term)
groups = []
for term in dict.fromkeys(query):
if term in vocabulary:
groups.append(frozenset({term}))
continue
groups.append(
frozenset(
candidate
for candidate in by_prefix.get(term[:MIN_SHARED_PREFIX], ())
if tokens_match(term, candidate, stems=vocabulary)
)
)
return groups
@dataclass(frozen=True)
class Ranking:
"""Every concept best first, as `(concept, fused, lexical)`, plus where
each concept's best window starts in the body that was searched."""
ranked: list[tuple[Concept, float, int]]
best_window: dict[str, int]
def _fuse(fused: dict[str, float], scores: dict[str, float]) -> None:
"""Add one signal's reciprocal ranks; equal scores share the first rank."""
order = sorted(scores, key=lambda key: (-scores[key], key))
start = 0
while start < len(order):
stop = start
while stop < len(order) and scores[order[stop]] == scores[order[start]]:
stop += 1
contribution = 1.0 / (RRF_K + start + 1)
for concept_id in order[start:stop]:
fused[concept_id] += contribution
start = stop
def rank(
concepts: Sequence[Concept],
question: str,
*,
bodies: Sequence[str] | None = None,
) -> Ranking:
"""Rank `concepts` for `question`.
`bodies` is the text searched per concept (defaults to each `body`); the
caller passes the body without the door's link line, so what is searched
is what the older ranking searched.
"""
texts = list(bodies) if bodies is not None else [concept.body for concept in concepts]
query = tokens(question)
shared = _shared_segments([concept.concept_id for concept in concepts])
own_source = len({concept.source_file for concept in concepts}) > 1
field_documents = [
tokens(field_text(concept, text, shared=shared, own_source=own_source))
for concept, text in zip(concepts, texts, strict=True)
]
field_index = Index(field_documents)
vocabulary = frozenset(field_index.postings)
groups = query_groups(query, vocabulary)
field = {
concepts[position].concept_id: score
for position, score in field_index.scores(groups).items()
}
owners: list[int] = []
starts: list[int] = []
passages: list[list[str]] = []
for position, text in enumerate(texts):
cut = windows(text) or [(0, concepts[position].title)]
for start, chunk in cut:
owners.append(position)
starts.append(start)
passages.append(tokens(chunk))
passage_index = Index(passages)
passage: dict[str, float] = {}
best_window: dict[str, int] = {}
for window, score in sorted(passage_index.scores(groups).items()):
concept_id = concepts[owners[window]].concept_id
if score > passage.get(concept_id, 0.0):
passage[concept_id] = score
best_window[concept_id] = starts[window]
fused = {concept.concept_id: 0.0 for concept in concepts}
_fuse(fused, passage)
_fuse(fused, field)
asked = [group for group in groups if field_index.idf(group) > 0.0]
lexical = {
concept.concept_id: sum(1 for group in asked if group & held)
for concept, held in (
(concept, set(document))
for concept, document in zip(concepts, field_documents, strict=True)
)
}
by_id = {concept.concept_id: concept for concept in concepts}
order = sorted(fused, key=lambda key: (-fused[key], key))
return Ranking(
ranked=[(by_id[key], fused[key], lexical[key]) for key in order],
best_window=best_window,
)

296
src/llm_ingestion_okf/consume.py
View file

@ -54,6 +54,7 @@ from dataclasses import dataclass, replace
from pathlib import Path
from typing import Literal
from . import bm25
from .connectors import safe_resolve
from .corpus import LOG_NAME
from .errors import SourceError
@ -1476,6 +1477,76 @@ def shared_id_prefix(concept_ids: Sequence[str]) -> int:
return shared
def _partitioned(
ranked_ids: list[str],
concepts: Sequence[Concept],
question: str,
*,
title_covered: bool,
lookup: bool,
) -> list[str]:
"""The two partitions every ranking lands under: a concept whose title the
question covers rises, and a concept the question NAMES reads first.
Shared by the fusion and by BM25 so the two readings differ in how they
SCORE and in nothing else.
"""
question_tokens = normalise(question)
by_id = {concept.concept_id: concept for concept in concepts}
covered = set(title_covered_hits(concepts, question)) if title_covered else set()
if covered:
# A PARTITION, not a signal, and it lands BELOW the lookup partition
# so a question that NAMES a concept still reads that one first. See
# `DEFAULT_TITLE_COVERED` for the arithmetic that rules a signal out,
# and `tests/test_title_covered.py` for the mechanism on a fixture.
#
# BOUNDED BY RECALL since round 17: a covered concept RISES through the
# fusion's order and stops beneath the first concept whose title
# answers MORE question tokens, by equality, than the covered title
# holds -- or beneath a covered concept the fusion put above it. With
# nothing above it answering more, it reaches the top exactly where
# round 16's plain partition put it. `tests/test_title_covered_rise.py`
# holds the known-negative that bound exists for.
#
# STABLE: the covered concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
asked = frozenset(question_tokens)
answered = {
concept_id: len(set(normalise(by_id[concept_id].title)) & asked)
for concept_id in ranked_ids
}
risen: list[str] = []
for concept_id in ranked_ids:
stop = len(risen)
if concept_id in covered:
while (
stop
and risen[stop - 1] not in covered
and answered[risen[stop - 1]] <= answered[concept_id]
):
stop -= 1
risen.insert(stop, concept_id)
ranked_ids = risen
named = set(lookup_hits(concepts, question)) if lookup else set()
if named:
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
# partition rather than a fourth signal. The form was chosen by
# measurement, not by preference: a fourth RRF signal was simulated on
# the same three bundles first and put the named concept at rank
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
# consumes RANKS ONLY, so any single signal contributes at most
# `1/(RRF_K + 1)` however certain it is, and a concept the question
# NAMES cannot outbid three signals that merely describe it
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
#
# STABLE: the named concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
ranked_ids = [key for key in ranked_ids if key in named] + [
key for key in ranked_ids if key not in named
]
return ranked_ids
def concept_scores(
concepts: Sequence[Concept],
question: str,
@ -1648,58 +1719,13 @@ def concept_scores(
}
)
by_id = {concept.concept_id: concept for concept in concepts}
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
covered = set(title_covered_hits(concepts, question)) if title_covered else set()
if covered:
# A PARTITION, not a signal, and it lands BELOW the lookup partition
# so a question that NAMES a concept still reads that one first. See
# `DEFAULT_TITLE_COVERED` for the arithmetic that rules a signal out,
# and `tests/test_title_covered.py` for the mechanism on a fixture.
#
# BOUNDED BY RECALL since round 17: a covered concept RISES through the
# fusion's order and stops beneath the first concept whose title
# answers MORE question tokens, by equality, than the covered title
# holds -- or beneath a covered concept the fusion put above it. With
# nothing above it answering more, it reaches the top exactly where
# round 16's plain partition put it. `tests/test_title_covered_rise.py`
# holds the known-negative that bound exists for.
#
# STABLE: the covered concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
asked = frozenset(question_tokens)
answered = {
concept_id: len(set(normalise(by_id[concept_id].title)) & asked)
for concept_id in ranked_ids
}
risen: list[str] = []
for concept_id in ranked_ids:
stop = len(risen)
if concept_id in covered:
while (
stop
and risen[stop - 1] not in covered
and answered[risen[stop - 1]] <= answered[concept_id]
):
stop -= 1
risen.insert(stop, concept_id)
ranked_ids = risen
named = set(lookup_hits(concepts, question)) if lookup else set()
if named:
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
# partition rather than a fourth signal. The form was chosen by
# measurement, not by preference: a fourth RRF signal was simulated on
# the same three bundles first and put the named concept at rank
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
# consumes RANKS ONLY, so any single signal contributes at most
# `1/(RRF_K + 1)` however certain it is, and a concept the question
# NAMES cannot outbid three signals that merely describe it
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
#
# STABLE: the named concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
ranked_ids = [key for key in ranked_ids if key in named] + [
key for key in ranked_ids if key not in named
]
ranked_ids = _partitioned(
sorted(fused, key=lambda key: (-fused[key], key)),
concepts,
question,
title_covered=title_covered,
lookup=lookup,
)
return [
(by_id[concept_id], fused[concept_id], lexical[concept_id]) for concept_id in ranked_ids
]
@ -1820,6 +1846,75 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
return excerpt
#: The most characters of a concept's body one excerpt carries (v1.1 C3).
#: Chosen as 4 000 because the default `k` of 8 excerpts then carries at most
#: 32 000 characters of text -- an answer that stays well under a tool
#: response's 25 000-token limit even when four sub-questions are merged into
#: it -- while a 500-character answering window keeps 3 500 characters of
#: surroundings, enough for the place to read alone. A concept at or under it
#: is delivered whole, byte for byte as before.
PASSAGE_CHARS = 4_000
#: Room for the heading line a passage is prefixed with and the two markers
#: that say text was cut before or after it. A heading longer than this is cut.
PASSAGE_HEADING_ALLOWANCE = 240
#: Written where a passage leaves text out, at the start or the end.
PASSAGE_ELISION = "[...]"
_HEADING_LINE = re.compile(r"^#{1,6} \S.*$", re.MULTILINE)
def as_passage(excerpt: dict[str, object], window: int) -> dict[str, object]:
"""`excerpt` with its `text` cut to the span around `window`, when too long.
The span is `PASSAGE_CHARS` wide, centred on the answering window the
ranking found, and snapped inward to whole lines. The nearest heading above
the span is carried as its first line when the span does not already hold
it, so a table row or a paragraph is read under the section it belongs to.
`passage` records `{start, end, of}` in characters of the WHOLE delivered
text, and `text_sha256` is recomputed over what is delivered -- `sha256`
stays the concept file's, so a reader can tell the excerpt from the whole
and fetch the whole by `concept_id`.
"""
text = excerpt["text"]
assert isinstance(text, str)
if len(text) <= PASSAGE_CHARS:
return excerpt
centre = min(max(window, 0), len(text)) + 250
start = max(0, min(centre - PASSAGE_CHARS // 2, len(text) - PASSAGE_CHARS))
end = min(len(text), start + PASSAGE_CHARS)
if start > 0:
# Inward to the next line start, so no line arrives cut in half.
newline = text.find("\n", start, end)
start = newline + 1 if newline != -1 else start
if end < len(text):
newline = text.rfind("\n", start, end)
end = newline if newline > start else end
span = text[start:end]
heading = ""
if start > 0:
above = [match.group(0) for match in _HEADING_LINE.finditer(text, 0, start)]
if above and above[-1] not in span.split("\n", 1)[0]:
heading = above[-1][: PASSAGE_HEADING_ALLOWANCE - 2 * len(PASSAGE_ELISION) - 4]
parts = []
if heading:
parts.append(heading)
if start > 0:
parts.append(PASSAGE_ELISION)
parts.append(span)
if end < len(text):
parts.append(PASSAGE_ELISION)
delivered = "\n".join(parts)
out = dict(excerpt)
out["passage"] = {"start": start, "end": end, "of": len(text)}
out["text_sha256"] = hashlib.sha256(delivered.encode("utf-8")).hexdigest()
# `text` stays the LAST key, as `excerpt_for` writes it.
del out["text"]
out["text"] = delivered
return out
def excerpt_weight(excerpt: Mapping[str, object]) -> int:
"""What this excerpt costs by the gate's own instrument.
@ -2010,6 +2105,7 @@ def cut(
limit: int,
reserve_top_rank: bool = False,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
windows: Mapping[str, int] | None = None,
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...], tuple[str, int] | None]:
"""The ranked concepts split into delivered excerpts, named drops, and the
reservation that was made, if any.
@ -2054,6 +2150,9 @@ def cut(
if excerpt is None:
withheld.append((concept.concept_id, "verified_unreadable"))
continue
if windows is not None:
# BEFORE the weight is read: the budget pays for what is delivered.
excerpt = as_passage(excerpt, windows.get(concept.concept_id, 0))
weight = excerpt_weight(excerpt)
if weight > limit:
withheld.append((concept.concept_id, "over_budget_alone"))
@ -2152,6 +2251,14 @@ WITHHELD_NEAREST_DEFAULT = 20
#: 300-item DP to discover the same answer.
DEFAULT_K = 8
#: The two readings `build_payload` can rank with. `bm25` is the default since
#: v1.1 (see `llm_ingestion_okf.bm25` for the mechanism and why it replaced the
#: fusion); `fusion` is the older three-signal RRF, kept reachable because its
#: mechanisms are pinned by tests of their own and a caller may need the old
#: order. The two share the lookup and title partitions and the cut.
RANKINGS = ("bm25", "fusion")
DEFAULT_RANKING = "bm25"
def root_bundle_id_of(bundle_root: Path, *, profile: BundleProfile = DEFAULT_PROFILE) -> str:
"""The `bundle_id` the root index declares, or a refusal naming which half
@ -2299,6 +2406,7 @@ def build_payload(
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
follow_parent: bool = DEFAULT_FOLLOW_PARENT,
link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL,
ranking: str = DEFAULT_RANKING,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -2373,25 +2481,66 @@ def build_payload(
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters.
stems = frozenset(token for tokens in tokenised for token in tokens) if stem_prefix else None
weights = rarity_weights(normalise(question), texts, stems=stems) if rarity_weight else None
ranked = concept_scores(
concepts,
question,
document_scores(
bundle_root,
if ranking not in RANKINGS:
raise ConsumeError(
f"unknown ranking {ranking!r}; expected one of {', '.join(RANKINGS)}",
code="ranking_invalid",
)
if ranking == "bm25" and (cost_vocabulary or rarity_weight):
# Both widen a signal only the fusion has. Accepting them here would be
# a flag that silently does nothing, which reads as a measurement.
raise ConsumeError(
"--cost-vocabulary and --rarity-weight are widenings of the fusion "
"ranking; ask for it with ranking='fusion'",
code="ranking_flag_conflict",
)
if ranking == "bm25":
result = bm25.rank(
concepts,
question,
profile=profile,
# The DELIVERED form of each body, so a window's offset points at
# the same characters `as_passage` cuts from.
bodies=[
delivered_text(
concept.body if link_in_signal else body_without_link_line(concept.body)
)
for concept in concepts
],
)
by_id = {concept.concept_id: concept for concept in concepts}
scored = {concept.concept_id: (score, lexical) for concept, score, lexical in result.ranked}
windows: Mapping[str, int] | None = result.best_window
ranked = [
(by_id[concept_id], *scored[concept_id])
for concept_id in _partitioned(
[concept.concept_id for concept, _, _ in result.ranked],
concepts,
question,
title_covered=title_covered,
lookup=True,
)
]
else:
windows = None
weights = rarity_weights(normalise(question), texts, stems=stems) if rarity_weight else None
ranked = concept_scores(
concepts,
question,
document_scores(
bundle_root,
question,
profile=profile,
cost_vocabulary=cost_vocabulary,
weights=weights,
stems=stems,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
tie_shared_rank=tie_shared_rank,
title_covered=title_covered,
stems=stems,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
tie_shared_rank=tie_shared_rank,
title_covered=title_covered,
stems=stems,
link_in_signal=link_in_signal,
)
link_in_signal=link_in_signal,
)
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(
@ -2400,6 +2549,7 @@ def build_payload(
limit=limit,
reserve_top_rank=reserve_top_rank,
source_quota=source_quota,
windows=windows,
)
if follow_parent:
# After the cut and never inside it: see `attach_parent_text`.
@ -2669,6 +2819,17 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"budget the payload reports"
),
)
parser.add_argument(
"--ranking",
choices=RANKINGS,
default=DEFAULT_RANKING,
help=(
f"how concepts are ordered before the cut. Default {DEFAULT_RANKING}: "
"passage and title BM25 fused by rank, a large concept delivered as "
"its answering passage. `fusion` is the older three-signal ranking; "
"--cost-vocabulary and --rarity-weight widen it and need it"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument(
"--ref",
@ -2710,6 +2871,7 @@ def main(argv: list[str] | None = None) -> int:
withheld_nearest=args.withheld_nearest,
withheld_full=args.withheld_full,
follow_parent=args.follow_parent,
ranking=args.ranking,
)
except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr)