open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage

Browse source 
C1. `okf consume` and MCP's `okf_ask` now rank with BM25 (`bm25.py`) instead
of the three-signal fusion. Two signals, fused by reciprocal rank:

- passage: every body cut into 500-character windows every 250, a concept
  scored by its BEST window -- a narrow question is answered in one place;
- field: title three times, the id path and source name twice, then the body
  -- a broad question is answered by what a section is called.

The document prior and the rarity weight are gone from the default: the first
favoured big documents full of common words, the second gave its largest
weight to a word the collection does not hold. Under BM25 such a word weighs
exactly zero. A signal that scores a concept zero adds nothing to it, and ties
share a rank, so alphabetical order lifts nothing either.

Three rules carried over from the fusion, each with its own test, because the
suite showed what BM25 alone lost:
- a directory every concept shares is not read (K3-20's defect, one signal on);
- a number a section is known by (`4.2`, `10.2-2`) is kept as one token, or
  a question naming a section by its number matches nothing in it;
- a question word the collection does NOT hold is read as the collection's
  words it shares a leading word with (`consume.tokens_match`) -- Norwegian
  inflection and compounding -- at that word's idf, never at its own.

The lookup and title-covered partitions are shared with the fusion
(`_partitioned`). `ranking="fusion"` / `--ranking fusion` keeps the old order
reachable; `--cost-vocabulary` and `--rarity-weight` widen only the fusion and
are refused with the default (`ranking_flag_conflict`) rather than ignored.

C3. A concept longer than `PASSAGE_CHARS` (4 000) is delivered as the span
around its best window, snapped to whole lines, under the nearest heading
above it, with `[...]` where text was left out. `passage: {start, end, of}`
says so, `text_sha256` covers what was delivered, and `sha256` stays the
file's, so the whole can be fetched by `concept_id`. 4 000 because eight
excerpts of it stay far under a tool response's limit even with several
sub-questions merged, while a 500-character window keeps 3 500 characters of
surroundings. The budget pays for the passage, not the file.

Tests moved with the default, each stated rather than silenced:
- fusion-mechanism tests (cost vocabulary, rarity weight, reservation, shared
  rank, the reference-bundle pins) ask for `ranking="fusion"`, the order they
  were measured on; the BM25 reading of the reference bundle is a separate
  measurement, kept in local state;
- the retrieval gate still measures the shipped default. Row 1 holds. Four of
  its premises were built against the fusion (a concept forced below k that
  BM25 now delivers, a quota that no longer decides, mutants patching fusion
  code) and are `xfail(strict=True)` until the fixtures are re-measured;
- the shipped example payload is regenerated; the shipped skill is unchanged.

README's Consume section and CLAUDE.md state the new default and that the
flags described after it belong to the fusion.

The search gate's table for this commit is kept in local state: the question
sets belong to a consumer whose content does not go on a public mirror.

Suite on a clean tree after `git add`: 2390 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-21 06:22:56 +02:00
commit 735468f600
11 changed files with 988 additions and 118 deletions
Download patch file Download diff file Expand all files Collapse all files

13
CLAUDE.md
View file

@ -1530,6 +1530,19 @@ R761 **8** (S1-S6 + KP + KN), vegnormal **32** questions / **43**
one-to-many **0 / 0**, today's per-bundle skill **1 / 1 per consuming
project** (it refuses out loud through `bundle_mismatch`, so its cost is not
silence), generic skill **0 / 0**.
- **THE DEFAULT RANKING IS BM25 SINCE v1.1 ORDER C (C1+C3), 2026-09-21**
(`bm25.py`, `consume.DEFAULT_RANKING`): the best 500-character passage and a
title/path-weighted field, fused by rank; the document prior and the rarity
weight are out of the default. Three rules came over from the fusion because
the suite showed what BM25 alone lost: a directory every id shares is not
read, an identifier (`4.2`) stays one token, and a question word the bundle
lacks is read as the bundle's words it shares a leading WORD with
(`tokens_match`). A concept over `PASSAGE_CHARS` (4 000) is delivered as its
answering passage (`as_passage`, `passage: {start, end, of}`).
`--ranking fusion` keeps everything below reachable; the fusion-mechanism
tests and the reference-bundle pins ask for it by name. The retrieval gate
still measures the DEFAULT, so four of its fusion-built premises are
`xfail(strict=True)` until re-measured. What follows describes the fusion.
- Consume a bundle: `okf consume <bundle> --question "<q>"
[--k N] [--limit N] [--out PATH] [--ref IDENTITY]` — the **pre-pass**
`docs/consumption-contract.md` § 1 defines, and the only reading direction

11
README.md
View file

@ -816,6 +816,17 @@ package, and takes no clock: the same bundle bytes and the same
`(question, k, limit, cost_vocabulary, reserve_top_rank, rarity_weight)`
produce byte-identical output.
**The ranking is BM25 since v1.1** (`--ranking bm25`, the default). Two signals
are fused by rank: each concept's best 500-character passage, and its title and
id path weighted above its body. A word the bundle does not hold weighs nothing
by itself; one it holds in another form — a Norwegian inflection or compound —
is read as that form. A concept longer than 4 000 characters is delivered as
the passage that answers, under the nearest heading above it, marked with
`passage: {start, end, of}`, so the whole can be fetched by its `concept_id`.
`--ranking fusion` is the older three-signal ranking; the flags below that say
they widen a signal (`--cost-vocabulary`, `--rarity-weight`) belong to it and
are refused without it. The rest of this section describes the fusion.
`--cost-vocabulary` is off by default and widens one question class: it lets a
declared list of cost/price/quantity terms bridge a question and a document that
name money with different words. The gate is the question — one naming no such

46
skills/okf-consume/references/example-payload.json
View file

@ -40,6 +40,28 @@
]
},
"excerpts": [
{
"bundle_id": "b-golden-segmented-okf-v0-2",
"concept_id": "veiledning",
"sha256": "8f1f4d940887b96b39496d0aceb7a4fa212553510165917427e4f7e9efb98b03",
"adjudication": "proposed",
"trust_tier": "unverified",
"bundle_id_inherited": false,
"title": "Veiledning for eksempelbundel",
"sources": [
{
"resource": "veiledning.md",
"title": "veiledning.md"
}
],
"source_file": "veiledning.md",
"source_sha256": "6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d",
"source_lines": "[1, 4]",
"source_offset": "[0, 94]",
"text_sha256": "82d67ff01367a1a56532130e077956048004ac047e6062eae2143d96704c4a92",
"text": "\n# Veiledning for eksempelbundel\n\nDenne teksten er syntetisk og finnes kun for a pinne bytes.",
"rank": 1
},
{
"bundle_id": "b-golden-segmented-okf-v0-2",
"concept_id": "krav/1-1/foerste-krav",
@ -64,7 +86,7 @@
},
"text_sha256": "08447c690087834883a78fb9f40d474c68b2526ed57d0dafe01c4868db3c04f0",
"text": "\n## 1.1 Foerste krav\n\nEt krav som beskriver den foerste betingelsen i eksempelet.",
"rank": 1
"rank": 2
},
{
"bundle_id": "b-golden-segmented-okf-v0-2",
@ -90,28 +112,6 @@
},
"text_sha256": "4c181e1debb141d3cb1a7bd47249fdac1c0b2c5db1b47cf255ef8f04f83ec840",
"text": "\n## 1.2 Andre krav\n\nEt krav som beskriver den andre betingelsen i eksempelet.",
"rank": 2
},
{
"bundle_id": "b-golden-segmented-okf-v0-2",
"concept_id": "veiledning",
"sha256": "8f1f4d940887b96b39496d0aceb7a4fa212553510165917427e4f7e9efb98b03",
"adjudication": "proposed",
"trust_tier": "unverified",
"bundle_id_inherited": false,
"title": "Veiledning for eksempelbundel",
"sources": [
{
"resource": "veiledning.md",
"title": "veiledning.md"
}
],
"source_file": "veiledning.md",
"source_sha256": "6906ec0acbcfc246e825bda9863c716eb5611b465020e8204eeb448c32343f7d",
"source_lines": "[1, 4]",
"source_offset": "[0, 94]",
"text_sha256": "82d67ff01367a1a56532130e077956048004ac047e6062eae2143d96704c4a92",
"text": "\n# Veiledning for eksempelbundel\n\nDenne teksten er syntetisk og finnes kun for a pinne bytes.",
"rank": 3
}
],

329
src/llm_ingestion_okf/bm25.py Normal file
View file

@ -0,0 +1,329 @@
"""BM25 over a bundle's concepts: the default ranking of `okf consume` (v1.1 C1).
WHY IT REPLACED THE FUSION. The earlier ranking fused three signals -- token
overlap with the title, token overlap with the body, and a document prior --
and two of them rewarded the wrong thing on a large collection: the document
prior favoured big documents full of common words, and the rarity weight gave
its MAXIMUM weight to a word that occurs nowhere in the collection. BM25 has
neither property, needs no new dependency, and ranks in milliseconds. The
measurement that chose it lives with the measurement; this module states the
mechanism.
TWO SIGNALS, FUSED BY RANK.
- **Passage** -- every body is cut into windows of `WINDOW_CHARS` characters
every `WINDOW_STEP`, each window is a BM25 document, and a concept scores its
BEST window (pure max). A narrow factual question is answered by one place
in a concept, and a sum over windows -- even a damped one -- rewards a long
concept for mentioning a word often.
- **Field** -- one BM25 document per concept: its title three times, its
directory path and source file name twice, then its body. A broad question
is answered by what a section is CALLED, and a passage never sees the title.
Fused by reciprocal rank (`RRF_K`, the constant the old fusion used). A signal
that scored a concept zero contributes nothing to it, and concepts that tie
within a signal share the group's first rank -- so neither a word the
collection lacks nor alphabetical order can lift a concept.
**A word the collection does not hold weighs exactly zero** (`idf` of a term
with `df == 0`), which is the property the old rarity weight had backwards.
Deterministic: every sort breaks ties by `concept_id`, and no float leaves this
module except as an ordering key.
"""
from __future__ import annotations
import functools
import math
import re
import unicodedata
from collections import Counter
from collections.abc import Sequence
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from llm_ingestion_okf.consume import Concept
#: BM25's two parameters, at the textbook values. Not tuned: the measurement
#: that chose this ranking used them as they are.
K1 = 1.2
B = 0.75
#: The passage window, in characters, and the step between window starts --
#: half a window, so every sentence sits whole inside at least one window.
WINDOW_CHARS = 500
WINDOW_STEP = 250
#: How many times the field signal repeats a concept's title, and its path.
TITLE_WEIGHT = 3
PATH_WEIGHT = 2
#: The rank-fusion constant, the same one the older fusion used.
RRF_K = 60
_TOKEN = re.compile(r"[0-9a-zà-öø-ÿ]+(?:[-_][0-9a-zà-öø-ÿ]+)*")
#: English and Norwegian function words. Norwegian because an operator asks in
#: Norwegian against a collection that may be English; a Norwegian function
#: word is then noise that could only ever match by accident.
STOPWORDS = frozenset(
"""a an and are as at be but by for if in into is it its of on or such that the
their then there these they this to was will with what which how when where who why
do does did can could should would i you your my me we our us not no yes from over
under about after before more most other some only own same so than too very s t just
don now am been being have has had having he she him her his hers them up out off
again further once here all any both each few nor
og i jeg det at en et den til er som på de med han av ikke der så var meg seg
men ett har om vi min mitt ha hadde hun nå over da ved fra du ut sin dem oss opp man
kan hans hvor eller hva skal selv her alle vil bli ble blitt kunne inn når være kom
noen noe ville dere hvordan gjør gjøre lar la flere""".split()
)
_SUFFIXES = ("ingly", "edly", "ing", "ies", "ied", "ed", "es", "s", "ly", "er", "est")
@functools.lru_cache(maxsize=None)
def _stem(token: str) -> str:
"""A light English suffix strip, applied alike to question and text.
Memoised because it is pure and a collection repeats its words: every
question re-tokenises every window, so the same words are stemmed again
and again.
"""
if len(token) <= 3:
return token
for suffix in _SUFFIXES:
if token.endswith(suffix) and len(token) - len(suffix) >= 3:
return token[: -len(suffix)]
return token
#: A number a document is known by -- `4.2`, `10.2-2`, `2.1.219` -- kept as ONE
#: token beside its pieces. Split on the dot, `4.2` is two single characters,
#: which are dropped, so a question naming a section by its number would match
#: nothing in the section it names. The shape is `consume`'s identifier rule.
_IDENTIFIER = re.compile(r"[0-9a-zà-öø-ÿ]*[0-9]+(?:[.-][0-9]+)+")
_DASH_TO_HYPHEN = str.maketrans(dict.fromkeys("‐‑‒–—―−", "-"))
def tokens(text: str) -> list[str]:
"""Casefolded, NFC, stopwords and single characters out, lightly stemmed,
plus every identifier whole and unstemmed."""
folded = unicodedata.normalize("NFC", text).casefold().translate(_DASH_TO_HYPHEN)
words = [
_stem(token)
for token in _TOKEN.findall(folded)
if len(token) > 1 and token not in STOPWORDS
]
return words + [token for token in _IDENTIFIER.findall(folded) if token not in words]
class Index:
"""Okapi BM25 over a list of token lists."""
def __init__(self, documents: Sequence[Sequence[str]]) -> None:
self.size = len(documents)
self.lengths = [len(document) for document in documents]
self.average = sum(self.lengths) / self.size if self.size else 0.0
self.postings: dict[str, list[tuple[int, int]]] = {}
for position, document in enumerate(documents):
for term, frequency in sorted(Counter(document).items()):
self.postings.setdefault(term, []).append((position, frequency))
def idf(self, terms: frozenset[str]) -> float:
"""Zero for terms no document holds: absence lifts nothing."""
df = len(self.holders(terms))
if df == 0:
return 0.0
return math.log(1 + (self.size - df + 0.5) / (df + 0.5))
def holders(self, terms: frozenset[str]) -> set[int]:
"""Every document position holding at least one of `terms`."""
return {position for term in terms for position, _ in self.postings.get(term, ())}
def scores(self, query: Sequence[frozenset[str]]) -> dict[int, float]:
"""Positive scores only, keyed by document position.
Each query element is a GROUP of terms read as one: its frequency in a
document is the sum over the group, its `df` the documents holding any.
A group of one is plain BM25.
"""
out: dict[int, float] = {}
for group in query:
weight = self.idf(group)
if weight == 0.0:
continue
frequencies: Counter[int] = Counter()
for term in sorted(group):
for position, frequency in self.postings.get(term, ()):
frequencies[position] += frequency
for position, frequency in sorted(frequencies.items()):
norm = K1 * (1 - B + B * self.lengths[position] / self.average)
out[position] = out.get(position, 0.0) + weight * (
frequency * (K1 + 1) / (frequency + norm)
)
return out
def windows(body: str) -> list[tuple[int, str]]:
"""`(start, text)` for every window of a body; one empty-start window if blank."""
if not body.strip():
return []
out = []
for start in range(0, len(body), WINDOW_STEP):
chunk = body[start : start + WINDOW_CHARS]
if chunk.strip():
out.append((start, chunk))
if start + WINDOW_CHARS >= len(body):
break
return out
def field_text(concept: Concept, body: str, *, shared: int = 0, own_source: bool = True) -> str:
"""Title and path weighted up, then the body: the field signal's document.
`shared` is how many leading id segments EVERY concept carries, and
`own_source` whether the source file name separates this concept from any
other. What every concept carries separates nothing, and a question naming
it would match them all -- the defect `consume.shared_id_prefix` closed for
the older ranking, and the reason both are dropped here.
"""
path = " ".join(concept.concept_id.split("/")[shared:]).replace("-", " ")
source = concept.source_file.removesuffix(".md").replace("-", " ") if own_source else ""
return f"{concept.title} " * TITLE_WEIGHT + f"{path} {source} " * PATH_WEIGHT + body
def _shared_segments(concept_ids: Sequence[str]) -> int:
"""How many leading DIRECTORY segments every id shares (never the leaf)."""
if not concept_ids:
return 0
split = [concept_id.split("/")[:-1] for concept_id in concept_ids]
count = 0
for segments in zip(*split):
if len(set(segments)) != 1:
break
count += 1
return count
def query_groups(query: Sequence[str], vocabulary: frozenset[str]) -> list[frozenset[str]]:
"""Each question term as the group of collection terms it is read as.
A term the collection holds is read as itself and nothing else. A term it
does NOT hold -- an inflection, a compound, a word in another language --
is read as the collection's words it shares a leading WORD with, by
`consume.tokens_match`, the rule the older ranking measured for Norwegian
inflection and compounding. So an absent word lifts nothing by itself: it
reaches the collection only through a relative the collection uses, at that
relative's `idf` and never at a weight of its own.
"""
from llm_ingestion_okf.consume import MIN_SHARED_PREFIX, tokens_match
by_prefix: dict[str, list[str]] = {}
for term in sorted(vocabulary):
by_prefix.setdefault(term[:MIN_SHARED_PREFIX], []).append(term)
groups = []
for term in dict.fromkeys(query):
if term in vocabulary:
groups.append(frozenset({term}))
continue
groups.append(
frozenset(
candidate
for candidate in by_prefix.get(term[:MIN_SHARED_PREFIX], ())
if tokens_match(term, candidate, stems=vocabulary)
)
)
return groups
@dataclass(frozen=True)
class Ranking:
"""Every concept best first, as `(concept, fused, lexical)`, plus where
each concept's best window starts in the body that was searched."""
ranked: list[tuple[Concept, float, int]]
best_window: dict[str, int]
def _fuse(fused: dict[str, float], scores: dict[str, float]) -> None:
"""Add one signal's reciprocal ranks; equal scores share the first rank."""
order = sorted(scores, key=lambda key: (-scores[key], key))
start = 0
while start < len(order):
stop = start
while stop < len(order) and scores[order[stop]] == scores[order[start]]:
stop += 1
contribution = 1.0 / (RRF_K + start + 1)
for concept_id in order[start:stop]:
fused[concept_id] += contribution
start = stop
def rank(
concepts: Sequence[Concept],
question: str,
*,
bodies: Sequence[str] | None = None,
) -> Ranking:
"""Rank `concepts` for `question`.
`bodies` is the text searched per concept (defaults to each `body`); the
caller passes the body without the door's link line, so what is searched
is what the older ranking searched.
"""
texts = list(bodies) if bodies is not None else [concept.body for concept in concepts]
query = tokens(question)
shared = _shared_segments([concept.concept_id for concept in concepts])
own_source = len({concept.source_file for concept in concepts}) > 1
field_documents = [
tokens(field_text(concept, text, shared=shared, own_source=own_source))
for concept, text in zip(concepts, texts, strict=True)
]
field_index = Index(field_documents)
vocabulary = frozenset(field_index.postings)
groups = query_groups(query, vocabulary)
field = {
concepts[position].concept_id: score
for position, score in field_index.scores(groups).items()
}
owners: list[int] = []
starts: list[int] = []
passages: list[list[str]] = []
for position, text in enumerate(texts):
cut = windows(text) or [(0, concepts[position].title)]
for start, chunk in cut:
owners.append(position)
starts.append(start)
passages.append(tokens(chunk))
passage_index = Index(passages)
passage: dict[str, float] = {}
best_window: dict[str, int] = {}
for window, score in sorted(passage_index.scores(groups).items()):
concept_id = concepts[owners[window]].concept_id
if score > passage.get(concept_id, 0.0):
passage[concept_id] = score
best_window[concept_id] = starts[window]
fused = {concept.concept_id: 0.0 for concept in concepts}
_fuse(fused, passage)
_fuse(fused, field)
asked = [group for group in groups if field_index.idf(group) > 0.0]
lexical = {
concept.concept_id: sum(1 for group in asked if group & held)
for concept, held in (
(concept, set(document))
for concept, document in zip(concepts, field_documents, strict=True)
)
}
by_id = {concept.concept_id: concept for concept in concepts}
order = sorted(fused, key=lambda key: (-fused[key], key))
return Ranking(
ranked=[(by_id[key], fused[key], lexical[key]) for key in order],
best_window=best_window,
)

296
src/llm_ingestion_okf/consume.py
View file

@ -54,6 +54,7 @@ from dataclasses import dataclass, replace
from pathlib import Path
from typing import Literal
from . import bm25
from .connectors import safe_resolve
from .corpus import LOG_NAME
from .errors import SourceError
@ -1476,6 +1477,76 @@ def shared_id_prefix(concept_ids: Sequence[str]) -> int:
return shared
def _partitioned(
ranked_ids: list[str],
concepts: Sequence[Concept],
question: str,
*,
title_covered: bool,
lookup: bool,
) -> list[str]:
"""The two partitions every ranking lands under: a concept whose title the
question covers rises, and a concept the question NAMES reads first.
Shared by the fusion and by BM25 so the two readings differ in how they
SCORE and in nothing else.
"""
question_tokens = normalise(question)
by_id = {concept.concept_id: concept for concept in concepts}
covered = set(title_covered_hits(concepts, question)) if title_covered else set()
if covered:
# A PARTITION, not a signal, and it lands BELOW the lookup partition
# so a question that NAMES a concept still reads that one first. See
# `DEFAULT_TITLE_COVERED` for the arithmetic that rules a signal out,
# and `tests/test_title_covered.py` for the mechanism on a fixture.
#
# BOUNDED BY RECALL since round 17: a covered concept RISES through the
# fusion's order and stops beneath the first concept whose title
# answers MORE question tokens, by equality, than the covered title
# holds -- or beneath a covered concept the fusion put above it. With
# nothing above it answering more, it reaches the top exactly where
# round 16's plain partition put it. `tests/test_title_covered_rise.py`
# holds the known-negative that bound exists for.
#
# STABLE: the covered concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
asked = frozenset(question_tokens)
answered = {
concept_id: len(set(normalise(by_id[concept_id].title)) & asked)
for concept_id in ranked_ids
}
risen: list[str] = []
for concept_id in ranked_ids:
stop = len(risen)
if concept_id in covered:
while (
stop
and risen[stop - 1] not in covered
and answered[risen[stop - 1]] <= answered[concept_id]
):
stop -= 1
risen.insert(stop, concept_id)
ranked_ids = risen
named = set(lookup_hits(concepts, question)) if lookup else set()
if named:
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
# partition rather than a fourth signal. The form was chosen by
# measurement, not by preference: a fourth RRF signal was simulated on
# the same three bundles first and put the named concept at rank
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
# consumes RANKS ONLY, so any single signal contributes at most
# `1/(RRF_K + 1)` however certain it is, and a concept the question
# NAMES cannot outbid three signals that merely describe it
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
#
# STABLE: the named concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
ranked_ids = [key for key in ranked_ids if key in named] + [
key for key in ranked_ids if key not in named
]
return ranked_ids
def concept_scores(
concepts: Sequence[Concept],
question: str,
@ -1648,58 +1719,13 @@ def concept_scores(
}
)
by_id = {concept.concept_id: concept for concept in concepts}
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
covered = set(title_covered_hits(concepts, question)) if title_covered else set()
if covered:
# A PARTITION, not a signal, and it lands BELOW the lookup partition
# so a question that NAMES a concept still reads that one first. See
# `DEFAULT_TITLE_COVERED` for the arithmetic that rules a signal out,
# and `tests/test_title_covered.py` for the mechanism on a fixture.
#
# BOUNDED BY RECALL since round 17: a covered concept RISES through the
# fusion's order and stops beneath the first concept whose title
# answers MORE question tokens, by equality, than the covered title
# holds -- or beneath a covered concept the fusion put above it. With
# nothing above it answering more, it reaches the top exactly where
# round 16's plain partition put it. `tests/test_title_covered_rise.py`
# holds the known-negative that bound exists for.
#
# STABLE: the covered concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
asked = frozenset(question_tokens)
answered = {
concept_id: len(set(normalise(by_id[concept_id].title)) & asked)
for concept_id in ranked_ids
}
risen: list[str] = []
for concept_id in ranked_ids:
stop = len(risen)
if concept_id in covered:
while (
stop
and risen[stop - 1] not in covered
and answered[risen[stop - 1]] <= answered[concept_id]
):
stop -= 1
risen.insert(stop, concept_id)
ranked_ids = risen
named = set(lookup_hits(concepts, question)) if lookup else set()
if named:
# THE LOOKUP LANDS BEFORE THE FUSION'S OUTPUT IS READ, and it is a
# partition rather than a fourth signal. The form was chosen by
# measurement, not by preference: a fourth RRF signal was simulated on
# the same three bundles first and put the named concept at rank
# **26 / 15 / 19** of 446 / 1 133 / 270 -- none of them delivered. RRF
# consumes RANKS ONLY, so any single signal contributes at most
# `1/(RRF_K + 1)` however certain it is, and a concept the question
# NAMES cannot outbid three signals that merely describe it
# (`docs/2026-09-08-sjeldenhetsvekt.md` SS 4 predicted exactly this).
#
# STABLE: the named concepts keep the order the fusion gave them, and
# so does everything else, so nothing here depends on dict order.
ranked_ids = [key for key in ranked_ids if key in named] + [
key for key in ranked_ids if key not in named
]
ranked_ids = _partitioned(
sorted(fused, key=lambda key: (-fused[key], key)),
concepts,
question,
title_covered=title_covered,
lookup=lookup,
)
return [
(by_id[concept_id], fused[concept_id], lexical[concept_id]) for concept_id in ranked_ids
]
@ -1820,6 +1846,75 @@ def excerpt_for(concept: Concept) -> dict[str, object] | None:
return excerpt
#: The most characters of a concept's body one excerpt carries (v1.1 C3).
#: Chosen as 4 000 because the default `k` of 8 excerpts then carries at most
#: 32 000 characters of text -- an answer that stays well under a tool
#: response's 25 000-token limit even when four sub-questions are merged into
#: it -- while a 500-character answering window keeps 3 500 characters of
#: surroundings, enough for the place to read alone. A concept at or under it
#: is delivered whole, byte for byte as before.
PASSAGE_CHARS = 4_000
#: Room for the heading line a passage is prefixed with and the two markers
#: that say text was cut before or after it. A heading longer than this is cut.
PASSAGE_HEADING_ALLOWANCE = 240
#: Written where a passage leaves text out, at the start or the end.
PASSAGE_ELISION = "[...]"
_HEADING_LINE = re.compile(r"^#{1,6} \S.*$", re.MULTILINE)
def as_passage(excerpt: dict[str, object], window: int) -> dict[str, object]:
"""`excerpt` with its `text` cut to the span around `window`, when too long.
The span is `PASSAGE_CHARS` wide, centred on the answering window the
ranking found, and snapped inward to whole lines. The nearest heading above
the span is carried as its first line when the span does not already hold
it, so a table row or a paragraph is read under the section it belongs to.
`passage` records `{start, end, of}` in characters of the WHOLE delivered
text, and `text_sha256` is recomputed over what is delivered -- `sha256`
stays the concept file's, so a reader can tell the excerpt from the whole
and fetch the whole by `concept_id`.
"""
text = excerpt["text"]
assert isinstance(text, str)
if len(text) <= PASSAGE_CHARS:
return excerpt
centre = min(max(window, 0), len(text)) + 250
start = max(0, min(centre - PASSAGE_CHARS // 2, len(text) - PASSAGE_CHARS))
end = min(len(text), start + PASSAGE_CHARS)
if start > 0:
# Inward to the next line start, so no line arrives cut in half.
newline = text.find("\n", start, end)
start = newline + 1 if newline != -1 else start
if end < len(text):
newline = text.rfind("\n", start, end)
end = newline if newline > start else end
span = text[start:end]
heading = ""
if start > 0:
above = [match.group(0) for match in _HEADING_LINE.finditer(text, 0, start)]
if above and above[-1] not in span.split("\n", 1)[0]:
heading = above[-1][: PASSAGE_HEADING_ALLOWANCE - 2 * len(PASSAGE_ELISION) - 4]
parts = []
if heading:
parts.append(heading)
if start > 0:
parts.append(PASSAGE_ELISION)
parts.append(span)
if end < len(text):
parts.append(PASSAGE_ELISION)
delivered = "\n".join(parts)
out = dict(excerpt)
out["passage"] = {"start": start, "end": end, "of": len(text)}
out["text_sha256"] = hashlib.sha256(delivered.encode("utf-8")).hexdigest()
# `text` stays the LAST key, as `excerpt_for` writes it.
del out["text"]
out["text"] = delivered
return out
def excerpt_weight(excerpt: Mapping[str, object]) -> int:
"""What this excerpt costs by the gate's own instrument.
@ -2010,6 +2105,7 @@ def cut(
limit: int,
reserve_top_rank: bool = False,
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
windows: Mapping[str, int] | None = None,
) -> tuple[tuple[dict[str, object], ...], tuple[tuple[str, str], ...], tuple[str, int] | None]:
"""The ranked concepts split into delivered excerpts, named drops, and the
reservation that was made, if any.
@ -2054,6 +2150,9 @@ def cut(
if excerpt is None:
withheld.append((concept.concept_id, "verified_unreadable"))
continue
if windows is not None:
# BEFORE the weight is read: the budget pays for what is delivered.
excerpt = as_passage(excerpt, windows.get(concept.concept_id, 0))
weight = excerpt_weight(excerpt)
if weight > limit:
withheld.append((concept.concept_id, "over_budget_alone"))
@ -2152,6 +2251,14 @@ WITHHELD_NEAREST_DEFAULT = 20
#: 300-item DP to discover the same answer.
DEFAULT_K = 8
#: The two readings `build_payload` can rank with. `bm25` is the default since
#: v1.1 (see `llm_ingestion_okf.bm25` for the mechanism and why it replaced the
#: fusion); `fusion` is the older three-signal RRF, kept reachable because its
#: mechanisms are pinned by tests of their own and a caller may need the old
#: order. The two share the lookup and title partitions and the cut.
RANKINGS = ("bm25", "fusion")
DEFAULT_RANKING = "bm25"
def root_bundle_id_of(bundle_root: Path, *, profile: BundleProfile = DEFAULT_PROFILE) -> str:
"""The `bundle_id` the root index declares, or a refusal naming which half
@ -2299,6 +2406,7 @@ def build_payload(
source_quota: int | None = DEFAULT_SOURCE_QUOTA,
follow_parent: bool = DEFAULT_FOLLOW_PARENT,
link_in_signal: bool = DEFAULT_LINK_IN_SIGNAL,
ranking: str = DEFAULT_RANKING,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -2373,25 +2481,66 @@ def build_payload(
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters.
stems = frozenset(token for tokens in tokenised for token in tokens) if stem_prefix else None
weights = rarity_weights(normalise(question), texts, stems=stems) if rarity_weight else None
ranked = concept_scores(
concepts,
question,
document_scores(
bundle_root,
if ranking not in RANKINGS:
raise ConsumeError(
f"unknown ranking {ranking!r}; expected one of {', '.join(RANKINGS)}",
code="ranking_invalid",
)
if ranking == "bm25" and (cost_vocabulary or rarity_weight):
# Both widen a signal only the fusion has. Accepting them here would be
# a flag that silently does nothing, which reads as a measurement.
raise ConsumeError(
"--cost-vocabulary and --rarity-weight are widenings of the fusion "
"ranking; ask for it with ranking='fusion'",
code="ranking_flag_conflict",
)
if ranking == "bm25":
result = bm25.rank(
concepts,
question,
profile=profile,
# The DELIVERED form of each body, so a window's offset points at
# the same characters `as_passage` cuts from.
bodies=[
delivered_text(
concept.body if link_in_signal else body_without_link_line(concept.body)
)
for concept in concepts
],
)
by_id = {concept.concept_id: concept for concept in concepts}
scored = {concept.concept_id: (score, lexical) for concept, score, lexical in result.ranked}
windows: Mapping[str, int] | None = result.best_window
ranked = [
(by_id[concept_id], *scored[concept_id])
for concept_id in _partitioned(
[concept.concept_id for concept, _, _ in result.ranked],
concepts,
question,
title_covered=title_covered,
lookup=True,
)
]
else:
windows = None
weights = rarity_weights(normalise(question), texts, stems=stems) if rarity_weight else None
ranked = concept_scores(
concepts,
question,
document_scores(
bundle_root,
question,
profile=profile,
cost_vocabulary=cost_vocabulary,
weights=weights,
stems=stems,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
tie_shared_rank=tie_shared_rank,
title_covered=title_covered,
stems=stems,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
tie_shared_rank=tie_shared_rank,
title_covered=title_covered,
stems=stems,
link_in_signal=link_in_signal,
)
link_in_signal=link_in_signal,
)
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(
@ -2400,6 +2549,7 @@ def build_payload(
limit=limit,
reserve_top_rank=reserve_top_rank,
source_quota=source_quota,
windows=windows,
)
if follow_parent:
# After the cut and never inside it: see `attach_parent_text`.
@ -2669,6 +2819,17 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"budget the payload reports"
),
)
parser.add_argument(
"--ranking",
choices=RANKINGS,
default=DEFAULT_RANKING,
help=(
f"how concepts are ordered before the cut. Default {DEFAULT_RANKING}: "
"passage and title BM25 fused by rank, a large concept delivered as "
"its answering passage. `fusion` is the older three-signal ranking; "
"--cost-vocabulary and --rarity-weight widen it and need it"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument(
"--ref",
@ -2710,6 +2871,7 @@ def main(argv: list[str] | None = None) -> int:
withheld_nearest=args.withheld_nearest,
withheld_full=args.withheld_full,
follow_parent=args.follow_parent,
ranking=args.ranking,
)
except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr)

191
tests/test_bm25_ranking.py Normal file
View file

@ -0,0 +1,191 @@
"""The BM25 ranking (v1.1 order C, C1): the default reading of `okf consume`.
Each test states one property of the mechanism on a synthetic bundle small
enough to reason about by hand. The measurement against a real collection is
the search gate's (`tools/okf_soek_gate.py`), not this file's.
"""
from __future__ import annotations
from pathlib import Path
from llm_ingestion_okf import bm25, consume
def _concept(concept_id: str, title: str, body: str) -> consume.Concept:
return consume.Concept(
path=Path(f"{concept_id}.md"),
concept_id=concept_id,
bundle_id="b",
bundle_id_inherited=True,
sha256="0" * 64,
okf_type="concept",
title=title,
source_file=f"{concept_id.split('/')[0]}.md",
adjudication="unknown",
adjudication_present=False,
req_number="",
sources=(),
sources_present=False,
locators={},
frontmatter={},
body=body,
)
def test_the_tokeniser_drops_stopwords_and_single_characters_and_folds_case() -> None:
assert bm25.tokens("What IS the Default model for a teammate?") == [
"default",
"model",
"teammate",
]
def test_the_tokeniser_keeps_a_norwegian_word_whole() -> None:
# A letter outside ASCII must not split a word into fragments that can
# match something unrelated in an English collection.
assert bm25.tokens("første") == ["første"]
def test_a_word_the_collection_does_not_hold_lifts_nothing() -> None:
concepts = [
_concept("a/one", "One", "the alpha feature is described here"),
_concept("b/two", "Two", "the beta feature is described here"),
]
plain = bm25.rank(concepts, "alpha feature")
padded = bm25.rank(concepts, "alpha feature zzqqxx")
assert [c.concept_id for c, _, _ in plain.ranked] == [c.concept_id for c, _, _ in padded.ranked]
assert [score for _, score, _ in plain.ranked] == [score for _, score, _ in padded.ranked]
def test_length_normalisation_prefers_the_short_concept_on_one_shared_term() -> None:
filler = " ".join(f"word{i}" for i in range(400))
concepts = [
_concept("a/long", "Long", f"rotation {filler}"),
_concept("b/short", "Short", "rotation of keys"),
]
ranked = bm25.rank(concepts, "rotation").ranked
assert ranked[0][0].concept_id == "b/short"
def test_the_title_field_separates_two_equal_bodies() -> None:
# The passage signal reads bodies only, so two equal bodies tie there and
# the field signal -- title and path weighted up -- decides.
concepts = [
_concept("a/body", "Unrelated", "notes about the sandbox and its settings"),
_concept("b/title", "Sandbox", "notes about the sandbox and its settings"),
]
assert bm25.rank(concepts, "sandbox").ranked[0][0].concept_id == "b/title"
def test_the_best_window_wins_rather_than_the_sum() -> None:
# Ten scattered mentions sum to more than one dense window; the rule is
# `max`, so the concept whose ONE window answers the question leads.
scattered = " ".join(["hooks"] + [f"pad{i}" for i in range(120)]) * 10
dense = "hooks configure hooks per event, hooks run commands"
concepts = [
_concept("a/scattered", "Scattered", scattered),
_concept("b/dense", "Dense", dense),
]
ranking = bm25.rank(concepts, "hooks configure event commands")
assert ranking.ranked[0][0].concept_id == "b/dense"
def test_the_best_window_offset_points_at_the_answering_text() -> None:
body = "x " * 2000 + "the answer about retention lives here " + "y " * 2000
ranking = bm25.rank([_concept("a/doc", "Doc", body)], "retention answer")
start = ranking.best_window["a/doc"]
assert "retention" in body[start : start + bm25.WINDOW_CHARS]
def test_a_concept_matching_nothing_carries_zero_lexical_and_sorts_by_id() -> None:
concepts = [
_concept("c/none", "C", "nothing relevant"),
_concept("b/none", "B", "nothing relevant"),
_concept("a/hit", "A", "the keyword appears"),
]
ranked = bm25.rank(concepts, "keyword").ranked
assert [(c.concept_id, lexical) for c, _, lexical in ranked] == [
("a/hit", 1),
("b/none", 0),
("c/none", 0),
]
def test_two_rankings_of_the_same_input_are_identical() -> None:
concepts = [_concept(f"d{i}/c", f"T{i}", f"shared term {i} " * (i + 1)) for i in range(12)]
first = bm25.rank(concepts, "shared term")
second = bm25.rank(list(reversed(concepts)), "shared term")
assert [(c.concept_id, s) for c, s, _ in first.ranked] == [
(c.concept_id, s) for c, s, _ in second.ranked
]
def test_the_default_ranking_is_bm25_and_the_fusion_is_still_reachable() -> None:
assert consume.DEFAULT_RANKING == "bm25"
assert set(consume.RANKINGS) == {"bm25", "fusion"}
def test_a_directory_every_concept_shares_matches_nothing() -> None:
# K3-20's defect, one signal over: in a one-document bundle every id
# carries the document's directory, so a question naming the document
# would otherwise match every concept.
concepts = [
_concept("handbook/intro", "Intro", "welcome to the club"),
_concept("handbook/fees", "Fees", "membership costs money"),
]
ranked = bm25.rank(concepts, "handbook").ranked
assert [lexical for _, _, lexical in ranked] == [0, 0]
assert [score for _, score, _ in ranked] == [0.0, 0.0]
def test_a_directory_that_separates_concepts_still_counts() -> None:
concepts = [
_concept("billing/overview", "Overview", "general words"),
_concept("security/overview", "Overview", "general words"),
]
assert bm25.rank(concepts, "billing").ranked[0][0].concept_id == "billing/overview"
def test_an_absent_inflection_reaches_the_form_the_collection_holds() -> None:
# `vinterberedskapen` occurs nowhere; `vinterberedskap` does, and it is the
# shared prefix -- a WORD of this collection -- that bridges the two
# (`consume.tokens_match`, the rule the older ranking measured for
# Norwegian inflection and compounding).
concepts = [
_concept("a/winter", "Vinterberedskap", "vinterberedskap kontrolleres hver host"),
_concept("b/summer", "Sommer", "sommerdrift og vedlikehold"),
]
ranked = bm25.rank(concepts, "Når kontrolleres vinterberedskapen?").ranked
assert ranked[0][0].concept_id == "a/winter"
assert ranked[0][2] == 2
def test_a_word_the_collection_holds_is_matched_as_itself_only() -> None:
# `mode` is in the collection, so it is never widened to `model`: the
# bridge is for a word that is absent, never a second reading of one that
# is present.
concepts = [
_concept("a/mode", "Fast mode", "fast mode speeds output"),
_concept("b/model", "Model", "model selection and model aliases"),
]
ranked = bm25.rank(concepts, "mode").ranked
assert [(c.concept_id, lexical) for c, _, lexical in ranked] == [("a/mode", 1), ("b/model", 0)]
def test_a_number_a_document_is_known_by_is_kept_whole() -> None:
# `4.2` split on the dot is two single characters, and single characters
# are dropped: without the whole token a question naming a section by its
# number matches nothing in the section it names.
assert "4.2" in bm25.tokens("Hva staar i punkt 4.2?")
assert "10.2-2" in bm25.tokens("Krav 10.2—2")
def test_a_concept_named_by_its_number_carries_a_lexical_match() -> None:
concepts = [
_concept("r/vakthold-4-2", "Vakthold 4.2", "Vakten gaar fra fredag til soendag."),
_concept("r/notat", "Notat om ettersyn", "Kontrollen av hytta foeres i skjema."),
]
ranked = bm25.rank(concepts, "Hva staar i punkt 4.2 om kontrollen av hytta?").ranked
lexical = {concept.concept_id: count for concept, _, count in ranked}
assert lexical["r/vakthold-4-2"] >= 1

17
tests/test_default_bundle_pin.py
View file

@ -1,5 +1,10 @@
"""The bundle the DEFAULT build produces, pinned where a regression goes red.
READ WITH THE FUSION RANKING since v1.1: these ranks were measured on the
three-signal fusion, which `okf consume` no longer uses by default. The
default BM25 reading of these bytes is a separate measurement and is not
pinned here.
`tests/test_okf_consume.py` pinned hit@8 against the Arm B bundle alone -- the
configuration `okf build` stopped emitting on 2026-09-08. A published number
measured on a bundle nobody produces is a number that cannot regress, so the
@ -120,7 +125,9 @@ def test_hit_at_eight_holds_rank_one_on_every_row_it_held() -> None:
assert len(questions) == len(EXPECTED_RANKS), "the gold set changed shape"
ranks = []
for entry in questions:
payload = okf_consume.build_payload(DEFAULT_BUNDLE, question=entry["question"])
payload = okf_consume.build_payload(
DEFAULT_BUNDLE, question=entry["question"], ranking="fusion"
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
@ -153,7 +160,7 @@ def test_the_reading_default_is_what_holds_row_one_on_these_bytes() -> None:
ranks = []
for entry in questions:
payload = okf_consume.build_payload(
DEFAULT_BUNDLE, question=entry["question"], tie_shared_rank=False
DEFAULT_BUNDLE, question=entry["question"], ranking="fusion", tie_shared_rank=False
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
@ -185,7 +192,9 @@ def test_the_stem_rule_holds_every_rank_on_the_shipped_bytes() -> None:
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
ranks = []
for entry in questions:
payload = okf_consume.build_payload(DEFAULT_BUNDLE, question=entry["question"])
payload = okf_consume.build_payload(
DEFAULT_BUNDLE, question=entry["question"], ranking="fusion"
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
@ -209,7 +218,7 @@ def test_the_document_quota_is_what_reaches_row_six_on_these_bytes() -> None:
ranks = []
for entry in questions:
payload = okf_consume.build_payload(
DEFAULT_BUNDLE, question=entry["question"], source_quota=None
DEFAULT_BUNDLE, question=entry["question"], ranking="fusion", source_quota=None
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)

31
tests/test_okf_consume.py
View file

@ -949,7 +949,7 @@ def test_every_gold_document_in_the_local_set_is_reached_or_named_as_a_miss() ->
assert len(questions) >= 5, "fewer than five questions is not the measurement"
hits = 0
for entry in questions:
payload = okf_consume.build_payload(K2_BUNDLE, question=entry["question"])
payload = okf_consume.build_payload(K2_BUNDLE, question=entry["question"], ranking="fusion")
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
if okf_consume_measure.hit_rank(excerpts, entry["gold_document"]) is not None:
@ -1484,7 +1484,7 @@ def test_a_cost_question_reaches_no_price_concept_without_the_flag() -> None:
def test_the_cost_vocabulary_flag_bridges_a_question_and_a_document_that_share_no_word() -> None:
payload = okf_consume.build_payload(
FIXTURE, question="Hvor kan vi kutte kostnader?", cost_vocabulary=True
FIXTURE, question="Hvor kan vi kutte kostnader?", cost_vocabulary=True, ranking="fusion"
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
@ -1506,9 +1506,13 @@ def test_the_flag_changes_nothing_when_the_question_names_no_such_term() -> None
# The GATE is the question, never the flag: a question with no cost term
# gets byte-identical bytes whether the flag is set or not.
question = "Hvor ofte er den årlige kontrollen?"
off = okf_consume.serialise(okf_consume.build_payload(FIXTURE, question=question))
off = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, ranking="fusion")
)
on = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, cost_vocabulary=True)
okf_consume.build_payload(
FIXTURE, question=question, cost_vocabulary=True, ranking="fusion"
)
)
assert off == on
@ -1559,13 +1563,15 @@ def test_the_cli_exposes_the_flag_and_omitting_it_reproduces_the_default_bytes()
question = "Hvordan skal prisene fylles ut?"
plain = _run(str(FIXTURE), "--question", question)
assert plain.returncode == 0
flagged = _run(str(FIXTURE), "--question", question, "--cost-vocabulary")
flagged = _run(str(FIXTURE), "--question", question, "--cost-vocabulary", "--ranking", "fusion")
assert flagged.returncode == 0
assert plain.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question)
)
assert flagged.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, cost_vocabulary=True)
okf_consume.build_payload(
FIXTURE, question=question, cost_vocabulary=True, ranking="fusion"
)
)
@ -1631,6 +1637,9 @@ def _eviction_payload(
k=k,
limit=limit,
reserve_top_rank=reserve_top_rank,
# The knapsack and the reservation are measured on the fusion, whose
# order these fixtures were written against.
ranking="fusion",
)
@ -1852,6 +1861,8 @@ def test_the_cli_exposes_the_reservation_and_omitting_it_reproduces_the_default_
"16",
"--limit",
str(EVICTION_LIMIT),
"--ranking",
"fusion",
)
plain = _run(*common)
assert plain.returncode == 0
@ -2115,14 +2126,14 @@ def test_the_weight_is_off_by_default_and_the_default_payload_is_unmoved() -> No
def test_the_cli_exposes_the_weight_and_omitting_it_reproduces_the_default_bytes() -> None:
question = "Hva krever Krav 10.2-2 om sentrale vilkår?"
plain = _run(str(FIXTURE), "--question", question)
weighted = _run(str(FIXTURE), "--question", question, "--rarity-weight")
weighted = _run(str(FIXTURE), "--question", question, "--rarity-weight", "--ranking", "fusion")
assert plain.returncode == 0, plain.stderr
assert weighted.returncode == 0, weighted.stderr
assert plain.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question)
)
assert weighted.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, rarity_weight=True)
okf_consume.build_payload(FIXTURE, question=question, rarity_weight=True, ranking="fusion")
)
@ -2161,8 +2172,8 @@ def test_build_payload_hands_the_same_weights_to_the_document_prior(
monkeypatch.setattr(okf_consume, "document_scores", spy)
question = "Hvordan skal prisene fylles ut?"
okf_consume.build_payload(FIXTURE, question=question)
okf_consume.build_payload(FIXTURE, question=question, rarity_weight=True)
okf_consume.build_payload(FIXTURE, question=question, ranking="fusion")
okf_consume.build_payload(FIXTURE, question=question, rarity_weight=True, ranking="fusion")
assert seen[0] is None
expected = okf_consume.rarity_weights(
okf_consume.normalise(question), okf_consume.searchable_text(_fixture_concepts())

117
tests/test_passage_delivery.py Normal file
View file

@ -0,0 +1,117 @@
"""A large concept is delivered as its RELEVANT PASSAGE (v1.1 order C, C3).
The reader receives the place that answers, with the heading it sits under and
enough surroundings to read alone, plus the concept's name so the whole can be
fetched. A concept at or under `PASSAGE_CHARS` is delivered whole, as before.
"""
from __future__ import annotations
import hashlib
import sys
from pathlib import Path
import pytest
from llm_ingestion_okf import consume
TOOLS = Path(__file__).resolve().parent.parent / "tools"
if str(TOOLS) not in sys.path:
sys.path.insert(0, str(TOOLS))
import okf_retrieval_gate as retrieval # noqa: E402
ANSWER = "The retention window for archived sessions is ninety days."
def _long_body() -> str:
filler = "\n".join(
f"Paragraph {i} describes an unrelated setting in detail." for i in range(300)
)
tail = "\n".join(f"Closing note {i} about something else." for i in range(300))
return f"{filler}\n\n## Session retention\n\n{ANSWER}\n\n{tail}"
def _excerpt(text: str) -> dict[str, object]:
return {
"concept_id": "doc/big",
"text": text,
"text_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
}
def test_a_short_excerpt_is_delivered_whole() -> None:
excerpt = _excerpt("A short body.")
assert consume.as_passage(dict(excerpt), 0) == excerpt
def test_a_long_excerpt_is_cut_to_the_window_with_its_heading() -> None:
body = _long_body()
window = body.index(ANSWER)
out = consume.as_passage(_excerpt(body), window)
text = out["text"]
assert isinstance(text, str)
assert ANSWER in text
assert "## Session retention" in text
assert len(text) <= consume.PASSAGE_CHARS + consume.PASSAGE_HEADING_ALLOWANCE
assert out["text_sha256"] == hashlib.sha256(text.encode("utf-8")).hexdigest()
passage = out["passage"]
assert isinstance(passage, dict)
assert passage["of"] == len(body)
assert 0 < passage["start"] <= window < passage["end"] <= len(body)
assert body[passage["start"] : passage["end"]] in text
def test_the_heading_is_carried_even_when_it_lies_before_the_span() -> None:
body = (
"# Top\n\n## Far heading\n\n" + ("filler line here\n" * 600) + ANSWER + "\n" + "x\n" * 600
)
out = consume.as_passage(_excerpt(body), body.index(ANSWER))
text = out["text"]
assert isinstance(text, str)
assert text.startswith("## Far heading\n")
assert ANSWER in text
def test_the_passage_is_cut_at_line_boundaries() -> None:
body = _long_body()
out = consume.as_passage(_excerpt(body), body.index(ANSWER))
passage = out["passage"]
assert isinstance(passage, dict)
assert passage["start"] == 0 or body[passage["start"] - 1] == "\n"
assert passage["end"] == len(body) or body[passage["end"]] == "\n"
@pytest.fixture(scope="module")
def bundle(tmp_path_factory: pytest.TempPathFactory) -> Path:
spec = retrieval.BundleSpec(
"passage-synthetic",
(
retrieval.DocumentSpec(
"manual",
"manual.md",
(retrieval.ConceptSpec(slug="big", title="Operations", body=_long_body()),),
),
),
)
return retrieval.build_bundle(tmp_path_factory.mktemp("passage") / "bundle", spec)
def test_the_payload_delivers_the_answering_passage_of_a_large_concept(bundle: Path) -> None:
payload = consume.build_payload(bundle, question="retention window archived sessions")
excerpts = payload["excerpts"]
assert isinstance(excerpts, list) and len(excerpts) == 1
text = excerpts[0]["text"]
assert ANSWER in text
assert len(text) <= consume.PASSAGE_CHARS + consume.PASSAGE_HEADING_ALLOWANCE
assert "passage" in excerpts[0]
def test_the_fusion_ranking_still_delivers_the_whole_concept(bundle: Path) -> None:
payload = consume.build_payload(
bundle, question="retention window archived sessions", ranking="fusion"
)
excerpts = payload["excerpts"]
assert isinstance(excerpts, list) and len(excerpts) == 1
assert "passage" not in excerpts[0]
assert len(excerpts[0]["text"]) > consume.PASSAGE_CHARS

35
tests/test_retrieval_gate.py
View file

@ -130,6 +130,19 @@ def test_row_one_is_green_when_the_ranker_delivers_every_fasit(tmp_path: Path) -
assert (row.k, row.m, row.status) == (9, 9, gate.GREEN)
#: v1.1 C1 moved the default ranking to BM25, and four of this gate's premises
#: were built against the fusion it replaced: `set-miss` and B1/B2 force a
#: concept BELOW k that BM25 now delivers, `set-quota` needs the source quota to
#: decide a cut it no longer decides, and row 7's mutants patch fusion code the
#: default no longer runs. The gate says so itself (`premise broken`, `NOT
#: RUN`, `8 of 14`). STRICT, so the day the fixtures are re-measured for BM25
#: these flip loudly instead of staying skipped.
FUSION_PREMISE = pytest.mark.xfail(
strict=True,
reason="fixture premise built against the fusion ranking; re-measure for BM25",
)
def test_row_one_is_red_when_a_fasit_is_not_delivered(tmp_path: Path) -> None:
path, sha = _set_file(
tmp_path / "set.json",
@ -138,7 +151,9 @@ def test_row_one_is_red_when_a_fasit_is_not_delivered(tmp_path: Path) -> None:
questions=[
{
"id": "R1",
"question": "Hvor mange medlemmer maa stemme for en endring av vedtektene?",
# Shares no word with the fasit concept, so no ranking can
# deliver it -- the row's red state is forced by the fixture.
"question": "Hvem eier kanoen ved brygga?",
"fasit": [
{
"by": "concept",
@ -165,6 +180,7 @@ def test_row_one_never_counts_a_question_that_declares_the_class_it_forces(
# --- row 2 --------------------------------------------------------------------
@FUSION_PREMISE
def test_row_two_is_green_when_every_class_is_the_one_its_fixture_forces(
tmp_path: Path,
) -> None:
@ -260,6 +276,7 @@ class _AlwaysTheQuota(dict[str, str]):
return "source_quota_exceeded"
@FUSION_PREMISE
def test_row_three_is_green_on_the_shipped_code(tmp_path: Path) -> None:
"""And the same three fixtures, unmutated, are the green direction.
@ -475,6 +492,7 @@ def _noop_mutant() -> gate.Mutant:
return gate.Mutant("N01 nothing is changed", 1, lambda: gate._patched())
@FUSION_PREMISE
def test_row_seven_is_green_when_every_mutant_is_felled(tmp_path: Path) -> None:
cases, _ = gate.synthetic_cases(tmp_path / "bundles", FIXTURES)
baseline = gate.deterministic_rows(cases)
@ -648,13 +666,13 @@ def test_the_gate_is_red_today_and_says_which_rows(tmp_path: Path) -> None:
rows = gate.evaluate(tmp_path / "bundles")
by_number = {row.number: row for row in rows}
assert sorted(by_number) == [1, 2, 3, 4, 5, 6, 7, 8, 9]
assert [row.number for row in rows if row.fails] == [5, 7, 8, 9]
# 10, not 9: `set-quota.json` adds row 3's known-positive, one question the
# source quota genuinely decides, and it is a hit.
# Rows 2, 3 and 7 went red with v1.1 C1's BM25 default: see
# `FUSION_PREMISE`. Row 1 holds every fasit it held.
assert [row.number for row in rows if row.fails] == [2, 3, 5, 7, 8, 9]
assert (by_number[1].k, by_number[1].m) == (10, 10)
assert (by_number[2].k, by_number[2].m) == (7, 7)
assert (by_number[3].k, by_number[3].m) == (5, 5)
assert (by_number[6].k, by_number[6].m) == (10, 10)
assert (by_number[2].k, by_number[2].m) == (5, 7)
assert (by_number[3].k, by_number[3].m) == (3, 5)
assert (by_number[6].k, by_number[6].m) == (12, 12)
def test_the_same_tree_measures_the_same_twice(tmp_path: Path) -> None:
@ -670,7 +688,7 @@ def test_the_command_exits_one_and_prints_every_row(
printed = capsys.readouterr().out
for number in range(1, 10):
assert f"\n{number} " in f"\n{printed}"
assert "GATE RED: rows 5, 7, 8, 9" in printed
assert "GATE RED: rows 2, 3, 5, 7, 8, 9" in printed
def test_the_json_form_carries_the_same_rows(capsys: pytest.CaptureFixture[str]) -> None:
@ -1262,6 +1280,7 @@ def test_a_threshold_that_is_not_a_number_is_refused(tmp_path: Path) -> None:
assert row.fails
@FUSION_PREMISE
def test_the_threshold_is_compared_with_the_measured_hold_out(tmp_path: Path) -> None:
"""Both directions, from the same code path: a set the bundle answers
clears a threshold under it, and a set it does not answer falls under one

20
tests/test_tie_shared_rank.py
View file

@ -143,9 +143,11 @@ def test_the_payload_is_byte_identical_with_the_flag_on(tmp_path: Path) -> None:
must produce the same bytes -- only the value it names moved.
"""
root = _tie_bundle(tmp_path / "bundle")
without = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION))
without = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, ranking="fusion")
)
explicit_on = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, tie_shared_rank=True)
okf_consume.build_payload(root, question=QUESTION, ranking="fusion", tie_shared_rank=True)
)
assert without == explicit_on
@ -160,17 +162,23 @@ def test_the_opt_out_reproduces_the_order_the_default_used_to_give(tmp_path: Pat
bytes as the default on the very fixture built to separate them.
"""
root = _tie_bundle(tmp_path / "bundle")
default = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION, k=3))
default = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, ranking="fusion", k=3)
)
opted_out = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=False)
okf_consume.build_payload(
root, question=QUESTION, ranking="fusion", k=3, tie_shared_rank=False
)
)
assert default != opted_out
def test_the_flag_changes_the_payload_it_is_meant_to_change(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
off = okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=False)
on = okf_consume.build_payload(root, question=QUESTION, k=3)
off = okf_consume.build_payload(
root, question=QUESTION, ranking="fusion", k=3, tie_shared_rank=False
)
on = okf_consume.build_payload(root, question=QUESTION, ranking="fusion", k=3)
delivered_off = [excerpt["concept_id"] for excerpt in off["excerpts"]] # type: ignore[index]
delivered_on = [excerpt["concept_id"] for excerpt in on["excerpts"]] # type: ignore[index]
assert not any(str(cid).endswith("zz-gull") for cid in delivered_off)