feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage
C1. `okf consume` and MCP's `okf_ask` now rank with BM25 (`bm25.py`) instead
of the three-signal fusion. Two signals, fused by reciprocal rank:
- passage: every body cut into 500-character windows every 250, a concept
scored by its BEST window -- a narrow question is answered in one place;
- field: title three times, the id path and source name twice, then the body
-- a broad question is answered by what a section is called.
The document prior and the rarity weight are gone from the default: the first
favoured big documents full of common words, the second gave its largest
weight to a word the collection does not hold. Under BM25 such a word weighs
exactly zero. A signal that scores a concept zero adds nothing to it, and ties
share a rank, so alphabetical order lifts nothing either.
Three rules carried over from the fusion, each with its own test, because the
suite showed what BM25 alone lost:
- a directory every concept shares is not read (K3-20's defect, one signal on);
- a number a section is known by (`4.2`, `10.2-2`) is kept as one token, or
a question naming a section by its number matches nothing in it;
- a question word the collection does NOT hold is read as the collection's
words it shares a leading word with (`consume.tokens_match`) -- Norwegian
inflection and compounding -- at that word's idf, never at its own.
The lookup and title-covered partitions are shared with the fusion
(`_partitioned`). `ranking="fusion"` / `--ranking fusion` keeps the old order
reachable; `--cost-vocabulary` and `--rarity-weight` widen only the fusion and
are refused with the default (`ranking_flag_conflict`) rather than ignored.
C3. A concept longer than `PASSAGE_CHARS` (4 000) is delivered as the span
around its best window, snapped to whole lines, under the nearest heading
above it, with `[...]` where text was left out. `passage: {start, end, of}`
says so, `text_sha256` covers what was delivered, and `sha256` stays the
file's, so the whole can be fetched by `concept_id`. 4 000 because eight
excerpts of it stay far under a tool response's limit even with several
sub-questions merged, while a 500-character window keeps 3 500 characters of
surroundings. The budget pays for the passage, not the file.
Tests moved with the default, each stated rather than silenced:
- fusion-mechanism tests (cost vocabulary, rarity weight, reservation, shared
rank, the reference-bundle pins) ask for `ranking="fusion"`, the order they
were measured on; the BM25 reading of the reference bundle is a separate
measurement, kept in local state;
- the retrieval gate still measures the shipped default. Row 1 holds. Four of
its premises were built against the fusion (a concept forced below k that
BM25 now delivers, a quota that no longer decides, mutants patching fusion
code) and are `xfail(strict=True)` until the fixtures are re-measured;
- the shipped example payload is regenerated; the shipped skill is unchanged.
README's Consume section and CLAUDE.md state the new default and that the
flags described after it belong to the fusion.
The search gate's table for this commit is kept in local state: the question
sets belong to a consumer whose content does not go on a public mirror.
Suite on a clean tree after `git add`: 2390 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
cf21449ddb
commit
735468f600
11 changed files with 988 additions and 118 deletions
11
README.md
11
README.md
|
|
@ -816,6 +816,17 @@ package, and takes no clock: the same bundle bytes and the same
|
|||
`(question, k, limit, cost_vocabulary, reserve_top_rank, rarity_weight)`
|
||||
produce byte-identical output.
|
||||
|
||||
**The ranking is BM25 since v1.1** (`--ranking bm25`, the default). Two signals
|
||||
are fused by rank: each concept's best 500-character passage, and its title and
|
||||
id path weighted above its body. A word the bundle does not hold weighs nothing
|
||||
by itself; one it holds in another form — a Norwegian inflection or compound —
|
||||
is read as that form. A concept longer than 4 000 characters is delivered as
|
||||
the passage that answers, under the nearest heading above it, marked with
|
||||
`passage: {start, end, of}`, so the whole can be fetched by its `concept_id`.
|
||||
`--ranking fusion` is the older three-signal ranking; the flags below that say
|
||||
they widen a signal (`--cost-vocabulary`, `--rarity-weight`) belong to it and
|
||||
are refused without it. The rest of this section describes the fusion.
|
||||
|
||||
`--cost-vocabulary` is off by default and widens one question class: it lets a
|
||||
declared list of cost/price/quantity terms bridge a question and a document that
|
||||
name money with different words. The gate is the question — one naming no such
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue