History

Kjell Tore Guttormsen 735468f600 feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage C1. `okf consume` and MCP's `okf_ask` now rank with BM25 (`bm25.py`) instead of the three-signal fusion. Two signals, fused by reciprocal rank: - passage: every body cut into 500-character windows every 250, a concept scored by its BEST window -- a narrow question is answered in one place; - field: title three times, the id path and source name twice, then the body -- a broad question is answered by what a section is called. The document prior and the rarity weight are gone from the default: the first favoured big documents full of common words, the second gave its largest weight to a word the collection does not hold. Under BM25 such a word weighs exactly zero. A signal that scores a concept zero adds nothing to it, and ties share a rank, so alphabetical order lifts nothing either. Three rules carried over from the fusion, each with its own test, because the suite showed what BM25 alone lost: - a directory every concept shares is not read (K3-20's defect, one signal on); - a number a section is known by (`4.2`, `10.2-2`) is kept as one token, or a question naming a section by its number matches nothing in it; - a question word the collection does NOT hold is read as the collection's words it shares a leading word with (`consume.tokens_match`) -- Norwegian inflection and compounding -- at that word's idf, never at its own. The lookup and title-covered partitions are shared with the fusion (`_partitioned`). `ranking="fusion"` / `--ranking fusion` keeps the old order reachable; `--cost-vocabulary` and `--rarity-weight` widen only the fusion and are refused with the default (`ranking_flag_conflict`) rather than ignored. C3. A concept longer than `PASSAGE_CHARS` (4 000) is delivered as the span around its best window, snapped to whole lines, under the nearest heading above it, with `[...]` where text was left out. `passage: {start, end, of}` says so, `text_sha256` covers what was delivered, and `sha256` stays the file's, so the whole can be fetched by `concept_id`. 4 000 because eight excerpts of it stay far under a tool response's limit even with several sub-questions merged, while a 500-character window keeps 3 500 characters of surroundings. The budget pays for the passage, not the file. Tests moved with the default, each stated rather than silenced: - fusion-mechanism tests (cost vocabulary, rarity weight, reservation, shared rank, the reference-bundle pins) ask for `ranking="fusion"`, the order they were measured on; the BM25 reading of the reference bundle is a separate measurement, kept in local state; - the retrieval gate still measures the shipped default. Row 1 holds. Four of its premises were built against the fusion (a concept forced below k that BM25 now delivers, a quota that no longer decides, mutants patching fusion code) and are `xfail(strict=True)` until the fixtures are re-measured; - the shipped example payload is regenerated; the shipped skill is unchanged. README's Consume section and CLAUDE.md state the new default and that the flags described after it belong to the fusion. The search gate's table for this commit is kept in local state: the question sets belong to a consumer whose content does not go on a public mirror. Suite on a clean tree after `git add`: 2390 passed, 2 skipped, 4 xfailed. ruff, ruff format, mypy --strict clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-09-21 06:22:56 +02:00
..
example-payload.json	feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage	2026-09-21 06:22:56 +02:00
README.md	feat(consume): parent reaches the reader -- excerpt field, body link, checker rule	2026-09-11 12:36:23 +02:00

Kjell Tore Guttormsen 735468f600 feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage

C1. `okf consume` and MCP's `okf_ask` now rank with BM25 (`bm25.py`) instead
of the three-signal fusion. Two signals, fused by reciprocal rank:

- passage: every body cut into 500-character windows every 250, a concept
  scored by its BEST window -- a narrow question is answered in one place;
- field: title three times, the id path and source name twice, then the body
  -- a broad question is answered by what a section is called.

The document prior and the rarity weight are gone from the default: the first
favoured big documents full of common words, the second gave its largest
weight to a word the collection does not hold. Under BM25 such a word weighs
exactly zero. A signal that scores a concept zero adds nothing to it, and ties
share a rank, so alphabetical order lifts nothing either.

Three rules carried over from the fusion, each with its own test, because the
suite showed what BM25 alone lost:
- a directory every concept shares is not read (K3-20's defect, one signal on);
- a number a section is known by (`4.2`, `10.2-2`) is kept as one token, or
  a question naming a section by its number matches nothing in it;
- a question word the collection does NOT hold is read as the collection's
  words it shares a leading word with (`consume.tokens_match`) -- Norwegian
  inflection and compounding -- at that word's idf, never at its own.

The lookup and title-covered partitions are shared with the fusion
(`_partitioned`). `ranking="fusion"` / `--ranking fusion` keeps the old order
reachable; `--cost-vocabulary` and `--rarity-weight` widen only the fusion and
are refused with the default (`ranking_flag_conflict`) rather than ignored.

C3. A concept longer than `PASSAGE_CHARS` (4 000) is delivered as the span
around its best window, snapped to whole lines, under the nearest heading
above it, with `[...]` where text was left out. `passage: {start, end, of}`
says so, `text_sha256` covers what was delivered, and `sha256` stays the
file's, so the whole can be fetched by `concept_id`. 4 000 because eight
excerpts of it stay far under a tool response's limit even with several
sub-questions merged, while a 500-character window keeps 3 500 characters of
surroundings. The budget pays for the passage, not the file.

Tests moved with the default, each stated rather than silenced:
- fusion-mechanism tests (cost vocabulary, rarity weight, reservation, shared
  rank, the reference-bundle pins) ask for `ranking="fusion"`, the order they
  were measured on; the BM25 reading of the reference bundle is a separate
  measurement, kept in local state;
- the retrieval gate still measures the shipped default. Row 1 holds. Four of
  its premises were built against the fusion (a concept forced below k that
  BM25 now delivers, a quota that no longer decides, mutants patching fusion
  code) and are `xfail(strict=True)` until the fixtures are re-measured;
- the shipped example payload is regenerated; the shipped skill is unchanged.

README's Consume section and CLAUDE.md state the new default and that the
flags described after it belong to the fusion.

The search gate's table for this commit is kept in local state: the question
sets belong to a consumer whose content does not go on a public mirror.

Suite on a clean tree after `git add`: 2390 passed, 2 skipped, 4 xfailed.
ruff, ruff format, mypy --strict clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-09-21 06:22:56 +02:00

example-payload.json

feat(consume): BM25 ranking by passage and title, a large concept delivered as its passage

2026-09-21 06:22:56 +02:00

README.md

feat(consume): parent reaches the reader -- excerpt field, body link, checker rule

2026-09-11 12:36:23 +02:00

README.md

References

example-payload.json is a real payload, not an illustration, and ../SKILL.md is the skill generated for the same bundle. Both come from the three-concept golden bundle that ships in this repository, examples/ingest-golden-segmented-okf-v0-2/expected-bundle, so anyone reading this file can regenerate them byte for byte and compare. Neither carries content from any corpus.

Regenerate them from the repository root rather than editing either file, with okf being this checkout's own install (for example .venv/bin/okf):

okf skill examples/ingest-golden-segmented-okf-v0-2/expected-bundle \
  --out skills/okf-consume --force --example-question "Hva sier veiledningen om krav?"
python3 -c 'import os, pathlib; p = pathlib.Path("skills/okf-consume/SKILL.md"); p.write_text(p.read_text(encoding="utf-8").replace(os.path.realpath(".") + "/", ""), encoding="utf-8")'
okf check --skill skills/okf-consume/SKILL.md \
  --payload skills/okf-consume/references/example-payload.json

Why each part is there:

--force: the generator refuses to replace an existing SKILL.md (refused (target_occupied)), because a silent overwrite would destroy a hand-edited copy.
--example-question: without it the question is derived from the bundle's own titles and the payload is a different one. tests/test_okf_consume.py asserts this payload byte for byte against the pre-pass for exactly Hva sier veiledningen om krav?, so the question is part of what the file is.
The python3 line: okf skill writes the bundle root and the skill's own path absolute when --out is not under .claude/skills/, and this directory is not. Shipped as generated, the two commands in SKILL.md would name one checkout on one machine. The line strips that checkout's prefix and nothing else, and a test holds the shipped SKILL.md to the generator's output with exactly that prefix removed.
okf check should report conformant: 17 rules over 3 excerpts and 0 withheld entries, 0 findings and exit 0.

The generated name is b-golden-segmented-okf-v0-2-consume while this directory stays skills/okf-consume/. Claude Code takes a project or personal skill's command from its directory name and uses name as a display label, so a copy placed at .claude/skills/okf-consume/ is still /okf-consume.

The payload is here so that its shape can be read without running anything, and so that a reader can see what the members the contract does not name look like in practice: text and text_sha256 on every excerpt (§ 8 permits additional members; § 1 defines an excerpt as delivered content, and without a body the budget gate would measure a skeleton), rank, bundle_id_inherited, and the raw_bytes/encoding_delta pair that gives the known-positive a second, independent check.