open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): weight a lexical hit by its rarity, off by default and measured

Browse source 
O2b asked whether a requirement number can be made worth more than a common
word by weighting each hit with the token's rarity in the bundle, with no
hand-set constant and no declared token class. It can, on one of the three
bundles, and the two it cannot are decomposed rather than guessed.

The rule is log(N/df) over the concepts' own tokens, counted with the same
four-character prefix rule a hit is scored with. It enters the RANKING and
never the GATE: `lexical` stays a count, because `krav` weighs exactly 0 on
all three bundles and a weighted gate would drop every concept matching only
that word -- which is the gate 54a0bc2 falsified for other reasons. One df
table per bundle reaches every stage that scores the question against text,
including the document prior. One pass, 0.241 s over 1 133 concepts.

Measured on four corpora, before and after, with every published figure
reproduced first: gold fused rank 96 -> 103, 9 -> 8 (withheld -> DELIVERED at
rank 8) and 35 -> 35; K2's priced sheet candidate rank 10 -> 2 with the cost
vocabulary and 251 -> 78 without; Q-good unmoved at rank 1; hit@8 5 of 6 with
every rank identical; the S7 control payload byte-identical on the default
command.

DEFAULT OFF, decided by the number and not by taste: it does not win on all
four, because N100's gold loses seven rank positions. Off means the bytes that
were already published, and that is measured -- 8 of 8 payload digests
identical against a frozen copy of 56c1205 built with git archive.

Two limits, both someone else's mechanism and both named: MIN_SHARED_PREFIX=4
makes a unique identifier read as 135-of-446 common on N100, so the weight
correctly ranks a common adjective above the exact requirement number; and RRF
consumes RANKS only, so on N500 -- where the gold already leads the one signal
that can see the identifier, and the other two cannot see it at all -- no
weighting inside a signal can move anything.

Consumption-side only, so no rebuild: the K2 bundle ref 2f82fcfe... stands.

Report: docs/2026-09-08-sjeldenhetsvekt.md. 13 new tests, red first; 8
mutations, 8 red, two of them only after the survivors were read as code -- one
exposed a fixture that put the identifier where the real corpus does not, and
the corrected fixture is what found the RRF limit. Suite 1295 -> 1308.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 12:36:57 +02:00
commit 116d3e1007
5 changed files with 755 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

16
CLAUDE.md
View file

@ -268,8 +268,8 @@ and fixtures, never code.
measurement behind it, including the control that FAILED, is measurement behind it, including the control that FAILED, is
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this `docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
repository's own choice** — the contract binds a payload, not a retrieval repository's own choice** — the contract binds a payload, not a retrieval
algorithm (§ 10) — and it has TWO optional widenings, both **off by default** algorithm (§ 10) — and it has THREE optional widenings, all **off by default**
and both keeping the default payload byte-identical. `--cost-vocabulary`: a and all keeping the default payload byte-identical. `--cost-vocabulary`: a
declared cost/price/quantity vocabulary family that bridges a question and a declared cost/price/quantity vocabulary family that bridges a question and a
document naming money with different words, gated on the QUESTION carrying document naming money with different words, gated on the QUESTION carrying
such a term, so a question without one is byte-identical either way. It moves such a term, so a question without one is byte-identical either way. It moves
@ -286,6 +286,18 @@ and fixtures, never code.
the caller's decision, because deriving a limit from the corpus was measured the caller's decision, because deriving a limit from the corpus was measured
and falsified — two defensible derivations, 49x apart, one of them breaking and falsified — two defensible derivations, 49x apart, one of them breaking
the known-positive. `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`. the known-positive. `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
`--rarity-weight` is the third: each lexical hit weighs `log(N/df)` over the
bundle's own concepts instead of 1, so an identifier is not worth what a
common verb is worth. It enters the RANKING and never the GATE — `lexical`
stays a count, because a word every concept carries weighs exactly 0 and a
weighted gate is what `54a0bc2` falsified. Off by default BY MEASUREMENT: it
delivers one of three requirement lookups and takes a priced sheet from
candidate rank 10 to 2, leaves one gold unmoved and costs another seven rank
positions. Two limits are decomposed rather than guessed, and both are
someone else's mechanism: `MIN_SHARED_PREFIX = 4` makes a unique identifier
read as 135-of-446 common, and RRF consumes RANKS, so no weighting inside a
signal can move a gold that already leads it.
`docs/2026-09-08-sjeldenhetsvekt.md`.
## Workflow ## Workflow

15
README.md
View file

@ -121,8 +121,8 @@ cuts them to a bounded set and emits one payload. It decides nothing about the
question; the skill that reads the payload does the judgement. It calls no question; the skill that reads the payload does the judgement. It calls no
model, opens no socket, imports nothing outside the standard library and this model, opens no socket, imports nothing outside the standard library and this
package, and takes no clock: the same bundle bytes and the same package, and takes no clock: the same bundle bytes and the same
`(question, k, limit, cost_vocabulary, reserve_top_rank)` produce `(question, k, limit, cost_vocabulary, reserve_top_rank, rarity_weight)`
byte-identical output. produce byte-identical output.
`--cost-vocabulary` is off by default and widens one question class: it lets a `--cost-vocabulary` is off by default and widens one question class: it lets a
declared list of cost/price/quantity terms bridge a question and a document that declared list of cost/price/quantity terms bridge a question and a document that
@ -141,6 +141,17 @@ pre-exclusion, never before — and the payload then declares
24 measured combinations, both of them that eviction: 24 measured combinations, both of them that eviction:
`docs/2026-09-08-blindsone-laas2-budsjett-k2.md`. `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
`--rarity-weight` is off by default and weights each lexical hit by
`log(N/df)` over the bundle's own concepts instead of counting it as one, so a
requirement number is not worth what a common verb is worth. The default being
off is a measurement rather than a preference: on four corpora it took one gold
concept from withheld to delivered and a priced sheet from candidate rank 10 to
2, left one gold rank unmoved, and cost another seven rank positions — because
the four-character prefix matcher makes a unique identifier read as
135-of-446 common on that bundle. Where it cannot help is decomposed rather
than guessed: RRF fuses RANKS, so a weight moves nothing on a signal the gold
already leads. `docs/2026-09-08-sjeldenhetsvekt.md`.
It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a It emits the § 8 shape — `contract`, `bundle` (`bundle_id` plus a
`sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a `sha256-tree:` content identity), `budget` (unit, instrument, limit, spent and a
validated known-positive), `denominators`, `excerpts` and `withheld` — and every validated known-positive), `denominators`, `excerpts` and `withheld` — and every

287
docs/2026-09-08-sjeldenhetsvekt.md Normal file
View file

@ -0,0 +1,287 @@
# The rarity weight: what it delivered, what it cannot reach, 2026-09-08
The previous session made a requirement number VISIBLE to the consumption
pre-pass and could not make it WORTH more than a common word: `_overlap`
returns a count, so `10.2-2` scored exactly what `krever` scored, and the gold
concept stayed `below_k` on all three bundles
(`docs/2026-09-08-kravnummer-tokenisering.md` § 8). This session weights each
hit by the token's rarity in the bundle and measures what that buys.
It buys one of the three requirement lookups and a large move on K2. It does
not buy the other two, and the reason is structural rather than a matter of
tuning. Both halves are below, with the decomposition.
---
## 0. What IS measured, and what is NOT
**Measured.** Nine known-positive figures reproduced before anything was
changed; the document frequency of every question token on all three bundles,
under the matcher and under exact equality; the gold's fused rank and its
per-signal ranks, before and after, on three bundles; the delivered/withheld
verdict and payload digest per bundle; every K2 control the order named, both
values; the six hit@8 questions; the `df` pass's cost; eight mutations.
**Known-positive, run first.** Every figure this session builds on was
reproduced on this machine, with the committed tool, before any after-column
was read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | gold fused rank per bundle | previous session § 4 | **96, 9, 35** |
| 2 | concepts sharing the gold's lexical score on N500 | previous session § 8 | **70 of 270** |
| 3 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
| 4 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
| 5 | K2 S7 control, two commands | previous session § 5 | `2ae46f68…` 169 573 B, `8aaf8558…` 172 588 B |
| 6 | priced-sheet candidate rank, four rows, O1 bundle | O1 § 7 | 251 / 19 / 1 / 1 |
| 7 | the same four rows on the rebuilt bundle | previous session § 7 | 251 / **10** / 1 / 1 |
| 8 | K2 delivery costs, four commands | previous session § 7 | 169 583 / 183 798 / 190 438 / 108 877 B |
| 9 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6** |
**NOT measured.** Two things, both named rather than glossed:
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
machine, and this repository's single-runtime-dependency rule does not admit
one to produce a caption. Every figure below is therefore in the unit the
budget gate itself uses — `BUDGET_UNIT`, "utf-8 bytes of emitted JSON" — plus
the payload `sha256`, which is exact where a byte count is only indicative.
The previously published o200k figures are not restated as if re-measured.
- **Whether a live model answers better.** This measures a *ranking*.
"Delivered" is not "read"; that measurement belongs to the consumer.
---
## 1. The rule, and why it has no constant in it
`rarity_weights(question_tokens, corpus)` returns, per question token,
```
w(t) = log(N / df(t))
```
`N` is the number of concepts in the bundle. `df(t)` is how many of them bear
the token, counted with the **same** four-character prefix rule a hit is scored
with. Nothing is set by hand: a word every concept carries weighs `log(1) = 0`
of itself, and an identifier one concept carries takes the corpus's maximum of
itself. No token class is declared anywhere.
Three decisions inside that sentence, each of them a decision:
1. **`df` is counted over the text a hit can be scored on** — title, id and
body joined, the two lexical signals' union. Counting rarity over one field
and matching on another would weight a token by how rare it is somewhere it
is never read.
2. **The weight enters the RANKING and never the GATE.** `lexical` stays a
count when weights are supplied, at the price of one more pass over the same
two fields. This is load-bearing: `krav` weighs exactly 0 on all three
bundles, so a concept matching only `krav` would have a weighted sum of 0.0
and would fall to `no_lexical_match` — turning a ranking change into the
*gate* that `54a0bc2` falsified.
3. **One `df` table per bundle, reaching every stage that scores the question
against text**, including the document prior. A weight applied to two of
three fused signals would be a second ranker rather than one statistic.
**Why this is not the sweep `54a0bc2` refuted.** That order asked for smoothed
IDF as a **port**: a threshold below which a concept is withheld, required to
zero both known-negative questions. The sweep found the frontier empty — `neg2`
first reaches 0 at `T = 4.6`, by which point gold-hit reachability has already
fallen to 0 of 6 — and the order was returned rather than implemented. That
result stands and is not re-litigated. This is the other use: an ordering
*inside* the candidate set, with the gate untouched. A ranking cannot withhold
anything, so the failure mode that refuted the gate has no counterpart here.
**The pollution `54a0bc2` § 1 named is inherited deliberately**, and § 3 below
shows it doing exactly the damage that section predicted.
**Cost.** One pass. Measured: **0.062 s** over 446 concepts, **0.241 s** over
1 133, **0.073 s** over 270 — inside the order's one-second bound on the
largest bundle by a factor of four.
---
## 2. The three requirement lookups, before and after
Same instrument before and after: the gold's position in the fused `ranked`
list, and the payload's own verdict from `build_payload`. Command:
```
python3 tools/okf_consume.py <bundle> \
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
[--rarity-weight]
```
| | N100:2023 | N200:2024 | N500:2024 |
|---|---|---|---|
| concepts | 446 | 1 133 | 270 |
| gold fused rank, weight **off** | 96 | 9 | 35 |
| gold fused rank, weight **on** | **103** | **8** | **35** |
| gold verdict, off | `below_k` | `below_k` | `below_k` |
| gold verdict, on | `below_k` | **delivered, rank 8** | `below_k` |
| payload bytes, off → on | 61 566 → 61 452 | 142 861 → **146 641** | 40 626 → 38 869 |
| payload `sha256`, off | `49e9c718…` | `adc2eac7…` | `21c5bbfe…` |
| payload `sha256`, on | `ba09a56a…` | `403ee989…` | `1758f6ca…` |
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
**One of three delivers.** N200's gold moves from withheld position 204 of
1 125 to delivered at rank 8. N500 does not move at all. N100 moves the wrong
way, by seven positions.
---
## 3. Why N100 got worse: the matcher makes the identifier common
`df` under the matcher, against `df` under exact token equality:
| bundle | identifier | `df` under the matcher | `df` exact | `w(identifier)` | `w('sentrale')` |
|---|---|---|---|---|---|
| N100 | `3.3.1-13` | **135 of 446** | 1 | 1.195 | **3.798** |
| N200 | `2.9.2-12` | 17 of 1 133 | 1 | 4.199 | 4.835 |
| N500 | `10.2-2` | 5 of 270 | 1 | 3.989 | 3.519 |
The identifier is unique in all three bundles. It is *common* in N100 only
because `tokens_match` compares four leading characters, so `3.3.1-13` matches
every requirement number beginning `3.3.` — 135 of them. The weight then does
what it was asked to do and ranks a common adjective (`sentrale`, borne by 10
of 446) as more informative than the exact requirement number.
This is `54a0bc2` § 1 — "`df` measured over the colliding matcher measures
collision breadth, not rarity" — measured on the identifier itself rather than
on `brann*`. It is not a defect in the weight; it is the weight faithfully
reporting a `df` the matcher produced. Fixing it means changing the matcher,
which is a different change with its own measurement, and it is not attempted
here.
---
## 4. Why N500 did not move: RRF consumes ranks, and the gold already led
The gold's rank on each of the three fused signals, and how many concepts each
signal reorders:
| bundle | signal ranks, off | signal ranks, on | positions moved per signal |
|---|---|---|---|
| N100 | 39 / 186 / 127 | 42 / 184 / 127 | 122 / 367 / **0** of 446 |
| N200 | 8 / 4 / 212 | **4** / 10 / 212 | 37 / 277 / **0** of 1 133 |
| N500 | **1** / 137 / 74 | **1** / 137 / 74 | **0** / 127 / **0** of 270 |
On N500 the gold is **already rank 1** on the title-and-id signal, by counting,
before any weighting. Weighting raises its score from 3.000 to 3.989 and its
rank from 1 to 1. RRF was chosen precisely because it **consumes ranks only**
(`RRF_K`'s comment says so), which makes it invariant to any monotone transform
of a signal — so the whole increase is invisible to the fusion. What holds the
gold at 35 is the other two signals, and neither can see the identifier at all:
the body repeats `krav` and nothing else, and the document prior is a
two-document density whose order the weight never changes on any of the three
bundles (0 positions moved, three times).
**The general statement, and it is a limit rather than a complaint:** a rarity
weight can only move a fused rank where it reorders some signal. Where the
identifier appears in exactly one field and already leads that field, no
weighting scheme inside the signals can do anything. Closing N500 means
changing the fusion — giving a signal a weight, or letting a strong single
signal carry more than 1/(60+1) — and that is a different mechanism from this
one. It is named here and not built.
The same shape is in the test suite, twice, so neither half can quietly stop
being true: a synthetic corpus where the identifier is in the title alone
(fused rank 18 → 18, title-signal rank 1 → 1) and one where weighting reorders
the title signal from last to first (30 → 1) and moves the fused rank only from
30 to 18.
---
## 5. K2: every control holds, and the priced sheet moves 10 → 2
| control | weight off | weight on | verdict |
|---|---|---|---|
| Q-good (`Hvordan skal prisene fylles ut?`), candidate rank | 1 | **1** | holds |
| Q-good with `--cost-vocabulary` | 1 | **1** | holds |
| priced sheet, mandate question, vocabulary off | 251 of 271 | **78 of 271** | improves |
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **2 of 280** | improves |
| priced sheet, delivered by `--cost-vocabulary --k 12` | rank 10 | **rank 2** | improves |
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank identical | holds |
| S7 control question, default command | `2ae46f68…`, 169 573 B | **byte-identical** | holds |
| S7 control question, `--cost-vocabulary --k 12 --limit 160000` | `8aaf8558…`, 172 588 B | `b205a618…`, 183 462 B | **moves, +10 874 B** |
No named K2 control falls. The one figure that moves is a payload the flag
changes on purpose: with the weight on, that command delivers a different set
of excerpts and costs 6.3 % more bytes.
---
## 6. The decision, made by the number
The order's rule: default ON if the weight wins on all four corpora with no K2
control falling; behind a flag otherwise, naming what fell and by how much.
**It does not win on all four. N100's gold fused rank moves 96 → 103, a loss of
seven positions**, on a question where the gold is not delivered either way. So
the weight ships as `--rarity-weight`, **off by default**.
Off means the bytes that were already published, and that is measured rather
than asserted. Eight payloads — three N-bundle requirement lookups, three
commands on the rebuilt K2 bundle, two on the O1 bundle — were built from a
frozen copy of `56c1205` (`git archive` into a scratch tree, `PYTHONPATH` and
`__file__` both checked) and from this working tree with the flag omitted:
**8 of 8 payload digests identical.** No consumer's bytes move because this
change exists.
What the flag buys the caller who sets it, in one line each: N200's requirement
lookup goes from withheld to delivered; K2's priced sheet goes from candidate
rank 10 to 2; N100 and N500 are unchanged in verdict and worse and equal in
rank respectively.
---
## 7. What did NOT close
- **Two of the three requirement lookups are still `below_k`**, and the target
the order set — rank ≤ 8 on all three — is **not met**. It is met on N200.
- **N500 cannot be closed by any weighting inside the signals** (§ 4). The next
mechanism is a change to the fusion, not to the weight, and it has no
measurement behind it yet.
- **N100 needs the matcher, not the weight** (§ 3). `MIN_SHARED_PREFIX = 4`
makes a unique identifier read as 135-of-446 common. That is the same defect
`54a0bc2` § 1 named a year of sessions ago in another form, and it is now
measured on the case it costs most.
- **The 34 % skewed deletion in the orphan gate** is untouched; different axis.
- **"Delivered" is not "read".** N200's gold is now in the payload. Whether a
model then answers the engineer's question is the consumer's measurement.
---
## 8. Tests, mutations, gates
Thirteen new tests, red before the rule existed (6 of them failed on
`AttributeError` alone, and the known-positive — the gold held at rank 18 of 30
by counting — passed from the start, so the red was the rule's absence and not
a broken fixture).
**Eight mutations, eight red.** Two survived the first pass and neither was
banded blind:
- **M5** (the document prior stops receiving the weights) survived because on
all three real bundles it is behaviourally equivalent — the two-document
density is never reordered. The commitment "one statistic reaches every
stage" is real even where no output distinguishes it, so it is now asserted
at the call site instead of on an output.
- **M7** (the title signal stops being weighted) survived because the fixture
put the identifier in the body as well. The real corpus does not: the gold's
title answers three question tokens and its body answers one. The fixture was
wrong, was corrected to the measured shape — and the corrected fixture then
reproduced the real failure of § 4, which is how § 4 was found at all.
| gate | result |
|---|---|
| `pytest -q` | **1 308 passed** (1 295 before) |
| `ruff check src tests tools` | clean |
| `ruff format --check` | clean, 88 files |
| `mypy --strict src/` | clean, 17 files |
| `okf_contract_check` on the three weighted payloads | exit 0, 14 rules, 0 findings |
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
**No rebuild.** The change is consumption-side only, confirmed by
`git diff --stat`: no production module is touched, so the K2 bundle ref
`sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b`
is unchanged.

304
tests/test_okf_consume.py
View file

@ -1116,6 +1116,7 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-prisform-og-loggen-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-prisform-og-loggen-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-kravnummer-tokenisering.md", PROJECT_ROOT / "docs" / "2026-09-08-kravnummer-tokenisering.md",
PROJECT_ROOT / "docs" / "2026-09-08-sjeldenhetsvekt.md",
PROJECT_ROOT / "README.md", PROJECT_ROOT / "README.md",
PROJECT_ROOT / "CLAUDE.md", PROJECT_ROOT / "CLAUDE.md",
] ]
@ -1589,3 +1590,306 @@ def test_the_cli_exposes_the_reservation_and_omitting_it_reproduces_the_default_
assert plain.stdout == okf_consume.serialise(_eviction_payload(root)) assert plain.stdout == okf_consume.serialise(_eviction_payload(root))
assert reserved.stdout == okf_consume.serialise(_eviction_payload(root, reserve_top_rank=True)) assert reserved.stdout == okf_consume.serialise(_eviction_payload(root, reserve_top_rank=True))
assert plain.stdout != reserved.stdout assert plain.stdout != reserved.stdout
# --- Step 12: the rarity weight (O2b) -----------------------------------------
#: The situation the previous session measured and could not close, recreated
#: small: every concept carries the common word, exactly one carries the
#: identifier, and the identifier is worth the same as the common word because
#: `_overlap` counts. Synthetic rather than borrowed from a bundle, because a
#: fixture that is a corpus measures that corpus.
RARITY_QUESTION = "Hva krever Krav 10.2-2 i N500? Gjengi det sentrale vilkåret."
RARITY_GOLD = "krav/c-29"
def _synthetic(concept_id: str, title: str, body: str) -> okf_consume.Concept:
return okf_consume.Concept(
path=Path(concept_id),
concept_id=concept_id,
bundle_id="rarity-fixture",
bundle_id_inherited=False,
sha256="0" * 64,
okf_type="Krav",
title=title,
source_file="synthetic.md",
adjudication="unknown",
adjudication_present=False,
frontmatter={},
body=body,
)
def _rarity_corpus(*, identifier_in_body: bool) -> list[okf_consume.Concept]:
"""29 concepts bearing the common words, one bearing the identifier.
The gold's `concept_id` sorts LAST, so nothing but the score can lift it:
on a tie the declared tie-break puts it at the bottom.
`identifier_in_body` is the whole difference between the two shapes the
real corpora turned out to have, and it decides whether the weight can do
anything at all -- see the two tests below.
"""
common = "Kravet i N500 gjengir det sentrale vilkåret for anlegget."
concepts = [
_synthetic(f"krav/c-{index:02d}", f"Krav 3.{index}-1 Alminnelig bestemmelse", common)
for index in range(29)
]
body = (
"Krav 10.2-2 stiller vilkår om anlegget."
if identifier_in_body
else "Tekniske bygg stiller vilkår om anlegget."
)
concepts.append(_synthetic(RARITY_GOLD, "Krav 10.2-2 Tekniske bygg", body))
return concepts
def _rank_of(ranked: list[tuple[okf_consume.Concept, float, int]], concept_id: str) -> int:
return [concept.concept_id for concept, _, _ in ranked].index(concept_id) + 1
def _signal_rank(
concepts: list[okf_consume.Concept], question: str, gold: str, weights: object
) -> int:
"""Where the gold sits on the title-and-id signal alone, by the ranker's own rule."""
tokens = okf_consume.normalise(question)
scores = {
concept.concept_id: okf_consume._overlap(
tokens,
f"{concept.title} {concept.concept_id.replace('/', ' ')}",
weights=weights, # type: ignore[arg-type]
)
for concept in concepts
}
order = sorted(scores, key=lambda key: (-scores[key], key))
return order.index(gold) + 1
def test_counting_leaves_the_identifier_worth_no_more_than_the_common_word() -> None:
# The known-positive for the weight: without it, a rank of 1 with the
# weight on would prove nothing, because nothing would have been shown to
# hold the gold down in the first place.
for identifier_in_body in (False, True):
concepts = _rarity_corpus(identifier_in_body=identifier_in_body)
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {})
assert _rank_of(ranked, RARITY_GOLD) == 18
lexical = {concept.concept_id: value for concept, _, value in ranked}
assert lexical["krav/c-00"] == 6
def test_weighting_a_hit_by_its_rarity_lifts_the_identifier_into_the_cut() -> None:
# The arm where the weight can act: the identifier is in the body too, so
# weighting REORDERS the body signal. This is the shape N200 and the K2
# price sheet have, and both moved (withheld -> delivered rank 8; candidate
# rank 10 -> 2).
concepts = _rarity_corpus(identifier_in_body=True)
weights = okf_consume.rarity_weights(
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
)
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
assert _rank_of(ranked, RARITY_GOLD) <= 8
def test_a_rarity_weight_cannot_move_a_signal_the_gold_already_leads() -> None:
"""The falsification, kept as a test so it cannot quietly stop being true.
RRF consumes RANKS ONLY -- that is the documented reason it was chosen. So
a weight changes the fused order only where it changes some signal's ORDER.
When the identifier is in the title alone, the gold already leads the title
signal by counting, the weight makes its lead larger and its RANK identical,
and the two signals that hold it down cannot see the identifier at all.
Measured on the real bundles: N500 35 -> 35, N100 96 -> 103.
"""
concepts = _rarity_corpus(identifier_in_body=False)
weights = okf_consume.rarity_weights(
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
)
# The weight DOES do its job on the score: the gold's title is worth more
# than every other title, by more than counting made it worth.
plain = okf_consume._overlap(
okf_consume.normalise(RARITY_QUESTION), f"{concepts[-1].title} krav c-29"
)
lifted = okf_consume._overlap(
okf_consume.normalise(RARITY_QUESTION), f"{concepts[-1].title} krav c-29", weights=weights
)
assert lifted > plain
# And the rank it buys is the rank it already had, so the fusion sees none of it.
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, None) == 1
assert _signal_rank(concepts, RARITY_QUESTION, RARITY_GOLD, weights) == 1
ranked = okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
assert _rank_of(ranked, RARITY_GOLD) == 18
def test_the_weight_leaves_the_lexical_count_a_count_so_the_cut_is_untouched() -> None:
# SS: the gate is a different axis from the ranking, and a token every
# concept carries weighs exactly zero. Were `lexical` the weighted sum, a
# concept matching only that token would become `no_lexical_match` -- the
# gate `54a0bc2` falsified rarity weighting FOR.
concepts = _rarity_corpus(identifier_in_body=True)
weights = okf_consume.rarity_weights(
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
)
plain = {
c.concept_id: v for c, _, v in okf_consume.concept_scores(concepts, RARITY_QUESTION, {})
}
weighted = {
c.concept_id: v
for c, _, v in okf_consume.concept_scores(concepts, RARITY_QUESTION, {}, weights=weights)
}
assert plain == weighted
def _equal_frequency_corpus() -> list[okf_consume.Concept]:
"""16 concepts over three words, each word borne by exactly 8 of them.
Equal `df` is the condition under which the weight provably cannot reorder
anything: every hit is multiplied by the same positive number, and RRF
consumes ranks. Concepts carry one word or all three, so the order being
preserved is a real order and not a single tie.
"""
words = ("vilkåret", "anlegget", "kravet")
bodies = [" ".join(words)] * 4 + [words[0]] * 4 + [words[1]] * 4 + [words[2]] * 4
return [
_synthetic(f"felles/e-{index:02d}", f"Bestemmelse {index}", f"Denne teksten nevner {body}.")
for index, body in enumerate(bodies)
]
def test_a_question_of_equally_common_words_comes_back_in_the_order_it_had() -> None:
concepts = _equal_frequency_corpus()
question = "Hva sier vilkåret om anlegget og kravet?"
texts = okf_consume.searchable_text(concepts)
weights = okf_consume.rarity_weights(okf_consume.normalise(question), texts)
# The three borne words weigh the same; `hva` and `sier` are borne by no
# concept at all, and their weight is never consumed because they never hit.
assert len({round(weights[word], 12) for word in ("vilkåret", "anlegget", "kravet")}) == 1
plain = [c.concept_id for c, _, _ in okf_consume.concept_scores(concepts, question, {})]
weighted = [
c.concept_id
for c, _, _ in okf_consume.concept_scores(concepts, question, {}, weights=weights)
]
assert weighted == plain
assert len(set(plain)) == 16
def test_a_word_every_concept_carries_weighs_exactly_nothing() -> None:
concepts = _rarity_corpus(identifier_in_body=True)
weights = okf_consume.rarity_weights(
okf_consume.normalise(RARITY_QUESTION), okf_consume.searchable_text(concepts)
)
assert weights["krav"] == 0.0
assert weights["10.2-2"] > weights["n500"] > 0.0
def test_a_question_token_no_concept_carries_is_weighted_but_never_consumed() -> None:
concepts = _rarity_corpus(identifier_in_body=True)
weights = okf_consume.rarity_weights(("kabelbroer",), okf_consume.searchable_text(concepts))
assert weights["kabelbroer"] > 0.0
assert okf_consume._overlap(("kabelbroer",), concepts[0].body, weights=weights) == 0.0
def test_the_weight_is_computed_from_the_bundle_and_not_from_a_constant() -> None:
# Same question, two corpora, two different weights for the same token:
# the number comes from the bundle in hand or it comes from nowhere.
small = okf_consume.rarity_weights(("n500",), ["N500 gjelder", "noe helt annet"])
large = okf_consume.rarity_weights(("n500",), ["N500 gjelder"] + ["noe helt annet"] * 9)
assert large["n500"] > small["n500"]
def test_the_weight_is_off_by_default_and_the_default_payload_is_unmoved() -> None:
# The decision recorded as a test rather than as a sentence: measured on
# four corpora the weight delivers one gold and costs another seven rank
# positions, so it ships OFF, and OFF has to mean the bytes that were
# already published.
plain = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
explicit = okf_consume.build_payload(
FIXTURE, question="Hvordan skal prisene fylles ut?", rarity_weight=False
)
assert okf_consume.serialise(plain) == okf_consume.serialise(explicit)
def test_the_cli_exposes_the_weight_and_omitting_it_reproduces_the_default_bytes() -> None:
question = "Hva krever Krav 10.2-2 om sentrale vilkår?"
plain = _run(str(FIXTURE), "--question", question)
weighted = _run(str(FIXTURE), "--question", question, "--rarity-weight")
assert plain.returncode == 0, plain.stderr
assert weighted.returncode == 0, weighted.stderr
assert plain.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question)
)
assert weighted.stdout == okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, rarity_weight=True)
)
def test_the_weight_reaches_the_document_prior_and_not_only_the_concept_signals() -> None:
# One df table for the bundle, used wherever the question is scored against
# its text. A weight applied to two of three fused signals would be a
# second ranker rather than one statistic.
question = "Hvordan skal prisene fylles ut?"
weights = okf_consume.rarity_weights(
okf_consume.normalise(question), okf_consume.searchable_text(_fixture_concepts())
)
plain = okf_consume.document_scores(FIXTURE, question)
weighted = okf_consume.document_scores(FIXTURE, question, weights=weights)
assert plain.keys() == weighted.keys()
assert plain != weighted
def test_build_payload_hands_the_same_weights_to_the_document_prior(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""ONE df table for the bundle, reaching every stage that scores text.
Asserted on the call rather than on an output, and that is the point: on
all three real bundles the document prior is a two-document density whose
ORDER the weight does not change, so a version passing the weights to the
concept signals alone ranks identically there -- measured, 103/8/35 either
way. The commitment is still that one statistic reaches every stage, and a
commitment no output can distinguish has to be checked where it is made.
"""
seen: list[object] = []
original = okf_consume.document_scores
def spy(*args: object, **kwargs: object) -> dict[str, float]:
seen.append(kwargs.get("weights"))
return original(*args, **kwargs) # type: ignore[arg-type]
monkeypatch.setattr(okf_consume, "document_scores", spy)
question = "Hvordan skal prisene fylles ut?"
okf_consume.build_payload(FIXTURE, question=question)
okf_consume.build_payload(FIXTURE, question=question, rarity_weight=True)
assert seen[0] is None
expected = okf_consume.rarity_weights(
okf_consume.normalise(question), okf_consume.searchable_text(_fixture_concepts())
)
assert seen[1] == expected
def test_the_weight_reaches_the_title_signal_and_reorders_it() -> None:
# The gold's title answers TWO question tokens and every other title
# answers THREE, so counting puts the gold behind all of them; only the
# rarity of the identifier can turn that around, and only if the weight
# reaches the title-and-id signal. Bodies are identical, so the body signal
# decides nothing. This is the N200 shape, where the gold's title-signal
# rank moved 8 -> 4 and the gold went from withheld to delivered.
body = "Denne bestemmelsen gjelder for anlegget."
concepts = [
_synthetic(f"krav/t-{index:02d}", f"Krav 3.{index}-1 N500 gjengir bestemmelsen", body)
for index in range(29)
]
concepts.append(_synthetic("krav/t-29", "Krav 10.2-2 Tekniske bygg", body))
question = "Hva krever Krav 10.2-2 i N500? Gjengi det sentrale vilkåret."
weights = okf_consume.rarity_weights(
okf_consume.normalise(question), okf_consume.searchable_text(concepts)
)
assert _signal_rank(concepts, question, "krav/t-29", None) == 30
assert _signal_rank(concepts, question, "krav/t-29", weights) == 1
plain = okf_consume.concept_scores(concepts, question, {})
weighted = okf_consume.concept_scores(concepts, question, {}, weights=weights)
# Fused: 30 -> 18. The title signal is reordered from last to first and the
# fused rank moves by twelve, not to one -- the other two signals still
# cannot see the identifier. That gap IS the finding of this session, and
# the numbers are here so a change to either half shows up as a diff.
assert _rank_of(plain, "krav/t-29") == 30
assert _rank_of(weighted, "krav/t-29") == 18

153
tools/okf_consume.py
View file

@ -34,6 +34,7 @@ from __future__ import annotations
import argparse import argparse
import hashlib import hashlib
import json import json
import math
import re import re
import sys import sys
import unicodedata import unicodedata
@ -642,14 +643,80 @@ def question_uses_cost_vocabulary(question: str) -> bool:
return any(in_cost_vocabulary(token) for token in normalise(question)) return any(in_cost_vocabulary(token) for token in normalise(question))
def searchable_text(concepts: Sequence["Concept"]) -> list[str]:
"""The text a concept is scored against, one string per concept.
The same two fields the ranker's two lexical signals read -- title plus
id, and body -- joined, so a `df` counted here is a `df` over exactly what
a hit can be scored on. Counting rarity over one field and matching on
another would weight a token by how rare it is somewhere it is not read.
"""
return [
f"{concept.title} {concept.concept_id.replace('/', ' ')} {concept.body}"
for concept in concepts
]
def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dict[str, float]:
"""What one hit on each question token is worth, from the bundle alone.
`log(N / df)`: `N` concepts, and `df` the number of them bearing the token
under the SAME prefix rule a hit is scored with. No constant is set by
hand and no class of token is declared anywhere -- a word every concept
carries weighs exactly `log(1) == 0` of itself, and an identifier one
concept carries takes the corpus's maximum of itself.
**What this is NOT.** `54a0bc2` swept smoothed IDF as a GATE -- a threshold
below which a concept is withheld -- and falsified it: no threshold zeroed
both known-negatives while any positive question still reached its gold
document. That result stands and is not re-litigated here. This is the
other use: an ordering inside the candidate set, with the gate untouched
and `lexical` still a count. A ranking cannot withhold anything, so the
failure mode that refuted the gate has no counterpart here.
**`df` is measured over the colliding matcher, and so measures collision
breadth as well as rarity** (`54a0bc2` § 1: every `brann*` compound shares
four leading characters). Inherited deliberately rather than fixed here:
the weight must agree with the matcher it weights, and changing the matcher
is a different change with its own measurement.
One pass over the corpus. A token borne by no concept takes the weight of
a token borne by one -- it is never consumed, because a token that matches
nothing is never a hit.
"""
total = len(corpus)
if total == 0:
return {token: 0.0 for token in question_tokens}
counts = {token: 0 for token in question_tokens}
for text in corpus:
candidate_tokens = normalise(text)
for token in counts:
if any(tokens_match(token, other) for other in candidate_tokens):
counts[token] += 1
return {
token: math.log(total / count) if count else math.log(total)
for token, count in counts.items()
}
def _overlap( def _overlap(
question_tokens: Sequence[str], candidate: str, *, cost_vocabulary: bool = False question_tokens: Sequence[str],
) -> int: candidate: str,
"""How many of the question's tokens the candidate text answers to.""" *,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
) -> float:
"""What the candidate text answers of the question.
A COUNT when `weights` is None -- one per question token the candidate
answers to, which is what every caller got before rarity weighting existed
and what the cut still reads. With `weights`, the sum of those tokens'
rarity weights instead.
"""
candidate_tokens = normalise(candidate) candidate_tokens = normalise(candidate)
bridged = cost_vocabulary and any(in_cost_vocabulary(token) for token in candidate_tokens) bridged = cost_vocabulary and any(in_cost_vocabulary(token) for token in candidate_tokens)
return sum( return sum(
1 1 if weights is None else weights.get(token, 1.0)
for token in question_tokens for token in question_tokens
if any(tokens_match(token, other) for other in candidate_tokens) if any(tokens_match(token, other) for other in candidate_tokens)
or (bridged and in_cost_vocabulary(token)) or (bridged and in_cost_vocabulary(token))
@ -662,6 +729,7 @@ def document_scores(
*, *,
profile: BundleProfile = DEFAULT_PROFILE, profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False, cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
) -> dict[str, float]: ) -> dict[str, float]:
"""One score per top-level document, from the indexes and the paths alone. """One score per top-level document, from the indexes and the paths alone.
@ -696,14 +764,19 @@ def document_scores(
totals: dict[str, float] = {} totals: dict[str, float] = {}
units: dict[str, int] = {} units: dict[str, int] = {}
def record(document: str, overlap: int) -> None: def record(document: str, overlap: float) -> None:
totals[document] = totals.get(document, 0.0) + float(overlap) totals[document] = totals.get(document, 0.0) + float(overlap)
units[document] = units.get(document, 0) + 1 units[document] = units.get(document, 0) + 1
for concept_id in concepts: for concept_id in concepts:
record( record(
concept_id.split("/", 1)[0], concept_id.split("/", 1)[0],
_overlap(question_tokens, concept_id.replace("/", " "), cost_vocabulary=bridge), _overlap(
question_tokens,
concept_id.replace("/", " "),
cost_vocabulary=bridge,
weights=weights,
),
) )
for relative in indexes: for relative in indexes:
document = relative.split("/", 1)[0] document = relative.split("/", 1)[0]
@ -713,7 +786,10 @@ def document_scores(
entry = profile.index.parse_entry(line) entry = profile.index.parse_entry(line)
if entry is None: if entry is None:
continue continue
record(document, _overlap(question_tokens, entry.label, cost_vocabulary=bridge)) record(
document,
_overlap(question_tokens, entry.label, cost_vocabulary=bridge, weights=weights),
)
return {document: totals[document] / units[document] for document in totals} return {document: totals[document] / units[document] for document in totals}
@ -737,6 +813,7 @@ def concept_scores(
document_score: Mapping[str, float], document_score: Mapping[str, float],
*, *,
cost_vocabulary: bool = False, cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
) -> list[tuple[Concept, float, int]]: ) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF. """Every concept, ordered best first, fused from three signals by RRF.
@ -760,20 +837,25 @@ def concept_scores(
""" """
question_tokens = normalise(question) question_tokens = normalise(question)
bridge = cost_vocabulary and question_uses_cost_vocabulary(question) bridge = cost_vocabulary and question_uses_cost_vocabulary(question)
titles = {
concept.concept_id: f"{concept.title} {concept.concept_id.replace('/', ' ')}"
for concept in concepts
}
signals: list[dict[str, float]] = [ signals: list[dict[str, float]] = [
{ {
concept.concept_id: float( concept.concept_id: float(
_overlap( _overlap(
question_tokens, question_tokens,
f"{concept.title} {concept.concept_id.replace('/', ' ')}", titles[concept.concept_id],
cost_vocabulary=bridge, cost_vocabulary=bridge,
weights=weights,
) )
) )
for concept in concepts for concept in concepts
}, },
{ {
concept.concept_id: float( concept.concept_id: float(
_overlap(question_tokens, concept.body, cost_vocabulary=bridge) _overlap(question_tokens, concept.body, cost_vocabulary=bridge, weights=weights)
) )
for concept in concepts for concept in concepts
}, },
@ -789,10 +871,26 @@ def concept_scores(
order = sorted(signal, key=lambda key: (-signal[key], key)) order = sorted(signal, key=lambda key: (-signal[key], key))
for position, concept_id in enumerate(order, start=1): for position, concept_id in enumerate(order, start=1):
fused[concept_id] += 1.0 / (RRF_K + position) fused[concept_id] += 1.0 / (RRF_K + position)
lexical = { lexical = (
concept.concept_id: int(signals[0][concept.concept_id] + signals[1][concept.concept_id]) {
for concept in concepts concept.concept_id: int(signals[0][concept.concept_id] + signals[1][concept.concept_id])
} for concept in concepts
}
if weights is None
# A COUNT even when the signals are weighted. The cut reads this, and a
# word every concept carries weighs zero: were `lexical` the weighted
# sum, a concept matching only that word would fall to
# `no_lexical_match` -- turning a ranking change into the GATE
# `54a0bc2` falsified. The gate is a different axis and stays where it
# was, at the price of one more pass over the same two fields.
else {
concept.concept_id: int(
_overlap(question_tokens, titles[concept.concept_id], cost_vocabulary=bridge)
+ _overlap(question_tokens, concept.body, cost_vocabulary=bridge)
)
for concept in concepts
}
)
by_id = {concept.concept_id: concept for concept in concepts} by_id = {concept.concept_id: concept for concept in concepts}
ranked_ids = sorted(fused, key=lambda key: (-fused[key], key)) ranked_ids = sorted(fused, key=lambda key: (-fused[key], key))
return [ return [
@ -1003,12 +1101,13 @@ def build_payload(
profile: BundleProfile = DEFAULT_PROFILE, profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False, cost_vocabulary: bool = False,
reserve_top_rank: bool = False, reserve_top_rank: bool = False,
rarity_weight: bool = False,
) -> dict[str, object]: ) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload. """One bundle plus one question, cut to one contract-conformant payload.
Pure with respect to the clock and the network: the same Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank)` at `(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank,
the same bytes returns the same object, every time. rarity_weight)` at the same bytes returns the same object, every time.
""" """
case, expected, measured = known_positive() case, expected, measured = known_positive()
if expected != measured: if expected != measured:
@ -1046,11 +1145,21 @@ def build_payload(
) )
for concept_id in concept_ids for concept_id in concept_ids
] ]
weights = (
rarity_weights(normalise(question), searchable_text(concepts)) if rarity_weight else None
)
ranked = concept_scores( ranked = concept_scores(
concepts, concepts,
question, question,
document_scores(bundle_root, question, profile=profile, cost_vocabulary=cost_vocabulary), document_scores(
bundle_root,
question,
profile=profile,
cost_vocabulary=cost_vocabulary,
weights=weights,
),
cost_vocabulary=cost_vocabulary, cost_vocabulary=cost_vocabulary,
weights=weights,
) )
matched = sum(1 for _, _, lexical in ranked if lexical > 0) matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank) delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
@ -1163,6 +1272,17 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"still refused" "still refused"
), ),
) )
parser.add_argument(
"--rarity-weight",
action="store_true",
help=(
"weight each lexical hit by log(N/df) over the bundle's own "
"concepts instead of counting it as one. OFF by default, and the "
"default is a MEASUREMENT rather than a preference: measured on "
"four corpora it moved one gold rank 9->8, left one at 35 and made "
"one 96->103 worse. See docs/2026-09-08-sjeldenhetsvekt.md"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout") parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument( parser.add_argument(
"--ref", "--ref",
@ -1196,6 +1316,7 @@ def main(argv: list[str] | None = None) -> int:
limit=args.limit, limit=args.limit,
cost_vocabulary=args.cost_vocabulary, cost_vocabulary=args.cost_vocabulary,
reserve_top_rank=args.reserve_top_rank, reserve_top_rank=args.reserve_top_rank,
rarity_weight=args.rarity_weight,
) )
except ConsumeError as error: except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr) print(f"okf_consume: FAILED - {error}", file=sys.stderr)