llm-ingestion-okf/docs/2026-09-08-sjeldenhetsvekt.md

The rarity weight: what it delivered, what it cannot reach, 2026-09-08

The previous session made a requirement number VISIBLE to the consumption pre-pass and could not make it WORTH more than a common word: _overlap returns a count, so 10.2-2 scored exactly what krever scored, and the gold concept stayed below_k on all three bundles (docs/2026-09-08-kravnummer-tokenisering.md § 8). This session weights each hit by the token's rarity in the bundle and measures what that buys.

It buys one of the three requirement lookups and a large move on K2. It does not buy the other two, and the reason is structural rather than a matter of tuning. Both halves are below, with the decomposition.

---

0. What IS measured, and what is NOT

Measured. Nine known-positive figures reproduced before anything was changed; the document frequency of every question token on all three bundles, under the matcher and under exact equality; the gold's fused rank and its per-signal ranks, before and after, on three bundles; the delivered/withheld verdict and payload digest per bundle; every K2 control the order named, both values; the six hit@8 questions; the df pass's cost; eight mutations.

Known-positive, run first. Every figure this session builds on was reproduced on this machine, with the committed tool, before any after-column was read:

#	known-positive	source	measured here
1	gold fused rank per bundle	previous session § 4	96, 9, 35
2	concepts sharing the gold's lexical score on N500	previous session § 8	70 of 270
3	considered / withheld / delivered	consumer § 3	446/438/8, 1133/1125/8, 270/262/8
4	gold position in withheld	consumer § 3.1	119 of 438, 204 of 1125, 66 of 262
5	K2 S7 control, two commands	previous session § 5	2ae46f68… 169 573 B, 8aaf8558… 172 588 B
6	priced-sheet candidate rank, four rows, O1 bundle	O1 § 7	251 / 19 / 1 / 1
7	the same four rows on the rebuilt bundle	previous session § 7	251 / 10 / 1 / 1
8	K2 delivery costs, four commands	previous session § 7	169 583 / 183 798 / 190 438 / 108 877 B
9	hit@8 over the six published questions	2026-09-07 § hit@k	5 of 6

NOT measured. Two things, both named rather than glossed:

• o200k_base token counts. No o200k encoder is installed on this machine, and this repository's single-runtime-dependency rule does not admit one to produce a caption. Every figure below is therefore in the unit the budget gate itself uses — BUDGET_UNIT, "utf-8 bytes of emitted JSON" — plus the payload sha256, which is exact where a byte count is only indicative. The previously published o200k figures are not restated as if re-measured.
• Whether a live model answers better. This measures a ranking. "Delivered" is not "read"; that measurement belongs to the consumer.

---

1. The rule, and why it has no constant in it

rarity_weights(question_tokens, corpus) returns, per question token,

w(t) = log(N / df(t))

N is the number of concepts in the bundle. df(t) is how many of them bear the token, counted with the same four-character prefix rule a hit is scored with. Nothing is set by hand: a word every concept carries weighs log(1) = 0 of itself, and an identifier one concept carries takes the corpus's maximum of itself. No token class is declared anywhere.

Three decisions inside that sentence, each of them a decision:

1. df is counted over the text a hit can be scored on — title, id and body joined, the two lexical signals' union. Counting rarity over one field and matching on another would weight a token by how rare it is somewhere it is never read.
2. The weight enters the RANKING and never the GATE. lexical stays a count when weights are supplied, at the price of one more pass over the same two fields. This is load-bearing: krav weighs exactly 0 on all three bundles, so a concept matching only krav would have a weighted sum of 0.0 and would fall to no_lexical_match — turning a ranking change into the gate that 54a0bc2 falsified.
3. One df table per bundle, reaching every stage that scores the question against text, including the document prior. A weight applied to two of three fused signals would be a second ranker rather than one statistic.

Why this is not the sweep 54a0bc2 refuted. That order asked for smoothed IDF as a port: a threshold below which a concept is withheld, required to zero both known-negative questions. The sweep found the frontier empty — neg2 first reaches 0 at T = 4.6, by which point gold-hit reachability has already fallen to 0 of 6 — and the order was returned rather than implemented. That result stands and is not re-litigated. This is the other use: an ordering inside the candidate set, with the gate untouched. A ranking cannot withhold anything, so the failure mode that refuted the gate has no counterpart here.

The pollution 54a0bc2 § 1 named is inherited deliberately, and § 3 below shows it doing exactly the damage that section predicted.

Cost. One pass. Measured: 0.062 s over 446 concepts, 0.241 s over 1 133, 0.073 s over 270 — inside the order's one-second bound on the largest bundle by a factor of four.

---

2. The three requirement lookups, before and after

Same instrument before and after: the gold's position in the fused ranked list, and the payload's own verdict from build_payload. Command:

python3 tools/okf_consume.py <bundle> \
  --question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
  [--rarity-weight]

	N100:2023	N200:2024	N500:2024
concepts	446	1 133	270
gold fused rank, weight off	96	9	35
gold fused rank, weight on	103	8	35
gold verdict, off	below_k	below_k	below_k
gold verdict, on	below_k	delivered, rank 8	below_k
payload bytes, off → on	61 566 → 61 452	142 861 → 146 641	40 626 → 38 869
payload sha256, off	49e9c718…	adc2eac7…	21c5bbfe…
payload sha256, on	ba09a56a…	403ee989…	1758f6ca…
okf_contract_check	exit 0, 14 rules, 0 findings	idem	idem

One of three delivers. N200's gold moves from withheld position 204 of 1 125 to delivered at rank 8. N500 does not move at all. N100 moves the wrong way, by seven positions.

---

3. Why N100 got worse: the matcher makes the identifier common

df under the matcher, against df under exact token equality:

bundle	identifier	df under the matcher	df exact	w(identifier)	w('sentrale')
N100	3.3.1-13	135 of 446	1	1.195	3.798
N200	2.9.2-12	17 of 1 133	1	4.199	4.835
N500	10.2-2	5 of 270	1	3.989	3.519

The identifier is unique in all three bundles. It is common in N100 only because tokens_match compares four leading characters, so 3.3.1-13 matches every requirement number beginning 3.3. — 135 of them. The weight then does what it was asked to do and ranks a common adjective (sentrale, borne by 10 of 446) as more informative than the exact requirement number.

This is 54a0bc2 § 1 — "df measured over the colliding matcher measures collision breadth, not rarity" — measured on the identifier itself rather than on brann*. It is not a defect in the weight; it is the weight faithfully reporting a df the matcher produced. Fixing it means changing the matcher, which is a different change with its own measurement, and it is not attempted here.

---

4. Why N500 did not move: RRF consumes ranks, and the gold already led

The gold's rank on each of the three fused signals, and how many concepts each signal reorders:

bundle	signal ranks, off	signal ranks, on	positions moved per signal
N100	39 / 186 / 127	42 / 184 / 127	122 / 367 / 0 of 446
N200	8 / 4 / 212	4 / 10 / 212	37 / 277 / 0 of 1 133
N500	1 / 137 / 74	1 / 137 / 74	0 / 127 / 0 of 270

On N500 the gold is already rank 1 on the title-and-id signal, by counting, before any weighting. Weighting raises its score from 3.000 to 3.989 and its rank from 1 to 1. RRF was chosen precisely because it consumes ranks only (RRF_K's comment says so), which makes it invariant to any monotone transform of a signal — so the whole increase is invisible to the fusion. What holds the gold at 35 is the other two signals, and neither can see the identifier at all: the body repeats krav and nothing else, and the document prior is a two-document density whose order the weight never changes on any of the three bundles (0 positions moved, three times).

The general statement, and it is a limit rather than a complaint: a rarity weight can only move a fused rank where it reorders some signal. Where the identifier appears in exactly one field and already leads that field, no weighting scheme inside the signals can do anything. Closing N500 means changing the fusion — giving a signal a weight, or letting a strong single signal carry more than 1/(60+1) — and that is a different mechanism from this one. It is named here and not built.

The same shape is in the test suite, twice, so neither half can quietly stop being true: a synthetic corpus where the identifier is in the title alone (fused rank 18 → 18, title-signal rank 1 → 1) and one where weighting reorders the title signal from last to first (30 → 1) and moves the fused rank only from 30 to 18.

---

5. K2: every control holds, and the priced sheet moves 10 → 2

control	weight off	weight on	verdict
Q-good (Hvordan skal prisene fylles ut?), candidate rank	1	1	holds
Q-good with --cost-vocabulary	1	1	holds
priced sheet, mandate question, vocabulary off	251 of 271	78 of 271	improves
priced sheet, --cost-vocabulary --k 12	10 of 280	2 of 280	improves
priced sheet, delivered by --cost-vocabulary --k 12	rank 10	rank 2	improves
hit@8 over the six published questions	5 of 6	5 of 6, every rank identical	holds
S7 control question, default command	2ae46f68…, 169 573 B	byte-identical	holds
S7 control question, --cost-vocabulary --k 12 --limit 160000	8aaf8558…, 172 588 B	b205a618…, 183 462 B	moves, +10 874 B

No named K2 control falls. The one figure that moves is a payload the flag changes on purpose: with the weight on, that command delivers a different set of excerpts and costs 6.3 % more bytes.

---

6. The decision, made by the number

The order's rule: default ON if the weight wins on all four corpora with no K2 control falling; behind a flag otherwise, naming what fell and by how much.

It does not win on all four. N100's gold fused rank moves 96 → 103, a loss of seven positions, on a question where the gold is not delivered either way. So the weight ships as --rarity-weight, off by default.

Off means the bytes that were already published, and that is measured rather than asserted. Eight payloads — three N-bundle requirement lookups, three commands on the rebuilt K2 bundle, two on the O1 bundle — were built from a frozen copy of 56c1205 (git archive into a scratch tree, PYTHONPATH and __file__ both checked) and from this working tree with the flag omitted:

8 of 8 payload digests identical. No consumer's bytes move because this change exists.

What the flag buys the caller who sets it, in one line each: N200's requirement lookup goes from withheld to delivered; K2's priced sheet goes from candidate rank 10 to 2; N100 and N500 are unchanged in verdict and worse and equal in rank respectively.

---

7. What did NOT close

• Two of the three requirement lookups are still below_k, and the target the order set — rank ≤ 8 on all three — is not met. It is met on N200.
• N500 cannot be closed by any weighting inside the signals (§ 4). The next mechanism is a change to the fusion, not to the weight, and it has no measurement behind it yet.
• N100 needs the matcher, not the weight (§ 3). MIN_SHARED_PREFIX = 4 makes a unique identifier read as 135-of-446 common. That is the same defect 54a0bc2 § 1 named a year of sessions ago in another form, and it is now measured on the case it costs most.
• The 34 % skewed deletion in the orphan gate is untouched; different axis.
• "Delivered" is not "read". N200's gold is now in the payload. Whether a model then answers the engineer's question is the consumer's measurement.

---

8. Tests, mutations, gates

Thirteen new tests, red before the rule existed (6 of them failed on AttributeError alone, and the known-positive — the gold held at rank 18 of 30 by counting — passed from the start, so the red was the rule's absence and not a broken fixture).

Eight mutations, eight red. Two survived the first pass and neither was banded blind:

• M5 (the document prior stops receiving the weights) survived because on all three real bundles it is behaviourally equivalent — the two-document density is never reordered. The commitment "one statistic reaches every stage" is real even where no output distinguishes it, so it is now asserted at the call site instead of on an output.
• M7 (the title signal stops being weighted) survived because the fixture put the identifier in the body as well. The real corpus does not: the gold's title answers three question tokens and its body answers one. The fixture was wrong, was corrected to the measured shape — and the corrected fixture then reproduced the real failure of § 4, which is how § 4 was found at all.

gate	result
pytest -q	1 308 passed (1 295 before)
ruff check src tests tools	clean
ruff format --check	clean, 88 files
mypy --strict src/	clean, 17 files
okf_contract_check on the three weighted payloads	exit 0, 14 rules, 0 findings
git diff --stat	tools/okf_consume.py, tests/test_okf_consume.py

No rebuild. The change is consumption-side only, confirmed by git diff --stat: no production module is touched, so the K2 bundle ref sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b is unchanged.