open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(consume): weight a lexical hit by its rarity, off by default and measured

Browse source 
O2b asked whether a requirement number can be made worth more than a common
word by weighting each hit with the token's rarity in the bundle, with no
hand-set constant and no declared token class. It can, on one of the three
bundles, and the two it cannot are decomposed rather than guessed.

The rule is log(N/df) over the concepts' own tokens, counted with the same
four-character prefix rule a hit is scored with. It enters the RANKING and
never the GATE: `lexical` stays a count, because `krav` weighs exactly 0 on
all three bundles and a weighted gate would drop every concept matching only
that word -- which is the gate 54a0bc2 falsified for other reasons. One df
table per bundle reaches every stage that scores the question against text,
including the document prior. One pass, 0.241 s over 1 133 concepts.

Measured on four corpora, before and after, with every published figure
reproduced first: gold fused rank 96 -> 103, 9 -> 8 (withheld -> DELIVERED at
rank 8) and 35 -> 35; K2's priced sheet candidate rank 10 -> 2 with the cost
vocabulary and 251 -> 78 without; Q-good unmoved at rank 1; hit@8 5 of 6 with
every rank identical; the S7 control payload byte-identical on the default
command.

DEFAULT OFF, decided by the number and not by taste: it does not win on all
four, because N100's gold loses seven rank positions. Off means the bytes that
were already published, and that is measured -- 8 of 8 payload digests
identical against a frozen copy of 56c1205 built with git archive.

Two limits, both someone else's mechanism and both named: MIN_SHARED_PREFIX=4
makes a unique identifier read as 135-of-446 common on N100, so the weight
correctly ranks a common adjective above the exact requirement number; and RRF
consumes RANKS only, so on N500 -- where the gold already leads the one signal
that can see the identifier, and the other two cannot see it at all -- no
weighting inside a signal can move anything.

Consumption-side only, so no rebuild: the K2 bundle ref 2f82fcfe... stands.

Report: docs/2026-09-08-sjeldenhetsvekt.md. 13 new tests, red first; 8
mutations, 8 red, two of them only after the survivors were read as code -- one
exposed a fixture that put the identifier where the real corpus does not, and
the corrected fixture is what found the RRF limit. Suite 1295 -> 1308.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 12:36:57 +02:00
commit 116d3e1007
5 changed files with 755 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

287
docs/2026-09-08-sjeldenhetsvekt.md Normal file
View file

@ -0,0 +1,287 @@
# The rarity weight: what it delivered, what it cannot reach, 2026-09-08
The previous session made a requirement number VISIBLE to the consumption
pre-pass and could not make it WORTH more than a common word: `_overlap`
returns a count, so `10.2-2` scored exactly what `krever` scored, and the gold
concept stayed `below_k` on all three bundles
(`docs/2026-09-08-kravnummer-tokenisering.md` § 8). This session weights each
hit by the token's rarity in the bundle and measures what that buys.
It buys one of the three requirement lookups and a large move on K2. It does
not buy the other two, and the reason is structural rather than a matter of
tuning. Both halves are below, with the decomposition.
---
## 0. What IS measured, and what is NOT
**Measured.** Nine known-positive figures reproduced before anything was
changed; the document frequency of every question token on all three bundles,
under the matcher and under exact equality; the gold's fused rank and its
per-signal ranks, before and after, on three bundles; the delivered/withheld
verdict and payload digest per bundle; every K2 control the order named, both
values; the six hit@8 questions; the `df` pass's cost; eight mutations.
**Known-positive, run first.** Every figure this session builds on was
reproduced on this machine, with the committed tool, before any after-column
was read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | gold fused rank per bundle | previous session § 4 | **96, 9, 35** |
| 2 | concepts sharing the gold's lexical score on N500 | previous session § 8 | **70 of 270** |
| 3 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
| 4 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
| 5 | K2 S7 control, two commands | previous session § 5 | `2ae46f68…` 169 573 B, `8aaf8558…` 172 588 B |
| 6 | priced-sheet candidate rank, four rows, O1 bundle | O1 § 7 | 251 / 19 / 1 / 1 |
| 7 | the same four rows on the rebuilt bundle | previous session § 7 | 251 / **10** / 1 / 1 |
| 8 | K2 delivery costs, four commands | previous session § 7 | 169 583 / 183 798 / 190 438 / 108 877 B |
| 9 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6** |
**NOT measured.** Two things, both named rather than glossed:
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
machine, and this repository's single-runtime-dependency rule does not admit
one to produce a caption. Every figure below is therefore in the unit the
budget gate itself uses — `BUDGET_UNIT`, "utf-8 bytes of emitted JSON" — plus
the payload `sha256`, which is exact where a byte count is only indicative.
The previously published o200k figures are not restated as if re-measured.
- **Whether a live model answers better.** This measures a *ranking*.
"Delivered" is not "read"; that measurement belongs to the consumer.
---
## 1. The rule, and why it has no constant in it
`rarity_weights(question_tokens, corpus)` returns, per question token,
```
w(t) = log(N / df(t))
```
`N` is the number of concepts in the bundle. `df(t)` is how many of them bear
the token, counted with the **same** four-character prefix rule a hit is scored
with. Nothing is set by hand: a word every concept carries weighs `log(1) = 0`
of itself, and an identifier one concept carries takes the corpus's maximum of
itself. No token class is declared anywhere.
Three decisions inside that sentence, each of them a decision:
1. **`df` is counted over the text a hit can be scored on** — title, id and
body joined, the two lexical signals' union. Counting rarity over one field
and matching on another would weight a token by how rare it is somewhere it
is never read.
2. **The weight enters the RANKING and never the GATE.** `lexical` stays a
count when weights are supplied, at the price of one more pass over the same
two fields. This is load-bearing: `krav` weighs exactly 0 on all three
bundles, so a concept matching only `krav` would have a weighted sum of 0.0
and would fall to `no_lexical_match` — turning a ranking change into the
*gate* that `54a0bc2` falsified.
3. **One `df` table per bundle, reaching every stage that scores the question
against text**, including the document prior. A weight applied to two of
three fused signals would be a second ranker rather than one statistic.
**Why this is not the sweep `54a0bc2` refuted.** That order asked for smoothed
IDF as a **port**: a threshold below which a concept is withheld, required to
zero both known-negative questions. The sweep found the frontier empty — `neg2`
first reaches 0 at `T = 4.6`, by which point gold-hit reachability has already
fallen to 0 of 6 — and the order was returned rather than implemented. That
result stands and is not re-litigated. This is the other use: an ordering
*inside* the candidate set, with the gate untouched. A ranking cannot withhold
anything, so the failure mode that refuted the gate has no counterpart here.
**The pollution `54a0bc2` § 1 named is inherited deliberately**, and § 3 below
shows it doing exactly the damage that section predicted.
**Cost.** One pass. Measured: **0.062 s** over 446 concepts, **0.241 s** over
1 133, **0.073 s** over 270 — inside the order's one-second bound on the
largest bundle by a factor of four.
---
## 2. The three requirement lookups, before and after
Same instrument before and after: the gold's position in the fused `ranked`
list, and the payload's own verdict from `build_payload`. Command:
```
python3 tools/okf_consume.py <bundle> \
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
[--rarity-weight]
```
| | N100:2023 | N200:2024 | N500:2024 |
|---|---|---|---|
| concepts | 446 | 1 133 | 270 |
| gold fused rank, weight **off** | 96 | 9 | 35 |
| gold fused rank, weight **on** | **103** | **8** | **35** |
| gold verdict, off | `below_k` | `below_k` | `below_k` |
| gold verdict, on | `below_k` | **delivered, rank 8** | `below_k` |
| payload bytes, off → on | 61 566 → 61 452 | 142 861 → **146 641** | 40 626 → 38 869 |
| payload `sha256`, off | `49e9c718…` | `adc2eac7…` | `21c5bbfe…` |
| payload `sha256`, on | `ba09a56a…` | `403ee989…` | `1758f6ca…` |
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
**One of three delivers.** N200's gold moves from withheld position 204 of
1 125 to delivered at rank 8. N500 does not move at all. N100 moves the wrong
way, by seven positions.
---
## 3. Why N100 got worse: the matcher makes the identifier common
`df` under the matcher, against `df` under exact token equality:
| bundle | identifier | `df` under the matcher | `df` exact | `w(identifier)` | `w('sentrale')` |
|---|---|---|---|---|---|
| N100 | `3.3.1-13` | **135 of 446** | 1 | 1.195 | **3.798** |
| N200 | `2.9.2-12` | 17 of 1 133 | 1 | 4.199 | 4.835 |
| N500 | `10.2-2` | 5 of 270 | 1 | 3.989 | 3.519 |
The identifier is unique in all three bundles. It is *common* in N100 only
because `tokens_match` compares four leading characters, so `3.3.1-13` matches
every requirement number beginning `3.3.` — 135 of them. The weight then does
what it was asked to do and ranks a common adjective (`sentrale`, borne by 10
of 446) as more informative than the exact requirement number.
This is `54a0bc2` § 1 — "`df` measured over the colliding matcher measures
collision breadth, not rarity" — measured on the identifier itself rather than
on `brann*`. It is not a defect in the weight; it is the weight faithfully
reporting a `df` the matcher produced. Fixing it means changing the matcher,
which is a different change with its own measurement, and it is not attempted
here.
---
## 4. Why N500 did not move: RRF consumes ranks, and the gold already led
The gold's rank on each of the three fused signals, and how many concepts each
signal reorders:
| bundle | signal ranks, off | signal ranks, on | positions moved per signal |
|---|---|---|---|
| N100 | 39 / 186 / 127 | 42 / 184 / 127 | 122 / 367 / **0** of 446 |
| N200 | 8 / 4 / 212 | **4** / 10 / 212 | 37 / 277 / **0** of 1 133 |
| N500 | **1** / 137 / 74 | **1** / 137 / 74 | **0** / 127 / **0** of 270 |
On N500 the gold is **already rank 1** on the title-and-id signal, by counting,
before any weighting. Weighting raises its score from 3.000 to 3.989 and its
rank from 1 to 1. RRF was chosen precisely because it **consumes ranks only**
(`RRF_K`'s comment says so), which makes it invariant to any monotone transform
of a signal — so the whole increase is invisible to the fusion. What holds the
gold at 35 is the other two signals, and neither can see the identifier at all:
the body repeats `krav` and nothing else, and the document prior is a
two-document density whose order the weight never changes on any of the three
bundles (0 positions moved, three times).
**The general statement, and it is a limit rather than a complaint:** a rarity
weight can only move a fused rank where it reorders some signal. Where the
identifier appears in exactly one field and already leads that field, no
weighting scheme inside the signals can do anything. Closing N500 means
changing the fusion — giving a signal a weight, or letting a strong single
signal carry more than 1/(60+1) — and that is a different mechanism from this
one. It is named here and not built.
The same shape is in the test suite, twice, so neither half can quietly stop
being true: a synthetic corpus where the identifier is in the title alone
(fused rank 18 → 18, title-signal rank 1 → 1) and one where weighting reorders
the title signal from last to first (30 → 1) and moves the fused rank only from
30 to 18.
---
## 5. K2: every control holds, and the priced sheet moves 10 → 2
| control | weight off | weight on | verdict |
|---|---|---|---|
| Q-good (`Hvordan skal prisene fylles ut?`), candidate rank | 1 | **1** | holds |
| Q-good with `--cost-vocabulary` | 1 | **1** | holds |
| priced sheet, mandate question, vocabulary off | 251 of 271 | **78 of 271** | improves |
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **2 of 280** | improves |
| priced sheet, delivered by `--cost-vocabulary --k 12` | rank 10 | **rank 2** | improves |
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank identical | holds |
| S7 control question, default command | `2ae46f68…`, 169 573 B | **byte-identical** | holds |
| S7 control question, `--cost-vocabulary --k 12 --limit 160000` | `8aaf8558…`, 172 588 B | `b205a618…`, 183 462 B | **moves, +10 874 B** |
No named K2 control falls. The one figure that moves is a payload the flag
changes on purpose: with the weight on, that command delivers a different set
of excerpts and costs 6.3 % more bytes.
---
## 6. The decision, made by the number
The order's rule: default ON if the weight wins on all four corpora with no K2
control falling; behind a flag otherwise, naming what fell and by how much.
**It does not win on all four. N100's gold fused rank moves 96 → 103, a loss of
seven positions**, on a question where the gold is not delivered either way. So
the weight ships as `--rarity-weight`, **off by default**.
Off means the bytes that were already published, and that is measured rather
than asserted. Eight payloads — three N-bundle requirement lookups, three
commands on the rebuilt K2 bundle, two on the O1 bundle — were built from a
frozen copy of `56c1205` (`git archive` into a scratch tree, `PYTHONPATH` and
`__file__` both checked) and from this working tree with the flag omitted:
**8 of 8 payload digests identical.** No consumer's bytes move because this
change exists.
What the flag buys the caller who sets it, in one line each: N200's requirement
lookup goes from withheld to delivered; K2's priced sheet goes from candidate
rank 10 to 2; N100 and N500 are unchanged in verdict and worse and equal in
rank respectively.
---
## 7. What did NOT close
- **Two of the three requirement lookups are still `below_k`**, and the target
the order set — rank ≤ 8 on all three — is **not met**. It is met on N200.
- **N500 cannot be closed by any weighting inside the signals** (§ 4). The next
mechanism is a change to the fusion, not to the weight, and it has no
measurement behind it yet.
- **N100 needs the matcher, not the weight** (§ 3). `MIN_SHARED_PREFIX = 4`
makes a unique identifier read as 135-of-446 common. That is the same defect
`54a0bc2` § 1 named a year of sessions ago in another form, and it is now
measured on the case it costs most.
- **The 34 % skewed deletion in the orphan gate** is untouched; different axis.
- **"Delivered" is not "read".** N200's gold is now in the payload. Whether a
model then answers the engineer's question is the consumer's measurement.
---
## 8. Tests, mutations, gates
Thirteen new tests, red before the rule existed (6 of them failed on
`AttributeError` alone, and the known-positive — the gold held at rank 18 of 30
by counting — passed from the start, so the red was the rule's absence and not
a broken fixture).
**Eight mutations, eight red.** Two survived the first pass and neither was
banded blind:
- **M5** (the document prior stops receiving the weights) survived because on
all three real bundles it is behaviourally equivalent — the two-document
density is never reordered. The commitment "one statistic reaches every
stage" is real even where no output distinguishes it, so it is now asserted
at the call site instead of on an output.
- **M7** (the title signal stops being weighted) survived because the fixture
put the identifier in the body as well. The real corpus does not: the gold's
title answers three question tokens and its body answers one. The fixture was
wrong, was corrected to the measured shape — and the corrected fixture then
reproduced the real failure of § 4, which is how § 4 was found at all.
| gate | result |
|---|---|
| `pytest -q` | **1 308 passed** (1 295 before) |
| `ruff check src tests tools` | clean |
| `ruff format --check` | clean, 88 files |
| `mypy --strict src/` | clean, 17 files |
| `okf_contract_check` on the three weighted payloads | exit 0, 14 rules, 0 findings |
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
**No rebuild.** The change is consumption-side only, confirmed by
`git diff --stat`: no production module is touched, so the K2 bundle ref
`sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b`
is unchanged.