Kjell Tore Guttormsen
116d3e1007
O2b asked whether a requirement number can be made worth more than a common word by weighting each hit with the token's rarity in the bundle, with no hand-set constant and no declared token class. It can, on one of the three bundles, and the two it cannot are decomposed rather than guessed. The rule is log(N/df) over the concepts' own tokens, counted with the same four-character prefix rule a hit is scored with. It enters the RANKING and never the GATE: `lexical` stays a count, because `krav` weighs exactly 0 on all three bundles and a weighted gate would drop every concept matching only that word -- which is the gate54a0bc2falsified for other reasons. One df table per bundle reaches every stage that scores the question against text, including the document prior. One pass, 0.241 s over 1 133 concepts. Measured on four corpora, before and after, with every published figure reproduced first: gold fused rank 96 -> 103, 9 -> 8 (withheld -> DELIVERED at rank 8) and 35 -> 35; K2's priced sheet candidate rank 10 -> 2 with the cost vocabulary and 251 -> 78 without; Q-good unmoved at rank 1; hit@8 5 of 6 with every rank identical; the S7 control payload byte-identical on the default command. DEFAULT OFF, decided by the number and not by taste: it does not win on all four, because N100's gold loses seven rank positions. Off means the bytes that were already published, and that is measured -- 8 of 8 payload digests identical against a frozen copy of56c1205built with git archive. Two limits, both someone else's mechanism and both named: MIN_SHARED_PREFIX=4 makes a unique identifier read as 135-of-446 common on N100, so the weight correctly ranks a common adjective above the exact requirement number; and RRF consumes RANKS only, so on N500 -- where the gold already leads the one signal that can see the identifier, and the other two cannot see it at all -- no weighting inside a signal can move anything. Consumption-side only, so no rebuild: the K2 bundle ref 2f82fcfe... stands. Report: docs/2026-09-08-sjeldenhetsvekt.md. 13 new tests, red first; 8 mutations, 8 red, two of them only after the survivors were read as code -- one exposed a fixture that put the identifier where the real corpus does not, and the corrected fixture is what found the RRF limit. Suite 1295 -> 1308. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
287 lines
14 KiB
Markdown
287 lines
14 KiB
Markdown
# The rarity weight: what it delivered, what it cannot reach, 2026-09-08
|
|
|
|
The previous session made a requirement number VISIBLE to the consumption
|
|
pre-pass and could not make it WORTH more than a common word: `_overlap`
|
|
returns a count, so `10.2-2` scored exactly what `krever` scored, and the gold
|
|
concept stayed `below_k` on all three bundles
|
|
(`docs/2026-09-08-kravnummer-tokenisering.md` § 8). This session weights each
|
|
hit by the token's rarity in the bundle and measures what that buys.
|
|
|
|
It buys one of the three requirement lookups and a large move on K2. It does
|
|
not buy the other two, and the reason is structural rather than a matter of
|
|
tuning. Both halves are below, with the decomposition.
|
|
|
|
---
|
|
|
|
## 0. What IS measured, and what is NOT
|
|
|
|
**Measured.** Nine known-positive figures reproduced before anything was
|
|
changed; the document frequency of every question token on all three bundles,
|
|
under the matcher and under exact equality; the gold's fused rank and its
|
|
per-signal ranks, before and after, on three bundles; the delivered/withheld
|
|
verdict and payload digest per bundle; every K2 control the order named, both
|
|
values; the six hit@8 questions; the `df` pass's cost; eight mutations.
|
|
|
|
**Known-positive, run first.** Every figure this session builds on was
|
|
reproduced on this machine, with the committed tool, before any after-column
|
|
was read:
|
|
|
|
| # | known-positive | source | measured here |
|
|
|---|---|---|---|
|
|
| 1 | gold fused rank per bundle | previous session § 4 | **96, 9, 35** |
|
|
| 2 | concepts sharing the gold's lexical score on N500 | previous session § 8 | **70 of 270** |
|
|
| 3 | considered / withheld / delivered | consumer § 3 | 446/438/8, 1133/1125/8, 270/262/8 |
|
|
| 4 | gold position in `withheld` | consumer § 3.1 | 119 of 438, 204 of 1125, 66 of 262 |
|
|
| 5 | K2 S7 control, two commands | previous session § 5 | `2ae46f68…` 169 573 B, `8aaf8558…` 172 588 B |
|
|
| 6 | priced-sheet candidate rank, four rows, O1 bundle | O1 § 7 | 251 / 19 / 1 / 1 |
|
|
| 7 | the same four rows on the rebuilt bundle | previous session § 7 | 251 / **10** / 1 / 1 |
|
|
| 8 | K2 delivery costs, four commands | previous session § 7 | 169 583 / 183 798 / 190 438 / 108 877 B |
|
|
| 9 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6** |
|
|
|
|
**NOT measured.** Two things, both named rather than glossed:
|
|
|
|
- **`o200k_base` token counts.** No `o200k` encoder is installed on this
|
|
machine, and this repository's single-runtime-dependency rule does not admit
|
|
one to produce a caption. Every figure below is therefore in the unit the
|
|
budget gate itself uses — `BUDGET_UNIT`, "utf-8 bytes of emitted JSON" — plus
|
|
the payload `sha256`, which is exact where a byte count is only indicative.
|
|
The previously published o200k figures are not restated as if re-measured.
|
|
- **Whether a live model answers better.** This measures a *ranking*.
|
|
"Delivered" is not "read"; that measurement belongs to the consumer.
|
|
|
|
---
|
|
|
|
## 1. The rule, and why it has no constant in it
|
|
|
|
`rarity_weights(question_tokens, corpus)` returns, per question token,
|
|
|
|
```
|
|
w(t) = log(N / df(t))
|
|
```
|
|
|
|
`N` is the number of concepts in the bundle. `df(t)` is how many of them bear
|
|
the token, counted with the **same** four-character prefix rule a hit is scored
|
|
with. Nothing is set by hand: a word every concept carries weighs `log(1) = 0`
|
|
of itself, and an identifier one concept carries takes the corpus's maximum of
|
|
itself. No token class is declared anywhere.
|
|
|
|
Three decisions inside that sentence, each of them a decision:
|
|
|
|
1. **`df` is counted over the text a hit can be scored on** — title, id and
|
|
body joined, the two lexical signals' union. Counting rarity over one field
|
|
and matching on another would weight a token by how rare it is somewhere it
|
|
is never read.
|
|
2. **The weight enters the RANKING and never the GATE.** `lexical` stays a
|
|
count when weights are supplied, at the price of one more pass over the same
|
|
two fields. This is load-bearing: `krav` weighs exactly 0 on all three
|
|
bundles, so a concept matching only `krav` would have a weighted sum of 0.0
|
|
and would fall to `no_lexical_match` — turning a ranking change into the
|
|
*gate* that `54a0bc2` falsified.
|
|
3. **One `df` table per bundle, reaching every stage that scores the question
|
|
against text**, including the document prior. A weight applied to two of
|
|
three fused signals would be a second ranker rather than one statistic.
|
|
|
|
**Why this is not the sweep `54a0bc2` refuted.** That order asked for smoothed
|
|
IDF as a **port**: a threshold below which a concept is withheld, required to
|
|
zero both known-negative questions. The sweep found the frontier empty — `neg2`
|
|
first reaches 0 at `T = 4.6`, by which point gold-hit reachability has already
|
|
fallen to 0 of 6 — and the order was returned rather than implemented. That
|
|
result stands and is not re-litigated. This is the other use: an ordering
|
|
*inside* the candidate set, with the gate untouched. A ranking cannot withhold
|
|
anything, so the failure mode that refuted the gate has no counterpart here.
|
|
|
|
**The pollution `54a0bc2` § 1 named is inherited deliberately**, and § 3 below
|
|
shows it doing exactly the damage that section predicted.
|
|
|
|
**Cost.** One pass. Measured: **0.062 s** over 446 concepts, **0.241 s** over
|
|
1 133, **0.073 s** over 270 — inside the order's one-second bound on the
|
|
largest bundle by a factor of four.
|
|
|
|
---
|
|
|
|
## 2. The three requirement lookups, before and after
|
|
|
|
Same instrument before and after: the gold's position in the fused `ranked`
|
|
list, and the payload's own verdict from `build_payload`. Command:
|
|
|
|
```
|
|
python3 tools/okf_consume.py <bundle> \
|
|
--question "Hva krever Krav <n> i <normal>? Gjengi det sentrale vilkåret." \
|
|
[--rarity-weight]
|
|
```
|
|
|
|
| | N100:2023 | N200:2024 | N500:2024 |
|
|
|---|---|---|---|
|
|
| concepts | 446 | 1 133 | 270 |
|
|
| gold fused rank, weight **off** | 96 | 9 | 35 |
|
|
| gold fused rank, weight **on** | **103** | **8** | **35** |
|
|
| gold verdict, off | `below_k` | `below_k` | `below_k` |
|
|
| gold verdict, on | `below_k` | **delivered, rank 8** | `below_k` |
|
|
| payload bytes, off → on | 61 566 → 61 452 | 142 861 → **146 641** | 40 626 → 38 869 |
|
|
| payload `sha256`, off | `49e9c718…` | `adc2eac7…` | `21c5bbfe…` |
|
|
| payload `sha256`, on | `ba09a56a…` | `403ee989…` | `1758f6ca…` |
|
|
| `okf_contract_check` | exit 0, 14 rules, 0 findings | idem | idem |
|
|
|
|
**One of three delivers.** N200's gold moves from withheld position 204 of
|
|
1 125 to delivered at rank 8. N500 does not move at all. N100 moves the wrong
|
|
way, by seven positions.
|
|
|
|
---
|
|
|
|
## 3. Why N100 got worse: the matcher makes the identifier common
|
|
|
|
`df` under the matcher, against `df` under exact token equality:
|
|
|
|
| bundle | identifier | `df` under the matcher | `df` exact | `w(identifier)` | `w('sentrale')` |
|
|
|---|---|---|---|---|---|
|
|
| N100 | `3.3.1-13` | **135 of 446** | 1 | 1.195 | **3.798** |
|
|
| N200 | `2.9.2-12` | 17 of 1 133 | 1 | 4.199 | 4.835 |
|
|
| N500 | `10.2-2` | 5 of 270 | 1 | 3.989 | 3.519 |
|
|
|
|
The identifier is unique in all three bundles. It is *common* in N100 only
|
|
because `tokens_match` compares four leading characters, so `3.3.1-13` matches
|
|
every requirement number beginning `3.3.` — 135 of them. The weight then does
|
|
what it was asked to do and ranks a common adjective (`sentrale`, borne by 10
|
|
of 446) as more informative than the exact requirement number.
|
|
|
|
This is `54a0bc2` § 1 — "`df` measured over the colliding matcher measures
|
|
collision breadth, not rarity" — measured on the identifier itself rather than
|
|
on `brann*`. It is not a defect in the weight; it is the weight faithfully
|
|
reporting a `df` the matcher produced. Fixing it means changing the matcher,
|
|
which is a different change with its own measurement, and it is not attempted
|
|
here.
|
|
|
|
---
|
|
|
|
## 4. Why N500 did not move: RRF consumes ranks, and the gold already led
|
|
|
|
The gold's rank on each of the three fused signals, and how many concepts each
|
|
signal reorders:
|
|
|
|
| bundle | signal ranks, off | signal ranks, on | positions moved per signal |
|
|
|---|---|---|---|
|
|
| N100 | 39 / 186 / 127 | 42 / 184 / 127 | 122 / 367 / **0** of 446 |
|
|
| N200 | 8 / 4 / 212 | **4** / 10 / 212 | 37 / 277 / **0** of 1 133 |
|
|
| N500 | **1** / 137 / 74 | **1** / 137 / 74 | **0** / 127 / **0** of 270 |
|
|
|
|
On N500 the gold is **already rank 1** on the title-and-id signal, by counting,
|
|
before any weighting. Weighting raises its score from 3.000 to 3.989 and its
|
|
rank from 1 to 1. RRF was chosen precisely because it **consumes ranks only**
|
|
(`RRF_K`'s comment says so), which makes it invariant to any monotone transform
|
|
of a signal — so the whole increase is invisible to the fusion. What holds the
|
|
gold at 35 is the other two signals, and neither can see the identifier at all:
|
|
the body repeats `krav` and nothing else, and the document prior is a
|
|
two-document density whose order the weight never changes on any of the three
|
|
bundles (0 positions moved, three times).
|
|
|
|
**The general statement, and it is a limit rather than a complaint:** a rarity
|
|
weight can only move a fused rank where it reorders some signal. Where the
|
|
identifier appears in exactly one field and already leads that field, no
|
|
weighting scheme inside the signals can do anything. Closing N500 means
|
|
changing the fusion — giving a signal a weight, or letting a strong single
|
|
signal carry more than 1/(60+1) — and that is a different mechanism from this
|
|
one. It is named here and not built.
|
|
|
|
The same shape is in the test suite, twice, so neither half can quietly stop
|
|
being true: a synthetic corpus where the identifier is in the title alone
|
|
(fused rank 18 → 18, title-signal rank 1 → 1) and one where weighting reorders
|
|
the title signal from last to first (30 → 1) and moves the fused rank only from
|
|
30 to 18.
|
|
|
|
---
|
|
|
|
## 5. K2: every control holds, and the priced sheet moves 10 → 2
|
|
|
|
| control | weight off | weight on | verdict |
|
|
|---|---|---|---|
|
|
| Q-good (`Hvordan skal prisene fylles ut?`), candidate rank | 1 | **1** | holds |
|
|
| Q-good with `--cost-vocabulary` | 1 | **1** | holds |
|
|
| priced sheet, mandate question, vocabulary off | 251 of 271 | **78 of 271** | improves |
|
|
| priced sheet, `--cost-vocabulary --k 12` | 10 of 280 | **2 of 280** | improves |
|
|
| priced sheet, delivered by `--cost-vocabulary --k 12` | rank 10 | **rank 2** | improves |
|
|
| hit@8 over the six published questions | 5 of 6 | **5 of 6**, every rank identical | holds |
|
|
| S7 control question, default command | `2ae46f68…`, 169 573 B | **byte-identical** | holds |
|
|
| S7 control question, `--cost-vocabulary --k 12 --limit 160000` | `8aaf8558…`, 172 588 B | `b205a618…`, 183 462 B | **moves, +10 874 B** |
|
|
|
|
No named K2 control falls. The one figure that moves is a payload the flag
|
|
changes on purpose: with the weight on, that command delivers a different set
|
|
of excerpts and costs 6.3 % more bytes.
|
|
|
|
---
|
|
|
|
## 6. The decision, made by the number
|
|
|
|
The order's rule: default ON if the weight wins on all four corpora with no K2
|
|
control falling; behind a flag otherwise, naming what fell and by how much.
|
|
|
|
**It does not win on all four. N100's gold fused rank moves 96 → 103, a loss of
|
|
seven positions**, on a question where the gold is not delivered either way. So
|
|
the weight ships as `--rarity-weight`, **off by default**.
|
|
|
|
Off means the bytes that were already published, and that is measured rather
|
|
than asserted. Eight payloads — three N-bundle requirement lookups, three
|
|
commands on the rebuilt K2 bundle, two on the O1 bundle — were built from a
|
|
frozen copy of `56c1205` (`git archive` into a scratch tree, `PYTHONPATH` and
|
|
`__file__` both checked) and from this working tree with the flag omitted:
|
|
|
|
**8 of 8 payload digests identical.** No consumer's bytes move because this
|
|
change exists.
|
|
|
|
What the flag buys the caller who sets it, in one line each: N200's requirement
|
|
lookup goes from withheld to delivered; K2's priced sheet goes from candidate
|
|
rank 10 to 2; N100 and N500 are unchanged in verdict and worse and equal in
|
|
rank respectively.
|
|
|
|
---
|
|
|
|
## 7. What did NOT close
|
|
|
|
- **Two of the three requirement lookups are still `below_k`**, and the target
|
|
the order set — rank ≤ 8 on all three — is **not met**. It is met on N200.
|
|
- **N500 cannot be closed by any weighting inside the signals** (§ 4). The next
|
|
mechanism is a change to the fusion, not to the weight, and it has no
|
|
measurement behind it yet.
|
|
- **N100 needs the matcher, not the weight** (§ 3). `MIN_SHARED_PREFIX = 4`
|
|
makes a unique identifier read as 135-of-446 common. That is the same defect
|
|
`54a0bc2` § 1 named a year of sessions ago in another form, and it is now
|
|
measured on the case it costs most.
|
|
- **The 34 % skewed deletion in the orphan gate** is untouched; different axis.
|
|
- **"Delivered" is not "read".** N200's gold is now in the payload. Whether a
|
|
model then answers the engineer's question is the consumer's measurement.
|
|
|
|
---
|
|
|
|
## 8. Tests, mutations, gates
|
|
|
|
Thirteen new tests, red before the rule existed (6 of them failed on
|
|
`AttributeError` alone, and the known-positive — the gold held at rank 18 of 30
|
|
by counting — passed from the start, so the red was the rule's absence and not
|
|
a broken fixture).
|
|
|
|
**Eight mutations, eight red.** Two survived the first pass and neither was
|
|
banded blind:
|
|
|
|
- **M5** (the document prior stops receiving the weights) survived because on
|
|
all three real bundles it is behaviourally equivalent — the two-document
|
|
density is never reordered. The commitment "one statistic reaches every
|
|
stage" is real even where no output distinguishes it, so it is now asserted
|
|
at the call site instead of on an output.
|
|
- **M7** (the title signal stops being weighted) survived because the fixture
|
|
put the identifier in the body as well. The real corpus does not: the gold's
|
|
title answers three question tokens and its body answers one. The fixture was
|
|
wrong, was corrected to the measured shape — and the corrected fixture then
|
|
reproduced the real failure of § 4, which is how § 4 was found at all.
|
|
|
|
| gate | result |
|
|
|---|---|
|
|
| `pytest -q` | **1 308 passed** (1 295 before) |
|
|
| `ruff check src tests tools` | clean |
|
|
| `ruff format --check` | clean, 88 files |
|
|
| `mypy --strict src/` | clean, 17 files |
|
|
| `okf_contract_check` on the three weighted payloads | exit 0, 14 rules, 0 findings |
|
|
| `git diff --stat` | `tools/okf_consume.py`, `tests/test_okf_consume.py` |
|
|
|
|
**No rebuild.** The change is consumption-side only, confirmed by
|
|
`git diff --stat`: no production module is touched, so the K2 bundle ref
|
|
`sha256-tree:2f82fcfea91c3bd3f8ef7147f80cd613227d3ca7975c41d88810233f3f79ab4b`
|
|
is unchanged.
|