open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(consume): refutation sweep — smoothed-IDF rarity weighting cannot zero both known-negatives

Browse source 
Order 20260907T074541Z-939882634-from-.claude asked for smoothed-IDF rarity
weighting so no_lexical_match gates on informative tokens, with both
known-negative controls (sveising/titan/vakuum; pingviner/Bouvetoeya) required
to reach delivered == 0 on the real K2 bundle (629 concepts, 39 documents).

Swept the threshold across the full observed IDF range (1.5-7.9) before
building anything, per Verifiseringsloven face 3: verify an input premise
against ground truth before acting on it. The frontier is empty. neg2 first
reaches 0 at T=4.6, and gold-hit reachability has ALREADY fallen to 0 of 6 at
that same T. neg1 does not reach 0 until T=6.4, by which point every positive
question has been unreachable for eighteen threshold steps. The two rarest
tokens in the entire gold set (titan idf 6.059, vakuum idf 6.347) belong to
the sveising question, so rarity weighting scores them as maximally
informative rather than as noise.

Two structural causes, not just a bad threshold pick: (1) df is measured over
the same four-character prefix matcher that collapses brannalarm/branntrapp/
branncelle/brannkonsept into one bucket, so a real topic word reads as the
LEAST rare term in its own question; (2) MIN_TOKEN_LENGTH=3 admits tokens
MIN_SHARED_PREFIX=4 can never match (hva/for/den are structurally df=0),
leaving some positive questions resting on a single surviving token.

No production code changed — the mechanism the order named is falsified
before implementation, so there is no red test to write against it. Returning
the order with this measurement as the reason.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 09:56:36 +02:00
commit 54a0bc26ef
1 changed files with 80 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

80
docs/2026-09-07-okf-konsumskill-maaling.md
View file

@ -111,6 +111,86 @@ this build does not implement it. It is not attempted here because this step's
scope fence freezes the instrument before it is measured — tuning a ranker scope fence freezes the instrument before it is measured — tuning a ranker
against the questions it is being scored on produces a number about the tuning. against the questions it is being scored on produces a number about the tuning.
## Rarity weighting, measured before being built — the named mechanism does not exist
**Date:** 2026-09-07, follow-up order `20260907T074541Z-939882634-from-.claude`.
The order named a specific mechanism — smoothed IDF over the corpus the walk
already reads, gating `no_lexical_match` so a delivered concept must match at
least one *informative* token — and required both known-negative controls
above to reach `delivered == 0` on the real K2 bundle. **This is a refutation
sweep, not a tuning sweep**: every threshold in the full observed range is
reported, not a value chosen to pass. No threshold was picked to make the
negatives zero; the table below is what falsifies that a passing threshold
exists at all, and the order is returned on that basis rather than implemented.
**Method.** `idf(t) = ln((N+1)/(df(t)+1)) + 1`, `N = 629`. `df(t)` counted with
the *same* four-character prefix rule `tokens_match` already applies (the
instrument `okf_consume_measure.token_reach` already uses), over
title + concept-id + body across all 629 concepts. For every concept, `M` is the
highest `idf` among the question tokens that concept matches at all (or none,
if the concept matches nothing). A candidate threshold `T` keeps a concept only
when `M > T`. Swept `T` from 1.5 to 7.9 in steps of 0.1 — the corpus's full
observed range for these two questions' tokens.
| `T` | neg1 (sveising) passing | neg2 (pingviner) passing | gold hits still reachable (of 6) |
|---|---|---|---|
| 1.5 – 2.5 | 31 | 49 | 6 |
| 2.6 – 3.5 | 31 | 49 | 4 (brann, branntegninger lost) |
| 3.6 – 3.7 | 31 | 49 | 3 (akustikk lost) |
| 3.8 – 4.2 | 31 | 17 | 3 |
| 4.3 | 31 | 17 | 2 (solcelle lost) |
| 4.4 – 4.5 | 31 | 17 | 1 (prisene only) |
| **4.6 – 5.0** | **14** | **0** | **0** |
| 5.1 – 6.0 | 5 | 0 | 0 |
| 6.1 – 6.3 | 2 | 0 | 0 |
| **6.4 – 7.9** | **0** | **0** | **0** |
**The frontier is empty, and it closes from the wrong side.** neg2 first
reaches 0 at `T = 4.6` — and at that *exact same* `T`, gold-hit reachability has
already fallen to **0 of 6**: no positive question's gold document is reachable
any more. neg1 has not moved yet at `T = 4.6` (still 14 concepts passing —
`titan` idf 6.059, `vakuum` idf 6.347, `reglene` idf 5.048, `sveising` idf 4.555:
the two rarest tokens in the entire gold set belong to the sveising question,
so rarity weighting scores them as maximally informative) and does not reach 0
until `T = 6.4`, by which point gold-hit reachability has been at 0 for eighteen
threshold steps. There is no `T` where both controls read 0 while any positive
question still reaches its gold document — the interval where neg2 is
satisfied and the interval where neg1 is satisfied do not overlap, and the
positive constraint is already violated before either control interval begins.
**Why, structurally, not just numerically.**
1. **`df` measured over the colliding matcher measures collision breadth, not
rarity.** The four-character prefix rule that lets `varene` match
`varemottak` also collapses every `brann*` compound into one bucket:
`brannkonseptet` reads `df=132`, reaching 18 of the corpus's 39 documents,
because `brannalarm`, `branntrapp`, `branncelle` and `branntetting` all
share the same four leading characters and none of them is the same word.
Rarity weighting inherits this pollution rather than fixing it — the
*most* topical word in the corpus for the brann question reads as the
*least* rare.
2. **`MIN_TOKEN_LENGTH = 3` admits tokens `MIN_SHARED_PREFIX = 4` guarantees
can never match.** `hva`, `for`, `den` are three characters; `tokens_match`
requires the shorter side to reach the four-character floor before any
comparison runs, so these tokens are structurally `df = 0` regardless of
what the corpus contains. The brann question therefore rests on its single
remaining token (`brannkonseptet`) the moment `sier` — which matches three
concepts, none in the gold document — is set aside.
3. **The sveising question's match is a real word, not noise.** `sveising`
matches `sveisede` in eight architectural-drawing concepts describing
welded floor-covering seams — correct lexically, unrelated to the
question's actual subject. No token-rarity statistic distinguishes this
from a genuine topical match, because by the rarity measure it *is* one
(`df = 17`, concentrated in a single document, `doc_reach = 1/39`).
**What is not built.** A coverage rule — require **two or more** informative
tokens to match, rather than any one — probably zeros both controls, since
neither negative question has two tokens that jointly land in the same small
concept set. It is named and explicitly **not implemented here**: choosing it
now would be picking a mechanism because it was checked against these two
answers, which is exactly what this order's own fence forbids. It goes to the
operator as an option, unmeasured.
## Honesty limits ## Honesty limits
Six, and the first three matter most. Six, and the first three matter most.