Order 20260907T074541Z-939882634-from-.claude asked for smoothed-IDF rarity
weighting so no_lexical_match gates on informative tokens, with both
known-negative controls (sveising/titan/vakuum; pingviner/Bouvetoeya) required
to reach delivered == 0 on the real K2 bundle (629 concepts, 39 documents).
Swept the threshold across the full observed IDF range (1.5-7.9) before
building anything, per Verifiseringsloven face 3: verify an input premise
against ground truth before acting on it. The frontier is empty. neg2 first
reaches 0 at T=4.6, and gold-hit reachability has ALREADY fallen to 0 of 6 at
that same T. neg1 does not reach 0 until T=6.4, by which point every positive
question has been unreachable for eighteen threshold steps. The two rarest
tokens in the entire gold set (titan idf 6.059, vakuum idf 6.347) belong to
the sveising question, so rarity weighting scores them as maximally
informative rather than as noise.
Two structural causes, not just a bad threshold pick: (1) df is measured over
the same four-character prefix matcher that collapses brannalarm/branntrapp/
branncelle/brannkonsept into one bucket, so a real topic word reads as the
LEAST rare term in its own question; (2) MIN_TOKEN_LENGTH=3 admits tokens
MIN_SHARED_PREFIX=4 can never match (hva/for/den are structurally df=0),
leaving some positive questions resting on a single surviving token.
No production code changed — the mechanism the order named is falsified
before implementation, so there is no red test to write against it. Returning
the order with this measurement as the reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hit@8 = 5 of 6, every hit at rank 1, against a chance baseline of 1.35 of 6
over a denominator of 629 concepts per question. Wall time 0.51-0.56 s per
question; spent 17 970 - 74 838 bytes against a 120 000 limit.
Two things this measurement did NOT establish, both in the report:
- BOTH known-negative controls FAILED. A question the bundle has no answer to
still returns eight excerpts, because no natural Norwegian question is
lexically disjoint from a 629-concept corpus under a four-character
shared-prefix rule -- measured per token, the interrogative `hvor` reaches 40
concepts, `brukes` 83. So `no_lexical_match` works per concept and not as a
whole-question gate: an empty excerpt list is evidence of absence, a full one
is not evidence of presence. The fix is named (rarity weighting) and NOT
built, because this step's fence freezes the instrument before it is measured.
- The question texts were written during execution, after the ranker existed.
The plan recorded the gold documents' SIZE profile -- its per-row baselines
sum to 1.35 and the sizes used here reproduce that exactly, which is an
independent check that this is the set the plan profiled -- but it recorded no
question texts, and three of six gold documents could not be pinned uniquely
from the sizes. Not a blind evaluation, and the report says so.
The scorer is a tool, not a script in a document: `tools/okf_consume_measure.py`
takes the gold set as an INPUT because it is tracked in a public repository and
an answer key names a consumer's documents. hit_rank, both chance baselines and
the document-size census are unit-tested; the corpus run is a measurement.
Public-file rule, checked with a pattern DERIVED from the corpus's own 39
document names rather than hand-picked, and shown able to find first (67 hits on
the bundle's own index): zero corpus document names in any tracked file in this
repository. One leak was found and removed on the way -- a corpus concept name
in a code comment and a hardcoded corpus path in a test.
Suite run after git add: 1230 passed, mypy --strict clean on 27 files,
ruff clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
