"""The shared prefix has to be a WORD, or `bilateral` reaches 400 of 453. THE FINDING, RE-MEASURED. `MIN_SHARED_PREFIX = 4` exists for Norwegian compounding -- `vare|ne` and `vare|mottak` share a stem -- and it has a known-positive in the suite. It also matches on four characters that are not a stem at all. Measured on the pinned 453-concept bundle, with the control run FIRST every time: word equality prefix share under (control) 79 172 38.0 % undersjoisk 0 172 38.0 % bilateral 0 400 88.3 % standhaftig 0 219 48.3 % The two extra known-negative words were FOUND rather than chosen: every 4-character prefix in the bundle was ranked by document frequency, and a real Norwegian word was taken from the widest ones -- `bila` (400 of 453, through `bilag`) and `stan` (219, through `standard`). THREE CANDIDATES WERE MEASURED AND ALL THREE FAILED, each on the SAME row. a. a longer floor (5, 6, 7, 8) row 1 falls 1 -> 2 on the default bundle and 1 -> None on Arm B b. coverage >= 0.5 / 0.6 / 0.7 / 0.8 every negative to 0, control still 88 > 79, and row 1 still falls c. prefix only for words >= 6/8/10/12 only >= 6 holds the ranks, and it fixes `bilateral` alone: 137 and 219 remain WHY THEY ALL FAIL ON ONE ROW, decomposed rather than guessed. Row 1's question token `prisene` reaches its gold document through `pris|sammenstilling` and `pris|skjema` on the four characters `pris`. That is 0.57 coverage of the question word and 0.22 of the document word -- so a coverage rule at 0.60 cuts exactly the match that finds the price sheet. The over-match and the wanted match are the same mechanism seen from two sides, which is why length and coverage cannot separate them. WHAT DOES SEPARATE THEM. `pris` is a word; `bila` and `stan` are not. The fourth candidate requires the shared prefix to occur as a token in the bundle's own concepts, and it is the only one that clears both criteria at once: bundle bilateral standhaftig undersjoisk control ranks default-20260912 400 -> 0 219 -> 56 172 -> 162 172 (1,1,1,1,1,None) armB-20260903 512 -> 0 235 -> 33 174 -> 160 174 (1,1,1,1,1,None) WHAT REMAINS, AND WHY IT IS NOT A CEILING BUT A DIFFERENT ANSWER. `undersjoisk` still reaches 162 concepts, because it shares `under` with them and `under` IS a word in this corpus -- a productive Norwegian prefix. `standhaftig` still reaches 56 through `stand`. Those are genuine shared morphemes, and a rule that cut them would be cutting the behaviour `MIN_SHARED_PREFIX` was built for. The residual is a statement about Norwegian, not about the implementation. HONESTY LIMITS: n = 6 questions, one rater, one gold set, and row 6 misses on every bundle and every configuration measured so far. The three known-negative words were chosen by this session from the bundle's own vocabulary, not by an independent source. The vocabulary is the BUNDLE's, so this rule makes a payload depend on the corpus the way `rarity_weights` already does. """ from __future__ import annotations from llm_ingestion_okf import consume #: The corpus vocabulary in miniature. `pris` and `under` stand on their own; #: `bila` and `stan` never do, which is the whole discriminator. STEMS = frozenset( { "pris", "prisene", "prissammenstilling", "prisskjema", "under", "underbygning", "bilag", "standard", "varemottak", "varene", "vare", } ) def test_a_shared_prefix_that_is_not_a_word_no_longer_matches() -> None: """`bila` is four characters of `bilag` and of `bilateral`, and no word.""" assert consume.tokens_match("bilateral", "bilag") is True, "the shipped rule matches" assert consume.tokens_match("bilateral", "bilag", stems=STEMS) is False def test_a_shared_prefix_that_is_a_word_still_matches() -> None: """The known-positive the repair must not kill: row 1 depends on it. `pris` is 0.57 of `prisene` and 0.22 of `prissammenstilling`, so every coverage rule measured cut it -- and cutting it is what cost row 1 its rank on both bundles. """ assert consume.tokens_match("prisene", "prissammenstilling", stems=STEMS) is True assert consume.tokens_match("varene", "varemottak", stems=STEMS) is True def test_the_productive_norwegian_prefix_is_kept_and_that_is_the_answer() -> None: """`under` is a word, so `undersjoisk` keeps reaching `underbygning`. Stated as a decision rather than left as a residual: this rule does not claim to separate two words that genuinely share a morpheme, and the 162 concepts `undersjoisk` still reaches are that morpheme, not a defect. """ assert consume.tokens_match("undersjøisk", "underbygning", stems=STEMS) is True def test_an_identifier_is_untouched_by_the_stem_rule() -> None: """The 2026-09-08 identifier fix stays exactly as it was.""" assert consume.tokens_match("3.3.1-13", "3.3.1-13", stems=STEMS) is True assert consume.tokens_match("3.3.1-13", "3.3.1-14", stems=STEMS) is False def test_equality_never_needs_a_stem() -> None: """A token always matches itself, whatever the vocabulary says.""" assert consume.tokens_match("zzzznotinvocab", "zzzznotinvocab", stems=STEMS) is True def test_without_stems_the_function_is_byte_for_byte_the_shipped_one() -> None: """The opt-out reproduces the pre-round-10 matcher on its own material.""" for left, right in ( ("bilateral", "bilag"), ("prisene", "prissammenstilling"), ("standhaftig", "standard"), ("krav", "kraft"), ("pris", "pri"), ): assert consume.tokens_match(left, right) == consume.tokens_match(left, right, stems=None)