llm-ingestion-okf/tests/test_stem_prefix.py

"""The shared prefix has to be a WORD, or `bilateral` reaches 400 of 453.

THE FINDING, RE-MEASURED. `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding -- `vare|ne` and `vare|mottak` share a stem -- and it has a
known-positive in the suite. It also matches on four characters that are not a
stem at all. Measured on the pinned 453-concept bundle, with the control run
FIRST every time:

    word            equality   prefix   share
    under (control)       79      172   38.0 %
    under…            0      172   38.0 %
    bilateral              0      400   88.3 %
    standhaftig            0      219   48.3 %

The two extra known-negative words were FOUND rather than chosen: every
4-character prefix in the bundle was ranked by document frequency, and a real
Norwegian word was taken from the widest ones -- `bila` (400 of 453, through
`bilag`) and `stan` (219, through `standard`).

THREE CANDIDATES WERE MEASURED AND ALL THREE FAILED, each on the SAME row.

    a. a longer floor (5, 6, 7, 8)          row 1 falls 1 -> 2 on the default
                                            bundle and 1 -> None on Arm B
    b. coverage >= 0.5 / 0.6 / 0.7 / 0.8    every negative to 0, control still
                                            88 > 79, and row 1 still falls
    c. prefix only for words >= 6/8/10/12   only >= 6 holds the ranks, and it
                                            fixes `bilateral` alone: 137 and
                                            219 remain

WHY THEY ALL FAIL ON ONE ROW, decomposed rather than guessed. Row 1's question
token `prisene` reaches its gold document through `pris|sammenstilling` and
`pris|skjema` on the four characters `pris`. That is 0.57 coverage of the
question word and 0.22 of the document word -- so a coverage rule at 0.60 cuts
exactly the match that finds the price sheet. The over-match and the wanted
match are the same mechanism seen from two sides, which is why length and
coverage cannot separate them.

WHAT DOES SEPARATE THEM. `pris` is a word; `bila` and `stan` are not. The
fourth candidate requires the shared prefix to occur as a token in the bundle's
own concepts, and it is the only one that clears both criteria at once:

    bundle              bilateral  standhaftig  under…  control  ranks
    default-20260912    400 ->  0   219 ->  56   172 -> 162   172     (1,1,1,1,1,None)
    armB-20260903       512 ->  0   235 ->  33   174 -> 160   174     (1,1,1,1,1,None)

WHAT REMAINS, AND WHY IT IS NOT A CEILING BUT A DIFFERENT ANSWER. `under…`
still reaches 162 concepts, because it shares `under` with them and `under` IS
a word in this corpus -- a productive Norwegian prefix. `standhaftig` still
reaches 56 through `stand`. Those are genuine shared morphemes, and a rule that
cut them would be cutting the behaviour `MIN_SHARED_PREFIX` was built for. The
residual is a statement about Norwegian, not about the implementation.

HONESTY LIMITS: n = 6 questions, one rater, one gold set, and row 6 misses on
every bundle and every configuration measured so far. The three known-negative
words were chosen by this session from the bundle's own vocabulary, not by an
independent source. The vocabulary is the BUNDLE's, so this rule makes a
payload depend on the corpus the way `rarity_weights` already does.
"""

from __future__ import annotations

from llm_ingestion_okf import consume

#: The corpus vocabulary in miniature. `pris` and `under` stand on their own;
#: `bila` and `stan` never do, which is the whole discriminator.
STEMS = frozenset(
    {
        "pris",
        "prisene",
        "prissammenstilling",
        "prisskjema",
        "under",
        "underbygning",
        "bilag",
        "standard",
        "varemottak",
        "varene",
        "vare",
    }
)


def test_a_shared_prefix_that_is_not_a_word_no_longer_matches() -> None:
    """`bila` is four characters of `bilag` and of `bilateral`, and no word."""
    assert consume.tokens_match("bilateral", "bilag") is True, "the shipped rule matches"
    assert consume.tokens_match("bilateral", "bilag", stems=STEMS) is False


def test_a_shared_prefix_that_is_a_word_still_matches() -> None:
    """The known-positive the repair must not kill: row 1 depends on it.

    `pris` is 0.57 of `prisene` and 0.22 of `prissammenstilling`, so every
    coverage rule measured cut it -- and cutting it is what cost row 1 its
    rank on both bundles.
    """
    assert consume.tokens_match("prisene", "prissammenstilling", stems=STEMS) is True
    assert consume.tokens_match("varene", "varemottak", stems=STEMS) is True


def test_the_productive_norwegian_prefix_is_kept_and_that_is_the_answer() -> None:
    """`under` is a word, so `under…` keeps reaching `underholdning`.

    Stated as a decision rather than left as a residual: this rule does not
    claim to separate two words that genuinely share a morpheme, and the 162
    concepts `under…` still reaches are that morpheme, not a defect.
    """
    assert consume.tokens_match("undertøy", "underholdning", stems=STEMS) is True


def test_an_identifier_is_untouched_by_the_stem_rule() -> None:
    """The 2026-09-08 identifier fix stays exactly as it was."""
    assert consume.tokens_match("3.3.1-13", "3.3.1-13", stems=STEMS) is True
    assert consume.tokens_match("3.3.1-13", "3.3.1-14", stems=STEMS) is False


def test_equality_never_needs_a_stem() -> None:
    """A token always matches itself, whatever the vocabulary says."""
    assert consume.tokens_match("zzzznotinvocab", "zzzznotinvocab", stems=STEMS) is True


def test_without_stems_the_function_is_byte_for_byte_the_shipped_one() -> None:
    """The opt-out reproduces the pre-round-10 matcher on its own material."""
    for left, right in (
        ("bilateral", "bilag"),
        ("prisene", "prissammenstilling"),
        ("standhaftig", "standard"),
        ("krav", "kraft"),
        ("pris", "pri"),
    ):
        assert consume.tokens_match(left, right) == consume.tokens_match(left, right, stems=None)