open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Browse source 
Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

130
tests/test_stem_prefix.py Normal file
View file

@ -0,0 +1,130 @@
"""The shared prefix has to be a WORD, or `bilateral` reaches 400 of 453.
THE FINDING, RE-MEASURED. `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding -- `vare|ne` and `vare|mottak` share a stem -- and it has a
known-positive in the suite. It also matches on four characters that are not a
stem at all. Measured on the pinned 453-concept bundle, with the control run
FIRST every time:
word equality prefix share
under (control) 79 172 38.0 %
undersjoisk 0 172 38.0 %
bilateral 0 400 88.3 %
standhaftig 0 219 48.3 %
The two extra known-negative words were FOUND rather than chosen: every
4-character prefix in the bundle was ranked by document frequency, and a real
Norwegian word was taken from the widest ones -- `bila` (400 of 453, through
`bilag`) and `stan` (219, through `standard`).
THREE CANDIDATES WERE MEASURED AND ALL THREE FAILED, each on the SAME row.
a. a longer floor (5, 6, 7, 8) row 1 falls 1 -> 2 on the default
bundle and 1 -> None on Arm B
b. coverage >= 0.5 / 0.6 / 0.7 / 0.8 every negative to 0, control still
88 > 79, and row 1 still falls
c. prefix only for words >= 6/8/10/12 only >= 6 holds the ranks, and it
fixes `bilateral` alone: 137 and
219 remain
WHY THEY ALL FAIL ON ONE ROW, decomposed rather than guessed. Row 1's question
token `prisene` reaches its gold document through `pris|sammenstilling` and
`pris|skjema` on the four characters `pris`. That is 0.57 coverage of the
question word and 0.22 of the document word -- so a coverage rule at 0.60 cuts
exactly the match that finds the price sheet. The over-match and the wanted
match are the same mechanism seen from two sides, which is why length and
coverage cannot separate them.
WHAT DOES SEPARATE THEM. `pris` is a word; `bila` and `stan` are not. The
fourth candidate requires the shared prefix to occur as a token in the bundle's
own concepts, and it is the only one that clears both criteria at once:
bundle bilateral standhaftig undersjoisk control ranks
default-20260912 400 -> 0 219 -> 56 172 -> 162 172 (1,1,1,1,1,None)
armB-20260903 512 -> 0 235 -> 33 174 -> 160 174 (1,1,1,1,1,None)
WHAT REMAINS, AND WHY IT IS NOT A CEILING BUT A DIFFERENT ANSWER. `undersjoisk`
still reaches 162 concepts, because it shares `under` with them and `under` IS
a word in this corpus -- a productive Norwegian prefix. `standhaftig` still
reaches 56 through `stand`. Those are genuine shared morphemes, and a rule that
cut them would be cutting the behaviour `MIN_SHARED_PREFIX` was built for. The
residual is a statement about Norwegian, not about the implementation.
HONESTY LIMITS: n = 6 questions, one rater, one gold set, and row 6 misses on
every bundle and every configuration measured so far. The three known-negative
words were chosen by this session from the bundle's own vocabulary, not by an
independent source. The vocabulary is the BUNDLE's, so this rule makes a
payload depend on the corpus the way `rarity_weights` already does.
"""
from __future__ import annotations
from llm_ingestion_okf import consume
#: The corpus vocabulary in miniature. `pris` and `under` stand on their own;
#: `bila` and `stan` never do, which is the whole discriminator.
STEMS = frozenset(
{
"pris",
"prisene",
"prissammenstilling",
"prisskjema",
"under",
"underbygning",
"bilag",
"standard",
"varemottak",
"varene",
"vare",
}
)
def test_a_shared_prefix_that_is_not_a_word_no_longer_matches() -> None:
"""`bila` is four characters of `bilag` and of `bilateral`, and no word."""
assert consume.tokens_match("bilateral", "bilag") is True, "the shipped rule matches"
assert consume.tokens_match("bilateral", "bilag", stems=STEMS) is False
def test_a_shared_prefix_that_is_a_word_still_matches() -> None:
"""The known-positive the repair must not kill: row 1 depends on it.
`pris` is 0.57 of `prisene` and 0.22 of `prissammenstilling`, so every
coverage rule measured cut it -- and cutting it is what cost row 1 its
rank on both bundles.
"""
assert consume.tokens_match("prisene", "prissammenstilling", stems=STEMS) is True
assert consume.tokens_match("varene", "varemottak", stems=STEMS) is True
def test_the_productive_norwegian_prefix_is_kept_and_that_is_the_answer() -> None:
"""`under` is a word, so `undersjoisk` keeps reaching `underbygning`.
Stated as a decision rather than left as a residual: this rule does not
claim to separate two words that genuinely share a morpheme, and the 162
concepts `undersjoisk` still reaches are that morpheme, not a defect.
"""
assert consume.tokens_match("undersjøisk", "underbygning", stems=STEMS) is True
def test_an_identifier_is_untouched_by_the_stem_rule() -> None:
"""The 2026-09-08 identifier fix stays exactly as it was."""
assert consume.tokens_match("3.3.1-13", "3.3.1-13", stems=STEMS) is True
assert consume.tokens_match("3.3.1-13", "3.3.1-14", stems=STEMS) is False
def test_equality_never_needs_a_stem() -> None:
"""A token always matches itself, whatever the vocabulary says."""
assert consume.tokens_match("zzzznotinvocab", "zzzznotinvocab", stems=STEMS) is True
def test_without_stems_the_function_is_byte_for_byte_the_shipped_one() -> None:
"""The opt-out reproduces the pre-round-10 matcher on its own material."""
for left, right in (
("bilateral", "bilag"),
("prisene", "prissammenstilling"),
("standhaftig", "standard"),
("krav", "kraft"),
("pris", "pri"),
):
assert consume.tokens_match(left, right) == consume.tokens_match(left, right, stems=None)