open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Browse source 
Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

134
src/llm_ingestion_okf/consume.py
View file

@ -745,9 +745,36 @@ def is_identifier(token: str) -> bool:
return _IDENTIFIER_RE.fullmatch(token) is not None
def tokens_match(left: str, right: str) -> bool:
def tokens_match(left: str, right: str, *, stems: frozenset[str] | None = None) -> bool:
"""Whether two tokens share a leading prefix of at least `MIN_SHARED_PREFIX`.
**THE SHARED PREFIX MUST BE A WORD** when `stems` is given, and that is
round 10's repair. The floor alone matched four characters that are not a
stem at all: measured on the pinned 453-concept bundle with the control run
first, `under` occurs 79 times by equality and matches 172 concepts by
prefix, while `undersjoisk` occurs 0 times and matches the same 172;
`bilateral` occurs 0 times and matches 400 of 453 through `bilag`;
`standhaftig` 0 and 219 through `standard`.
Three repairs were measured and all three failed on the same row: a longer
floor (5-8), a coverage share of the question word (0.5-0.8), and a floor
applied only to long words (>= 6, 8, 10, 12). Row 1's question token
`prisene` reaches its gold document through `pris|sammenstilling` on the
four characters `pris` -- 0.57 of the question word and 0.22 of the
document word -- so the over-match and the wanted match are the same
mechanism seen from two sides, and no threshold on length or coverage
separates them.
What separates them is that `pris` is a word and `bila` is not. With
`stems`, `bilateral` falls to 0 on both bundles and every hit@8 row keeps
its rank. `undersjoisk` still reaches 162 because it shares `under`, which
IS a word here -- a genuine Norwegian morpheme, so that residual is a
different answer rather than a ceiling.
The vocabulary is the BUNDLE's own, which makes a payload depend on the
corpus the way `rarity_weights` already does. Passing `None` reproduces the
pre-round-10 matcher exactly.
Symmetric, and it degrades to equality for short tokens: two 4-character
tokens match only if they are the same word.
@ -778,7 +805,15 @@ def tokens_match(left: str, right: str) -> bool:
shared = 0
while shared < limit and left[shared] == right[shared]:
shared += 1
return shared >= MIN_SHARED_PREFIX
if shared < MIN_SHARED_PREFIX:
return False
if stems is None:
return True
# Equality first, and only inside this branch. A token always answers to
# itself, whatever the corpus contains -- but the check must NOT move above
# the floor, where it would make `veg`/`veg` match and change the shipped
# rule for every token shorter than `MIN_SHARED_PREFIX`.
return left == right or left[:shared] in stems
#: One declared vocabulary family, spelled once: within it, any term answers to
@ -846,7 +881,12 @@ def searchable_text(concepts: Sequence["Concept"]) -> list[str]:
]
def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dict[str, float]:
def rarity_weights(
question_tokens: Sequence[str],
corpus: Sequence[str],
*,
stems: frozenset[str] | None = None,
) -> dict[str, float]:
"""What one hit on each question token is worth, from the bundle alone.
`log(N / df)`: `N` concepts, and `df` the number of them bearing the token
@ -880,7 +920,7 @@ def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dic
for text in corpus:
candidate_tokens = normalise(text)
for token in counts:
if any(tokens_match(token, other) for other in candidate_tokens):
if any(tokens_match(token, other, stems=stems) for other in candidate_tokens):
counts[token] += 1
return {
token: math.log(total / count) if count else math.log(total)
@ -931,6 +971,7 @@ def _overlap(
*,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
stems: frozenset[str] | None = None,
) -> float:
"""What the candidate text answers of the question.
@ -944,7 +985,7 @@ def _overlap(
return sum(
1 if weights is None else weights.get(token, 1.0)
for token in question_tokens
if any(tokens_match(token, other) for other in candidate_tokens)
if any(tokens_match(token, other, stems=stems) for other in candidate_tokens)
or (bridged and in_cost_vocabulary(token))
)
@ -978,6 +1019,7 @@ def document_scores(
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
stems: frozenset[str] | None = None,
) -> dict[str, float]:
"""One score per top-level document, from the indexes and the paths alone.
@ -1030,6 +1072,7 @@ def document_scores(
concept_id.replace("/", " "),
cost_vocabulary=bridge,
weights=weights,
stems=stems,
),
)
for relative in indexes:
@ -1042,7 +1085,13 @@ def document_scores(
continue
record(
document,
_overlap(question_tokens, entry.label, cost_vocabulary=bridge, weights=weights),
_overlap(
question_tokens,
entry.label,
cost_vocabulary=bridge,
weights=weights,
stems=stems,
),
)
return {
document: totals[document] / units[document] ** DOCUMENT_PRIOR_EXPONENT
@ -1087,6 +1136,28 @@ RRF_K = 60
DEFAULT_TIE_SHARED_RANK = True
#: Round 10. `MIN_SHARED_PREFIX = 4` matches on four characters whether or not
#: they are a stem. Measured on the pinned 453-concept bundle, control first:
#: `bilateral` occurs 0 times by equality and matches 400 of 453 through
#: `bilag`; `standhaftig` 0 and 219 through `standard`; `undersjoisk` 0 and 172
#: through `under`. Requiring the shared prefix to occur as a token in the
#: bundle's own concepts takes the first to 0 and the second to 56 on the
#: default bundle (0 and 33 on Arm B) with every hit@8 row keeping rank 1 on
#: BOTH bundles.
#:
#: ON by measurement, not by taste, and the measurement is that the other three
#: candidates are not available: a longer floor (5-8), a coverage share
#: (0.5-0.8) and a floor for long words only (>= 8) each cost row 1 its rank on
#: the default bundle and the whole row on Arm B. Row 1 reaches its gold
#: document through `pris|sammenstilling` on the four characters `pris`, so the
#: over-match and the wanted match are one mechanism; only "is the prefix a
#: word" separates them.
#:
#: LIKE `--tie-shared-rank`, THIS MOVES A PAYLOAD WITH NO BUNDLE CHANGING. A
#: consumer pinned to the previous excerpt order needs `--no-stem-prefix`.
DEFAULT_STEM_PREFIX = True
def concept_scores(
concepts: Sequence[Concept],
question: str,
@ -1096,6 +1167,7 @@ def concept_scores(
weights: Mapping[str, float] | None = None,
lookup: bool = True,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
stems: frozenset[str] | None = None,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -1161,13 +1233,20 @@ def concept_scores(
titles[concept.concept_id],
cost_vocabulary=bridge,
weights=weights,
stems=stems,
)
)
for concept in concepts
},
{
concept.concept_id: float(
_overlap(question_tokens, concept.body, cost_vocabulary=bridge, weights=weights)
_overlap(
question_tokens,
concept.body,
cost_vocabulary=bridge,
weights=weights,
stems=stems,
)
)
for concept in concepts
},
@ -1212,8 +1291,10 @@ def concept_scores(
# was, at the price of one more pass over the same two fields.
else {
concept.concept_id: int(
_overlap(question_tokens, titles[concept.concept_id], cost_vocabulary=bridge)
+ _overlap(question_tokens, concept.body, cost_vocabulary=bridge)
_overlap(
question_tokens, titles[concept.concept_id], cost_vocabulary=bridge, stems=stems
)
+ _overlap(question_tokens, concept.body, cost_vocabulary=bridge, stems=stems)
)
for concept in concepts
}
@ -1489,6 +1570,7 @@ def build_payload(
rarity_weight: bool = False,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
withheld_titles: bool = False,
stem_prefix: bool = DEFAULT_STEM_PREFIX,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -1534,8 +1616,19 @@ def build_payload(
)
for concept_id in concept_ids
]
# The bundle's OWN vocabulary, and the reason the rule is a set rather than
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters. One
# pass, over the same text the ranking reads.
stems = (
frozenset(token for text in searchable_text(concepts) for token in normalise(text))
if stem_prefix
else None
)
weights = (
rarity_weights(normalise(question), searchable_text(concepts)) if rarity_weight else None
rarity_weights(normalise(question), searchable_text(concepts), stems=stems)
if rarity_weight
else None
)
ranked = concept_scores(
concepts,
@ -1546,10 +1639,12 @@ def build_payload(
profile=profile,
cost_vocabulary=cost_vocabulary,
weights=weights,
stems=stems,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
tie_shared_rank=tie_shared_rank,
stems=stems,
)
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
@ -1705,6 +1800,24 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
dest="tie_shared_rank",
help="The rule's explicit opt-out, reproducing the pre-2026-09-10 order",
)
parser.add_argument(
"--stem-prefix",
action="store_true",
default=DEFAULT_STEM_PREFIX,
help=(
"require a shared PREFIX to be a word the bundle uses, so four "
"coincidental characters no longer match. Measured on the pinned "
"453-concept bundle: `bilateral` occurs 0 times and reached 400 of "
"453 through `bilag`; with this it reaches 0, and every hit@8 row "
"keeps rank 1 on both bundles. ON since 2026-09-09"
),
)
parser.add_argument(
"--no-stem-prefix",
action="store_false",
dest="stem_prefix",
help="The rule's explicit opt-out, reproducing the pre-round-10 matcher",
)
parser.add_argument(
"--withheld-titles",
action="store_true",
@ -1750,6 +1863,7 @@ def main(argv: list[str] | None = None) -> int:
reserve_top_rank=args.reserve_top_rank,
rarity_weight=args.rarity_weight,
tie_shared_rank=args.tie_shared_rank,
stem_prefix=args.stem_prefix,
withheld_titles=args.withheld_titles,
)
except ConsumeError as error: