feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions

View file

@ -152,3 +152,73 @@ def test_the_reading_default_is_what_holds_row_one_on_these_bytes() -> None:
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
assert ranks[0] == 2, "the known-negative stopped being negative"
assert tuple(ranks[1:]) == EXPECTED_RANKS[1:]
@requires_default_bundle
def test_the_stem_rule_holds_every_rank_on_the_shipped_bytes() -> None:
"""Round 10's reading-side default, pinned the way `--tie-shared-rank` is.
The rule requires a shared prefix to be a word the bundle uses. It moves a
payload with NO bundle changing, which is the same class of change as the
tie-break above, so it gets the same treatment: the ranks are pinned WITH
it, and the test below pins what it costs when it is off.
Measured on these bytes, control run first: `under` occurs 79 times by
equality and matches 172 concepts by prefix; `bilateral` occurs 0 times and
matched 400 of 453 before this rule and 0 after; `standhaftig` 0 and
219 -> 56. `undersjoisk` goes 172 -> 162 and stops there, because `under`
IS a word here -- a genuine Norwegian morpheme rather than a residual.
"""
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
ranks = []
for entry in questions:
payload = okf_consume.build_payload(DEFAULT_BUNDLE, question=entry["question"])
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
assert tuple(ranks) == EXPECTED_RANKS
@requires_default_bundle
def test_the_stem_rule_is_what_keeps_bilateral_out_of_four_hundred_concepts() -> None:
"""The known-negative for the stem rule, on the shipped bundle.
Without it the pin above would be green with no stated cause. The number
that names the cause is the one the rule was built for: a word occurring
ZERO times reaching 400 of 453 concepts.
The control is measured FIRST and in the same loop, because round 9's first
attempt at this measurement returned 0 for the probe AND for the control --
`enumerate_concepts` yields `str`, not `Path` -- and a zero that is really a
broken query is indistinguishable from a zero that is a result.
"""
root_id = okf_consume.root_bundle_id_of(DEFAULT_BUNDLE)
concepts = [
okf_consume.read_concept(
DEFAULT_BUNDLE / f"{concept_id}.md",
bundle_root=DEFAULT_BUNDLE,
root_bundle_id=root_id,
)
for concept_id in okf_consume.enumerate_concepts(DEFAULT_BUNDLE)
]
documents = [okf_consume.normalise(text) for text in okf_consume.searchable_text(concepts)]
stems = frozenset(token for document in documents for token in document)
def reach(word: str, *, stemmed: bool) -> int:
token = okf_consume.normalise(word)[0]
return sum(
1
for document in documents
for _ in (0,)
if any(
okf_consume.tokens_match(token, other, stems=stems if stemmed else None)
for other in document
)
)
assert sum(1 for d in documents if okf_consume.normalise("under")[0] in d) == 79, (
"the CONTROL first: `under` really does occur, so the query can find"
)
assert reach("bilateral", stemmed=False) == 400, "the defect, on these bytes"
assert reach("bilateral", stemmed=True) == 0, "and what the rule does to it"
assert reach("under", stemmed=True) == 172, "a real word keeps its compounds"