feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions

View file

@ -13,6 +13,11 @@ this name and is the contract for the pre-pass.
No logic here, deliberately: a second copy of the ranking or the cut is a
second thing that can be right while the shipped one is wrong.
**IF YOU IMPORT THIS FILE BY PATH** (`importlib.util.spec_from_file_location`),
read the module back out of `sys.modules[<name>]` after `exec_module`, or import
`llm_ingestion_okf.consume` directly: the last line of this file replaces the
REGISTRY entry, which is not the object a path importer holds.
"""
from __future__ import annotations
@ -27,6 +32,24 @@ from llm_ingestion_okf import consume as _impl # noqa: E402
if __name__ == "__main__":
raise SystemExit(_impl.main())
# A PATH IMPORTER NEVER SEES THE LINE BELOW, so its names are copied here.
# `sys.modules[__name__] = _impl` replaces the REGISTRY entry; a caller using
# `importlib.util.spec_from_file_location` already holds the object
# `module_from_spec` made, and `exec_module` leaves that object with this
# file's own globals -- 3 public names against the packaged module's 76.
# Reported by `vegnormal-okf` after v0.7.0 and reproduced in
# `tests/test_okf_consume_shim.py` under two counting methods.
#
# THE DUNDER FILTER IS LOAD-BEARING. An unfiltered `vars(_impl)` overwrites
# `__name__` with `llm_ingestion_okf.consume`, and the next statement uses
# `__name__` as the `sys.modules` key -- the module would be aliased under the
# wrong one.
#
# This restores attribute ACCESS and NOT patch-through: a caller monkeypatching
# one of these copies patches a binding the implementation never reads. That is
# precisely why the alias below exists, and it is why the alias stays.
globals().update({k: v for k, v in vars(_impl).items() if not k.startswith("__")})
# ALIASED, not re-exported. `import okf_consume` must hand back the packaged module
# ITSELF: a re-export binds copies of the names into a second module object, so
# a caller patching one of them patches a binding the implementation never