Kjell Tore Guttormsen
191de89f41
Three of round 9's four measured holes, each closed with a rule chosen on a measurement rather than named as a limit. `rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has no heading style, so the author's title is bold text. The grammar is markdown, not `rtf`: the converter already writes that title as `**...**` in the same output every office row produces, so no `rtf`-only heading form exists. Three parameters were swept over 47 readable documents and ONE carried -- refusing a line that ends in terminal punctuation takes false-positive lines from 9-12 to 1-2. A maximum title length (unlimited/40/60/80/120) and a must-stand-between-blank-lines clause are both FLAT, so neither is in the rule. The last false positive is closed by G1, the principle `_gate_outline` already carries: recovery yields to declaration. False positives are then 0 of the 31 declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39 corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind `--bold-title`, default OFF pending the hit@8 measurement; the default bundle is byte-identical without it. BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid fixture DECLARES heading styles in a stylesheet and the converter discards them, emitting the same bold line -- so "read the declared headings out of the markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX headings on that same document, because the loss is in the `rtf` READER before any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with the fasit written first; they live in their own directory because Door B walks a drop directory recursively and `k2-office/` reads its N off the listing. THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW. Re-measured on the pinned 453-concept bundle with the control run first: `under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0 and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra known-negatives were FOUND, not chosen -- every 4-character prefix ranked by document frequency, then a real word taken from the widest. A longer floor (5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost row 1 its rank on the default bundle and the whole row on Arm B. Decomposed: row 1's token `prisene` reaches its gold document through `pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the other -- so the over-match and the wanted match are one mechanism. THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0, `standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine Norwegian morpheme, so that residual is a different answer, not a ceiling. ON by default (`--no-stem-prefix`), pinned with its own known-negative on the shipped bytes. THE SHIM: a path importer holds the object `module_from_spec` made, and `sys.modules[__name__] = _impl` never reaches it. Measured under both counting methods -- 3 of 76 public names by `vars()`. One line copies the public names into this file's globals; the dunder filter is load-bearing, because an unfiltered copy overwrites `__name__` before the next line uses it as the alias key. It restores attribute ACCESS and not patch-through, which is why the alias stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since what the consumer asked for was the note. Suite 1515 -> 1535. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
105 lines
4.1 KiB
Python
105 lines
4.1 KiB
Python
"""A path importer holds the module object, and `sys.modules` is not it.
|
|
|
|
WHAT WAS REPORTED, AND BY WHOM. `vegnormal-okf` reported after v0.7.0 that the
|
|
shim broke a caller importing it with `importlib.util.spec_from_file_location`.
|
|
Reproduced here, and it is not a spelling mistake: `sys.modules[__name__] =
|
|
_impl` replaces the REGISTRY entry, and a path importer already holds a
|
|
different module object -- the one `module_from_spec` made and `exec_module`
|
|
ran. That object keeps whatever the file's own globals ended up with, which is
|
|
four public names, while the registry entry has ninety.
|
|
|
|
TWO COUNTING METHODS, BOTH IN THE TEST. `vars()` gives 4 against 90 and `dir()`
|
|
gives 3 against 75; the numbers differ because `dir()` on a module is sorted
|
|
and de-duplicated over a different set. Asserting only that ONE name appears
|
|
would be green over a nearly empty set -- which is how the defect survived a
|
|
release -- so the test asserts the COUNT under both methods.
|
|
|
|
WHAT THE FIX IS AND WHAT IT IS NOT. One line, copying the implementation's
|
|
public names into this module's globals BEFORE the alias. It restores attribute
|
|
ACCESS. It does NOT restore patch-through: a caller who monkeypatches the copy
|
|
patches a binding the implementation never reads, and that is exactly why the
|
|
alias exists and why it stays. The dunder filter is load-bearing -- an
|
|
unfiltered `vars(_impl)` overwrites `__name__` with
|
|
`llm_ingestion_okf.consume` before the next line reads it, and the module is
|
|
then aliased under the wrong key.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
import sys
|
|
from pathlib import Path
|
|
from types import ModuleType
|
|
|
|
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
|
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
|
|
|
SHIM = PROJECT_ROOT / "tools" / "okf_consume.py"
|
|
|
|
|
|
def _load_by_path(name: str) -> ModuleType:
|
|
"""Exactly what the reporting caller does, and nothing else."""
|
|
spec = importlib.util.spec_from_file_location(name, SHIM)
|
|
assert spec is not None and spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
saved = sys.modules.get(name)
|
|
try:
|
|
spec.loader.exec_module(module)
|
|
finally:
|
|
if saved is None:
|
|
sys.modules.pop(name, None)
|
|
else:
|
|
sys.modules[name] = saved
|
|
return module
|
|
|
|
|
|
def _public_by_vars(module: ModuleType) -> set[str]:
|
|
return {name for name in vars(module) if not name.startswith("_")}
|
|
|
|
|
|
def _public_by_dir(module: ModuleType) -> set[str]:
|
|
return {name for name in dir(module) if not name.startswith("_")}
|
|
|
|
|
|
def test_a_path_imported_shim_carries_the_implementations_public_names() -> None:
|
|
"""The defect, stated as the count rather than as one name."""
|
|
from llm_ingestion_okf import consume
|
|
|
|
module = _load_by_path("okf_consume_path_imported")
|
|
for method in (_public_by_vars, _public_by_dir):
|
|
held = method(module)
|
|
registry = method(consume)
|
|
missing = registry - held
|
|
assert not missing, (
|
|
f"{method.__name__}: the path-imported object is missing "
|
|
f"{len(missing)} of {len(registry)} public names"
|
|
)
|
|
|
|
|
|
def test_build_payload_is_reachable_on_the_path_imported_object() -> None:
|
|
"""The specific call the reporting caller makes."""
|
|
from llm_ingestion_okf import consume
|
|
|
|
module = _load_by_path("okf_consume_path_imported_two")
|
|
assert module.build_payload is consume.build_payload
|
|
|
|
|
|
def test_the_dunder_filter_leaves_the_modules_own_name_alone() -> None:
|
|
"""The known-negative for the filter: without it the alias key is wrong.
|
|
|
|
An unfiltered copy would set `__name__` to `llm_ingestion_okf.consume`,
|
|
and the very next line uses `__name__` as the `sys.modules` key.
|
|
"""
|
|
name = "okf_consume_path_imported_three"
|
|
module = _load_by_path(name)
|
|
assert module.__name__ == name
|
|
assert module.__file__ is not None and module.__file__.endswith("okf_consume.py")
|
|
|
|
|
|
def test_importing_it_by_name_still_hands_back_the_packaged_module() -> None:
|
|
"""The alias is unchanged: patch-through is what it buys and it stays."""
|
|
import okf_consume
|
|
|
|
from llm_ingestion_okf import consume
|
|
|
|
assert okf_consume is consume
|