open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Browse source 
Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

49
README.md
View file

@ -146,6 +146,21 @@ pre-2026-09-10 bundle byte for byte — measured with `diff -rq`, 0 differences,
not asserted. Each line below carries the number it was measured at, and
nothing beyond it.
A **tenth** flag, `--bold-title`, is **off** by default. It is the rule for the
type whose container declares nothing: `rtf` has no heading style, so an
author's title is bold text, and the row was measured at 0 of 0 declared
headings, 0 concepts and 1368 of 1368 characters in no segment. The grammar is
markdown, not `rtf` — the converter already writes that title as `**…**` in the
same output every office row produces — and it is gated by the principle
`--outline-gate` already carries: recovery yields to declaration. Measured over
47 readable documents, three parameters were swept and one carried; false
positives are **0 of the 31 documents that declare**, and the rule reaches
**2 of 39** corpus documents, both `docx`, **0 of 33 `pdf`** and **0 of 2
`xlsx`**. On the `rtf` fixture set it recovers **6 of 6 authored titles over
N = 4** with **0** false titles and **0 of 1994** characters in no segment. On
a five-document folder it moves 26 → 27 concepts, replacing a mechanical
`tabell-linje-30` with two named concepts.
**A re-run is what this costs a consumer, and it is not a small one:** on the
43-document reference corpus the default bundle goes from **629 concepts in
1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move,
@ -290,6 +305,24 @@ than guessed: RRF fuses RANKS, so a weight moves nothing on a signal the gold
already leads. `docs/2026-09-08-sjeldenhetsvekt.md`. Its published figures were
measured under the pre-2026-09-10 tie-break and are not re-measured.
`--stem-prefix` is **on** by default since 2026-09-09 (opt out with
`--no-stem-prefix`), and like `--tie-shared-rank` below it alters a payload
with no bundle changing. `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding, and it also matches four characters that are not a stem: measured
on the pinned 453-concept bundle with the control run first, `under` occurs 79
times by equality and matches 172 concepts by prefix, while `bilateral` occurs
**0** times and matched **400 of 453** through `bilag`, and `standhaftig` 0 and
219 through `standard`. Three repairs were measured and all three failed on the
same row — a longer floor (5–8), a coverage share (0.5–0.8) and a
long-words-only floor (≥ 8) each cost row 1 its rank on the default bundle and
the whole row on Arm B, because row 1's token `prisene` reaches its gold
document through `pris|sammenstilling` on four characters. The rule that works
asks whether the shared prefix is a WORD the bundle uses: `bilateral` 400 → 0
and 512 → 0, `standhaftig` 219 → 56 and 235 → 33, **every hit@8 row keeping
rank 1 on both bundles**. `undersjøisk` stops at 162 because `under` is a word
here — a genuine Norwegian morpheme, so the residual is a different answer and
not a ceiling.
`--tie-shared-rank` is **on** by default since 2026-09-10 (opt out with
`--no-tie-shared-rank`), and it is the one change in this library that alters a
payload with no bundle changing — a consumer pinned to the previous excerpt
@ -674,11 +707,17 @@ an uninstalled extra run in both. The suite is not shipped in an installed
distribution — `tests/` lives at the repository root, so this command needs a
clone rather than a `pip install`.
**Lint is measured against the lockfile.** `uv sync --frozen` resolves ruff
0.15.22, under which `ruff check src tests tools` is clean. A loose install
resolves ruff 0.16.6, under which the same tree reports 144 lint errors and 2
files that would be reformatted — new rules, not new defects. Add
`--extra extract` to that sync or `mypy src` cannot find `pdfplumber`.
**The lint acceptance is BOTH of ruff's gates.** `ruff check src tests tools`
*and* `ruff format --check .`, both named in a report with the version they ran
under. The second was not in the acceptance until 2026-09-09, and it had gone
red unseen: an acceptance that names one of a tool's two gates reports green
for the gate it never ran. Add `--extra extract` to the sync or `mypy src`
cannot find `pdfplumber`.
**Use `uv run --frozen`, not `uv run`.** The committed lockfile predates the
`[ocr]` extra, so any unlocked resolve writes that extra's whole transitive
tree back into `uv.lock` — measured, 681 insertions over 4 deletions, pulling
in packages this project does not use.
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
install-time *channel*, not the pin: the range above stays the declared