open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Browse source 
Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

10
CHANGELOG.md
View file

@ -13,6 +13,16 @@ The first screen an agent reads, the three shapes of request the skill answers,
and one defect that made the command the first screen recommends build a worse
bundle than the command it claims to be.
> **Note for path importers, added 2026-09-09 after a report from a consumer.**
> `tools/okf_consume.py` ends by replacing its own `sys.modules` entry with the
> packaged `llm_ingestion_okf.consume`. A caller importing it with
> `importlib.util.spec_from_file_location` holds the object `module_from_spec`
> returned, which that line does not reach: read the module back out of
> `sys.modules[<name>]` after `exec_module`, or import `llm_ingestion_okf.consume`
> directly. From the commit that adds this note the file also copies the
> implementation's public names into its own globals, so a path-imported object
> carries them -- but that restores attribute ACCESS only, never patch-through.
### Fixed
- **`okf project` built a bundle two rules behind `okf build`.** `cli.build`'s

41
CLAUDE.md
View file

@ -322,7 +322,25 @@ and fixtures, never code.
of 49. Corpus 429 -> 447 candidates, K2 436 -> 453 concepts / 832 -> 865 md
(`21af4a1aa98315cf...`), the 12-position reference label-identical in BOTH
readings and hit@8 `[1,1,1,1,1,None]` on the new bundle AND Arm B. Opt-out
`--no-contents-name`. The nine below are unchanged -- `--outline-run 3`, `--table-grid` and
`--no-contents-name`.
**An ELEVENTH flag, `--bold-title`, is OFF** (round 10, 2026-09-09): the rule
for the type whose container declares nothing. `rtf` measured 0 of 0 declared
headings, 0 concepts, 1368 of 1368 characters in no segment. The grammar is
MARKDOWN, not `rtf` -- the converter already writes the author's bold title
as `**...**` in the same output every office row produces, so no `rtf`-only
heading grammar exists, the same shape of decision as the PDF font reader's
ATX form. Three parameters swept over 47 readable documents and ONE carried
(refusing a line that ends in terminal punctuation: false-positive lines
9-12 -> 1-2); a maximum title length and a stand-between-blank-lines clause
are both FLAT and neither is in the rule. The last false positive is closed
by G1, `_gate_outline`'s own principle, so false positives are **0 of the 31
declaring documents**. Reach **2 of 39** corpus documents, both `docx`, **0
of 33 `pdf`**. **BOTH alternatives the order named were measured and
FELLED**: a hand-laid fixture that DECLARES a heading style has it discarded
by the converter, and `rtf` -> `docx` -> markdown yields 0 ATX headings on
that same document, because the loss is in the `rtf` READER before any
writer.
The nine below are unchanged -- `--outline-run 3`, `--table-grid` and
`--unit-fold` since 2026-09-08, `--drop-wrapped-outline` and
`--outline-gate` since 2026-09-09, `--sheet-section-rows`,
`--keep-table-heading` and `--first-span-from-zero` since 2026-09-10, and
@ -559,6 +577,27 @@ and fixtures, never code.
the caller's decision, because deriving a limit from the corpus was measured
and falsified — two defensible derivations, 49x apart, one of them breaking
the known-positive. `docs/2026-09-08-blindsone-laas2-budsjett-k2.md`.
**A SIXTH flag, `--stem-prefix`, is ON since 2026-09-09** and is the second
change here that alters a payload with NO bundle changing (opt-out
`--no-stem-prefix`). `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding and also matches four characters that are not a stem: on the
pinned 453-concept bundle, control first, `under` occurs 79 times by equality
and matches 172 by prefix, `bilateral` occurs **0** and matched **400 of
453** through `bilag`, `standhaftig` 0 and 219 through `standard`. The two
extra known-negatives were FOUND, not chosen -- every 4-character prefix
ranked by document frequency, then a real word taken from the widest. **Three
candidates were measured and all three failed on the SAME row**: a longer
floor (5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8).
Decomposed, row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so **the over-match and the wanted match are one mechanism** and no
threshold on length or coverage separates them. The fourth candidate does:
the shared prefix must be a WORD the bundle uses. `bilateral` 400 -> 0 and
512 -> 0, `standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank
1 on BOTH bundles. `undersjoisk` stops at **162** because `under` IS a word
here -- a genuine Norwegian morpheme, so that residual is a different answer,
never a ceiling. The vocabulary is the BUNDLE's own, so the rule makes a
payload corpus-dependent the way `rarity_weights` already is.
`--rarity-weight` is the third: each lexical hit weighs `log(N/df)` over the
bundle's own concepts instead of 1, so an identifier is not worth what a
common verb is worth. It enters the RANKING and never the GATE — `lexical`

49
README.md
View file

@ -146,6 +146,21 @@ pre-2026-09-10 bundle byte for byte — measured with `diff -rq`, 0 differences,
not asserted. Each line below carries the number it was measured at, and
nothing beyond it.
A **tenth** flag, `--bold-title`, is **off** by default. It is the rule for the
type whose container declares nothing: `rtf` has no heading style, so an
author's title is bold text, and the row was measured at 0 of 0 declared
headings, 0 concepts and 1368 of 1368 characters in no segment. The grammar is
markdown, not `rtf` — the converter already writes that title as `**…**` in the
same output every office row produces — and it is gated by the principle
`--outline-gate` already carries: recovery yields to declaration. Measured over
47 readable documents, three parameters were swept and one carried; false
positives are **0 of the 31 documents that declare**, and the rule reaches
**2 of 39** corpus documents, both `docx`, **0 of 33 `pdf`** and **0 of 2
`xlsx`**. On the `rtf` fixture set it recovers **6 of 6 authored titles over
N = 4** with **0** false titles and **0 of 1994** characters in no segment. On
a five-document folder it moves 26 → 27 concepts, replacing a mechanical
`tabell-linje-30` with two named concepts.
**A re-run is what this costs a consumer, and it is not a small one:** on the
43-document reference corpus the default bundle goes from **629 concepts in
1108 files** (the 2026-09-03 tree) to **492 in 944** after the 2026-09-08 move,
@ -290,6 +305,24 @@ than guessed: RRF fuses RANKS, so a weight moves nothing on a signal the gold
already leads. `docs/2026-09-08-sjeldenhetsvekt.md`. Its published figures were
measured under the pre-2026-09-10 tie-break and are not re-measured.
`--stem-prefix` is **on** by default since 2026-09-09 (opt out with
`--no-stem-prefix`), and like `--tie-shared-rank` below it alters a payload
with no bundle changing. `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding, and it also matches four characters that are not a stem: measured
on the pinned 453-concept bundle with the control run first, `under` occurs 79
times by equality and matches 172 concepts by prefix, while `bilateral` occurs
**0** times and matched **400 of 453** through `bilag`, and `standhaftig` 0 and
219 through `standard`. Three repairs were measured and all three failed on the
same row — a longer floor (5–8), a coverage share (0.5–0.8) and a
long-words-only floor (≥ 8) each cost row 1 its rank on the default bundle and
the whole row on Arm B, because row 1's token `prisene` reaches its gold
document through `pris|sammenstilling` on four characters. The rule that works
asks whether the shared prefix is a WORD the bundle uses: `bilateral` 400 → 0
and 512 → 0, `standhaftig` 219 → 56 and 235 → 33, **every hit@8 row keeping
rank 1 on both bundles**. `undersjøisk` stops at 162 because `under` is a word
here — a genuine Norwegian morpheme, so the residual is a different answer and
not a ceiling.
`--tie-shared-rank` is **on** by default since 2026-09-10 (opt out with
`--no-tie-shared-rank`), and it is the one change in this library that alters a
payload with no bundle changing — a consumer pinned to the previous excerpt
@ -674,11 +707,17 @@ an uninstalled extra run in both. The suite is not shipped in an installed
distribution — `tests/` lives at the repository root, so this command needs a
clone rather than a `pip install`.
**Lint is measured against the lockfile.** `uv sync --frozen` resolves ruff
0.15.22, under which `ruff check src tests tools` is clean. A loose install
resolves ruff 0.16.6, under which the same tree reports 144 lint errors and 2
files that would be reformatted — new rules, not new defects. Add
`--extra extract` to that sync or `mypy src` cannot find `pdfplumber`.
**The lint acceptance is BOTH of ruff's gates.** `ruff check src tests tools`
*and* `ruff format --check .`, both named in a report with the version they ran
under. The second was not in the acceptance until 2026-09-09, and it had gone
red unseen: an acceptance that names one of a tool's two gates reports green
for the gate it never ran. Add `--extra extract` to the sync or `mypy src`
cannot find `pdfplumber`.
**Use `uv run --frozen`, not `uv run`.** The committed lockfile predates the
`[ocr]` extra, so any unlocked resolve writes that extra's whole transitive
tree back into `uv.lock` — measured, 681 insertions over 4 deletions, pulling
in packages this project does not use.
A git URL is a PEP 508 direct reference and pins one exact tag, so it is an
install-time *channel*, not the pin: the range above stays the declared

35
src/llm_ingestion_okf/cli.py
View file

@ -183,6 +183,15 @@ DEFAULT_CLOSE_SPAN_GAPS = True
#: them from a contents entry. It only ever REMOVES members from a run, so it
#: can only add concepts, never take one away. Opt-out `--no-contents-name`.
DEFAULT_CONTENTS_NAME = True
#: Round 10's rule for the type that declares nothing. `rtf` came back at 0 of
#: 0 declared headings, 0 concepts and 1368 of 1368 characters in no segment:
#: the container has no heading style, so the author's title is bold text.
#: The grammar is markdown, not `rtf` -- the converter already writes that
#: title as `**...**` in the same output every office row produces -- and it is
#: gated by the principle Arm D already carries, that recovery yields to
#: declaration. Three parameters were swept over 47 readable documents and one
#: carried (see `tests/test_bold_title.py`). Default set by measurement below.
DEFAULT_BOLD_TITLE = False
#: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and
#: 6 by a RETRIEVAL regression rather than by the reference: they take the
@ -250,6 +259,7 @@ def _propose_plans(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -287,6 +297,7 @@ def _propose_plans(
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -323,6 +334,7 @@ def build(
first_span_from_zero: bool = DEFAULT_FIRST_SPAN_FROM_ZERO,
close_span_gaps: bool = DEFAULT_CLOSE_SPAN_GAPS,
contents_name: bool = DEFAULT_CONTENTS_NAME,
bold_title: bool = DEFAULT_BOLD_TITLE,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
@ -394,6 +406,7 @@ def build(
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -725,6 +738,27 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"pre-2026-09-09 contents-run predicate byte for byte"
),
)
build_parser.add_argument(
"--bold-title",
action="store_true",
default=DEFAULT_BOLD_TITLE,
help=(
"Read a line that is ONE bold span as a title, in a document that "
"declares no heading of its own. For the type whose container has "
"no heading style at all: `rtf` reached round 9 at 0 of 0 declared "
"headings, 0 concepts and 1368 of 1368 characters in no segment. "
"The grammar is markdown, so it reaches every type the converter "
"writes bold for, and it is inert for `pdf`, which never goes "
"through the converter. Measured over 47 readable documents: 0 "
"false positives on the 31 that declare, by the gate"
),
)
build_parser.add_argument(
"--no-bold-title",
action="store_false",
dest="bold_title",
help="The rule's explicit opt-out",
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font", "font-reserve"),
@ -799,6 +833,7 @@ def main(argv: list[str] | None = None) -> int:
first_span_from_zero=args.first_span_from_zero,
close_span_gaps=args.close_span_gaps,
contents_name=args.contents_name,
bold_title=args.bold_title,
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,

134
src/llm_ingestion_okf/consume.py
View file

@ -745,9 +745,36 @@ def is_identifier(token: str) -> bool:
return _IDENTIFIER_RE.fullmatch(token) is not None
def tokens_match(left: str, right: str) -> bool:
def tokens_match(left: str, right: str, *, stems: frozenset[str] | None = None) -> bool:
"""Whether two tokens share a leading prefix of at least `MIN_SHARED_PREFIX`.
**THE SHARED PREFIX MUST BE A WORD** when `stems` is given, and that is
round 10's repair. The floor alone matched four characters that are not a
stem at all: measured on the pinned 453-concept bundle with the control run
first, `under` occurs 79 times by equality and matches 172 concepts by
prefix, while `undersjoisk` occurs 0 times and matches the same 172;
`bilateral` occurs 0 times and matches 400 of 453 through `bilag`;
`standhaftig` 0 and 219 through `standard`.
Three repairs were measured and all three failed on the same row: a longer
floor (5-8), a coverage share of the question word (0.5-0.8), and a floor
applied only to long words (>= 6, 8, 10, 12). Row 1's question token
`prisene` reaches its gold document through `pris|sammenstilling` on the
four characters `pris` -- 0.57 of the question word and 0.22 of the
document word -- so the over-match and the wanted match are the same
mechanism seen from two sides, and no threshold on length or coverage
separates them.
What separates them is that `pris` is a word and `bila` is not. With
`stems`, `bilateral` falls to 0 on both bundles and every hit@8 row keeps
its rank. `undersjoisk` still reaches 162 because it shares `under`, which
IS a word here -- a genuine Norwegian morpheme, so that residual is a
different answer rather than a ceiling.
The vocabulary is the BUNDLE's own, which makes a payload depend on the
corpus the way `rarity_weights` already does. Passing `None` reproduces the
pre-round-10 matcher exactly.
Symmetric, and it degrades to equality for short tokens: two 4-character
tokens match only if they are the same word.
@ -778,7 +805,15 @@ def tokens_match(left: str, right: str) -> bool:
shared = 0
while shared < limit and left[shared] == right[shared]:
shared += 1
return shared >= MIN_SHARED_PREFIX
if shared < MIN_SHARED_PREFIX:
return False
if stems is None:
return True
# Equality first, and only inside this branch. A token always answers to
# itself, whatever the corpus contains -- but the check must NOT move above
# the floor, where it would make `veg`/`veg` match and change the shipped
# rule for every token shorter than `MIN_SHARED_PREFIX`.
return left == right or left[:shared] in stems
#: One declared vocabulary family, spelled once: within it, any term answers to
@ -846,7 +881,12 @@ def searchable_text(concepts: Sequence["Concept"]) -> list[str]:
]
def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dict[str, float]:
def rarity_weights(
question_tokens: Sequence[str],
corpus: Sequence[str],
*,
stems: frozenset[str] | None = None,
) -> dict[str, float]:
"""What one hit on each question token is worth, from the bundle alone.
`log(N / df)`: `N` concepts, and `df` the number of them bearing the token
@ -880,7 +920,7 @@ def rarity_weights(question_tokens: Sequence[str], corpus: Sequence[str]) -> dic
for text in corpus:
candidate_tokens = normalise(text)
for token in counts:
if any(tokens_match(token, other) for other in candidate_tokens):
if any(tokens_match(token, other, stems=stems) for other in candidate_tokens):
counts[token] += 1
return {
token: math.log(total / count) if count else math.log(total)
@ -931,6 +971,7 @@ def _overlap(
*,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
stems: frozenset[str] | None = None,
) -> float:
"""What the candidate text answers of the question.
@ -944,7 +985,7 @@ def _overlap(
return sum(
1 if weights is None else weights.get(token, 1.0)
for token in question_tokens
if any(tokens_match(token, other) for other in candidate_tokens)
if any(tokens_match(token, other, stems=stems) for other in candidate_tokens)
or (bridged and in_cost_vocabulary(token))
)
@ -978,6 +1019,7 @@ def document_scores(
profile: BundleProfile = DEFAULT_PROFILE,
cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None,
stems: frozenset[str] | None = None,
) -> dict[str, float]:
"""One score per top-level document, from the indexes and the paths alone.
@ -1030,6 +1072,7 @@ def document_scores(
concept_id.replace("/", " "),
cost_vocabulary=bridge,
weights=weights,
stems=stems,
),
)
for relative in indexes:
@ -1042,7 +1085,13 @@ def document_scores(
continue
record(
document,
_overlap(question_tokens, entry.label, cost_vocabulary=bridge, weights=weights),
_overlap(
question_tokens,
entry.label,
cost_vocabulary=bridge,
weights=weights,
stems=stems,
),
)
return {
document: totals[document] / units[document] ** DOCUMENT_PRIOR_EXPONENT
@ -1087,6 +1136,28 @@ RRF_K = 60
DEFAULT_TIE_SHARED_RANK = True
#: Round 10. `MIN_SHARED_PREFIX = 4` matches on four characters whether or not
#: they are a stem. Measured on the pinned 453-concept bundle, control first:
#: `bilateral` occurs 0 times by equality and matches 400 of 453 through
#: `bilag`; `standhaftig` 0 and 219 through `standard`; `undersjoisk` 0 and 172
#: through `under`. Requiring the shared prefix to occur as a token in the
#: bundle's own concepts takes the first to 0 and the second to 56 on the
#: default bundle (0 and 33 on Arm B) with every hit@8 row keeping rank 1 on
#: BOTH bundles.
#:
#: ON by measurement, not by taste, and the measurement is that the other three
#: candidates are not available: a longer floor (5-8), a coverage share
#: (0.5-0.8) and a floor for long words only (>= 8) each cost row 1 its rank on
#: the default bundle and the whole row on Arm B. Row 1 reaches its gold
#: document through `pris|sammenstilling` on the four characters `pris`, so the
#: over-match and the wanted match are one mechanism; only "is the prefix a
#: word" separates them.
#:
#: LIKE `--tie-shared-rank`, THIS MOVES A PAYLOAD WITH NO BUNDLE CHANGING. A
#: consumer pinned to the previous excerpt order needs `--no-stem-prefix`.
DEFAULT_STEM_PREFIX = True
def concept_scores(
concepts: Sequence[Concept],
question: str,
@ -1096,6 +1167,7 @@ def concept_scores(
weights: Mapping[str, float] | None = None,
lookup: bool = True,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
stems: frozenset[str] | None = None,
) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF.
@ -1161,13 +1233,20 @@ def concept_scores(
titles[concept.concept_id],
cost_vocabulary=bridge,
weights=weights,
stems=stems,
)
)
for concept in concepts
},
{
concept.concept_id: float(
_overlap(question_tokens, concept.body, cost_vocabulary=bridge, weights=weights)
_overlap(
question_tokens,
concept.body,
cost_vocabulary=bridge,
weights=weights,
stems=stems,
)
)
for concept in concepts
},
@ -1212,8 +1291,10 @@ def concept_scores(
# was, at the price of one more pass over the same two fields.
else {
concept.concept_id: int(
_overlap(question_tokens, titles[concept.concept_id], cost_vocabulary=bridge)
+ _overlap(question_tokens, concept.body, cost_vocabulary=bridge)
_overlap(
question_tokens, titles[concept.concept_id], cost_vocabulary=bridge, stems=stems
)
+ _overlap(question_tokens, concept.body, cost_vocabulary=bridge, stems=stems)
)
for concept in concepts
}
@ -1489,6 +1570,7 @@ def build_payload(
rarity_weight: bool = False,
tie_shared_rank: bool = DEFAULT_TIE_SHARED_RANK,
withheld_titles: bool = False,
stem_prefix: bool = DEFAULT_STEM_PREFIX,
) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload.
@ -1534,8 +1616,19 @@ def build_payload(
)
for concept_id in concept_ids
]
# The bundle's OWN vocabulary, and the reason the rule is a set rather than
# a threshold: `pris` is a word here and `bila` is not, which is what
# separates a Norwegian compound from four coincidental characters. One
# pass, over the same text the ranking reads.
stems = (
frozenset(token for text in searchable_text(concepts) for token in normalise(text))
if stem_prefix
else None
)
weights = (
rarity_weights(normalise(question), searchable_text(concepts)) if rarity_weight else None
rarity_weights(normalise(question), searchable_text(concepts), stems=stems)
if rarity_weight
else None
)
ranked = concept_scores(
concepts,
@ -1546,10 +1639,12 @@ def build_payload(
profile=profile,
cost_vocabulary=cost_vocabulary,
weights=weights,
stems=stems,
),
cost_vocabulary=cost_vocabulary,
weights=weights,
tie_shared_rank=tie_shared_rank,
stems=stems,
)
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0)
@ -1705,6 +1800,24 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
dest="tie_shared_rank",
help="The rule's explicit opt-out, reproducing the pre-2026-09-10 order",
)
parser.add_argument(
"--stem-prefix",
action="store_true",
default=DEFAULT_STEM_PREFIX,
help=(
"require a shared PREFIX to be a word the bundle uses, so four "
"coincidental characters no longer match. Measured on the pinned "
"453-concept bundle: `bilateral` occurs 0 times and reached 400 of "
"453 through `bilag`; with this it reaches 0, and every hit@8 row "
"keeps rank 1 on both bundles. ON since 2026-09-09"
),
)
parser.add_argument(
"--no-stem-prefix",
action="store_false",
dest="stem_prefix",
help="The rule's explicit opt-out, reproducing the pre-round-10 matcher",
)
parser.add_argument(
"--withheld-titles",
action="store_true",
@ -1750,6 +1863,7 @@ def main(argv: list[str] | None = None) -> int:
reserve_top_rank=args.reserve_top_rank,
rarity_weight=args.rarity_weight,
tie_shared_rank=args.tie_shared_rank,
stem_prefix=args.stem_prefix,
withheld_titles=args.withheld_titles,
)
except ConsumeError as error:

100
src/llm_ingestion_okf/propose.py
View file

@ -123,6 +123,18 @@ RULE_TABLE_GRID = "rule:table-grid"
#: the two compose in one order: Arm E decides how far a block extends, this
#: decides where it is cut inside.
RULE_SHEET_SECTION = "rule:sheet-section"
#: Round 10 only, and its axis is a fifth one. Arm C names SIZE, Arm D what the
#: DOCUMENT declared, Arm E what the CONVERTER emitted, D3 what the SHEET
#: labelled; this names what the AUTHOR set in bold where the container gave
#: them no heading style to declare with. `rtf` is the row that forced it -- it
#: has no heading style at all -- but the grammar is markdown, not `rtf`, and
#: the rule reaches every type the converter writes bold for.
#:
#: It is a RECOVERY, like Arm D, so it carries Arm D's gate: a document that
#: declares a heading of its own admits none of these. Measured over 47
#: readable documents, that gate is what takes the false-positive count to 0
#: of the 31 that declare.
RULE_BOLD_TITLE = "rule:bold-title"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -131,6 +143,7 @@ RULE_NAMES = (
RULE_OUTLINE,
RULE_TABLE_GRID,
RULE_SHEET_SECTION,
RULE_BOLD_TITLE,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -246,6 +259,20 @@ _SHEET_SECTION_LABEL = re.compile(r"^\d+(?:[+./-]\d+)*$")
# A pipe that pandoc did not escape. Splitting a row on a bare `|` would cut a
# cell containing a literal pipe in half and misread the FIRST cell of the row
# after it, which is the only cell this rule judges.
#: A line that is ONE bold span and nothing else. The whole discriminator of
#: round 10's rule: bold inside a paragraph is `text **bold** text`, which is
#: not a whole line, so the anchors are what separate a title from emphasis.
#: Both markers are read because the converter writes `**` and a hand-authored
#: markdown document may carry `__`.
_BOLD_LINE = re.compile(r"^\s*(?:\*\*|__)(?P<title>\S.*?)(?:\*\*|__)\s*$")
#: What a bold line must NOT end in. The one parameter of three that carried a
#: measurement: over 47 readable documents it takes false-positive lines from
#: 9-12 down to 1-2, and the candidates it drops are a contract cover page's
#: sentence fragments (`er inngatt mellom:`, `Sted og dato:`). A heading names
#: a section; a fragment set in bold ends the way a sentence does.
_TERMINAL_PUNCTUATION = (".", ":", ",", ";", "!", "?")
_UNESCAPED_PIPE = re.compile(r"(?<!\\)\|")
@ -442,6 +469,40 @@ def declares_headings(candidates: Iterable[Candidate]) -> bool:
return any(candidate.rule == RULE_HEADING for candidate in candidates)
def _gate_bold_title(
marked: list[tuple[int, Candidate]], joined: set[int]
) -> tuple[list[tuple[int, Candidate]], set[int]]:
"""G1 for round 10's rule: a document that declares admits no bold titles.
The same principle `_gate_outline` carries and the same predicate,
`declares_headings`, so there is one definition of "this document declared
a heading of its own" and not two that can drift. Applied to `marked` --
BEFORE the orphan pass -- for the reason stated there: the second pass
closes each span at the NEXT mark, so removing a mark here lets the
preceding span reach through the text it used to open, where filtering the
finished entries would leave that text in no segment at all.
There is NO G2 half. Arm D's share clause exists because a recovered
outline can carry a document whose declarations are incidental; a single
bold line in a document that already declares is not that case, and the one
false positive measured over 47 documents is exactly it. Adding a share
threshold here would be a knob no measurement asks for.
`declares_headings` counts `RULE_HEADING` alone, so admitting bold titles
can never change what this gate -- or the outline gate below it -- decides.
"""
if not declares_headings(candidate for _, candidate in marked):
return marked, joined
kept: list[tuple[int, Candidate]] = []
remap: dict[int, int] = {}
for position, entry in enumerate(marked):
if entry[1].rule == RULE_BOLD_TITLE:
continue
remap[position] = len(kept)
kept.append(entry)
return kept, {remap[p] for p in joined if p in remap}
def _gate_outline(
marked: list[tuple[int, Candidate]], joined: set[int], end_of_text: int, length: int
) -> tuple[list[tuple[int, Candidate]], set[int]]:
@ -549,6 +610,7 @@ def find_candidates(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -591,6 +653,14 @@ def find_candidates(
REMOVE members from a run, which is why it only ever rescues candidates and
never discards one the shipped rule kept.
`bold_title` is round 10's gate and it is OFF at False, where the branch is
not even evaluated. On, a line that is ONE bold span, does not end in
terminal punctuation and is not made only of stop words proposes a
boundary -- but ONLY in a document that declares no heading of its own.
It exists for `rtf`, whose container has no heading style at all, and it
reads markdown rather than `rtf`: the converter already writes the author's
bold title as `**...**` in the same output every office row produces.
`close_span_gaps` is OFF at False, where a mark removed after its
neighbour's span was closed takes that text out of the plan entirely. On,
a span runs to the next SURVIVING candidate's start and the last one runs
@ -752,6 +822,30 @@ def find_candidates(
)
continue
if bold_title:
bold = _BOLD_LINE.match(line)
if bold is not None:
title = bold.group("title")
# An inner marker means the line is two or more bold RUNS with
# text between them, not one title set in bold.
nested = "**" in title or "__" in title
fragment = title.rstrip().endswith(_TERMINAL_PUNCTUATION)
if not nested and not fragment and not _is_stop_word_only(title):
marked.append(
(
index,
Candidate(
title=title,
level=1,
number=None,
rule=RULE_BOLD_TITLE,
start=offsets[index],
end=end_of_text,
),
)
)
continue
atx = _ATX.match(line)
numbered = _NUMBERED.match(line)
if atx is None and numbered is None:
@ -788,6 +882,8 @@ def find_candidates(
)
)
if bold_title:
marked, joined = _gate_bold_title(marked, joined)
if outline_gate:
marked, joined = _gate_outline(marked, joined, end_of_text, len(text))
@ -1232,6 +1328,7 @@ def build_plan(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1249,6 +1346,7 @@ def build_plan(
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -1326,6 +1424,7 @@ def run(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -1402,6 +1501,7 @@ def run(
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan

59
tests/fixtures/k2-office-fasit.json vendored
View file

@ -38,5 +38,64 @@
"strings": 59,
"container": "flat stream: title paragraph, intro paragraph, 20 table rows, caption paragraph, 4 table rows"
}
},
"rtf_variants": {
"_note": [
"Hand-counted 2026-09-09 from the AUTHORED content in",
"make_k2_office.py, BEFORE round 10's bold-title rule was measured",
"against it -- never from a converter's output. `authored_titles` is",
"what a reader of the source would call a section name; it is NOT a",
"count of what any container manages to declare, which is the whole",
"point: rtf has no heading style, so the author's title is bold text.",
"`bold_runs_that_are_not_titles` is the known-negative denominator."
],
"counted_by": "hand, from tests/fixtures/make_k2_office.py",
"counted_at": "2026-09-09",
"documents": {
"krav-rikt-tekstformat.rtf": {
"authored_titles": 1,
"titles": [
"Kravspesifikasjon for tunnelbelysning"
],
"bold_runs_that_are_not_titles": 0,
"declares_heading_style": false,
"container": "title bold AND at 16pt, then intro, the 20-row table, the caption and the 4x4 grid"
},
"krav-rikt-tekstformat-fet-alene.rtf": {
"authored_titles": 2,
"titles": [
"Kravspesifikasjon for tunnelbelysning",
"Luminanskrav per trafikklasse"
],
"bold_runs_that_are_not_titles": 0,
"declares_heading_style": false,
"container": "both titles bold at BODY size, no point-size change anywhere"
},
"krav-rikt-tekstformat-fet-i-avsnitt.rtf": {
"authored_titles": 1,
"titles": [
"Kravspesifikasjon for tunnelbelysning"
],
"bold_runs_that_are_not_titles": 1,
"declares_heading_style": false,
"container": "one bold title, then a sentence carrying one bold phrase that continues past it"
},
"krav-rikt-tekstformat-stil.rtf": {
"authored_titles": 2,
"titles": [
"Kravspesifikasjon for tunnelbelysning",
"Luminanskrav per trafikklasse"
],
"bold_runs_that_are_not_titles": 0,
"declares_heading_style": true,
"container": "a stylesheet declaring \\s1 and \\s2, applied to the two title paragraphs"
}
},
"totals": {
"documents": 4,
"authored_titles": 6,
"bold_runs_that_are_not_titles": 1,
"documents_declaring_a_heading_style": 1
}
}
}

5
tests/fixtures/k2-rtf-variants/krav-rikt-tekstformat-fet-alene.rtf vendored Normal file
View file

@ -0,0 +1,5 @@
{\rtf1\ansi\ansicpg1252\deff0{\fonttbl{\f0\froman Times New Roman;}}\pard\sa180\b Kravspesifikasjon for tunnelbelysning\b0\par
\pard\sa180 Dokumentet samler kravene til belysning i vegtunneler over 500 meter.\par
\pard\sa180\b Luminanskrav per trafikklasse\b0\par
\pard\sa180 Terskelluminansen er 145 candela og gjelder i hele overgangssonen.\par
}

4
tests/fixtures/k2-rtf-variants/krav-rikt-tekstformat-fet-i-avsnitt.rtf vendored Normal file
View file

@ -0,0 +1,4 @@
{\rtf1\ansi\ansicpg1252\deff0{\fonttbl{\f0\froman Times New Roman;}}\pard\sa180\b Kravspesifikasjon for tunnelbelysning\b0\par
\pard\sa180 Dokumentet samler kravene til belysning i vegtunneler over 500 meter.\par
\pard\sa180 Avvik fra kravene krever soknad om \b fravik\b0 fra byggherren for arbeidet starter.\par
}

5
tests/fixtures/k2-rtf-variants/krav-rikt-tekstformat-stil.rtf vendored Normal file
View file

@ -0,0 +1,5 @@
{\rtf1\ansi\ansicpg1252\deff0{\fonttbl{\f0\froman Times New Roman;}}{\stylesheet{\s0 Normal;}{\s1\b\fs32 heading 1;}{\s2\b\fs28 heading 2;}}\pard\s1\b\fs32 Kravspesifikasjon for tunnelbelysning\b0\fs24\par
\pard\s0\sa180 Dokumentet samler kravene til belysning i vegtunneler over 500 meter.\par
\pard\s2\b\fs28 Luminanskrav per trafikklasse\b0\fs24\par
\pard\s0\sa180 Terskelluminansen er 145 candela og gjelder i hele overgangssonen.\par
}

110
tests/fixtures/make_k2_office.py vendored
View file

@ -26,6 +26,11 @@ from pathlib import Path
HERE = Path(__file__).parent
OUT = HERE / "k2-office"
#: Round 10's rtf variants live in their OWN directory, and that is not tidiness.
#: `test_k2_office_fixtures.py` reads N off `k2-office/` by listing it, because
#: Door B walks a drop directory RECURSIVELY -- anything parked beside the three
#: containers would enter that run and the denominator would stop being three.
RTF_OUT = HERE / "k2-rtf-variants"
TITLE = "Kravspesifikasjon for tunnelbelysning"
INTRO = "Dokumentet samler kravene til belysning i vegtunneler over 500 meter."
@ -307,6 +312,95 @@ def _rtf_row(row: tuple[str, ...]) -> str:
return "\\trowd\\trgaph108" + borders + "\n" + cells + "\\row\n"
# --- the three rtf VARIANTS round 10 added ------------------------------------
#
# WHY THEY EXIST. Round 9 measured the `rtf` row on ONE document and it came
# back at 0 of 0 declared headings, 0 concepts, 1368 of 1368 characters in no
# segment. A repair proposed on N = 1 is a repair fitted to one file, so round
# 10's order required N >= 3 before any rule was written. Three lawful sources
# were offered; this is the first of them, and it is the one this repository's
# committed-fixture policy allows: hand-laid here, in the same file, with the
# fasit written BEFORE the measurement.
#
# WHAT EACH ONE VARIES, and it is one axis each:
#
# -fet-alene the title is bold at BODY size, with no point-size change
# at all. Variant A (`krav-rikt-tekstformat.rtf`) sets its
# title bold AND at 16pt, so without this document a rule
# reading "bold" could be passing on the size instead.
# -fet-i-avsnitt a bold PHRASE inside a running sentence, which must NOT
# become a boundary. The document's own known-negative.
# -stil the container DECLARES heading styles (`\\s1`, `\\s2`) in a
# stylesheet. This is the control for the whole repair, and
# it is a control that FAILED in an informative direction:
# measured, the vendored converter discards the style and
# emits the same bold line, so "read the declared style"
# is not a route that exists for `rtf`. The fixture is kept
# precisely because it pins that.
#
# The bodies are deliberately short. These documents answer where a boundary
# is, not how much text survives -- variant A already carries the coverage and
# fidelity counts for the container.
RTF_VARIANT_TITLE = "Kravspesifikasjon for tunnelbelysning"
RTF_VARIANT_SECTION = "Luminanskrav per trafikklasse"
RTF_VARIANT_BODY_ONE = "Dokumentet samler kravene til belysning i vegtunneler over 500 meter."
RTF_VARIANT_BODY_TWO = "Terskelluminansen er 145 candela og gjelder i hele overgangssonen."
#: The bold RUN in `-fet-i-avsnitt`, set inside a sentence that continues past
#: it. Authored as emphasis, never as a title.
RTF_VARIANT_EMPHASIS = "fravik"
def _rtf_document(paragraphs: list[str]) -> bytes:
"""Wrap hand-written paragraph bodies in the minimal rtf container."""
header = "{\\rtf1\\ansi\\ansicpg1252\\deff0{\\fonttbl{\\f0\\froman Times New Roman;}}"
return (header + "".join(paragraphs) + "}").encode("ascii")
def rtf_bold_only_bytes() -> bytes:
"""Title and section name bold at BODY size -- no point-size change."""
return _rtf_document(
[
"\\pard\\sa180\\b " + _rtf_escape(RTF_VARIANT_TITLE) + "\\b0\\par\n",
"\\pard\\sa180 " + _rtf_escape(RTF_VARIANT_BODY_ONE) + "\\par\n",
"\\pard\\sa180\\b " + _rtf_escape(RTF_VARIANT_SECTION) + "\\b0\\par\n",
"\\pard\\sa180 " + _rtf_escape(RTF_VARIANT_BODY_TWO) + "\\par\n",
]
)
def rtf_bold_inside_paragraph_bytes() -> bytes:
"""One bold title, and one bold phrase mid-sentence that is not a title."""
sentence = (
"Avvik fra kravene krever soknad om "
+ "\\b "
+ _rtf_escape(RTF_VARIANT_EMPHASIS)
+ "\\b0 "
+ " fra byggherren for arbeidet starter."
)
return _rtf_document(
[
"\\pard\\sa180\\b " + _rtf_escape(RTF_VARIANT_TITLE) + "\\b0\\par\n",
"\\pard\\sa180 " + _rtf_escape(RTF_VARIANT_BODY_ONE) + "\\par\n",
"\\pard\\sa180 " + sentence + "\\par\n",
]
)
def rtf_styled_bytes() -> bytes:
"""The container DECLARES heading styles. The converter discards them."""
stylesheet = "{\\stylesheet{\\s0 Normal;}{\\s1\\b\\fs32 heading 1;}{\\s2\\b\\fs28 heading 2;}}"
return _rtf_document(
[
stylesheet,
"\\pard\\s1\\b\\fs32 " + _rtf_escape(RTF_VARIANT_TITLE) + "\\b0\\fs24\\par\n",
"\\pard\\s0\\sa180 " + _rtf_escape(RTF_VARIANT_BODY_ONE) + "\\par\n",
"\\pard\\s2\\b\\fs28 " + _rtf_escape(RTF_VARIANT_SECTION) + "\\b0\\fs24\\par\n",
"\\pard\\s0\\sa180 " + _rtf_escape(RTF_VARIANT_BODY_TWO) + "\\par\n",
]
)
def rtf_bytes() -> bytes:
body = [
"{\\rtf1\\ansi\\ansicpg1252\\deff0",
@ -323,11 +417,15 @@ def rtf_bytes() -> bytes:
if __name__ == "__main__":
OUT.mkdir(parents=True, exist_ok=True)
RTF_OUT.mkdir(parents=True, exist_ok=True)
written = {
"krav-presentasjon.pptx": build_container(pptx_parts()),
"krav-tekstdokument.odt": build_container(odt_parts(), stored_first="mimetype"),
"krav-rikt-tekstformat.rtf": rtf_bytes(),
OUT / "krav-presentasjon.pptx": build_container(pptx_parts()),
OUT / "krav-tekstdokument.odt": build_container(odt_parts(), stored_first="mimetype"),
OUT / "krav-rikt-tekstformat.rtf": rtf_bytes(),
RTF_OUT / "krav-rikt-tekstformat-fet-alene.rtf": rtf_bold_only_bytes(),
RTF_OUT / "krav-rikt-tekstformat-fet-i-avsnitt.rtf": rtf_bold_inside_paragraph_bytes(),
RTF_OUT / "krav-rikt-tekstformat-stil.rtf": rtf_styled_bytes(),
}
for name, payload in sorted(written.items()):
(OUT / name).write_bytes(payload)
print(f"wrote {name} ({len(payload)} bytes)")
for path, payload in sorted(written.items()):
path.write_bytes(payload)
print(f"wrote {path.parent.name}/{path.name} ({len(payload)} bytes)")

242
tests/test_bold_title.py Normal file
View file

@ -0,0 +1,242 @@
"""A bold standalone line is a title where the document declares nothing else.
THE HOLE THIS CLOSES. Round 9 met `rtf` end to end for the first time and the
row came back at 0 of 0 declared headings, 0 concepts, 1368 of 1368 characters
in no segment, N = 1: the container has no heading style, so the author's title
is bold text, the proposer proposes nothing, and the document reaches Door B's
inbox as one flat concept. Content preserved, structure zero.
WHAT THE GRAMMAR READS, AND WHERE IT LIVES. Not `rtf` markup. The converter
already emits that bold title as `**Kravspesifikasjon for tunnelbelysning**` in
the SAME markdown the other office rows produce, so this rule is a markdown
rule and no `rtf`-only grammar exists -- the same shape of decision as the PDF
font reader emitting ATX rather than a PDF-only heading form. Bold INSIDE a
paragraph is `text **bold** text` and is not a whole line, which is the
discriminator the rule rests on.
THREE PARAMETERS WERE SWEPT AND ONE CARRIES. Over 47 readable documents (3
hand-built office fixtures, 5 in the operator's folder, 39 of the 43 corpus
files), with the false-positive denominator measured FIRST:
* a maximum title length (unlimited / 40 / 60 / 80 / 120 characters) is
FLAT -- identical counts at every value;
* requiring the line to stand alone between blank lines is FLAT -- the
converter already writes one paragraph per line;
* refusing a line that ends in terminal punctuation carries: false-positive
lines fall 9-12 to 1-2, and the four candidates it drops are
`er inngatt mellom:`, `Sted og dato:` and their kind -- sentence fragments
a contract's cover page sets in bold, never section names.
A parameter measured flat buys nothing, so neither of the two is in the rule.
THE GATE IS G1, ALREADY SHIPPED. One false positive survived the punctuation
clause: a corpus `docx` that DECLARES two headings of its own and sets one
later line (`EGENERKLAERING`) in bold. The repair is not a fourth parameter --
it is the principle `_gate_outline` and `fold_units` clause 2 already carry,
that recovery yields to declaration. A bold-title candidate is admitted only
where the document declares no heading of its own, which takes the
false-positive count to 0 of 31 declaring documents BY CONSTRUCTION.
WHAT WAS MEASURED AND NOT ADDED. `heading_reserve_applies` yields to an
admitted Arm D run as well as to a declaration, and that axis was measured
here before being left out: over all 4 documents that reach this rule, the
outline gate admits 0 runs and the reserve applies on every one. n = 4, which
is small; the axis is flat over it, so coupling this rule to `outline_run`
would add a knob no measurement asks for.
HONESTY LIMIT: the `rtf` row rests on hand-built documents, not corpus files.
The corpus contains ZERO `rtf`. Its evidence class stays `constructed`.
"""
from __future__ import annotations
import json
from pathlib import Path
from llm_ingestion_okf import cli
from llm_ingestion_okf.extract import extract_text
from llm_ingestion_okf.propose import RULE_BOLD_TITLE, RULE_HEADING, find_candidates
#: Variant A stays with the three-container set; round 10's three variants live
#: in their own directory, because `test_k2_office_fixtures.py` reads N off that
#: one by listing it and Door B walks a drop directory recursively.
FIXTURES = Path(__file__).parent / "fixtures" / "k2-office"
RTF_VARIANTS = Path(__file__).parent / "fixtures" / "k2-rtf-variants"
FASIT_PATH = Path(__file__).parent / "fixtures" / "k2-office-fasit.json"
DEFAULT = dict(
outline_run=cli.DEFAULT_OUTLINE_RUN,
table_grid=cli.DEFAULT_TABLE_GRID,
unit_fold=cli.DEFAULT_UNIT_FOLD,
keep_table_heading=cli.DEFAULT_KEEP_TABLE_HEADING,
sheet_section_rows=cli.DEFAULT_SHEET_SECTION_ROWS,
drop_wrapped_outline=cli.DEFAULT_DROP_WRAPPED_OUTLINE,
outline_gate=cli.DEFAULT_OUTLINE_GATE,
first_span_from_zero=cli.DEFAULT_FIRST_SPAN_FROM_ZERO,
close_span_gaps=cli.DEFAULT_CLOSE_SPAN_GAPS,
contents_name=cli.DEFAULT_CONTENTS_NAME,
)
#: The shape the converter hands back for the `rtf` fixture: a bold title, a
#: prose paragraph, then a grid table of label/value pairs.
BOLD_TITLED = """**Kravspesifikasjon for tunnelbelysning**
Dokumentet samler kravene til belysning i vegtunneler over 500 meter.
----------------------- ------------------------------
Dokumentnummer: SVV-2026-0417
Tittel: Tunnelbelysning i hovedlopet
----------------------- ------------------------------
**Luminansmatrise**
Matrisen gjelder per trafikklasse og er bindende.
"""
#: The known-negative for the whole-line clause. `**fet**` here is a phrase
#: inside a running sentence, which is what a bold RUN looks like and what the
#: rule must never read as a boundary.
BOLD_INSIDE_A_PARAGRAPH = """Dokumentet samler kravene, og et **fremhevet** ord
staar midt i en setning som fortsetter forbi det.
Neste avsnitt naevner **ogsaa** noe fremhevet i loepende tekst.
"""
#: The false-positive denominator in miniature: a document that declares its
#: own heading AND sets a later line in bold. The bold line must not become a
#: boundary here -- recovery yields to declaration.
DECLARES_AND_ALSO_BOLDS = """# Konkurransegrunnlag
Innledende avsnitt som beskriver konkurransen og dens omfang.
**EGENERKLAERING**
Leverandoren bekrefter at opplysningene i skjemaet er riktige.
"""
#: The punctuation clause's own material: a contract cover page whose bold
#: lines are sentence fragments, not section names.
BOLD_SENTENCE_FRAGMENTS = """**er inngaatt mellom:**
Byggherren og leverandoren, representert ved sine kontaktpersoner.
**Sted og dato:**
Oslo, 1. mars 2026, med signatur fra begge parter i to eksemplarer.
"""
def _fixture(name: str) -> Path:
"""Variant A in the three-container set, the three variants beside it."""
return FIXTURES / name if (FIXTURES / name).is_file() else RTF_VARIANTS / name
def _titles(text: str, **overrides: object) -> list[str]:
keywords = {**DEFAULT, **overrides}
return [c.title for c in find_candidates(text, **keywords) if c.rule == RULE_BOLD_TITLE]
def test_a_bold_standalone_line_becomes_a_title_candidate() -> None:
"""The `rtf` row's whole finding: the title the container never declared."""
assert _titles(BOLD_TITLED, bold_title=True) == [
"Kravspesifikasjon for tunnelbelysning",
"Luminansmatrise",
]
def test_bold_inside_a_paragraph_is_not_a_title_candidate() -> None:
"""The known-negative for the whole-line clause, in the same file."""
assert _titles(BOLD_INSIDE_A_PARAGRAPH, bold_title=True) == []
def test_a_declared_heading_closes_the_rule_for_the_whole_document() -> None:
"""G1: recovery yields to declaration, so the false positive cannot occur.
The document's own `# Konkurransegrunnlag` survives; the bold line below it
proposes nothing. This is the single false positive measured over the 47
documents, reduced to its mechanism.
"""
assert _titles(DECLARES_AND_ALSO_BOLDS, bold_title=True) == []
declared = [
c.title
for c in find_candidates(DECLARES_AND_ALSO_BOLDS, **{**DEFAULT, "bold_title": True})
if c.rule == RULE_HEADING
]
assert declared == ["Konkurransegrunnlag"], "the gate must not touch the declared heading"
def test_a_bold_line_ending_in_terminal_punctuation_is_not_a_title() -> None:
"""The one swept parameter that carried, pinned with its own material."""
assert _titles(BOLD_SENTENCE_FRAGMENTS, bold_title=True) == []
def test_the_rule_proposes_nothing_when_the_flag_is_off() -> None:
"""The opt-out reproduces the pre-rule bytes on the rule's own material."""
assert _titles(BOLD_TITLED, bold_title=False) == []
assert find_candidates(BOLD_TITLED, **{**DEFAULT, "bold_title": False}) == find_candidates(
BOLD_TITLED, **DEFAULT
)
def test_the_rtf_row_recovers_every_authored_title_over_four_documents() -> None:
"""The row this round exists to move, pinned against the hand-written fasit.
Round 9: 0 of 0 declared headings, 0 concepts, 1368 of 1368 characters in
no segment, N = 1. Here: 6 of 6 AUTHORED titles over N = 4, 0 false bold
titles, 0 of 1994 characters in no segment.
`authored_titles` is deliberately not a count of what the container
declares. `rtf` has no heading style, so a "declared heading" count for
this type is 0 by construction and says nothing about whether the document
has sections. The fasit counts what a reader of the SOURCE would call a
section name, hand-written before this measurement ran.
The fourth document is a control that failed informatively: it DECLARES
`\\s1` and `\\s2` in a stylesheet, and the vendored converter discards the
style and emits the same bold line. "Read the declared style" is therefore
not a route that exists for this container -- which is why the rule reads
markdown and not `rtf`.
"""
fasit = json.loads(FASIT_PATH.read_text(encoding="utf-8"))["rtf_variants"]
recovered = authored = false_titles = unsegmented = total = 0
for name, want in sorted(fasit["documents"].items()):
path = _fixture(name)
text = extract_text(path.name, path.read_bytes())
found = find_candidates(text, **{**DEFAULT, "bold_title": True})
titles = [c.title for c in found]
authored += want["authored_titles"]
recovered += sum(1 for t in want["titles"] if t in titles)
false_titles += sum(
1 for c in found if c.rule == RULE_BOLD_TITLE and c.title not in want["titles"]
)
total += len(text)
unsegmented += len(text) - sum(c.end - c.start for c in found)
assert (recovered, authored) == (6, 6), "every authored title, over four containers"
assert false_titles == 0, "the bold RUN inside a sentence must not become a title"
assert (unsegmented, total) == (0, 1994), "no character left outside a segment"
def test_the_row_is_zero_without_the_rule_which_is_round_nine_reproduced() -> None:
"""The known-negative for the row: turn the rule off and the finding returns."""
fasit = json.loads(FASIT_PATH.read_text(encoding="utf-8"))["rtf_variants"]
for name in sorted(fasit["documents"]):
path = _fixture(name)
text = extract_text(path.name, path.read_bytes())
found = find_candidates(text, **{**DEFAULT, "bold_title": False})
assert found == [], f"{name} proposes nothing without the rule"
def test_the_container_declaring_a_heading_style_still_reaches_us_as_bold() -> None:
"""The measurement that felled both alternatives the order named.
Alternative (i) was "read the declared headings out of the converter's
markdown". There are none: this document declares `\\s1`/`\\s2` and the
converter emits `**...**`. Alternative (ii) was "convert `rtf` to `docx`
and let the `docx` grammar work" -- measured separately, that yields 0 ATX
headings on this same document, because the loss happens in the `rtf`
READER before any writer sees the style.
"""
path = _fixture("krav-rikt-tekstformat-stil.rtf")
text = extract_text(path.name, path.read_bytes())
assert "**Kravspesifikasjon for tunnelbelysning**" in text
assert [line for line in text.splitlines() if line.startswith("#")] == []

70
tests/test_default_bundle_pin.py
View file

@ -152,3 +152,73 @@ def test_the_reading_default_is_what_holds_row_one_on_these_bytes() -> None:
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
assert ranks[0] == 2, "the known-negative stopped being negative"
assert tuple(ranks[1:]) == EXPECTED_RANKS[1:]
@requires_default_bundle
def test_the_stem_rule_holds_every_rank_on_the_shipped_bytes() -> None:
"""Round 10's reading-side default, pinned the way `--tie-shared-rank` is.
The rule requires a shared prefix to be a word the bundle uses. It moves a
payload with NO bundle changing, which is the same class of change as the
tie-break above, so it gets the same treatment: the ranks are pinned WITH
it, and the test below pins what it costs when it is off.
Measured on these bytes, control run first: `under` occurs 79 times by
equality and matches 172 concepts by prefix; `bilateral` occurs 0 times and
matched 400 of 453 before this rule and 0 after; `standhaftig` 0 and
219 -> 56. `undersjoisk` goes 172 -> 162 and stops there, because `under`
IS a word here -- a genuine Norwegian morpheme rather than a residual.
"""
questions = json.loads(GOLD_SET.read_text(encoding="utf-8"))["questions"]
ranks = []
for entry in questions:
payload = okf_consume.build_payload(DEFAULT_BUNDLE, question=entry["question"])
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
ranks.append(okf_consume_measure.hit_rank(excerpts, entry["gold_document"]))
assert tuple(ranks) == EXPECTED_RANKS
@requires_default_bundle
def test_the_stem_rule_is_what_keeps_bilateral_out_of_four_hundred_concepts() -> None:
"""The known-negative for the stem rule, on the shipped bundle.
Without it the pin above would be green with no stated cause. The number
that names the cause is the one the rule was built for: a word occurring
ZERO times reaching 400 of 453 concepts.
The control is measured FIRST and in the same loop, because round 9's first
attempt at this measurement returned 0 for the probe AND for the control --
`enumerate_concepts` yields `str`, not `Path` -- and a zero that is really a
broken query is indistinguishable from a zero that is a result.
"""
root_id = okf_consume.root_bundle_id_of(DEFAULT_BUNDLE)
concepts = [
okf_consume.read_concept(
DEFAULT_BUNDLE / f"{concept_id}.md",
bundle_root=DEFAULT_BUNDLE,
root_bundle_id=root_id,
)
for concept_id in okf_consume.enumerate_concepts(DEFAULT_BUNDLE)
]
documents = [okf_consume.normalise(text) for text in okf_consume.searchable_text(concepts)]
stems = frozenset(token for document in documents for token in document)
def reach(word: str, *, stemmed: bool) -> int:
token = okf_consume.normalise(word)[0]
return sum(
1
for document in documents
for _ in (0,)
if any(
okf_consume.tokens_match(token, other, stems=stems if stemmed else None)
for other in document
)
)
assert sum(1 for d in documents if okf_consume.normalise("under")[0] in d) == 79, (
"the CONTROL first: `under` really does occur, so the query can find"
)
assert reach("bilateral", stemmed=False) == 400, "the defect, on these bytes"
assert reach("bilateral", stemmed=True) == 0, "and what the rule does to it"
assert reach("under", stemmed=True) == 172, "a real word keeps its compounds"

105
tests/test_okf_consume_shim.py Normal file
View file

@ -0,0 +1,105 @@
"""A path importer holds the module object, and `sys.modules` is not it.
WHAT WAS REPORTED, AND BY WHOM. `vegnormal-okf` reported after v0.7.0 that the
shim broke a caller importing it with `importlib.util.spec_from_file_location`.
Reproduced here, and it is not a spelling mistake: `sys.modules[__name__] =
_impl` replaces the REGISTRY entry, and a path importer already holds a
different module object -- the one `module_from_spec` made and `exec_module`
ran. That object keeps whatever the file's own globals ended up with, which is
four public names, while the registry entry has ninety.
TWO COUNTING METHODS, BOTH IN THE TEST. `vars()` gives 4 against 90 and `dir()`
gives 3 against 75; the numbers differ because `dir()` on a module is sorted
and de-duplicated over a different set. Asserting only that ONE name appears
would be green over a nearly empty set -- which is how the defect survived a
release -- so the test asserts the COUNT under both methods.
WHAT THE FIX IS AND WHAT IT IS NOT. One line, copying the implementation's
public names into this module's globals BEFORE the alias. It restores attribute
ACCESS. It does NOT restore patch-through: a caller who monkeypatches the copy
patches a binding the implementation never reads, and that is exactly why the
alias exists and why it stays. The dunder filter is load-bearing -- an
unfiltered `vars(_impl)` overwrites `__name__` with
`llm_ingestion_okf.consume` before the next line reads it, and the module is
then aliased under the wrong key.
"""
from __future__ import annotations
import importlib.util
import sys
from pathlib import Path
from types import ModuleType
PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
SHIM = PROJECT_ROOT / "tools" / "okf_consume.py"
def _load_by_path(name: str) -> ModuleType:
"""Exactly what the reporting caller does, and nothing else."""
spec = importlib.util.spec_from_file_location(name, SHIM)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
saved = sys.modules.get(name)
try:
spec.loader.exec_module(module)
finally:
if saved is None:
sys.modules.pop(name, None)
else:
sys.modules[name] = saved
return module
def _public_by_vars(module: ModuleType) -> set[str]:
return {name for name in vars(module) if not name.startswith("_")}
def _public_by_dir(module: ModuleType) -> set[str]:
return {name for name in dir(module) if not name.startswith("_")}
def test_a_path_imported_shim_carries_the_implementations_public_names() -> None:
"""The defect, stated as the count rather than as one name."""
from llm_ingestion_okf import consume
module = _load_by_path("okf_consume_path_imported")
for method in (_public_by_vars, _public_by_dir):
held = method(module)
registry = method(consume)
missing = registry - held
assert not missing, (
f"{method.__name__}: the path-imported object is missing "
f"{len(missing)} of {len(registry)} public names"
)
def test_build_payload_is_reachable_on_the_path_imported_object() -> None:
"""The specific call the reporting caller makes."""
from llm_ingestion_okf import consume
module = _load_by_path("okf_consume_path_imported_two")
assert module.build_payload is consume.build_payload
def test_the_dunder_filter_leaves_the_modules_own_name_alone() -> None:
"""The known-negative for the filter: without it the alias key is wrong.
An unfiltered copy would set `__name__` to `llm_ingestion_okf.consume`,
and the very next line uses `__name__` as the `sys.modules` key.
"""
name = "okf_consume_path_imported_three"
module = _load_by_path(name)
assert module.__name__ == name
assert module.__file__ is not None and module.__file__.endswith("okf_consume.py")
def test_importing_it_by_name_still_hands_back_the_packaged_module() -> None:
"""The alias is unchanged: patch-through is what it buys and it stays."""
import okf_consume
from llm_ingestion_okf import consume
assert okf_consume is consume

130
tests/test_stem_prefix.py Normal file
View file

@ -0,0 +1,130 @@
"""The shared prefix has to be a WORD, or `bilateral` reaches 400 of 453.
THE FINDING, RE-MEASURED. `MIN_SHARED_PREFIX = 4` exists for Norwegian
compounding -- `vare|ne` and `vare|mottak` share a stem -- and it has a
known-positive in the suite. It also matches on four characters that are not a
stem at all. Measured on the pinned 453-concept bundle, with the control run
FIRST every time:
word equality prefix share
under (control) 79 172 38.0 %
undersjoisk 0 172 38.0 %
bilateral 0 400 88.3 %
standhaftig 0 219 48.3 %
The two extra known-negative words were FOUND rather than chosen: every
4-character prefix in the bundle was ranked by document frequency, and a real
Norwegian word was taken from the widest ones -- `bila` (400 of 453, through
`bilag`) and `stan` (219, through `standard`).
THREE CANDIDATES WERE MEASURED AND ALL THREE FAILED, each on the SAME row.
a. a longer floor (5, 6, 7, 8) row 1 falls 1 -> 2 on the default
bundle and 1 -> None on Arm B
b. coverage >= 0.5 / 0.6 / 0.7 / 0.8 every negative to 0, control still
88 > 79, and row 1 still falls
c. prefix only for words >= 6/8/10/12 only >= 6 holds the ranks, and it
fixes `bilateral` alone: 137 and
219 remain
WHY THEY ALL FAIL ON ONE ROW, decomposed rather than guessed. Row 1's question
token `prisene` reaches its gold document through `pris|sammenstilling` and
`pris|skjema` on the four characters `pris`. That is 0.57 coverage of the
question word and 0.22 of the document word -- so a coverage rule at 0.60 cuts
exactly the match that finds the price sheet. The over-match and the wanted
match are the same mechanism seen from two sides, which is why length and
coverage cannot separate them.
WHAT DOES SEPARATE THEM. `pris` is a word; `bila` and `stan` are not. The
fourth candidate requires the shared prefix to occur as a token in the bundle's
own concepts, and it is the only one that clears both criteria at once:
bundle bilateral standhaftig undersjoisk control ranks
default-20260912 400 -> 0 219 -> 56 172 -> 162 172 (1,1,1,1,1,None)
armB-20260903 512 -> 0 235 -> 33 174 -> 160 174 (1,1,1,1,1,None)
WHAT REMAINS, AND WHY IT IS NOT A CEILING BUT A DIFFERENT ANSWER. `undersjoisk`
still reaches 162 concepts, because it shares `under` with them and `under` IS
a word in this corpus -- a productive Norwegian prefix. `standhaftig` still
reaches 56 through `stand`. Those are genuine shared morphemes, and a rule that
cut them would be cutting the behaviour `MIN_SHARED_PREFIX` was built for. The
residual is a statement about Norwegian, not about the implementation.
HONESTY LIMITS: n = 6 questions, one rater, one gold set, and row 6 misses on
every bundle and every configuration measured so far. The three known-negative
words were chosen by this session from the bundle's own vocabulary, not by an
independent source. The vocabulary is the BUNDLE's, so this rule makes a
payload depend on the corpus the way `rarity_weights` already does.
"""
from __future__ import annotations
from llm_ingestion_okf import consume
#: The corpus vocabulary in miniature. `pris` and `under` stand on their own;
#: `bila` and `stan` never do, which is the whole discriminator.
STEMS = frozenset(
{
"pris",
"prisene",
"prissammenstilling",
"prisskjema",
"under",
"underbygning",
"bilag",
"standard",
"varemottak",
"varene",
"vare",
}
)
def test_a_shared_prefix_that_is_not_a_word_no_longer_matches() -> None:
"""`bila` is four characters of `bilag` and of `bilateral`, and no word."""
assert consume.tokens_match("bilateral", "bilag") is True, "the shipped rule matches"
assert consume.tokens_match("bilateral", "bilag", stems=STEMS) is False
def test_a_shared_prefix_that_is_a_word_still_matches() -> None:
"""The known-positive the repair must not kill: row 1 depends on it.
`pris` is 0.57 of `prisene` and 0.22 of `prissammenstilling`, so every
coverage rule measured cut it -- and cutting it is what cost row 1 its
rank on both bundles.
"""
assert consume.tokens_match("prisene", "prissammenstilling", stems=STEMS) is True
assert consume.tokens_match("varene", "varemottak", stems=STEMS) is True
def test_the_productive_norwegian_prefix_is_kept_and_that_is_the_answer() -> None:
"""`under` is a word, so `undersjoisk` keeps reaching `underbygning`.
Stated as a decision rather than left as a residual: this rule does not
claim to separate two words that genuinely share a morpheme, and the 162
concepts `undersjoisk` still reaches are that morpheme, not a defect.
"""
assert consume.tokens_match("undersjøisk", "underbygning", stems=STEMS) is True
def test_an_identifier_is_untouched_by_the_stem_rule() -> None:
"""The 2026-09-08 identifier fix stays exactly as it was."""
assert consume.tokens_match("3.3.1-13", "3.3.1-13", stems=STEMS) is True
assert consume.tokens_match("3.3.1-13", "3.3.1-14", stems=STEMS) is False
def test_equality_never_needs_a_stem() -> None:
"""A token always matches itself, whatever the vocabulary says."""
assert consume.tokens_match("zzzznotinvocab", "zzzznotinvocab", stems=STEMS) is True
def test_without_stems_the_function_is_byte_for_byte_the_shipped_one() -> None:
"""The opt-out reproduces the pre-round-10 matcher on its own material."""
for left, right in (
("bilateral", "bilag"),
("prisene", "prissammenstilling"),
("standhaftig", "standard"),
("krav", "kraft"),
("pris", "pri"),
):
assert consume.tokens_match(left, right) == consume.tokens_match(left, right, stems=None)

23
tools/okf_consume.py
View file

@ -13,6 +13,11 @@ this name and is the contract for the pre-pass.
No logic here, deliberately: a second copy of the ranking or the cut is a
second thing that can be right while the shipped one is wrong.
**IF YOU IMPORT THIS FILE BY PATH** (`importlib.util.spec_from_file_location`),
read the module back out of `sys.modules[<name>]` after `exec_module`, or import
`llm_ingestion_okf.consume` directly: the last line of this file replaces the
REGISTRY entry, which is not the object a path importer holds.
"""
from __future__ import annotations
@ -27,6 +32,24 @@ from llm_ingestion_okf import consume as _impl # noqa: E402
if __name__ == "__main__":
raise SystemExit(_impl.main())
# A PATH IMPORTER NEVER SEES THE LINE BELOW, so its names are copied here.
# `sys.modules[__name__] = _impl` replaces the REGISTRY entry; a caller using
# `importlib.util.spec_from_file_location` already holds the object
# `module_from_spec` made, and `exec_module` leaves that object with this
# file's own globals -- 3 public names against the packaged module's 76.
# Reported by `vegnormal-okf` after v0.7.0 and reproduced in
# `tests/test_okf_consume_shim.py` under two counting methods.
#
# THE DUNDER FILTER IS LOAD-BEARING. An unfiltered `vars(_impl)` overwrites
# `__name__` with `llm_ingestion_okf.consume`, and the next statement uses
# `__name__` as the `sys.modules` key -- the module would be aliased under the
# wrong one.
#
# This restores attribute ACCESS and NOT patch-through: a caller monkeypatching
# one of these copies patches a binding the implementation never reads. That is
# precisely why the alias below exists, and it is why the alias stays.
globals().update({k: v for k, v in vars(_impl).items() if not k.startswith("__")})
# ALIASED, not re-exported. `import okf_consume` must hand back the packaged module
# ITSELF: a re-export binds copies of the names into a second module object, so
# a caller patching one of them patches a binding the implementation never