open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

feat(propose,consume,tools): the type that declares nothing, and the prefix that is not a word

Browse source 
Three of round 9's four measured holes, each closed with a rule chosen on a
measurement rather than named as a limit.

`rtf` GIVES 0 SEGMENTS -> 6 of 6 AUTHORED TITLES over N = 4. The container has
no heading style, so the author's title is bold text. The grammar is markdown,
not `rtf`: the converter already writes that title as `**...**` in the same
output every office row produces, so no `rtf`-only heading form exists. Three
parameters were swept over 47 readable documents and ONE carried -- refusing a
line that ends in terminal punctuation takes false-positive lines from 9-12 to
1-2. A maximum title length (unlimited/40/60/80/120) and a
must-stand-between-blank-lines clause are both FLAT, so neither is in the rule.
The last false positive is closed by G1, the principle `_gate_outline` already
carries: recovery yields to declaration. False positives are then 0 of the 31
declaring documents by construction, and 0 of 27 on the corpus. Reach: 2 of 39
corpus documents, both `docx`, 0 of 33 `pdf` and 0 of 2 `xlsx`. Behind
`--bold-title`, default OFF pending the hit@8 measurement; the default bundle
is byte-identical without it.

BOTH ALTERNATIVES THE ORDER NAMED WERE MEASURED AND FELLED. A fourth hand-laid
fixture DECLARES heading styles in a stylesheet and the converter discards
them, emitting the same bold line -- so "read the declared headings out of the
markdown" has nothing to read. `rtf` -> `docx` -> markdown yields 0 ATX
headings on that same document, because the loss is in the `rtf` READER before
any writer sees the style. Fixtures are hand-laid in `make_k2_office.py` with
the fasit written first; they live in their own directory because Door B walks
a drop directory recursively and `k2-office/` reads its N off the listing.

THE PREFIX OVER-MATCH: THREE CANDIDATES MEASURED, ALL THREE FAILED ON ONE ROW.
Re-measured on the pinned 453-concept bundle with the control run first:
`under` occurs 79 times by equality and matches 172 by prefix, `undersjoisk` 0
and 172, `bilateral` 0 and 400 of 453, `standhaftig` 0 and 219. The two extra
known-negatives were FOUND, not chosen -- every 4-character prefix ranked by
document frequency, then a real word taken from the widest. A longer floor
(5-8), a coverage share (0.5-0.8) and a long-words-only floor (>= 8) each cost
row 1 its rank on the default bundle and the whole row on Arm B. Decomposed:
row 1's token `prisene` reaches its gold document through
`pris|sammenstilling` on four characters -- 0.57 of one word and 0.22 of the
other -- so the over-match and the wanted match are one mechanism.

THE FOURTH CANDIDATE IS THE ANSWER: the shared prefix must be a WORD the bundle
uses. `pris` is; `bila` and `stan` are not. `bilateral` 400 -> 0 and 512 -> 0,
`standhaftig` 219 -> 56 and 235 -> 33, every hit@8 row keeping rank 1 on BOTH
bundles. `undersjoisk` stops at 162 because `under` IS a word here -- a genuine
Norwegian morpheme, so that residual is a different answer, not a ceiling. ON
by default (`--no-stem-prefix`), pinned with its own known-negative on the
shipped bytes.

THE SHIM: a path importer holds the object `module_from_spec` made, and
`sys.modules[__name__] = _impl` never reaches it. Measured under both counting
methods -- 3 of 76 public names by `vars()`. One line copies the public names
into this file's globals; the dunder filter is load-bearing, because an
unfiltered copy overwrites `__name__` before the next line uses it as the alias
key. It restores attribute ACCESS and not patch-through, which is why the alias
stays. A CHANGELOG note under 0.7.0 and a shim docstring line say so, since
what the consumer asked for was the note.

Suite 1515 -> 1535.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 23:05:45 +02:00
commit 191de89f41
16 changed files with 1100 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

100
src/llm_ingestion_okf/propose.py
View file

@ -123,6 +123,18 @@ RULE_TABLE_GRID = "rule:table-grid"
#: the two compose in one order: Arm E decides how far a block extends, this
#: decides where it is cut inside.
RULE_SHEET_SECTION = "rule:sheet-section"
#: Round 10 only, and its axis is a fifth one. Arm C names SIZE, Arm D what the
#: DOCUMENT declared, Arm E what the CONVERTER emitted, D3 what the SHEET
#: labelled; this names what the AUTHOR set in bold where the container gave
#: them no heading style to declare with. `rtf` is the row that forced it -- it
#: has no heading style at all -- but the grammar is markdown, not `rtf`, and
#: the rule reaches every type the converter writes bold for.
#:
#: It is a RECOVERY, like Arm D, so it carries Arm D's gate: a document that
#: declares a heading of its own admits none of these. Measured over 47
#: readable documents, that gate is what takes the false-positive count to 0
#: of the 31 that declare.
RULE_BOLD_TITLE = "rule:bold-title"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -131,6 +143,7 @@ RULE_NAMES = (
RULE_OUTLINE,
RULE_TABLE_GRID,
RULE_SHEET_SECTION,
RULE_BOLD_TITLE,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -246,6 +259,20 @@ _SHEET_SECTION_LABEL = re.compile(r"^\d+(?:[+./-]\d+)*$")
# A pipe that pandoc did not escape. Splitting a row on a bare `|` would cut a
# cell containing a literal pipe in half and misread the FIRST cell of the row
# after it, which is the only cell this rule judges.
#: A line that is ONE bold span and nothing else. The whole discriminator of
#: round 10's rule: bold inside a paragraph is `text **bold** text`, which is
#: not a whole line, so the anchors are what separate a title from emphasis.
#: Both markers are read because the converter writes `**` and a hand-authored
#: markdown document may carry `__`.
_BOLD_LINE = re.compile(r"^\s*(?:\*\*|__)(?P<title>\S.*?)(?:\*\*|__)\s*$")
#: What a bold line must NOT end in. The one parameter of three that carried a
#: measurement: over 47 readable documents it takes false-positive lines from
#: 9-12 down to 1-2, and the candidates it drops are a contract cover page's
#: sentence fragments (`er inngatt mellom:`, `Sted og dato:`). A heading names
#: a section; a fragment set in bold ends the way a sentence does.
_TERMINAL_PUNCTUATION = (".", ":", ",", ";", "!", "?")
_UNESCAPED_PIPE = re.compile(r"(?<!\\)\|")
@ -442,6 +469,40 @@ def declares_headings(candidates: Iterable[Candidate]) -> bool:
return any(candidate.rule == RULE_HEADING for candidate in candidates)
def _gate_bold_title(
marked: list[tuple[int, Candidate]], joined: set[int]
) -> tuple[list[tuple[int, Candidate]], set[int]]:
"""G1 for round 10's rule: a document that declares admits no bold titles.
The same principle `_gate_outline` carries and the same predicate,
`declares_headings`, so there is one definition of "this document declared
a heading of its own" and not two that can drift. Applied to `marked` --
BEFORE the orphan pass -- for the reason stated there: the second pass
closes each span at the NEXT mark, so removing a mark here lets the
preceding span reach through the text it used to open, where filtering the
finished entries would leave that text in no segment at all.
There is NO G2 half. Arm D's share clause exists because a recovered
outline can carry a document whose declarations are incidental; a single
bold line in a document that already declares is not that case, and the one
false positive measured over 47 documents is exactly it. Adding a share
threshold here would be a knob no measurement asks for.
`declares_headings` counts `RULE_HEADING` alone, so admitting bold titles
can never change what this gate -- or the outline gate below it -- decides.
"""
if not declares_headings(candidate for _, candidate in marked):
return marked, joined
kept: list[tuple[int, Candidate]] = []
remap: dict[int, int] = {}
for position, entry in enumerate(marked):
if entry[1].rule == RULE_BOLD_TITLE:
continue
remap[position] = len(kept)
kept.append(entry)
return kept, {remap[p] for p in joined if p in remap}
def _gate_outline(
marked: list[tuple[int, Candidate]], joined: set[int], end_of_text: int, length: int
) -> tuple[list[tuple[int, Candidate]], set[int]]:
@ -549,6 +610,7 @@ def find_candidates(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -591,6 +653,14 @@ def find_candidates(
REMOVE members from a run, which is why it only ever rescues candidates and
never discards one the shipped rule kept.
`bold_title` is round 10's gate and it is OFF at False, where the branch is
not even evaluated. On, a line that is ONE bold span, does not end in
terminal punctuation and is not made only of stop words proposes a
boundary -- but ONLY in a document that declares no heading of its own.
It exists for `rtf`, whose container has no heading style at all, and it
reads markdown rather than `rtf`: the converter already writes the author's
bold title as `**...**` in the same output every office row produces.
`close_span_gaps` is OFF at False, where a mark removed after its
neighbour's span was closed takes that text out of the plan entirely. On,
a span runs to the next SURVIVING candidate's start and the last one runs
@ -752,6 +822,30 @@ def find_candidates(
)
continue
if bold_title:
bold = _BOLD_LINE.match(line)
if bold is not None:
title = bold.group("title")
# An inner marker means the line is two or more bold RUNS with
# text between them, not one title set in bold.
nested = "**" in title or "__" in title
fragment = title.rstrip().endswith(_TERMINAL_PUNCTUATION)
if not nested and not fragment and not _is_stop_word_only(title):
marked.append(
(
index,
Candidate(
title=title,
level=1,
number=None,
rule=RULE_BOLD_TITLE,
start=offsets[index],
end=end_of_text,
),
)
)
continue
atx = _ATX.match(line)
numbered = _NUMBERED.match(line)
if atx is None and numbered is None:
@ -788,6 +882,8 @@ def find_candidates(
)
)
if bold_title:
marked, joined = _gate_bold_title(marked, joined)
if outline_gate:
marked, joined = _gate_outline(marked, joined, end_of_text, len(text))
@ -1232,6 +1328,7 @@ def build_plan(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1249,6 +1346,7 @@ def build_plan(
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -1326,6 +1424,7 @@ def run(
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
bold_title: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -1402,6 +1501,7 @@ def run(
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
bold_title=bold_title,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan