open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms

Browse source 
K3 round 2, per file type. Order 20260908T143513Z-6327528123-from-.claude, carrying two operator decisions taken beforehand: D1 the orphan-gate variant goes behind a flag, D2 the arms become reachable from `okf build`. No default moved. Report: docs/2026-09-08-k3-runde2-per-filtype.md.

THE REPRODUCTION HELD, all three numbers, before any edit: `okf build` on the five-document tender folder gives 31 markdown files with both PDFs flat and 5/5 merged; the tender PDF gives no boundary without a flag and 9 with `--outline-run 3` (reference 9); the price sheet gives 1 on HEAD. Both proposer runs had to go through `bash -c` -- zsh does not word-split an unquoted `$flags`, so a sweep hands `--outline-run 3` to argparse as one token and every row comes back exit 2.

D2 -- `cli.py:_propose_plans` called the proposer with no arm argument, so the build path ran Arm B while `tools/okf_propose_segments.py` could run D, E and F. It now passes `--outline-run`, `--table-grid`, `--unit-fold` and `--keep-table-heading` through unchanged. THE DEFAULT DOES NOT MOVE and that is measured, not asserted: same folder, no flags, before and after the change, digest 3af10770...8fbbe2 both times and `diff -rq` clean. The "before" bundle was built before the first edit, because the editable install reads src/ live. Red test on the PLANS and on titles rather than a count, with the same fixture and no flags as its control. Per-document table for B/D/E/F/F2 is in the README and the report; the tender PDF is 1 under the default and 9 under every arm above it, and the reference is 9.

D1 -- a sheet heading with a table opening under it has an empty body, so the orphan check drops it: the NAME survives (carried onto the table block), the LINE does not. `--keep-table-heading` lets the heading survive and absorb the table instead. Price sheet 1 -> 1 concepts, `source_offset` [34, 11048] -> [0, 11048], body now starting at the heading. ELEVEN IS NOT REACHABLE THIS WAY and the number says why: the sheet is one heading and one continuous pipe-table block, and the eleven cost groups are eleven ROWS inside it (lines 10-20 of 103). What is missing is a section-row rule inside a sheet -- the opposite of `--table-grid`. Corpus: the flag changes 2 of 39 documents, both `.xlsx`, under arms B, E and F alike; known-negative 0 of 32 `pdf` and 0 of 5 `docx`. With it off, Arm E over all 43 is byte-identical to session 109's tree (33 plans, 43 `.err`, 4 FAILED, diff exit 0, counts asserted first).

THE PDF REMAINDER, one at a time. Position 9: clause 1 read the list AFTER the orphan check, and a contents list without dot leaders is a run of bodiless headings, so all but the last entry were already gone and the run was one. The run is now measured on the pre-orphan list, predicate written once and read in both places. 11 -> 10. Position 7: the same clause required siblings, and a numbered report's contents list interleaves 1.1/1.1.1/2.1 -- its 34 entries are one block that the level condition cut into runs of 9, 1, 1, 1, 5, 2, 10, 2 and 3, so the short runs survived. The level condition is dropped; the run LENGTH, which is what the CONTENTS_RUN sweep bought, is unchanged. Measured outward: the relaxation changes 1 document of 39 and removes exactly the leftover line. 34 -> 33.

TWO REMAINDERS ARE DECLINED WITH NUMBERS RATHER THAN FIXED. Position 1: the three level-1 candidates are 3 of 3 `rule:outline`, same level, same grammar, and the operator keeps one of them by prose alone -- there is no property to read. Position 4: a title-length rule was measured on paper and falsified -- a real chapter is 56 characters and a real heading in a document the arms already score correct is 88, sitting between position 4's 86 and 91, so no threshold separates the classes. Position 0 stays an extraction failure.

ONE SHIPPED EXPECTATION MOVED and is stated rather than quietly updated: `Innhold 1` is now discarded with the contents list it heads. Its body is in no segment afterwards, which is a real cost on a fixture where that heading has one.

Nine new tests: five red before the implementation, four green by construction and named as such. Three mutations, three red, unmutated control green each time -- restoring the level condition, computing the run post-orphan, absorbing a table unconditionally. 1379 -> 1388 tests. ruff clean, mypy --strict clean on 17 files. K2 bundle untouched (1108 files, 9cd74519...). The K2 ranking control is NOT measured: no bundle was rebuilt with the flag, so the rank is a prediction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 18:21:25 +02:00
commit ff79cfa19b
7 changed files with 989 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

139
src/llm_ingestion_okf/propose.py
View file

@ -239,6 +239,15 @@ class Candidate:
#: a rule line -- a single-row grid table joins nothing and stays
#: byte-identical to Arm D.
grid: bool = False
#: True when this candidate belongs to a run of page-numbered siblings that
#: was a contents list BEFORE the orphan check thinned it. Set only when
#: Arm F is on, because it exists only for Arm F's clause 1 to read: a
#: contents list without dot leaders is a run of bodiless headings, so the
#: orphan check deletes all but the last and the run clause 1 looks for is
#: gone by the time `fold_units` sees the list. Computed where the whole
#: pre-orphan list is still in hand, and nowhere else -- no candidate
#: carrying it survives clause 1, so it never reaches an artifact.
contents: bool = False
def _is_stop_word_only(title: str) -> bool:
@ -312,6 +321,7 @@ def find_candidates(
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -338,6 +348,14 @@ def find_candidates(
table block, so one grid table proposes one candidate instead of one per row
group. It only ever REMOVES marks, which is what keeps every surviving
candidate's `start` fixed and the orphan check monotone.
`keep_table_heading` is D1's gate and it is OFF at False. On, a heading
whose body is empty ONLY because a table block opens under it keeps that
table instead of being dropped: the table is absorbed into the heading's
span rather than emitted, so the concept starts at the heading line. The
count does not move -- one candidate either way -- and the first byte does.
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
"""
lines = text.splitlines(keepends=True)
offsets: list[int] = []
@ -479,8 +497,24 @@ def find_candidates(
# is still carried by a live candidate, so copying its title onto the table
# as well would put one name on two concepts and rescue none.
orphaned_name: tuple[str, str | None] | None = None
# D1. Which table blocks a heading ABSORBS, decided before the pass that
# consumes it: once a table is absorbed the heading's body is no longer
# empty, so the orphan check below stops firing on it by itself and no
# branch is needed there. Computed only when the caller asked, so every
# other arm's `marked` -> `candidates` mapping is untouched code.
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = _contents_run_positions(marked) if unit_fold else set()
for position_in_list, (_, candidate) in enumerate(marked):
following = marked[position_in_list + 1 :]
if position_in_list in absorbed:
continue
following = [
entry
for position, entry in enumerate(marked)
if position > position_in_list and position not in absorbed
]
end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end]
# The orphan check: everything after the heading line itself.
@ -503,11 +537,77 @@ def find_candidates(
start=candidate.start,
end=end,
grid=position_in_list in joined,
contents=position_in_list in contents_run,
)
)
return fold_units(candidates) if unit_fold else candidates
def _absorbed_tables(
text: str,
marked: list[tuple[int, Candidate]],
offsets: list[int],
end_of_text: int,
) -> set[int]:
"""D1: positions in `marked` of table blocks a bodiless heading keeps.
The condition is the orphan check's own, evaluated on the UNABSORBED
neighbour distance -- a heading is eligible only when the very next mark is
a table block and there is nothing between them but the heading line. That
is what keeps the variant from merging a section into a table it merely
contains: a heading with a paragraph of its own is not orphaned, so it
absorbs nothing.
"""
absorbed: set[int] = set()
for position, (_, candidate) in enumerate(marked):
if candidate.rule == RULE_TABLE_BLOCK or position + 1 >= len(marked):
continue
line_index, following = marked[position + 1]
if following.rule != RULE_TABLE_BLOCK:
continue
body = text[candidate.start : offsets[line_index]]
if body.splitlines()[1:] and "".join(body.splitlines()[1:]).strip():
continue
absorbed.add(position + 1)
return absorbed
def _contents_run_positions(marked: list[tuple[int, Candidate]]) -> set[int]:
"""Arm F clause 1, measured on the PRE-orphan list. Same predicate as there.
A run of at least `CONTENTS_RUN` consecutive page-numbered headings, table
blocks excluded and never a single line. Written once here and read by
`fold_units` through `Candidate.contents`, so the two cannot drift into two
definitions of what a contents list is.
LEVEL IS NOT PART OF THE PREDICATE, and that is the second round-2 change.
A numbered report's contents list interleaves `1`, `1.1`, `1.1.1`, `2`, so
requiring the members to be siblings breaks the run at every level change:
on K3 position 7 the level-2 entries formed runs long enough to discard and
`6.2.2 Tverrfaglig kontroll` did not, so one contents line was emitted as a
concept while its neighbours were not. What still bounds the rule is the run
LENGTH, which is what the `CONTENTS_RUN` sweep bought and is unchanged.
"""
inside: set[int] = set()
index = 0
while index < len(marked):
candidate = marked[index][1]
if candidate.rule == RULE_TABLE_BLOCK or not _TRAILING_PAGE_NUMBER.search(candidate.title):
index += 1
continue
end = index
while (
end < len(marked)
and marked[end][1].rule != RULE_TABLE_BLOCK
and _TRAILING_PAGE_NUMBER.search(marked[end][1].title)
):
end += 1
if end - index >= CONTENTS_RUN:
inside.update(range(index, end))
index = end
return inside
#: Arm F clause 1. How many CONSECUTIVE same-level page-numbered headings make
#: a contents list. Three, and it is swept rather than guessed: at 1 and 2 the
#: rule deletes body chapters (a heading like `... i henhold til TEK 17` ends
@ -527,6 +627,9 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
`CONTENTS_RUN` consecutive same-level headings each ending in a page
number is a contents list and is discarded. A run of siblings, never a
single line: one body heading ending in a number is not a contents list.
The run is measured on this list AND on the list before the orphan check
(`Candidate.contents`), because a contents list whose entries carry no
dot leaders is bodiless and reaches here as one surviving line.
2. "hvert h2-kapittel med sine h3" -- the unit level is the SHALLOWEST
heading level occurring more than once; anything deeper folds into the
preceding candidate at or above that level, which EXTENDS the parent's
@ -543,8 +646,13 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
if not candidates:
return candidates
# Clause 1.
drop: set[int] = set()
# Clause 1. Two inputs, one predicate. `contents` carries the run measured
# on the list BEFORE the orphan check thinned it -- a contents list with no
# dot leaders is a run of bodiless headings, so only its last entry reaches
# here and a run of one is below `CONTENTS_RUN`. The scan below still has
# work to do: a contents list WITH dot leaders keeps every entry, and that
# run exists only in this list.
drop: set[int] = {position for position, c in enumerate(candidates) if c.contents}
index = 0
while index < len(candidates):
candidate = candidates[index]
@ -760,13 +868,18 @@ def build_plan(
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
extractor_id = source.suffix.lower().lstrip(".") or "none"
entries: list[dict[str, Any]] = []
candidates = find_candidates(
text, outline_run=outline_run, table_grid=table_grid, unit_fold=unit_fold
text,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -837,6 +950,7 @@ def run(
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -889,6 +1003,7 @@ def run(
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
@ -997,6 +1112,21 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"either"
),
)
parser.add_argument(
"--keep-table-heading",
action="store_true",
help=(
"D1: keep a heading whose body is empty ONLY because a table block "
"opens under it, and absorb that table into the heading's span "
"instead of emitting it. Without this the orphan check drops the "
"heading, the table inherits its NAME, and the concept starts at "
"the first table row -- so the heading line is in no concept's "
"body. Measured on a spreadsheet: the concept count does not move "
"(1 -> 1), the first byte does. Absent (the default) is OFF and "
"leaves every artifact byte-identical. Its own flag rather than "
"part of an arm: the orphan check is reached by every file type"
),
)
parser.add_argument(
"--proposed-at",
default="1970-01-01T00:00:00Z",
@ -1018,6 +1148,7 @@ def main(argv: list[str] | None = None) -> int:
outline_run=args.outline_run,
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
)
except ProposerError as exc:
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)