open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms

Browse source 
K3 round 2, per file type. Order 20260908T143513Z-6327528123-from-.claude, carrying two operator decisions taken beforehand: D1 the orphan-gate variant goes behind a flag, D2 the arms become reachable from `okf build`. No default moved. Report: docs/2026-09-08-k3-runde2-per-filtype.md.

THE REPRODUCTION HELD, all three numbers, before any edit: `okf build` on the five-document tender folder gives 31 markdown files with both PDFs flat and 5/5 merged; the tender PDF gives no boundary without a flag and 9 with `--outline-run 3` (reference 9); the price sheet gives 1 on HEAD. Both proposer runs had to go through `bash -c` -- zsh does not word-split an unquoted `$flags`, so a sweep hands `--outline-run 3` to argparse as one token and every row comes back exit 2.

D2 -- `cli.py:_propose_plans` called the proposer with no arm argument, so the build path ran Arm B while `tools/okf_propose_segments.py` could run D, E and F. It now passes `--outline-run`, `--table-grid`, `--unit-fold` and `--keep-table-heading` through unchanged. THE DEFAULT DOES NOT MOVE and that is measured, not asserted: same folder, no flags, before and after the change, digest 3af10770...8fbbe2 both times and `diff -rq` clean. The "before" bundle was built before the first edit, because the editable install reads src/ live. Red test on the PLANS and on titles rather than a count, with the same fixture and no flags as its control. Per-document table for B/D/E/F/F2 is in the README and the report; the tender PDF is 1 under the default and 9 under every arm above it, and the reference is 9.

D1 -- a sheet heading with a table opening under it has an empty body, so the orphan check drops it: the NAME survives (carried onto the table block), the LINE does not. `--keep-table-heading` lets the heading survive and absorb the table instead. Price sheet 1 -> 1 concepts, `source_offset` [34, 11048] -> [0, 11048], body now starting at the heading. ELEVEN IS NOT REACHABLE THIS WAY and the number says why: the sheet is one heading and one continuous pipe-table block, and the eleven cost groups are eleven ROWS inside it (lines 10-20 of 103). What is missing is a section-row rule inside a sheet -- the opposite of `--table-grid`. Corpus: the flag changes 2 of 39 documents, both `.xlsx`, under arms B, E and F alike; known-negative 0 of 32 `pdf` and 0 of 5 `docx`. With it off, Arm E over all 43 is byte-identical to session 109's tree (33 plans, 43 `.err`, 4 FAILED, diff exit 0, counts asserted first).

THE PDF REMAINDER, one at a time. Position 9: clause 1 read the list AFTER the orphan check, and a contents list without dot leaders is a run of bodiless headings, so all but the last entry were already gone and the run was one. The run is now measured on the pre-orphan list, predicate written once and read in both places. 11 -> 10. Position 7: the same clause required siblings, and a numbered report's contents list interleaves 1.1/1.1.1/2.1 -- its 34 entries are one block that the level condition cut into runs of 9, 1, 1, 1, 5, 2, 10, 2 and 3, so the short runs survived. The level condition is dropped; the run LENGTH, which is what the CONTENTS_RUN sweep bought, is unchanged. Measured outward: the relaxation changes 1 document of 39 and removes exactly the leftover line. 34 -> 33.

TWO REMAINDERS ARE DECLINED WITH NUMBERS RATHER THAN FIXED. Position 1: the three level-1 candidates are 3 of 3 `rule:outline`, same level, same grammar, and the operator keeps one of them by prose alone -- there is no property to read. Position 4: a title-length rule was measured on paper and falsified -- a real chapter is 56 characters and a real heading in a document the arms already score correct is 88, sitting between position 4's 86 and 91, so no threshold separates the classes. Position 0 stays an extraction failure.

ONE SHIPPED EXPECTATION MOVED and is stated rather than quietly updated: `Innhold 1` is now discarded with the contents list it heads. Its body is in no segment afterwards, which is a real cost on a fixture where that heading has one.

Nine new tests: five red before the implementation, four green by construction and named as such. Three mutations, three red, unmutated control green each time -- restoring the level condition, computing the run post-orphan, absorbing a table unconditionally. 1379 -> 1388 tests. ruff clean, mypy --strict clean on 17 files. K2 bundle untouched (1108 files, 9cd74519...). The K2 ranking control is NOT measured: no bundle was rebuilt with the flag, so the rank is a prediction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 18:21:25 +02:00
commit ff79cfa19b
7 changed files with 989 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

100
src/llm_ingestion_okf/cli.py
View file

@ -38,11 +38,21 @@ the replay dated differently passes both explicitly.
## What it does not decide
Arm C (`--max-segment-chars`), Arm D (`--outline-run`) and Arm E
(`--table-grid`) are OFF here and are not exposed: they are measurement arms,
all off by default by operator decision, and a build command is not where an
unadjudicated segmentation heuristic should become one flag away. `tools/`
still reaches them.
**Every arm is OFF unless the caller asks, and this command does not move a
default.** Which arm should ship as the default is the operator's decision and
is not taken here.
The arms are, however, REACHABLE from here, and that is a change of 2026-09-08.
Until then `_propose_plans` called the proposer with no arm flag at all, so
`okf build` ran Arm B while `tools/okf_propose_segments.py` could run Arm D, E
and F -- a build path a full arm behind the proposer, reachable only by
retyping the loop the packaging removed. Measured on a five-document folder:
one tender PDF lands as ONE concept from the build path and as NINE with
`--outline-run 3`, and nine is what the operator's unit worksheet asks for.
Arm C (`--max-segment-chars`) stays unexposed: it is a character cap whose
value nobody has measured against a reference, so it has no number to offer a
caller.
"""
from __future__ import annotations
@ -71,7 +81,16 @@ DEFAULT_STAMP = "1970-01-01T00:00:00Z"
def _propose_plans(
inbox: Path, bundle: Path, plans_dir: Path, *, proposed_at: str, okf_type: str
inbox: Path,
bundle: Path,
plans_dir: Path,
*,
proposed_at: str,
okf_type: str,
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -96,6 +115,10 @@ def _propose_plans(
okf_type=okf_type,
proposed_at=proposed_at,
path_prefix=relative.with_suffix("").as_posix(),
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -119,6 +142,10 @@ def build(
segments: bool = True,
plans_dir: Path | None = None,
okf_type: str = "reference",
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -157,7 +184,15 @@ def build(
target = plans_dir if plans_dir is not None else Path(scratch)
target.mkdir(parents=True, exist_ok=True)
written, nothing, failed = _propose_plans(
inbox, bundle, target, proposed_at=proposed_at, okf_type=okf_type
inbox,
bundle,
target,
proposed_at=proposed_at,
okf_type=okf_type,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -257,6 +292,53 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"the run replayed"
),
)
build_parser.add_argument(
"--outline-run",
type=int,
default=0,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. 0 (the default) is OFF and "
"leaves the bundle byte-identical. Measured on a tender PDF whose "
"headings are bare integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
"--table-grid",
action="store_true",
help=(
"Arm E, passed to the proposer unchanged: a pandoc grid-table rule "
"line no longer closes an open table block, so one grid table is one "
"concept instead of one per row group. Absent (the default) is OFF. "
"Measured on the K3 sample: it changes a `.docx` experience list from "
"21 concepts to 6"
),
)
build_parser.add_argument(
"--unit-fold",
action="store_true",
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. Absent (the default) is OFF. Measured on the K3 "
"sample: 5 of 12 documents match the operator's unit worksheet, "
"against 2 for the shipped default, with no cell worse"
),
)
build_parser.add_argument(
"--keep-table-heading",
action="store_true",
help=(
"D1, passed to the proposer unchanged: keep a heading whose body is "
"empty only because a table opens under it, and absorb that table "
"into its span. Absent (the default) is OFF. Measured on a tender "
"price sheet: the concept count does not move (1 -> 1) and the "
"concept gains the heading line it was missing"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
@ -277,6 +359,10 @@ def main(argv: list[str] | None = None) -> int:
segments=args.segments == "on",
plans_dir=args.plans_dir,
okf_type=args.okf_type,
outline_run=args.outline_run,
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

139
src/llm_ingestion_okf/propose.py
View file

@ -239,6 +239,15 @@ class Candidate:
#: a rule line -- a single-row grid table joins nothing and stays
#: byte-identical to Arm D.
grid: bool = False
#: True when this candidate belongs to a run of page-numbered siblings that
#: was a contents list BEFORE the orphan check thinned it. Set only when
#: Arm F is on, because it exists only for Arm F's clause 1 to read: a
#: contents list without dot leaders is a run of bodiless headings, so the
#: orphan check deletes all but the last and the run clause 1 looks for is
#: gone by the time `fold_units` sees the list. Computed where the whole
#: pre-orphan list is still in hand, and nowhere else -- no candidate
#: carrying it survives clause 1, so it never reaches an artifact.
contents: bool = False
def _is_stop_word_only(title: str) -> bool:
@ -312,6 +321,7 @@ def find_candidates(
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -338,6 +348,14 @@ def find_candidates(
table block, so one grid table proposes one candidate instead of one per row
group. It only ever REMOVES marks, which is what keeps every surviving
candidate's `start` fixed and the orphan check monotone.
`keep_table_heading` is D1's gate and it is OFF at False. On, a heading
whose body is empty ONLY because a table block opens under it keeps that
table instead of being dropped: the table is absorbed into the heading's
span rather than emitted, so the concept starts at the heading line. The
count does not move -- one candidate either way -- and the first byte does.
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
"""
lines = text.splitlines(keepends=True)
offsets: list[int] = []
@ -479,8 +497,24 @@ def find_candidates(
# is still carried by a live candidate, so copying its title onto the table
# as well would put one name on two concepts and rescue none.
orphaned_name: tuple[str, str | None] | None = None
# D1. Which table blocks a heading ABSORBS, decided before the pass that
# consumes it: once a table is absorbed the heading's body is no longer
# empty, so the orphan check below stops firing on it by itself and no
# branch is needed there. Computed only when the caller asked, so every
# other arm's `marked` -> `candidates` mapping is untouched code.
absorbed = _absorbed_tables(text, marked, offsets, end_of_text) if keep_table_heading else set()
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = _contents_run_positions(marked) if unit_fold else set()
for position_in_list, (_, candidate) in enumerate(marked):
following = marked[position_in_list + 1 :]
if position_in_list in absorbed:
continue
following = [
entry
for position, entry in enumerate(marked)
if position > position_in_list and position not in absorbed
]
end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end]
# The orphan check: everything after the heading line itself.
@ -503,11 +537,77 @@ def find_candidates(
start=candidate.start,
end=end,
grid=position_in_list in joined,
contents=position_in_list in contents_run,
)
)
return fold_units(candidates) if unit_fold else candidates
def _absorbed_tables(
text: str,
marked: list[tuple[int, Candidate]],
offsets: list[int],
end_of_text: int,
) -> set[int]:
"""D1: positions in `marked` of table blocks a bodiless heading keeps.
The condition is the orphan check's own, evaluated on the UNABSORBED
neighbour distance -- a heading is eligible only when the very next mark is
a table block and there is nothing between them but the heading line. That
is what keeps the variant from merging a section into a table it merely
contains: a heading with a paragraph of its own is not orphaned, so it
absorbs nothing.
"""
absorbed: set[int] = set()
for position, (_, candidate) in enumerate(marked):
if candidate.rule == RULE_TABLE_BLOCK or position + 1 >= len(marked):
continue
line_index, following = marked[position + 1]
if following.rule != RULE_TABLE_BLOCK:
continue
body = text[candidate.start : offsets[line_index]]
if body.splitlines()[1:] and "".join(body.splitlines()[1:]).strip():
continue
absorbed.add(position + 1)
return absorbed
def _contents_run_positions(marked: list[tuple[int, Candidate]]) -> set[int]:
"""Arm F clause 1, measured on the PRE-orphan list. Same predicate as there.
A run of at least `CONTENTS_RUN` consecutive page-numbered headings, table
blocks excluded and never a single line. Written once here and read by
`fold_units` through `Candidate.contents`, so the two cannot drift into two
definitions of what a contents list is.
LEVEL IS NOT PART OF THE PREDICATE, and that is the second round-2 change.
A numbered report's contents list interleaves `1`, `1.1`, `1.1.1`, `2`, so
requiring the members to be siblings breaks the run at every level change:
on K3 position 7 the level-2 entries formed runs long enough to discard and
`6.2.2 Tverrfaglig kontroll` did not, so one contents line was emitted as a
concept while its neighbours were not. What still bounds the rule is the run
LENGTH, which is what the `CONTENTS_RUN` sweep bought and is unchanged.
"""
inside: set[int] = set()
index = 0
while index < len(marked):
candidate = marked[index][1]
if candidate.rule == RULE_TABLE_BLOCK or not _TRAILING_PAGE_NUMBER.search(candidate.title):
index += 1
continue
end = index
while (
end < len(marked)
and marked[end][1].rule != RULE_TABLE_BLOCK
and _TRAILING_PAGE_NUMBER.search(marked[end][1].title)
):
end += 1
if end - index >= CONTENTS_RUN:
inside.update(range(index, end))
index = end
return inside
#: Arm F clause 1. How many CONSECUTIVE same-level page-numbered headings make
#: a contents list. Three, and it is swept rather than guessed: at 1 and 2 the
#: rule deletes body chapters (a heading like `... i henhold til TEK 17` ends
@ -527,6 +627,9 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
`CONTENTS_RUN` consecutive same-level headings each ending in a page
number is a contents list and is discarded. A run of siblings, never a
single line: one body heading ending in a number is not a contents list.
The run is measured on this list AND on the list before the orphan check
(`Candidate.contents`), because a contents list whose entries carry no
dot leaders is bodiless and reaches here as one surviving line.
2. "hvert h2-kapittel med sine h3" -- the unit level is the SHALLOWEST
heading level occurring more than once; anything deeper folds into the
preceding candidate at or above that level, which EXTENDS the parent's
@ -543,8 +646,13 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
if not candidates:
return candidates
# Clause 1.
drop: set[int] = set()
# Clause 1. Two inputs, one predicate. `contents` carries the run measured
# on the list BEFORE the orphan check thinned it -- a contents list with no
# dot leaders is a run of bodiless headings, so only its last entry reaches
# here and a run of one is below `CONTENTS_RUN`. The scan below still has
# work to do: a contents list WITH dot leaders keeps every entry, and that
# run exists only in this list.
drop: set[int] = {position for position, c in enumerate(candidates) if c.contents}
index = 0
while index < len(candidates):
candidate = candidates[index]
@ -760,13 +868,18 @@ def build_plan(
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
extractor_id = source.suffix.lower().lstrip(".") or "none"
entries: list[dict[str, Any]] = []
candidates = find_candidates(
text, outline_run=outline_run, table_grid=table_grid, unit_fold=unit_fold
text,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -837,6 +950,7 @@ def run(
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -889,6 +1003,7 @@ def run(
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
@ -997,6 +1112,21 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"either"
),
)
parser.add_argument(
"--keep-table-heading",
action="store_true",
help=(
"D1: keep a heading whose body is empty ONLY because a table block "
"opens under it, and absorb that table into the heading's span "
"instead of emitting it. Without this the orphan check drops the "
"heading, the table inherits its NAME, and the concept starts at "
"the first table row -- so the heading line is in no concept's "
"body. Measured on a spreadsheet: the concept count does not move "
"(1 -> 1), the first byte does. Absent (the default) is OFF and "
"leaves every artifact byte-identical. Its own flag rather than "
"part of an arm: the orphan check is reached by every file type"
),
)
parser.add_argument(
"--proposed-at",
default="1970-01-01T00:00:00Z",
@ -1018,6 +1148,7 @@ def main(argv: list[str] | None = None) -> int:
outline_run=args.outline_run,
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
)
except ProposerError as exc:
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)