open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,propose): reach the arms from okf build, keep a sheet heading behind a flag, fix two PDF contents mechanisms

Browse source 
K3 round 2, per file type. Order 20260908T143513Z-6327528123-from-.claude, carrying two operator decisions taken beforehand: D1 the orphan-gate variant goes behind a flag, D2 the arms become reachable from `okf build`. No default moved. Report: docs/2026-09-08-k3-runde2-per-filtype.md.

THE REPRODUCTION HELD, all three numbers, before any edit: `okf build` on the five-document tender folder gives 31 markdown files with both PDFs flat and 5/5 merged; the tender PDF gives no boundary without a flag and 9 with `--outline-run 3` (reference 9); the price sheet gives 1 on HEAD. Both proposer runs had to go through `bash -c` -- zsh does not word-split an unquoted `$flags`, so a sweep hands `--outline-run 3` to argparse as one token and every row comes back exit 2.

D2 -- `cli.py:_propose_plans` called the proposer with no arm argument, so the build path ran Arm B while `tools/okf_propose_segments.py` could run D, E and F. It now passes `--outline-run`, `--table-grid`, `--unit-fold` and `--keep-table-heading` through unchanged. THE DEFAULT DOES NOT MOVE and that is measured, not asserted: same folder, no flags, before and after the change, digest 3af10770...8fbbe2 both times and `diff -rq` clean. The "before" bundle was built before the first edit, because the editable install reads src/ live. Red test on the PLANS and on titles rather than a count, with the same fixture and no flags as its control. Per-document table for B/D/E/F/F2 is in the README and the report; the tender PDF is 1 under the default and 9 under every arm above it, and the reference is 9.

D1 -- a sheet heading with a table opening under it has an empty body, so the orphan check drops it: the NAME survives (carried onto the table block), the LINE does not. `--keep-table-heading` lets the heading survive and absorb the table instead. Price sheet 1 -> 1 concepts, `source_offset` [34, 11048] -> [0, 11048], body now starting at the heading. ELEVEN IS NOT REACHABLE THIS WAY and the number says why: the sheet is one heading and one continuous pipe-table block, and the eleven cost groups are eleven ROWS inside it (lines 10-20 of 103). What is missing is a section-row rule inside a sheet -- the opposite of `--table-grid`. Corpus: the flag changes 2 of 39 documents, both `.xlsx`, under arms B, E and F alike; known-negative 0 of 32 `pdf` and 0 of 5 `docx`. With it off, Arm E over all 43 is byte-identical to session 109's tree (33 plans, 43 `.err`, 4 FAILED, diff exit 0, counts asserted first).

THE PDF REMAINDER, one at a time. Position 9: clause 1 read the list AFTER the orphan check, and a contents list without dot leaders is a run of bodiless headings, so all but the last entry were already gone and the run was one. The run is now measured on the pre-orphan list, predicate written once and read in both places. 11 -> 10. Position 7: the same clause required siblings, and a numbered report's contents list interleaves 1.1/1.1.1/2.1 -- its 34 entries are one block that the level condition cut into runs of 9, 1, 1, 1, 5, 2, 10, 2 and 3, so the short runs survived. The level condition is dropped; the run LENGTH, which is what the CONTENTS_RUN sweep bought, is unchanged. Measured outward: the relaxation changes 1 document of 39 and removes exactly the leftover line. 34 -> 33.

TWO REMAINDERS ARE DECLINED WITH NUMBERS RATHER THAN FIXED. Position 1: the three level-1 candidates are 3 of 3 `rule:outline`, same level, same grammar, and the operator keeps one of them by prose alone -- there is no property to read. Position 4: a title-length rule was measured on paper and falsified -- a real chapter is 56 characters and a real heading in a document the arms already score correct is 88, sitting between position 4's 86 and 91, so no threshold separates the classes. Position 0 stays an extraction failure.

ONE SHIPPED EXPECTATION MOVED and is stated rather than quietly updated: `Innhold 1` is now discarded with the contents list it heads. Its body is in no segment afterwards, which is a real cost on a fixture where that heading has one.

Nine new tests: five red before the implementation, four green by construction and named as such. Three mutations, three red, unmutated control green each time -- restoring the level condition, computing the run post-orphan, absorbing a table unconditionally. 1379 -> 1388 tests. ruff clean, mypy --strict clean on 17 files. K2 bundle untouched (1108 files, 9cd74519...). The K2 ranking control is NOT measured: no bundle was rebuilt with the flag, so the rank is a prediction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 18:21:25 +02:00
commit ff79cfa19b
7 changed files with 989 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

100
src/llm_ingestion_okf/cli.py
View file

@ -38,11 +38,21 @@ the replay dated differently passes both explicitly.
## What it does not decide
Arm C (`--max-segment-chars`), Arm D (`--outline-run`) and Arm E
(`--table-grid`) are OFF here and are not exposed: they are measurement arms,
all off by default by operator decision, and a build command is not where an
unadjudicated segmentation heuristic should become one flag away. `tools/`
still reaches them.
**Every arm is OFF unless the caller asks, and this command does not move a
default.** Which arm should ship as the default is the operator's decision and
is not taken here.
The arms are, however, REACHABLE from here, and that is a change of 2026-09-08.
Until then `_propose_plans` called the proposer with no arm flag at all, so
`okf build` ran Arm B while `tools/okf_propose_segments.py` could run Arm D, E
and F -- a build path a full arm behind the proposer, reachable only by
retyping the loop the packaging removed. Measured on a five-document folder:
one tender PDF lands as ONE concept from the build path and as NINE with
`--outline-run 3`, and nine is what the operator's unit worksheet asks for.
Arm C (`--max-segment-chars`) stays unexposed: it is a character cap whose
value nobody has measured against a reference, so it has no number to offer a
caller.
"""
from __future__ import annotations
@ -71,7 +81,16 @@ DEFAULT_STAMP = "1970-01-01T00:00:00Z"
def _propose_plans(
inbox: Path, bundle: Path, plans_dir: Path, *, proposed_at: str, okf_type: str
inbox: Path,
bundle: Path,
plans_dir: Path,
*,
proposed_at: str,
okf_type: str,
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -96,6 +115,10 @@ def _propose_plans(
okf_type=okf_type,
proposed_at=proposed_at,
path_prefix=relative.with_suffix("").as_posix(),
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -119,6 +142,10 @@ def build(
segments: bool = True,
plans_dir: Path | None = None,
okf_type: str = "reference",
outline_run: int = 0,
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -157,7 +184,15 @@ def build(
target = plans_dir if plans_dir is not None else Path(scratch)
target.mkdir(parents=True, exist_ok=True)
written, nothing, failed = _propose_plans(
inbox, bundle, target, proposed_at=proposed_at, okf_type=okf_type
inbox,
bundle,
target,
proposed_at=proposed_at,
okf_type=okf_type,
outline_run=outline_run,
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -257,6 +292,53 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"the run replayed"
),
)
build_parser.add_argument(
"--outline-run",
type=int,
default=0,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. 0 (the default) is OFF and "
"leaves the bundle byte-identical. Measured on a tender PDF whose "
"headings are bare integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
"--table-grid",
action="store_true",
help=(
"Arm E, passed to the proposer unchanged: a pandoc grid-table rule "
"line no longer closes an open table block, so one grid table is one "
"concept instead of one per row group. Absent (the default) is OFF. "
"Measured on the K3 sample: it changes a `.docx` experience list from "
"21 concepts to 6"
),
)
build_parser.add_argument(
"--unit-fold",
action="store_true",
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. Absent (the default) is OFF. Measured on the K3 "
"sample: 5 of 12 documents match the operator's unit worksheet, "
"against 2 for the shipped default, with no cell worse"
),
)
build_parser.add_argument(
"--keep-table-heading",
action="store_true",
help=(
"D1, passed to the proposer unchanged: keep a heading whose body is "
"empty only because a table opens under it, and absorb that table "
"into its span. Absent (the default) is OFF. Measured on a tender "
"price sheet: the concept count does not move (1 -> 1) and the "
"concept gains the heading line it was missing"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
@ -277,6 +359,10 @@ def main(argv: list[str] | None = None) -> int:
segments=args.segments == "on",
plans_dir=args.plans_dir,
okf_type=args.okf_type,
outline_run=args.outline_run,
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)