open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved

Browse source 
K3 round 3, order 20260908T170037Z-3622420612-from-.claude, plus the PM
injection of 17:55Z carrying the operator's D3 answer.

TWO NEW RULES, both OFF by default.

`--sheet-section-rows` cuts an open table block at the rows that label its
sections: a RUN of at least three rows whose first cell is a bare numeric
label. It is the first rule here that opens a span INSIDE a table, and the
opposite direction from Arm E -- that arm decides how far a block extends,
this decides where it is cut inside. The spreadsheet is the one file type no
arm had ever moved: 1 concept -> 12 on the priced sheet, of which 11 are the
reference's 11 cost groups. The run guard buys exactly three candidates on the
corpus (the rows stating a computation basis), and each section carries its
own `source_sheet` + `source_rows`, verified on the artifact.

`--drop-wrapped-outline` declines an outline candidate whose line continues
onto the next one. Round 2 falsified the LENGTH axis on this case; this one is
not about size. Over the 12-document sample it separates 8 of 34 candidates --
the quoted regulation paragraphs and the risk-table rows -- and none of the 26
the operator kept. Position 4: 4 concepts -> 1, the reference. The cell is
`treffer` by count and lossy by bytes, and the report says so.

Whole corpus, one cached extraction: 1 of 39 documents changes under the first
rule (0 of 32 pdf, 0 of 5 docx), 5 of 39 under the second (all pdf). Arm B and
F2 are byte-identical against a frozen c3b645b, both halves by `diff -r`.

THE `okf build` DEFAULT MOVED, and this is the operator's decision executed,
not a side effect: no flag now means `--outline-run 3 --unit-fold`. Each arm
keeps an explicit opt-out (`--outline-run 0`, `--no-unit-fold`) and the pair
reproduces the pre-move bytes exactly. The proposer's own defaults do NOT
move: they are what the goldens and every published reproduction block are
pinned to, so the two layers now disagree on purpose and `cli.py` says where.
Two shipped expectations moved with it and both are stated in place. MEASURED
AND REPORTED BACK: the configuration made default scores 2 of 12 on the unit
worksheet, `docx` 0 of 3 -- Arm F's published 5 of 12 was measured with
`--table-grid` ON, and without it the fold has no joined table to fold.

Position 1 is declined for the second round on the axis the order named: 3 of
3 recovered chapters have body under them, so "heading-on-heading" separates 0
of 3.

The K2 ranking control round 2 could not measure: two bundles from frozen
c3b645b differing only in `--keep-table-heading`, 2 of 1108 files apart. The
priced concept ranks 96 of 629 in BOTH and is delivered at rank 10 in both, so
the flag buys 35 payload bytes and zero rank positions. Round 2's prediction
is falsified: the concept's NAME was already restored by the orphan-name
inheritance; the flag adds the heading LINE.

The stale S7 literal is restated where it stood (`c759a657...`, 171 614 B,
re-measured on this HEAD), so the next order cannot inherit the superseded one.

1397 -> 1414 tests. Report: docs/2026-09-08-k3-runde3-per-filtype.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 20:33:52 +02:00
commit 47f6619e6f
11 changed files with 1381 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

85
src/llm_ingestion_okf/cli.py
View file

@ -73,6 +73,25 @@ __all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
CLI_ID = "okf build"
#: What `okf build` runs when no flag is given. Moved 2026-09-08 by the
#: operator, from Arm B to Arm D plus Arm F, on the round-2 and round-3
#: measurements: the previous default was a full arm behind the proposer on
#: the one document with a reference, and the fold matches 5 of 12 documents
#: against the default's 2 with no cell worse.
#:
#: THE VALUE LIVES HERE AND NOT IN `propose.py`. That module's rules stay OFF
#: by default: they are what the golden fixtures and every published
#: reproduction block are pinned to, and the proposer's own CLI is the
#: measuring instrument. What moved is this COMMAND's default, and the two
#: layers disagreeing on purpose is why the disagreement is written down here
#: rather than discovered later.
#:
#: Each arm keeps an explicit opt-out: `--outline-run 0` (the number was
#: always its own switch) and `--no-unit-fold`. A default a caller cannot turn
#: off is not a default.
DEFAULT_OUTLINE_RUN = 3
DEFAULT_UNIT_FOLD = True
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -91,6 +110,8 @@ def _propose_plans(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -119,6 +140,8 @@ def _propose_plans(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -142,10 +165,12 @@ def build(
segments: bool = True,
plans_dir: Path | None = None,
okf_type: str = "reference",
outline_run: int = 0,
outline_run: int = DEFAULT_OUTLINE_RUN,
table_grid: bool = False,
unit_fold: bool = False,
unit_fold: bool = DEFAULT_UNIT_FOLD,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -193,6 +218,8 @@ def build(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -295,14 +322,15 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--outline-run",
type=int,
default=0,
default=DEFAULT_OUTLINE_RUN,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. 0 (the default) is OFF and "
"leaves the bundle byte-identical. Measured on a tender PDF whose "
"headings are bare integers: no boundary at 0, nine at 3"
"sustain an ascending run of at least N. The default is 3 (operator, "
"2026-09-08); pass 0 to turn the arm off, which is this arm's "
"explicit opt-out. Measured on a tender PDF whose headings are bare "
"integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
@ -319,13 +347,25 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--unit-fold",
action="store_true",
default=DEFAULT_UNIT_FOLD,
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. Absent (the default) is OFF. Measured on the K3 "
"sample: 5 of 12 documents match the operator's unit worksheet, "
"against 2 for the shipped default, with no cell worse"
"only reduce a plan. ON by default since 2026-09-08 (operator); the "
"flag is kept so a call site that passes it stays valid. Measured on "
"the K3 sample: 5 of 12 documents match the operator's unit "
"worksheet, against 2 for the arm below it, with no cell worse"
),
)
build_parser.add_argument(
"--no-unit-fold",
action="store_false",
dest="unit_fold",
help=(
"Arm F's explicit opt-out. With --outline-run 0 it reproduces the "
"pre-2026-09-08 default byte for byte, which is what keeps the move "
"reversible for a consumer who needs the old bytes"
),
)
build_parser.add_argument(
@ -339,6 +379,31 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"concept gains the heading line it was missing"
),
)
build_parser.add_argument(
"--sheet-section-rows",
action="store_true",
help=(
"D3, passed to the proposer unchanged: cut an open table block at "
"the rows that label its sections -- a RUN of rows whose first cell "
"is a bare numeric label. The opposite direction from Arm E, which "
"decides how far a block extends; this decides where it is cut "
"inside. Absent (the default) is OFF. Measured on a tender price "
"sheet whose whole body is one table block: 1 concept -> 12, "
"against a reference of 11 cost groups plus the sheet's preamble"
),
)
build_parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
help=(
"D3, passed to the proposer unchanged: do not admit an outline "
"candidate whose line continues onto the next one. Recovered "
"candidates only, never a heading the document declares. Absent "
"(the default) is OFF. Measured on the K3 sample: 8 of 34 outline "
"candidates wrap, and they are the quoted regulation paragraphs "
"and the risk-table rows the operator wrote off"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
@ -363,6 +428,8 @@ def main(argv: list[str] | None = None) -> int:
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)