open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,cli): the sheet's rows, the wrapped sentence, and a default that moved

Browse source 
K3 round 3, order 20260908T170037Z-3622420612-from-.claude, plus the PM
injection of 17:55Z carrying the operator's D3 answer.

TWO NEW RULES, both OFF by default.

`--sheet-section-rows` cuts an open table block at the rows that label its
sections: a RUN of at least three rows whose first cell is a bare numeric
label. It is the first rule here that opens a span INSIDE a table, and the
opposite direction from Arm E -- that arm decides how far a block extends,
this decides where it is cut inside. The spreadsheet is the one file type no
arm had ever moved: 1 concept -> 12 on the priced sheet, of which 11 are the
reference's 11 cost groups. The run guard buys exactly three candidates on the
corpus (the rows stating a computation basis), and each section carries its
own `source_sheet` + `source_rows`, verified on the artifact.

`--drop-wrapped-outline` declines an outline candidate whose line continues
onto the next one. Round 2 falsified the LENGTH axis on this case; this one is
not about size. Over the 12-document sample it separates 8 of 34 candidates --
the quoted regulation paragraphs and the risk-table rows -- and none of the 26
the operator kept. Position 4: 4 concepts -> 1, the reference. The cell is
`treffer` by count and lossy by bytes, and the report says so.

Whole corpus, one cached extraction: 1 of 39 documents changes under the first
rule (0 of 32 pdf, 0 of 5 docx), 5 of 39 under the second (all pdf). Arm B and
F2 are byte-identical against a frozen c3b645b, both halves by `diff -r`.

THE `okf build` DEFAULT MOVED, and this is the operator's decision executed,
not a side effect: no flag now means `--outline-run 3 --unit-fold`. Each arm
keeps an explicit opt-out (`--outline-run 0`, `--no-unit-fold`) and the pair
reproduces the pre-move bytes exactly. The proposer's own defaults do NOT
move: they are what the goldens and every published reproduction block are
pinned to, so the two layers now disagree on purpose and `cli.py` says where.
Two shipped expectations moved with it and both are stated in place. MEASURED
AND REPORTED BACK: the configuration made default scores 2 of 12 on the unit
worksheet, `docx` 0 of 3 -- Arm F's published 5 of 12 was measured with
`--table-grid` ON, and without it the fold has no joined table to fold.

Position 1 is declined for the second round on the axis the order named: 3 of
3 recovered chapters have body under them, so "heading-on-heading" separates 0
of 3.

The K2 ranking control round 2 could not measure: two bundles from frozen
c3b645b differing only in `--keep-table-heading`, 2 of 1108 files apart. The
priced concept ranks 96 of 629 in BOTH and is delivered at rank 10 in both, so
the flag buys 35 payload bytes and zero rank positions. Round 2's prediction
is falsified: the concept's NAME was already restored by the orphan-name
inheritance; the flag adds the heading LINE.

The stale S7 literal is restated where it stood (`c759a657...`, 171 614 B,
re-measured on this HEAD), so the next order cannot inherit the superseded one.

1397 -> 1414 tests. Report: docs/2026-09-08-k3-runde3-per-filtype.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 20:33:52 +02:00
commit 47f6619e6f
11 changed files with 1381 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

85
src/llm_ingestion_okf/cli.py
View file

@ -73,6 +73,25 @@ __all__ = ["DEFAULT_STAMP", "build", "main", "measure"]
CLI_ID = "okf build"
#: What `okf build` runs when no flag is given. Moved 2026-09-08 by the
#: operator, from Arm B to Arm D plus Arm F, on the round-2 and round-3
#: measurements: the previous default was a full arm behind the proposer on
#: the one document with a reference, and the fold matches 5 of 12 documents
#: against the default's 2 with no cell worse.
#:
#: THE VALUE LIVES HERE AND NOT IN `propose.py`. That module's rules stay OFF
#: by default: they are what the golden fixtures and every published
#: reproduction block are pinned to, and the proposer's own CLI is the
#: measuring instrument. What moved is this COMMAND's default, and the two
#: layers disagreeing on purpose is why the disagreement is written down here
#: rather than discovered later.
#:
#: Each arm keeps an explicit opt-out: `--outline-run 0` (the number was
#: always its own switch) and `--no-unit-fold`. A default a caller cannot turn
#: off is not a default.
DEFAULT_OUTLINE_RUN = 3
DEFAULT_UNIT_FOLD = True
#: The timestamp written when the caller passes none, for the ingest stamp and
#: the proposal stamp alike. ONE constant: two independently-defaulted literals
#: drift, and the drift shows up only as two bundles differing in a field
@ -91,6 +110,8 @@ def _propose_plans(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> tuple[int, int, int]:
"""Propose a plan per dropped file. Returns (written, nothing, failed).
@ -119,6 +140,8 @@ def _propose_plans(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
except ProposerError as exc:
print(f"{CLI_ID}: {relative.as_posix()}: {exc}", file=sys.stderr)
@ -142,10 +165,12 @@ def build(
segments: bool = True,
plans_dir: Path | None = None,
okf_type: str = "reference",
outline_run: int = 0,
outline_run: int = DEFAULT_OUTLINE_RUN,
table_grid: bool = False,
unit_fold: bool = False,
unit_fold: bool = DEFAULT_UNIT_FOLD,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> CorpusReport:
"""Folder in, bundle out. The whole command, minus argument parsing.
@ -193,6 +218,8 @@ def build(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
print(
f"{CLI_ID}: proposed {written} plan(s); {nothing} document(s) with no boundary; "
@ -295,14 +322,15 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--outline-run",
type=int,
default=0,
default=DEFAULT_OUTLINE_RUN,
metavar="N",
help=(
"Arm D, passed to the proposer unchanged: also propose a boundary at "
"each line of the document's own numbered outline, where the integers "
"sustain an ascending run of at least N. 0 (the default) is OFF and "
"leaves the bundle byte-identical. Measured on a tender PDF whose "
"headings are bare integers: no boundary at 0, nine at 3"
"sustain an ascending run of at least N. The default is 3 (operator, "
"2026-09-08); pass 0 to turn the arm off, which is this arm's "
"explicit opt-out. Measured on a tender PDF whose headings are bare "
"integers: no boundary at 0, nine at 3"
),
)
build_parser.add_argument(
@ -319,13 +347,25 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
build_parser.add_argument(
"--unit-fold",
action="store_true",
default=DEFAULT_UNIT_FOLD,
help=(
"Arm F, passed to the proposer unchanged: discard a contents-list run, "
"fold a deeper heading into its parent, fold a table back into the "
"shorter heading that introduces it. It adds no boundary, so it can "
"only reduce a plan. Absent (the default) is OFF. Measured on the K3 "
"sample: 5 of 12 documents match the operator's unit worksheet, "
"against 2 for the shipped default, with no cell worse"
"only reduce a plan. ON by default since 2026-09-08 (operator); the "
"flag is kept so a call site that passes it stays valid. Measured on "
"the K3 sample: 5 of 12 documents match the operator's unit "
"worksheet, against 2 for the arm below it, with no cell worse"
),
)
build_parser.add_argument(
"--no-unit-fold",
action="store_false",
dest="unit_fold",
help=(
"Arm F's explicit opt-out. With --outline-run 0 it reproduces the "
"pre-2026-09-08 default byte for byte, which is what keeps the move "
"reversible for a consumer who needs the old bytes"
),
)
build_parser.add_argument(
@ -339,6 +379,31 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"concept gains the heading line it was missing"
),
)
build_parser.add_argument(
"--sheet-section-rows",
action="store_true",
help=(
"D3, passed to the proposer unchanged: cut an open table block at "
"the rows that label its sections -- a RUN of rows whose first cell "
"is a bare numeric label. The opposite direction from Arm E, which "
"decides how far a block extends; this decides where it is cut "
"inside. Absent (the default) is OFF. Measured on a tender price "
"sheet whose whole body is one table block: 1 concept -> 12, "
"against a reference of 11 cost groups plus the sheet's preamble"
),
)
build_parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
help=(
"D3, passed to the proposer unchanged: do not admit an outline "
"candidate whose line continues onto the next one. Recovered "
"candidates only, never a heading the document declares. Absent "
"(the default) is OFF. Measured on the K3 sample: 8 of 34 outline "
"candidates wrap, and they are the quoted regulation paragraphs "
"and the risk-table rows the operator wrote off"
),
)
build_parser.add_argument("--report", type=Path, default=None, help="also write the report")
return parser.parse_args(argv)
@ -363,6 +428,8 @@ def main(argv: list[str] | None = None) -> int:
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
)
except (IngestError, OSError, ValueError) as exc:
print(f"{CLI_ID}: FAILED - {exc}", file=sys.stderr)

211
src/llm_ingestion_okf/propose.py
View file

@ -108,6 +108,20 @@ RULE_OUTLINE = "rule:outline"
#: joined -- never on one whose span merely happens to contain a rule line, so
#: a single-row grid table stays byte-identical to Arm D.
RULE_TABLE_GRID = "rule:table-grid"
#: D3 round 3 only, and it is the first rule in this module that opens a span
#: INSIDE a table rather than at one. Like Arm C, D and E it is NOT one of Topic
#: 2's ported rules and NOT defined upstream -- `docs/2026-09-02-k3-k4-k5-metode.md`
#: contains no occurrence of the word "arm" at all -- so this definition was
#: written for order 20260908T170037Z-3622420612-from-.claude and is reported as
#: the author's. Its axis is a fourth one. Arm C names SIZE, Arm D what the
#: DOCUMENT declared, Arm E what the CONVERTER emitted; this names what the
#: SHEET labelled: a row whose first cell is a bare numeric label, in a run of
#: such rows. It is the opposite DIRECTION from Arm E -- that one stops a rule
#: line from closing a block so a grid table proposes one candidate instead of
#: many, this one cuts an open block at the rows that label its sections -- and
#: the two compose in one order: Arm E decides how far a block extends, this
#: decides where it is cut inside.
RULE_SHEET_SECTION = "rule:sheet-section"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
@ -115,6 +129,7 @@ RULE_NAMES = (
RULE_SIZE_SPLIT,
RULE_OUTLINE,
RULE_TABLE_GRID,
RULE_SHEET_SECTION,
)
#: How many characters of context each side of a quote anchor carries. Enough
@ -211,6 +226,16 @@ _OUTLINE = re.compile(r"^\s{0,4}(?P<number>\d{1,2})[.)]?\s+(?P<title>\S.*?)\s*$"
# through `_segment_path` -- an unstripped page number would become part of a
# filename.
_TRAILING_PAGE_NUMBER = re.compile(r"[\s.]+\d{1,4}\s*$")
# D3's grammar, and it reads a CELL rather than a line. A sheet's section label
# is a bare number, optionally joined to another by a separator where two groups
# were merged (`11+12`), and nothing else: no letters, so a row opening with a
# word is not a section, and no word list, so the rule knows nothing about which
# numbers any real sheet uses.
_SHEET_SECTION_LABEL = re.compile(r"^\d+(?:[+./-]\d+)*$")
# A pipe that pandoc did not escape. Splitting a row on a bare `|` would cut a
# cell containing a literal pipe in half and misread the FIRST cell of the row
# after it, which is the only cell this rule judges.
_UNESCAPED_PIPE = re.compile(r"(?<!\\)\|")
class ProposerError(Exception):
@ -315,6 +340,79 @@ def outline_runs(
return [run for run in runs if len(run) >= minimum]
#: D3. How many CONSECUTIVE numbered rows make a sectioned table. Three, and it
#: is the same bounding device -- and the same number -- `CONTENTS_RUN` uses,
#: for the same reason: a single numbered row is a stated quantity, not a
#: section, and a rule that read one would cut a sheet at every computation
#: basis. NOT swept on a reference, because no reference exists for its value;
#: the corpus SENSITIVITY of the choice is published instead, in
#: `docs/2026-09-08-k3-runde3-per-filtype.md`. Not a CLI knob for the reason
#: `CONTENTS_RUN` is not one.
SHEET_SECTION_RUN = 3
def _wraps_onto_next_line(lines: list[str], index: int) -> bool:
"""True when the line at `index` is a sentence that continues below it.
The test is the next line's first character being lower case. A heading is
a complete line -- the line under it opens a new sentence, is blank, or is
a bullet -- while a hard-wrapped paragraph carries its own continuation.
This is deliberately NOT a length test. Round 2 sorted every outline title
in the K3 sample and measured the two classes overlapping: a real chapter
heading of 88 characters against quoted sentences of 86, 91, 92 and 100, so
no threshold separates them. Measured on the same sample, this axis
separates 8 of 34 -- the four quoted regulation paragraphs and four risk
table rows, and none of the 26 headings the operator kept.
"""
if index + 1 >= len(lines):
return False
following = lines[index + 1].strip()
return bool(following) and following[0].islower()
def _sheet_section_rows(lines: list[str]) -> dict[int, tuple[str, str]]:
"""D3: line index -> (label, row text) for every section row, or empty.
A section row is a table row whose FIRST cell is a bare numeric label and
which carries at least one other non-empty cell -- the label alone names
nothing, and the first non-empty cell after it is what the section is
called. The rows must come in a run of at least :data:`SHEET_SECTION_RUN`
consecutive lines: that is what separates a labelled section column from a
quantity stated on its own row, and it is the module's existing guard
rather than a new one.
Computed over the whole line list before the marking loop, for the reason
the outline runs are: a run is a property of the text, and a forward scan
that decided one row at a time could not know whether the run it is inside
is long enough.
"""
labelled: dict[int, tuple[str, str]] = {}
for index, line in enumerate(lines):
if not _TABLE_ROW.match(line):
continue
cells = [cell.strip() for cell in _UNESCAPED_PIPE.split(line.strip().strip("|"))]
if not cells or not _SHEET_SECTION_LABEL.match(cells[0]):
continue
rest = [cell for cell in cells[1:] if cell]
if not rest:
continue
labelled[index] = (cells[0], rest[0])
sections: dict[int, tuple[str, str]] = {}
ordered = sorted(labelled)
start = 0
while start < len(ordered):
end = start + 1
while end < len(ordered) and ordered[end] == ordered[end - 1] + 1:
end += 1
if end - start >= SHEET_SECTION_RUN:
for position in range(start, end):
sections[ordered[position]] = labelled[ordered[position]]
start = end
return sections
def find_candidates(
text: str,
*,
@ -322,6 +420,8 @@ def find_candidates(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -356,6 +456,14 @@ def find_candidates(
count does not move -- one candidate either way -- and the first byte does.
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
`sheet_section_rows` is D3's gate and it is OFF at False, where the scan is
not run at all. On, a RUN of numbered rows inside an open table block cuts
it: each such row opens a candidate that reaches the next section row, or
the end of the block. It is the only rule here that opens a span inside a
table, and it is its own flag for the same reason D1 is -- a sheet is the
one file type whose units are rows, and every other type reaches this scan
too.
"""
lines = text.splitlines(keepends=True)
offsets: list[int] = []
@ -384,6 +492,10 @@ def find_candidates(
# ordering rather than a property of this corpus.
admitted = {index: title for index, _, title in runs[-1]}
# D3's input, and the same whole-text reasoning as `admitted` above: a run
# is a property of the line list, not of a line.
sections = _sheet_section_rows(lines) if sheet_section_rows else {}
marked: list[tuple[int, Candidate]] = []
in_table = False
# Arm E's state, and all three of these are cleared together on the
@ -398,6 +510,36 @@ def find_candidates(
joined: set[int] = set()
for index, line in enumerate(lines):
if _TABLE_ROW.match(line):
section = sections.get(index)
if section is not None:
label, name = section
# A section row never ALSO opens a table block, even when the
# run starts on the block's first line: two marks on one line
# would give the second an empty span, and the orphan check
# deletes an empty span silently. `open_block` is cleared with
# it, because a block cut here is no longer one span an Arm E
# join could extend.
in_table = True
open_block = None
rule_pending = False
marked.append(
(
index,
Candidate(
title=f"{label} {name}",
# The table sentinel, not a heading depth: a sheet
# row declares no level, and letting it vote in
# Arm F's clause 2 would make a sectioned sheet a
# one-level document.
level=9,
number=label,
rule=RULE_SHEET_SECTION,
start=offsets[index],
end=end_of_text,
),
)
)
continue
if not in_table:
in_table = True
open_block = len(marked)
@ -431,6 +573,13 @@ def find_candidates(
outline_title = admitted.get(index)
if outline_title is not None:
if drop_wrapped_outline and _wraps_onto_next_line(lines, index):
# Filtered at ADMISSION rather than at run selection: the run
# this document sustains is a property of its numbering, and
# re-selecting it from a thinned list would move boundaries on
# documents where nothing wraps. The rule declines candidates;
# it does not rewrite which run won.
continue
outline_match = _OUTLINE.match(line)
assert outline_match is not None, "an admitted index still matches the grammar"
marked.append(
@ -518,7 +667,19 @@ def find_candidates(
end = offsets[following[0][0]] if following else end_of_text
body = text[candidate.start : end]
# The orphan check: everything after the heading line itself.
if not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip():
#
# D3 IS EXEMPT, and the exception is stated rather than worked around:
# the check asks whether anything stands UNDER a candidate's first
# line, which is the right question for a heading and the wrong one for
# a row. A one-row section carries its content in its own cells, so
# every section but the last would be read as bodiless and deleted --
# the rule could not fire at all. It is the same shape as a contents
# list without dot leaders, and it is the reason that one needed
# `Candidate.contents`.
orphan = candidate.rule != RULE_SHEET_SECTION and (
not body.splitlines()[1:] or not "".join(body.splitlines()[1:]).strip()
)
if orphan:
orphaned_name = (candidate.title, candidate.number)
continue
inherited, orphaned_name = orphaned_name, None
@ -869,6 +1030,8 @@ def build_plan(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -880,6 +1043,8 @@ def build_plan(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -951,6 +1116,8 @@ def run(
table_grid: bool = False,
unit_fold: bool = False,
keep_table_heading: bool = False,
sheet_section_rows: bool = False,
drop_wrapped_outline: bool = False,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
@ -1004,6 +1171,8 @@ def run(
table_grid=table_grid,
unit_fold=unit_fold,
keep_table_heading=keep_table_heading,
sheet_section_rows=sheet_section_rows,
drop_wrapped_outline=drop_wrapped_outline,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
@ -1127,6 +1296,44 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"part of an arm: the orphan check is reached by every file type"
),
)
parser.add_argument(
"--sheet-section-rows",
action="store_true",
help=(
"D3: cut an open table block at the rows that label its sections. "
"A section row is one of a RUN of at least SHEET_SECTION_RUN "
"consecutive rows whose first cell is a bare numeric label and "
"which carry at least one other non-empty cell; each opens a "
"candidate reaching the next section row or the end of the block. "
"The opposite direction from Arm E, which stops a grid rule line "
"from CLOSING a block: that arm decides how far a block extends, "
"this rule where it is cut inside, and they compose in that order. "
"Written for a spreadsheet whose whole body is one table block and "
"whose units are rows. Absent (the default) is OFF and leaves every "
"artifact byte-identical. A boolean: the run length is a module "
"constant, not a knob a caller can turn. D3 is the author's "
"definition, written for order "
"20260908T170037Z-3622420612-from-.claude; it is not defined "
"upstream, and the K3 method file does not name it either"
),
)
parser.add_argument(
"--drop-wrapped-outline",
action="store_true",
help=(
"D3: do not admit an outline candidate whose line continues onto "
"the next one, because a wrapped sentence is not a heading. Judges "
"RECOVERED candidates only, never a heading the document declares "
"for itself. Written for quoted regulation text, whose numbered "
"paragraphs match Arm D's grammar exactly; a TITLE LENGTH rule was "
"tried first and falsified, because a real 88-character heading "
"sits between the quoted sentences at 86 and 91. Absent (the "
"default) is OFF and leaves every artifact byte-identical. It is "
"the author's definition, written for order "
"20260908T170037Z-3622420612-from-.claude; it is not defined "
"upstream, and the K3 method file does not name it either"
),
)
parser.add_argument(
"--proposed-at",
default="1970-01-01T00:00:00Z",
@ -1149,6 +1356,8 @@ def main(argv: list[str] | None = None) -> int:
table_grid=args.table_grid,
unit_fold=args.unit_fold,
keep_table_heading=args.keep_table_heading,
sheet_section_rows=args.sheet_section_rows,
drop_wrapped_outline=args.drop_wrapped_outline,
)
except ProposerError as exc:
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)