open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose): Arm C cuts an over-long span at a paragraph boundary

Browse source 
Arm C is NOT defined in docs/2026-09-02-k3-k4-k5-metode.md -- that file
contains no occurrence of the word, and neither Arm A nor Arm B is defined
there either. The definition implemented here was written for order
20260904T145630Z and is reported as the author's, never as a ratified one.

    Arm C = Arm B's mechanical rules, plus one deterministic rule that cuts
    any proposed span longer than a declared cap at the nearest paragraph
    boundary at or before it, the whole document counting as one span when
    the rules find no boundary at all.

One rule and not two, on purpose. The two failure modes the K2 rebuild
measured -- a PDF with no outline (Bilag 9.1, 217 472 characters) and a PDF
whose headings are its table of contents, so the trailing segment absorbs
the body (Bilag 3.1, Bilag 1) -- are the same failure of SIZE, and a second
rule aimed at each would confound which one moved the number.

`--max-segment-chars` defaults to 0, which is OFF: the artifact is then
byte-identical to Arm B's, pinned by a test that writes both and compares
bytes. The standard profile does not move, and the K2 bundle a consumer is
running against right now is not rebuilt.

What Arm C deliberately does NOT change: the region before the first
candidate is still covered by no segment. That is a real coverage defect --
the K3 baseline's blind rater named it -- and fixing it here would put two
changes behind one measurement.

A part carries TWO rule names in `derived`: the heading rule that opened the
span, and `rule:size-split` for the cut. Dropping the first would make a
part traceable to arithmetic and nothing else.

Tests first: 13 red, then green. 1055 -> 1068.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-04 17:06:58 +02:00
commit 0c7a485c11
2 changed files with 340 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

158
tools/okf_propose_segments.py
View file

@ -74,7 +74,18 @@ PROPOSER_VERSION = "1"
RULE_HEADING = "rule:heading"
RULE_TABLE_BLOCK = "rule:table-block"
RULE_POPPLER_SIZE_AND_BOLD = "rule:poppler-size-and-bold"
RULE_NAMES = (RULE_HEADING, RULE_TABLE_BLOCK, RULE_POPPLER_SIZE_AND_BOLD)
#: Arm C only. NOT one of Topic 2's ported rules and not a heading rule at
#: all: it names the fact that a span was cut because it was too long, which
#: is a judgement about SIZE and says nothing about where a unit of knowledge
#: begins. It is emitted ALONGSIDE the rule that proposed the origin span, so
#: an operator reading a part can still see what opened it.
RULE_SIZE_SPLIT = "rule:size-split"
RULE_NAMES = (
RULE_HEADING,
RULE_TABLE_BLOCK,
RULE_POPPLER_SIZE_AND_BOLD,
RULE_SIZE_SPLIT,
)
#: How many characters of context each side of a quote anchor carries. Enough
#: to separate two occurrences of a repeated heading, short enough that an
@ -148,6 +159,10 @@ class Candidate:
rule: str
start: int
end: int
#: True when this candidate is one PART of a longer span that Arm C cut.
#: Kept on the candidate rather than recomputed at write time so the entry
#: and the reason it exists cannot drift apart.
split: bool = False
def _is_stop_word_only(title: str) -> bool:
@ -249,6 +264,102 @@ def find_candidates(text: str) -> list[Candidate]:
return candidates
def _cut_points(text: str, start: int, end: int, cap: int) -> list[int]:
"""Where to cut `text[start:end]` so no part exceeds `cap` characters.
The cut prefers a PARAGRAPH boundary (a blank line) inside the window, then
a line boundary, and only then cuts mid-line. The order is the whole
content of the rule: a cut that lands mid-sentence splits one unit of
knowledge for no reason other than arithmetic, and the K3 categories count
that as `too fine`. The last resort exists anyway, because a document whose
body is one unbroken line is exactly where a cap that quietly stopped
binding would be least defensible.
"""
cuts: list[int] = []
position = start
while end - position > cap:
window_end = position + cap
paragraph = text.rfind("\n\n", position, window_end)
if paragraph != -1:
cut = paragraph + 2
else:
line = text.rfind("\n", position, window_end)
cut = line + 1 if line != -1 else window_end
# rfind can only return an index at or after `position`, so every
# branch advances. The assertion states that rather than trusting it:
# a cut that did not advance would loop forever on a corpus run.
assert cut > position, f"cut {cut} did not advance past {position}"
cuts.append(cut)
position = cut
return cuts
def subdivide(text: str, candidates: list[Candidate], cap: int) -> list[Candidate]:
"""Arm C. Arm B's candidates, with every over-long span cut down to `cap`.
ARM C IS NOT DEFINED IN `docs/2026-09-02-k3-k4-k5-metode.md`; that file
contains no occurrence of the word. This definition was written for order
20260904T145630Z and is reported as the author's, not as a ratified one.
Two callers' cases, one rule. When Arm B found boundaries but a span still
runs long (a PDF whose headings are its table of contents, so the trailing
segment absorbs the body), the span is cut. When Arm B found NO boundary at
all, the whole document is that span -- which is the `no declared
structure` case § 10 names, and 23 of 33 PDFs in the K2 corpus are in it.
A document with no boundaries that is already under the cap proposes
NOTHING, exactly as Arm B does. Arm C fires on size; where size is not the
problem it has nothing to say, and a one-entry plan would only dress a
single concept in a plan file.
"""
if cap <= 0:
return candidates
if not candidates:
if len(text) <= cap:
return []
# The synthetic span. Its rule is the size rule alone, because no
# heading rule proposed it -- there was no heading.
candidates = [
Candidate(
title="Del",
level=1,
number=None,
rule=RULE_SIZE_SPLIT,
start=0,
end=len(text),
split=False,
)
]
unnumbered_parts = True
else:
unnumbered_parts = False
out: list[Candidate] = []
for candidate in candidates:
cuts = _cut_points(text, candidate.start, candidate.end, cap)
if not cuts:
out.append(candidate)
continue
edges = [candidate.start, *cuts, candidate.end]
for part, (start, end) in enumerate(zip(edges, edges[1:]), start=1):
if unnumbered_parts:
title = f"Del {part}"
else:
title = candidate.title if part == 1 else f"{candidate.title} (del {part})"
out.append(
Candidate(
title=title,
level=candidate.level,
number=candidate.number,
rule=candidate.rule,
start=start,
end=end,
split=True,
)
)
return out
def _segment_path(candidate: Candidate, taken: set[str], prefix: str = "") -> str:
title = unicodedata.normalize("NFC", candidate.title)
# The section number becomes the DIRECTORY, so leaving it in the stem too
@ -281,12 +392,13 @@ def build_plan(
okf_type: str,
proposed_at: str,
path_prefix: str = "",
max_segment_chars: int = 0,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
extractor_id = source.suffix.lower().lstrip(".") or "none"
entries: list[dict[str, Any]] = []
for candidate in find_candidates(text):
for candidate in subdivide(text, find_candidates(text), max_segment_chars):
entries.append(
{
"segment_id": f"p{len(entries) + 1}",
@ -309,7 +421,16 @@ def build_plan(
# this library's existing "which of these did we infer" marker,
# so a consumer that already distrusts derived fields
# distrusts these by construction.
"derived": [PROPOSED_MARKER, candidate.rule],
# PROPOSED first, then the rule that proposed the span, then
# -- for an Arm C part only -- the size rule that cut it. Two
# names rather than one on those entries: the heading rule is
# still what opened the span, and dropping it would make a part
# untraceable to anything but arithmetic.
"derived": (
[PROPOSED_MARKER, candidate.rule, RULE_SIZE_SPLIT]
if candidate.split and candidate.rule != RULE_SIZE_SPLIT
else [PROPOSED_MARKER, candidate.rule]
),
}
)
return {
@ -334,7 +455,20 @@ def build_plan(
}
def run(source: Path, out: Path, *, okf_type: str, proposed_at: str, path_prefix: str = "") -> int:
def run(
source: Path,
out: Path,
*,
okf_type: str,
proposed_at: str,
path_prefix: str = "",
max_segment_chars: int = 0,
) -> int:
if max_segment_chars < 0:
raise ProposerError(
f"--max-segment-chars {max_segment_chars} is negative; the cap is a "
"character count, and 0 means off (Arm B)"
)
# Reduced HERE, before anything is read: a prefix that survives to the
# entries as an empty component would produce exactly the unscoped paths
# the caller asked to avoid, and would do it silently.
@ -363,6 +497,7 @@ def run(source: Path, out: Path, *, okf_type: str, proposed_at: str, path_prefix
okf_type=okf_type,
proposed_at=proposed_at,
path_prefix=scope,
max_segment_chars=max_segment_chars,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan
@ -408,6 +543,20 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"the bundle"
),
)
parser.add_argument(
"--max-segment-chars",
type=int,
default=0,
metavar="N",
help=(
"Arm C: cut any proposed span longer than N characters at the nearest "
"paragraph boundary, the whole document counting as one span when the "
"mechanical rules find no boundary at all. 0 (the default) is OFF and "
"leaves the artifact byte-identical to Arm B. Arm C is the author's "
"definition, written for order 20260904T145630Z; it is not defined in "
"the K3 method file"
),
)
parser.add_argument(
"--proposed-at",
default="1970-01-01T00:00:00Z",
@ -425,6 +574,7 @@ def main(argv: list[str] | None = None) -> int:
okf_type=args.okf_type,
proposed_at=args.proposed_at,
path_prefix=args.path_prefix,
max_segment_chars=args.max_segment_chars,
)
except ProposerError as exc:
print(f"{PROPOSER_ID}: FAILED - {exc}", file=sys.stderr)