open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,extract,cli): a title that ends in a number, and a converter's own anchor in a concept id

Browse source 
Round 9: the four rests in STATE's NESTE that needed no operator decision.

CLAUSE 1 CLASSIFIED BY THE NUMBER, NOT THE TITLE. `_TRAILING_PAGE_NUMBER`
admitted a candidate into a contents run by asking whether the title ended in
an integer -- a question about the number. A drawing's dimension chain, a
schematic's labels, a door schedule, a coordinate column and a soil-layer
table all end in integers and name nothing. Measured over the 43-document
corpus: 68 candidates discarded over 11 of 39 readable documents, of which
19 over 5 documents are data rows.

That corrects round 8's own decomposition. Its "four misclassified numeric
tables and seven real contents listings" needs each document on one side, and
two of the eleven are both. Read across all 68 titles rather than the
three-title sample: 5 documents carry a data row, 8 carry a real entry.

`--contents-name` requires a NAME to survive stripping the page number. The
threshold is SWEPT, not chosen, and collapses at both ends: at an alphabetic
run of 1 a door schedule keeps a stray `V` and 13 of 19 are rescued; at 3 the
two-letter section name `VA` stops being a name, falls out of run membership,
and takes `RIB`, `MMI` and `Tittelfelt` below `CONTENTS_RUN` with it -- one
acronym costing four REAL entries. At 2: 16 of 19 rescued, 0 of 49 regressed.
The three not rescued carry a real word and are named rather than rounded off.

THE CONVERTER'S ANCHOR WAS IN THE CONCEPT ID. Pandoc writes a sheet as
`## <name> {#sheet-N}` and a titled slide as `## <title> {#slide-N}`. Because
a filename is reduced FROM the title, the anchor reached both. Operator
authorised the strip 2026-09-09 after the exposure was counted: 2 of 810
concepts on the previous default bundle, 2 of 1108 on Arm B, 1 of 26 on the
operator's folder. Two ids renamed, one of which `portfolio-optimiser` has
cited in writing; both are in the report so that message can be sent.

One rule in one function, read by BOTH title-forming sites -- a rule in only
one would leave the id and the title naming the same concept differently. The
known-negative is the point: `Mal for {kundenavn}` is a title an author wrote.

odt/rtf/pptx MEASURED END TO END FOR THE FIRST TIME, on hand-built documents,
because the corpus denominator is genuinely zero (86 files: 66 pdf, 10 docx,
4 xlsx, 2 zip, 2 smc, 2 doc). `_EVIDENCE` gains a third class rather than
stretching an existing one: `constructed` means the row has met a document,
but not one anyone wrote for their own purposes. odt 1 of 1 declared headings;
pptx 2 of 2 on a deck that declares slide titles and 0 of 2 on one that does
not -- round 7's reading of pptx was a fixture property, not the format; rtf
0 segments, because the container has no heading style and the author's title
is bold text. rtf is the one open finding.

ACCEPTANCE, all four. The 12-position reference is label-identical in BOTH
readings (pdf 7/8, docx 3/3, xlsx 0/1 or 1/1, sheet 10/12 or 11/12). One K2
bundle carrying both changes: 453 concepts / 865 md, hit@8 [1,1,1,1,1,None]
on it AND on Arm B, with the known-negative still reproducing on the new
bytes. `okf project` byte-equal to `okf build`, `diff -r` empty. Consumer
cost is a re-run: 436/832 -> 453/865, digest 21af4a1aa98315cf.

Three published numbers corrected: README's 596 tests (1515), README's "15
concepts out" for `okf project` (that was the O6 defect; it is 26), and O6's
print-mode method, which does not reproduce without --allowedTools.

Report: docs/2026-09-09-k3-runde9-restene.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 21:45:08 +02:00
commit 6ff18fd703
15 changed files with 964 additions and 67 deletions
Download patch file Download diff file Expand all files Collapse all files

42
src/llm_ingestion_okf/cli.py
View file

@ -170,6 +170,19 @@ DEFAULT_FIRST_SPAN_FROM_ZERO = True
#: rule closes each span against the next SURVIVOR. It adds no boundary, so the
#: concept count is unchanged and the cost to a consumer is a re-run.
DEFAULT_CLOSE_SPAN_GAPS = True
#: Round 9. Clause 1 asked whether a title ENDED in a number, which is a
#: question about the number rather than about the title: a drawing's dimension
#: chain, a P&ID's schematic labels, a door schedule and a borehole log's
#: coordinate column all end in integers and name nothing. Measured over the
#: 43-document corpus, clause 1 discarded 68 candidates over 11 of 39 readable
#: documents, and 19 of those over 5 of the 11 were data rows of that kind.
#: The rule requires a NAME to survive stripping the page number; the threshold
#: is swept rather than chosen (`propose.CONTENTS_NAME_RUN`). It rescues 16 of
#: the 19 and 0 of the 49 real contents entries -- the three it does not rescue
#: carry a real word (`- 8.225m Fjell 41`) and this predicate cannot separate
#: them from a contents entry. It only ever REMOVES members from a run, so it
#: can only add concepts, never take one away. Opt-out `--no-contents-name`.
DEFAULT_CONTENTS_NAME = True
#: Round 3's two spreadsheet rules (D1 and D3), held back through rounds 5 and
#: 6 by a RETRIEVAL regression rather than by the reference: they take the
@ -236,6 +249,7 @@ def _propose_plans(
outline_gate: bool = False,
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -272,6 +286,7 @@ def _propose_plans(
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -307,6 +322,7 @@ def build(
outline_gate: bool = DEFAULT_OUTLINE_GATE,
first_span_from_zero: bool = DEFAULT_FIRST_SPAN_FROM_ZERO,
close_span_gaps: bool = DEFAULT_CLOSE_SPAN_GAPS,
contents_name: bool = DEFAULT_CONTENTS_NAME,
pdf_headings: bool = DEFAULT_PDF_HEADINGS,
pdf_headings_reserve: bool = DEFAULT_PDF_HEADINGS_RESERVE,
ocr: bool = DEFAULT_OCR,
@ -377,6 +393,7 @@ def build(
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
pdf_headings=pdf_headings,
pdf_headings_reserve=pdf_headings_reserve,
ocr=ocr,
@ -684,6 +701,30 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"pre-2026-09-11 spans byte for byte"
),
)
build_parser.add_argument(
"--contents-name",
action="store_true",
default=DEFAULT_CONTENTS_NAME,
help=(
"Admit a title into a contents run only when a NAME survives "
"stripping its trailing page number, so a run of data rows is not "
"read as a contents listing and discarded. Measured over the "
"39-document corpus: clause 1 dropped 68 candidates over 11 "
"documents, of which 19 over 5 documents were data rows; the rule "
"rescues 16 of the 19 and 0 of the 49 real contents entries. It "
"only removes members from a run, so it can only add concepts. "
"ON since 2026-09-09"
),
)
build_parser.add_argument(
"--no-contents-name",
action="store_false",
dest="contents_name",
help=(
"The repair's explicit opt-out, for a consumer who needs the "
"pre-2026-09-09 contents-run predicate byte for byte"
),
)
build_parser.add_argument(
"--pdf-headings",
choices=("none", "font", "font-reserve"),
@ -757,6 +798,7 @@ def main(argv: list[str] | None = None) -> int:
outline_gate=args.outline_gate,
first_span_from_zero=args.first_span_from_zero,
close_span_gaps=args.close_span_gaps,
contents_name=args.contents_name,
pdf_headings=args.pdf_headings == "font",
pdf_headings_reserve=args.pdf_headings == "font-reserve",
ocr=args.ocr,

77
src/llm_ingestion_okf/extract.py
View file

@ -66,16 +66,42 @@ _PANDOC_FORMATS: dict[str, str] = {
}
# What each row's behaviour actually rests on, asserted in the suite rather
# than written in a comment that rots. `measured` means real corpus files and a
# hand-counted fasit; `unmeasured` means the corpus contains ZERO files of that
# type, so the row works by construction and has never been checked against a
# document anyone wrote. An unmeasured row must not read as a supported one.
# than written in a comment that rots. THREE classes, and the third exists
# because the first two could not tell the truth about these rows:
#
# measured real CORPUS files and a hand-counted fasit. Someone wrote the
# document for their own purposes and we counted what we got.
# constructed hand-built or generator-built documents with a hand-written
# fasit, and no corpus file at all. The row has now met a
# document end to end -- it is not `unmeasured` -- but a
# document written to exercise it is not a document anyone
# wrote, so it is not `measured` either.
# unmeasured the corpus contains ZERO files of the type AND no document
# has ever been put through the row. It works by construction.
#
# An unmeasured row must not read as a supported one, and neither must a
# constructed one.
#
# THE THREE OFFICE ROWS MOVED unmeasured -> constructed ON 2026-09-09, each on
# its own hand-built document (N = 1, except `pptx` at N = 2):
#
# .odt 1 of 1 declared headings recovered, 1 concept, 0 characters in no
# segment.
# .pptx 2 of 2 declared slide titles recovered on a deck that DECLARES them
# (`<p:ph type="title"/>`), 0 of 2 on one that does not -- the latter
# lands as `Slide 1`/`Slide 2`, which is the converter naming a slide
# it has no title for, not a segmentation failure.
# .rtf 0 declared headings, because the container has no heading style and
# the author's title is bold text. The proposer therefore proposes
# nothing and the document reaches Door B's INBOX as one concept --
# content preserved, structure zero. That is the row's honest result
# and it is the one open finding of the three.
_EVIDENCE: dict[str, str] = {
".docx": "measured",
".xlsx": "measured",
".pptx": "unmeasured",
".odt": "unmeasured",
".rtf": "unmeasured",
".pptx": "constructed",
".odt": "constructed",
".rtf": "constructed",
}
# Load-bearing, all three, and none of them hygiene:
@ -703,6 +729,20 @@ _OPTIONAL_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
# containing a `#` cannot be read as a sheet.
_SHEET_HEADING = re.compile(r"^#{1,6} (?P<name>.*?) \{#sheet-\d+\}$")
# Pandoc's ATTRIBUTE syntax at the end of a heading, which is what the sheet
# and slide anchors above are an instance of. Deliberately NARROW, because the
# known-negative is the whole point: an author writing `Mal for {kundenavn}` or
# `Feltet {"id": 4}` wrote a title, and stripping that would be this same
# defect pointed the other way.
#
# The three narrowings, each doing work: the block must be at the END of the
# title (`$`), it must OPEN with `#` (pandoc's identifier -- `{.class}` and
# `{key=val}` alone are not what any converter here emits, and matching them
# would reach further than measured), and the identifier is the restricted
# character set pandoc actually generates, so a brace holding a space, a quote
# or a colon is not an attribute.
_CONVERTER_ATTRIBUTE = re.compile(r"\s*\{#[A-Za-z0-9_.:-]+\}\s*$")
# A line the converter wrote as part of a pipe table. Whether one of them is
# the table's SEPARATOR is decided by POSITION, never by content: an empty
# spreadsheet row renders as `| | |` and a separator as `|----|----|`, and
@ -713,6 +753,29 @@ _SHEET_HEADING = re.compile(r"^#{1,6} (?P<name>.*?) \{#sheet-\d+\}$")
_TABLE_LINE = "|"
def strip_converter_attribute(title: str) -> str:
"""Remove a trailing pandoc attribute anchor from a heading's title.
ONE definition, read by both title-forming sites: `propose` names a
segment from an ATX heading, `structure` derives a document title from its
leading heading, and a rule living in only one of them would strip the
attribute on one path and leave it on the other -- with the id and the
title then disagreeing about the same concept.
Lives HERE because the attribute is a CONVERTER artefact: `_SHEET_HEADING`
above is the same syntax read for a different purpose, and this module is
the one that knows what pandoc writes. That reading must keep its
attribute, which is why the strip is applied to a title downstream and
never to the extracted text.
RENAMES CONCEPT IDS, by design and with the operator's authorisation
(2026-09-09): a filename is reduced FROM the title, so the two move
together. Measured exposure at the time: 2 of 810 concepts on the default
K2 bundle and 2 of 1108 on Arm B.
"""
return _CONVERTER_ATTRIBUTE.sub("", title)
@dataclass(frozen=True)
class SourceUnits:
"""Where in the ORIGINAL each stretch of the extracted text came from.

93
src/llm_ingestion_okf/propose.py
View file

@ -62,7 +62,7 @@ from pathlib import Path
from typing import Any
from .errors import IngestError
from .extract import extract_text
from .extract import extract_text, strip_converter_attribute
from .materialize import reduce_to_id_grammar
from .segmentation import observed_extractor_version
@ -227,6 +227,16 @@ _OUTLINE = re.compile(r"^\s{0,4}(?P<number>\d{1,2})[.)]?\s+(?P<title>\S.*?)\s*$"
# through `_segment_path` -- an unstripped page number would become part of a
# filename.
_TRAILING_PAGE_NUMBER = re.compile(r"[\s.]+\d{1,4}\s*$")
#: Clause 1's name part. An alphabetic run of at least this many characters is
#: a NAME; anything shorter is a stray letter in a data row. SWEPT over the
#: 43-document corpus at 1, 2 and 3, and it collapses at both ends: at 1 a
#: drawing's door schedule keeps a bare `V` and stays misclassified (13 of 19
#: data rows rescued, not 16), and at 3 the two-letter section name `VA` stops
#: being a name, falls out of run membership, and takes `RIB`, `MMI` and
#: `Tittelfelt` below `CONTENTS_RUN` with it -- one acronym costing four REAL
#: contents entries. At 2 the rescue is 16 of 19 and the regression 0 of 49.
CONTENTS_NAME_RUN = 2
_NAME_PART = re.compile(r"[^\W\d_]{%d,}" % CONTENTS_NAME_RUN)
# D3's grammar, and it reads a CELL rather than a line. A sheet's section label
# is a bare number, optionally joined to another by a separator where two groups
# were merged (`11+12`), and nothing else: no letters, so a row opening with a
@ -292,6 +302,29 @@ def _strip_page_number(title: str) -> str:
return _TRAILING_PAGE_NUMBER.sub("", title)
def bears_a_name(title: str) -> bool:
"""Does a NAME survive stripping the trailing page number?
Clause 1's admission predicate, and the whole of round 9's repair. The
shipped rule asked whether `_TRAILING_PAGE_NUMBER` MATCHED, which is a
question about the number and not about the title -- a drawing's dimension
chain, a P&ID's schematic labels, a door schedule and a borehole log's
coordinate column all end in integers, and none of them names anything.
Measured on the corpus, 19 of the 68 candidates clause 1 discarded over 5
of the 11 documents were rows of that kind.
Module level and importable for the same reason `outline_lines` is: the
reach instrument measures THIS predicate, and an instrument re-implementing
the grammar it measures is measuring a second definition.
Applied to the REST, never to the whole title: `40.23` bears no name and
`Poseprover posisjon 24 7` does. Three corpus data rows carry a real word
(`- 8.225m Fjell 41`) and this predicate cannot separate them from a
contents entry -- that is a stated limit, not a rounding error.
"""
return _NAME_PART.search(_strip_page_number(title).strip()) is not None
def outline_lines(text: str) -> list[tuple[int, int, str]]:
"""Every line the outline grammar admits, as `(line index, integer, title)`.
@ -515,6 +548,7 @@ def find_candidates(
outline_gate: bool = False,
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
) -> list[Candidate]:
"""Every boundary the mechanical rules propose, in document order.
@ -550,6 +584,13 @@ def find_candidates(
It is its own flag and not part of an arm because the orphan check is
reached by every file type, and moving it is a decision about all of them.
`contents_name` is round 9's gate and it is OFF at False, where clause 1
behaves exactly as it did before the rule existed. On, a title joins a
contents run only when a NAME survives stripping its page number, so a run
of data rows is no longer read as a contents listing. It can only ever
REMOVE members from a run, which is why it only ever rescues candidates and
never discards one the shipped rule kept.
`close_span_gaps` is OFF at False, where a mark removed after its
neighbour's span was closed takes that text out of the plan entirely. On,
a span runs to the next SURVIVING candidate's start and the last one runs
@ -716,7 +757,11 @@ def find_candidates(
if atx is None and numbered is None:
continue
if atx is not None:
title = atx.group("title")
# The converter's own anchor is not part of the title, and it must
# come off HERE rather than where the filename is reduced: the id
# is reduced FROM the title, so stripping it downstream would leave
# the two naming the same concept differently.
title = strip_converter_attribute(atx.group("title"))
level = len(atx.group("hashes"))
inner = _NUMBERED.match(title)
number = inner.group("number") if inner else None
@ -772,7 +817,9 @@ def find_candidates(
# Arm F clause 1's input, and it must be read HERE: the orphan pass below
# deletes every bodiless heading, which is every entry of a contents list
# but the last, and a run of one is below `CONTENTS_RUN`.
contents_run = _contents_run_positions(marked) if unit_fold else set()
contents_run = (
_contents_run_positions(marked, contents_name=contents_name) if unit_fold else set()
)
for position_in_list, (_, candidate) in enumerate(marked):
if position_in_list in absorbed:
continue
@ -818,7 +865,7 @@ def find_candidates(
contents=position_in_list in contents_run,
)
)
resolved = fold_units(candidates) if unit_fold else candidates
resolved = fold_units(candidates, contents_name=contents_name) if unit_fold else candidates
if first_span_from_zero and resolved and resolved[0].start > 0:
# Applied AFTER the fold, so the concept that SURVIVES is the one that
# opens at 0: the fold can discard a contents run, and moving the start
@ -869,7 +916,9 @@ def _absorbed_tables(
return absorbed
def _contents_run_positions(marked: list[tuple[int, Candidate]]) -> set[int]:
def _contents_run_positions(
marked: list[tuple[int, Candidate]], *, contents_name: bool = False
) -> set[int]:
"""Arm F clause 1, measured on the PRE-orphan list. Same predicate as there.
A run of at least `CONTENTS_RUN` consecutive page-numbered headings, table
@ -885,19 +934,22 @@ def _contents_run_positions(marked: list[tuple[int, Candidate]]) -> set[int]:
concept while its neighbours were not. What still bounds the rule is the run
LENGTH, which is what the `CONTENTS_RUN` sweep bought and is unchanged.
"""
def member(candidate: Candidate) -> bool:
if candidate.rule == RULE_TABLE_BLOCK:
return False
if not _TRAILING_PAGE_NUMBER.search(candidate.title):
return False
return bears_a_name(candidate.title) if contents_name else True
inside: set[int] = set()
index = 0
while index < len(marked):
candidate = marked[index][1]
if candidate.rule == RULE_TABLE_BLOCK or not _TRAILING_PAGE_NUMBER.search(candidate.title):
if not member(candidate):
index += 1
continue
end = index
while (
end < len(marked)
and marked[end][1].rule != RULE_TABLE_BLOCK
and _TRAILING_PAGE_NUMBER.search(marked[end][1].title)
):
while end < len(marked) and member(marked[end][1]):
end += 1
if end - index >= CONTENTS_RUN:
inside.update(range(index, end))
@ -914,7 +966,7 @@ def _contents_run_positions(marked: list[tuple[int, Candidate]]) -> set[int]:
CONTENTS_RUN = 3
def fold_units(candidates: list[Candidate]) -> list[Candidate]:
def fold_units(candidates: list[Candidate], *, contents_name: bool = False) -> list[Candidate]:
"""Arm F: ONE rule, three clauses, and it only MERGES or DISCARDS.
Derived from the three rules the operator wrote across the K3 unit
@ -950,18 +1002,25 @@ def fold_units(candidates: list[Candidate]) -> list[Candidate]:
# work to do: a contents list WITH dot leaders keeps every entry, and that
# run exists only in this list.
drop: set[int] = {position for position, c in enumerate(candidates) if c.contents}
def member(candidate: Candidate) -> bool:
if candidate.rule == RULE_TABLE_BLOCK:
return False
if not _TRAILING_PAGE_NUMBER.search(candidate.title):
return False
return bears_a_name(candidate.title) if contents_name else True
index = 0
while index < len(candidates):
candidate = candidates[index]
if candidate.rule == RULE_TABLE_BLOCK or not _TRAILING_PAGE_NUMBER.search(candidate.title):
if not member(candidate):
index += 1
continue
end = index
while (
end < len(candidates)
and candidates[end].rule != RULE_TABLE_BLOCK
and candidates[end].level == candidate.level
and _TRAILING_PAGE_NUMBER.search(candidates[end].title)
and member(candidates[end])
):
end += 1
if end - index >= CONTENTS_RUN:
@ -1171,6 +1230,7 @@ def build_plan(
outline_gate: bool = False,
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
) -> dict[str, Any]:
"""The artifact. Every entry PROPOSED, the plan itself never adjudicated."""
taken: set[str] = set()
@ -1187,6 +1247,7 @@ def build_plan(
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
)
for candidate in subdivide(text, candidates, max_segment_chars):
entries.append(
@ -1263,6 +1324,7 @@ def run(
outline_gate: bool = False,
first_span_from_zero: bool = False,
close_span_gaps: bool = False,
contents_name: bool = False,
pdf_headings: bool = False,
pdf_headings_reserve: bool = False,
ocr: bool = False,
@ -1338,6 +1400,7 @@ def run(
outline_gate=outline_gate,
first_span_from_zero=first_span_from_zero,
close_span_gaps=close_span_gaps,
contents_name=contents_name,
)
# Nothing to propose is an OUTCOME, and it is not an artifact. An empty
# plan cannot be replayed -- `process_inbox` refuses one, because a plan

8
src/llm_ingestion_okf/structure.py
View file

@ -34,6 +34,8 @@ from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from .extract import strip_converter_attribute
# A document number is either an alpha-prefixed identifier (`N500`, `V720`,
# `R610.4`) or a dotted numeric section (`4.2.1`). A BARE integer is
# deliberately not a number: `12-things.md` and `2026-notes.md` are ordinary
@ -168,7 +170,11 @@ def _leading_heading(body: str) -> str | None:
# Only a LEADING heading is the document's title. A heading further
# down is a section of the document, and taking it would retitle every
# document whose body happens to open with prose.
return line[2:].strip() if line.startswith("# ") else None
if not line.startswith("# "):
return None
# Same rule as the segment path reads, from the same function: a
# converter anchor must not become a derived document title either.
return strip_converter_attribute(line[2:].strip()).strip()
return None