open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index

Browse source 
Two producer-side findings from the consumer's S7c acid test (ordre 20260908T063454Z-3648220855-from-.claude), both measured on K2 before and after, both with the corpus rebuilt from scratch.

FUNN 3 -- THE FORM. The converter's default markdown writer emits simple tables, which pad every cell out to the width of the widest cell in its column. Measured on the tender's price sheet: one 594-character prose cell produced a 67 244-character whitespace carpet with runs of up to 887 characters between a label and its amount, 19 integral amounts carrying a converter `.0`, and a header row naming one column. The bytes reached a live model in 2 of 11 prompts and 0 of 11 answers. The spreadsheet row now writes pipe tables with `--columns=1` (load-bearing: the pipe writer pads to a width computed from it, so at the default 72 a narrow table gains runs of up to 45). Same sheet after: 11 048 characters, longest run 2, one row per line, 0 artificial `.0`. Spreadsheet-only, and the scoping is pinned by three digests -- the same change moves the odt fixture 1366 -> 1105, so it can fail.

The `.0` rewrite is bounded twice: to a cell whose whole content is such a number, anchored between unescaped pipes, and skipped when the literal is in the workbook's shared string table -- the converter renders the number 92 and the TEXT "92.0" identically, so the output alone cannot tell them apart. Read with zipfile and xml.etree; no new dependency.

FUNN 2 -- THE LOG LINK. `link_log_in_root_index` (95eb271) is removed. Consumption contract SS 9.2 forbids a consumer from enumerating the bundle directory unless the profile says the index is derived, so the index tree is the entire map a consumer may use and everything it links is a document: their navigator returned 630 where our pre-pass counts 629, and a corpus run's own log was citable as content. The log is still written to the bundle root (SPEC section 9); `tools/okf_consume.py` keeps its exclusion for the bundles already built with the link.

K2 rebuilt twice. BEFORE reproduces the consumer's ref exactly (`sha256-tree:f14872a0...c8a92a`, 629 concepts) and their three consume figures to the token (57 289 / 62 149 / 58 401). AFTER: 629 concepts, `merged + coded rejections = 43 = N`, new ref `sha256-tree:c26eed6a...e3261f`, 627 of 629 concepts byte-identical, 1104 of 1108 files identical to the delivered bundle.

ONE REGRESSION, MEASURED AND NOT FIXED: on the mandate-shaped question with the vocabulary bridge the priced concept moves from candidate rank 10 to 19, so `--k 12` withholds it `below_k`; `--cost-vocabulary --k 20` delivers it at 65 912 o200k. The cause is measured rather than argued -- restoring only the concept's title on the new short body ranks it 10 again. The chain ends at the orphan check (`propose.py:461`), which drops the sheet heading once a table block opens two lines below it. That is the already-reported orphan gate, and changing it is a default-ON segmentation rule affecting every document type. The specific question is unaffected: rank 1 before and after. The priced excerpt's budget share falls from 56.5 % to 9.7 %.

11 new tests (RED first), 8 mutations, 8 red, with an unmutated control green each time. One mutation survived twice before the fixture could make it fire, and both survivals are written down. 1279 -> 1287 tests. mypy --strict clean on 28 files. ruff clean. Both proposer goldens byte-unchanged. One frozen literal moved with the fix and is reported rather than hidden.

Report: docs/2026-09-08-prisform-og-loggen-k2.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 10:06:58 +02:00
commit 56ae274246
13 changed files with 1018 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

3
src/llm_ingestion_okf/cli.py
View file

@ -52,7 +52,7 @@ import sys
import tempfile
from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, link_log_in_root_index, load_plans, measure
from .corpus import LOG_NAME, CorpusReport, load_plans, measure
from .errors import IngestError
from .inbox import walk_inbox
from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
@ -187,7 +187,6 @@ def _write_log(bundle: Path, report: CorpusReport, *, profile: BundleProfile) ->
"""
bundle.mkdir(parents=True, exist_ok=True)
(bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="")
link_log_in_root_index(bundle, profile)
def parse_args(argv: list[str] | None) -> argparse.Namespace:

60
src/llm_ingestion_okf/corpus.py
View file

@ -376,45 +376,26 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
return parser.parse_args(argv)
def link_log_in_root_index(bundle: Path, profile: BundleProfile) -> None:
"""Point the root index at the log, so the walk section 8 supports reaches it.
Measured on the K2 artifact: the bundle carried a conformant root `log.md`
that no index named, so a consumer entering at `index.md` never reached the
one file carrying `N`.
A LOCAL choice, not a conformance requirement, and the difference is worth
stating rather than implying. Section 9 puts `log.md` at any level and
section 8 has an index enumerate its directory's contents, but upstream's
own bundles do not link it: measured at `9a15b13`, 0 of the 24 shipped
`index.md` files name the single `log.md` in the set. Upstream therefore
shows the link is not REQUIRED -- not that it is disallowed.
It belongs to the harness and not the library. The log's content IS the
run's outcome, so it cannot exist when the indexes are projected; an index
that enumerated it off the directory would gain the link only from the
second run onward and break rebuild-equals-incremental, the property the
segmented bundle is built on. Writing it after the log instead keeps both
runs identical.
THE MEMBERSHIP TEST IS LOAD-BEARING, and measured rather than assumed: the
two reprojections do not treat this line the same way. The per-directory
one drops every managed line before re-emitting its block, so the link is
gone by the time this runs. The flat one keeps a managed line whose target
is not an owned concept -- deliberately, because claiming somebody else's
link on the strength of a regex would delete curated content -- so `log.md`
survives there. Appending unconditionally therefore doubled the entry on
the second unsegmented run. Re-writing the line only when it is absent is
idempotent under both, without either side having to know about the other.
"""
index_path = bundle / profile.index.name
if not index_path.is_file():
return
body = index_path.read_text(encoding="utf-8")
link = profile.index.render_link(LOG_TITLE, LOG_NAME) + "\n"
if link in body.splitlines(keepends=True):
return
index_path.write_text(body + link, encoding="utf-8", newline="")
# `link_log_in_root_index` LIVED HERE AND WAS REMOVED (2026-09-08).
#
# It appended `- [Corpus run history](log.md)` to the root index (`95eb271`) so
# a reader entering at `index.md` could reach the one file carrying `N`. That
# was a LOCAL choice, said so at the time, and upstream never linked its own
# logs -- measured at `9a15b13`, 0 of 24 shipped `index.md` files name the one
# `log.md` in the set.
#
# The cost was measured on K2 by the first consumer to walk a bundle of ours
# with a live model: consumption contract SS 9.2 forbids a consumer from
# enumerating the bundle directory unless the profile says the index is
# derived, so the index tree IS the entire map a consumer may use, and anything
# it links is a document. Their navigator returned 630 where our own pre-pass
# counts 629, and a corpus run's own log became readable and citable as
# content. `5a0c879` (F2) excluded `log.md` from OUR walk, which fixed the
# count on one side of a disagreement produced on the other.
#
# The log itself is still written to the bundle root, which is where SS 9 puts
# it and all F2 ever needed. Reported in
# `docs/2026-09-08-prisform-og-loggen-k2.md`.
def main(argv: list[str] | None = None) -> int:
@ -471,7 +452,6 @@ def main(argv: list[str] | None = None) -> int:
# as it was before.
bundle.mkdir(parents=True, exist_ok=True)
(bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="")
link_log_in_root_index(bundle, profile)
print(report.render())
if report.unaccounted or report.merged + report.rejected != report.n:
print(

91
src/llm_ingestion_okf/extract.py
View file

@ -26,8 +26,11 @@ from __future__ import annotations
import csv
import functools
import io
import re
import tempfile
import warnings
import zipfile
from xml.etree import ElementTree
from collections.abc import Callable, Sequence
from html.parser import HTMLParser
from pathlib import Path
@ -85,6 +88,39 @@ _EVIDENCE: dict[str, str] = {
_PANDOC_WRITER = "markdown"
_PANDOC_ARGS = ("--eol=lf", "--wrap=none")
# The spreadsheet row writes PIPE tables, and it is the only row that does.
#
# The default writer prefers simple tables, which pad every cell out to the
# width of the widest cell in its column. Measured on the K2 price sheet: one
# 594-character prose cell turned every other row in that column into a run of
# up to 887 spaces between a label and its amount, 100 795 characters in all,
# and the header row named ONE column because only the first cell of the source
# row 1 is filled. The bytes reached the reader and the structure did not. The
# same sheet through this writer is 11 221 characters with no whitespace run
# longer than two, one row per line, each source column its own cell.
#
# `--columns=1` is load-bearing rather than cosmetic: the pipe writer pads cells
# out to the column width it computes from that setting, so at the default 72 a
# NARROW table gains runs of up to 45 spaces -- the same defect at a smaller
# scale. Measured across every office fixture and every K2 office file, the
# longest whitespace run with it is 2.
#
# SPREADSHEET-ONLY, deliberately. The other four rows have the same defect
# available to the same one-line fix (measured: the odt fixture 1366 -> 1105
# characters), but a spreadsheet IS a grid with no prose fallback, while moving
# the prose rows would move a corpus denominator that nothing has measured.
# `tests/test_extract.py` pins that scoping with three digests.
_SPREADSHEET_WRITER = "markdown-simple_tables-multiline_tables-grid_tables"
_SPREADSHEET_ARGS = (*_PANDOC_ARGS, "--columns=1")
# SpreadsheetML's namespace, needed to read the workbook's shared string table.
_SSML = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
# A table cell whose whole content is an integer with the converter's trailing
# `.0`. Bounded by unescaped pipes on both sides so a cell containing an escaped
# `\|` can never be split in the middle.
_INTEGRAL_CELL = re.compile(r"(?<!\\)\|(\s*)(-?\d+)\.0(\s*)(?=(?<!\\)\|)")
# Conversion recovers text, on the same terms as PDF extraction: a drawing has
# no text to recover. Said out loud on every conversion rather than detected
# per document, for the same reason.
@ -290,8 +326,11 @@ def _extract_office(suffix: str, data: bytes) -> str:
except ImportError as exc:
raise _extra_missing(suffix) from exc
spreadsheet = suffix == ".xlsx"
writer = _SPREADSHEET_WRITER if spreadsheet else _PANDOC_WRITER
args = _SPREADSHEET_ARGS if spreadsheet else _PANDOC_ARGS
try:
text = _convert_bytes(data, _PANDOC_WRITER, _PANDOC_FORMATS[suffix], _PANDOC_ARGS)
text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
except ExtractionError:
raise
except Exception as exc: # noqa: BLE001 - third-party converter, wrapped never leaked
@ -308,12 +347,62 @@ def _extract_office(suffix: str, data: bytes) -> str:
code="extractor_empty_conversion",
)
if spreadsheet:
text = _drop_converter_decimals(text, data)
# After the parse, not before: a run that produced no text has nothing to
# be lossy about, and warning there would just add noise to a failure.
warnings.warn(_OFFICE_LOSSY_WARNING, ExtractionWarning, stacklevel=3)
return text
def _shared_strings(data: bytes) -> frozenset[str]:
"""Every literal in a workbook's shared string table, or nothing.
Read for one purpose: to tell a NUMBER from TEXT THAT LOOKS LIKE ONE. The
converter renders a numeric cell as a double, so an integral value arrives
as `5647500.0` -- and a text cell reading `92.0` arrives as `92.0` too. The
output alone cannot separate them, and rewriting on the output alone would
silently edit somebody's authored text.
Shared strings are the only text the converter recovers from a sheet at
all: an inline string (`t="inlineStr"`) is read as an EMPTY cell, measured
while the first xlsx fixture was built (`tests/fixtures/README.md`). So a
`<digits>.0` that is not in this set did not come from text.
Every failure returns the empty set, which makes the rewrite a no-op rather
than a guess: a workbook this cannot read keeps its converter decimals.
"""
try:
with zipfile.ZipFile(io.BytesIO(data)) as archive:
raw = archive.read("xl/sharedStrings.xml")
root = ElementTree.fromstring(raw)
except (KeyError, OSError, zipfile.BadZipFile, ElementTree.ParseError):
return frozenset()
return frozenset(
"".join(node.text or "" for node in item.iter(f"{{{_SSML}}}t")) for item in root
)
def _drop_converter_decimals(text: str, data: bytes) -> str:
"""Undo the converter's `N.0` on cells the workbook stores as integers.
Cell-scoped and never applied to prose: the pattern is anchored between two
unescaped pipes, so only a cell whose ENTIRE content is an integer with a
trailing `.0` is rewritten, and only when that same literal is absent from
the shared string table.
"""
literals = _shared_strings(data)
def rewrite(match: re.Match[str]) -> str:
digits = match.group(2)
if f"{digits}.0" in literals:
return match.group(0)
return f"|{match.group(1)}{digits}{match.group(3)}"
return _INTEGRAL_CELL.sub(rewrite, text)
_CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
".md": _extract_passthrough,
".txt": _extract_passthrough,