open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index

Browse source 
Two producer-side findings from the consumer's S7c acid test (ordre 20260908T063454Z-3648220855-from-.claude), both measured on K2 before and after, both with the corpus rebuilt from scratch.

FUNN 3 -- THE FORM. The converter's default markdown writer emits simple tables, which pad every cell out to the width of the widest cell in its column. Measured on the tender's price sheet: one 594-character prose cell produced a 67 244-character whitespace carpet with runs of up to 887 characters between a label and its amount, 19 integral amounts carrying a converter `.0`, and a header row naming one column. The bytes reached a live model in 2 of 11 prompts and 0 of 11 answers. The spreadsheet row now writes pipe tables with `--columns=1` (load-bearing: the pipe writer pads to a width computed from it, so at the default 72 a narrow table gains runs of up to 45). Same sheet after: 11 048 characters, longest run 2, one row per line, 0 artificial `.0`. Spreadsheet-only, and the scoping is pinned by three digests -- the same change moves the odt fixture 1366 -> 1105, so it can fail.

The `.0` rewrite is bounded twice: to a cell whose whole content is such a number, anchored between unescaped pipes, and skipped when the literal is in the workbook's shared string table -- the converter renders the number 92 and the TEXT "92.0" identically, so the output alone cannot tell them apart. Read with zipfile and xml.etree; no new dependency.

FUNN 2 -- THE LOG LINK. `link_log_in_root_index` (95eb271) is removed. Consumption contract SS 9.2 forbids a consumer from enumerating the bundle directory unless the profile says the index is derived, so the index tree is the entire map a consumer may use and everything it links is a document: their navigator returned 630 where our pre-pass counts 629, and a corpus run's own log was citable as content. The log is still written to the bundle root (SPEC section 9); `tools/okf_consume.py` keeps its exclusion for the bundles already built with the link.

K2 rebuilt twice. BEFORE reproduces the consumer's ref exactly (`sha256-tree:f14872a0...c8a92a`, 629 concepts) and their three consume figures to the token (57 289 / 62 149 / 58 401). AFTER: 629 concepts, `merged + coded rejections = 43 = N`, new ref `sha256-tree:c26eed6a...e3261f`, 627 of 629 concepts byte-identical, 1104 of 1108 files identical to the delivered bundle.

ONE REGRESSION, MEASURED AND NOT FIXED: on the mandate-shaped question with the vocabulary bridge the priced concept moves from candidate rank 10 to 19, so `--k 12` withholds it `below_k`; `--cost-vocabulary --k 20` delivers it at 65 912 o200k. The cause is measured rather than argued -- restoring only the concept's title on the new short body ranks it 10 again. The chain ends at the orphan check (`propose.py:461`), which drops the sheet heading once a table block opens two lines below it. That is the already-reported orphan gate, and changing it is a default-ON segmentation rule affecting every document type. The specific question is unaffected: rank 1 before and after. The priced excerpt's budget share falls from 56.5 % to 9.7 %.

11 new tests (RED first), 8 mutations, 8 red, with an unmutated control green each time. One mutation survived twice before the fixture could make it fire, and both survivals are written down. 1279 -> 1287 tests. mypy --strict clean on 28 files. ruff clean. Both proposer goldens byte-unchanged. One frozen literal moved with the fix and is reported rather than hidden.

Report: docs/2026-09-08-prisform-og-loggen-k2.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 10:06:58 +02:00
commit 56ae274246
13 changed files with 1018 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

174
tests/test_extract.py
View file

@ -9,7 +9,9 @@ call — it is pure, deterministic plumbing.
from __future__ import annotations
import hashlib
import importlib.util
import re
import sys
from pathlib import Path
@ -325,10 +327,12 @@ DOCX_TEXT = "# Krav til helning\n\n60 og 70 1:15"
# xlsx: the sheet name becomes a heading and the rows become a table. The
# label/value pairing survives on one row, which is the property that matters.
XLSX_TEXT = (
"## Krav {#sheet-1}\n\n Krav til helning \n"
" ------------------ ------\n 60 og 70 1:15"
)
#
# THIS LITERAL MOVED ONCE, deliberately, and the move is the fix reported in
# `docs/2026-09-08-prisform-og-loggen-k2.md`: the spreadsheet row now writes
# pipe tables, so the cells arrive delimited instead of padded. Every character
# of content is the same; only the table form changed.
XLSX_TEXT = "## Krav {#sheet-1}\n\n| Krav til helning | |\n|----|----|\n| 60 og 70 | 1:15 |"
# The negative control, committed rather than described: the SAME document
# without `word/styles.xml`. The body survives and the heading marker does not.
@ -457,3 +461,165 @@ def test_empty_csv_fails_fast() -> None:
with pytest.raises(ExtractionError) as excinfo:
extract_text("empty.csv", b"")
assert excinfo.value.code == "extractor_empty_csv"
# --- xlsx: the spreadsheet form a consumer has to read ---
#
# Measured on K2 and reported upstream by the first consumer to read the bundle
# with a live model (`docs/2026-09-08-prisform-og-loggen-k2.md`): the converter's
# DEFAULT markdown writer emits simple tables, which pad every cell out to the
# width of the widest cell in its column. One long prose cell therefore turns
# every other row in that column into a whitespace carpet -- 887 characters
# between a label and its amount on the real sheet -- while the header row names
# a single column because only the first source cell in row 1 is filled. The
# bytes reach the reader and the STRUCTURE does not.
#
# `prisark.xlsx` is that shape in miniature, hand-laid rather than recorded, and
# it carries its own negative control on a second sheet.
PRISARK = "prisark.xlsx"
def _table_rows(text: str) -> list[list[str]]:
"""Every pipe-table row in `text`, as its cells, in order."""
rows = []
for line in text.split("\n"):
stripped = line.strip()
if not (stripped.startswith("|") and stripped.endswith("|")):
continue
cells = [cell.strip() for cell in stripped[1:-1].split("|")]
if all(set(cell) <= set("-:") and cell for cell in cells):
continue # the header separator is punctuation, not a row
rows.append(cells)
return rows
@requires_pandoc
def test_a_spreadsheet_keeps_its_columns_one_row_per_line() -> None:
"""The label and the amount arrive as separate cells on one line.
This is the property the whole change exists for. Asserted as properties
rather than only as a frozen literal, because a literal pins bytes and says
nothing about which of them was the point.
"""
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
text = extract_text(PRISARK, data)
rows = _table_rows(text)
assert [
"01",
"Rigging og drift av byggeplass, medregnet alt som ikke er "
"priset spesifikt nedenfor og alt som er innkalkulert i de angitte "
"prisene",
"5647500",
] in rows
assert ["02", "Andel", "12.5"] in rows
longest = max((len(run) for run in re.findall(r" {2,}", text)), default=0)
assert longest <= 8, f"a whitespace run of {longest} is a carpet, not a column"
@requires_pandoc
def test_an_integral_amount_loses_the_converters_decimal_and_a_real_one_keeps_it() -> None:
"""`5647500` is a number; `92.0` in the same sheet is TEXT.
The converter renders both as `<digits>.0`, so the output alone cannot tell
them apart. The shared string table can, and is what the rewrite consults --
which is why this test asserts both directions from ONE document.
"""
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
text = extract_text(PRISARK, data)
assert "5647500.0" not in text
assert "250000.0" not in text
assert "12.5" in text, "a genuine decimal is a value, not a converter artefact"
assert ["03", "92.0", "250000"] in _table_rows(text), (
"a shared-string cell reading 92.0 is author text and survives verbatim"
)
assert "Kode 4 \\| 5.0" in text, (
"a `5.0` INSIDE a cell is not a cell: the delimiter test is what sees that"
)
@requires_pandoc
def test_a_single_column_sheet_gains_no_columns() -> None:
"""The negative control, in the same document as the case it controls.
Sheet 2 has ONE column in the source. There is nothing to recover, so the
fix must not invent a second cell anywhere on it. Its three values arrive
in order and alone.
It is NOT byte-identical before and after the change, and that is measured
rather than glossed: the writer emits a pipe table for every table it
writes, so a one-column table changes delimiter form too. What must not
change is the cell content and the column count.
"""
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
text = extract_text(PRISARK, data)
single = text.split("## Enkeltkolonne")[1]
rows = _table_rows(single)
assert [cells for cells in rows if any(cells)] == [
["Notat"],
["Ingen kolonner her"],
["Sum ikke oppgitt"],
]
# The scoping control. The same writer change applied to the other four office
# rows was MEASURED to move them (the odt fixture 1366 -> 1105 characters), so
# this digest can fail; it is not a tautology. The change is deliberately
# spreadsheet-only: a spreadsheet IS a grid and has no prose fallback, while
# moving docx/pptx/odt/rtf would move a corpus denominator nothing has measured.
# A red here means the writer stopped being scoped -- read the diff and decide.
OFFICE_TEXT_DIGESTS = {
"k2-office/krav-tekstdokument.odt": (
"58c9776f0d7f2b2a3a9d2774e4ae243b265c31b5b6b96914ef4db419fa66e4e2"
),
"k2-office/krav-presentasjon.pptx": (
"752420a04d651a416938ff9f0b3c2de5849bd2ea1d52063dd88aaae65ab99b90"
),
"k2-office/krav-rikt-tekstformat.rtf": (
"79cbf756eb482bb603f82c171d11efe74b9ba62ab9ef679ecd6bb8c3b3740ffb"
),
}
@requires_pandoc
@pytest.mark.parametrize("relative", sorted(OFFICE_TEXT_DIGESTS))
def test_the_other_office_rows_are_untouched_by_the_spreadsheet_writer(relative: str) -> None:
path = FIXTURES / relative
with pytest.warns(ExtractionWarning):
text = extract_text(path.name, path.read_bytes())
assert hashlib.sha256(text.encode("utf-8")).hexdigest() == OFFICE_TEXT_DIGESTS[relative]
# The whole fixture, frozen against the same named converter version as the
# literals above. The property tests say WHAT matters; this one catches any
# other byte moving without anybody noticing.
PRISARK_TEXT = (
"## Prisark {#sheet-1}\n\n"
"| Prisskjema | | |\n"
"|----|----|----|\n"
"| Post | Beskrivelse | Sum |\n"
"| 01 | Rigging og drift av byggeplass, medregnet alt som ikke er priset "
"spesifikt nedenfor og alt som er innkalkulert i de angitte prisene | 5647500 |\n"
"| 02 | Andel | 12.5 |\n"
"| 03 | 92.0 | 250000 |\n"
"| 04 | Kode 4 \\| 5.0 | |\n\n"
"## Enkeltkolonne {#sheet-2}\n\n"
"| Notat |\n"
"|----|\n"
"| Ingen kolonner her |\n"
"| Sum ikke oppgitt |"
)
@requires_pandoc
def test_prisark_extracts_to_its_frozen_text() -> None:
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
assert extract_text(PRISARK, data) == PRISARK_TEXT