open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index

Browse source 
Two producer-side findings from the consumer's S7c acid test (ordre 20260908T063454Z-3648220855-from-.claude), both measured on K2 before and after, both with the corpus rebuilt from scratch.

FUNN 3 -- THE FORM. The converter's default markdown writer emits simple tables, which pad every cell out to the width of the widest cell in its column. Measured on the tender's price sheet: one 594-character prose cell produced a 67 244-character whitespace carpet with runs of up to 887 characters between a label and its amount, 19 integral amounts carrying a converter `.0`, and a header row naming one column. The bytes reached a live model in 2 of 11 prompts and 0 of 11 answers. The spreadsheet row now writes pipe tables with `--columns=1` (load-bearing: the pipe writer pads to a width computed from it, so at the default 72 a narrow table gains runs of up to 45). Same sheet after: 11 048 characters, longest run 2, one row per line, 0 artificial `.0`. Spreadsheet-only, and the scoping is pinned by three digests -- the same change moves the odt fixture 1366 -> 1105, so it can fail.

The `.0` rewrite is bounded twice: to a cell whose whole content is such a number, anchored between unescaped pipes, and skipped when the literal is in the workbook's shared string table -- the converter renders the number 92 and the TEXT "92.0" identically, so the output alone cannot tell them apart. Read with zipfile and xml.etree; no new dependency.

FUNN 2 -- THE LOG LINK. `link_log_in_root_index` (95eb271) is removed. Consumption contract SS 9.2 forbids a consumer from enumerating the bundle directory unless the profile says the index is derived, so the index tree is the entire map a consumer may use and everything it links is a document: their navigator returned 630 where our pre-pass counts 629, and a corpus run's own log was citable as content. The log is still written to the bundle root (SPEC section 9); `tools/okf_consume.py` keeps its exclusion for the bundles already built with the link.

K2 rebuilt twice. BEFORE reproduces the consumer's ref exactly (`sha256-tree:f14872a0...c8a92a`, 629 concepts) and their three consume figures to the token (57 289 / 62 149 / 58 401). AFTER: 629 concepts, `merged + coded rejections = 43 = N`, new ref `sha256-tree:c26eed6a...e3261f`, 627 of 629 concepts byte-identical, 1104 of 1108 files identical to the delivered bundle.

ONE REGRESSION, MEASURED AND NOT FIXED: on the mandate-shaped question with the vocabulary bridge the priced concept moves from candidate rank 10 to 19, so `--k 12` withholds it `below_k`; `--cost-vocabulary --k 20` delivers it at 65 912 o200k. The cause is measured rather than argued -- restoring only the concept's title on the new short body ranks it 10 again. The chain ends at the orphan check (`propose.py:461`), which drops the sheet heading once a table block opens two lines below it. That is the already-reported orphan gate, and changing it is a default-ON segmentation rule affecting every document type. The specific question is unaffected: rank 1 before and after. The priced excerpt's budget share falls from 56.5 % to 9.7 %.

11 new tests (RED first), 8 mutations, 8 red, with an unmutated control green each time. One mutation survived twice before the fixture could make it fire, and both survivals are written down. 1279 -> 1287 tests. mypy --strict clean on 28 files. ruff clean. Both proposer goldens byte-unchanged. One frozen literal moved with the fix and is reported rather than hidden.

Report: docs/2026-09-08-prisform-og-loggen-k2.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 10:06:58 +02:00
commit 56ae274246
13 changed files with 1018 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

22
README.md
View file

@ -80,7 +80,7 @@ than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them.
Measured 2026-09-07 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
three files no reader accepts), one
`okf build` invocation replacing the shell loop over `tools/` that produced the
same corpus's bundle on 2026-09-03:
@ -92,15 +92,19 @@ same corpus's bundle on 2026-09-03:
| coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) |
| K1b | `39 + 4 = 43 = N`, exit `0` |
| files written | 1108 |
| identical to the 2026-09-03 bundle | 1107/1108 |
| wall time | 779.43 s total, 18.126 s per file |
| identical to the 2026-09-03 bundle | 1104/1108 |
| wall time | 842.82 s total, 19.600 s per file (re-measured 2026-09-08) |
The one file that differs is the root `index.md`, by exactly one line: the link
to the bundle's own `log.md`, added by a commit that postdates the stored
artifact by fifteen hours. Appending that line to the stored `index.md`
reproduces the new one byte for byte. The command's own byte-identity test
compares it against the two scripts at the current commit, where the two agree
over the whole tree.
The six files that differ are all in the corpus's two spreadsheet documents, and
they are the change reported in `docs/2026-09-08-prisform-og-loggen-k2.md`: a
spreadsheet's tables are now written as pipe tables, so each row is one line
with its cells delimited rather than padded out to the widest cell in the
column. Two concept files are renamed by it, two are removed under their old
names, and the two documents' own `index.md` follow. The root `index.md` is
identical to the stored one again, because this library no longer links the
bundle's `log.md` from it. The command's own byte-identity test compares it
against the two scripts at the current commit, where the two agree over the
whole tree.
## Consume