open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index

Browse source 
Two producer-side findings from the consumer's S7c acid test (ordre 20260908T063454Z-3648220855-from-.claude), both measured on K2 before and after, both with the corpus rebuilt from scratch.

FUNN 3 -- THE FORM. The converter's default markdown writer emits simple tables, which pad every cell out to the width of the widest cell in its column. Measured on the tender's price sheet: one 594-character prose cell produced a 67 244-character whitespace carpet with runs of up to 887 characters between a label and its amount, 19 integral amounts carrying a converter `.0`, and a header row naming one column. The bytes reached a live model in 2 of 11 prompts and 0 of 11 answers. The spreadsheet row now writes pipe tables with `--columns=1` (load-bearing: the pipe writer pads to a width computed from it, so at the default 72 a narrow table gains runs of up to 45). Same sheet after: 11 048 characters, longest run 2, one row per line, 0 artificial `.0`. Spreadsheet-only, and the scoping is pinned by three digests -- the same change moves the odt fixture 1366 -> 1105, so it can fail.

The `.0` rewrite is bounded twice: to a cell whose whole content is such a number, anchored between unescaped pipes, and skipped when the literal is in the workbook's shared string table -- the converter renders the number 92 and the TEXT "92.0" identically, so the output alone cannot tell them apart. Read with zipfile and xml.etree; no new dependency.

FUNN 2 -- THE LOG LINK. `link_log_in_root_index` (95eb271) is removed. Consumption contract SS 9.2 forbids a consumer from enumerating the bundle directory unless the profile says the index is derived, so the index tree is the entire map a consumer may use and everything it links is a document: their navigator returned 630 where our pre-pass counts 629, and a corpus run's own log was citable as content. The log is still written to the bundle root (SPEC section 9); `tools/okf_consume.py` keeps its exclusion for the bundles already built with the link.

K2 rebuilt twice. BEFORE reproduces the consumer's ref exactly (`sha256-tree:f14872a0...c8a92a`, 629 concepts) and their three consume figures to the token (57 289 / 62 149 / 58 401). AFTER: 629 concepts, `merged + coded rejections = 43 = N`, new ref `sha256-tree:c26eed6a...e3261f`, 627 of 629 concepts byte-identical, 1104 of 1108 files identical to the delivered bundle.

ONE REGRESSION, MEASURED AND NOT FIXED: on the mandate-shaped question with the vocabulary bridge the priced concept moves from candidate rank 10 to 19, so `--k 12` withholds it `below_k`; `--cost-vocabulary --k 20` delivers it at 65 912 o200k. The cause is measured rather than argued -- restoring only the concept's title on the new short body ranks it 10 again. The chain ends at the orphan check (`propose.py:461`), which drops the sheet heading once a table block opens two lines below it. That is the already-reported orphan gate, and changing it is a default-ON segmentation rule affecting every document type. The specific question is unaffected: rank 1 before and after. The priced excerpt's budget share falls from 56.5 % to 9.7 %.

11 new tests (RED first), 8 mutations, 8 red, with an unmutated control green each time. One mutation survived twice before the fixture could make it fire, and both survivals are written down. 1279 -> 1287 tests. mypy --strict clean on 28 files. ruff clean. Both proposer goldens byte-unchanged. One frozen literal moved with the fix and is reported rather than hidden.

Report: docs/2026-09-08-prisform-og-loggen-k2.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 10:06:58 +02:00
commit 56ae274246
13 changed files with 1018 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

65
tests/test_corpus_run.py
View file

@ -402,30 +402,25 @@ def test_a_second_run_into_the_same_bundle_reproduces_it_byte_for_byte(tmp_path:
assert second == first
def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None:
"""A log nothing links is a file on disk, not a member of the bundle.
def test_the_root_index_does_not_link_the_bundles_own_log(tmp_path: Path) -> None:
"""The log is in the bundle and out of the index. Both halves matter.
Measured on the artifact: the K2 bundle carried a conformant root `log.md`
that no index named, so a consumer walking the bundle from `index.md` --
which is the walk section 8 exists to support -- never reached the one file
carrying `N`.
The harness DID link it (`95eb271`), so a consumer entering at `index.md`
could reach the one file carrying `N`. That was a LOCAL choice and said so:
upstream's own reference bundles do not link their logs -- measured at
`9a15b13`, 0 of the 24 shipped `index.md` files name the single `log.md` in
the bundle set -- so upstream showed the link was permitted, never required.
This is a LOCAL choice, not a conformance requirement, and the distinction
is worth keeping straight. Section 9 lets `log.md` sit at any level and
section 8 has an index enumerate its directory's contents, but upstream's
own reference bundles do not link it: measured at `9a15b13`, 0 of the 24
shipped `index.md` files name the single `log.md` in the bundle set. So
upstream proves the link is not required, not that it is disallowed.
It was removed on 2026-09-08 because the cost was measured. Consumption
contract SS 9.2 forbids a consumer from enumerating the bundle directory
unless the named profile says the index is derived, so the index tree is the
ENTIRE map a consumer may use and everything it links is a document. A
consumer walking K2 that way returned 630 documents where our own pre-pass
counts 629, and the corpus run's own log was readable and citable as
content. `5a0c879` fixed the count on our side of a disagreement produced on
this one.
It is made HERE, in the harness, because the library cannot make it. The
log's content is the run's outcome, so it cannot be written before the
indexes are projected -- and an index that enumerated `log.md` off the
directory would gain the link only on the SECOND run, breaking the
rebuild-equals-incremental property the segmented bundle is built on. The
harness instead writes the link after the log, and only when it is not
already there -- a test, not an append, because the two reprojections
disagree about this line: the per-directory one drops it as a managed
entry, the flat one keeps it because its target is not an owned concept.
Section 9 still puts `log.md` at any level, and it is still written there.
"""
root = corpus(tmp_path, {"doc.md": SEGMENTABLE, "flat.md": SUBSTANTIVE})
plans = tmp_path / "plans"
@ -450,18 +445,12 @@ def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None:
]
assert okf_corpus_run.main(argv) == 0
index = (bundle / "index.md").read_text(encoding="utf-8")
assert "](log.md)" in index
assert index.count("](log.md)") == 1
assert (bundle / "log.md").is_file(), "the log is still written to the bundle"
for index_path in bundle.rglob("index.md"):
assert "log.md" not in index_path.read_text(encoding="utf-8"), index_path
# The link is to the log in THIS directory, so a nested index must not
# carry one: there is no `log.md` beside it to reach.
for nested in bundle.rglob("*/index.md"):
assert "](log.md)" not in nested.read_text(encoding="utf-8")
# Rebuild equals incremental, still. This is what discriminates the two
# ways the append could be wrong: a link the reprojection keeps would be
# doubled here, and one the harness forgot to re-write would vanish.
# Rebuild equals incremental, still -- the property the removed append was
# most likely to break, and the one that has to survive its removal too.
first = {
path.relative_to(bundle).as_posix(): path.read_bytes()
for path in sorted(bundle.rglob("*"))
@ -476,15 +465,14 @@ def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None:
assert second == first
def test_the_log_link_holds_on_the_unsegmented_path_too(tmp_path: Path) -> None:
def test_the_unsegmented_path_leaves_the_log_out_of_the_index_too(tmp_path: Path) -> None:
"""The two run modes reproject through different code, so both are pinned.
A run without `--plans-dir` uses `STRUCTURED_V1`, whose index is not
per-directory and is rewritten by the singular reprojection rather than the
per-directory one. The harness writes `log.md` in both modes, so a link
that only held on the segmented path would leave the plainer bundle with
exactly the orphan this closes -- and if that path kept the line instead of
dropping it, the second run would carry two.
per-directory one. The removed append behaved differently under the two --
the per-directory reprojection dropped the line, the flat one kept it -- so
a removal verified on one path only would prove nothing about the other.
"""
root = corpus(tmp_path, {"a.md": SUBSTANTIVE, "b.md": SUBSTANTIVE})
bundle = tmp_path / "bundle"
@ -500,8 +488,9 @@ def test_the_log_link_holds_on_the_unsegmented_path_too(tmp_path: Path) -> None:
]
assert okf_corpus_run.main(argv) == 0
assert (bundle / "log.md").is_file()
index = (bundle / "index.md").read_text(encoding="utf-8")
assert "](log.md)" in index
assert "log.md" not in index
assert okf_corpus_run.main(argv) == 0
assert (bundle / "index.md").read_text(encoding="utf-8") == index