open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index

Browse source 
Two producer-side findings from the consumer's S7c acid test (ordre 20260908T063454Z-3648220855-from-.claude), both measured on K2 before and after, both with the corpus rebuilt from scratch.

FUNN 3 -- THE FORM. The converter's default markdown writer emits simple tables, which pad every cell out to the width of the widest cell in its column. Measured on the tender's price sheet: one 594-character prose cell produced a 67 244-character whitespace carpet with runs of up to 887 characters between a label and its amount, 19 integral amounts carrying a converter `.0`, and a header row naming one column. The bytes reached a live model in 2 of 11 prompts and 0 of 11 answers. The spreadsheet row now writes pipe tables with `--columns=1` (load-bearing: the pipe writer pads to a width computed from it, so at the default 72 a narrow table gains runs of up to 45). Same sheet after: 11 048 characters, longest run 2, one row per line, 0 artificial `.0`. Spreadsheet-only, and the scoping is pinned by three digests -- the same change moves the odt fixture 1366 -> 1105, so it can fail.

The `.0` rewrite is bounded twice: to a cell whose whole content is such a number, anchored between unescaped pipes, and skipped when the literal is in the workbook's shared string table -- the converter renders the number 92 and the TEXT "92.0" identically, so the output alone cannot tell them apart. Read with zipfile and xml.etree; no new dependency.

FUNN 2 -- THE LOG LINK. `link_log_in_root_index` (95eb271) is removed. Consumption contract SS 9.2 forbids a consumer from enumerating the bundle directory unless the profile says the index is derived, so the index tree is the entire map a consumer may use and everything it links is a document: their navigator returned 630 where our pre-pass counts 629, and a corpus run's own log was citable as content. The log is still written to the bundle root (SPEC section 9); `tools/okf_consume.py` keeps its exclusion for the bundles already built with the link.

K2 rebuilt twice. BEFORE reproduces the consumer's ref exactly (`sha256-tree:f14872a0...c8a92a`, 629 concepts) and their three consume figures to the token (57 289 / 62 149 / 58 401). AFTER: 629 concepts, `merged + coded rejections = 43 = N`, new ref `sha256-tree:c26eed6a...e3261f`, 627 of 629 concepts byte-identical, 1104 of 1108 files identical to the delivered bundle.

ONE REGRESSION, MEASURED AND NOT FIXED: on the mandate-shaped question with the vocabulary bridge the priced concept moves from candidate rank 10 to 19, so `--k 12` withholds it `below_k`; `--cost-vocabulary --k 20` delivers it at 65 912 o200k. The cause is measured rather than argued -- restoring only the concept's title on the new short body ranks it 10 again. The chain ends at the orphan check (`propose.py:461`), which drops the sheet heading once a table block opens two lines below it. That is the already-reported orphan gate, and changing it is a default-ON segmentation rule affecting every document type. The specific question is unaffected: rank 1 before and after. The priced excerpt's budget share falls from 56.5 % to 9.7 %.

11 new tests (RED first), 8 mutations, 8 red, with an unmutated control green each time. One mutation survived twice before the fixture could make it fire, and both survivals are written down. 1279 -> 1287 tests. mypy --strict clean on 28 files. ruff clean. Both proposer goldens byte-unchanged. One frozen literal moved with the fix and is reported rather than hidden.

Report: docs/2026-09-08-prisform-og-loggen-k2.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 10:06:58 +02:00
commit 56ae274246
13 changed files with 1018 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

66
CHANGELOG.md
View file

@ -17,25 +17,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
dated from `ingested_at`, so `merged + sum(coded rejections) == N` is
checkable from the bundle alone rather than only from a report that does not
travel with it. Without `--plans-dir` a run is unchanged.
- **The corpus harness links that `log.md` from the bundle's root `index.md`.**
Measured on the K2 artifact by a consumer: the log was on disk and no index
named it, so a reader entering the bundle at `index.md` -- the walk section 8
exists to support -- never reached the one file carrying `N`. A LOCAL choice
rather than conformance, and stated as one: upstream's own bundles do not
link their log (measured at `9a15b13`, 0 of 24 shipped `index.md` files name
the single `log.md` in the set), which shows the link is not required, not
that it is disallowed. It lives in the harness because the library cannot
make it -- the log's content is the run's outcome, so an index that
enumerated it off the directory would gain the link only from the second run
and break rebuild-equals-incremental. Written only when absent, because the
two reprojections differ: the per-directory one drops the line as a managed
entry, the flat one keeps it (its target is not an owned concept), and
appending unconditionally doubled it on the second unsegmented run.
Measured cause: the harness passed `STRUCTURED_V1` and no plans, so a
43-document corpus arrived as 39 flat concepts with no `adjudication` key
anywhere. Rebuilt, the same corpus yields 629 concepts, 618 of them
`adjudication: proposed`. Record: `docs/2026-09-03-k2-bundle-rebuild.md`.
- **The K2 rebuild that measured all of this.** The harness had passed
`STRUCTURED_V1` and no plans, so a 43-document corpus arrived as 39 flat
concepts with no `adjudication` key anywhere. Rebuilt, the same corpus yields
629 concepts, 618 of them `adjudication: proposed`. Record:
`docs/2026-09-03-k2-bundle-rebuild.md`.
- **`tools/okf_propose_segments.py` takes `--path-prefix`.** Section numbering
is document-local, so across 39 documents 618 proposed entries claimed only
@ -45,6 +31,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **A spreadsheet's tables are written as PIPE tables, so a row survives as a
row.** The converter's default markdown writer emits SIMPLE tables, which pad
every cell out to the width of the widest cell in its column. Measured on a
real 43-document corpus: one 594-character prose cell turned the sheet holding
the tender's prices into a 67 244-character whitespace carpet with runs of up
to **887 characters between a label and its amount**, and the header row named
a single column because only the first cell of the source's row 1 is filled.
The bytes reached a live consumer's model in two of eleven prompts and
appeared in none of its eleven answers. The same sheet through the pipe writer
is **11 048 characters with no whitespace run longer than two**, one row per
line, each source column its own cell. `--columns=1` is part of the fix and
not cosmetic: the pipe writer pads to a width computed from that setting, so
at the default a NARROW table gains runs of up to 45. Spreadsheet-only: the
other four office rows have the same defect available to the same one-line
change, but a spreadsheet is a grid with no prose fallback, while moving the
prose rows would move a corpus denominator nothing has measured.
`tests/test_extract.py` pins that scoping with three digests. Record:
`docs/2026-09-08-prisform-og-loggen-k2.md`.
- **An integral spreadsheet number loses the converter's trailing `.0`, and the
shared string table is what makes that safe.** The converter renders a numeric
cell as a double, so `5647500` arrives as `5647500.0` -- and a TEXT cell
reading `92.0` arrives as `92.0` too, which the output alone cannot tell
apart. The rewrite is bounded to a table cell whose entire content is such a
number, by unescaped pipes on both sides, and it is skipped whenever the same
literal is in the workbook's shared string table. Any workbook this cannot
read keeps its converter decimals rather than being guessed at.
- **The bundle's root `index.md` no longer links `log.md`.** The harness added
that link (`95eb271`) so a reader entering at `index.md` could reach the one
file carrying `N`; it was a LOCAL choice and said so. Consumption contract
SS 9.2 forbids a consumer from enumerating the bundle directory unless the
named profile says the index is derived, which makes the index tree the entire
map a consumer may use -- so everything it links is a document. Measured: a
consumer walking a 629-concept bundle that way returned **630**, and a corpus
run's own log became readable and citable as content. `5a0c879` excluded
`log.md` from OUR walk, which fixed the count on one side of a disagreement
produced on the other. The log is still written to the bundle root, which is
where SPEC section 9 puts it, and `tools/okf_consume.py` still excludes a
linked `log.md` -- every bundle built between `95eb271` and this change
carries the link.
- **The pinned `llm-ingestion-guard` moves to `v1.3.0`** (`[tool.uv.sources]`
and `uv.lock`; the `>=1.2,<2.0` range in `[project.dependencies]` already
covered it and is unchanged). What this fixes is that the guard could not