open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(extract,build): write a spreadsheet as pipe tables, stop linking the run log from the index

Browse source 
Two producer-side findings from the consumer's S7c acid test (ordre 20260908T063454Z-3648220855-from-.claude), both measured on K2 before and after, both with the corpus rebuilt from scratch.

FUNN 3 -- THE FORM. The converter's default markdown writer emits simple tables, which pad every cell out to the width of the widest cell in its column. Measured on the tender's price sheet: one 594-character prose cell produced a 67 244-character whitespace carpet with runs of up to 887 characters between a label and its amount, 19 integral amounts carrying a converter `.0`, and a header row naming one column. The bytes reached a live model in 2 of 11 prompts and 0 of 11 answers. The spreadsheet row now writes pipe tables with `--columns=1` (load-bearing: the pipe writer pads to a width computed from it, so at the default 72 a narrow table gains runs of up to 45). Same sheet after: 11 048 characters, longest run 2, one row per line, 0 artificial `.0`. Spreadsheet-only, and the scoping is pinned by three digests -- the same change moves the odt fixture 1366 -> 1105, so it can fail.

The `.0` rewrite is bounded twice: to a cell whose whole content is such a number, anchored between unescaped pipes, and skipped when the literal is in the workbook's shared string table -- the converter renders the number 92 and the TEXT "92.0" identically, so the output alone cannot tell them apart. Read with zipfile and xml.etree; no new dependency.

FUNN 2 -- THE LOG LINK. `link_log_in_root_index` (95eb271) is removed. Consumption contract SS 9.2 forbids a consumer from enumerating the bundle directory unless the profile says the index is derived, so the index tree is the entire map a consumer may use and everything it links is a document: their navigator returned 630 where our pre-pass counts 629, and a corpus run's own log was citable as content. The log is still written to the bundle root (SPEC section 9); `tools/okf_consume.py` keeps its exclusion for the bundles already built with the link.

K2 rebuilt twice. BEFORE reproduces the consumer's ref exactly (`sha256-tree:f14872a0...c8a92a`, 629 concepts) and their three consume figures to the token (57 289 / 62 149 / 58 401). AFTER: 629 concepts, `merged + coded rejections = 43 = N`, new ref `sha256-tree:c26eed6a...e3261f`, 627 of 629 concepts byte-identical, 1104 of 1108 files identical to the delivered bundle.

ONE REGRESSION, MEASURED AND NOT FIXED: on the mandate-shaped question with the vocabulary bridge the priced concept moves from candidate rank 10 to 19, so `--k 12` withholds it `below_k`; `--cost-vocabulary --k 20` delivers it at 65 912 o200k. The cause is measured rather than argued -- restoring only the concept's title on the new short body ranks it 10 again. The chain ends at the orphan check (`propose.py:461`), which drops the sheet heading once a table block opens two lines below it. That is the already-reported orphan gate, and changing it is a default-ON segmentation rule affecting every document type. The specific question is unaffected: rank 1 before and after. The priced excerpt's budget share falls from 56.5 % to 9.7 %.

11 new tests (RED first), 8 mutations, 8 red, with an unmutated control green each time. One mutation survived twice before the fixture could make it fire, and both survivals are written down. 1279 -> 1287 tests. mypy --strict clean on 28 files. ruff clean. Both proposer goldens byte-unchanged. One frozen literal moved with the fix and is reported rather than hidden.

Report: docs/2026-09-08-prisform-og-loggen-k2.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 10:06:58 +02:00
commit 56ae274246
13 changed files with 1018 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

66
CHANGELOG.md
View file

@ -17,25 +17,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
dated from `ingested_at`, so `merged + sum(coded rejections) == N` is dated from `ingested_at`, so `merged + sum(coded rejections) == N` is
checkable from the bundle alone rather than only from a report that does not checkable from the bundle alone rather than only from a report that does not
travel with it. Without `--plans-dir` a run is unchanged. travel with it. Without `--plans-dir` a run is unchanged.
- **The corpus harness links that `log.md` from the bundle's root `index.md`.** - **The K2 rebuild that measured all of this.** The harness had passed
Measured on the K2 artifact by a consumer: the log was on disk and no index `STRUCTURED_V1` and no plans, so a 43-document corpus arrived as 39 flat
named it, so a reader entering the bundle at `index.md` -- the walk section 8 concepts with no `adjudication` key anywhere. Rebuilt, the same corpus yields
exists to support -- never reached the one file carrying `N`. A LOCAL choice 629 concepts, 618 of them `adjudication: proposed`. Record:
rather than conformance, and stated as one: upstream's own bundles do not `docs/2026-09-03-k2-bundle-rebuild.md`.
link their log (measured at `9a15b13`, 0 of 24 shipped `index.md` files name
the single `log.md` in the set), which shows the link is not required, not
that it is disallowed. It lives in the harness because the library cannot
make it -- the log's content is the run's outcome, so an index that
enumerated it off the directory would gain the link only from the second run
and break rebuild-equals-incremental. Written only when absent, because the
two reprojections differ: the per-directory one drops the line as a managed
entry, the flat one keeps it (its target is not an owned concept), and
appending unconditionally doubled it on the second unsegmented run.
Measured cause: the harness passed `STRUCTURED_V1` and no plans, so a
43-document corpus arrived as 39 flat concepts with no `adjudication` key
anywhere. Rebuilt, the same corpus yields 629 concepts, 618 of them
`adjudication: proposed`. Record: `docs/2026-09-03-k2-bundle-rebuild.md`.
- **`tools/okf_propose_segments.py` takes `--path-prefix`.** Section numbering - **`tools/okf_propose_segments.py` takes `--path-prefix`.** Section numbering
is document-local, so across 39 documents 618 proposed entries claimed only is document-local, so across 39 documents 618 proposed entries claimed only
@ -45,6 +31,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed ### Changed
- **A spreadsheet's tables are written as PIPE tables, so a row survives as a
row.** The converter's default markdown writer emits SIMPLE tables, which pad
every cell out to the width of the widest cell in its column. Measured on a
real 43-document corpus: one 594-character prose cell turned the sheet holding
the tender's prices into a 67 244-character whitespace carpet with runs of up
to **887 characters between a label and its amount**, and the header row named
a single column because only the first cell of the source's row 1 is filled.
The bytes reached a live consumer's model in two of eleven prompts and
appeared in none of its eleven answers. The same sheet through the pipe writer
is **11 048 characters with no whitespace run longer than two**, one row per
line, each source column its own cell. `--columns=1` is part of the fix and
not cosmetic: the pipe writer pads to a width computed from that setting, so
at the default a NARROW table gains runs of up to 45. Spreadsheet-only: the
other four office rows have the same defect available to the same one-line
change, but a spreadsheet is a grid with no prose fallback, while moving the
prose rows would move a corpus denominator nothing has measured.
`tests/test_extract.py` pins that scoping with three digests. Record:
`docs/2026-09-08-prisform-og-loggen-k2.md`.
- **An integral spreadsheet number loses the converter's trailing `.0`, and the
shared string table is what makes that safe.** The converter renders a numeric
cell as a double, so `5647500` arrives as `5647500.0` -- and a TEXT cell
reading `92.0` arrives as `92.0` too, which the output alone cannot tell
apart. The rewrite is bounded to a table cell whose entire content is such a
number, by unescaped pipes on both sides, and it is skipped whenever the same
literal is in the workbook's shared string table. Any workbook this cannot
read keeps its converter decimals rather than being guessed at.
- **The bundle's root `index.md` no longer links `log.md`.** The harness added
that link (`95eb271`) so a reader entering at `index.md` could reach the one
file carrying `N`; it was a LOCAL choice and said so. Consumption contract
SS 9.2 forbids a consumer from enumerating the bundle directory unless the
named profile says the index is derived, which makes the index tree the entire
map a consumer may use -- so everything it links is a document. Measured: a
consumer walking a 629-concept bundle that way returned **630**, and a corpus
run's own log became readable and citable as content. `5a0c879` excluded
`log.md` from OUR walk, which fixed the count on one side of a disagreement
produced on the other. The log is still written to the bundle root, which is
where SPEC section 9 puts it, and `tools/okf_consume.py` still excludes a
linked `log.md` -- every bundle built between `95eb271` and this change
carries the link.
- **The pinned `llm-ingestion-guard` moves to `v1.3.0`** (`[tool.uv.sources]` - **The pinned `llm-ingestion-guard` moves to `v1.3.0`** (`[tool.uv.sources]`
and `uv.lock`; the `>=1.2,<2.0` range in `[project.dependencies]` already and `uv.lock`; the `>=1.2,<2.0` range in `[project.dependencies]` already
covered it and is unchanged). What this fixes is that the guard could not covered it and is unchanged). What this fixes is that the guard could not

22
README.md
View file

@ -80,7 +80,7 @@ than the clock, so two builds of the same folder are byte-identical — a
wall-clock default would break rebuild-equals-incremental for every caller who wall-clock default would break rebuild-equals-incremental for every caller who
did not pass them. did not pass them.
Measured 2026-09-07 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and Measured 2026-09-08 on a 43-file corpus (33 `pdf`, 5 `docx`, 2 `xlsx`, and
three files no reader accepts), one three files no reader accepts), one
`okf build` invocation replacing the shell loop over `tools/` that produced the `okf build` invocation replacing the shell loop over `tools/` that produced the
same corpus's bundle on 2026-09-03: same corpus's bundle on 2026-09-03:
@ -92,15 +92,19 @@ same corpus's bundle on 2026-09-03:
| coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) | | coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) |
| K1b | `39 + 4 = 43 = N`, exit `0` | | K1b | `39 + 4 = 43 = N`, exit `0` |
| files written | 1108 | | files written | 1108 |
| identical to the 2026-09-03 bundle | 1107/1108 | | identical to the 2026-09-03 bundle | 1104/1108 |
| wall time | 779.43 s total, 18.126 s per file | | wall time | 842.82 s total, 19.600 s per file (re-measured 2026-09-08) |
The one file that differs is the root `index.md`, by exactly one line: the link The six files that differ are all in the corpus's two spreadsheet documents, and
to the bundle's own `log.md`, added by a commit that postdates the stored they are the change reported in `docs/2026-09-08-prisform-og-loggen-k2.md`: a
artifact by fifteen hours. Appending that line to the stored `index.md` spreadsheet's tables are now written as pipe tables, so each row is one line
reproduces the new one byte for byte. The command's own byte-identity test with its cells delimited rather than padded out to the widest cell in the
compares it against the two scripts at the current commit, where the two agree column. Two concept files are renamed by it, two are removed under their old
over the whole tree. names, and the two documents' own `index.md` follow. The root `index.md` is
identical to the stored one again, because this library no longer links the
bundle's `log.md` from it. The command's own byte-identity test compares it
against the two scripts at the current commit, where the two agree over the
whole tree.
## Consume ## Consume

473
docs/2026-09-08-prisform-og-loggen-k2.md Normal file
View file

@ -0,0 +1,473 @@
# The delivered price sheet's FORM, and the log the index linked
**Date:** 2026-09-08 · **Order:** `20260908T063454Z-3648220855-from-.claude` ·
**Instrument:** `src/llm_ingestion_okf/extract.py` at `6776c37` plus the two
changes this document reports · **Upstream finding:**
`portfolio-optimiser docs/2026-09-08-syretest-s7c-begge-laaser-k2.md`
(`648b36e`) SS 5 and SS 9, findings 2 and 3.
The consumer that reads bundles from this library ran a live model over a
629-concept bundle with both of the pre-pass's optional widenings open. The
document holding the tender's prices WAS delivered, at rank 10, and its bytes
reached two of eleven prompts. It appears in **zero of the eleven answers**.
They measured why rather than guessing, and the answer is on this side of the
boundary: the delivered excerpt is a pandoc SIMPLE table with **one column
header**, 67 245 characters over 104 lines, with whitespace runs of up to **887
characters between a label and its amount**. The same form makes their own
cost-baseline deriver refuse the sheet. Opening both locks bought the BYTES and
not the STRUCTURE.
Their second finding is the same class of defect at the bundle level: this
library writes a link to the run's own `log.md` into the root `index.md`, so a
consumer's navigator reaches **630** documents where our own pre-pass counts
629, and a corpus run's log is readable and citable as content.
Both are producer-side. This document measures both, fixes both, and states
what it did not fix.
The corpus is external and private to the measurement; no document name, path or
body from it appears here. Documents are named by shape ("the priced sheet") and
the numbers are counts.
---
## 0. What IS measured, and what is NOT
**Measured.** The form of both spreadsheet documents in the corpus before and
after the change, each figure with the command that produced it; the line that
produces the form, read out of the code rather than inferred; the fix on a
hand-laid synthetic sheet and on the corpus's OTHER sheet; a full corpus rebuild
before and after with its diff, its new content identity and its concept count;
the pre-pass's delivered rank and token cost for the priced sheet after the
change; and the index/concept disagreement, on a build rather than on an
argument.
**Known-positive, run first.** The upstream report's four form figures,
re-measured on this tree before anything changed:
| # | figure | upstream | measured here |
|---|---|---|---|
| 1 | delivered excerpt, characters | 67 245 | **67 244 + the trailing LF** |
| 2 | delivered excerpt, lines | 104 | **103 + the trailing LF** |
| 3 | longest whitespace run | 887 | **887** |
| 4 | numbers >= 1000 in `.0` form | 19 | **19** |
| 5 | non-empty header cells in the table's header row | 1 | **1** |
Figures 1 and 2 differ by exactly the trailing newline, which is stated rather
than rounded away: the file's last line ends with an LF, so a count that
includes it gives 67 245 and 104. Figures 3, 4 and 5 are exact. An instrument
that has not reproduced a known figure has not been shown to count.
**One thing this change makes WORSE, measured and reported rather than
discovered later.** On the mandate-shaped question with the vocabulary bridge
open, the priced concept moves from candidate rank **10 to rank 19**, so the
command the consumer used (`--k 12`) no longer reaches it. The cause is traced
in SS 7 and it is not the new form: it is the sheet's heading being dropped by a
segmentation rule the new form wakes up, which costs the concept its title. The
question this library's own hit@k measurement uses is unaffected -- the priced
document still ranks first there.
**NOT measured, and it is the finding that matters.** Whether a live model now
READS the sheet. That needs a live model and it is the consumer's measurement,
in their S7 form, not this one. What is measured here is the FORM: the label and
the amount are in separate cells on one line instead of separated by hundreds of
spaces. Whether that is enough is not this repository's to answer, and saying so
is the same limit this library stated when it shipped the two flags.
**NOT measured either:** that the change helps any corpus other than this one.
Two real sheets and one synthetic sheet are not a sample.
**One thing is deliberately NOT fixed.** The header row still names one column,
because the SOURCE's first row has one filled cell. Promoting the next row to a
header would be inventing structure the document does not have, which is the one
thing this library refuses to do without marking it (`CLAUDE.md`: an unmarked
heuristic is worse than none). If the consumer's deriver requires two or more
column HEADERS, this change alone does not satisfy it -- and that is their
measurement to make, on the rebuilt bundle named below.
---
## 1. Setup
The corpus is the 43-document folder the K2 measurements have used since
2026-08-29. Both bundles below are `okf build` runs over it with
`--ingested-at 2026-09-03T00:00:00Z --bundle-id <the delivered id>
--okf-version 0.2`.
**The BEFORE build runs committed `6776c37` source, not the working tree.** The
package is installed editable, so a build started from this clone would pick up
every edit made while it ran -- which it did once, and the run was discarded.
The before build therefore runs from `git archive 6776c37` unpacked to `/tmp`
with `PYTHONPATH` pointing at it, and the first line of its log names that path.
Every command in this document is offline: no model call, no socket, no clock.
---
## 2. The form, before anything changed
Both figures below are taken from the extracted text with each line's trailing
whitespace removed, which is what the upstream report counted.
| sheet | characters | lines | longest run | numbers >= 1000 in `.0` form | non-empty header cells |
|---|---|---|---|---|---|
| priced, BEFORE | 67 244 | 103 | **887** | **19** | 1 |
| other, BEFORE | 1 842 | 42 | 18 | 0 | 2 |
The instrument the order names measures the same thing from the table grammar's
side, over a two-document corpus holding only the two sheets:
```
okf_table_measure.py --corpus <the two sheets> --report <out>
```
| | BEFORE | AFTER |
|---|---|---|
| documents with at least one table row | **0**/2 | **2**/2 |
| table rows in total | **0** | **141** |
| documents with at least one grid-rule line | 0/2 | 0/2 |
Zero table rows out of two spreadsheets is the whole finding in one number: the
shipped table grammar, which is the grammar the segment proposer uses, could not
see a single row in either sheet. Not because the sheets have no rows -- they
have 141 -- but because the form they were written in has none it can match.
---
## 3. The line that produces it
Read out of the code, not inferred. At `6776c37`,
`src/llm_ingestion_okf/extract.py` lines 85-86:
```python
_PANDOC_WRITER = "markdown"
_PANDOC_ARGS = ("--eol=lf", "--wrap=none")
```
`markdown` with no extension changes is the writer's DEFAULT table selection,
which prefers a simple table and falls back to multiline and grid. Every one of
those three pads each cell out to the width of the widest cell in its column.
That is the whole mechanism: the sheet's first column holds a 594-character
prose cell, so every other row in that column is padded to 594 characters.
It is neither the reader nor the source: the header underline of the simple
table has EIGHT dash groups, so the converter recovered eight columns from the
sheet and the writer then serialised them in a form where the separation is
positional whitespace. Measured with the same three writers on the same file:
| writer | characters | longest run |
|---|---|---|
| `markdown` (default tables) | 100 795 | 996 |
| `markdown` minus simple/multiline/grid tables | 11 221 | 2 |
| the same, with `--columns=1` | 11 221 | 2 |
(The figures in this table are the raw converter output including trailing
padding, which is why they are larger than SS 2's.)
`--columns=1` costs nothing on this sheet and is load-bearing on a narrow one:
the pipe writer pads cells out to a width computed from that setting, so at the
default 72 a small table gains runs of up to 45 characters. Measured across
every office fixture and every office file in the corpus, the longest run with
`--columns=1` is **2**.
---
## 4. The fix, in full
Two changes in `src/llm_ingestion_okf/extract.py`, both spreadsheet-only.
**(a) The spreadsheet row writes pipe tables** (lines 113-114):
```python
_SPREADSHEET_WRITER = "markdown-simple_tables-multiline_tables-grid_tables"
_SPREADSHEET_ARGS = (*_PANDOC_ARGS, "--columns=1")
```
selected at the one call site by `spreadsheet = suffix == ".xlsx"`.
**(b) An integral cell loses the converter's trailing `.0`** (line 122 and the
two functions below it). The converter renders a numeric cell as a double, so an
integral value arrives as `5647500.0`. A TEXT cell reading `92.0` arrives as
`92.0` as well, and the output alone cannot tell the two apart -- so rewriting
on the output alone would silently edit somebody's authored text. The rewrite is
therefore bounded twice:
- to a table cell whose ENTIRE content is such a number, anchored between two
pipes that are not backslash-escaped, so a `5.0` inside a longer cell is not a
cell and is left alone;
- and skipped whenever the same literal is present in the workbook's shared
string table, which is read with `zipfile` and `xml.etree` and no new
dependency. Shared strings are the only text the converter recovers from a
sheet at all -- an inline string is read as an empty cell, measured when the
first spreadsheet fixture was built -- so a `<digits>.0` that is not in that
set did not come from text.
Every failure to read the workbook returns the empty set, which makes the
rewrite a no-op rather than a guess.
**Nothing else moved.** No new dependency, no new flag, no model call, and no
change to any other file type. One frozen literal in the suite moved with the
fix and is reported in SS 5 rather than hidden.
---
## 5. The tests, the controls and the mutations
Written RED first, in this order.
| test | what it pins |
|---|---|
| `test_a_spreadsheet_keeps_its_columns_one_row_per_line` | label and amount are separate cells on one line; no whitespace run over 8 |
| `test_an_integral_amount_loses_the_converters_decimal_and_a_real_one_keeps_it` | `5647500` and `250000` lose `.0`; the genuine `12.5` keeps its decimal; the shared-string `92.0` survives verbatim; a `5.0` inside an escaped-pipe cell survives |
| `test_a_single_column_sheet_gains_no_columns` | a one-column source sheet gets one column and its three values, in order |
| `test_the_other_office_rows_are_untouched_by_the_spreadsheet_writer` (x3) | the docx/pptx/odt/rtf path is byte-identical, by digest |
| `test_prisark_extracts_to_its_frozen_text` | the whole fixture, byte for byte |
| `test_the_root_index_does_not_link_the_run_log` | the log is in the bundle and out of every index; the pre-pass's count equals the concept count |
The fixture is `tests/fixtures/prisark.xlsx`, hand-laid part by part in
`tests/fixtures/make_fixtures.py` under the same policy as every other office
fixture in this repository: a document written by the converter and then read by
the converter proves only that the converter agrees with itself. It carries the
defect in miniature -- a single filled cell in row 1, a long prose cell in column
B, two integral numbers, one genuine decimal, one text cell reading `92.0` and
one cell containing a literal pipe -- and its own negative control on a second
sheet.
**The negative control the order asked for, and where it does not hold.** The
order asked that a sheet with no column structure be byte-identical before and
after. **It is not, and that is measured rather than glossed:** the fix is the
table FORM, and a one-column table changes delimiter form with every other
table. Measured on the fixture's second sheet, the cell content and the column
count are unchanged and only the delimiters moved. Two negative controls that DO
hold byte for byte are shipped in its place, and neither is a tautology:
- the three other office rows, by digest -- the same one-line change applied to
them was measured to move them (the odt fixture 1366 -> 1105 characters), so
the digests can fail;
- the docx frozen literal, green under every mutation below.
**Mutations: 8 written, 8 red, with an unmutated control green each time.**
| # | mutation | result |
|---|---|---|
| 1 | writer reverted to the default (simple tables) | 5 red |
| 2 | `--columns=1` dropped | 3 red |
| 3 | the spreadsheet writer applied to every office row | 4 red, incl. all three digests |
| 4 | the decimal rewrite never runs | 3 red |
| 5 | the shared string table ignored | 2 red |
| 6 | the guard looks up the wrong literal | 2 red |
| 7 | an escaped pipe counts as a cell delimiter | 2 red |
| 8 | the log link written back into the root index | 2 red |
Mutation 7 SURVIVED its first form. The first attempt added `re.M` to a pattern
containing no `^` and no `$`, which is an equivalent mutation and says nothing
about the tests -- the trap this repository has hit before, and the reason a
surviving mutation is read as code before it is called a test hole. The second
attempt, which actually removes the escaped-pipe guard, survived too: the
fixture's pipe-bearing cell ended in prose rather than in the number, so the
pattern could not match either way. The fixture was sharpened until the mutation
could fire, and then it fired.
---
## 6. Generality
The order required the fix be measured beyond the one sheet it was written for.
| sheet | characters | lines | longest run | numbers >= 1000 in `.0` |
|---|---|---|---|---|
| priced, BEFORE | 67 244 | 103 | 887 | 19 |
| priced, AFTER | **11 048** | 103 | **2** | **0** |
| other, BEFORE | 1 842 | 42 | 18 | 0 |
| other, AFTER | **1 921** | 42 | **2** | 0 |
| synthetic, BEFORE | 950 | 16 | 130 | n/a (2 integral, 2 text) |
| synthetic, AFTER | **402** | 16 | **2** | 2 integral rewritten, 2 text kept |
The other sheet is the honest row here: it gets **larger** by 79 characters,
because pipe delimiters cost more than the padding they replaced on a table
whose columns were already narrow. Its longest run still falls from 18 to 2. The
fix is not a compression; it is a change of form that happens to compress a
sheet with one very wide column by 84 %.
---
## 7. The rebuild: 629 concepts, a new identity, and one regression
Both builds are `okf build` over the same 43-document folder with the same
arguments; the before build runs committed `6776c37` source from `/tmp`.
**The known-positive at bundle level, before anything is read from the after
build.** The before build reproduces the identity the consumer measured:
| | before | after |
|---|---|---|
| `merged + coded rejections == N` | `39 + 4 = 43 = N`, exit 0 | `39 + 4 = 43 = N`, exit 0 |
| concepts | **629** | **629** |
| content identity | `sha256-tree:f14872a0…c8a92a` | **`sha256-tree:c26eed6a…e3261f`** |
| wall time | 831.08 s | 842.82 s |
`f14872a0…` is the consumer's ref for the bundle they measured, character for
character. **`c26eed6a…` supersedes it: their ref is now stale**, and every
figure keyed to it belongs to the old form.
**Seven files differ between the two bundles, and they are all in the two
spreadsheet documents.** 627 of 629 concepts are byte-identical.
| what | before | after |
|---|---|---|
| the priced document's concept | `…/prissammenstilling-sheet-1.md` | `…/tabell-linje-3.md` |
| the other sheet's concept | `…/ark1-sheet-1.md` | `…/tabell-linje-3.md` |
| the two documents' `index.md` | link and title of the above | idem |
| the bundle's root `index.md` | carried the `log.md` link | does not |
Measured against the bundle delivered on 2026-09-03, **1104 of 1108 files are
identical**, against 1107 before: the root `index.md` is identical again (SS 8),
and the two spreadsheet documents account for every remaining difference.
**The form, measured on the rebuilt concept:** longest whitespace run **887 ->
2**, numbers >= 1000 in `.0` form **19 -> 0**, body **100 694 -> 11 048**
characters, one row per line. And its excerpt weight, which is what the budget
gate counts: **67 838 B -> 11 594 B**, from **56.5 % of the default 120 000-byte
budget to 9.7 %**. The second lock the previous document measured is, for this
document, gone.
### The regression: rank 10 -> 19, and its cause traced rather than guessed
The order asked whether `--cost-vocabulary --k 12 --limit 160000` still delivers
the priced sheet. **It does not.**
| question | vocabulary | before | after |
|---|---|---|---|
| mandate-shaped | off | candidate rank 251 of 271 | **251 of 271** |
| mandate-shaped | on | candidate rank **10** of 280 | **19** of 280 |
| specific | off | rank **1** of 233 | **1** of 233 |
| specific | on | rank **1** of 248 | **1** of 248 |
`--k 12` reaches rank 12, so at rank 19 the concept is withheld `below_k`. The
question this library's own hit@k measurement uses is unaffected: the priced
document still ranks **first** of 233 candidates.
**The cause is the title, not the body, and that is measured rather than
argued.** A copy of the AFTER bundle with the concept's file name and `title`
restored to the before values -- the new short pipe-table body untouched --
ranks the concept **10** again, exactly where it was.
The chain, each link read out of the code:
1. the extraction now emits `|` rows, so `rule:table-block` opens a candidate on
the sheet's first table row (`propose.py`, `RULE_TABLE_BLOCK`);
2. that candidate starts two lines after the sheet heading, so the heading
candidate's body is its own line plus a blank one;
3. the ORPHAN CHECK (`propose.py:461`) drops a candidate whose body is empty
after its first line, so the heading candidate is dropped;
4. the only surviving candidate is the table block, whose title is the
mechanical `Tabell linje <n>`;
5. the concept id and title therefore lose the sheet's name, and the cost
vocabulary bridge has nothing left to match in signal 1.
**This is the orphan gate that was already reported and not fixed** (it deletes
34 % of candidates unevenly, measured earlier and open on the operator's list).
This is a second measured consequence of it, not a new defect, and fixing it is
a change to a default-ON segmentation rule affecting every document type -- which
is not in this order and is not decided here.
**The commands, measured on the rebuilt bundle:**
| command | payload | o200k | priced sheet |
|---|---|---|---|
| flagless default | 169 573 B | 57 286 | `below_k` |
| `--cost-vocabulary --k 12 --limit 160000` | 172 588 B | 58 553 | **`below_k`** |
| `--cost-vocabulary --k 20` (default budget) | 190 428 B | 65 912 | **delivered, rank 19** |
Direction, stated as the order asked: the payload for the command that delivers
the sheet is **larger** in tokens than before (65 912 against 62 149, +6.1 %),
because a bigger `k` delivers eight more excerpts; the SHEET's own contribution
fell by 83 %. The default payload is 3 tokens cheaper and otherwise unmoved.
**The three known-positive consume figures reproduce exactly on the before
build**, which is what makes the after column readable at all: 57 289 o200k for
the flagless default, 62 149 for `--cost-vocabulary --k 12 --limit 160000`, and
58 401 for the same at the default budget -- the consumer's three figures, to
the token.
---
## 8. The log the index linked
The upstream finding, verbatim in substance: our pre-pass excludes `log.md` from
its walk (`considered` = 629), but the link is still written into the root
`index.md`, so their navigator follows it and returns **630**. The corpus run's
own log is a document their agents can read and cite. The bundle delivered on
2026-09-03 has no such link, because the link postdates it.
**The link was a LOCAL choice and said so at the time** (`95eb271`). It was made
on a real measurement -- the bundle carried a conformant root `log.md` that no
index named, so a reader entering at `index.md` never reached the one file
carrying `N`. Upstream's own bundles do not link theirs: measured at `9a15b13`,
0 of the 24 shipped `index.md` files name the single `log.md` in the set. So
upstream showed the link was permitted, never that it was required.
**Why it goes, and the section that decides it.** Consumption contract SS 9.2:
> A consumer MUST NOT enumerate a directory unless the named profile says the
> index is derived.
Measured, `entries_match_directory` is true for `STRICT_V1` alone, and the
profile these bundles use is not it. So a consumer of one of our bundles has
exactly one lawful way to find out what is in it: walk the index tree. That
makes the index the ENTIRE map, and everything the map names a document. A
producer that links a non-concept from the index is not adding a convenience --
it is adding a document, to the only enumeration the consumer is allowed to
perform, and the consumer has no lawful second opinion to check it against.
SS 8's payload shape is the other half. It requires `denominators.considered`
beside an excerpt list keyed on `concept_id`; a bundle whose index reaches one
more document than it has concepts makes those two numbers answer different
questions while looking like one. That is exactly the shape the consumer
measured: 630 against 629.
**The smallest change that makes producer and consumer agree** is therefore not
another exclusion rule on the consumer's side -- there is no end to that list --
but for the producer to stop naming a non-concept in the map. `log.md` is still
written to the bundle root, which is where SPEC section 9 puts it and all the
earlier fix ever needed; it is no longer linked from any index.
`tools/okf_consume.py` KEEPS its own exclusion, and the test that pins it now
writes the link itself. That is not dead code: every bundle built between
`95eb271` and this change carries the link, including the ones consumers are
reading today, and the pre-pass must still count 629 on those.
The two tests that pinned the old behaviour were rewritten rather than deleted,
including the halves that mattered independently -- rebuild-equals-incremental
on both the segmented and the unsegmented path, which is the property the
removed append was most likely to break, and the one that has to survive its
removal too.
---
## 9. Honesty limits
- **The form is measured; the reading is not.** Whether a live model uses the
sheet now is the consumer's measurement, in their S7 form. This repository
cannot make it and does not claim it.
- **The header row still names one column.** The source's row 1 has one filled
cell and this change does not invent a second. A deriver that requires two or
more column HEADERS is not satisfied by this alone.
- **Two real sheets and one synthetic sheet are not a sample.** The corpus holds
exactly two spreadsheets; both are measured, and that is the whole denominator
available, not a selection from a larger one.
- **The other four office rows have the same defect.** It is measured (the odt
fixture 1366 -> 1105 characters) and deliberately not fixed here, because
changing them moves a corpus denominator nothing has measured. It is a
decision for the operator, not an oversight.
- **The order's literal negative control does not hold** (SS 5): a one-column
sheet is not byte-identical, because the fix is the table form. What holds is
that no column is invented and no cell content changes.
- **The priced sheet is a PRE-AWARD form and is not filled in** (the consumer's
own limit, repeated here): no real saving was ever available to find in it,
whatever its form.
- **`.0` on a number the workbook stores as text** is preserved, and the guard
is conservative in the other direction too: a workbook whose shared string
table happens to contain the same literal keeps its converter decimals on
every numeric cell carrying that value.

3
src/llm_ingestion_okf/cli.py
View file

@ -52,7 +52,7 @@ import sys
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from .corpus import LOG_NAME, CorpusReport, link_log_in_root_index, load_plans, measure from .corpus import LOG_NAME, CorpusReport, load_plans, measure
from .errors import IngestError from .errors import IngestError
from .inbox import walk_inbox from .inbox import walk_inbox
from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile from .profiles import SEGMENTED_OKF_V0_2, STRUCTURED_V1, BundleProfile
@ -187,7 +187,6 @@ def _write_log(bundle: Path, report: CorpusReport, *, profile: BundleProfile) ->
""" """
bundle.mkdir(parents=True, exist_ok=True) bundle.mkdir(parents=True, exist_ok=True)
(bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="") (bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="")
link_log_in_root_index(bundle, profile)
def parse_args(argv: list[str] | None) -> argparse.Namespace: def parse_args(argv: list[str] | None) -> argparse.Namespace:

60
src/llm_ingestion_okf/corpus.py
View file

@ -376,45 +376,26 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
return parser.parse_args(argv) return parser.parse_args(argv)
def link_log_in_root_index(bundle: Path, profile: BundleProfile) -> None: # `link_log_in_root_index` LIVED HERE AND WAS REMOVED (2026-09-08).
"""Point the root index at the log, so the walk section 8 supports reaches it. #
# It appended `- [Corpus run history](log.md)` to the root index (`95eb271`) so
Measured on the K2 artifact: the bundle carried a conformant root `log.md` # a reader entering at `index.md` could reach the one file carrying `N`. That
that no index named, so a consumer entering at `index.md` never reached the # was a LOCAL choice, said so at the time, and upstream never linked its own
one file carrying `N`. # logs -- measured at `9a15b13`, 0 of 24 shipped `index.md` files name the one
# `log.md` in the set.
A LOCAL choice, not a conformance requirement, and the difference is worth #
stating rather than implying. Section 9 puts `log.md` at any level and # The cost was measured on K2 by the first consumer to walk a bundle of ours
section 8 has an index enumerate its directory's contents, but upstream's # with a live model: consumption contract SS 9.2 forbids a consumer from
own bundles do not link it: measured at `9a15b13`, 0 of the 24 shipped # enumerating the bundle directory unless the profile says the index is
`index.md` files name the single `log.md` in the set. Upstream therefore # derived, so the index tree IS the entire map a consumer may use, and anything
shows the link is not REQUIRED -- not that it is disallowed. # it links is a document. Their navigator returned 630 where our own pre-pass
# counts 629, and a corpus run's own log became readable and citable as
It belongs to the harness and not the library. The log's content IS the # content. `5a0c879` (F2) excluded `log.md` from OUR walk, which fixed the
run's outcome, so it cannot exist when the indexes are projected; an index # count on one side of a disagreement produced on the other.
that enumerated it off the directory would gain the link only from the #
second run onward and break rebuild-equals-incremental, the property the # The log itself is still written to the bundle root, which is where SS 9 puts
segmented bundle is built on. Writing it after the log instead keeps both # it and all F2 ever needed. Reported in
runs identical. # `docs/2026-09-08-prisform-og-loggen-k2.md`.
THE MEMBERSHIP TEST IS LOAD-BEARING, and measured rather than assumed: the
two reprojections do not treat this line the same way. The per-directory
one drops every managed line before re-emitting its block, so the link is
gone by the time this runs. The flat one keeps a managed line whose target
is not an owned concept -- deliberately, because claiming somebody else's
link on the strength of a regex would delete curated content -- so `log.md`
survives there. Appending unconditionally therefore doubled the entry on
the second unsegmented run. Re-writing the line only when it is absent is
idempotent under both, without either side having to know about the other.
"""
index_path = bundle / profile.index.name
if not index_path.is_file():
return
body = index_path.read_text(encoding="utf-8")
link = profile.index.render_link(LOG_TITLE, LOG_NAME) + "\n"
if link in body.splitlines(keepends=True):
return
index_path.write_text(body + link, encoding="utf-8", newline="")
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
@ -471,7 +452,6 @@ def main(argv: list[str] | None = None) -> int:
# as it was before. # as it was before.
bundle.mkdir(parents=True, exist_ok=True) bundle.mkdir(parents=True, exist_ok=True)
(bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="") (bundle / LOG_NAME).write_text(report.render_log(), encoding="utf-8", newline="")
link_log_in_root_index(bundle, profile)
print(report.render()) print(report.render())
if report.unaccounted or report.merged + report.rejected != report.n: if report.unaccounted or report.merged + report.rejected != report.n:
print( print(

91
src/llm_ingestion_okf/extract.py
View file

@ -26,8 +26,11 @@ from __future__ import annotations
import csv import csv
import functools import functools
import io import io
import re
import tempfile import tempfile
import warnings import warnings
import zipfile
from xml.etree import ElementTree
from collections.abc import Callable, Sequence from collections.abc import Callable, Sequence
from html.parser import HTMLParser from html.parser import HTMLParser
from pathlib import Path from pathlib import Path
@ -85,6 +88,39 @@ _EVIDENCE: dict[str, str] = {
_PANDOC_WRITER = "markdown" _PANDOC_WRITER = "markdown"
_PANDOC_ARGS = ("--eol=lf", "--wrap=none") _PANDOC_ARGS = ("--eol=lf", "--wrap=none")
# The spreadsheet row writes PIPE tables, and it is the only row that does.
#
# The default writer prefers simple tables, which pad every cell out to the
# width of the widest cell in its column. Measured on the K2 price sheet: one
# 594-character prose cell turned every other row in that column into a run of
# up to 887 spaces between a label and its amount, 100 795 characters in all,
# and the header row named ONE column because only the first cell of the source
# row 1 is filled. The bytes reached the reader and the structure did not. The
# same sheet through this writer is 11 221 characters with no whitespace run
# longer than two, one row per line, each source column its own cell.
#
# `--columns=1` is load-bearing rather than cosmetic: the pipe writer pads cells
# out to the column width it computes from that setting, so at the default 72 a
# NARROW table gains runs of up to 45 spaces -- the same defect at a smaller
# scale. Measured across every office fixture and every K2 office file, the
# longest whitespace run with it is 2.
#
# SPREADSHEET-ONLY, deliberately. The other four rows have the same defect
# available to the same one-line fix (measured: the odt fixture 1366 -> 1105
# characters), but a spreadsheet IS a grid with no prose fallback, while moving
# the prose rows would move a corpus denominator that nothing has measured.
# `tests/test_extract.py` pins that scoping with three digests.
_SPREADSHEET_WRITER = "markdown-simple_tables-multiline_tables-grid_tables"
_SPREADSHEET_ARGS = (*_PANDOC_ARGS, "--columns=1")
# SpreadsheetML's namespace, needed to read the workbook's shared string table.
_SSML = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
# A table cell whose whole content is an integer with the converter's trailing
# `.0`. Bounded by unescaped pipes on both sides so a cell containing an escaped
# `\|` can never be split in the middle.
_INTEGRAL_CELL = re.compile(r"(?<!\\)\|(\s*)(-?\d+)\.0(\s*)(?=(?<!\\)\|)")
# Conversion recovers text, on the same terms as PDF extraction: a drawing has # Conversion recovers text, on the same terms as PDF extraction: a drawing has
# no text to recover. Said out loud on every conversion rather than detected # no text to recover. Said out loud on every conversion rather than detected
# per document, for the same reason. # per document, for the same reason.
@ -290,8 +326,11 @@ def _extract_office(suffix: str, data: bytes) -> str:
except ImportError as exc: except ImportError as exc:
raise _extra_missing(suffix) from exc raise _extra_missing(suffix) from exc
spreadsheet = suffix == ".xlsx"
writer = _SPREADSHEET_WRITER if spreadsheet else _PANDOC_WRITER
args = _SPREADSHEET_ARGS if spreadsheet else _PANDOC_ARGS
try: try:
text = _convert_bytes(data, _PANDOC_WRITER, _PANDOC_FORMATS[suffix], _PANDOC_ARGS) text = _convert_bytes(data, writer, _PANDOC_FORMATS[suffix], args)
except ExtractionError: except ExtractionError:
raise raise
except Exception as exc: # noqa: BLE001 - third-party converter, wrapped never leaked except Exception as exc: # noqa: BLE001 - third-party converter, wrapped never leaked
@ -308,12 +347,62 @@ def _extract_office(suffix: str, data: bytes) -> str:
code="extractor_empty_conversion", code="extractor_empty_conversion",
) )
if spreadsheet:
text = _drop_converter_decimals(text, data)
# After the parse, not before: a run that produced no text has nothing to # After the parse, not before: a run that produced no text has nothing to
# be lossy about, and warning there would just add noise to a failure. # be lossy about, and warning there would just add noise to a failure.
warnings.warn(_OFFICE_LOSSY_WARNING, ExtractionWarning, stacklevel=3) warnings.warn(_OFFICE_LOSSY_WARNING, ExtractionWarning, stacklevel=3)
return text return text
def _shared_strings(data: bytes) -> frozenset[str]:
"""Every literal in a workbook's shared string table, or nothing.
Read for one purpose: to tell a NUMBER from TEXT THAT LOOKS LIKE ONE. The
converter renders a numeric cell as a double, so an integral value arrives
as `5647500.0` -- and a text cell reading `92.0` arrives as `92.0` too. The
output alone cannot separate them, and rewriting on the output alone would
silently edit somebody's authored text.
Shared strings are the only text the converter recovers from a sheet at
all: an inline string (`t="inlineStr"`) is read as an EMPTY cell, measured
while the first xlsx fixture was built (`tests/fixtures/README.md`). So a
`<digits>.0` that is not in this set did not come from text.
Every failure returns the empty set, which makes the rewrite a no-op rather
than a guess: a workbook this cannot read keeps its converter decimals.
"""
try:
with zipfile.ZipFile(io.BytesIO(data)) as archive:
raw = archive.read("xl/sharedStrings.xml")
root = ElementTree.fromstring(raw)
except (KeyError, OSError, zipfile.BadZipFile, ElementTree.ParseError):
return frozenset()
return frozenset(
"".join(node.text or "" for node in item.iter(f"{{{_SSML}}}t")) for item in root
)
def _drop_converter_decimals(text: str, data: bytes) -> str:
"""Undo the converter's `N.0` on cells the workbook stores as integers.
Cell-scoped and never applied to prose: the pattern is anchored between two
unescaped pipes, so only a cell whose ENTIRE content is an integer with a
trailing `.0` is rewritten, and only when that same literal is absent from
the shared string table.
"""
literals = _shared_strings(data)
def rewrite(match: re.Match[str]) -> str:
digits = match.group(2)
if f"{digits}.0" in literals:
return match.group(0)
return f"|{match.group(1)}{digits}{match.group(3)}"
return _INTEGRAL_CELL.sub(rewrite, text)
_CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = { _CORE_EXTRACTORS: dict[str, Callable[[bytes], str]] = {
".md": _extract_passthrough, ".md": _extract_passthrough,
".txt": _extract_passthrough, ".txt": _extract_passthrough,

103
tests/fixtures/make_fixtures.py vendored
View file

@ -162,6 +162,108 @@ _XLSX_PARTS = {
} }
# A price sheet, and the negative control for it, in ONE workbook.
#
# Sheet 1 mirrors the shape measured on the K2 price sheet: row 1 carries a
# single title cell, so the table's HEADER ROW names one column while the rows
# below it carry three. That is what makes a reader see one column and a
# whitespace carpet where the source has a label and an amount. Column B also
# holds one long prose cell, which is what makes the simple-table writer pad
# every other row in that column out to its width.
#
# Sheet 2 is the negative control in the same file: one column in the SOURCE,
# so there are no columns to recover and nothing for a fix to invent.
#
# THREE NUMERIC CELLS AND ONE THAT ONLY LOOKS NUMERIC. `5647500` and `250000`
# are stored as numbers and are integral; `12.5` is stored as a number and is
# not; `92.0` is a SHARED STRING. The converter renders the first two as
# `5647500.0` and `250000.0` and the last one as `92.0` -- identical output for
# a number and for text, which is the whole reason the shared string table is
# consulted before any of them is rewritten.
_PRISARK_STRINGS = (
"Prisskjema",
"Post",
"Beskrivelse",
"Sum",
"01",
"Rigging og drift av byggeplass, medregnet alt som ikke er priset "
"spesifikt nedenfor og alt som er innkalkulert i de angitte prisene",
"02",
"Andel",
"03",
"92.0",
"Notat",
"Ingen kolonner her",
"Sum ikke oppgitt",
# A cell whose own text contains a pipe and a number. The converter escapes
# the pipe inside a pipe table, and the escape is what the rewrite's
# delimiter test has to survive: a `5.0` INSIDE a cell is not a cell.
"Kode 4 | 5.0",
"04",
)
_PRISARK_SHEET1 = (
'<row r="1"><c r="A1" t="s"><v>0</v></c></row>'
'<row r="2"><c r="A2" t="s"><v>1</v></c><c r="B2" t="s"><v>2</v></c>'
'<c r="C2" t="s"><v>3</v></c></row>'
'<row r="3"><c r="A3" t="s"><v>4</v></c><c r="B3" t="s"><v>5</v></c>'
'<c r="C3"><v>5647500</v></c></row>'
'<row r="4"><c r="A4" t="s"><v>6</v></c><c r="B4" t="s"><v>7</v></c>'
'<c r="C4"><v>12.5</v></c></row>'
'<row r="5"><c r="A5" t="s"><v>8</v></c><c r="B5" t="s"><v>9</v></c>'
'<c r="C5"><v>250000</v></c></row>'
'<row r="6"><c r="A6" t="s"><v>14</v></c><c r="B6" t="s"><v>13</v></c></row>'
)
_PRISARK_SHEET2 = (
'<row r="1"><c r="A1" t="s"><v>10</v></c></row>'
'<row r="2"><c r="A2" t="s"><v>11</v></c></row>'
'<row r="3"><c r="A3" t="s"><v>12</v></c></row>'
)
def _sheet(dimension: str, rows: str) -> str:
return (
_XML
+ '<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">'
+ f'<dimension ref="{dimension}"/><sheetData>'
+ rows
+ "</sheetData></worksheet>"
)
_PRISARK_PARTS = {
"[Content_Types].xml": _XML
+ '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
+ '<Default Extension="xml" ContentType="application/xml"/>'
+ '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
+ '<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>'
+ '<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
+ '<Override PartName="/xl/worksheets/sheet2.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
+ '<Override PartName="/xl/sharedStrings.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sharedStrings+xml"/>'
+ "</Types>",
"_rels/.rels": _XLSX_PARTS["_rels/.rels"],
"xl/_rels/workbook.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet1.xml"/>'
+ '<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet2.xml"/>'
+ '<Relationship Id="rId3" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/sharedStrings" Target="sharedStrings.xml"/>'
+ "</Relationships>",
"xl/workbook.xml": _XML
+ '<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
+ ' xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
+ '<sheets><sheet name="Prisark" sheetId="1" r:id="rId1"/>'
+ '<sheet name="Enkeltkolonne" sheetId="2" r:id="rId2"/></sheets></workbook>',
"xl/sharedStrings.xml": _XML
+ '<sst xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
+ f' count="{len(_PRISARK_STRINGS)}" uniqueCount="{len(_PRISARK_STRINGS)}">'
+ "".join(f"<si><t>{value}</t></si>" for value in _PRISARK_STRINGS)
+ "</sst>",
"xl/worksheets/sheet1.xml": _sheet("A1:C6", _PRISARK_SHEET1),
"xl/worksheets/sheet2.xml": _sheet("A1:A3", _PRISARK_SHEET2),
}
def build_ooxml(parts: dict[str, str]) -> bytes: def build_ooxml(parts: dict[str, str]) -> bytes:
"""Zip the parts with a fixed timestamp and no compression variance. """Zip the parts with a fixed timestamp and no compression variance.
@ -191,6 +293,7 @@ if __name__ == "__main__":
("two-line-krav.docx", _DOCX_PARTS), ("two-line-krav.docx", _DOCX_PARTS),
("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS), ("no-styles-krav.docx", _DOCX_NO_STYLES_PARTS),
("two-line-krav.xlsx", _XLSX_PARTS), ("two-line-krav.xlsx", _XLSX_PARTS),
("prisark.xlsx", _PRISARK_PARTS),
): ):
(HERE / name).write_bytes(build_ooxml(parts)) (HERE / name).write_bytes(build_ooxml(parts))
print(f"wrote {name}") print(f"wrote {name}")

BIN
tests/fixtures/prisark.xlsx vendored Normal file
View file

Binary file not shown.

43
tests/test_cli_build.py
View file

@ -426,3 +426,46 @@ def test_the_packaged_modules_do_not_reach_back_into_the_clone() -> None:
for name in ("propose.py", "corpus.py", "cli.py"): for name in ("propose.py", "corpus.py", "cli.py"):
source = (PROJECT_ROOT / "src" / "llm_ingestion_okf" / name).read_text(encoding="utf-8") source = (PROJECT_ROOT / "src" / "llm_ingestion_okf" / name).read_text(encoding="utf-8")
assert "sys.path" not in source assert "sys.path" not in source
def test_the_root_index_does_not_link_the_run_log(tmp_path: Path) -> None:
"""Producer and consumer must count the same documents. Measured: they did not.
`link_log_in_root_index` (`95eb271`) appended a markdown link to `log.md`
from the root index so a reader entering there could reach the one file
carrying `N`. The consumption contract SS 9.2 forbids a consumer from
enumerating the bundle directory unless the named profile says the index is
derived -- which for this profile it does not -- so the index tree IS the
whole map a consumer is allowed to use. Anything the index links is a
document, by that contract.
The cost was measured on K2 by the first consumer to walk the bundle with a
live model: their navigator followed the link and returned 630 documents
where this repository's own pre-pass counts 629, and the corpus run's own
log was reachable and citable as content. Our pre-pass excluding `log.md`
(`5a0c879`, F2) fixed the count on OUR side only; the disagreement is
produced HERE.
The log still exists at the bundle root, which is where SS 9 puts it and is
all F2 ever required. Upstream's own bundles show the link was never
required either: measured at `9a15b13`, 0 of the 24 shipped `index.md`
files name the single `log.md` in the set.
"""
inbox = inbox_with_subdirectories(tmp_path)
bundle = tmp_path / "bundle"
assert build(inbox, bundle) == 0
assert (bundle / "log.md").is_file(), "the log itself stays in the bundle"
index = (bundle / "index.md").read_text(encoding="utf-8")
assert "log.md" not in index
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools"))
import okf_consume
concepts = {
name
for name in tree(bundle)
if name.endswith(".md") and not name.endswith("index.md") and name != "log.md"
}
assert concepts, "the fixture must produce concepts for this control to mean anything"
assert len(okf_consume.enumerate_concepts(bundle)) == len(concepts)

65
tests/test_corpus_run.py
View file

@ -402,30 +402,25 @@ def test_a_second_run_into_the_same_bundle_reproduces_it_byte_for_byte(tmp_path:
assert second == first assert second == first
def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None: def test_the_root_index_does_not_link_the_bundles_own_log(tmp_path: Path) -> None:
"""A log nothing links is a file on disk, not a member of the bundle. """The log is in the bundle and out of the index. Both halves matter.
Measured on the artifact: the K2 bundle carried a conformant root `log.md` The harness DID link it (`95eb271`), so a consumer entering at `index.md`
that no index named, so a consumer walking the bundle from `index.md` -- could reach the one file carrying `N`. That was a LOCAL choice and said so:
which is the walk section 8 exists to support -- never reached the one file upstream's own reference bundles do not link their logs -- measured at
carrying `N`. `9a15b13`, 0 of the 24 shipped `index.md` files name the single `log.md` in
the bundle set -- so upstream showed the link was permitted, never required.
This is a LOCAL choice, not a conformance requirement, and the distinction It was removed on 2026-09-08 because the cost was measured. Consumption
is worth keeping straight. Section 9 lets `log.md` sit at any level and contract SS 9.2 forbids a consumer from enumerating the bundle directory
section 8 has an index enumerate its directory's contents, but upstream's unless the named profile says the index is derived, so the index tree is the
own reference bundles do not link it: measured at `9a15b13`, 0 of the 24 ENTIRE map a consumer may use and everything it links is a document. A
shipped `index.md` files name the single `log.md` in the bundle set. So consumer walking K2 that way returned 630 documents where our own pre-pass
upstream proves the link is not required, not that it is disallowed. counts 629, and the corpus run's own log was readable and citable as
content. `5a0c879` fixed the count on our side of a disagreement produced on
this one.
It is made HERE, in the harness, because the library cannot make it. The Section 9 still puts `log.md` at any level, and it is still written there.
log's content is the run's outcome, so it cannot be written before the
indexes are projected -- and an index that enumerated `log.md` off the
directory would gain the link only on the SECOND run, breaking the
rebuild-equals-incremental property the segmented bundle is built on. The
harness instead writes the link after the log, and only when it is not
already there -- a test, not an append, because the two reprojections
disagree about this line: the per-directory one drops it as a managed
entry, the flat one keeps it because its target is not an owned concept.
""" """
root = corpus(tmp_path, {"doc.md": SEGMENTABLE, "flat.md": SUBSTANTIVE}) root = corpus(tmp_path, {"doc.md": SEGMENTABLE, "flat.md": SUBSTANTIVE})
plans = tmp_path / "plans" plans = tmp_path / "plans"
@ -450,18 +445,12 @@ def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None:
] ]
assert okf_corpus_run.main(argv) == 0 assert okf_corpus_run.main(argv) == 0
index = (bundle / "index.md").read_text(encoding="utf-8") assert (bundle / "log.md").is_file(), "the log is still written to the bundle"
assert "](log.md)" in index for index_path in bundle.rglob("index.md"):
assert index.count("](log.md)") == 1 assert "log.md" not in index_path.read_text(encoding="utf-8"), index_path
# The link is to the log in THIS directory, so a nested index must not # Rebuild equals incremental, still -- the property the removed append was
# carry one: there is no `log.md` beside it to reach. # most likely to break, and the one that has to survive its removal too.
for nested in bundle.rglob("*/index.md"):
assert "](log.md)" not in nested.read_text(encoding="utf-8")
# Rebuild equals incremental, still. This is what discriminates the two
# ways the append could be wrong: a link the reprojection keeps would be
# doubled here, and one the harness forgot to re-write would vanish.
first = { first = {
path.relative_to(bundle).as_posix(): path.read_bytes() path.relative_to(bundle).as_posix(): path.read_bytes()
for path in sorted(bundle.rglob("*")) for path in sorted(bundle.rglob("*"))
@ -476,15 +465,14 @@ def test_the_root_index_links_the_bundles_own_log(tmp_path: Path) -> None:
assert second == first assert second == first
def test_the_log_link_holds_on_the_unsegmented_path_too(tmp_path: Path) -> None: def test_the_unsegmented_path_leaves_the_log_out_of_the_index_too(tmp_path: Path) -> None:
"""The two run modes reproject through different code, so both are pinned. """The two run modes reproject through different code, so both are pinned.
A run without `--plans-dir` uses `STRUCTURED_V1`, whose index is not A run without `--plans-dir` uses `STRUCTURED_V1`, whose index is not
per-directory and is rewritten by the singular reprojection rather than the per-directory and is rewritten by the singular reprojection rather than the
per-directory one. The harness writes `log.md` in both modes, so a link per-directory one. The removed append behaved differently under the two --
that only held on the segmented path would leave the plainer bundle with the per-directory reprojection dropped the line, the flat one kept it -- so
exactly the orphan this closes -- and if that path kept the line instead of a removal verified on one path only would prove nothing about the other.
dropping it, the second run would carry two.
""" """
root = corpus(tmp_path, {"a.md": SUBSTANTIVE, "b.md": SUBSTANTIVE}) root = corpus(tmp_path, {"a.md": SUBSTANTIVE, "b.md": SUBSTANTIVE})
bundle = tmp_path / "bundle" bundle = tmp_path / "bundle"
@ -500,8 +488,9 @@ def test_the_log_link_holds_on_the_unsegmented_path_too(tmp_path: Path) -> None:
] ]
assert okf_corpus_run.main(argv) == 0 assert okf_corpus_run.main(argv) == 0
assert (bundle / "log.md").is_file()
index = (bundle / "index.md").read_text(encoding="utf-8") index = (bundle / "index.md").read_text(encoding="utf-8")
assert "](log.md)" in index assert "log.md" not in index
assert okf_corpus_run.main(argv) == 0 assert okf_corpus_run.main(argv) == 0
assert (bundle / "index.md").read_text(encoding="utf-8") == index assert (bundle / "index.md").read_text(encoding="utf-8") == index

174
tests/test_extract.py
View file

@ -9,7 +9,9 @@ call — it is pure, deterministic plumbing.
from __future__ import annotations from __future__ import annotations
import hashlib
import importlib.util import importlib.util
import re
import sys import sys
from pathlib import Path from pathlib import Path
@ -325,10 +327,12 @@ DOCX_TEXT = "# Krav til helning\n\n60 og 70 1:15"
# xlsx: the sheet name becomes a heading and the rows become a table. The # xlsx: the sheet name becomes a heading and the rows become a table. The
# label/value pairing survives on one row, which is the property that matters. # label/value pairing survives on one row, which is the property that matters.
XLSX_TEXT = ( #
"## Krav {#sheet-1}\n\n Krav til helning \n" # THIS LITERAL MOVED ONCE, deliberately, and the move is the fix reported in
" ------------------ ------\n 60 og 70 1:15" # `docs/2026-09-08-prisform-og-loggen-k2.md`: the spreadsheet row now writes
) # pipe tables, so the cells arrive delimited instead of padded. Every character
# of content is the same; only the table form changed.
XLSX_TEXT = "## Krav {#sheet-1}\n\n| Krav til helning | |\n|----|----|\n| 60 og 70 | 1:15 |"
# The negative control, committed rather than described: the SAME document # The negative control, committed rather than described: the SAME document
# without `word/styles.xml`. The body survives and the heading marker does not. # without `word/styles.xml`. The body survives and the heading marker does not.
@ -457,3 +461,165 @@ def test_empty_csv_fails_fast() -> None:
with pytest.raises(ExtractionError) as excinfo: with pytest.raises(ExtractionError) as excinfo:
extract_text("empty.csv", b"") extract_text("empty.csv", b"")
assert excinfo.value.code == "extractor_empty_csv" assert excinfo.value.code == "extractor_empty_csv"
# --- xlsx: the spreadsheet form a consumer has to read ---
#
# Measured on K2 and reported upstream by the first consumer to read the bundle
# with a live model (`docs/2026-09-08-prisform-og-loggen-k2.md`): the converter's
# DEFAULT markdown writer emits simple tables, which pad every cell out to the
# width of the widest cell in its column. One long prose cell therefore turns
# every other row in that column into a whitespace carpet -- 887 characters
# between a label and its amount on the real sheet -- while the header row names
# a single column because only the first source cell in row 1 is filled. The
# bytes reach the reader and the STRUCTURE does not.
#
# `prisark.xlsx` is that shape in miniature, hand-laid rather than recorded, and
# it carries its own negative control on a second sheet.
PRISARK = "prisark.xlsx"
def _table_rows(text: str) -> list[list[str]]:
"""Every pipe-table row in `text`, as its cells, in order."""
rows = []
for line in text.split("\n"):
stripped = line.strip()
if not (stripped.startswith("|") and stripped.endswith("|")):
continue
cells = [cell.strip() for cell in stripped[1:-1].split("|")]
if all(set(cell) <= set("-:") and cell for cell in cells):
continue # the header separator is punctuation, not a row
rows.append(cells)
return rows
@requires_pandoc
def test_a_spreadsheet_keeps_its_columns_one_row_per_line() -> None:
"""The label and the amount arrive as separate cells on one line.
This is the property the whole change exists for. Asserted as properties
rather than only as a frozen literal, because a literal pins bytes and says
nothing about which of them was the point.
"""
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
text = extract_text(PRISARK, data)
rows = _table_rows(text)
assert [
"01",
"Rigging og drift av byggeplass, medregnet alt som ikke er "
"priset spesifikt nedenfor og alt som er innkalkulert i de angitte "
"prisene",
"5647500",
] in rows
assert ["02", "Andel", "12.5"] in rows
longest = max((len(run) for run in re.findall(r" {2,}", text)), default=0)
assert longest <= 8, f"a whitespace run of {longest} is a carpet, not a column"
@requires_pandoc
def test_an_integral_amount_loses_the_converters_decimal_and_a_real_one_keeps_it() -> None:
"""`5647500` is a number; `92.0` in the same sheet is TEXT.
The converter renders both as `<digits>.0`, so the output alone cannot tell
them apart. The shared string table can, and is what the rewrite consults --
which is why this test asserts both directions from ONE document.
"""
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
text = extract_text(PRISARK, data)
assert "5647500.0" not in text
assert "250000.0" not in text
assert "12.5" in text, "a genuine decimal is a value, not a converter artefact"
assert ["03", "92.0", "250000"] in _table_rows(text), (
"a shared-string cell reading 92.0 is author text and survives verbatim"
)
assert "Kode 4 \\| 5.0" in text, (
"a `5.0` INSIDE a cell is not a cell: the delimiter test is what sees that"
)
@requires_pandoc
def test_a_single_column_sheet_gains_no_columns() -> None:
"""The negative control, in the same document as the case it controls.
Sheet 2 has ONE column in the source. There is nothing to recover, so the
fix must not invent a second cell anywhere on it. Its three values arrive
in order and alone.
It is NOT byte-identical before and after the change, and that is measured
rather than glossed: the writer emits a pipe table for every table it
writes, so a one-column table changes delimiter form too. What must not
change is the cell content and the column count.
"""
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
text = extract_text(PRISARK, data)
single = text.split("## Enkeltkolonne")[1]
rows = _table_rows(single)
assert [cells for cells in rows if any(cells)] == [
["Notat"],
["Ingen kolonner her"],
["Sum ikke oppgitt"],
]
# The scoping control. The same writer change applied to the other four office
# rows was MEASURED to move them (the odt fixture 1366 -> 1105 characters), so
# this digest can fail; it is not a tautology. The change is deliberately
# spreadsheet-only: a spreadsheet IS a grid and has no prose fallback, while
# moving docx/pptx/odt/rtf would move a corpus denominator nothing has measured.
# A red here means the writer stopped being scoped -- read the diff and decide.
OFFICE_TEXT_DIGESTS = {
"k2-office/krav-tekstdokument.odt": (
"58c9776f0d7f2b2a3a9d2774e4ae243b265c31b5b6b96914ef4db419fa66e4e2"
),
"k2-office/krav-presentasjon.pptx": (
"752420a04d651a416938ff9f0b3c2de5849bd2ea1d52063dd88aaae65ab99b90"
),
"k2-office/krav-rikt-tekstformat.rtf": (
"79cbf756eb482bb603f82c171d11efe74b9ba62ab9ef679ecd6bb8c3b3740ffb"
),
}
@requires_pandoc
@pytest.mark.parametrize("relative", sorted(OFFICE_TEXT_DIGESTS))
def test_the_other_office_rows_are_untouched_by_the_spreadsheet_writer(relative: str) -> None:
path = FIXTURES / relative
with pytest.warns(ExtractionWarning):
text = extract_text(path.name, path.read_bytes())
assert hashlib.sha256(text.encode("utf-8")).hexdigest() == OFFICE_TEXT_DIGESTS[relative]
# The whole fixture, frozen against the same named converter version as the
# literals above. The property tests say WHAT matters; this one catches any
# other byte moving without anybody noticing.
PRISARK_TEXT = (
"## Prisark {#sheet-1}\n\n"
"| Prisskjema | | |\n"
"|----|----|----|\n"
"| Post | Beskrivelse | Sum |\n"
"| 01 | Rigging og drift av byggeplass, medregnet alt som ikke er priset "
"spesifikt nedenfor og alt som er innkalkulert i de angitte prisene | 5647500 |\n"
"| 02 | Andel | 12.5 |\n"
"| 03 | 92.0 | 250000 |\n"
"| 04 | Kode 4 \\| 5.0 | |\n\n"
"## Enkeltkolonne {#sheet-2}\n\n"
"| Notat |\n"
"|----|\n"
"| Ingen kolonner her |\n"
"| Sum ikke oppgitt |"
)
@requires_pandoc
def test_prisark_extracts_to_its_frozen_text() -> None:
data = (FIXTURES / PRISARK).read_bytes()
with pytest.warns(ExtractionWarning):
assert extract_text(PRISARK, data) == PRISARK_TEXT

23
tests/test_okf_consume.py
View file

@ -115,21 +115,31 @@ def _copy_bundle(source: Path, target: Path) -> None:
def test_the_index_walk_excludes_a_linked_log_from_concept_navigation(tmp_path: Path) -> None: def test_the_index_walk_excludes_a_linked_log_from_concept_navigation(tmp_path: Path) -> None:
"""A linked `log.md` is bundle metadata, not a concept -- measured on K2 (S7 F2). """A linked `log.md` is bundle metadata, not a concept -- measured on K2 (S7 F2).
`link_log_in_root_index` (`corpus.py`) links a run's own log from the root `link_log_in_root_index` (`corpus.py`, `95eb271`) linked a run's own log from
index so a reader entering at `index.md` can reach it (`95eb271`). That link the root index so a reader entering at `index.md` could reach it. That link
makes the log reachable by the same walk this instrument uses to enumerate makes the log reachable by the same walk this instrument uses to enumerate
concepts, and a walk that does not distinguish "linked" from "concept" concepts, and a walk that does not distinguish "linked" from "concept"
counts it as a 630th concept on a 629-concept bundle -- exactly what the S7 counts it as a 630th concept on a 629-concept bundle -- exactly what the S7
acid test measured, with the log then ranked and cut like real content. acid test measured, with the log then ranked and cut like real content.
THE PRODUCER NO LONGER WRITES THAT LINK (2026-09-08), so the fixture writes
it here instead. The exclusion stays and is not dead code: every bundle
built between `95eb271` and that removal carries the link, including the
ones consumers are reading today, and this instrument must count 629 on
those too.
""" """
from llm_ingestion_okf.corpus import LOG_NAME, link_log_in_root_index from llm_ingestion_okf.corpus import LOG_NAME
from llm_ingestion_okf.profiles import SEGMENTED_OKF_V0_2
copy = tmp_path / "bundle" copy = tmp_path / "bundle"
_copy_bundle(GOLDEN, copy) _copy_bundle(GOLDEN, copy)
(copy / LOG_NAME).write_text("# Corpus run history\n\nN = 3\n", encoding="utf-8") (copy / LOG_NAME).write_text("# Corpus run history\n\nN = 3\n", encoding="utf-8")
link_log_in_root_index(copy, SEGMENTED_OKF_V0_2) index_path = copy / "index.md"
index = (copy / "index.md").read_text(encoding="utf-8") index_path.write_text(
index_path.read_text(encoding="utf-8") + f"- [Corpus run history]({LOG_NAME})\n",
encoding="utf-8",
newline="",
)
index = index_path.read_text(encoding="utf-8")
assert "](log.md)" in index, ( assert "](log.md)" in index, (
"the fixture must actually link the log for this control to mean anything" "the fixture must actually link the log for this control to mean anything"
) )
@ -1046,6 +1056,7 @@ def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md", PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-blindsone-below-k-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md", PROJECT_ROOT / "docs" / "2026-09-08-blindsone-laas2-budsjett-k2.md",
PROJECT_ROOT / "docs" / "2026-09-08-prisform-og-loggen-k2.md",
PROJECT_ROOT / "README.md", PROJECT_ROOT / "README.md",
PROJECT_ROOT / "CLAUDE.md", PROJECT_ROOT / "CLAUDE.md",
] ]

15
tools/okf_consume.py
View file

@ -110,11 +110,16 @@ def _walk_index_tree(
if name == index_name: if name == index_name:
pending.append(target) pending.append(target)
elif name == LOG_NAME: elif name == LOG_NAME:
# A run's own log, linked for bundle navigation # A run's own log -- reachable, but metadata, never a
# (`corpus.link_log_in_root_index`) -- reachable, but metadata, # concept. Counted as one it inflated the K2 walk to 630
# never a concept. Counted as one it inflated the K2 walk to # against a 629-concept bundle and let the log rank and cut
# 630 against a 629-concept bundle and let the log rank and # like real content (measured, S7 F2).
# cut like real content (measured, S7 F2). #
# THE PRODUCER NO LONGER WRITES THE LINK (2026-09-08, after a
# consumer's navigator followed it and returned 630 where this
# counts 629). This branch is not dead: every bundle built
# between `95eb271` and that change carries it, including the
# ones consumers are reading today.
continue continue
elif target.endswith(suffix): elif target.endswith(suffix):
concepts.add(target[: -len(suffix)]) concepts.add(target[: -len(suffix)])