llm-ingestion-okf/docs/2026-09-03-k2-bundle-rebuild.md

Rebuilding the K2 bundle so it meets the consumption contract, 2026-09-03

A consumer measured the bundle this repository's corpus harness built on 2026-09-02 and found four things missing (docs/2026-09-03-syretest-s7a-k2.md in portfolio-optimiser): no adjudication key in any of the 39 concepts, no log.md, so N was not recoverable from the artifact, and two concepts above 100 000 tokens — one file, one concept, for the largest PDFs.

This is the rebuild, the cause, and the numbers. Counts only: the corpus is public procurement material, but nothing here needs a document body or a full filename list to be checkable.

The cause: one, not four

The harness passed profile=STRUCTURED_V1 and no segmentation plans at all.

grep -n "STRUCTURED_V1\|SEGMENTED\|segmentations" tools/okf_corpus_run.py
git log --oneline -- tools/okf_corpus_run.py

Before this session that printed two hits, both STRUCTURED_V1, and a single commit. STRUCTURED_V1 does not declare the segmentation capability, so a plan passed to it would have been refused outright rather than ignored — and none was passed, because nothing in the run path produced one.

All three findings follow from that. adjudication is written only inside the plan-covered branch (inbox.py), so a run with no plans cannot emit it. A document with no plan lands as one concept, which is what the >100 000-token concepts were. log.md was never written by anything.

The plan this work came from says so directly: step 17's Reuses names process_inbox "with the per-document plan mapping from Step 15". The harness shipped without that wiring. It is an omission in one file, not a design decision that was later regretted.

What changed

Three commits, each test-first.

1. The harness replays plans and writes the bundle's log. --plans-dir selects the proposals to replay and the profile follows from it; --bundle-id and --okf-version are arguments, because a profile names a key and the caller owns its value (decision E1). log.md is written in SPEC §9 form and dated from ingested_at, never the wall clock. Without --plans-dir the run is the flat STRUCTURED_V1 run that produced the published K1/K2 numbers.
2. The proposer scopes a document's segments under a caller's prefix. Measured first: 39 documents proposed 618 entries under 601 distinct paths — 17 paths claimed by two documents each. Section numbering is document-local, so this is structural. Every collision reaches Door B's gate, which refuses per document, so those documents would have become coded rejections instead of concepts. With --path-prefix set to each document's stem: 618 entries, 618 distinct paths, 0 collisions.
3. Nothing to propose writes no artifact. 11 of the 39 documents proposed zero segments. The proposer wrote an empty artifact for each and exited 0; process_inbox refuses an empty plan by design, and the first rebuild attempt stopped on segmentation_plan_invalid before writing anything. Exit 1 and no file, distinct from exit 2, so a driver can tell "this document lands as one flat concept" from "stop".

The rebuild

Bundle: ~/corpora/okf-telling-20260829/K2-bundle-20260903/ — 5.5 MB, 1108 files: 629 concepts, 478 index.md (one per directory, the profile sets per_directory), and one log.md. Plans: ~/corpora/okf-telling-20260829/K2-plans-20260903/ — 28 of them, one per document that had something to propose. Both outside the repository, both durable.

Reproduce into a FRESH plans directory. A plan is selected by source_sha256, so re-running the proposer into a directory that already holds these plans leaves two files claiming the same hash, and _resolve_plans refuses that rather than picking one:

PLANS=~/corpora/okf-telling-20260829/K2-plans-$(date +%Y%m%d-%H%M%S); mkdir -p "$PLANS"
i=0
for f in ~/corpora/okf-telling-20260829/K2/trinn1/*; do
  i=$((i+1)); b=$(basename "$f")
  .venv/bin/python tools/okf_propose_segments.py "$f" \
    --out "$PLANS/$(printf '%02d' $i).json" \
    --path-prefix "${b%.*}" --proposed-at 2026-09-03T00:00:00Z
done   # exit 1 for the 11 with nothing to propose, exit 2 for the 4 unreadable
.venv/bin/python tools/okf_corpus_run.py \
  --corpus ~/corpora/okf-telling-20260829/K2/trinn1 \
  --report ~/corpora/okf-telling-20260829/K2-bundle-20260903-report.md \
  --bundle ~/corpora/okf-telling-20260829/K2-bundle-20260903 \
  --ingested-at 2026-09-03T00:00:00Z \
  --plans-dir "$PLANS" \
  --bundle-id k2-trinn1-20260903 --okf-version 0.2

Converter as the harness resolved it: the vendored pypandoc binary, version 3.9 — not the host's 3.10.2.

The numbers, each with its denominator

figure	value
N (corpus directory file count, computed)	43
merged	39/43
coded rejections	4/43 (extractor_unknown 3, extractor_empty_pdf 1)
concepts	629
concepts carrying adjudication	618/629, every one of them proposed
body characters, max	217 472
body characters, median	441
concepts over 100 000 characters	4/629
wall time	784.29 s total, 18.239 s per file
K1b	39 + 4 = 43 = N, run exited 0
rebuild == incremental	diff -r exit 0 over all 1108 files

The previous bundle, re-measured here rather than quoted: 39 concepts, 0/39 carrying adjudication, max body 267 548 characters, no log.md.

log.md is a file the run path writes into a directory Door B enumerates on the next round, so the rebuild property was re-measured rather than assumed: the whole corpus was run a second time into the same bundle and compared against a snapshot with diff -r, which exited 0. A test in tests/test_corpus_run.py pins the same property on a synthetic corpus, so it fails in seconds rather than in 13 minutes.

K1b is now recoverable from the bundle alone, which was the point of §9:

python - <<'PY'
import re; log=open('.../K2-bundle-20260903/log.md').read()
n=int(re.search(r'N = (\d+)',log).group(1))
merged=int(re.search(r'merged = (\d+)',log).group(1))
codes=[int(m.group(2)) for m in re.finditer(r'`([a-z_]+)`: (\d+)',log)]
assert merged+sum(codes)==n; print(merged,'+',sum(codes),'=',n)
PY

Two numbers that need their units stated

The consumer measured tokens (max 121 462); the figures above are characters, which is what this repository can count without adopting somebody's tokenizer. Their own two numbers imply ≈2.20 characters per token on this corpus. Applied to the maximum here that is ≈98 700 tokens — a derived figure, and the only one in this document that is not a direct count.

The 11 concepts with no adjudication

They are exactly the 11 documents that proposed zero segments: no plan, so the ordinary one-concept path, so no key. Marking them proposed would claim a proposal that was never made. A consumer distinguishes three states — proposed, adjudicated, and absent — and absent is the honest one here. Reported rather than changed: the wire form is a ratified contract (docs/plan/office-intake.md §4–5) and is not this session's to move.

Point 4: why the large concepts were not split — and which one still is not

Not "the proposer refused". It was never run. Given plans, the two documents the consumer named do split:

document	entries proposed	largest segment (chars)
Bilag 3.1 (was 270 572 chars, one concept)	34	180 604
Bilag 1 (was 257 072 chars, one concept)	18	142 667

Both are now below the 100 000-token line. In each the trailing segment still absorbs most of the document, because the headings the mechanical rules find in a PDF are largely table-of-contents lines near the front.

The largest concept in the new bundle is a different document: Bilag 9.1, at 217 472 characters, one of the 11 with no proposable structure at all. This is §10's "no declared structure" as Topic 1b measured it — 23 of 33 PDFs carry no outline, and 95 % of the outline entries that do exist are AutoCAD export metadata. The mechanical rules cannot reach it, and nothing here pretends otherwise. Reported, not fixed: Arm C is its own decision.

What this does not measure

• The bundle is not re-measured against the consumption contract's checkable half by tools/okf_contract_check.py; that check reads a consumption skill, not a bundle.
• pptx, odt and rtf still have denominator 0 in this corpus. Unmeasured, not passing.
• Segmentation quality is unchanged. K3 measured 7 of 12 splits as too coarse and that number is a floor; the entries here are proposed precisely because no one has judged them.