docs(measure): the K2 bundle rebuild, its cause and its numbers
One cause behind all four of the consumer's findings: the harness ran under STRUCTURED_V1 with no plans. 629 concepts, 618 of them carrying `adjudication: proposed`, K1b recoverable from the bundle's own log.md, and the two >100k-token concepts split into 34 and 18 entries. The largest concept is now a different document that has no derivable structure at all -- reported, not fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
bca722fa85
commit
503a7a08da
1 changed files with 163 additions and 0 deletions
163
docs/2026-09-03-k2-bundle-rebuild.md
Normal file
163
docs/2026-09-03-k2-bundle-rebuild.md
Normal file
|
|
@ -0,0 +1,163 @@
|
||||||
|
# Rebuilding the K2 bundle so it meets the consumption contract, 2026-09-03
|
||||||
|
|
||||||
|
A consumer measured the bundle this repository's corpus harness built on
|
||||||
|
2026-09-02 and found four things missing (`docs/2026-09-03-syretest-s7a-k2.md`
|
||||||
|
in `portfolio-optimiser`): no `adjudication` key in any of the 39 concepts, no
|
||||||
|
`log.md`, so `N` was not recoverable from the artifact, and two concepts above
|
||||||
|
100 000 tokens — one file, one concept, for the largest PDFs.
|
||||||
|
|
||||||
|
This is the rebuild, the cause, and the numbers. Counts only: the corpus is
|
||||||
|
public procurement material, but nothing here needs a document body or a full
|
||||||
|
filename list to be checkable.
|
||||||
|
|
||||||
|
## The cause: one, not four
|
||||||
|
|
||||||
|
The harness passed `profile=STRUCTURED_V1` and no segmentation plans at all.
|
||||||
|
|
||||||
|
```
|
||||||
|
grep -n "STRUCTURED_V1\|SEGMENTED\|segmentations" tools/okf_corpus_run.py
|
||||||
|
git log --oneline -- tools/okf_corpus_run.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Before this session that printed two hits, both `STRUCTURED_V1`, and a single
|
||||||
|
commit. `STRUCTURED_V1` does not declare the segmentation capability, so a plan
|
||||||
|
passed to it would have been refused outright rather than ignored — and none
|
||||||
|
was passed, because nothing in the run path produced one.
|
||||||
|
|
||||||
|
All three findings follow from that. `adjudication` is written only inside the
|
||||||
|
plan-covered branch (`inbox.py`), so a run with no plans cannot emit it. A
|
||||||
|
document with no plan lands as one concept, which is what the >100 000-token
|
||||||
|
concepts were. `log.md` was never written by anything.
|
||||||
|
|
||||||
|
The plan this work came from says so directly: step 17's *Reuses* names
|
||||||
|
`process_inbox` "with the per-document plan mapping from Step 15". The harness
|
||||||
|
shipped without that wiring. It is an omission in one file, not a design
|
||||||
|
decision that was later regretted.
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
Three commits, each test-first.
|
||||||
|
|
||||||
|
1. **The harness replays plans and writes the bundle's log.** `--plans-dir`
|
||||||
|
selects the proposals to replay and the profile follows from it;
|
||||||
|
`--bundle-id` and `--okf-version` are arguments, because a profile names a
|
||||||
|
key and the caller owns its value (decision E1). `log.md` is written in
|
||||||
|
SPEC §9 form and dated from `ingested_at`, never the wall clock. Without
|
||||||
|
`--plans-dir` the run is the flat `STRUCTURED_V1` run that produced the
|
||||||
|
published K1/K2 numbers.
|
||||||
|
2. **The proposer scopes a document's segments under a caller's prefix.**
|
||||||
|
Measured first: 39 documents proposed 618 entries under **601** distinct
|
||||||
|
paths — 17 paths claimed by two documents each. Section numbering is
|
||||||
|
document-local, so this is structural. Every collision reaches Door B's
|
||||||
|
gate, which refuses per document, so those documents would have become
|
||||||
|
coded rejections instead of concepts. With `--path-prefix` set to each
|
||||||
|
document's stem: 618 entries, **618** distinct paths, 0 collisions.
|
||||||
|
3. **Nothing to propose writes no artifact.** 11 of the 39 documents proposed
|
||||||
|
zero segments. The proposer wrote an empty artifact for each and exited 0;
|
||||||
|
`process_inbox` refuses an empty plan by design, and the first rebuild
|
||||||
|
attempt stopped on `segmentation_plan_invalid` before writing anything.
|
||||||
|
Exit 1 and no file, distinct from exit 2, so a driver can tell "this
|
||||||
|
document lands as one flat concept" from "stop".
|
||||||
|
|
||||||
|
## The rebuild
|
||||||
|
|
||||||
|
Bundle: `~/corpora/okf-telling-20260829/K2-bundle-20260903/` (5.5 MB, 29
|
||||||
|
directories). Plans: `~/corpora/okf-telling-20260829/K2-plans-20260903/`.
|
||||||
|
Both outside the repository, both durable.
|
||||||
|
|
||||||
|
```
|
||||||
|
for f in ~/corpora/okf-telling-20260829/K2/trinn1/*; do
|
||||||
|
b=$(basename "$f"); .venv/bin/python tools/okf_propose_segments.py "$f" \
|
||||||
|
--out ~/corpora/okf-telling-20260829/K2-plans-20260903/"${b%.*}".json \
|
||||||
|
--path-prefix "${b%.*}" --proposed-at 2026-09-03T00:00:00Z
|
||||||
|
done
|
||||||
|
.venv/bin/python tools/okf_corpus_run.py \
|
||||||
|
--corpus ~/corpora/okf-telling-20260829/K2/trinn1 \
|
||||||
|
--report ~/corpora/okf-telling-20260829/K2-bundle-20260903-report.md \
|
||||||
|
--bundle ~/corpora/okf-telling-20260829/K2-bundle-20260903 \
|
||||||
|
--ingested-at 2026-09-03T00:00:00Z \
|
||||||
|
--plans-dir ~/corpora/okf-telling-20260829/K2-plans-20260903 \
|
||||||
|
--bundle-id k2-trinn1-20260903 --okf-version 0.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Converter as the harness resolved it: the vendored `pypandoc` binary, version
|
||||||
|
**3.9** — not the host's 3.10.2.
|
||||||
|
|
||||||
|
### The numbers, each with its denominator
|
||||||
|
|
||||||
|
| figure | value |
|
||||||
|
|--------|-------|
|
||||||
|
| `N` (corpus directory file count, computed) | 43 |
|
||||||
|
| merged | 39/43 |
|
||||||
|
| coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) |
|
||||||
|
| concepts | **629** |
|
||||||
|
| concepts carrying `adjudication` | **618/629**, every one of them `proposed` |
|
||||||
|
| body characters, max | 217 472 |
|
||||||
|
| body characters, median | 441 |
|
||||||
|
| concepts over 100 000 characters | 4/629 |
|
||||||
|
| wall time | 784.29 s total, 18.239 s per file |
|
||||||
|
| K1b | `39 + 4 = 43 = N`, run exited `0` |
|
||||||
|
|
||||||
|
The previous bundle, re-measured here rather than quoted: 39 concepts, **0/39**
|
||||||
|
carrying `adjudication`, max body 267 548 characters, no `log.md`.
|
||||||
|
|
||||||
|
**K1b is now recoverable from the bundle alone**, which was the point of §9:
|
||||||
|
|
||||||
|
```
|
||||||
|
python - <<'PY'
|
||||||
|
import re; log=open('.../K2-bundle-20260903/log.md').read()
|
||||||
|
n=int(re.search(r'N = (\d+)',log).group(1))
|
||||||
|
merged=int(re.search(r'merged = (\d+)',log).group(1))
|
||||||
|
codes=[int(m.group(2)) for m in re.finditer(r'`([a-z_]+)`: (\d+)',log)]
|
||||||
|
assert merged+sum(codes)==n; print(merged,'+',sum(codes),'=',n)
|
||||||
|
PY
|
||||||
|
```
|
||||||
|
|
||||||
|
### Two numbers that need their units stated
|
||||||
|
|
||||||
|
The consumer measured **tokens** (max 121 462); the figures above are
|
||||||
|
**characters**, which is what this repository can count without adopting
|
||||||
|
somebody's tokenizer. Their own two numbers imply ≈2.20 characters per token on
|
||||||
|
this corpus. Applied to the maximum here that is ≈98 700 tokens — a **derived**
|
||||||
|
figure, and the only one in this document that is not a direct count.
|
||||||
|
|
||||||
|
### The 11 concepts with no `adjudication`
|
||||||
|
|
||||||
|
They are exactly the 11 documents that proposed zero segments: no plan, so the
|
||||||
|
ordinary one-concept path, so no key. Marking them `proposed` would claim a
|
||||||
|
proposal that was never made. A consumer distinguishes three states — `proposed`,
|
||||||
|
`adjudicated`, and absent — and absent is the honest one here. Reported rather
|
||||||
|
than changed: the wire form is a ratified contract (`docs/plan/office-intake.md`
|
||||||
|
§4–5) and is not this session's to move.
|
||||||
|
|
||||||
|
## Point 4: why the large concepts were not split — and which one still is not
|
||||||
|
|
||||||
|
Not "the proposer refused". It was never run. Given plans, the two documents
|
||||||
|
the consumer named **do** split:
|
||||||
|
|
||||||
|
| document | entries proposed | largest segment (chars) |
|
||||||
|
|----------|------------------|-------------------------|
|
||||||
|
| Bilag 3.1 (was 270 572 chars, one concept) | 34 | 180 604 |
|
||||||
|
| Bilag 1 (was 257 072 chars, one concept) | 18 | 142 667 |
|
||||||
|
|
||||||
|
Both are now below the 100 000-token line. In each the trailing segment still
|
||||||
|
absorbs most of the document, because the headings the mechanical rules find in
|
||||||
|
a PDF are largely table-of-contents lines near the front.
|
||||||
|
|
||||||
|
The largest concept in the new bundle is a **different** document: Bilag 9.1, at
|
||||||
|
217 472 characters, one of the 11 with no proposable structure at all. This is
|
||||||
|
§10's "no declared structure" as Topic 1b measured it — 23 of 33 PDFs carry no
|
||||||
|
outline, and 95 % of the outline entries that do exist are AutoCAD export
|
||||||
|
metadata. The mechanical rules cannot reach it, and nothing here pretends
|
||||||
|
otherwise. Reported, not fixed: Arm C is its own decision.
|
||||||
|
|
||||||
|
## What this does not measure
|
||||||
|
|
||||||
|
- The bundle is not re-measured against the consumption contract's checkable
|
||||||
|
half by `tools/okf_contract_check.py`; that check reads a consumption skill,
|
||||||
|
not a bundle.
|
||||||
|
- `pptx`, `odt` and `rtf` still have denominator **0** in this corpus. Unmeasured,
|
||||||
|
not passing.
|
||||||
|
- Segmentation quality is unchanged. K3 measured 7 of 12 splits as too coarse
|
||||||
|
and that number is a floor; the entries here are `proposed` precisely because
|
||||||
|
no one has judged them.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue