docs(measure): the K2 bundle rebuild, its cause and its numbers
One cause behind all four of the consumer's findings: the harness ran under STRUCTURED_V1 with no plans. 629 concepts, 618 of them carrying `adjudication: proposed`, K1b recoverable from the bundle's own log.md, and the two >100k-token concepts split into 34 and 18 entries. The largest concept is now a different document that has no derivable structure at all -- reported, not fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
bca722fa85
commit
503a7a08da
1 changed files with 163 additions and 0 deletions
163
docs/2026-09-03-k2-bundle-rebuild.md
Normal file
163
docs/2026-09-03-k2-bundle-rebuild.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# Rebuilding the K2 bundle so it meets the consumption contract, 2026-09-03
|
||||
|
||||
A consumer measured the bundle this repository's corpus harness built on
|
||||
2026-09-02 and found four things missing (`docs/2026-09-03-syretest-s7a-k2.md`
|
||||
in `portfolio-optimiser`): no `adjudication` key in any of the 39 concepts, no
|
||||
`log.md`, so `N` was not recoverable from the artifact, and two concepts above
|
||||
100 000 tokens — one file, one concept, for the largest PDFs.
|
||||
|
||||
This is the rebuild, the cause, and the numbers. Counts only: the corpus is
|
||||
public procurement material, but nothing here needs a document body or a full
|
||||
filename list to be checkable.
|
||||
|
||||
## The cause: one, not four
|
||||
|
||||
The harness passed `profile=STRUCTURED_V1` and no segmentation plans at all.
|
||||
|
||||
```
|
||||
grep -n "STRUCTURED_V1\|SEGMENTED\|segmentations" tools/okf_corpus_run.py
|
||||
git log --oneline -- tools/okf_corpus_run.py
|
||||
```
|
||||
|
||||
Before this session that printed two hits, both `STRUCTURED_V1`, and a single
|
||||
commit. `STRUCTURED_V1` does not declare the segmentation capability, so a plan
|
||||
passed to it would have been refused outright rather than ignored — and none
|
||||
was passed, because nothing in the run path produced one.
|
||||
|
||||
All three findings follow from that. `adjudication` is written only inside the
|
||||
plan-covered branch (`inbox.py`), so a run with no plans cannot emit it. A
|
||||
document with no plan lands as one concept, which is what the >100 000-token
|
||||
concepts were. `log.md` was never written by anything.
|
||||
|
||||
The plan this work came from says so directly: step 17's *Reuses* names
|
||||
`process_inbox` "with the per-document plan mapping from Step 15". The harness
|
||||
shipped without that wiring. It is an omission in one file, not a design
|
||||
decision that was later regretted.
|
||||
|
||||
## What changed
|
||||
|
||||
Three commits, each test-first.
|
||||
|
||||
1. **The harness replays plans and writes the bundle's log.** `--plans-dir`
|
||||
selects the proposals to replay and the profile follows from it;
|
||||
`--bundle-id` and `--okf-version` are arguments, because a profile names a
|
||||
key and the caller owns its value (decision E1). `log.md` is written in
|
||||
SPEC §9 form and dated from `ingested_at`, never the wall clock. Without
|
||||
`--plans-dir` the run is the flat `STRUCTURED_V1` run that produced the
|
||||
published K1/K2 numbers.
|
||||
2. **The proposer scopes a document's segments under a caller's prefix.**
|
||||
Measured first: 39 documents proposed 618 entries under **601** distinct
|
||||
paths — 17 paths claimed by two documents each. Section numbering is
|
||||
document-local, so this is structural. Every collision reaches Door B's
|
||||
gate, which refuses per document, so those documents would have become
|
||||
coded rejections instead of concepts. With `--path-prefix` set to each
|
||||
document's stem: 618 entries, **618** distinct paths, 0 collisions.
|
||||
3. **Nothing to propose writes no artifact.** 11 of the 39 documents proposed
|
||||
zero segments. The proposer wrote an empty artifact for each and exited 0;
|
||||
`process_inbox` refuses an empty plan by design, and the first rebuild
|
||||
attempt stopped on `segmentation_plan_invalid` before writing anything.
|
||||
Exit 1 and no file, distinct from exit 2, so a driver can tell "this
|
||||
document lands as one flat concept" from "stop".
|
||||
|
||||
## The rebuild
|
||||
|
||||
Bundle: `~/corpora/okf-telling-20260829/K2-bundle-20260903/` (5.5 MB, 29
|
||||
directories). Plans: `~/corpora/okf-telling-20260829/K2-plans-20260903/`.
|
||||
Both outside the repository, both durable.
|
||||
|
||||
```
|
||||
for f in ~/corpora/okf-telling-20260829/K2/trinn1/*; do
|
||||
b=$(basename "$f"); .venv/bin/python tools/okf_propose_segments.py "$f" \
|
||||
--out ~/corpora/okf-telling-20260829/K2-plans-20260903/"${b%.*}".json \
|
||||
--path-prefix "${b%.*}" --proposed-at 2026-09-03T00:00:00Z
|
||||
done
|
||||
.venv/bin/python tools/okf_corpus_run.py \
|
||||
--corpus ~/corpora/okf-telling-20260829/K2/trinn1 \
|
||||
--report ~/corpora/okf-telling-20260829/K2-bundle-20260903-report.md \
|
||||
--bundle ~/corpora/okf-telling-20260829/K2-bundle-20260903 \
|
||||
--ingested-at 2026-09-03T00:00:00Z \
|
||||
--plans-dir ~/corpora/okf-telling-20260829/K2-plans-20260903 \
|
||||
--bundle-id k2-trinn1-20260903 --okf-version 0.2
|
||||
```
|
||||
|
||||
Converter as the harness resolved it: the vendored `pypandoc` binary, version
|
||||
**3.9** — not the host's 3.10.2.
|
||||
|
||||
### The numbers, each with its denominator
|
||||
|
||||
| figure | value |
|
||||
|--------|-------|
|
||||
| `N` (corpus directory file count, computed) | 43 |
|
||||
| merged | 39/43 |
|
||||
| coded rejections | 4/43 (`extractor_unknown` 3, `extractor_empty_pdf` 1) |
|
||||
| concepts | **629** |
|
||||
| concepts carrying `adjudication` | **618/629**, every one of them `proposed` |
|
||||
| body characters, max | 217 472 |
|
||||
| body characters, median | 441 |
|
||||
| concepts over 100 000 characters | 4/629 |
|
||||
| wall time | 784.29 s total, 18.239 s per file |
|
||||
| K1b | `39 + 4 = 43 = N`, run exited `0` |
|
||||
|
||||
The previous bundle, re-measured here rather than quoted: 39 concepts, **0/39**
|
||||
carrying `adjudication`, max body 267 548 characters, no `log.md`.
|
||||
|
||||
**K1b is now recoverable from the bundle alone**, which was the point of §9:
|
||||
|
||||
```
|
||||
python - <<'PY'
|
||||
import re; log=open('.../K2-bundle-20260903/log.md').read()
|
||||
n=int(re.search(r'N = (\d+)',log).group(1))
|
||||
merged=int(re.search(r'merged = (\d+)',log).group(1))
|
||||
codes=[int(m.group(2)) for m in re.finditer(r'`([a-z_]+)`: (\d+)',log)]
|
||||
assert merged+sum(codes)==n; print(merged,'+',sum(codes),'=',n)
|
||||
PY
|
||||
```
|
||||
|
||||
### Two numbers that need their units stated
|
||||
|
||||
The consumer measured **tokens** (max 121 462); the figures above are
|
||||
**characters**, which is what this repository can count without adopting
|
||||
somebody's tokenizer. Their own two numbers imply ≈2.20 characters per token on
|
||||
this corpus. Applied to the maximum here that is ≈98 700 tokens — a **derived**
|
||||
figure, and the only one in this document that is not a direct count.
|
||||
|
||||
### The 11 concepts with no `adjudication`
|
||||
|
||||
They are exactly the 11 documents that proposed zero segments: no plan, so the
|
||||
ordinary one-concept path, so no key. Marking them `proposed` would claim a
|
||||
proposal that was never made. A consumer distinguishes three states — `proposed`,
|
||||
`adjudicated`, and absent — and absent is the honest one here. Reported rather
|
||||
than changed: the wire form is a ratified contract (`docs/plan/office-intake.md`
|
||||
§4–5) and is not this session's to move.
|
||||
|
||||
## Point 4: why the large concepts were not split — and which one still is not
|
||||
|
||||
Not "the proposer refused". It was never run. Given plans, the two documents
|
||||
the consumer named **do** split:
|
||||
|
||||
| document | entries proposed | largest segment (chars) |
|
||||
|----------|------------------|-------------------------|
|
||||
| Bilag 3.1 (was 270 572 chars, one concept) | 34 | 180 604 |
|
||||
| Bilag 1 (was 257 072 chars, one concept) | 18 | 142 667 |
|
||||
|
||||
Both are now below the 100 000-token line. In each the trailing segment still
|
||||
absorbs most of the document, because the headings the mechanical rules find in
|
||||
a PDF are largely table-of-contents lines near the front.
|
||||
|
||||
The largest concept in the new bundle is a **different** document: Bilag 9.1, at
|
||||
217 472 characters, one of the 11 with no proposable structure at all. This is
|
||||
§10's "no declared structure" as Topic 1b measured it — 23 of 33 PDFs carry no
|
||||
outline, and 95 % of the outline entries that do exist are AutoCAD export
|
||||
metadata. The mechanical rules cannot reach it, and nothing here pretends
|
||||
otherwise. Reported, not fixed: Arm C is its own decision.
|
||||
|
||||
## What this does not measure
|
||||
|
||||
- The bundle is not re-measured against the consumption contract's checkable
|
||||
half by `tools/okf_contract_check.py`; that check reads a consumption skill,
|
||||
not a bundle.
|
||||
- `pptx`, `odt` and `rtf` still have denominator **0** in this corpus. Unmeasured,
|
||||
not passing.
|
||||
- Segmentation quality is unchanged. K3 measured 7 of 12 splits as too coarse
|
||||
and that number is a floor; the entries here are `proposed` precisely because
|
||||
no one has judged them.
|
||||
Loading…
Add table
Add a link
Reference in a new issue