open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(cli,consume): the first span, and the fusion that punished fine-graining

Browse source 
Round 6 left two things behind: a measured coverage hole with the repair built
but not the number to move it, and a retrieval regression that kept two
spreadsheet rules off the default. Both are measured; four defaults move.

THE HOLE IS BIGGER THAN THE RULE BUILT FOR IT. Measured over the 39-document
corpus, the pre-move default left 207 435 characters -- 11.92 % -- in no
segment: 163 804 above the first entry, 26 041 between entries, 17 590 after
the last. `--first-span-from-zero` closes the first part entirely, 79 % of the
whole, leaving 43 631 (2.51 %) over 8 of 32 documents. It adds no boundary and
the K2 concept count is identical with and without it (425 = 425); hit@8 holds
[1,1,1,1,1,-] under both tie-breaks and the 12-position reference does not move
one cell.

THE RANKER WAS PUNISHING FINE-GRAINING FOR BEING FINE-GRAINED. Round 6 held
`--sheet-section-rows --keep-table-heading` back because on a bundle built with
them row 1 fell rank 1 -> 2, and ruled out the document prior because its rank
over 39 DOCUMENTS was 1 on both bundles. RRF reads a rank over CONCEPTS:
splitting the gold document 1 -> 12 puts its own twelve concepts in that
signal's whole top tie group, so the one leading the body signal takes position
11, contributing 1/71 where the undivided concept contributed 1/61. A signal
that scored them all EQUALLY still emitted twelve ranks, ordered by concept_id,
and the fusion read alphabetical order as a measurement.

AND THE REPAIR WAS ALREADY IN THE TREE, OFF FOR A COST THAT HAD BEEN REMOVED.
`--tie-shared-rank` shipped off 2026-09-08 because hit@8 fell 5 of 6 to 4 of 6.
Swept over 2 prior exponents x 3 bundles x 6 rows: that fall exists only at
DOCUMENT_PRIOR_EXPONENT 1.0. Round 6 moved the exponent to 0.5 for an unrelated
reason and correctly reported it moved no hit@8 row -- nobody measured the
pair. A flag's "off by measurement" is a measurement of a CONFIGURATION, not a
property of the flag.

Shipped, each with an explicit opt-out and both directions measured rather than
asserted: --no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading reproduces the previous bundle byte for byte (diff -rq,
0 differences), and the no-flag build equals the explicit-flag build.
--no-tie-shared-rank reproduces the previous excerpt order -- the one change
here that alters a payload with no bundle changing.

K2 default: 436 concepts / 832 files, digest 8dff8a8e6c15d2f7..., hit@8 5 of 6
ranks [1,1,1,1,1,-]. The pin now holds its own known-negative on those bytes:
read with --no-tie-shared-rank the same bundle gives [2,1,1,1,1,-], so the
green assertion names its cause. Consumer cost is a re-run: 425/810 -> 436/832
on the reference corpus, 15/30 -> 26/52 on a five-document folder.

pptx and md measured end to end for the first time, on two hand-built
documents: md recovers 3 of 4 declared headings, pptx segments per slide only
where the converter recognised a title placeholder. A converter attribute leaks
into titles ({#slide-N}, {#sheet-1}) on 2 of 810 K2 files and 1 of 30 on a
five-document folder -- not fixed, because a filename is reduced from its title
and the fix renames concept ids a consumer has cited.

Suite 1486 (1478 before), ruff and mypy --strict clean over 21 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 15:40:01 +02:00
commit a364ef4c7d
11 changed files with 768 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

304
docs/2026-09-10-k3-runde7-forste-spenn-og-rangeringen.md Normal file
View file

@ -0,0 +1,304 @@
# K3 round 7: the first span, and the fusion that punished fine-graining
Round 6 shipped the outline gate and left two things behind: a measured
coverage hole it had built the repair for but not the number to move it, and a
retrieval regression that kept two spreadsheet rules off the default. This
round measured both. Three defaults moved and one file type turned out never to
have been measured at all.
Everything here is measured on this repository at `38104b7` plus this round's
changes, with `~/repos/llm-ingestion-okf/.venv/bin/python`, against the cached
rendering `/tmp/k3r3/text43` (39 readable documents) and the criterion in
`/tmp/k3r5/score5.py` imported unchanged. Bundle digests are
`find . -type f | sort | xargs shasum -a 256 | shasum -a 256` from inside the
bundle, so they are path-independent.
## 0. One divergence from the numbers this round was handed
The order states the coverage loss as **159 704 characters, 9.18 % of the
corpus**. Measured here with the shipped default over the same 39 documents,
the text above every plan's first entry is **163 804 characters, 9.42 %** — a
difference of 4 100 characters, 2.5 % of the figure. Both agree on the
denominator (39 documents, 32 with a plan) and on the shape of the finding.
The method used here is in § 1 and the 4 100 are not chased further; where the
two disagree the number in this report is the one measured today.
The same measurement also found more of the hole than the figure it was handed
covers, which is § 1.
## 1. Coverage: the hole is bigger than the first span, and the first span was 79 % of it
Characters lying in NO segment, over the 39-document corpus (1 739 511
characters, 32 documents with a plan):
| configuration | in no segment | share | documents | above first | between | after last |
|---|---|---|---|---|---|---|
| pre-round-7 default | 207 435 | **11.92 %** | 32 of 32 | 163 804 | 26 041 | 17 590 |
| + `--first-span-from-zero` | 43 631 | 2.51 % | 8 of 32 | **0** | 26 041 | 17 590 |
| round 7 default (all three) | 43 631 | **2.51 %** | 8 of 32 | 0 | 26 041 | 17 590 |
`--first-span-from-zero` closes **163 804 of 207 435**, 79 % of the hole, and
it closes the whole of the part that hits every document. What remains is
**43 631 characters, 2.51 % of the corpus, over 8 of 32 documents**, in two
named parts that are NOT the same defect and are not fixed here:
* **26 041 characters between one entry's end and the next one's start.** The
mechanism is § 4's: a `rule:table-block` candidate opens BELOW the heading
line that introduces it, so the heading's own line falls between two spans.
* **17 590 characters after the last entry ends.** Not investigated this round.
Both are written into STATE with their mechanism. Neither is a ceiling; each is
a measurement with a denominator and a named cause.
### 1.1 What it cost to move it
`--first-span-from-zero` adds no boundary, and the measurement says so twice
over. On the 12-position reference it changes **not one cell** — `pdf` 7 of 8,
`docx` 3 of 3, `xlsx` 0 of 1, sheet 10 of 12, identical to the default without
it, per position and per label. On a K2 bundle built with it: **425 concepts /
810 files**, the same count as the default without it, and hit@8 `[1, 1, 1, 1,
1, None]` = 5 of 6 — **under both tie-breaks**, so its acceptance does not
borrow from § 2's move.
## 2. The fusion punished fine-graining, and the repair was already in the tree
### 2.1 The decomposition
Round 6 measured that a K2 bundle built with `--sheet-section-rows
--keep-table-heading` takes row 1 of the hit@8 set from rank 1 to rank 2, and
held the two rules back for it. It also measured that the DOCUMENT-prior rank
is 1 on both bundles, which ruled the prior out — correctly, and that is why
the cause stayed unfound: the prior's rank over 39 DOCUMENTS is not the prior
signal's rank over 425 CONCEPTS, and RRF reads the second.
Per-signal, row 1, gold concept against the concept that takes fused rank 1:
| bundle | gold doc's units | gold: title / body / prior rank | competitor: title / body / prior | fused |
|---|---|---|---|---|
| default (425) | 1 | 6 / 4 / **1** | — (gold IS rank 1) | gold 1 |
| D1+D3 (436) | 12 | 16 / 4 / **11** | 5 / 9 / 13 | gold **2** |
The prior signal on the split bundle scores all twelve of the gold document's
concepts identically (2.8735) and they take positions **1 through 12**. The
one that leads the body signal lands at position 11 and contributes `1/71`
where the undivided concept contributed `1/61`. The competitor is a
one-concept document that leads nothing and is beaten in every signal it can
be compared on — it wins because its evidence is in one unit and the gold's is
spread over twelve, and because the gold's own siblings fill the tie group
ahead of it.
So the order's hypothesis is right in its conclusion and needed one correction
in its mechanism: it is not that twelve concepts are each individually weaker.
It is that a signal which scores them all EQUALLY still emits twelve distinct
ranks, ordered by `concept_id`, and the fusion reads that alphabetical order as
if it were a measurement.
### 2.2 That rule already existed and was off for a cost that had been removed
`--tie-shared-rank` — shipped 2026-09-08, off by default — is exactly this
repair. Its published reason for being off is that hit@8 fell 5 of 6 to 4 of 6.
**That figure is real and it is conditional.** Swept over the prior exponent,
3 bundles x 6 rows:
| exponent | bundle | `tie_shared_rank` off | on |
|---|---|---|---|
| 1.0 | Arm B (629) | `1,1,1,1,1,–` 5/6 | `1,–,1,1,1,–` **4/6** |
| 1.0 | default (425) | `1,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
| 1.0 | D1+D3 (436) | `2,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
| **0.5** | Arm B (629) | `1,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` **5/6** |
| **0.5** | default (425) | `1,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
| **0.5** | D1+D3 (436) | `2,1,1,1,1,–` 5/6 | `1,1,1,1,1,–` 5/6 |
Round 6 moved `DOCUMENT_PRIOR_EXPONENT` from 1.0 to 0.5 for an unrelated
reason and reported, correctly, that it moved no hit@8 row end to end. What
nobody measured is the PAIR: the exponent move removed the only cost keeping
`--tie-shared-rank` off. A rule sat behind a published number that had stopped
being true in the same commit that was reported as changing nothing.
**The lesson is the durable one from this round**: a flag's "off by
measurement" is a measurement of a configuration, not a property of the flag.
When a constant that flag interacts with moves, its default is unmeasured
again — and nothing in the tree said so, because the two decisions live in
different files.
### 2.3 What shipped, and the control in the same measurement
`consume.DEFAULT_TIE_SHARED_RANK = True`, opt-out `--no-tie-shared-rank`. With
it, all three of the round's acceptance conditions hold at once, and the
build-side rules could move:
| bundle | flags over the pre-round-7 default | concepts | files | digest | hit@8 | ranks |
|---|---|---|---|---|---|---|
| pre-round-7 default | — | 425 | 810 | `bdf4977ca5a443c4…` | 5 of 6 | 1,1,1,1,1,– |
| + first span | `--first-span-from-zero` | 425 | 810 | — | 5 of 6 | 1,1,1,1,1,– |
| **round 7 default** | the above `--sheet-section-rows --keep-table-heading` | **436** | **832** | `8dff8a8e6c15d2f7…` | **5 of 6** | **1,1,1,1,1,–** |
| round 7 default, `--no-tie-shared-rank` | (same bytes, read differently) | 436 | 832 | same | 5 of 6 | **2**,1,1,1,1,– |
The last row is the known-negative and it is on the SHIPPED bytes rather than a
fixture: the regression round 6 refused to ship is reproduced on the bundle
that now ships, by turning off the one thing that removes it.
`tests/test_default_bundle_pin.py` holds both rows.
## 3. The reference sheet, and the two readings of `xlsx`
Reproduced before any code moved, criterion unchanged:
| configuration | `pdf` | `docx` | `xlsx` | sheet |
|---|---|---|---|---|
| pre-round-7 default | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 |
| + `--first-span-from-zero` | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 (not one cell moves) |
| + `--sheet-section-rows --keep-table-heading`, `|F|` = 11 | 7 of 8 | 3 of 3 | 0 of 1 | 10 of 12 |
| + the same, `|F|` = 12 | 7 of 8 | 3 of 3 | **1 of 1** | **11 of 12** |
Per position on the default, all at distance 0: pos 1 `treffer`/15, pos 2 /9,
pos 4 /1, pos 6 /6, pos 7 /30, pos 8 /8, pos 9 /4, pos 5, 10, 11 `docx`
`treffer`. Pos 0 is `for grovt` with U=1 and `|F|` undefined. Pos 3 is
`for grovt` with U=0, S=1.
**D-2 is unanswered and this report does not answer it.** `F[3]` was not
touched in the criterion. What the measurement adds is that the choice between
the two readings does not decide whether the rules ship: under `|F|` = 11 the
sheet rules leave the sheet at 10 of 12 and take position 3's distance from
**12 to 1** (U=12 against 11, `for fint`); under `|F|` = 12 they take it to
`treffer`. Neither reading is worse off with them on.
## 4. `pptx` and `md`: the mandate's weakest axis, now measured
`extract.py:76` marks `.pptx` `unmeasured` and the 12-position reference is 8
`pdf`, 3 `docx`, 1 `xlsx` — no `pptx`, no `md`. Two documents with a
hand-known structure were built for this (a markdown file with 1 + 3 declared
headings, and the same content written to `.pptx` through the converter). The
fasit is NOT invented: each document's structure is what its author wrote, and
the question asked is only whether the default recovers it.
**`pptx` extracts, and its segmentation depends on a property of the deck.**
Two decks, both extracted with rc 0 and no rejection code:
| deck | structure | concepts | matches |
|---|---|---|---|
| section slide + 3 content slides | 4 slides | **1** | no |
| 4 title slides | 4 slides | **4** | yes |
The mechanism: the converter emits one ATX heading per slide **that carries a
title placeholder it recognised**, and plain body lines for the rest. In deck 1
only slide 1 got a heading, so the document DECLARES one heading — and the
outline gate, correctly by its own rule, then admits none of Arm D's recovered
ones. The whole deck lands as one concept. This is round 6's principle working
exactly as designed on an input its measurement never contained.
**`md` extracts and loses one of four declared headings**: 3 concepts against
the 4 the document declares. Decomposed below.
**A converter attribute leaks into concept titles, and it is not a `pptx`
curiosity.** Every `pptx` concept measured carries `{#slide-N}` in its `title`,
and on real corpora the same leak appears as `{#sheet-1}`: **2 of 810** files
on the previous default K2 bundle, **2 of 1108** on Arm B, and **1 of 30** on
the operator's own test folder. Because a concept's FILENAME is reduced from
its title, the leak is in a concept id a consumer has already cited by name.
Not fixed here — see § 6.
## 5. § 4's `md` defect is § 1's remaining 26 041 characters
The markdown document declares `# Prosjektbeskrivelse` and `## 1`, `## 2`,
`## 3`. Candidates, decomposed:
* with `--no-unit-fold`: four candidates, and the fourth is
`rule:table-block` starting at 503 — **not** `rule:heading` at 489. The
heading line `## 3 Prising` is 14 characters that belong to no segment.
* with the fold on (the default): three candidates. The table block, having no
heading of its own, is folded into section 2, whose span now runs to the end
of the file. The boundary is gone.
The root cause is in the first step, not the fold: **a table-block candidate
displaces a heading the document DECLARED**, and opens below it. That is the
same principle round 6 established for Arm D against declared headings —
recovery yields to declaration — not yet applied between the table rule and the
heading rule.
Reach over the 39-document corpus, so this is a measurement and not an
anecdote: **5 of 39** documents carry a table-block candidate at all; the fold
changes the entry count on **17 of 39**, removing **198** entries; and with the
fold off, **18 527 characters over 15 of 39 documents** sit between one
candidate's end and the next one's start. Under the shipped default that
between-gap is **26 041 characters**.
Not fixed this round: it changes what the proposer emits on 5 of 39 documents,
so it needs its own K2 build and its own hit@8 read, which is one full arm's
worth of acceptance. It is in STATE with this mechanism and these numbers.
## 6. What did NOT move, and why
* **The `{#…}` leak.** The fix is small; the consequence is not. A concept's
filename is reduced from its title, so stripping the attribute RENAMES the
affected concepts — including one an external consumer has already cited by
its full id in writing. Renaming a concept id another repository names is not
this round's call to make; it is a question, with the exposure measured (2 of
810, 2 of 1108, 1 of 30).
* **`MIN_SHARED_PREFIX` and the prefix over-match.** Untouched, and
deliberately: § 2 changes the FUSION and the rarity weight changes a signal's
scores. Moving both would measure two changes with one number. The df figure
it rests on (159 of 270) is round 5's and is still not re-measured.
* **The remaining 17 590 characters after the last entry.** Named, not
investigated.
* **`--pdf-headings`, `--ocr`, `--rarity-weight`, `--cost-vocabulary`,
`--reserve-top-rank`, `--withheld-titles`, `--max-segment-chars`.** All
unchanged and all still off.
## 7. Cost to a consumer
A re-run, and it is not small. The 43-document reference corpus:
* **629 concepts / 1 108 files** (Arm B, delivered 2026-09-03)
* **492 / 944** after the 2026-09-08 move
* **425 / 810** after the 2026-09-09 move
* **436 / 832** after this one — digest `8dff8a8e6c15d2f7…`, default flags,
default epoch stamp.
On the operator's own five-document test folder the same move is **15 concepts
/ 30 files → 26 / 52**.
Every moved default has an explicit opt-out, and the opt-outs were measured
rather than asserted: `--no-first-span-from-zero --no-sheet-section-rows
--no-keep-table-heading` reproduces the pre-round-7 bundle **byte for byte**
(`diff -rq`, 0 differences), and `--no-tie-shared-rank` reproduces the previous
excerpt order.
The reading-side change is the one to flag hardest, because it has no bundle to
diff: `okf consume` now returns a different excerpt ORDER on any bundle with a
tied signal, without any bundle changing. `--no-tie-shared-rank` is the way
back.
## 8. Honesty limits
* **n = 12 positions on the reference sheet**, stratified 8 `pdf`, 3 `docx`,
1 `xlsx`. Every per-type fraction here is a fraction of 8, 3 or 1. A single
`xlsx` position carries the entire `xlsx` claim.
* **n = 6 questions on the retrieval side, one rater, one gold set.** Row 6
misses on every bundle and every configuration measured so far, so "5 of 6"
has always been "5 of 5 reachable rows plus one that no configuration has
reached".
* **Every bundle here is the SAME 43-document corpus read again.** "General"
in this report means "holds across configurations of one corpus", never
"holds across corpora". The `pptx`/`md` documents in § 4 are the only inputs
measured this round that are not from it, and they were written for it.
* **`pptx`, `md`, `odt` and `rtf` remain outside the reference sheet.** § 4
measures two hand-built documents, which is more than zero and is not a
fasit. `odt` and `rtf` are still `unmeasured` with no document behind them
at all.
* **`|F|` for position 0 is undefined and position 3's is unanswered.** Both
are operator decisions with the numbers already delivered; neither is
assumed here.
* **"Delivered" is not "answered".** hit@8 measures whether a concept from the
gold document reaches the payload, not whether the payload answers the
question. The last measurement of that, on 3 bundles, was rank 1 of 8 on 3 of
3 and a correct answer on 1 of 3.
* **The synthetic adverse case for § 2 is recorded, not hidden.** On a
30-concept fixture where one signal separates and two do not, shared ranks
move a gold from rank 18 to rank 30. It is in
`tests/test_okf_consume.py` with that number. Nothing like it appears in the
three real bundles, which is an absence over three corpora, not a proof.
* **`docs/2026-09-08-sjeldenhetsvekt.md`'s figures were measured under the
pre-2026-09-10 tie-break** and are not re-measured here. On one fixture the
change takes the weight's gold from fused rank 18 to rank 1, so at least one
of that report's numbers no longer describes the default.
* **The 4 100-character divergence in § 0 is unexplained.**