open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(propose,consume,profiles,importer): recovery yields to declaration, and 9 % of the corpus that was in no segment

Browse source 
One rule explains every remaining `pdf` miss on the twelve-position reference:
where a document DECLARES headings, Arm D's RECOVERED headings are the whole of
the excess, and every declared one is a unit the reference wants. `--outline-gate`
admits recovery only where the document declares none of its own, plus any one
recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units`
clause 2's own principle moved from voting to admission, and it filters at
ADMISSION so the text a removed mark opened is carried by the mark above it --
the post-filter form scores identically on all twelve positions and loses that
text, which is why only one of them shipped.

`--outline-gate` and `--drop-wrapped-outline` become the package default, one
decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8
alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of
3. Each keeps an explicit opt-out. The bar the move had to clear was not the
reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-,
no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of
12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2.
Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810.

DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum
measures size and a density is diluted by every unit carrying none of the
question, so a document split 1 -> 12 lost its prior by 12. Swept over five
values on 18 rows it is at least as good as the delivered density everywhere and
strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row
on any of four bundles, so it did not solve the knot it was adopted for -- what
did is that the `pdf` gain never needed `--sheet-section-rows`.

`--first-span-from-zero` is off and repairs a measured loss found while chasing
one position's 940 characters: 32 of the 32 documents that get a plan leave the
text above their first concept in no segment -- 159 704 characters, 9.18 % of
the corpus, 45 841 from one document. It changes nothing on the reference. Off
because it moves the first span of essentially every bundle with no hit@8 number
behind it yet.

vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every
concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads`
carries it for the SEGMENTED profiles, read-only, after the emitted pattern
misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2).

vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1,
`bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`,
keyword-only, rendered before any disk mutation, written only when the index is
created -- Door B's mechanism and ordering.

Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md.
Suite 1478 passed (1449 before), ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-09 14:17:00 +02:00
commit 38104b7df5
16 changed files with 1301 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

356
docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md Normal file
View file

@ -0,0 +1,356 @@
# K3 round 6: the outline gate, the sublinear prior, and 9 % of the corpus that was in no segment
Session 116, order `20260909T110525Z-388129821`. Everything below was measured
on this machine on 2026-09-09 against `b01492b`; every figure carries its
denominator, and where a figure disagrees with the one the order carried, the
one measured here is the one that stands and the divergence is named at the
point it occurs.
## 0. What this round found, in one paragraph
One rule explains every remaining `pdf` miss on the twelve-position reference:
where a document DECLARES headings, Arm D's RECOVERED headings are the whole
of the excess. Gating recovery behind declaration, plus re-admitting a single
recovered heading that covers a fifth of its document, takes `pdf` from 2 of 8
to **7 of 8** and the sheet from 5 of 12 to **10 of 12** -- and with D1 and D3
on top, to **11 of 12**, every position but the one that is undefined by
construction. Two things the round was told to expect did not happen. The
sublinear document prior is at least as good as the delivered one on all 18
prior-rank rows, exactly as the order said -- and end to end it moves **not one
hit@8 row on any of three bundles**, so it did not solve the K2 knot and
nothing here claims it did; what solved that knot is that the `pdf` gain does
not need `--sheet-section-rows` at all. And chasing one position's 940 lost
characters found that the loss is not one position: **32 of the 32 documents
that get a plan lose the text above their first concept, 159 704 characters,
9.18 % of the corpus.**
## 1. Reproduction of the starting state
Measured before any code moved.
| claim | measured here | verdict |
|---|---|---|
| `git rev-parse HEAD` | `b01492b` | matches |
| working tree | 0 modified files | matches |
| `pytest -q` | **1449 passed**, 0 failed | matches (the order's two failures are export artefacts, absent in a work tree) |
| `ruff check src tests tools` | clean | matches |
| `mypy src` | clean, 21 files | matches |
| operator folder, no flags | **30** md | matches |
| operator folder, three flags | **52** md | matches |
| `/tmp/k3r3/text43/*.txt` | **39** | matches |
| `/tmp/k3r4/font43/*.txt` | **39** | matches |
| `/tmp/k3r4/k2-default` | **944** md | matches |
| `/tmp/k3r5/k2-f3` | **942** md | matches |
| Arm B bundle | **1108** md | matches |
**The push list in STATE was stale, and the remote says so.** `git ls-remote
origin refs/heads/main` returns `b01492b`, which is HEAD; `v0.6.0` is on the
remote and points at `f6fea13`, one commit behind. Nothing was unpushed. The
tag was not moved and no new one was made.
## 2. The outline gate
### 2.1 The finding
In each of the four missing `pdf` positions the excess consists exclusively of
`rule:outline` entries, while every `rule:heading` entry is a unit the
reference wants. Reproduced here on the F3 plans:
| pos | \|F\| | entries | `rule:heading` | `rule:outline` |
|---|---|---|---|---|
| 1 | 15 | 17 | 14 | **3** |
| 2 | 9 | 9 | **0** | 9 |
| 4 | 1 | 1 | **0** | 1 |
| 6 | 6 | 7 | 6 | **1** |
| 7 | 30 | 33 | 30 | **3** |
| 8 | 8 | 8 | **0** | 8 |
| 9 | 4 | 6 | 4 | **2** |
The two `pdf` cells that already hit -- 2 and 8 -- carry **no** declared
heading at all, and there the recovered numbering is the reference exactly.
**The precedent is the module's own.** `propose.fold_units`' clause 2 already
excludes `RULE_OUTLINE` from voting on the unit level, and its comment names
positions 1, 7 and 9: *"a document whose structure was recovered rather than
declared has no unit level to read"*. G1 is that sentence moved from VOTING to
ADMISSION.
### 2.2 G1 and G2
- **G1:** Arm D's recovery is admitted only where the delivered heading
grammar (`_ATX` and `_NUMBERED`, i.e. `RULE_HEADING`) proposes nothing.
- **G2:** after G1, a recovered heading whose span covers at least
`OUTLINE_SHARE` of the document is re-admitted.
G2 exists because G1 alone lands position 1 on 14 against a reference of 15.
The eight outline candidates in the four gated positions separate cleanly by
span share:
| pos | shares | kept by the reference |
|---|---|---|
| 1 | 0.011 · 0.035 · **0.316** | only the 0.316 |
| 6 | 0.039 | none |
| 7 | 0.006 · 0.025 · 0.004 | none |
| 9 | 0.094 · 0.065 | none |
`OUTLINE_SHARE = 0.20`, chosen the way `OCR_CID_SHARE` was: bounded by an
empty region on one side and by a measured collapse on the other. Swept end to
end, `pdf` is 7 of 8 at 0.10, 0.15, 0.20, 0.25 and 0.30, and **6 of 8 at 0.05**
(position 9's 0.094 candidate enters) and **6 of 8 at 0.35** (position 1's
0.316 candidate leaves). 0.20 is the middle of that plateau and of the empty
region alike. The 0.35 end is measured here and was not in the order.
### 2.3 What it scores
Criterion imported unchanged from round 5's `score5.py`; only its trailing
round-5 driver call is not executed.
| configuration | `pdf` | `docx` | `xlsx` | sheet |
|---|---|---|---|---|
| shipped default | 2 of 8 | 3 of 3 | 0 of 1 | 5 of 12 |
| F3 (default + the three flags) | 3 of 8 | 3 of 3 | 0 of 1 | 6 of 12 |
| default + G1 | **5 of 8** | 3 of 3 | 0 of 1 | 8 of 12 |
| default + `--drop-wrapped-outline` + G1 | **6 of 8** | 3 of 3 | 0 of 1 | 9 of 12 |
| default + `--drop-wrapped-outline` + G1 + G2 | **7 of 8** | 3 of 3 | 0 of 1 | **10 of 12** |
| the row above + D1 + D3, reference re-read to 12 | **7 of 8** | 3 of 3 | **1 of 1** | **11 of 12** |
Every one of the twelve positions except position 0 lands on its reference
count exactly, with distance 0.
**The `pdf` gain does not need `--sheet-section-rows`.** That is the rule with
a measured K2 cost, and the 7-of-8 row does not contain it.
### 2.4 Admission, not post-filtering
The gate removes marks BEFORE spans are closed, so the text a removed mark
opened is carried by the mark above it. Filtering finished entries instead
leaves that text in no segment -- the silent loss this library refuses
elsewhere. Both forms were measured; they agree label for label on all twelve
positions and on the whole corpus reach, and differ only in the spans.
### 2.5 Corpus reach, and two divergences from the order's figures
| configuration | gate fires | plan changes | entries removed | plans |
|---|---|---|---|---|
| default | **25** of 39 | 15 of 39 | **64** of 485 | 32 -> 32 |
| default + `--drop-wrapped-outline` | 25 of 39 | 15 of 39 | 55 of 473 | 32 -> 32 |
| F3 | **27** of 39 | 15 of 39 | **55** of 484 | 32 -> 32 |
The order gave 27 fires and 66 removals for the default. Both divergences are
explained rather than left standing: **27 is F3's fire count** (D1 rescues an
orphaned heading in two documents, so two more documents declare), and **66 is
the G1-ONLY removal count** -- decomposed here, G1 alone removes 66 of 485 on
the default and 57 of 484 on F3, matching the order exactly, while G1+G2
removes 64 and 55 because G2 re-admits two candidates. No figure contradicts
another; two of them named a different configuration than the row they sat in.
## 3. The document prior
### 3.1 The mechanism
`consume.document_scores` returned `total / n` -- a density. A sum grows with
unit count and therefore measures size, which is why the density replaced it;
but a density is `n**0` and is diluted by every unit carrying none of the
question, so a document split from 1 concept into 12 has its prior divided by
12. That is where the segmentation side and the retrieval side compete over
one number.
`DOCUMENT_PRIOR_EXPONENT = 0.5`: `total / n**0.5`, the classical length
normalisation between the two.
### 3.2 The sweep, on the gold document's rank under the prior
6 questions x 3 bundles = 18 rows.
| exponent | Arm B (629) | default (492) | D1/D3 (491) | sum |
|---|---|---|---|---|
| 0.0 (a sum) | 4,1,1,1,1,3 | 7,1,1,1,1,3 | 1,1,1,1,1,3 | 33 |
| 0.25 | 1,1,1,1,1,3 | **2**,1,1,1,1,3 | 1,1,1,1,1,3 | 25 |
| **0.5** | **1,1,1,1,1,3** | **1,1,1,1,1,3** | **1,1,1,1,1,3** | **24** |
| 0.75 | 1,1,1,1,1,3 | 1,1,1,1,1,3 | 1,1,1,1,1,3 | **24** |
| 1.0 (delivered) | 1,**2**,1,**2**,1,3 | 1,1,1,1,1,3 | **2**,1,1,1,1,3 | 27 |
This reproduces the order's table row for row. One addition: **0.75 is
identical to 0.5**, so the plateau is [0.5, 0.75] rather than a point, and 0.5
is its lower edge. 0.5 is chosen for having a name -- length normalisation --
rather than for being the middle of a two-sample plateau, and that is a
weaker reason than the sweep looks.
### 3.3 The end-to-end measurement, which is the one that decides
Both exponents, three bundles, one process, `tools/okf_consume_measure.py`'s
own `hit_rank` over `build_payload`:
| exponent | Arm B (629) | default (492) | D1/D3 (491) |
|---|---|---|---|
| 1.0 (delivered) | 1,1,1,1,1,– · 5 of 6 | 1,1,1,1,1,– · 5 of 6 | **2**,1,1,1,1,– · 5 of 6 |
| 0.5 | 1,1,1,1,1,– · 5 of 6 | 1,1,1,1,1,– · 5 of 6 | **2**,1,1,1,1,– · 5 of 6 |
**The prior changed no row on any bundle.** It fixes the gold document's rank
under the prior (D1/D3 row 1: 2 -> 1) and that improvement does not survive
RRF, which fuses the prior with two other signals. So the order's premise --
that the sqrt prior solves the K2 knot -- is **not** what was measured. It
holds every row, which is the order's own condition for adopting it, and it is
adopted; it does not do what it was adopted for. What actually removes the knot
is section 2.3: the `pdf` gain never needed the rule that caused it.
The prior is not a no-op in general. It reorders the shipped consumption
skill's reference payload on the golden fixture bundle, which was regenerated
in this commit -- 14 lines, same byte count, a different excerpt selection.
## 4. The `xlsx` reference, and a cheaper route to it
Reproduced, all three readings of position 3 under `--sheet-section-rows`:
| reading | U | S | distance | label |
|---|---|---|---|---|
| \|F\| = 11, lead block as S | 11 | 1 | 1 | `for fint` (what rounds 3-5 published) |
| \|F\| = 12, lead block as S | 11 | 1 | **2** | `for grovt` -- worse |
| \|F\| = 12, lead block as a UNIT | 12 | 0 | **0** | **`treffer`** |
So the order is right that both halves must move -- **if `--keep-table-heading`
is off.** Measured here, it need not be: under `--sheet-section-rows
--keep-table-heading` the lead block comes out as `rule:heading` rather than
`rule:table-block`, so the criterion as it stands already counts it as a unit,
and the cell hits at \|F\| = 12 with **the criterion untouched**. D1 does in
the artifact what the second half of the re-read describes in the rater's
rules. The operator's decision is therefore narrower than it looked: it is the
count 11 -> 12, and nothing else.
Nothing published so far depends on the outcome: rounds 3, 4 and 5 all used
the reading "11" and all reported distance 1.
## 5. The text above the first concept
Round 3 recorded one position losing 940 characters under
`--drop-wrapped-outline`. Measured over the whole corpus, both configurations:
| configuration | documents with a plan | documents losing their opening | characters lost | worst single document |
|---|---|---|---|---|
| shipped default | 32 of 39 | **32 of 32** | **159 704** of 1 739 511 (**9.18 %**) | 45 841 |
| proposed default | 32 of 39 | **32 of 32** | 163 804 (9.42 %) | 45 841 |
It is not one position and it is not a footnote. `--first-span-from-zero`
starts the first surviving concept at 0; it adds no boundary and removes none,
and on the twelve-position reference it changes **nothing** (10 of 12 either
way), which is what makes it a pure recovery.
**It is OFF, and the reason is this round's own acceptance criterion.** It
moves the first span of essentially every segmented bundle, and the session's
budget held the K2 builds that measured the outline gate. A default that
changed every bundle's first concept with no hit@8 number behind it would be
the move this round refused to make everywhere else. The measurement it needs
is named in STATE.
## 6. vegnormal-okf FUNN 1 and FUNN 2
Both reproduced against a known positive first, then fixed.
**FUNN 2 -- SPEC section 8's star row.** Confirmed:
`SEGMENTED_OKF_V0_2.index.parse_entry("* [Title](id-x.md) - description")`
returned `None`, so the section 9.2 index walk read it as curated prose and
every concept behind such a row was unreachable. Measured wider than reported:
the star marker fails **with or without** a description, so it is the bullet
character and not the separator. `IndexPolicy.also_reads` now carries the star
form for the SEGMENTED profiles, consulted by `parse_entry` only, after the
emitted pattern misses. Reading a form is not a licence to emit it -- the same
asymmetry `sources` already has. `DEFAULT` and `STRICT_V1` are untouched: they
state contracts owned in other repositories (O2). Index MAINTENANCE still keys
on `entry_pattern` alone, so a foreign row we read is never a row we rewrite.
**FUNN 1 -- Door C's outcome is not a consumable bundle.** Confirmed: the
known positive reads at exit 0, and Door C's own output was refused at exit 1
with `bundle_id_missing`. `import_bundle` now takes
`root_frontmatter_values`, keyword-only, defaulting to `None`, rendered before
any disk mutation and written only when the index is CREATED -- the mechanism
and the ordering Door B already has. The half of the finding that is not a
defect is stated rather than papered over: `DEFAULT` names no root key at all,
so a caller wanting a consumable outcome must also name a profile whose index
policy declares `bundle_id`. Widening `DEFAULT` would be this repository
editing commons' contract.
## 7. What still stands, unchanged
- **The prefix over-matches the other way** (round 3): `undersjøisk` matches
`under`, df 159 of 270. It belongs to `--rarity-weight` and was not touched;
this round changed `document_scores` and not `rarity_weights`.
- `--pdf-headings font` and `font-reserve` stay off, on round 4's and round
5's measurements.
- The `ocr` optional group **cannot be installed in this repository's own
`.venv`**: it is Python 3.14 on macOS x86_64 and `onnxruntime` publishes no
cp314 wheel for that platform. Round 5's `--ocr` figures stand; reproducing
them needs a 3.13 interpreter, which is what was used here.
## 8. Honesty limits
- **n = 12 positions**, stratified 8 `pdf`, 3 `docx`, 1 `xlsx`. Every
per-type fraction is a fraction of 8, 3 or 1. A single `xlsx` cell is one
document, and "1 of 1" is one document agreeing, not a rate.
- **One rater, one reference sheet**, and the sheet is internally
inconsistent: position 1 keeps an outline candidate titled `Vedlegg` while
positions 7 and 9 reject candidates with the **same title, same rule and
same level**. Same word, three documents, two answers. G2 resolves it on a
property the documents actually carry (0.316 against 0.004 and 0.065), so it
does not read the title -- but the inconsistency is the only reason position
1 needed a second half at all, and it is a fact about the reference rather
than about the code.
- **G2's threshold was chosen among 8 candidates in 4 documents.**
- **The prior's exponent was chosen among five values on 18 rows**, and the
plateau it sits at the edge of has two members.
- **n = 6 questions and 18 rows** on the retrieval side; one row misses on
every bundle under every configuration measured.
- The three bundles are the **same 43-document corpus read three ways**, not
three corpora.
- **"Delivered" is not "answered".** hit@8 says a gold document reached the
payload, not that the payload answered the question.
- The K2 figures in section 9 are one build per configuration, not a
distribution.
## 9. The K2 control, and the default move
Two bundles were built from the same corpus in this session, in parallel, each
`rc=0` in about 2 470 s. Digests are `find . -type f | sort | xargs shasum -a
256 | shasum -a 256` from inside the bundle, so they are path-independent; the
figure for the previous default is recomputed the same way here rather than
quoted, because STATE's `bdefa679…` was produced by a different command and the
two are not comparable.
| bundle | flags over the package default | concepts | files | digest | hit@8 | ranks |
|---|---|---|---|---|---|---|
| previous default | — | 492 | 944 | `862116da16e422f6…` | 5 of 6 | 1,1,1,1,1,– |
| **new default** | `--drop-wrapped-outline --outline-gate` | **425** | **810** | `bdf4977ca5a443c4…` | **5 of 6** | **1,1,1,1,1,–** |
| + D1 + D3 | the above `--sheet-section-rows --keep-table-heading` | 436 | 832 | `03b1c352548b0cb6…` | 5 of 6 | **2**,1,1,1,1,– |
**The acceptance criterion is met by the middle row and only by it**: the `pdf`
cells go 2 of 8 to 7 of 8 and no row loses rank 1, on either prior exponent. So
`--outline-gate` and `--drop-wrapped-outline` are the package default from
2026-09-09, each with an explicit opt-out (`--no-outline-gate`,
`--keep-wrapped-outline`), and the two of them off reproduce the 2026-09-08
bytes.
**The cost to a consumer is a re-run, and it is not small**: 492 concepts / 944
files becomes **425 / 810**.
### 9.1 Step 4: the D1/D3 cost is NOT gone, and here is exactly what it is
The order's step 4 asked whether the sublinear prior removes the cost that kept
`--sheet-section-rows` behind a flag. Measured on a bundle built for the
question:
| bundle | gold document's concepts | document-prior rank | excerpt rank |
|---|---|---|---|
| new default | 1 | 1 | **1** |
| + D1 + D3 | **12** | 1 | **2** |
The prior did exactly what it was adopted for and it was not enough. Under
`total/n` the gold document's PRIOR rank fell to 2; under `total/n**0.5` it is
**1 on both bundles** -- the regression the order named is gone at that layer.
The FUSED excerpt rank still falls from 1 to 2, on both exponents, because RRF
consumes ranks from three signals and the other two put another concept first.
So `--sheet-section-rows` and `--keep-table-heading` stay behind flags, `xlsx`
stays 0 of 1 on the default, and the row that loses is **row 1, rank 1 -> 2**,
with the gold document going from 1 concept to 12. That is the whole cost,
named rather than summarised. The 11-of-12 sheet in section 2.3 is reachable
today with two flags; what it is waiting on is a ranker that does not lose rank
1 when a document is split twelve ways -- not another segmentation rule.