open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(k3): round 11, and the corpus that rebuilt byte-identical [skip-docs]

Browse source 
The report for both deliveries, in S7 form, every measurement with its
denominator and an "Honesty limits" section.

Three of the order's own figures did not reproduce and are corrected in section
0 rather than smoothed over: the dominant document takes 22 of 43 places here
and not 23 of 42, the known-positive ranks 8 and not 12, and at k = 8 it takes 7
of 8 places and not 8 of 8. All three divergences point the same way and have one
likely cause -- SK1 measured a frozen export of `6ff18fd` and HEAD is `de9564d`,
with round 10's prefix work in between, which changes the matcher the ranking
reads. This session's numbers are the ones used.

A control caught a defect in this session's OWN measuring query: the first
dominance run reported 2 of 7 fasit prefixes missing from the bundle, and they
were not missing -- the harness read the last segment of a concept id where the
document is the first. Corrected to 0 of 7 missing, and only then were ranks
read.

Records the strong form of "no other file type moved": the whole 43-document
reference corpus rebuilt on this HEAD is byte-identical to the pinned bundle,
`diff -rq` 0 differences, digest 21af4a1aa98315cf... and 453 concepts. The file
census (0 of 86 corpus files are HTML) is an argument that the changed path is
never reached; the rebuild is a measurement of it, and round 10's own trap -- a
green aggregate over a set where the difference cannot appear -- is why the
census alone was not accepted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 01:10:59 +02:00
commit 8e1917946a
1 changed files with 358 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

358
docs/2026-09-10-k3-runde11-html-og-dominans.md Normal file
View file

@ -0,0 +1,358 @@
# K3 round 11 — the type that had never met a document, and the document that took every place
Two holes, both measured outside this repository, both with denominators,
neither a function of corpus size. Both are closed; neither is closed with a
ceiling.
- **A.** `.html` was in `_CORE_EXTRACTORS` and no HTML document could ever be
segmented: **828 of 828** real sections gave 0 plans and exit 2.
- **B.** One source document took **8 of 8** delivered places on a question and
**7 of 8** on the known-positive, on a 3206-concept bundle where it is
**0.87 %** of the concepts.
Both were reproduced here before either was touched, and three of the numbers
the order carried are corrected below.
---
## 0. What was reproduced first, and what did not reproduce
| claim | source | reproduced here |
|---|---|---|
| HTML: 0 plans, N with no boundary, exit 2 | SK1, frozen `6ff18fd` | **yes** — 83 of 83 at the 10 % point, exit 2 |
| markdown control: 83 plans, 343 concepts | SK1 | **yes**, exactly |
| smoke folder 52 md / 26 concepts, 0 of 5 rejected | order | **yes** |
| suite 1535, ruff + format + mypy clean | order | **yes** — 1535 passed, ruff 0.16.6 clean, `ruff format --check` 114 files, mypy 21 files |
| corpus counts 83 / 414 / 828 / 145 / 110 | order | **yes**, `find -type f` and suffix census |
| dominant document takes 23 of 42 places | SK1 | **close, not equal** — 22 of 43 at `k = 50`; see below |
| known-positive at rank 12 | SK1 | **no** — rank 8 at `k = 50` here |
| the document takes 8 of 8 places on the known-positive at `k = 8` | order | **no** — **7 of 8** here |
The three divergences are all in the same direction and have one likely cause:
SK1 measured a frozen export of `6ff18fd`, and HEAD is `de9564d` — two commits
of round 10's prefix work sit between them, and that work changes the matcher
the ranking reads. They are reported rather than smoothed over, and **the
numbers used below are this session's own**.
**A control caught a defect in this session's own measuring query.** The first
run of the dominance harness reported 2 of 7 fasit prefixes missing from the
bundle. They were not missing: the harness read the LAST segment of a concept id
where the document is the FIRST. With the query corrected the control is **0 of
7 missing**, and only then were any ranks read. A rank measured against a fasit
that is not in the bundle is a measurement of nothing.
---
## 1. Delivery A — `.html` (commit `732f84d`)
### The mechanism, read in the source
`_HTMLTextExtractor.text()` was:
```python
return " ".join("".join(self._parts).split())
```
`str.split()` with no argument splits on **all** whitespace, newlines included,
and `" ".join` puts it back together with spaces. Extraction of any HTML file
therefore returned **unconditionally one line**. Every boundary grammar in
`propose` is line-anchored — `_ATX` `^#{1,6}`, `_NUMBERED` `^\d+(\.\d+)+`,
`_TABLE_ROW` `^\s*\|`, `_GRID_RULE` `^\s*\+`, `_OUTLINE` `^\s{0,4}\d{1,2}` — so
on one line at most the first can match, and a match at line 0 opens no
**interior** boundary.
### The form of the fix, and why not the other one
Block tags open lines of their own; `h1`–`h6` open a line carrying the ATX
marker for **their own level**; `br` breaks the line; every other tag stays the
word boundary it already was. **The output grammar is markdown** — the same
markdown the office rows reach the proposer through, because those rows are
converted with `-t markdown` for exactly this reason. So **no HTML-only heading
grammar exists**, which is the same shape of decision as the PDF font reader's
ATX form in round 4.
A flat `#` for every heading was rejected on the mechanism, not on taste: it
would hand `_ATX` three top-level boundaries where the document declares one
section and two subsections.
**Never via the converter.** `.html` stays out of `_PANDOC_FORMATS` because
routing it there adds CVE-2025-51591 (SSRF via an iframe in HTML input),
unpatched in every converter version. The test asserting that exclusion is
untouched and green.
**The block set is wider than the five tags the corpus exercises**, and that is
a decision rather than an oversight. Block versus inline is a property of HTML,
not of one corpus: the measured corpus writes its prose in `p`, `li` and `tr`,
while a `div`-structured page — the ordinary shape of exported HTML — carries
the same prose in containers this corpus never uses, and a set restricted to
what was measured would leave that page collapsing into one line, which **is**
the defect. Adding a line break removes no text and invents no boundary on its
own: a boundary needs a line that MATCHES a grammar.
### Result, with denominators
| point | files | before | after | markdown path |
|---|---|---|---|---|
| 10 % | 83 `.html` | 0 plans, 83 of 83 no boundary, exit 2 | **83 plans, exit 0, 343 concepts** | 83 plans, **343** concepts |
| 50 % | 414 `.html` | 0 plans, exit 2 | **414 plans, exit 0, 1651 concepts** | **1651** |
| 100 % | 828 `.html` | 0 plans, exit 2 | **828 plans, exit 0, 3206 concepts / 6015 md** | **3206** / 6015 |
| coarser cut | 145 `.html` | 145 of 145 no boundary, exit 2 | **145 plans, exit 0, 953 concepts** | not built |
The acceptance bar was ±2 % of the markdown path. The deviation is **0.0 %** at
all three points — the HTML path and the markdown path produce the **same
concept count**, not a similar one. `merged + coded rejections = 828; N = 828`
closes at every point, 0 unreadable, 0 coded rejections.
### Text preservation as an exact invariant
The primary criterion is not a percentage. Strip the ATX markers this extractor
added and the sequence of **non-whitespace characters** must be identical to
what the old extractor returned for the same bytes.
| measure | value |
|---|---|
| files exact | **828 of 828** |
| characters, old vs new (markers stripped) | 1 634 454 vs 1 634 454 — ratio **1.000000** |
| SK1's own bar | ≥ 99.8 % |
| ATX markers added | 7 600 over 828 files |
| lines produced | **31 141**, where the old extractor produced 828 — one per file |
The invariant runs **in the suite over the whole corpus**, not over one
document, behind `OKF_HTML_CORPUS`: the path names a consumer's own export and
this repository is public, so the pointer is an environment variable rather than
a constant in a tracked file. The test asserts a **known-positive first** — the
marker pattern must find something — before its equality result counts.
`_SKIP_TAGS` is unchanged at `{script, style}`. Dropping `nav`/`header`/`footer`
would be a different change with a different guarantee: the exact invariant
above holds only while nothing is dropped, and a quiet widening would hide
exactly how many characters left the document.
### Nothing else moved, and that is measured rather than argued
| check | result |
|---|---|
| HTML files in the K2 corpus | **0 of 86** (66 pdf, 10 docx, 4 xlsx, 2 zip, 2 smc, 2 doc) |
| HTML files in the smoke folder | **0 of 5** |
| smoke bundle before vs after | `diff -r` **empty** — 52 md / 26 concepts, 0 of 5 rejected |
| `okf project` vs `okf build` | `diff -r` **empty** |
| **the whole 43-document reference corpus rebuilt on this HEAD** | **byte-identical to the pinned bundle** — `diff -rq` **0 differences**, digest `21af4a1aa98315cf...` matching the published one, **453** concepts |
| K2 hit@8, both bundles | unmoved by A |
The last row is the strong form of "nothing else moved". The census (0 of 86
files) is an argument that the changed code path is never reached; rebuilding
the entire reference corpus and getting the same bytes is a **measurement** of
it. Round 10's own trap — a green aggregate over a set where the difference
cannot appear — is why the census alone was not accepted as the answer.
### Provenance
`source_units` already routed `.html` through `_line_units`, but the table was
trivial: one line, so every offset in every HTML concept resolved to line 1. The
numbers now mean something, and what they mean is **a line of OUR extraction** —
a block — never a line of the original markup. A test pins a four-physical-line
source file whose extraction is three lines.
### Evidence class
`_EVIDENCE` had five rows, all converter formats, and no `.html` row even though
`.html` is core-supported. It now has one, at **`measured`**, chosen against the
class definitions rather than assumed: `measured` means "someone wrote the
document for their own purposes and we counted what we got", and the 828 files
are a consumer's own export of a real published handbook, produced for their
ingestion and not to exercise this row, with a fasit written before any lookup.
What the class does **not** claim travels with it in the source: one product,
one format, one publisher, and a generator's cut. README's sentence about
`constructed` rows is unchanged, because `.html` did not land there.
### Tests changed on purpose
The order expected three. **One moved.**
| test | verdict |
|---|---|
| `test_html_text_via_htmlparser` | **changed because the behaviour changed** — it asserted the collapsed one-line form, the very form that caused the defect. The half that still holds (inline tags are word boundaries) is still asserted. |
| `test_html_skips_script_and_style` | **unchanged**, re-read: `_SKIP_TAGS` still holds two members and a single block still renders as a single line. |
| `test_htm_is_an_html_alias` | **unchanged**, same reason. |
A test changed because the behaviour changed is legitimate; a test changed to go
green is not. These are the first kind and the third and fourth are neither.
---
## 2. Delivery B — one document taking every place (commit `1e9f38b`)
### The finding, re-measured here
On the 3206-concept bundle, at the default `k = 8`:
| question | fasit delivered at | dominant document's share of the 8 places |
|---|---|---|
| S1 | — | **8 of 8** |
| S2 | rank 1 | 4 of 8 |
| S3 | — | 4 of 8 |
| S4 | rank 3 | 6 of 8 |
| S5 | — | 4 of 8 |
| S6 | — | 5 of 8 |
| known-positive | — | **7 of 8** |
| known-negative | — (correctly) | 3 of 8 |
hit@8 over S1–S6: **2 of 6**. The document is the code's own process overview:
28 of 3206 concepts (**0.87 %**) and 8.0 % of the source characters. SK1 measured
the same 8-of-8 share at **343**, **1651** and **3206** concepts, so this is the
corpus's **composition** — it holds its own table of contents — and not its size.
A split would move the defect, not remove it; every corpus with a contents list,
a project overview or a summary document has the same property.
### Candidate (a): a quota per source document — **carries**
`--source-quota N` caps how many delivered places one `source_file` may take.
It cuts **where the shortlist is cut** (`shortlist = candidates[:k]`), before the
pack and never inside the DP — the DP maximises a sum over a set it is handed, so
a quota expressed there would be a different and slower problem. The freed place
goes to the next candidate, so `k` is still delivered in full.
**Sweep, `N` ∈ {2, 3, 4, off}, three bundles:**
| N | K2 pinned (453) | K2 Arm B (629) | handbook hit@8 (S1–S6) | KP rank | dominant share, S1 / KP / KN |
|---|---|---|---|---|---|
| off | `[1,1,1,1,1,–]` 5 of 6 | `[1,1,1,1,1,–]` 5 of 6 | 2 of 6 | not delivered | 8 / 7 / 3 of 8 |
| **2** | `[1,1,1,1,1,5]` **6 of 6** | `[1,1,1,1,1,5]` **6 of 6** | **4 of 6** | **4** | **2 / 2 / 2 of 8** |
| 3 | `[1,1,1,1,1,7]` 6 of 6 | `[1,1,1,1,1,7]` 6 of 6 | 4 of 6 | 5 | 3 / 3 / 3 of 8 |
| 4 | `[1,1,1,1,1,–]` 5 of 6 | `[1,1,1,1,1,–]` 5 of 6 | — | 6 | 4 / 4 / 3 of 8 |
The band that carries is {2, 3} and it collapses at both ends. **2 rather than 3
on rank alone**: the recovered rows come in at 5 and 4 rather than 7 and 5, and
the dominant share falls further.
**The veto was not merely cleared, it was improved.** The standing condition is
hit@8 5 of 6 on both K2 bundles with no row losing rank. Every one of the five
rank-1 rows is unmoved on both bundles, and the sixth row — which had missed on
**every bundle and every configuration measured until now** — is delivered at
rank 5. The default moved to **2**, with `--no-source-quota` as the opt-out.
**The known-negatives all still reproduce under the new default:**
| control | pinned bundle | Arm B |
|---|---|---|
| `--no-tie-shared-rank` | `[2,…]` — row 1 still falls 1 → 2 | `[1,…]` |
| `--no-stem-prefix` | all five rank-1 rows hold | hold |
| `--no-source-quota` (new) | row 6 falls back to **`None`** | falls back to `None` |
### Candidate (b): `--rarity-weight` — **measured and rejected for this defect**
It was run alone and combined with the quota, on the same bundles and the same
questions.
| configuration | K2 pinned | K2 Arm B | handbook: dominant share on S1 | KP |
|---|---|---|---|---|
| rarity alone | 5 of 6, ranks unmoved | 5 of 6 | **8 of 8 — unchanged** | not delivered |
| quota 3 + rarity | — | — | 3 of 8 | **not delivered** (worse than quota alone) |
The weight does not touch the mechanism: the dominant document still takes every
place on the question it floods, and the combination costs the known-positive
that the quota alone recovers. It stays off, and its default is now measured
against a second defect rather than one.
### The vocabulary stayed closed
`WITHHOLDING_RULES` goes six to seven with `source_quota_exceeded`. It is
published in all three places a consumer can read it — the constant,
`docs/consumption-contract.md` § 5.3, and the **generated** SKILL.md, verified by
reading the generated file rather than the code that writes it. The code is
distinct from `no_lexical_match` on purpose: this is a **diversity** drop, and
folding it into a relevance code would tell a consumer the question reached
nothing in a concept the question in fact reached.
`okf check --skill … --payload …` is **conformant, 0 findings over 15 rules**.
Editing the contract moved the § 7.4 known-positive, which is that coupling
working rather than breaking: 12 563 → **13 238** encoded, 12 227 → **12 893**
raw, delta 336 → **345**, updated in the constant, the instantiated skill and the
shipped example payload.
### The adverse case, named rather than found later
A bundle built from **one** document carries the same `source_file` on every
concept, so a quota applied literally would deliver 2 excerpts where `k` were
asked for — a rule against dominance turned into a rule against small bundles.
The shortlist is topped back up from the best-ranked over-quota candidates, which
makes such a bundle **byte-identical to the quota being off**, and a test holds
that identity.
### The O6 guard, on the reading side
`build_payload`'s signature defaults are now asserted equal to `okf consume`'s
argparse defaults for every same-named parameter. `okf project` shipped exactly
that disagreement for two rounds, and the reading side had no equivalent guard.
---
## 3. Honesty limits
- **The HTML corpus is machine-generated from ONE source by another session.**
It is not 828 documents anyone wrote: the cut follows the source's own `<sec>`
nesting, each file's `<h1>` is a generator's, and a different cut would give
different numbers. The **text** is a real published handbook's; the
**documents** are a rendering of it.
- **One product, one format, one publisher is not generality.** `okf build` over
828 markdown files says nothing about 828 PDFs: SK1 measured 0.0076 s per
document against K2's 19.2, which is an **extraction-cost** difference and not
a scaling one.
- **SK1's hit@k rests on 6 questions, one rater, one fasit**, written by the
session that ran the measurement, from the handbook's own contents list.
- **hit@8 is not neutral with respect to a document quota.** It asks whether the
gold **document** appears among the delivered excerpts, and a quota directly
raises how many distinct documents a payload holds. The claim that survives
that is the *absence of loss*: the five rows already at rank 1 are neutral
with respect to the rule, and none of them moved.
- **The K2 sheet is n = 12 positions**, of which 8 `pdf`, 3 `docx`, 1 `xlsx`, so
every per-type fraction is a fraction of 8, 3 or 1. **Both readings, never
one:** `|F| = 12` gives xlsx 1/1 and `|F| = 11` gives 0/1. The sheet was **not
re-rated this round**; delivery A cannot reach it (0 of 86 corpus files are
HTML, and the whole 43-document corpus rebuilds byte-identical on this HEAD)
and delivery B is a reading-side change that does not enter the sheet at all.
That is why no row can be worse: the bundle the sheet was rated on is the
bundle this HEAD produces, bit for bit.
- **"Delivered" is not "answered."** Every figure above is about what the cut
put in the payload. Whether it answers the question is the skill's judgement,
and the pre-pass decides nothing about the question by construction.
- **Still with the operator, and NOT moved by this round:** position 0's `|F|`,
the `xlsx` fasit's `|F| = 12`, the `pptx` fasit, and routing between bundles
on `--id`. They are blocked on a decision, not on code.
- **Three of this order's own figures did not reproduce** (§ 0), and this
session's numbers are the ones used.
---
## 4. V1 FERDIG — `html`
| | before | after |
|---|---|---|
| documents with a segmentation plan | **0 of 828** | **828 of 828** |
| exit code | 2 (`FAILED — no segmentation plans`) | **0** |
| concepts | 0 | **3206** — the markdown path's count exactly |
| md files | 0 | **6015** |
| conservation identity | never reached | `merged + coded rejections = 828; N = 828` |
| text preserved (exact, non-whitespace) | n/a | **828 of 828**, ratio **1.000000** |
| coarser 145-document cut | 0 of 145 | **145 of 145** |
| evidence class | absent from `_EVIDENCE` | **`measured`**, with its limit stated |
The row this round set out to move is moved, with the denominator it was
measured against.
---
## 5. Acceptance
Run after `git add` of every named file **and** after each commit.
| gate | result |
|---|---|
| `pytest -q` | **1546 passed, 1 skipped** (1535 before; the skip is the corpus invariant without `OKF_HTML_CORPUS`) |
| `ruff check src tests tools` (0.16.6) | clean |
| `ruff format --check .` | clean, 114 files |
| `mypy src` | clean, 21 files |
| smoke folder | 52 md / 26 concepts, 0 of 5 rejected, `diff -r` against the pre-change tree **empty** |
| `okf project` vs `okf build` | `diff -r` **empty** |
| `okf check --skill --payload` | conformant, 0 findings |