open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(k3): round 16, the whole title takes the route precision could not

Browse source 
The report, the CHANGELOG entry under [Unreleased] and the README's ranking
section. No version bump and no tag: v0.8.0 was released this same day.

R761, delivered ranks, before -> after: hit@1/8/50 3/6 - 5/6 - 5/6 -> 6/6 -
6/6 - 6/6 at default k and 3/6 - 5/6 - 6/6 -> 6/6 - 6/6 - 6/6 at --k 50, with
the known-positive at rank 1 on all four and the known-negative still not a
hit. --no-title-covered on the new binary reproduces the old row exactly.

Nothing else moves, measured on the bytes: the pinned K2 bundle and Arm B keep
(1,1,1,1,1,5) with payloads byte-identical on 6 of 6 questions each,
n100/n200/n500 payloads byte-identical on 5 each, the 828-file HTML corpus at
828 plans / 0 unreadable / 6 015 md with diff -rq empty, the five-document
folder diff -r-identical at 26/52, and okf project still byte-equal to okf
build.

README's test count corrected to what this round measures: 1583 collected,
1582 passed, 1 skipped. The published 1575 was the passed count.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 22:26:57 +02:00
commit 4d1b5222b7
4 changed files with 392 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

282
docs/2026-09-10-k3-runde16-hele-tittelen-tar-ruten.md Normal file
View file

@ -0,0 +1,282 @@
# K3 round 16 — a question that accounts for the WHOLE title takes that section first
Round 15's bundle of one standard delivered the answering section at rank 1 on
**3 of 6** scored questions, and none of the reading side's six flags moved that
number. This round decomposes the three misses, falsifies two candidate repairs
on the corpus, and ships one rule: `--title-covered`, on by default, opt-out
`--no-title-covered`.
Commits: `14ae36a` (red test) · `6bb2189` (the rule) · `ff06b92` (the two
guarantees this round states) · this report.
## 0 Deviations from the order, stated before anything is built on them
Every premise the order named was reproduced. Two readings differ, and in both
cases the number below is the one measured here.
| premise | order | measured here | command |
|---|---|---|---|
| HEAD | `02f9876` | `02f9876` | `git log -1` |
| tag `v0.8.0` | exists, annotated, on `4d1f9d3` | exists, on `4d1f9d3` | `git tag --points-at 4d1f9d3` |
| version | `0.8.0` | `0.8.0` | `grep -n '^version' pyproject.toml` |
| remote `main` | `078b7df` | `078b7df` | `git ls-remote origin refs/heads/main` |
| worktree | clean | clean, `uv.lock` untouched | `git status --short` |
| suite at HEAD | 1575 passed, 1 skipped | **1575 passed, 1 skipped**, 95.84 s | `pytest -q` |
| collected at HEAD | 1576 | **1576** | `pytest --collect-only -q` |
| R761 build | rc 0, 23.2 s, 2 737 dirs, 2 761 concepts | rc 0, **25 s**, 2 737 dirs, **2 761** concepts | `okf build inn-xml …` |
| R761 at default `k` | 3/6 · 5/6 · 5/6, KP 1 | **identical**, and S1 rank 4 · S5 not delivered · S6 rank 3 | `hitk_sk2.py` |
| R761 at `--k 50` | 3/6 · 5/6 · 6/6, KP 1 | **identical** | `hitk_sk2.py --k 50` |
**Deviation 1.** The order's table gives S1 rank **4**; that is its rank at
default `k`. At `--k 50` it is **3**. Both are measured here; the order states
only the first and the two are not in conflict.
**Deviation 2, and it is about an INSTRUMENT rather than about the code.** The
sweep harness written for this round ranks CANDIDATES and the published figures
are DELIVERED ranks, so the two do not agree and must not be read as one number.
On the same bytes the harness gives S1 5 · S3 2 · S6 3 where the CLI delivers
4 · 1 · 3: the cut runs after the fusion and removes candidates above the gold.
Every harness figure in § 3 is labelled `candidate rank`; every figure in § 4 and
§ 5 came out of `okf consume` itself.
Everything was measured in frozen `git archive` exports under
`/tmp/k3r16/{export,export2}`, each `uv sync --frozen --extra extract`, using
that export's own `.venv/bin/okf`. `~/repos/vegnormal-okf` and
`~/okf-test/dokumenter` were read and never written.
## 1 The diagnosis, per miss
The fusion has three signals. Two are lexical COVERAGE COUNTS — one per question
token the candidate answers to, over the title-and-id text and over the body.
The third is the document prior.
**The third signal carries no information on this bundle, and that is measured
rather than inferred.** `document_score` is keyed on
`concept_id.split("/", 1)[0]`, which is the source document; R761 is one XML
file, so the prior takes **one distinct value over all 2 761 concepts**. With
`tie_shared_rank` on, every concept receives the same constant from it. The
order's hypothesis is CONFIRMED. Its evidence — that `--no-tie-shared-rank`
drops hit@1 to 0/6 — is confirmed too: a signal that separates nothing still
imposes an alphabetical order when the tie-break is by id.
So on this bundle the ranking is decided by two coverage counts, and the
question is which of them turns each miss over. Decomposed per candidate, with
the question tokens each one answers:
**S1** — «Hvordan er Prosesskoden delt inn i hovedprosesser?», tokens
`(hvordan, prosesskoden, delt, inn, hovedprosesser)`.
| rank | concept | title cov | body cov | body tokens answered | bytes |
|---|---|---|---|---|---|
| 1 | `11/oversikt-over-prosesser` | 1 | 3 | prosesskoden, delt, hovedprosesser | 114 728 |
| 2 | `hovedprosess-81-l-smasser` | 1 | 3 | **hvordan**, prosesskoden, hovedprosesser | 2 103 |
| 3 | `hovedprosess-83-…` | 1 | 3 | — | 14 529 |
| 4 | `hovedprosess-84-betong` | 1 | 3 | **hvordan**, prosesskoden, hovedprosesser | 4 410 |
| **5** | **`2-1/hovedprosesser`** | 1 | **2** | prosesskoden, hovedprosesser | 1 047 |
The body signal turns it over, and the token it turns on is `hvordan` — an
interrogative pronoun. The title signal separates nothing: every one of these
scores 1 and sits in a tie group thousands of concepts wide.
**S5** — «Hvilke krav stilles til armering i betongkonstruksjoner?».
| rank | concept | title cov | body cov | body tokens answered | bytes |
|---|---|---|---|---|---|
| 1 | `84-4/betongst-p` | 1 | **5** | **hvilke**, krav, **stilles**, armering, betongkonstruksjoner | 19 658 |
| 2 | `34-43/armering-av-betonghvelv` | **2** | 3 | krav, armering, betongkonstruksjoner | 442 |
| … | | | | | |
| **12** | **`84-3/armering`** | 1 | 3 | krav, armering, betongkonstruksjoner | 2 759 |
Two mechanisms, not one. The rank-1 concept wins on `hvilke` and `stilles`,
both grammatical; its body is 7x the gold's, and a longer body can only reach
MORE distinct question tokens. The rank-2 concept wins on the TITLE, at
`betonghvelv ~ betongkonstruksjoner` through the four-character stem `betong` —
a longer title can only reach more too.
**S6** — «Hva omfatter inspeksjon av bruer og konstruksjoner?».
| rank | concept | title cov | body cov | body tokens answered | bytes |
|---|---|---|---|---|---|
| 1 | `88-14/enkel-inspeksjon` | 1 | 4 | omfatter, inspeksjon, bruer, **konstruksjoner** | 446 |
| 2 | `81-6/masser-…-konstruksjoner-under-vann` | 1 | 3 | omfatter, bruer, konstruksjoner | 715 |
| **3** | **`88-1/inspeksjon`** | 1 | 3 | omfatter, inspeksjon, bruer | 585 |
Rank 2 is a TIE with the gold on both signals, broken by `concept_id` — the
declared tie-break, doing what it is declared to do because nothing else
separates them.
**The order's reading of the pattern is confirmed and can be stated more
precisely.** The three golds carry the bare term as their title
(`Hovedprosesser`, `Armering`, `Inspeksjon`) and lose to a title that is that
term plus a qualifier, or to a much larger neighbour. The mechanism under all
three is one thing: **both signals measure how much of the QUESTION the
candidate answers, and nothing measures how much of the CANDIDATE the question
accounts for.** Coverage is recall with no precision beside it, so surface area
— a longer title, a longer body — can only help.
What does not fit the order's phrasing: on S6 the gold is not beaten by the
larger neighbour at all, it is beaten by `concept_id` order at an exact tie, and
on S1 the deciding token is a pronoun rather than a qualifier. Both are the same
defect seen from different sides.
**It is not a segmentation miss on this bundle, and the control says so.**
`bundle-xml/<doc>/2-1`, `/84-3`, `/88-1` and `/1` all exist as concept
directories. On the PDF arm (`build/sk2-bundle-default`, read only)
`find . -maxdepth 2 -type d -name '22' / '53' / '1'` are all EMPTY — only
`22-1`, `53-1` … exist — so S3/S4/KP are segmentation misses there. That is the
known-positive proving the presence query can find a difference.
## 2 Two candidate repairs, measured and FELLED
Both were measured on the corpus before the shipped rule was written.
**Length normalisation of the body signal — falsified.** The textbook answer to
"unnormalised coverage favours long documents" is pivoted length normalisation,
`cov / (1 - b + b*len/avg)`. Swept at b = 0.25, 0.5, 0.75, 1.0 over the eight
questions (candidate ranks):
| b | hit@1 | hit@8 | S1 | S3 | S4 | S5 | KP |
|---|---|---|---|---|---|---|---|
| off (baseline) | 2/6 | 5/6 | 5 | 2 | 1 | 12 | 1 |
| 0.25 | 1/6 | 3/6 | 3 | 55 | 72 | 42 | 1 |
| 0.5 | 1/6 | 1/6 | 32 | 90 | 61 | 183 | 1 |
| 1.0 | 0/6 | 0/6 | 59 | 96 | 64 | 200 | **49** |
It collapses at every value including the known-positive. The reason is the
corpus's shape: the median concept holds **22** tokens against a mean of 60, so
thousands of tiny concepts mentioning a term once are promoted over the section
that treats it. Length is not what is wrong with the body signal.
**Title precision as a signal — falsified on the control.** `|Q ∩ T| / |T|` over
the title alone reaches candidate hit@1 5/6 in one arrangement, and takes the
known-positive from rank 1 to **4** every time it does. KP's gold is titled
`Bruksområder for Prosesskoden` — a phrase, one of whose three tokens the
question answers — so a ratio that rewards short titles demotes it. The fasit's
own comment says a KP below rank 1 means the instrument is broken, so this is
not a trade to make.
## 3 Why a PARTITION and not a fourth signal
The order asks for this to be answered rather than assumed, and the arithmetic
answers it.
RRF consumes RANKS ALONE, so one signal contributes at most `1/(RRF_K + 1)`.
With shared ranks, a signal whose positive group has `m` members gives the group
position 1 and everyone else position `m + 1` — so its whole separating power is
`1/61 - 1/(61 + m)`. **A rule that fires on ONE concept of 2 761 is worth
0.00026**, while the body-signal gap it must close on S1 is
`1/61 - 1/74 = 0.0029`, an order of magnitude larger. **A precise rule is worth
LEAST under this fusion, exactly backwards from what precision is for.**
Measured, not just derived. The same title-subset predicate, as a signal:
| form | hit@1 (candidate) | hit@8 | S1 | S5 | S6 | KP |
|---|---|---|---|---|---|---|
| baseline | 2/6 | 5/6 | 5 | 12 | 3 | 1 |
| as a 3rd signal (replacing the dead prior) | 3/6 | 6/6 | 5 | 2 | 2 | 1 |
| as a 4th signal | 3/6 | 6/6 | 5 | 2 | 2 | 1 |
| **as a PARTITION** | **6/6** | **6/6** | **1** | **1** | **1** | **1** |
This is the same measurement `concept_scores` already records for `lookup_hits`:
a fourth RRF signal put the named concept at 26/15/19 of 446/1 133/270 and
delivered none of them, so the lookup became a partition. The rule shipped here
is that shape for the same reason, and the figures above are its own — not
borrowed from that one.
## 4 The rule
`title_covered_hits(concepts, question)` returns the concepts whose EVERY title
token is a token of the question. `concept_scores` reads them first, keeping the
fusion's order inside each part, BELOW the lookup partition.
**By equality, never by shared prefix.** `tokens_match` accepts four shared
leading characters. Under a prefix-matching form the group goes 1 → 6 on S1 and
9 → 31 on S5, and the gold falls to candidate rank 6 with the known-positive to
2. Under equality both hold rank 1. The precedent is `tokens_match`'s own rule
that an identifier matches by equality alone.
**Reads `title` and not the concept id.** An id segment is this library's own
reduction of the title, so counting it would let the same words qualify twice,
and on a single-document bundle the id carries a uuid that is in no question.
**Reach, measured, and it is narrow.** Group size per question on R761:
| S1 | S2 | S3 | S4 | S5 | S6 | KP | KN |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 1 | 1 | 9 | 2 | **0** | **0** |
It fires on 6 of 8 questions there and on **0 of 6** K2 questions and **0 of 15**
N-bundle cells (§ 5). It never fires on the known-negative.
## 5 The row, and every other row measured
**R761, delivered ranks from `okf consume`, 2 761 concepts, one source document:**
| | before, default `k` | after, default `k` | before, `--k 50` | after, `--k 50` |
|---|---|---|---|---|
| hit@1 | **3/6** | **6/6** | **3/6** | **6/6** |
| hit@8 | 5/6 | **6/6** | 5/6 | **6/6** |
| hit@50 | 5/6 | **6/6** | 6/6 | **6/6** |
| KP rank | 1 | **1** | 1 | **1** |
| KN | not a hit | **not a hit**, same top-1 | not a hit | not a hit |
| S1 · S5 · S6 | 4 · – · 3 | **1 · 1 · 1** | 3 · 11 · 3 | **1 · 1 · 1** |
**The known-negative on the same bytes.** `--no-title-covered` on the NEW binary
reproduces the old row exactly — S1 4, S5 not delivered, S6 3, hit 3/6 · 5/6 —
identical row for row to the OLD binary at shipped defaults. The pin names its
own cause.
**Nothing else moves, and it is measured on the bytes:**
| row | denominator | before | after |
|---|---|---|---|
| K2 pinned default bundle, hit@8 ranks | 6 questions, 453 concepts | `(1,1,1,1,1,5)` | **identical**, payloads **byte-identical on 6 of 6** |
| Arm B bundle, hit@8 ranks | 6 questions, 629 concepts | `(1,1,1,1,1,5)` | **identical**, payloads **byte-identical on 6 of 6** |
| `tests/test_default_bundle_pin.py` | 7 tests | 7 passed | **7 passed**, file unedited |
| n100-2023 | 5 questions, 446 concepts | — | payloads **byte-identical**, rule fires 0 of 5 |
| n200-2024 | 5 questions, 1 133 concepts | — | payloads **byte-identical**, rule fires 0 of 5 |
| n500-2024 | 5 questions, 270 concepts | — | payloads **byte-identical**, rule fires 0 of 5 |
| 828-file HTML corpus | 828 documents | 828 plans, 0 unreadable, 6 015 md | **828 plans, 0 unreadable, 6 015 md**; `diff -rq` empty apart from `log.md`'s converter PATH (same pandoc 3.9) |
| `~/okf-test/dokumenter` | 5 documents | 26 concepts / 52 md | **26 / 52**, `diff -r` **EMPTY** |
| `okf project` vs `okf build` | same folder | byte-equal | **byte-equal**, `diff -r` empty |
| suite | — | 1575 passed, 1 skipped | **1582 passed, 1 skipped**, 1583 collected |
| `ruff check src tests tools` · `ruff format --check .` · `mypy src` | ruff 0.16.6 | clean | **clean** (118 files, 21 source files) |
**hit@k on N100/N200/N500 is NOT MEASURED, and the denominator is why:** this
repository holds no gold set for those three bundles — 0 gold sets, not 0 hits.
What is measured there is that the payload does not change.
## 6 Honesty limits
- **N = 6.** hit@k here rests on ONE fasit with SIX scored questions on ONE
corpus from ONE publisher in ONE schema. A move from 3/6 to 4/6 would be one
question; this round moves three. Three questions is not a retrieval
evaluation.
- **The fasit is another repository's and was not touched.** It was set in
`vegnormal-okf` on 2026-09-10 before any lookup ran, and this round read it
and the instrument and wrote to neither.
- **KP and KN are CONTROLS, not results.** Neither enters hit@k. KP holding rank
1 is what says the instrument still counts; KN staying not-a-hit is what says
the rule invents nothing.
- **The PDF arm's zeros on the same questions are a SEGMENTATION miss**, not a
ranking one, and are not comparable with the XML arm's ranks. The presence
check in § 1 is what separates the two.
- **"Solved" means three measured misses were decomposed and the row moved.** It
does not mean a fourth failure shape does not exist. Coverage is still
unnormalised; this rule adds a precision statement about the title and none
about the body.
- **The rule's reach outside R761 is ZERO in everything measured.** "No
regression on K2 and the three N bundles" is really "the rule never fires
there" — 0 of 21 cells — not "it fires and is harmless". A corpus whose
section titles are phrases rather than terms gets nothing from it.
- **S5's group is 9 concepts, all titled `Armering`**, and the fusion's own
order put the right one first. That is one case, not a property: where a
publisher reuses a title, this rule floats all holders and decides nothing
between them.
- **The sweep harness is not the product.** Its candidate ranks and the CLI's
delivered ranks differ by the cut; the falsifications in § 2 and the
arithmetic in § 3 are harness figures, everything in § 4 and § 5 is not.
- **This is the fourth reading-side change that moves a payload with NO bundle
changing.** A consumer pinned to the previous excerpt order needs
`--no-title-covered`. Measured, that consumer is nobody on K2 or the N
bundles: their payloads are byte-identical either way.