open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(k3): round 17, a covered title stops beneath a title that answers more

Browse source 
The diagnosis, the eight forms measured and the rows that did not move.
What separates the known-negative from the standard's six questions is not
the group size, the title length or the share of the question: it is that the
title directly above the covered concept shares 3 question tokens against the
covered title's 1, while on the standard no title above a covered concept ever
shares more than it holds.

(a) title length sells hit@1 back to 3/6; (b) question share holds only for
1/9 < s <= 1/6; (c) and (d) cannot act on a group of one; (e) ships. The one
shape still read wrong (a constructed shortening of the known-negative) is
the same shape as a scored question with the opposite answer, and the form
that fixes it takes that question from rank 1 to 3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-11 01:45:53 +02:00
commit f5a002e4ed
1 changed files with 314 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

314
docs/2026-09-11-k3-runde17-dekningen-stopper-ved-en-bredere-tittel.md Normal file
View file

@ -0,0 +1,314 @@
# K3 round 17 — a covered title stops beneath a title that answers more of the question
Round 16's `--title-covered` took the 2 761-concept bundle of one standard from
hit@1 **3 of 6** to **6 of 6**. Re-measured on another bundle, it failed a
constructed known-negative: a short, generic one-token title that a long
question happens to contain took rank 1 from the section the question names.
This round decomposes why the partition hurt there and helped on the standard,
measures the four candidate repairs the order named plus four more, and ships
one form under the same flag. The standard stays at 6 of 6 at both `k`, the
known-negative returns to rank 1, and nothing else moves one byte.
Commits: `8cc2c54` (red test) · `e15de71` (the rule) · this report.
## 0 Deviations from the order, stated before anything is built on them
Every premise the order named was reproduced. Where a number differs, the
number below is the one measured here.
| premise | order | measured here | command |
|---|---|---|---|
| HEAD | `7cca9e0` | `7cca9e0` | `git rev-parse HEAD` |
| tags | `v0.8.0` on `4d1f9d3`, `v0.8.1` on `3daf983` | same; none points at HEAD | `git tag -l 'v0.8*'`, `git rev-list -n1`, `git tag --points-at HEAD` |
| version | `0.8.1` | `0.8.1` | `grep -n '^version' pyproject.toml` |
| remote `main` | unpushed (STATE: `078b7df`) | **`7cca9e0`, and the remote carries both `v0.8.0` and `v0.8.1`** | `git ls-remote origin refs/heads/main`, `git ls-remote --tags origin 'v0.8*'` |
| worktree | clean | clean at start; **` M uv.lock` appeared mid-session** | `git status --short` |
| suite at HEAD | 1593 passed, 1 skipped | **1593 passed, 1 skipped**, 90.5 s | `pytest -q` |
| `len(RULES)` | 16, at l. 434 | **16**, the tuple at **l. 519** | `python -c 'print(len(contract_check.RULES))'` |
| gates | ruff 0.16.6 clean, mypy clean | same | `ruff --version`, `ruff check`, `ruff format --check`, `mypy src` |
| known-negative bundle | rc 0, 2.91 s, 0 of 5 rejected, 52 md, 4 document dirs, 20 concept dirs, 26 concepts | rc 0, **15 s**, 0 of 5, 52, 4, 20, 26 | `okf build … --bundle-id t --okf-version 0.2` |
| known-negative | before rank 1 (rival rank 2), after rank 2, ranks 3-8 identical | **identical**, row for row | `okf consume`, with and without `--no-title-covered` |
| question tokens | 9, `er` and `og` dropped | **9**, the same nine | `normalise` |
| `title_covered_hits` | exactly one concept, one title token | **one**, one token | — |
| R761 build | ~23 s, 2 737 dirs, 2 761 concepts | **16 s**, 2 737, 2 761 | `okf build` |
| R761 before / after | 3/6·5/6·5/6 → 6/6·6/6·6/6 at default `k`; 3/6·5/6·6/6 → 6/6·6/6·6/6 at `--k 50` | **identical**, KP rank 1 at both | `hitk_sk2.py` |
| R761 group sizes | 1, 1, 1, 1, 9, 2, 0, 0 | **identical** | § 1 |
**Deviation 1, remote state.** STATE described `main` and both tags as
unpushed; the remote holds `7cca9e0` and both tags. The push list at the end of
this round is therefore the round's own three commits and nothing older.
**Deviation 2, `uv.lock`.** Mid-session the tracked lockfile showed one changed
line: the project's own version `0.7.0` → `0.8.1`. No command in this session
ran `uv` against the repository (the only `uv sync --frozen` ran inside a
scratch export), so its source is not identified. It was reverted with
`git checkout uv.lock` before any commit. The fact under it is real and outside
this round: the tracked `uv.lock` records the package at `0.7.0` while
`pyproject.toml` says `0.8.1`, so any non-frozen `uv` command rewrites it.
**Deviation 3, wall time.** The known-negative build took 15 s here against
2.91 s; the R761 build 16 s against ~23 s. Neither enters any result.
Everything below was measured in frozen `git archive` exports — `7cca9e0` under
`/tmp/k3r17/head`, `e15de71` under `/tmp/k3r17/new`, each
`uv sync --frozen --extra extract` — or through a scratch harness that imports
one of those `src` trees. `~/repos/vegnormal-okf`, its fasit, its instrument,
its XML source and `~/okf-test/dokumenter` were read and never written.
## 1 The diagnosis: what makes the partition harmful there and useful here
The harness reads the fusion's own order (`concept_scores` with
`title_covered=False, lookup=False`) and reports, for every concept in the
covered group, its title's token count, the share of the QUESTION that title
accounts for, the concept's rank in the fusion, and the largest number of
question tokens any title ABOVE it shares by equality. **Ranks in this table
are fusion (candidate) ranks, not delivered ranks.**
| case | question tokens | group | covered title tokens | share of question | fusion rank of covered | most question tokens a title above it shares | fusion rank of the right answer | K3-16 did |
|---|---|---|---|---|---|---|---|---|
| **known-negative** | 9 | **1** | **1** | **0.111** | 2 | **3** | 1 (10-token title) | **harm: 2 → 1** |
| P1 | 6 | 1 | 1 | 0.167 | 5 | 0 | 5 (is the covered one) | help: 5 → 1 |
| P2 | 6 | 0 | — | — | — | — | 1 | nothing |
| P3 | 6 | 1 | 1 | 0.167 | 3 | 0 | 3 (is the covered one) | help: 3 → 1 |
| R761 S1 | 5 | 1 | 1 | 0.2 | 5 | 0 | 5 (covered) | help |
| R761 S2 | 8 | 1 | 2 | 0.25 | 1 | 0 | 1 (covered) | nothing |
| R761 S3 | 5 | 1 | 2 | 0.4 | 2 | 0 | 2 (covered) | help |
| R761 S4 | 6 | 1 | 1 | 0.167 | 1 | 0 | 1 (covered) | nothing |
| R761 S5 | 6 | **9** | 1 each | 0.167 | 12, 27, 31, 71, 73, 89, 120, 126, 132 | 1 | 12 (covered, first of the nine) | help |
| R761 S6 | 5 | 2 | 1 each | 0.2 | 3, 29 | 1 | 3 (covered, first of the two) | help |
| R761 KP | 22 | 0 | — | — | — | — | 1 | nothing |
| R761 KN | 7 | 0 | — | — | — | — | — | nothing |
**Three properties do NOT separate the harmful case from the useful ones.**
The group size is 1 in the known-negative and 1 on four of the standard's six
questions. The title is one token in the known-negative and one token on S1,
S4, S5 and S6. The share of the question is 0.111 against 0.167-0.2 — a gap
of ONE question token of length, since a one-token title's share is `1/|Q|`.
**One property does.** In every useful case, no title above the covered
concept shares MORE question tokens than the covered title holds (0 or 1
against 1 or 2). In the known-negative the title directly above shares
**three** (`ritb`, `ns3935`, `prosjektering`) against the covered title's one.
The partition states the covered title's PRECISION — it says nothing the
question did not ask — and round 16 let that claim override the fusion without
ever comparing it with what the title it overrides answers. The fusion had it
right; the partition overwrote it.
**It is not a segmentation miss, and the presence check says so.** Both
concepts of the known-negative are concept directories of their own and both
are delivered in every run of this round, at ranks 1 and 2 — only their order
moves. On R761 the answering directories `2-1`, `84-3`, `88-1`, `1`, `8-4`,
`22` and `53` each exist once, and the control `99-9` does not, so the query
can find an absence.
## 2 Candidate repairs, measured against each other
Every rank here is a DELIVERED rank from `build_payload`, with the partition
replaced by the variant and everything else — the lookup partition, the cut,
the quota — the shipped code. The harness was validated first: its
re-implementation of round 16 is byte-identical to the shipped code on all 8
R761 payloads and all 5 known-negative-bundle payloads; its "off" is
byte-identical to `okf consume --no-title-covered` on the known-negative's
payload and reproduces the pre-0.8.1 R761 row rank for rank.
The known-negative bundle's four questions, verbatim, the three probes written
and their answers fixed at 22:55:36Z on 2026-09-10 before any of them was run:
- **KN** (the order's, verbatim): «Hva er RITB sitt ansvar etter NS3935 gjennom
prosjektering og prøvedrift?» — the 10-token section it names.
- **P1**: «Hvilken opplæring skal driftspersonellet få før idriftsettelse?» —
the one-token section on training.
- **P2**: «Hvordan skal utstyr og komponenter merkes i anlegget?» — the
one-token section on labelling.
- **P3**: «Hvilke serviceavtaler må entreprenøren tilby etter prøvedriften?» —
the one-token section on service agreements.
- **KN-short** (a constructed shortening of KN, a robustness row for (b) and
NOT one of the four): «Hva er RITB sitt ansvar i prøvedrift?» — the same
answer as KN.
The answers are named here by form only; the documents are the operator's, and
their titles stay out of a tracked file.
| form | KN | P1 | P2 | P3 | R761 hit@1/8/50, default `k` | R761, `--k 50` | KP | what it adds | KN-short |
|---|---|---|---|---|---|---|---|---|---|
| off (pre-0.8.1) | 1 | 5 | 1 | 3 | 3/5/5 | 3/5/6 | 1 | — | 1 |
| round 16 (0.8.1) | **2** | 1 | 1 | 1 | 6/6/6 | 6/6/6 | 1 | — | 2 |
| (a) title ≥ 2 tokens | 1 | 5 | 1 | 3 | **3/5/5** | **3/5/6** | 1 | a constant | 1 |
| (a) title ≥ 3 tokens | 1 | 5 | 1 | 3 | **3/5/5** | **3/5/6** | 1 | a constant | 1 |
| (b) share ≥ 0.125 | 1 | 1 | 1 | 1 | 6/6/6 | 6/6/6 | 1 | a constant | 2 |
| (b) share ≥ 0.15 | 1 | 1 | 1 | 1 | 6/6/6 | 6/6/6 | 1 | a constant | 2 |
| (b) share ≥ 0.2 | 1 | 5 | 1 | 3 | **5/5/5** (S5 lost) | **5/5/6** | 1 | a constant | 2 |
| (b) share ≥ 0.34 | 1 | 5 | 1 | 3 | **3/5/5** | **3/5/6** | 1 | a constant | 1 |
| (b) share ≥ 0.5 | 1 | 5 | 1 | 3 | **3/5/5** | **3/5/6** | 1 | a constant | 1 |
| (c) longest title first in the group | **2** | 1 | 1 | 1 | 6/6/6 | 6/6/6 | 1 | nothing | 2 |
| (d) title must hold a non-stop-word | **2** | 1 | 1 | 1 | 6/6/6 | 6/6/6 | 1 | **a closed stop list: new contract surface, new upkeep** | 2 |
| **(e) rise until blocked — SHIPPED** | **1** | **1** | **1** | **1** | **6/6/6** | **6/6/6** | **1** | **nothing** | 2 |
| (e) blocked on `≥` instead of `>` | 1 | 1 | 1 | 1 | **4/6/6** (S5 3, S6 3) | **4/6/6** (S5 2, S6 3) | 1 | — | 1 |
| (e) blocker counted by shared prefix | 1 | 1 | 1 | 1 | **5/6/6** (S5 3) | **5/6/6** (S5 2) | 1 | — | 1 |
| (f) blocked by any question token the covered title lacks | 1 | 1 | 1 | 1 | **5/6/6** (S6 3) | **5/6/6** (S6 3) | 1 | nothing | 1 |
| (g) global gate: no title anywhere shares more | 1 | 1 | 1 | 1 | 6/6/6 | 6/6/6 | 1 | nothing | 2 |
Read per candidate, each component alone first:
- **(a) felled.** Every answering title on the standard that the rule lifts is
one token (`Hovedprosesser`, `Armering`, `Inspeksjon`), so excluding
one-token titles sells hit@1 straight back to 3 of 6, at 2 and at 3 tokens
alike. It also loses P1 and P3.
- **(b) holds every measured row at 0.125 and 0.15, and is not shipped.** It
is a different quantity from the one `tokens_match`' docstring records as
felled in round 10 — that was the share of ONE question word a shared prefix
covered; this is the share of the whole question a title accounts for — and
it does not fall on the same row. It falls on arithmetic: a one-token
title's share is `1/|Q|`, so the rule admits it exactly while the question
has at most `1/s` tokens. The working band is `1/9 < s ≤ 1/6`, bounded by
the known-negative's nine tokens and S5's six — one question token wide, and
set by how many words the asker used rather than by anything about the
titles. At 0.2 S5 falls out of the cut entirely. KN-short, the same question
in five tokens, is admitted at every working value.
- **(c) cannot act.** The known-negative's group is ONE concept; there is no
inside to order. On S5 all nine members hold one token, so it reorders
nothing there either.
- **(d) cannot act, and costs.** No title involved is a function word, so a
closed list — measured with one of 33 Norwegian and English function words —
changes no row. It would add a vocabulary to the contract and to maintenance
for no measured effect.
- **(e) holds every row the order named.** Its two known-negatives say why each
of its choices is the one taken: blocking on EQUAL overlap takes S5 and S6
(an `Armering av …` or `Enkel inspeksjon` above the answer shares exactly
the one token the answer holds), and counting the blocker by shared prefix
takes S5 (`betonghvelv` reaches `betongkonstruksjoner` through the word
`betong`). Strict, and by equality — the matcher the covered group itself is
read with.
- **(f) and (g)** are the two neighbours of (e), measured to locate it. (f)
compares token SETS instead of counts and fixes KN-short at the price of S6;
§ 4 says why no title-only rule has both. (g) scores identically to (e) on
every row.
**Why (e) over (b) and (g), which score the same.** The choice rests on a
property, not on a number, and it is stated as such. (b) needs a constant whose
working band is one question token wide; (e) needs none and cannot see a word
that no title holds — held by `test_words_no_title_holds_do_not_change_what_the_rule_does`.
(g) switches the rule off whenever ANY title in the bundle shares more question
tokens, including one the fusion ranked far below the covered concept and that
the partition would never have passed; (e) compares a covered concept only
with the concepts it would move over, so it overrides the fusion exactly where
the title evidence dominates and nowhere else. That (g) would lose a row
somewhere (e) holds is an argument here, not a measurement.
## 3 The rule
`concept_scores`, under the same `--title-covered` / `--no-title-covered` flag,
no new parameter, no new constant:
> a covered concept RISES through the fusion's order and stops beneath the
> first concept whose title answers MORE question tokens, by equality, than the
> covered title holds — or beneath a covered concept the fusion put above it.
- **With nothing above it answering more, it reaches the top exactly where
round 16 put it.** That is why the standard's payloads do not move: in all 15
covered-group memberships over its eight questions (1+1+1+1+9+2), no title
above shares more than the covered title holds.
- **The lookup partition still lands above it**, unchanged.
- **Covered concepts keep the fusion's order among themselves**, as in round 16.
- **Nothing is withheld by it.** It reorders; it drops nothing, so no code joins
`WITHHOLDING_RULES` — SS 5.3's rule ("a drop with no rule is the silent cut")
is about drops, and there are none. The payload's § 8 shape and § 5
denominators are untouched: the same keys, the same counting. The contract
binds the payload, not the ranking (§ 10), and this is ranking.
- **0.8.1's unbounded order is reproducible by no flag.** It differs from this
one only where a covered concept has such a title above it: 1 of the 4
ordered questions on the known-negative bundle, 0 of 8 on the standard, 0 of
27 anywhere else measured (§ 4). `--no-title-covered` still reproduces the
pre-0.8.1 order, measured: the fix's export with the opt-out gives the
standard's old row digit for digit, 3/6·5/6·5/6, S1 4, S5 not delivered, S6 3.
## 4 The row, and every other row measured
**The standard (R761 XML, 2 761 concepts, one source document), delivered
ranks from `okf consume` through `hitk_sk2.py`, run from the `e15de71` export:**
| | off (pre-0.8.1) | round 16 (0.8.1) | round 17 |
|---|---|---|---|
| hit@1/8/50, default `k` | 3/6 · 5/6 · 5/6 | 6/6 · 6/6 · 6/6 | **6/6 · 6/6 · 6/6** |
| hit@1/8/50, `--k 50` | 3/6 · 5/6 · 6/6 | 6/6 · 6/6 · 6/6 | **6/6 · 6/6 · 6/6** |
| KP rank, both `k` | 1 | 1 | **1** |
| KN | not a hit | not a hit | **not a hit**, same top-1 |
| S1 · S5 · S6, default `k` | 4 · – · 3 | 1 · 1 · 1 | **1 · 1 · 1** |
| payloads identical to 0.8.1 | — | — | **8 of 8 at default `k`, 8 of 8 at `--k 50`** |
**The known-negative bundle (26 concepts, five documents), delivered ranks:**
| question | off (pre-0.8.1) | round 16 (0.8.1) | round 17 |
|---|---|---|---|
| KN | 1 | **2** | **1** — payload equal to `--no-title-covered`'s |
| P1 | 5 | 1 | **1** |
| P2 | 1 | 1 | **1** (the rule does not fire) |
| P3 | 3 | 1 | **1** |
| KN-short (robustness, not counted) | 1 | 2 | 2 |
**Nothing else moves, measured on the bytes:**
| row | denominator | result |
|---|---|---|
| K2 pinned default bundle | 6 questions, 453 concepts | payloads **identical 6 of 6**; group size 0 on 6 of 6 |
| `tests/test_default_bundle_pin.py` | 7 tests | **7 passed**, `EXPECTED_RANKS = (1, 1, 1, 1, 1, 5)`, file untouched |
| Arm B bundle | 6 questions, 629 concepts | payloads **identical 6 of 6**; group 0 on 6 |
| n100-2023 | 5 questions (round 16's), 446 concepts | **identical 5 of 5**; group 0 on 5 |
| n200-2024 | 5 questions, 1 133 concepts | **identical 5 of 5**; group 0 on 5 |
| n500-2024 | 5 questions, 270 concepts | **identical 5 of 5**; group 0 on 5 |
| 828-file HTML corpus | 828 documents | **828 plans, 0 of 828 rejected, 6 015 md**; `diff -r` **empty**, old and new source on one interpreter |
| `~/okf-test/dokumenter` | 5 documents | **26 concepts / 52 md**; `diff -r` **empty**, old and new source on one interpreter (built in two venvs, one line differs: `log.md`'s converter path) |
| `okf project` vs `okf build` | same folder, `--id d` | **byte-equal**, `diff -r` empty, 52 md |
| suite, working tree | — | 1593 passed, 1 skipped → **1600 passed, 1 skipped** (7 new) |
| suite, frozen export of `e15de71` | — | 1591 passed, 9 skipped, **1 failed** = `test_segmented_golden.py::test_the_four_existing_goldens_are_untouched`, the known export artefact (`git diff` with no `.git`); the 9 skips are the pin's 7 (gold set is local-only), the gold-set test, and `OKF_HTML_CORPUS` |
| gates | ruff 0.16.6 | `ruff check src tests tools` clean · `ruff format --check .` 120 files clean · `mypy src` clean, 21 files |
| `okf check` | `len(RULES)` | **16**, no rule added, `contract_check.py` untouched |
**hit@k on N100/N200/N500 is NOT MEASURED**, as in round 16: this repository
holds no gold set for them — 0 gold sets, not 0 hits. What is measured there is
that the payload does not change.
**The one shape still read wrong, and why no title-only rule has it.**
KN-short stays at rank 2, as under round 16. It shares ONE question token with
each of the two titles, and different tokens (`ritb` with the named section,
`prøvedrift` with the covered one), so under (e) neither title answers more
and precision decides. R761 S6 is the same shape with the opposite answer: the
title above the right one shares `konstruksjoner`, the right one's title
`inspeksjon` — one each, different tokens — and there the covered title IS the
answer. Form (f), which blocks on any question token the covered title lacks,
fixes KN-short and takes S6 from rank 1 to 3; (e) keeps S6 and leaves KN-short.
Reading titles alone the two are indistinguishable. The trade is explicit:
**(f) would buy KN-short, a row this round constructed, for S6's hit@1 on the
row the order requires to stand**; (e) was chosen. KN-short's answer was fixed
before the run, and its content is ambiguous as well: the covered section's
own body names the same role four times.
## 5 Honesty limits
- **N = 6 on the standard.** hit@k there rests on ONE fasit with SIX scored
questions on ONE corpus from ONE publisher in ONE schema; one question is
17 percentage points. This round holds that row; it does not add evidence
for it.
- **The known-negative bundle has 26 concepts and four questions, three of
them mine**, written and answered by the person measuring. That denominator
is small AND not independent of the measurer.
- **The known-negative is CONSTRUCTED to fell the rule.** It says nothing about
how often the shape occurs in a real corpus; neither does KN-short.
- **KP and KN in the standard's fasit are controls, not results.** Neither
enters hit@k.
- **"Solved" means the one measured downside is gone with the standard
standing.** It does not mean no other downside exists — § 4 names one this
round could not remove, and the rule's reach outside the standard and this
bundle is still ZERO in everything measured (0 of 27 payloads fire).
- **P1 and P3's gains are round 16's, not this round's.** This round keeps them.
- **Three forms score identically on every row** — (b) at 0.125/0.15, (e), (g).
The choice between them rests on a property argued in § 2, not on a number.
- **The sweep harness is scratch, not the product.** It lives outside the
repository; its fusion ranks in § 1 are candidate ranks. Every rank in § 2
came out of `build_payload`'s real cut, and every figure in § 4's first table
came out of `okf consume` itself, run by `vegnormal-okf`'s instrument, which
was read and not modified.
- **This is still a reading-side change that moves a payload with no bundle
changing** — on the one measured question where a covered title is blocked.
A consumer pinned to 0.8.1's excerpt order on such a question has no flag
that reproduces it.