open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume): the compound-word miss is a degenerate signal's tie-break, behind a flag

Browse source 
The consumer's question about `vann- og frostsikring` in a subsea tunnel
delivered 0 of the 16 concepts covering it, best of them at fused rank 14.
Reproduced with the denominator, then decomposed per signal before anything
was built.

It is not a matcher miss. `normalise("vann- og frostsikring")` already returns
`('vann', 'frostsikring')` on HEAD, the prefix rule already bridges the
inflections, and the best covering concept already answers 7 of 7 question
tokens -- more than any delivered one. A tokeniser rule had nothing to widen.

It is the fusion, but not a weight. RRF ranks every concept in every signal,
including a signal that scored them all the same, and the declared
`(-score, concept_id)` tie-break then orders that group by id. On N500 the
document prior has TWO distinct values over 270 concepts, so the third signal
contributed alphabetical UUID order spread from 1/61 to 1/329 -- enough to put
a concept leading the body signal behind concepts sharing only `tunnel` and
`vann`.

`--tie-shared-rank` lets concepts a signal scores equally share that group's
first rank. The miss closes: best covering 14 -> 3, 2 of 16 delivered. OFF BY
DEFAULT, by the order's own rule: the three requirement lookups hold at rank 1
and the K2 digest holds, but hit@8 over the six published questions falls 5 of
6 to 4 of 6. Decomposed rather than guessed -- K2's prior is coarse (6 values
over 39 documents) rather than degenerate, and one gold sat early in its tie
group. That benefit was never a measurement, but it is a published row.

`--withheld-titles` gives each withheld entry the concept's title, so a reader
can see WHAT was withheld without reading the bundle. 11 lines of code; the
bytes are why it is off. It grows an N500 payload 37.9 % and takes the
629-concept K2 bundle's BOOKKEEPING to 122 704 B -- past the 120 000-byte limit
itself -- which would falsify the breaking point published in the tracked
`skills/okf-consume/SKILL.md` on the day it shipped.

Defaults measured, not asserted: six payload digests built from a frozen
`ff79cfa` (`git archive`, `__file__` checked) and from this tree with both
flags omitted are 6 of 6 identical, and `okf_skill.py` output is identical
apart from the paths each copy writes about itself. Contract checker exit 0 on
eight payloads, both values.

One known-positive did not reproduce and is reported rather than matched: the
order's S7 literal `2ae46f68`/169 573 B is stale by three excerpt-form commits;
HEAD measures `c759a657`/171 614 B.

Suite 1388 -> 1397. Report: docs/2026-09-08-rangeringsbom-sammensatte-ord.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 18:53:18 +02:00
commit c3b645bccf
5 changed files with 630 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

290
docs/2026-09-08-rangeringsbom-sammensatte-ord.md Normal file
View file

@ -0,0 +1,290 @@
# The compound-word ranking miss: a degenerate signal, not a matcher, 2026-09-08
A consumer asked the N500 bundle about `vann- og frostsikring` in a subsea
tunnel and got eight excerpts, none of which covered the subject; the bundle
covers it on 16 concepts. The consumer behaved as the contract asks — it
answered `[sourced-not-sufficient]` and named its denominator — so the miss is
the ranker's (`docs/2026-09-08-claude-code-skill-vilkaarlig-bundle.md` § 5).
The order proposed two sites for one rule: the tokeniser, or the fusion. The
measurement chose for us, and it chose neither of the two mechanisms the order
named. **The tokeniser already resolves the compound**, and **the fusion's
defect is not a weight — it is the tie-break in a signal that separates
nothing.**
---
## 0. What IS measured, and what is NOT
**Measured.** The miss reproduced with a command and a denominator; the three
per-signal ranks of the covering concepts; the document prior's distinctness on
two corpora; the rule's effect on four cases; every known-negative the order
names, both values; six default payload digests against a frozen HEAD; the
contract checker on eight payloads; the skill generator's output before and
after; the full suite.
**Known-positive, run first.** Every figure this session builds on was
reproduced on this machine before any after-column was read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | files matching `frostsikring` | C1 § 5 | **17**, case-insensitively, `index.md` among them |
| 2 | files matching `vann- og frostsikring` | C1 § 5 | **10** |
| 3 | files matching `undersjøisk` | C1 § 5 | **13** |
| 4 | the miss itself: covering concepts in the cut | C1 § 5 | **0** |
| 5 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6**, ranks `1,1,1,1,1,–` |
| 6 | the three requirement lookups, rank of 8 | O2c § 3 | **1, 1, 1** |
| 7 | K2 bundle, whole tree | Arm C § "did not move" | **1108 files**, `9cd74519…` |
| 8 | suite before the session | STATE (round 110) | **1388 passed** |
**One known-positive did NOT reproduce, and the deviation is the order's own
literal.** The order names the S7 control payload as `2ae46f68…`, 169 573 B.
Measured on frozen `ff79cfa` (`git archive` into a scratch tree, `__file__`
asserted), that command produces **`c759a657…`, 171 614 B**. The literal is not
wrong, it is *stale*: it was published at `116d3e1`, and `a37d5ce`, `17c49fc`
and `c95d189` each widened the excerpt form afterwards. No document restates it.
The row is therefore measured as **"byte-identical against HEAD"**, which is
what it exists to prove, and the stale literal is reported rather than quietly
matched.
**NOT measured.**
- **Whether a live model answers better.** This measures a ranking. Delivered
is not read.
- **`o200k_base` token counts.** No encoder is installed and the
single-dependency rule does not admit one. Every figure is in `BUDGET_UNIT`
plus `sha256`.
- **Whether the rule helps a third corpus class.** Two were measured (the three
N-bundles and K2). A corpus whose document prior is neither degenerate nor
coarse is unmeasured, and the flag's default reflects that.
---
## 1. The reproduction, with the command and the numbers
The order asks for the question C1 § 5 used, quoted. **It is not in that
document** — § 5 describes the question in prose ("about water and frost
protection in a subsea tunnel") and quotes no line. The question below is the
order's own wording, and it is marked as such rather than presented as C1's:
```
python3 tools/okf_consume.py ~/repos/vegnormal-okf/build/ferdig/n500-2024 \
--question "Hvilke krav gjelder vann- og frostsikring i undersjøisk tunnel?" \
--out /tmp/o4-baseline.json
```
270 concepts, 8 delivered, 262 withheld. **0 of the 16 covering concepts** — the
concepts whose title or body says `frostsikring`, case-insensitively — is in the
cut. The best of them fuses to **rank 14**.
The eight delivered, and the three covering concepts nearest the cut, with the
per-signal ranks that decide the fusion:
| | fused | body | title+id | doc prior | lexical | title |
|---|---|---|---|---|---|---|
| delivered 1 | 1 | 17 | 6 | 4 | 6 | Krav 4.3—1 Tunnelportaler |
| delivered 2 | 2 | 24 | 1 | 50 | 7 | Krav 3.2—2 Utslipp av vann fra tunnel |
| delivered 3 | 3 | 138 | 5 | 1 | 4 | Krav 5.3.6—1 Fjernstyrte bommer |
| **covering 1** | **14** | **6** | 33 | **130** | **7** | Krav 8.4.2—1 Frostisolering med PE-skum eller XPS |
| covering 2 | 39 | 34 | 30 | 121 | 6 | Krav 9.3.2—2 Fundament, omfylling og frostsikring |
| covering 3 | 44 | 25 | 101 | 63 | 5 | Krav 1.1—2 Generelle bestemmelser |
The best covering concept answers **7 of 7** question tokens — more than any
delivered one — and leads the body signal at rank 6. It is not invisible to the
matcher. It is behind on one signal: the document prior, at rank 130.
---
## 2. Why it is not a matcher miss (the order's alternative A, falsified)
Three measurements, each of which alone rules the tokeniser out as the site:
1. **The compound is already resolved.** `normalise("vann- og frostsikring")`
returns `('vann', 'frostsikring')` on HEAD. The hyphen splits, `og` falls
below `MIN_TOKEN_LENGTH`. There is no coordination left for a rule to undo.
2. **The prefix rule already bridges the inflections.**
`tokens_match('frostsikring', 'frostsikringen')`,
`('vann', 'vannsikring')` and `('undersjøisk', 'undersjøiske')` are all
`True` at `MIN_SHARED_PREFIX = 4`.
3. **The covering concept already scores the maximum.** Its lexical overlap is
7 of 7 question tokens. A rule widening the tokeniser cannot raise 7.
Recorded as a test rather than as a memory:
`tests/test_tie_shared_rank.py::test_the_hyphen_and_og_coordination_is_already_resolved_by_the_tokeniser`,
green on HEAD and labelled a characterisation.
One thing the same measurement DID surface, and it is not fixed here: the
prefix rule over-matches in the other direction. `undersjøisk` matches `under`,
so its document frequency is **159 of 270** — a term that should be one of the
rarest in the question reads as one of the commonest. That belongs to the
rarity weight (`--rarity-weight`, already shipped and already off), and this
session was allowed one rule.
---
## 3. What it is: a signal that separates nothing still orders everything
RRF consumes ranks only, so `concept_scores` produces a rank for every concept
in every signal — including a signal that gave them all the same score. The
declared tie-break, `(-score, concept_id)`, then orders that group **by
`concept_id`**, and the fusion reads the result as though it were a
measurement.
On N500 the concept ids are UUIDs and the document prior is degenerate:
| corpus | documents | distinct prior values | concepts sharing one value |
|---|---|---|---|
| N500 | 2 | **2** | **269 of 270** |
| K2 | 39 | 6 | — |
So on N500 the third signal contributed alphabetical UUID order, spread from
`1/61` down to `1/329`. That spread is worth more than several positions on
the two signals that did the measuring, and it is what put a concept answering
7 of 7 tokens behind concepts sharing only `tunnel` and `vann`.
**The rule.** `tie_shared_rank`: every concept a signal scores equally takes
that score group's **first** position, so a signal that separates nothing
contributes an identical constant to every concept and orders none of them. It
is general — no word list, no corpus constant, no new number except the group
boundary the scores themselves define.
The first position rather than the middle, and that was measured too: the
midpoint variant put the best covering concept at rank 5 where the first puts
it at **3**, and neither changed the three lookups. First is kept because under
it a non-separating signal contributes an identical constant, which is the
whole claim; the midpoint still varies with the size of the group a concept
lands in.
---
## 4. The known-negative table, both values
Flag `--tie-shared-rank`, off and on. Every row measured in one run,
`/tmp/o4/kn.py`.
| row | off | on | verdict |
|---|---|---|---|
| N100 lookup `Krav 3.3.1-13`, rank of 8 | 1 | **1** | holds |
| N200 lookup `Krav 2.9.2-12`, rank of 8 | 1 | **1** | holds |
| N500 lookup `Krav 10.2-2`, rank of 8 | 1 | **1** | holds |
| hit@8 over the six published questions | **5 of 6** | **4 of 6** | **FALLS** |
| — per-question rank | `1,1,1,1,1,–` | `1,–,1,1,1,–` | row 2 lost |
| S7 control question, default command | `c759a657…` 171 614 B | `5f4cd665…` 145 228 B | moves (on purpose) |
| K2 bundle, whole tree | `9cd74519…` 1108 files | `9cd74519…` 1108 files | holds |
| `okf_contract_check`, 8 payloads | exit 0 | exit 0 | holds |
| **the O4 miss: best covering rank** | **14** | **3** | **closes** |
| **the O4 miss: covering concepts in the cut** | **0 of 16** | **2 of 16** | **closes** |
The two concepts the rule delivers are `Krav 8.4.2—1 Frostisolering med PE-skum
eller XPS` and `Krav 9.3.2—2 Fundament, omfylling og frostsikring for
ledninger`.
**Default stays OFF**, by the order's own rule: it requires every row to hold
*and* the miss to close. The miss closes; the hit@8 row does not.
### Why row 2 falls, decomposed rather than guessed
| row | gold concepts | best fused, off | best fused, on |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| **2** | 49 | **1** | **28** |
| 3 | 20 | 1 | 1 |
| 4 | 43 | 1 | 1 |
| 5 | 11 | 1 | 1 |
| 6 | 5 | 11 (not delivered) | 91 (not delivered) |
K2's document prior is **not** degenerate: 6 distinct values over 39 documents.
There the tie-break groups are large but the signal does separate, and row 2's
gold happened to sit early in its group — that is, the noise was, on that one
question, pointing the right way. Removing the noise removed a benefit that was
never a measurement. That is an honest reason to keep the rule available and an
equally honest reason not to make it the default: **a rule that is right in
principle and loses a published row is a caller's choice, not a library's.**
Row 6 was already the known 1-of-6 shortfall and is not delivered either way.
---
## 5. The default did not move, measured against a frozen HEAD
Six payloads built from `git archive ff79cfa` into `/tmp/o4/head` (with
`__file__` asserted to be under that tree, so an editable install cannot be
read by accident) and from this working tree with both new flags omitted:
| payload | digest |
|---|---|
| N100 lookup | `a2a91d8b99157a0c`, 58 082 B |
| N200 lookup | `0c828a5f8c432864`, 144 932 B |
| N500 lookup | `ad2b35a556278553`, 41 364 B |
| N500 O4 miss | `d4a75828c147d9e9`, 47 056 B |
| S7 control | `c759a657dbb41d71`, 171 614 B |
| K2 price question | `f17c8725ee6bbe1a`, 167 360 B |
**6 of 6 identical.** No consumer's bytes move because this session happened.
`tools/okf_skill.py` likewise: `SKILL.md` and `references/example-payload.json`
are byte-identical apart from the absolute paths each copy writes about itself.
---
## 6. `withheld` names what was dropped — behind a flag, and here is why
C1 § 5's second observation: a `withheld` entry carries `concept_id` and `rule`
and no title, so a reader told 262 concepts were withheld cannot tell *what*
without reading the bundle, which § 2.2 forbids.
The code is cheap — 11 lines, well inside the order's 20-line bar. **The bytes
are not**, and that is what moved the decision:
| | without title | with title |
|---|---|---|
| N500 payload | 41 364 B | 57 023 B (**+37.9 %**) |
| N100 payload | 58 082 B | 88 251 B |
| N200 payload | 144 932 B | 215 148 B |
| K2 bookkeeping (629 concepts, everything not an excerpt) | ~75 KB | **122 704 B** |
The last row is the one that decided it. `skills/okf-consume/SKILL.md` — a
**tracked** file — publishes the breaking point as "at 629 concepts it is ~75 KB
… at roughly **8 000 concepts** the `withheld` list alone approaches the
120 000-byte limit". With titles on by default that sentence becomes false
immediately: at 629 concepts the bookkeeping is already **past** the limit. The
generated skill's own figure moved from a 1206-concept ceiling to 762.
So `--withheld-titles`, off by default. On, every one of N500's 262 withheld
entries carries its title; off, the entries are `{concept_id, rule}` exactly as
before, and the eight contract-checked payloads all exit 0 either way.
---
## 7. What did NOT close
- **14 of the 16 covering concepts are still outside the cut** with the flag on.
Rank 3 is not rank 1, and `k = 8` over 270 concepts is a narrow door.
- **The over-matching prefix.** `undersjøisk` reads as 159-of-270 common. One
rule per session; this was not it.
- **hit@8 row 6** — unchanged, not delivered either way, both before and after.
- **The default.** Two flags shipped this session and both are off. Whether
either should be on is in the session's decision block, not decided here.
---
## 8. Tests and gates
New file `tests/test_tie_shared_rank.py`, six tests: two characterisations
(green on HEAD, marked as such), and four that were **red on HEAD** — the
ranking difference (`8` vs `1` on a synthetic bundle, two different numbers
rather than one predicate two branches share), the byte-identity of the default,
the delivered set under the flag, and the CLI defaults.
Three tests added to `tests/test_okf_consume.py` for `--withheld-titles`: the
titled form, the untitled default (`set(entry) == {"concept_id", "rule"}`), and
the byte cost. One existing assertion was widened from a whole-dict equality to
the two fields it is actually about, so an unrelated widening of the withheld
form cannot fail a cost-vocabulary test.
| gate | before | after |
|---|---|---|
| `pytest -q` | 1388 passed | **1397 passed** |
| `ruff check src tests tools` | clean | clean |
| `ruff format --check` | clean | clean (92 files) |
| `mypy --strict src` | 17 files, clean | **17 files, clean** |