open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

fix(consume): the compound-word miss is a degenerate signal's tie-break, behind a flag

Browse source 
The consumer's question about `vann- og frostsikring` in a subsea tunnel
delivered 0 of the 16 concepts covering it, best of them at fused rank 14.
Reproduced with the denominator, then decomposed per signal before anything
was built.

It is not a matcher miss. `normalise("vann- og frostsikring")` already returns
`('vann', 'frostsikring')` on HEAD, the prefix rule already bridges the
inflections, and the best covering concept already answers 7 of 7 question
tokens -- more than any delivered one. A tokeniser rule had nothing to widen.

It is the fusion, but not a weight. RRF ranks every concept in every signal,
including a signal that scored them all the same, and the declared
`(-score, concept_id)` tie-break then orders that group by id. On N500 the
document prior has TWO distinct values over 270 concepts, so the third signal
contributed alphabetical UUID order spread from 1/61 to 1/329 -- enough to put
a concept leading the body signal behind concepts sharing only `tunnel` and
`vann`.

`--tie-shared-rank` lets concepts a signal scores equally share that group's
first rank. The miss closes: best covering 14 -> 3, 2 of 16 delivered. OFF BY
DEFAULT, by the order's own rule: the three requirement lookups hold at rank 1
and the K2 digest holds, but hit@8 over the six published questions falls 5 of
6 to 4 of 6. Decomposed rather than guessed -- K2's prior is coarse (6 values
over 39 documents) rather than degenerate, and one gold sat early in its tie
group. That benefit was never a measurement, but it is a published row.

`--withheld-titles` gives each withheld entry the concept's title, so a reader
can see WHAT was withheld without reading the bundle. 11 lines of code; the
bytes are why it is off. It grows an N500 payload 37.9 % and takes the
629-concept K2 bundle's BOOKKEEPING to 122 704 B -- past the 120 000-byte limit
itself -- which would falsify the breaking point published in the tracked
`skills/okf-consume/SKILL.md` on the day it shipped.

Defaults measured, not asserted: six payload digests built from a frozen
`ff79cfa` (`git archive`, `__file__` checked) and from this tree with both
flags omitted are 6 of 6 identical, and `okf_skill.py` output is identical
apart from the paths each copy writes about itself. Contract checker exit 0 on
eight payloads, both values.

One known-positive did not reproduce and is reported rather than matched: the
order's S7 literal `2ae46f68`/169 573 B is stale by three excerpt-form commits;
HEAD measures `c759a657`/171 614 B.

Suite 1388 -> 1397. Report: docs/2026-09-08-rangeringsbom-sammensatte-ord.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-08 18:53:18 +02:00
commit c3b645bccf
5 changed files with 630 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

25
CLAUDE.md
View file

@ -314,7 +314,7 @@ and fixtures, never code.
measurement behind it, including the control that FAILED, is measurement behind it, including the control that FAILED, is
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this `docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
repository's own choice** — the contract binds a payload, not a retrieval repository's own choice** — the contract binds a payload, not a retrieval
algorithm (§ 10) — and it has THREE optional widenings, all **off by default** algorithm (§ 10) — and it has FOUR optional widenings, all **off by default**
and all keeping the default payload byte-identical. `--cost-vocabulary`: a and all keeping the default payload byte-identical. `--cost-vocabulary`: a
declared cost/price/quantity vocabulary family that bridges a question and a declared cost/price/quantity vocabulary family that bridges a question and a
document naming money with different words, gated on the QUESTION carrying document naming money with different words, gated on the QUESTION carrying
@ -344,6 +344,29 @@ and fixtures, never code.
read as 135-of-446 common, and RRF consumes RANKS, so no weighting inside a read as 135-of-446 common, and RRF consumes RANKS, so no weighting inside a
signal can move a gold that already leads it. signal can move a gold that already leads it.
`docs/2026-09-08-sjeldenhetsvekt.md`. `docs/2026-09-08-sjeldenhetsvekt.md`.
`--tie-shared-rank` is the fourth, and it is a correction to the TIE-BREAK
rather than a weight: RRF ranks every concept in every signal, including a
signal that scored them all the same, and the declared `(-score, concept_id)`
tie-break then orders that group by id. Measured on N500, whose document
prior has **two** distinct values over 270 concepts, that signal contributed
alphabetical UUID order and put a concept answering 7 of 7 question tokens at
fused rank 14 — outside the cut — behind concepts sharing only `tunnel` and
`vann`. Under shared ranks it is rank 3 and 2 of the 16 covering concepts are
delivered. Off by default BY MEASUREMENT: the three requirement lookups hold
at rank 1 and the K2 digest holds, but hit@8 over the six published questions
falls 5 of 6 to 4 of 6, because K2's prior is coarse rather than degenerate
(6 values over 39 documents) and one gold sat early in its tie group — a
benefit that was never a measurement, but a published row all the same.
`docs/2026-09-08-rangeringsbom-sammensatte-ord.md`.
A FIFTH flag is not a ranking widening and is listed apart: `--withheld-titles`
gives each `withheld` entry the concept's `title`, so a reader can see WHAT
was withheld without reading the bundle (§ 2.2 forbids going to look). The
code is 11 lines; the bytes are the reason it is off. Measured, it grows an
N500 payload 37.9 % and takes the 629-concept K2 bundle's BOOKKEEPING to
122 704 B — past the 120 000-byte limit itself — which would make the
breaking point published in the tracked `skills/okf-consume/SKILL.md`
("~75 KB at 629 concepts … at roughly 8 000 concepts") false on the day it
shipped.
## Workflow ## Workflow

290
docs/2026-09-08-rangeringsbom-sammensatte-ord.md Normal file
View file

@ -0,0 +1,290 @@
# The compound-word ranking miss: a degenerate signal, not a matcher, 2026-09-08
A consumer asked the N500 bundle about `vann- og frostsikring` in a subsea
tunnel and got eight excerpts, none of which covered the subject; the bundle
covers it on 16 concepts. The consumer behaved as the contract asks — it
answered `[sourced-not-sufficient]` and named its denominator — so the miss is
the ranker's (`docs/2026-09-08-claude-code-skill-vilkaarlig-bundle.md` § 5).
The order proposed two sites for one rule: the tokeniser, or the fusion. The
measurement chose for us, and it chose neither of the two mechanisms the order
named. **The tokeniser already resolves the compound**, and **the fusion's
defect is not a weight — it is the tie-break in a signal that separates
nothing.**
---
## 0. What IS measured, and what is NOT
**Measured.** The miss reproduced with a command and a denominator; the three
per-signal ranks of the covering concepts; the document prior's distinctness on
two corpora; the rule's effect on four cases; every known-negative the order
names, both values; six default payload digests against a frozen HEAD; the
contract checker on eight payloads; the skill generator's output before and
after; the full suite.
**Known-positive, run first.** Every figure this session builds on was
reproduced on this machine before any after-column was read:
| # | known-positive | source | measured here |
|---|---|---|---|
| 1 | files matching `frostsikring` | C1 § 5 | **17**, case-insensitively, `index.md` among them |
| 2 | files matching `vann- og frostsikring` | C1 § 5 | **10** |
| 3 | files matching `undersjøisk` | C1 § 5 | **13** |
| 4 | the miss itself: covering concepts in the cut | C1 § 5 | **0** |
| 5 | hit@8 over the six published questions | 2026-09-07 § hit@k | **5 of 6**, ranks `1,1,1,1,1,–` |
| 6 | the three requirement lookups, rank of 8 | O2c § 3 | **1, 1, 1** |
| 7 | K2 bundle, whole tree | Arm C § "did not move" | **1108 files**, `9cd74519…` |
| 8 | suite before the session | STATE (round 110) | **1388 passed** |
**One known-positive did NOT reproduce, and the deviation is the order's own
literal.** The order names the S7 control payload as `2ae46f68…`, 169 573 B.
Measured on frozen `ff79cfa` (`git archive` into a scratch tree, `__file__`
asserted), that command produces **`c759a657…`, 171 614 B**. The literal is not
wrong, it is *stale*: it was published at `116d3e1`, and `a37d5ce`, `17c49fc`
and `c95d189` each widened the excerpt form afterwards. No document restates it.
The row is therefore measured as **"byte-identical against HEAD"**, which is
what it exists to prove, and the stale literal is reported rather than quietly
matched.
**NOT measured.**
- **Whether a live model answers better.** This measures a ranking. Delivered
is not read.
- **`o200k_base` token counts.** No encoder is installed and the
single-dependency rule does not admit one. Every figure is in `BUDGET_UNIT`
plus `sha256`.
- **Whether the rule helps a third corpus class.** Two were measured (the three
N-bundles and K2). A corpus whose document prior is neither degenerate nor
coarse is unmeasured, and the flag's default reflects that.
---
## 1. The reproduction, with the command and the numbers
The order asks for the question C1 § 5 used, quoted. **It is not in that
document** — § 5 describes the question in prose ("about water and frost
protection in a subsea tunnel") and quotes no line. The question below is the
order's own wording, and it is marked as such rather than presented as C1's:
```
python3 tools/okf_consume.py ~/repos/vegnormal-okf/build/ferdig/n500-2024 \
--question "Hvilke krav gjelder vann- og frostsikring i undersjøisk tunnel?" \
--out /tmp/o4-baseline.json
```
270 concepts, 8 delivered, 262 withheld. **0 of the 16 covering concepts** — the
concepts whose title or body says `frostsikring`, case-insensitively — is in the
cut. The best of them fuses to **rank 14**.
The eight delivered, and the three covering concepts nearest the cut, with the
per-signal ranks that decide the fusion:
| | fused | body | title+id | doc prior | lexical | title |
|---|---|---|---|---|---|---|
| delivered 1 | 1 | 17 | 6 | 4 | 6 | Krav 4.3—1 Tunnelportaler |
| delivered 2 | 2 | 24 | 1 | 50 | 7 | Krav 3.2—2 Utslipp av vann fra tunnel |
| delivered 3 | 3 | 138 | 5 | 1 | 4 | Krav 5.3.6—1 Fjernstyrte bommer |
| **covering 1** | **14** | **6** | 33 | **130** | **7** | Krav 8.4.2—1 Frostisolering med PE-skum eller XPS |
| covering 2 | 39 | 34 | 30 | 121 | 6 | Krav 9.3.2—2 Fundament, omfylling og frostsikring |
| covering 3 | 44 | 25 | 101 | 63 | 5 | Krav 1.1—2 Generelle bestemmelser |
The best covering concept answers **7 of 7** question tokens — more than any
delivered one — and leads the body signal at rank 6. It is not invisible to the
matcher. It is behind on one signal: the document prior, at rank 130.
---
## 2. Why it is not a matcher miss (the order's alternative A, falsified)
Three measurements, each of which alone rules the tokeniser out as the site:
1. **The compound is already resolved.** `normalise("vann- og frostsikring")`
returns `('vann', 'frostsikring')` on HEAD. The hyphen splits, `og` falls
below `MIN_TOKEN_LENGTH`. There is no coordination left for a rule to undo.
2. **The prefix rule already bridges the inflections.**
`tokens_match('frostsikring', 'frostsikringen')`,
`('vann', 'vannsikring')` and `('undersjøisk', 'undersjøiske')` are all
`True` at `MIN_SHARED_PREFIX = 4`.
3. **The covering concept already scores the maximum.** Its lexical overlap is
7 of 7 question tokens. A rule widening the tokeniser cannot raise 7.
Recorded as a test rather than as a memory:
`tests/test_tie_shared_rank.py::test_the_hyphen_and_og_coordination_is_already_resolved_by_the_tokeniser`,
green on HEAD and labelled a characterisation.
One thing the same measurement DID surface, and it is not fixed here: the
prefix rule over-matches in the other direction. `undersjøisk` matches `under`,
so its document frequency is **159 of 270** — a term that should be one of the
rarest in the question reads as one of the commonest. That belongs to the
rarity weight (`--rarity-weight`, already shipped and already off), and this
session was allowed one rule.
---
## 3. What it is: a signal that separates nothing still orders everything
RRF consumes ranks only, so `concept_scores` produces a rank for every concept
in every signal — including a signal that gave them all the same score. The
declared tie-break, `(-score, concept_id)`, then orders that group **by
`concept_id`**, and the fusion reads the result as though it were a
measurement.
On N500 the concept ids are UUIDs and the document prior is degenerate:
| corpus | documents | distinct prior values | concepts sharing one value |
|---|---|---|---|
| N500 | 2 | **2** | **269 of 270** |
| K2 | 39 | 6 | — |
So on N500 the third signal contributed alphabetical UUID order, spread from
`1/61` down to `1/329`. That spread is worth more than several positions on
the two signals that did the measuring, and it is what put a concept answering
7 of 7 tokens behind concepts sharing only `tunnel` and `vann`.
**The rule.** `tie_shared_rank`: every concept a signal scores equally takes
that score group's **first** position, so a signal that separates nothing
contributes an identical constant to every concept and orders none of them. It
is general — no word list, no corpus constant, no new number except the group
boundary the scores themselves define.
The first position rather than the middle, and that was measured too: the
midpoint variant put the best covering concept at rank 5 where the first puts
it at **3**, and neither changed the three lookups. First is kept because under
it a non-separating signal contributes an identical constant, which is the
whole claim; the midpoint still varies with the size of the group a concept
lands in.
---
## 4. The known-negative table, both values
Flag `--tie-shared-rank`, off and on. Every row measured in one run,
`/tmp/o4/kn.py`.
| row | off | on | verdict |
|---|---|---|---|
| N100 lookup `Krav 3.3.1-13`, rank of 8 | 1 | **1** | holds |
| N200 lookup `Krav 2.9.2-12`, rank of 8 | 1 | **1** | holds |
| N500 lookup `Krav 10.2-2`, rank of 8 | 1 | **1** | holds |
| hit@8 over the six published questions | **5 of 6** | **4 of 6** | **FALLS** |
| — per-question rank | `1,1,1,1,1,–` | `1,–,1,1,1,–` | row 2 lost |
| S7 control question, default command | `c759a657…` 171 614 B | `5f4cd665…` 145 228 B | moves (on purpose) |
| K2 bundle, whole tree | `9cd74519…` 1108 files | `9cd74519…` 1108 files | holds |
| `okf_contract_check`, 8 payloads | exit 0 | exit 0 | holds |
| **the O4 miss: best covering rank** | **14** | **3** | **closes** |
| **the O4 miss: covering concepts in the cut** | **0 of 16** | **2 of 16** | **closes** |
The two concepts the rule delivers are `Krav 8.4.2—1 Frostisolering med PE-skum
eller XPS` and `Krav 9.3.2—2 Fundament, omfylling og frostsikring for
ledninger`.
**Default stays OFF**, by the order's own rule: it requires every row to hold
*and* the miss to close. The miss closes; the hit@8 row does not.
### Why row 2 falls, decomposed rather than guessed
| row | gold concepts | best fused, off | best fused, on |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| **2** | 49 | **1** | **28** |
| 3 | 20 | 1 | 1 |
| 4 | 43 | 1 | 1 |
| 5 | 11 | 1 | 1 |
| 6 | 5 | 11 (not delivered) | 91 (not delivered) |
K2's document prior is **not** degenerate: 6 distinct values over 39 documents.
There the tie-break groups are large but the signal does separate, and row 2's
gold happened to sit early in its group — that is, the noise was, on that one
question, pointing the right way. Removing the noise removed a benefit that was
never a measurement. That is an honest reason to keep the rule available and an
equally honest reason not to make it the default: **a rule that is right in
principle and loses a published row is a caller's choice, not a library's.**
Row 6 was already the known 1-of-6 shortfall and is not delivered either way.
---
## 5. The default did not move, measured against a frozen HEAD
Six payloads built from `git archive ff79cfa` into `/tmp/o4/head` (with
`__file__` asserted to be under that tree, so an editable install cannot be
read by accident) and from this working tree with both new flags omitted:
| payload | digest |
|---|---|
| N100 lookup | `a2a91d8b99157a0c`, 58 082 B |
| N200 lookup | `0c828a5f8c432864`, 144 932 B |
| N500 lookup | `ad2b35a556278553`, 41 364 B |
| N500 O4 miss | `d4a75828c147d9e9`, 47 056 B |
| S7 control | `c759a657dbb41d71`, 171 614 B |
| K2 price question | `f17c8725ee6bbe1a`, 167 360 B |
**6 of 6 identical.** No consumer's bytes move because this session happened.
`tools/okf_skill.py` likewise: `SKILL.md` and `references/example-payload.json`
are byte-identical apart from the absolute paths each copy writes about itself.
---
## 6. `withheld` names what was dropped — behind a flag, and here is why
C1 § 5's second observation: a `withheld` entry carries `concept_id` and `rule`
and no title, so a reader told 262 concepts were withheld cannot tell *what*
without reading the bundle, which § 2.2 forbids.
The code is cheap — 11 lines, well inside the order's 20-line bar. **The bytes
are not**, and that is what moved the decision:
| | without title | with title |
|---|---|---|
| N500 payload | 41 364 B | 57 023 B (**+37.9 %**) |
| N100 payload | 58 082 B | 88 251 B |
| N200 payload | 144 932 B | 215 148 B |
| K2 bookkeeping (629 concepts, everything not an excerpt) | ~75 KB | **122 704 B** |
The last row is the one that decided it. `skills/okf-consume/SKILL.md` — a
**tracked** file — publishes the breaking point as "at 629 concepts it is ~75 KB
… at roughly **8 000 concepts** the `withheld` list alone approaches the
120 000-byte limit". With titles on by default that sentence becomes false
immediately: at 629 concepts the bookkeeping is already **past** the limit. The
generated skill's own figure moved from a 1206-concept ceiling to 762.
So `--withheld-titles`, off by default. On, every one of N500's 262 withheld
entries carries its title; off, the entries are `{concept_id, rule}` exactly as
before, and the eight contract-checked payloads all exit 0 either way.
---
## 7. What did NOT close
- **14 of the 16 covering concepts are still outside the cut** with the flag on.
Rank 3 is not rank 1, and `k = 8` over 270 concepts is a narrow door.
- **The over-matching prefix.** `undersjøisk` reads as 159-of-270 common. One
rule per session; this was not it.
- **hit@8 row 6** — unchanged, not delivered either way, both before and after.
- **The default.** Two flags shipped this session and both are off. Whether
either should be on is in the session's decision block, not decided here.
---
## 8. Tests and gates
New file `tests/test_tie_shared_rank.py`, six tests: two characterisations
(green on HEAD, marked as such), and four that were **red on HEAD** — the
ranking difference (`8` vs `1` on a synthetic bundle, two different numbers
rather than one predicate two branches share), the byte-identity of the default,
the delivered set under the flag, and the CLI defaults.
Three tests added to `tests/test_okf_consume.py` for `--withheld-titles`: the
titled form, the untitled default (`set(entry) == {"concept_id", "rule"}`), and
the byte cost. One existing assertion was widened from a whole-dict equality to
the two fields it is actually about, so an unrelated widening of the withheld
form cannot fail a cost-vocabulary test.
| gate | before | after |
|---|---|---|
| `pytest -q` | 1388 passed | **1397 passed** |
| `ruff check src tests tools` | clean | clean |
| `ruff format --check` | clean | clean (92 files) |
| `mypy --strict src` | 17 files, clean | **17 files, clean** |

52
tests/test_okf_consume.py
View file

@ -609,6 +609,51 @@ def test_a_concept_whose_verified_cannot_be_read_is_withheld_by_name() -> None:
assert dict(withheld)["dyp/nivaa/blokkform-verifisert"] == "verified_unreadable" assert dict(withheld)["dyp/nivaa/blokkform-verifisert"] == "verified_unreadable"
def test_a_withheld_entry_names_what_was_dropped_under_the_flag() -> None:
# A reader who is told 262 concepts were withheld, by id and rule alone,
# cannot tell WHAT was withheld without reading the bundle -- which SS 2.2
# forbids. The title closes that, and it is emitted only where the concept
# carries one.
payload = okf_consume.build_payload(
FIXTURE, question="Hvordan skal prisene fylles ut?", withheld_titles=True
)
entries = payload["withheld"]
assert isinstance(entries, list) and entries
titled = [entry for entry in entries if "title" in entry]
assert titled, "no withheld entry carried a title, so the rule measures nothing"
concepts = {concept.concept_id: concept for concept in _fixture_concepts()}
for entry in entries:
concept = concepts[str(entry["concept_id"])]
if concept.title:
assert entry["title"] == concept.title
else:
assert "title" not in entry
def test_no_withheld_entry_names_anything_without_the_flag() -> None:
# The default is what every consumer already runs, and this is the
# measurement that keeps it theirs: a title on every withheld entry grew a
# 270-concept payload by 37.9 % and pushed a 629-concept bundle's
# bookkeeping past the budget limit itself.
payload = okf_consume.build_payload(FIXTURE, question="Hvordan skal prisene fylles ut?")
entries = payload["withheld"]
assert isinstance(entries, list) and entries
assert all(set(entry) == {"concept_id", "rule"} for entry in entries)
def test_the_withheld_title_flag_costs_bytes_and_the_default_pays_none() -> None:
question = "Hvordan skal prisene fylles ut?"
off = okf_consume.serialise(okf_consume.build_payload(FIXTURE, question=question))
explicit_off = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, withheld_titles=False)
)
on = okf_consume.serialise(
okf_consume.build_payload(FIXTURE, question=question, withheld_titles=True)
)
assert off == explicit_off
assert len(on.encode("utf-8")) > len(off.encode("utf-8"))
def test_delivered_and_withheld_partition_the_considered_set() -> None: def test_delivered_and_withheld_partition_the_considered_set() -> None:
delivered, withheld, considered = _cut_fixture() delivered, withheld, considered = _cut_fixture()
delivered_ids = {excerpt["concept_id"] for excerpt in delivered} delivered_ids = {excerpt["concept_id"] for excerpt in delivered}
@ -1224,7 +1269,12 @@ def test_a_cost_question_reaches_no_price_concept_without_the_flag() -> None:
counts, withheld = payload["denominators"], payload["withheld"] counts, withheld = payload["denominators"], payload["withheld"]
assert isinstance(counts, dict) and isinstance(withheld, list) assert isinstance(counts, dict) and isinstance(withheld, list)
assert counts["delivered"] == 0 assert counts["delivered"] == 0
assert {"concept_id": "krav/pristabell", "rule": "no_lexical_match"} in withheld # By the two fields this test is about, not by the whole entry: the entry
# also carries the concept's title, and pinning the exact dict here would
# make an unrelated widening of the withheld form fail a cost-question test.
assert {(entry["concept_id"], entry["rule"]) for entry in withheld} >= {
("krav/pristabell", "no_lexical_match")
}
def test_the_cost_vocabulary_flag_bridges_a_question_and_a_document_that_share_no_word() -> None: def test_the_cost_vocabulary_flag_bridges_a_question_and_a_document_that_share_no_word() -> None:

167
tests/test_tie_shared_rank.py Normal file
View file

@ -0,0 +1,167 @@
"""The fusion's tie-break, measured rather than assumed.
`concept_scores` fuses three signals by RRF, and RRF consumes RANKS ONLY. A
rank is therefore produced for every concept in every signal -- including a
signal that does not separate them. The tie-break is declared
(`(-score, concept_id)`), so when a signal gives 269 of 270 concepts the same
score, that signal's contribution to the fusion is the concepts' own ids in
lexicographic order: a UUID, which is noise, weighted exactly as heavily as
the two signals that did the measuring.
MEASURED 2026-09-08 on the N500 bundle (270 concepts, `feae0c8`), for the
question about `vann- og frostsikring` in a subsea tunnel: the document prior
has **two** distinct values over the bundle, and 269 concepts share one of
them. The best covering concept answered **7 of 7** question tokens and led
the body signal at rank 6, and it fused to rank **14** -- outside the cut --
while concepts answering fewer tokens fused ahead of it on nothing but an
earlier id.
This file holds the two halves apart:
- The matcher is CHARACTERISED, not fixed. `normalise` already resolves the
hyphen-and-`og` coordination, so the alternative that would have widened the
tokeniser has nothing to widen. That is asserted here so the choice stays
falsifiable rather than remembered.
- The fusion gets one rule, behind one flag, off by default.
"""
from __future__ import annotations
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
import okf_consume # noqa: E402
QUESTION = "Hvilke krav gjelder vann- og frostsikring i undersjoeisk tunnel?"
_FRONTMATTER = (
"---\ntype: reference\ntitle: {title}\nsource_file: {slug}.md\n"
"source_sha256: {digest}\ningested_at: 2026-09-01T00:00:00Z\n"
"adjudication: proposed\nbundle_id: tie-fixture\n"
"verified: [{{ by: process:okf-check, at: 2026-09-01T00:00:00Z }}]\n---\n\n"
)
def _tie_bundle(root: Path, *, fillers: int = 30) -> Path:
"""One document, so the document prior cannot separate anything.
The gold concept's id sorts LAST and the fillers' ids sort first, which is
what makes the degenerate signal's tie-break work against the concept that
answers the question. The fillers answer `krav`, `gjelder`, `vann` and
`tunnel`; only the gold answers `frostsikring` and `undersjoeisk` too.
"""
(root / "krav").mkdir(parents=True)
(root / "index.md").write_text(
"---\nokf_version: 0.2\nbundle_id: tie-fixture\n---\n\n- [krav (index)](krav/index.md)\n",
encoding="utf-8",
)
entries: list[str] = []
def add(slug: str, title: str, body: str) -> None:
entries.append(f"- [{title}]({slug}.md) — adjudication: proposed\n")
(root / "krav" / f"{slug}.md").write_text(
_FRONTMATTER.format(title=title, slug=slug, digest="1" * 64) + f"## {title}\n\n" + body,
encoding="utf-8",
)
add(
"zz-gull",
"Krav om vann- og frostsikring i undersjoeisk tunnel",
"Kravet gjelder vannsikring og frostsikring i undersjoeisk tunnel.\n" * 4,
)
for number in range(1, fillers + 1):
add(
f"aa-{number:02d}",
f"Krav om tunnel og vann {number:02d}",
"Kravet gjelder tunnel og vann i anlegget.\n" * 4,
)
(root / "krav" / "index.md").write_text("".join(entries), encoding="utf-8")
return root
def _gold_rank(root: Path, *, tie_shared_rank: bool) -> int:
concepts = [
okf_consume.read_concept(
root / f"{concept_id}.md", bundle_root=root, root_bundle_id="tie-fixture"
)
for concept_id in okf_consume.enumerate_concepts(root)
]
ranked = okf_consume.concept_scores(
concepts,
QUESTION,
okf_consume.document_scores(root, QUESTION),
tie_shared_rank=tie_shared_rank,
)
for position, (concept, _, _) in enumerate(ranked, start=1):
if concept.concept_id.endswith("zz-gull"):
return position
raise AssertionError("the gold concept is not in the ranking at all")
# --- The half that is characterised, not fixed --------------------------------
def test_the_hyphen_and_og_coordination_is_already_resolved_by_the_tokeniser() -> None:
# CHARACTERISATION, green on HEAD. This is the measurement that ruled the
# tokeniser out as the site of the fix: there is no coordination left to
# resolve, so a rule widening it could not have moved the miss.
assert okf_consume.normalise("vann- og frostsikring") == ("vann", "frostsikring")
assert okf_consume.tokens_match("frostsikring", "frostsikringen")
assert okf_consume.tokens_match("vann", "vannsikring")
def test_a_signal_that_separates_nothing_still_ranks_every_concept(tmp_path: Path) -> None:
# CHARACTERISATION of the defect's mechanism: one distinct score, and one
# rank per concept all the same. The second number is the noise.
root = _tie_bundle(tmp_path / "bundle")
prior = okf_consume.document_scores(root, QUESTION)
concepts = okf_consume.enumerate_concepts(root)
scores = {prior[concept_id.split("/", 1)[0]] for concept_id in concepts}
assert len(scores) == 1
assert len(concepts) == 31
# --- The half that gets the rule ----------------------------------------------
def test_shared_rank_lifts_the_concept_the_measuring_signals_lead(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
# Two DIFFERENT numbers, not one predicate two branches share: the concept
# answering every question token sits at 8 while the degenerate signal
# orders by id, and at 1 once that signal stops ordering.
assert _gold_rank(root, tie_shared_rank=False) == 8
assert _gold_rank(root, tie_shared_rank=True) == 1
def test_the_payload_is_byte_identical_with_the_flag_off(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
without = okf_consume.serialise(okf_consume.build_payload(root, question=QUESTION))
explicit_off = okf_consume.serialise(
okf_consume.build_payload(root, question=QUESTION, tie_shared_rank=False)
)
assert without == explicit_off
def test_the_flag_changes_the_payload_it_is_meant_to_change(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
off = okf_consume.build_payload(root, question=QUESTION, k=3)
on = okf_consume.build_payload(root, question=QUESTION, k=3, tie_shared_rank=True)
delivered_off = [excerpt["concept_id"] for excerpt in off["excerpts"]] # type: ignore[index]
delivered_on = [excerpt["concept_id"] for excerpt in on["excerpts"]] # type: ignore[index]
assert not any(str(cid).endswith("zz-gull") for cid in delivered_off)
assert str(delivered_on[0]).endswith("zz-gull")
def test_the_cli_exposes_the_flag_and_defaults_it_off(tmp_path: Path) -> None:
root = _tie_bundle(tmp_path / "bundle")
parsed = okf_consume.parse_args([str(root), "--question", QUESTION])
assert parsed.tie_shared_rank is False
parsed_on = okf_consume.parse_args([str(root), "--question", QUESTION, "--tie-shared-rank"])
assert parsed_on.tie_shared_rank is True
# The other flag this session added, asserted here so "both default off"
# is one measurement rather than two files' worth of trust.
assert parsed.withheld_titles is False

98
tools/okf_consume.py
View file

@ -1004,6 +1004,7 @@ def concept_scores(
cost_vocabulary: bool = False, cost_vocabulary: bool = False,
weights: Mapping[str, float] | None = None, weights: Mapping[str, float] | None = None,
lookup: bool = True, lookup: bool = True,
tie_shared_rank: bool = False,
) -> list[tuple[Concept, float, int]]: ) -> list[tuple[Concept, float, int]]:
"""Every concept, ordered best first, fused from three signals by RRF. """Every concept, ordered best first, fused from three signals by RRF.
@ -1030,6 +1031,30 @@ def concept_scores(
separately: a concept that answers nothing in the question, sitting in a separately: a concept that answers nothing in the question, sitting in a
document that does, is a GUESS, and a guess is the one thing a declared cut document that does, is a GUESS, and a guess is the one thing a declared cut
must not deliver. must not deliver.
**`tie_shared_rank` is the one rule this fusion has for a signal that does
not separate**, off by default, and it is a correction to what the declared
tie-break does rather than a weight. RRF consumes ranks, so a rank is
produced for EVERY concept in EVERY signal -- including a signal that gave
them all the same score. The tie-break then orders that group by
`concept_id`, and the fusion reads the result as if it were a measurement.
MEASURED 2026-09-08 on the N500 bundle (270 concepts): the document prior
has **two** distinct values there and 269 concepts share one, so that
signal contributed the concepts' UUIDs in alphabetical order, spread across
`1/61` to `1/329`. The best concept covering `vann- og frostsikring` in a
subsea tunnel answered **7 of 7** question tokens and led the body signal
at rank 6; it fused to rank 14, outside the cut, behind concepts sharing
only `tunnel` and `vann` whose ids sorted earlier. With shared ranks it
fuses to rank 3.
The rule takes the FIRST position of a score group rather than its middle.
Both were measured on the same four cases; the middle put the same concept
at rank 5 where the first puts it at 3, and neither changed the three
known-positive lookups. First is kept because it is the reading under which
a signal that separates nothing contributes an identical constant to every
concept -- which is the whole claim -- where the middle still varies with
the size of the group a concept lands in.
""" """
question_tokens = normalise(question) question_tokens = normalise(question)
bridge = cost_vocabulary and question_uses_cost_vocabulary(question) bridge = cost_vocabulary and question_uses_cost_vocabulary(question)
@ -1065,8 +1090,23 @@ def concept_scores(
# Sort by score descending, then by id ascending -- the declared # Sort by score descending, then by id ascending -- the declared
# tie-break, applied before a rank is ever read. # tie-break, applied before a rank is ever read.
order = sorted(signal, key=lambda key: (-signal[key], key)) order = sorted(signal, key=lambda key: (-signal[key], key))
if not tie_shared_rank:
for position, concept_id in enumerate(order, start=1): for position, concept_id in enumerate(order, start=1):
fused[concept_id] += 1.0 / (RRF_K + position) fused[concept_id] += 1.0 / (RRF_K + position)
continue
# SHARED RANK: every concept a signal scores EQUALLY takes that score
# group's first position, so the signal contributes the same amount to
# each of them and orders none of them. See this function's docstring
# and `tests/test_tie_shared_rank.py` for what it is for.
start = 0
while start < len(order):
stop = start
while stop < len(order) and signal[order[stop]] == signal[order[start]]:
stop += 1
contribution = 1.0 / (RRF_K + start + 1)
for concept_id in order[start:stop]:
fused[concept_id] += contribution
start = stop
lexical = ( lexical = (
{ {
concept.concept_id: int(signals[0][concept.concept_id] + signals[1][concept.concept_id]) concept.concept_id: int(signals[0][concept.concept_id] + signals[1][concept.concept_id])
@ -1356,12 +1396,30 @@ def build_payload(
cost_vocabulary: bool = False, cost_vocabulary: bool = False,
reserve_top_rank: bool = False, reserve_top_rank: bool = False,
rarity_weight: bool = False, rarity_weight: bool = False,
tie_shared_rank: bool = False,
withheld_titles: bool = False,
) -> dict[str, object]: ) -> dict[str, object]:
"""One bundle plus one question, cut to one contract-conformant payload. """One bundle plus one question, cut to one contract-conformant payload.
Pure with respect to the clock and the network: the same Pure with respect to the clock and the network: the same
`(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank, `(bundle_root, question, k, limit, cost_vocabulary, reserve_top_rank,
rarity_weight)` at the same bytes returns the same object, every time. rarity_weight, tie_shared_rank, withheld_titles)` at the same bytes returns
the same object, every time.
**`withheld_titles` (default off) names what was dropped.** A `withheld`
entry carries `concept_id` and `rule` and no title, so a reader told that
262 concepts were withheld cannot tell WHAT was withheld without reading
the bundle -- which SS 2.2 forbids. The title closes that.
OFF BY DEFAULT BY MEASUREMENT, not by taste. Measured 2026-09-08: on N500
the payload grows 41 364 -> 57 023 bytes (+37.9 %), and on the 629-concept
K2 bundle the bookkeeping -- everything that is not an excerpt -- grows to
**122 704 bytes, past the 120 000-byte limit itself**. The instantiated
skill publishes that breaking point as "~75 KB at 629 concepts, reached at
roughly 8 000 concepts"; on by default would make that sentence false and
would move every consumer's bytes for a field none of them asked for.
Whether the naming is worth the bookkeeping is the caller's call, and the
flag is how it stays one.
""" """
case, expected, measured = known_positive() case, expected, measured = known_positive()
if expected != measured: if expected != measured:
@ -1400,7 +1458,9 @@ def build_payload(
), ),
cost_vocabulary=cost_vocabulary, cost_vocabulary=cost_vocabulary,
weights=weights, weights=weights,
tie_shared_rank=tie_shared_rank,
) )
titles_by_id = {concept.concept_id: concept.title for concept in concepts}
matched = sum(1 for _, _, lexical in ranked if lexical > 0) matched = sum(1 for _, _, lexical in ranked if lexical > 0)
delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank) delivered, withheld, reserved = cut(ranked, k=k, limit=limit, reserve_top_rank=reserve_top_rank)
spent = sum(excerpt_weight(excerpt) for excerpt in delivered) spent = sum(excerpt_weight(excerpt) for excerpt in delivered)
@ -1460,7 +1520,19 @@ def build_payload(
}, },
"question": question, "question": question,
"excerpts": list(delivered), "excerpts": list(delivered),
"withheld": [{"concept_id": concept_id, "rule": rule} for concept_id, rule in withheld], # Emitted only under the flag, and then only where the concept carries
# a title, so a bundle whose concepts have none produces the same bytes
# either way. See this function's docstring for the measurement that
# keeps the default off.
"withheld": [
{"concept_id": concept_id, "rule": rule}
| (
{"title": titles_by_id[concept_id]}
if withheld_titles and titles_by_id.get(concept_id)
else {}
)
for concept_id, rule in withheld
],
} }
@ -1523,6 +1595,26 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
"one 96->103 worse. See docs/2026-09-08-sjeldenhetsvekt.md" "one 96->103 worse. See docs/2026-09-08-sjeldenhetsvekt.md"
), ),
) )
parser.add_argument(
"--tie-shared-rank",
action="store_true",
help=(
"let concepts a signal scores EQUALLY share that score group's "
"first rank, so a signal that separates nothing contributes the "
"same constant to each of them instead of ordering them by id. OFF "
"by default. See docs/2026-09-08-rangeringsbom-sammensatte-ord.md"
),
)
parser.add_argument(
"--withheld-titles",
action="store_true",
help=(
"give each withheld entry the concept's title, so a reader can see "
"WHAT was withheld without reading the bundle. OFF by default: it "
"grew a 270-concept payload by 37.9 %% and pushed a 629-concept "
"bundle's bookkeeping past the budget limit itself"
),
)
parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout") parser.add_argument("--out", type=Path, default=None, help="write here instead of stdout")
parser.add_argument( parser.add_argument(
"--ref", "--ref",
@ -1557,6 +1649,8 @@ def main(argv: list[str] | None = None) -> int:
cost_vocabulary=args.cost_vocabulary, cost_vocabulary=args.cost_vocabulary,
reserve_top_rank=args.reserve_top_rank, reserve_top_rank=args.reserve_top_rank,
rarity_weight=args.rarity_weight, rarity_weight=args.rarity_weight,
tie_shared_rank=args.tie_shared_rank,
withheld_titles=args.withheld_titles,
) )
except ConsumeError as error: except ConsumeError as error:
print(f"okf_consume: FAILED - {error}", file=sys.stderr) print(f"okf_consume: FAILED - {error}", file=sys.stderr)