llm-ingestion-okf/docs/2026-09-07-okf-konsumskill-maaling.md

# The consumption pre-pass, measured — hit@8 on a 629-concept bundle

**Date:** 2026-09-07 · **Instrument:** `tools/okf_consume.py` at `51735fa`,
scored by `tools/okf_consume_measure.py` · **Corpus:** a 629-concept, 39-document
OKF v0.2 segmented bundle built by this library's own doors on 2026-09-03.

This is the first time `docs/consumption-contract.md` has been run against a
bundle. The contract has existed since S5 with no instrument, and a rule with no
instrument is a rule nobody can be shown to have broken. What follows is that
instrument's first numbers, with the two things a first number needs: the
denominator it was measured over, and the chance baseline it should be read
against.

**Read the headline with its granularity attached.** The order this work came
from asked for "≥ 5 spørsmål der **fasit-konseptet** er kjent" — gold at
**concept** granularity. The operator answered the open question on it
(2026-09-07 04:33Z) with option (a): gold is the top-level **document** the
question names, taken from the corpus's own document titles and never from
anything the ranker computes. **Five of the six rows below are therefore
DOCUMENT-granularity, which is a weaker claim than the order asked for.** One row
— the price form — has a gold document holding exactly one concept, so document
and concept granularity coincide there, and it is the only concept-level result
in this report.

## The result

| figure | value |
|---|---|
| questions | 6 positive + 2 known-negative controls |
| **hit@8** | **5 of 6**, every hit at **rank 1** |
| expected by chance | **1.35 of 6** |
| denominator | **629 concepts** per question, every one accounted for |
| `k` | 8 |
| known-negative controls | **2 of 2 FAILED** — see below |
| `spent` per question | 17 970 – 74 838 bytes (median 20 182), limit 120 000 |
| whole payload per question | 109 951 – 165 109 bytes |
| wall time per question | 0.51 – 0.56 s (629 concepts, 1.95 MB of concept text) |

Five hits at rank 1 against an expectation of 1.35 is a real signal and a small
one: **n = 6**. Nothing here supports a claim about the ranker's behaviour on a
seventh question, and this report makes none.

### Per-row, with the baseline each row must be read against

The per-question gold ids are **not in this file**. They name documents in a
consumer's corpus, and this repository keeps corpus paths and document titles out
of tracked files; they live in the local project directory beside the run, and go
to `portfolio-optimiser` by coord. What is publishable is the shape:

| row | gold concepts | hit@8 | rank | chance, analytic | chance, empirical |
|---|---|---|---|---|---|
| 1 — **the concept-level row** | 1 | ✅ | 1 | 0.013 | 0.013 |
| 2 | 49 | ✅ | 1 | 0.479 | 0.483 |
| 3 | 20 | ✅ | 1 | 0.229 | 0.231 |
| 4 | 43 | ✅ | 1 | 0.434 | 0.438 |
| 5 | 11 | ✅ | 1 | 0.132 | 0.134 |
| 6 | 5 | ❌ | — | 0.062 | 0.063 |

**Why every row carries its own baseline.** A document-prefix hit is *easier for
a large gold document*: with gold sets spanning 1 to 49 concepts, a raw count of
4/6 could be produced by a ranker that had learned nothing. Row 2 alone is 48 %
likely by chance; row 1 is 1.3 %. The two hardest rows (1 and 6, at 0.013 and
0.062) are one hit and one miss.

The empirical baseline is 20 000 draws at seed 20260907. It agrees with the
analytic form to within about half a percentage point — **not** to three decimal
places, and this report does not claim it does.

**The miss, diagnosed rather than absorbed.** The gold's best concept ranked
**11 overall**, just outside `k = 8`, and comes back at rank 11 for `k = 12` and
`k = 16`. Cause: the profile matches two tokens when they share four leading
characters, so a compound question term matches two much larger sibling
documents in the same subject area, and those two fill all eight slots. The
document prior does not rescue it (0.333 against 0.433 and 0.415). This is a
near-miss at the reported `k`, not an absence — and it is the shape of failure to
expect from this ranker: crowding by large same-prefix neighbours.

## The controls that failed, stated first rather than last

The order required a known-negative: a question with no answer in the bundle
must produce an empty or "not found" payload, never a confident guess. **Both
known-negative questions produced eight excerpts.** The control failed.

| control | tokens reaching nothing | delivered |
|---|---|---|
| the question chosen as planned | 2 of 6 | 8 |
| the most lexically disjoint question found | 4 of 6 | 8 |

**Why, measured rather than guessed.** The pre-pass has a per-concept relevance
rule (`no_lexical_match`) that withholds a concept the question does not reach,
and it works: it fires on 600-odd concepts per run. It does **not** work as a
whole-question gate, because no natural Norwegian question is lexically disjoint
from a 629-concept corpus under a four-character shared-prefix rule. Measured
per token: the interrogative `hvor` reaches 40 concepts, `mange` 17, `brukes` 83,
`etablert` 38. Four candidate questions were probed and none reached zero on
every token. The first control's content words did genuinely match — one of them
reached 17 concepts describing welded seams in floor covering, which is a correct
lexical answer and a useless topical one.

**Consequence for anyone using this, stated in the skill as well as here: an
empty `excerpts` list is evidence of absence; a full one is not evidence of
presence.** When the delivered excerpts do not answer the question, the right
output is `[sourced-not-sufficient]` plus a statement that the cut found nothing
responsive — not eight answers.

**What would fix it, named and not built.** Rarity weighting: a token matching a
large fraction of the corpus carries little information, and requiring a
delivered concept to match at least one *informative* token would make the gate
work. Smoothed IDF is on the plan's own list of mechanisms worth lifting and
this build does not implement it. It is not attempted here because this step's
scope fence freezes the instrument before it is measured — tuning a ranker
against the questions it is being scored on produces a number about the tuning.

## Rarity weighting, measured before being built — the named mechanism does not exist

**Date:** 2026-09-07, follow-up order `20260907T074541Z-939882634-from-.claude`.
The order named a specific mechanism — smoothed IDF over the corpus the walk
already reads, gating `no_lexical_match` so a delivered concept must match at
least one *informative* token — and required both known-negative controls
above to reach `delivered == 0` on the real K2 bundle. **This is a refutation
sweep, not a tuning sweep**: every threshold in the full observed range is
reported, not a value chosen to pass. No threshold was picked to make the
negatives zero; the table below is what falsifies that a passing threshold
exists at all, and the order is returned on that basis rather than implemented.

**Method.** `idf(t) = ln((N+1)/(df(t)+1)) + 1`, `N = 629`. `df(t)` counted with
the *same* four-character prefix rule `tokens_match` already applies (the
instrument `okf_consume_measure.token_reach` already uses), over
title + concept-id + body across all 629 concepts. For every concept, `M` is the
highest `idf` among the question tokens that concept matches at all (or none,
if the concept matches nothing). A candidate threshold `T` keeps a concept only
when `M > T`. Swept `T` from 1.5 to 7.9 in steps of 0.1 — the corpus's full
observed range for these two questions' tokens.

| `T` | neg1 (sveising) passing | neg2 (pingviner) passing | gold hits still reachable (of 6) |
|---|---|---|---|
| 1.5 – 2.5 | 31 | 49 | 6 |
| 2.6 – 3.5 | 31 | 49 | 4 (brann, branntegninger lost) |
| 3.6 – 3.7 | 31 | 49 | 3 (akustikk lost) |
| 3.8 – 4.2 | 31 | 17 | 3 |
| 4.3 | 31 | 17 | 2 (solcelle lost) |
| 4.4 – 4.5 | 31 | 17 | 1 (prisene only) |
| **4.6 – 5.0** | **14** | **0** | **0** |
| 5.1 – 6.0 | 5 | 0 | 0 |
| 6.1 – 6.3 | 2 | 0 | 0 |
| **6.4 – 7.9** | **0** | **0** | **0** |

**The frontier is empty, and it closes from the wrong side.** neg2 first
reaches 0 at `T = 4.6` — and at that *exact same* `T`, gold-hit reachability has
already fallen to **0 of 6**: no positive question's gold document is reachable
any more. neg1 has not moved yet at `T = 4.6` (still 14 concepts passing —
`titan` idf 6.059, `vakuum` idf 6.347, `reglene` idf 5.048, `sveising` idf 4.555:
the two rarest tokens in the entire gold set belong to the sveising question,
so rarity weighting scores them as maximally informative) and does not reach 0
until `T = 6.4`, by which point gold-hit reachability has been at 0 for eighteen
threshold steps. There is no `T` where both controls read 0 while any positive
question still reaches its gold document — the interval where neg2 is
satisfied and the interval where neg1 is satisfied do not overlap, and the
positive constraint is already violated before either control interval begins.

**Why, structurally, not just numerically.**

1. **`df` measured over the colliding matcher measures collision breadth, not
   rarity.** The four-character prefix rule that lets `varene` match
   `varemottak` also collapses every `brann*` compound into one bucket:
   `brannkonseptet` reads `df=132`, reaching 18 of the corpus's 39 documents,
   because `brannalarm`, `branntrapp`, `branncelle` and `branntetting` all
   share the same four leading characters and none of them is the same word.
   Rarity weighting inherits this pollution rather than fixing it — the
   *most* topical word in the corpus for the brann question reads as the
   *least* rare.
2. **`MIN_TOKEN_LENGTH = 3` admits tokens `MIN_SHARED_PREFIX = 4` guarantees
   can never match.** `hva`, `for`, `den` are three characters; `tokens_match`
   requires the shorter side to reach the four-character floor before any
   comparison runs, so these tokens are structurally `df = 0` regardless of
   what the corpus contains. The brann question therefore rests on its single
   remaining token (`brannkonseptet`) the moment `sier` — which matches three
   concepts, none in the gold document — is set aside.
3. **The sveising question's match is a real word, not noise.** `sveising`
   matches `sveisede` in eight architectural-drawing concepts describing
   welded floor-covering seams — correct lexically, unrelated to the
   question's actual subject. No token-rarity statistic distinguishes this
   from a genuine topical match, because by the rarity measure it *is* one
   (`df = 17`, concentrated in a single document, `doc_reach = 1/39`).

**What is not built.** A coverage rule — require **two or more** informative
tokens to match, rather than any one — probably zeros both controls, since
neither negative question has two tokens that jointly land in the same small
concept set. It is named and explicitly **not implemented here**: choosing it
now would be picking a mechanism because it was checked against these two
answers, which is exactly what this order's own fence forbids. It goes to the
operator as an option, unmeasured.

## Honesty limits

Six, and the first three matter most.

1. **n = 6.** Six questions is a small sample and the table says so on every
   reading. The chance baseline is what keeps it from being meaningless, not
   what makes it large.
2. **The question texts were written after the ranker existed.** The plan fixed
   the gold documents' *size profile* before any ranker existed — its recorded
   per-row baselines sum to 1.35, and the sizes used here reproduce that total
   exactly, which is an independent check that this is the gold set the plan
   profiled. But the plan recorded no question **texts**, so those were written
   during execution. Three of the six gold documents also could not be pinned
   uniquely from the recorded sizes and were chosen among equal-sized
   candidates. The gold documents themselves came from the corpus's own titles,
   per the operator's answer, and not from anything the ranker computes — but
   this is not a blind evaluation and must not be quoted as one.
3. **One ranking change was made with a gold answer visible.** The stage-one
   document prior originally *summed* lexical overlap across a document's units,
   which measures document size: measured on this corpus, a 79-concept document
   scored 6.0 (0.076 each) against the price document's 2.0 over 1 concept, and
   the price gold fell outside the top 8. The prior is now a density. The fix is
   justified by the scoring function's own arithmetic rather than by the answer
   — a prior that grows with size is measuring size — but it was found by running
   row 1, and the ranker is therefore not blind to that row.
4. **Five of six rows are DOCUMENT granularity**, not the concept granularity
   the order asked for. Row 1 is the exception and is labelled throughout.
5. **Two of the three states this contract makes consumers distinguish have
   denominator zero in this corpus.** `adjudicated` never occurs (all 618 present
   values are `proposed`; 11 concepts carry no key and read as `unknown`), and
   `verified` is absent on all 629 — anchored and unanchored — so every excerpt
   is `unverified`. `type: verdict` is likewise absent on all 629, which makes
   the § 9.1 exclusion **vacuous here**. All three are exercised only against a
   synthetic fixture, and the skill states that rather than implying coverage.
6. **The contract checker is a lint, not the acceptance.** Measured: it returns
   0 findings on a payload declaring `considered=0, withheld=0, delivered=0`
   paired with the *unfilled* template. A green checker means the payload carries
   what a claim would have to rest on — not that the claim is sound. The real
   gates are the negative controls, the partition assertion, the placeholder scan
   and the digest test, and none of those is something the checker performs.

## What the contract's own gates reported

| gate | result |
|---|---|
| SS 5.2, `considered == withheld + delivered` | closes on every run, by construction |
| SS 5.3, every withheld concept names its rule | 621 of 621 on the row-1 run, from a closed set of six |
| SS 7.4, the instrument reproduced a known-positive before reporting | 10 349 expected, 10 349 measured, plus a second route (`wc -c` = 10 060, delta 289) |
| SS 7.2, `spent` is the **delivered set** | 74 838 spent against a 120 000 limit on row 1 — where a whole-payload reading would have put 165 109 against the same limit and refused |
| SS 3.3, the ref is a content identity, not a declared version | `sha256-tree:` over the reachable tree; `--ref` asserts and never overrides |
| SS 9.2, no directory enumerated | the index walk reaches 629 concepts, exactly the set the forbidden method finds |
| contract check on a real payload | `conformant: 14 rules over 8 excerpts and 621 withheld entries, 0 findings`, exit 0 |
| negative control: broken identity | `NOT conformant`, 2 findings, exit 1 |
| negative control: missing payload | exit 2 |

## Reproduce

```sh
# One payload, and the checker on it.
.venv/bin/python tools/okf_consume.py <bundle> \
  --question "Hvordan skal prisene fylles ut?" --out payload.json
.venv/bin/python tools/okf_contract_check.py \
  --skill skills/okf-consume/SKILL.md --payload payload.json

# The whole table. The gold set is an INPUT and is not in this repository.
.venv/bin/python tools/okf_consume_measure.py <bundle> --questions <gold-set.json>
```

The gold-set file's shape is in `tools/okf_consume_measure.py`'s module
docstring. Determinism is asserted by the suite, not by inspection: two runs of
the same `(bundle, question, k, limit)` produce byte-identical output, and the
module reaches no clock.