llm-ingestion-okf/docs/2026-09-07-okf-konsumskill-maaling.md

The consumption pre-pass, measured — hit@8 on a 629-concept bundle

Date: 2026-09-07 · Instrument: tools/okf_consume.py at 51735fa, scored by tools/okf_consume_measure.py · Corpus: a 629-concept, 39-document OKF v0.2 segmented bundle built by this library's own doors on 2026-09-03.

This is the first time docs/consumption-contract.md has been run against a bundle. The contract has existed since S5 with no instrument, and a rule with no instrument is a rule nobody can be shown to have broken. What follows is that instrument's first numbers, with the two things a first number needs: the denominator it was measured over, and the chance baseline it should be read against.

Read the headline with its granularity attached. The order this work came from asked for "≥ 5 spørsmål der fasit-konseptet er kjent" — gold at concept granularity. The operator answered the open question on it (2026-09-07 04:33Z) with option (a): gold is the top-level document the question names, taken from the corpus's own document titles and never from anything the ranker computes. Five of the six rows below are therefore DOCUMENT-granularity, which is a weaker claim than the order asked for. One row — the price form — has a gold document holding exactly one concept, so document and concept granularity coincide there, and it is the only concept-level result in this report.

The result

figure	value
questions	6 positive + 2 known-negative controls
hit@8	5 of 6, every hit at rank 1
expected by chance	1.35 of 6
denominator	629 concepts per question, every one accounted for
k	8
known-negative controls	2 of 2 FAILED — see below
spent per question	17 970 – 74 838 bytes (median 20 182), limit 120 000
whole payload per question	109 951 – 165 109 bytes
wall time per question	0.51 – 0.56 s (629 concepts, 1.95 MB of concept text)

Five hits at rank 1 against an expectation of 1.35 is a real signal and a small one: n = 6. Nothing here supports a claim about the ranker's behaviour on a seventh question, and this report makes none.

Per-row, with the baseline each row must be read against

The per-question gold ids are not in this file. They name documents in a consumer's corpus, and this repository keeps corpus paths and document titles out of tracked files; they live in the local project directory beside the run, and go to portfolio-optimiser by coord. What is publishable is the shape:

row	gold concepts	hit@8	rank	chance, analytic	chance, empirical
1 — the concept-level row	1	✅	1	0.013	0.013
2	49	✅	1	0.479	0.483
3	20	✅	1	0.229	0.231
4	43	✅	1	0.434	0.438
5	11	✅	1	0.132	0.134
6	5	❌	—	0.062	0.063

Why every row carries its own baseline. A document-prefix hit is easier for a large gold document: with gold sets spanning 1 to 49 concepts, a raw count of 4/6 could be produced by a ranker that had learned nothing. Row 2 alone is 48 % likely by chance; row 1 is 1.3 %. The two hardest rows (1 and 6, at 0.013 and 0.062) are one hit and one miss.

The empirical baseline is 20 000 draws at seed 20260907. It agrees with the analytic form to within about half a percentage point — not to three decimal places, and this report does not claim it does.

The miss, diagnosed rather than absorbed. The gold's best concept ranked 11 overall, just outside k = 8, and comes back at rank 11 for k = 12 and k = 16. Cause: the profile matches two tokens when they share four leading characters, so a compound question term matches two much larger sibling documents in the same subject area, and those two fill all eight slots. The document prior does not rescue it (0.333 against 0.433 and 0.415). This is a near-miss at the reported k, not an absence — and it is the shape of failure to expect from this ranker: crowding by large same-prefix neighbours.

The controls that failed, stated first rather than last

The order required a known-negative: a question with no answer in the bundle must produce an empty or "not found" payload, never a confident guess. Both known-negative questions produced eight excerpts. The control failed.

control	tokens reaching nothing	delivered
the question chosen as planned	2 of 6	8
the most lexically disjoint question found	4 of 6	8

Why, measured rather than guessed. The pre-pass has a per-concept relevance rule (no_lexical_match) that withholds a concept the question does not reach, and it works: it fires on 600-odd concepts per run. It does not work as a whole-question gate, because no natural Norwegian question is lexically disjoint from a 629-concept corpus under a four-character shared-prefix rule. Measured per token: the interrogative hvor reaches 40 concepts, mange 17, brukes 83, etablert 38. Four candidate questions were probed and none reached zero on every token. The first control's content words did genuinely match — one of them reached 17 concepts describing welded seams in floor covering, which is a correct lexical answer and a useless topical one.

Consequence for anyone using this, stated in the skill as well as here: an empty excerpts list is evidence of absence; a full one is not evidence of presence. When the delivered excerpts do not answer the question, the right output is [sourced-not-sufficient] plus a statement that the cut found nothing responsive — not eight answers.

What would fix it, named and not built. Rarity weighting: a token matching a large fraction of the corpus carries little information, and requiring a delivered concept to match at least one informative token would make the gate work. Smoothed IDF is on the plan's own list of mechanisms worth lifting and this build does not implement it. It is not attempted here because this step's scope fence freezes the instrument before it is measured — tuning a ranker against the questions it is being scored on produces a number about the tuning.

Honesty limits

Six, and the first three matter most.

1. n = 6. Six questions is a small sample and the table says so on every reading. The chance baseline is what keeps it from being meaningless, not what makes it large.
2. The question texts were written after the ranker existed. The plan fixed the gold documents' size profile before any ranker existed — its recorded per-row baselines sum to 1.35, and the sizes used here reproduce that total exactly, which is an independent check that this is the gold set the plan profiled. But the plan recorded no question texts, so those were written during execution. Three of the six gold documents also could not be pinned uniquely from the recorded sizes and were chosen among equal-sized candidates. The gold documents themselves came from the corpus's own titles, per the operator's answer, and not from anything the ranker computes — but this is not a blind evaluation and must not be quoted as one.
3. One ranking change was made with a gold answer visible. The stage-one document prior originally summed lexical overlap across a document's units, which measures document size: measured on this corpus, a 79-concept document scored 6.0 (0.076 each) against the price document's 2.0 over 1 concept, and the price gold fell outside the top 8. The prior is now a density. The fix is justified by the scoring function's own arithmetic rather than by the answer — a prior that grows with size is measuring size — but it was found by running row 1, and the ranker is therefore not blind to that row.
4. Five of six rows are DOCUMENT granularity, not the concept granularity the order asked for. Row 1 is the exception and is labelled throughout.
5. Two of the three states this contract makes consumers distinguish have denominator zero in this corpus. adjudicated never occurs (all 618 present values are proposed; 11 concepts carry no key and read as unknown), and verified is absent on all 629 — anchored and unanchored — so every excerpt is unverified. type: verdict is likewise absent on all 629, which makes the § 9.1 exclusion vacuous here. All three are exercised only against a synthetic fixture, and the skill states that rather than implying coverage.
6. The contract checker is a lint, not the acceptance. Measured: it returns 0 findings on a payload declaring considered=0, withheld=0, delivered=0 paired with the unfilled template. A green checker means the payload carries what a claim would have to rest on — not that the claim is sound. The real gates are the negative controls, the partition assertion, the placeholder scan and the digest test, and none of those is something the checker performs.

What the contract's own gates reported

gate	result
SS 5.2, considered == withheld + delivered	closes on every run, by construction
SS 5.3, every withheld concept names its rule	621 of 621 on the row-1 run, from a closed set of six
SS 7.4, the instrument reproduced a known-positive before reporting	10 349 expected, 10 349 measured, plus a second route (wc -c = 10 060, delta 289)
SS 7.2, spent is the delivered set	74 838 spent against a 120 000 limit on row 1 — where a whole-payload reading would have put 165 109 against the same limit and refused
SS 3.3, the ref is a content identity, not a declared version	sha256-tree: over the reachable tree; --ref asserts and never overrides
SS 9.2, no directory enumerated	the index walk reaches 629 concepts, exactly the set the forbidden method finds
contract check on a real payload	conformant: 14 rules over 8 excerpts and 621 withheld entries, 0 findings, exit 0
negative control: broken identity	NOT conformant, 2 findings, exit 1
negative control: missing payload	exit 2

Reproduce

# One payload, and the checker on it.
.venv/bin/python tools/okf_consume.py <bundle> \
  --question "Hvordan skal prisene fylles ut?" --out payload.json
.venv/bin/python tools/okf_contract_check.py \
  --skill skills/okf-consume/SKILL.md --payload payload.json

# The whole table. The gold set is an INPUT and is not in this repository.
.venv/bin/python tools/okf_consume_measure.py <bundle> --questions <gold-set.json>

The gold-set file's shape is in tools/okf_consume_measure.py's module docstring. Determinism is asserted by the suite, not by inspection: two runs of the same (bundle, question, k, limit) produce byte-identical output, and the module reaches no clock.