test(consume): hit@8 over six questions against a random-ranker baseline

hit@8 = 5 of 6, every hit at rank 1, against a chance baseline of 1.35 of 6
over a denominator of 629 concepts per question. Wall time 0.51-0.56 s per
question; spent 17 970 - 74 838 bytes against a 120 000 limit.

Two things this measurement did NOT establish, both in the report:

- BOTH known-negative controls FAILED. A question the bundle has no answer to
  still returns eight excerpts, because no natural Norwegian question is
  lexically disjoint from a 629-concept corpus under a four-character
  shared-prefix rule -- measured per token, the interrogative `hvor` reaches 40
  concepts, `brukes` 83. So `no_lexical_match` works per concept and not as a
  whole-question gate: an empty excerpt list is evidence of absence, a full one
  is not evidence of presence. The fix is named (rarity weighting) and NOT
  built, because this step's fence freezes the instrument before it is measured.
- The question texts were written during execution, after the ranker existed.
  The plan recorded the gold documents' SIZE profile -- its per-row baselines
  sum to 1.35 and the sizes used here reproduce that exactly, which is an
  independent check that this is the set the plan profiled -- but it recorded no
  question texts, and three of six gold documents could not be pinned uniquely
  from the sizes. Not a blind evaluation, and the report says so.

The scorer is a tool, not a script in a document: `tools/okf_consume_measure.py`
takes the gold set as an INPUT because it is tracked in a public repository and
an answer key names a consumer's documents. hit_rank, both chance baselines and
the document-size census are unit-tested; the corpus run is a measurement.

Public-file rule, checked with a pattern DERIVED from the corpus's own 39
document names rather than hand-picked, and shown able to find first (67 hits on
the bundle's own index): zero corpus document names in any tracked file in this
repository. One leak was found and removed on the way -- a corpus concept name
in a code comment and a hardcoded corpus path in a test.

Suite run after git add: 1230 passed, mypy --strict clean on 27 files,
ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tests/fixtures/consume-bundle/krav/index.md

@@ -1,4 +1,4 @@
-- [Prissammenstilling](prissammenstilling.md) — adjudication: proposed
+- [Pristabell](pristabell.md) — adjudication: proposed
 - [Årlig kontroll av anlegget](aarlig-kontroll.md) — adjudication: adjudicated
 - [verdict-lookalike](verdict-lookalike.md) — adjudication: proposed
 - [Loggnotat](loggnotat.md) — adjudication: proposed

tests/fixtures/consume-bundle/krav/pristabell.md

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: Prissammenstilling
-source_file: Prisskjema.xlsx
+title: Pristabell
+source_file: Pristabell.xlsx
 source_sha256: 1111111111111111111111111111111111111111111111111111111111111111
 ingested_at: 2026-09-01T00:00:00Z
 adjudication: proposed
@@ -9,7 +9,7 @@ bundle_id: consume-fixture
 verified: [{ by: process:okf-check, at: 2026-09-01T00:00:00Z }]
 ---
 
-## Prissammenstilling
+## Pristabell
 
 Prisene fylles ut i dette skjemaet. Summen av alle poster overfoeres til
 tilbudsbrevet.

tests/test_okf_consume.py

@@ -37,6 +37,7 @@ PROJECT_ROOT = Path(__file__).resolve().parents[1]
 sys.path.insert(0, str(PROJECT_ROOT / "tools"))
 
 import okf_consume  # noqa: E402
+import okf_consume_measure  # noqa: E402
 import okf_contract_check  # noqa: E402
 
 from llm_ingestion_okf.materialize import parse_frontmatter  # noqa: E402
@@ -425,8 +426,8 @@ def test_a_concept_in_a_high_scoring_document_outranks_an_equally_lexical_one()
     question = "Hvordan skal prisene fylles ut?"
     lifted = okf_consume.concept_scores(concepts, question, {"krav": 10.0, "dyp": 0.0})
     dropped = okf_consume.concept_scores(concepts, question, {"krav": 0.0, "dyp": 10.0})
-    krav_first = [c.concept_id for c, _, _ in lifted].index("krav/prissammenstilling")
-    krav_later = [c.concept_id for c, _, _ in dropped].index("krav/prissammenstilling")
+    krav_first = [c.concept_id for c, _, _ in lifted].index("krav/pristabell")
+    krav_later = [c.concept_id for c, _, _ in dropped].index("krav/pristabell")
     assert krav_first < krav_later
 
 
@@ -442,7 +443,7 @@ def test_the_price_concept_leads_on_the_price_question_in_the_fixture() -> None:
     concepts = _fixture_concepts()
     scores = okf_consume.document_scores(FIXTURE, "Hvordan skal prisene fylles ut?")
     ranked = okf_consume.concept_scores(concepts, "Hvordan skal prisene fylles ut?", scores)
-    assert ranked[0][0].concept_id == "krav/prissammenstilling"
+    assert ranked[0][0].concept_id == "krav/pristabell"
 
 
 # --- Step 7: the cut ----------------------------------------------------------
@@ -562,7 +563,7 @@ def test_excerpts_come_back_in_rank_order_and_carry_that_rank() -> None:
     # second one.
     delivered, _, _ = _cut_fixture()
     assert [excerpt["rank"] for excerpt in delivered] == list(range(1, len(delivered) + 1))
-    assert delivered[0]["concept_id"] == "krav/prissammenstilling"
+    assert delivered[0]["concept_id"] == "krav/pristabell"
 
 
 # --- Step 8: the payload ------------------------------------------------------
@@ -765,16 +766,33 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
     assert int(budget["spent"]) <= int(budget["limit"])
 
 
+#: The gold set is LOCAL-ONLY: it names corpus documents, which never reach a
+#: tracked file here. The test reads it rather than restating it, so this file
+#: carries the assertion and not the answer key.
+GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
+
+
 @requires_k2
-def test_the_price_form_gold_is_delivered_for_the_price_question() -> None:
-    # SC6. The one question whose gold is confirmed by a signal from outside
-    # this repository: a live model reached that directory unprompted in three
-    # navigation steps on 2026-09-06.
-    payload = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
-    excerpts = payload["excerpts"]
-    assert isinstance(excerpts, list)
-    ids = [str(excerpt["concept_id"]) for excerpt in excerpts]
-    assert "del-ii-bilag-7-prisskjema/prissammenstilling-sheet-1" in ids
+@pytest.mark.skipif(not GOLD_SET.is_file(), reason=f"the local gold set is absent ({GOLD_SET})")
+def test_every_gold_document_in_the_local_set_is_reached_or_named_as_a_miss() -> None:
+    # SC5 and SC6 together, run against the answer key rather than a literal.
+    # Row 1's gold is the one confirmed by a signal from outside this
+    # repository -- a live model reached that document unprompted in three
+    # navigation steps on 2026-09-06 -- and its gold document holds exactly one
+    # concept, so it is also the one concept-granularity row.
+    spec = json.loads(GOLD_SET.read_text(encoding="utf-8"))
+    questions = spec["questions"]
+    assert len(questions) >= 5, "fewer than five questions is not the measurement"
+    hits = 0
+    for entry in questions:
+        payload = okf_consume.build_payload(K2_BUNDLE, question=entry["question"])
+        excerpts = payload["excerpts"]
+        assert isinstance(excerpts, list)
+        if okf_consume_measure.hit_rank(excerpts, entry["gold_document"]) is not None:
+            hits += 1
+    # The published bar, and the published number. A regression that drops a
+    # row goes red here rather than in a document nobody re-runs.
+    assert hits == 5, f"hit@8 moved: {hits} of {len(questions)}"
 
 
 # --- Step 9: the CLI ----------------------------------------------------------
@@ -970,22 +988,34 @@ def test_the_shipped_example_payload_is_current_and_regenerates_byte_for_byte()
     assert shipped == regenerated
 
 
-def test_no_k2_concept_path_or_document_title_reaches_the_tracked_skill() -> None:
-    # CLAUDE.md's public-file rule, with the pattern widened to the bare
-    # basenames a report is most likely to leak, and shown capable of finding
-    # against the bundle's own index before its zero here is believed.
-    leak = re.compile(
-        r"del-ii-bilag|del-i-vedlegg|del-i-konkurranse|prisskjema|prissammenstilling|stange",
-        re.IGNORECASE,
+@requires_k2
+def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
+    # CLAUDE.md's public-file rule. The pattern is DERIVED from the corpus's own
+    # top-level document names at run time rather than hand-picked, so it covers
+    # every document rather than the six someone thought of -- and so this
+    # tracked file carries no corpus name of its own.
+    documents = sorted(
+        {concept_id.split("/", 1)[0] for concept_id in okf_consume.enumerate_concepts(K2_BUNDLE)}
     )
+    assert len(documents) > 30, "too few documents to be the real corpus"
+    leak = re.compile("|".join(re.escape(name) for name in documents), re.IGNORECASE)
+
+    # The known-positive, first: the pattern must be shown able to find before
+    # its zero counts as a measurement.
+    control = (K2_BUNDLE / "index.md").read_text(encoding="utf-8")
+    assert leak.findall(control), "the pattern cannot find; the zeros below would mean nothing"
+
     tracked = [
         SKILL,
         SKILL.parent / "references" / "README.md",
         SKILL.parent / "references" / "example-payload.json",
+        PROJECT_ROOT / "tools" / "okf_consume.py",
+        PROJECT_ROOT / "tools" / "okf_consume_measure.py",
+        PROJECT_ROOT / "tests" / "test_okf_consume.py",
+        PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md",
+        PROJECT_ROOT / "README.md",
+        PROJECT_ROOT / "CLAUDE.md",
     ]
-    if K2_BUNDLE.is_dir():
-        control = (K2_BUNDLE / "index.md").read_text(encoding="utf-8")
-        assert leak.findall(control), "the pattern cannot find; its zero below would mean nothing"
     for path in tracked:
         assert leak.findall(path.read_text(encoding="utf-8")) == [], path
 
@@ -997,3 +1027,56 @@ def test_the_readme_consume_section_states_the_rule_count_the_code_emits() -> No
     assert len(okf_consume.WITHHOLDING_RULES) == 6
     assert "closed set of six" in readme
     assert "tools/okf_consume.py" in readme
+
+
+# --- Step 11: the measurement scorer -----------------------------------------
+
+
+def test_a_gold_at_rank_three_is_a_hit_at_eight_and_a_miss_at_two() -> None:
+    excerpts = [
+        {"concept_id": "other-doc/a", "rank": 1},
+        {"concept_id": "other-doc/b", "rank": 2},
+        {"concept_id": "gold-doc/c", "rank": 3},
+    ]
+    assert okf_consume_measure.hit_rank(excerpts, "gold-doc") == 3
+    assert okf_consume_measure.hit_rank(excerpts[:2], "gold-doc") is None
+
+
+def test_the_document_prefix_test_does_not_match_a_longer_document_name() -> None:
+    # Without the trailing slash, a gold document `bilag-7` would count a
+    # concept in `bilag-70` as a hit -- a false positive that inflates the
+    # headline number and looks like a correct answer.
+    excerpts = [{"concept_id": "bilag-70/a", "rank": 1}]
+    assert okf_consume_measure.hit_rank(excerpts, "bilag-7") is None
+    assert okf_consume_measure.hit_rank([{"concept_id": "bilag-7", "rank": 1}], "bilag-7") == 1
+
+
+def test_the_analytic_chance_baseline_is_exact_for_a_single_gold_concept() -> None:
+    # One gold concept in a 629-concept corpus, drawn 8: exactly 8/629, which
+    # is derivable by hand and pins the closed form.
+    assert okf_consume_measure.chance_analytic(1, 629, 8) == pytest.approx(8 / 629)
+    assert okf_consume_measure.chance_analytic(629, 629, 8) == 1.0
+    assert okf_consume_measure.chance_analytic(0, 629, 8) == 0.0
+
+
+def test_the_empirical_baseline_reproduces_the_analytic_one_on_a_fixed_seed() -> None:
+    # Two routes to one number. The agreement is about half a percentage point
+    # at 20 000 trials -- asserted at that tolerance rather than at the three
+    # decimal places the plan claimed, because the looser figure is the one
+    # that is true.
+    for gold in (1, 5, 11, 20, 43, 49):
+        analytic = okf_consume_measure.chance_analytic(gold, 629, 8)
+        empirical = okf_consume_measure.chance_empirical(gold, 629, 8)
+        assert abs(analytic - empirical) < 0.01, gold
+
+
+def test_document_sizes_counts_root_level_concepts_as_their_own_document() -> None:
+    sizes = okf_consume_measure.document_sizes(("a/1", "a/2", "b/1", "root-concept"))
+    assert sizes == {"a": 2, "b": 1, "root-concept": 1}
+
+
+def test_the_measurement_instrument_names_no_corpus_document() -> None:
+    # It is tracked and public; the answer key is an input, never a constant.
+    source = (PROJECT_ROOT / "tools" / "okf_consume_measure.py").read_text(encoding="utf-8")
+    leak = re.compile(r"del-ii-bilag|del-i-vedlegg|del-i-konkurranse|prisskjema|stange", re.I)
+    assert leak.findall(source) == []