test(consume): hit@8 over six questions against a random-ranker baseline

hit@8 = 5 of 6, every hit at rank 1, against a chance baseline of 1.35 of 6
over a denominator of 629 concepts per question. Wall time 0.51-0.56 s per
question; spent 17 970 - 74 838 bytes against a 120 000 limit.

Two things this measurement did NOT establish, both in the report:

- BOTH known-negative controls FAILED. A question the bundle has no answer to
  still returns eight excerpts, because no natural Norwegian question is
  lexically disjoint from a 629-concept corpus under a four-character
  shared-prefix rule -- measured per token, the interrogative `hvor` reaches 40
  concepts, `brukes` 83. So `no_lexical_match` works per concept and not as a
  whole-question gate: an empty excerpt list is evidence of absence, a full one
  is not evidence of presence. The fix is named (rarity weighting) and NOT
  built, because this step's fence freezes the instrument before it is measured.
- The question texts were written during execution, after the ranker existed.
  The plan recorded the gold documents' SIZE profile -- its per-row baselines
  sum to 1.35 and the sizes used here reproduce that exactly, which is an
  independent check that this is the set the plan profiled -- but it recorded no
  question texts, and three of six gold documents could not be pinned uniquely
  from the sizes. Not a blind evaluation, and the report says so.

The scorer is a tool, not a script in a document: `tools/okf_consume_measure.py`
takes the gold set as an INPUT because it is tracked in a public repository and
an answer key names a consumer's documents. hit_rank, both chance baselines and
the document-size census are unit-tested; the corpus run is a measurement.

Public-file rule, checked with a pattern DERIVED from the corpus's own 39
document names rather than hand-picked, and shown able to find first (67 hits on
the bundle's own index): zero corpus document names in any tracked file in this
repository. One leak was found and removed on the way -- a corpus concept name
in a code comment and a hardcoded corpus path in a test.

Suite run after git add: 1230 passed, mypy --strict clean on 27 files,
ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 09:37:14 +02:00
commit d7751c0b9a
6 changed files with 598 additions and 39 deletions

View file

@ -1,4 +1,4 @@
- [Prissammenstilling](prissammenstilling.md) — adjudication: proposed
- [Pristabell](pristabell.md) — adjudication: proposed
- [Årlig kontroll av anlegget](aarlig-kontroll.md) — adjudication: adjudicated
- [verdict-lookalike](verdict-lookalike.md) — adjudication: proposed
- [Loggnotat](loggnotat.md) — adjudication: proposed

View file

@ -1,7 +1,7 @@
---
type: reference
title: Prissammenstilling
source_file: Prisskjema.xlsx
title: Pristabell
source_file: Pristabell.xlsx
source_sha256: 1111111111111111111111111111111111111111111111111111111111111111
ingested_at: 2026-09-01T00:00:00Z
adjudication: proposed
@ -9,7 +9,7 @@ bundle_id: consume-fixture
verified: [{ by: process:okf-check, at: 2026-09-01T00:00:00Z }]
---
## Prissammenstilling
## Pristabell
Prisene fylles ut i dette skjemaet. Summen av alle poster overfoeres til
tilbudsbrevet.