open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

test(consume): hit@8 over six questions against a random-ranker baseline

Browse source 
hit@8 = 5 of 6, every hit at rank 1, against a chance baseline of 1.35 of 6
over a denominator of 629 concepts per question. Wall time 0.51-0.56 s per
question; spent 17 970 - 74 838 bytes against a 120 000 limit.

Two things this measurement did NOT establish, both in the report:

- BOTH known-negative controls FAILED. A question the bundle has no answer to
  still returns eight excerpts, because no natural Norwegian question is
  lexically disjoint from a 629-concept corpus under a four-character
  shared-prefix rule -- measured per token, the interrogative `hvor` reaches 40
  concepts, `brukes` 83. So `no_lexical_match` works per concept and not as a
  whole-question gate: an empty excerpt list is evidence of absence, a full one
  is not evidence of presence. The fix is named (rarity weighting) and NOT
  built, because this step's fence freezes the instrument before it is measured.
- The question texts were written during execution, after the ranker existed.
  The plan recorded the gold documents' SIZE profile -- its per-row baselines
  sum to 1.35 and the sizes used here reproduce that exactly, which is an
  independent check that this is the set the plan profiled -- but it recorded no
  question texts, and three of six gold documents could not be pinned uniquely
  from the sizes. Not a blind evaluation, and the report says so.

The scorer is a tool, not a script in a document: `tools/okf_consume_measure.py`
takes the gold set as an INPUT because it is tracked in a public repository and
an answer key names a consumer's documents. hit_rank, both chance baselines and
the document-size census are unit-tested; the corpus run is a measurement.

Public-file rule, checked with a pattern DERIVED from the corpus's own 39
document names rather than hand-picked, and shown able to find first (67 hits on
the bundle's own index): zero corpus document names in any tracked file in this
repository. One leak was found and removed on the way -- a corpus concept name
in a code comment and a hardcoded corpus path in a test.

Suite run after git add: 1230 passed, mypy --strict clean on 27 files,
ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 09:37:14 +02:00
commit d7751c0b9a
6 changed files with 598 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

185
docs/2026-09-07-okf-konsumskill-maaling.md Normal file
View file

@ -0,0 +1,185 @@
# The consumption pre-pass, measured — hit@8 on a 629-concept bundle
**Date:** 2026-09-07 · **Instrument:** `tools/okf_consume.py` at `51735fa`,
scored by `tools/okf_consume_measure.py` · **Corpus:** a 629-concept, 39-document
OKF v0.2 segmented bundle built by this library's own doors on 2026-09-03.
This is the first time `docs/consumption-contract.md` has been run against a
bundle. The contract has existed since S5 with no instrument, and a rule with no
instrument is a rule nobody can be shown to have broken. What follows is that
instrument's first numbers, with the two things a first number needs: the
denominator it was measured over, and the chance baseline it should be read
against.
**Read the headline with its granularity attached.** The order this work came
from asked for "≥ 5 spørsmål der **fasit-konseptet** er kjent" — gold at
**concept** granularity. The operator answered the open question on it
(2026-09-07 04:33Z) with option (a): gold is the top-level **document** the
question names, taken from the corpus's own document titles and never from
anything the ranker computes. **Five of the six rows below are therefore
DOCUMENT-granularity, which is a weaker claim than the order asked for.** One row
— the price form — has a gold document holding exactly one concept, so document
and concept granularity coincide there, and it is the only concept-level result
in this report.
## The result
| figure | value |
|---|---|
| questions | 6 positive + 2 known-negative controls |
| **hit@8** | **5 of 6**, every hit at **rank 1** |
| expected by chance | **1.35 of 6** |
| denominator | **629 concepts** per question, every one accounted for |
| `k` | 8 |
| known-negative controls | **2 of 2 FAILED** — see below |
| `spent` per question | 17 970 – 74 838 bytes (median 20 182), limit 120 000 |
| whole payload per question | 109 951 – 165 109 bytes |
| wall time per question | 0.51 – 0.56 s (629 concepts, 1.95 MB of concept text) |
Five hits at rank 1 against an expectation of 1.35 is a real signal and a small
one: **n = 6**. Nothing here supports a claim about the ranker's behaviour on a
seventh question, and this report makes none.
### Per-row, with the baseline each row must be read against
The per-question gold ids are **not in this file**. They name documents in a
consumer's corpus, and this repository keeps corpus paths and document titles out
of tracked files; they live in the local project directory beside the run, and go
to `portfolio-optimiser` by coord. What is publishable is the shape:
| row | gold concepts | hit@8 | rank | chance, analytic | chance, empirical |
|---|---|---|---|---|---|
| 1 — **the concept-level row** | 1 | ✅ | 1 | 0.013 | 0.013 |
| 2 | 49 | ✅ | 1 | 0.479 | 0.483 |
| 3 | 20 | ✅ | 1 | 0.229 | 0.231 |
| 4 | 43 | ✅ | 1 | 0.434 | 0.438 |
| 5 | 11 | ✅ | 1 | 0.132 | 0.134 |
| 6 | 5 | ❌ | — | 0.062 | 0.063 |
**Why every row carries its own baseline.** A document-prefix hit is *easier for
a large gold document*: with gold sets spanning 1 to 49 concepts, a raw count of
4/6 could be produced by a ranker that had learned nothing. Row 2 alone is 48 %
likely by chance; row 1 is 1.3 %. The two hardest rows (1 and 6, at 0.013 and
0.062) are one hit and one miss.
The empirical baseline is 20 000 draws at seed 20260907. It agrees with the
analytic form to within about half a percentage point — **not** to three decimal
places, and this report does not claim it does.
**The miss, diagnosed rather than absorbed.** The gold's best concept ranked
**11 overall**, just outside `k = 8`, and comes back at rank 11 for `k = 12` and
`k = 16`. Cause: the profile matches two tokens when they share four leading
characters, so a compound question term matches two much larger sibling
documents in the same subject area, and those two fill all eight slots. The
document prior does not rescue it (0.333 against 0.433 and 0.415). This is a
near-miss at the reported `k`, not an absence — and it is the shape of failure to
expect from this ranker: crowding by large same-prefix neighbours.
## The controls that failed, stated first rather than last
The order required a known-negative: a question with no answer in the bundle
must produce an empty or "not found" payload, never a confident guess. **Both
known-negative questions produced eight excerpts.** The control failed.
| control | tokens reaching nothing | delivered |
|---|---|---|
| the question chosen as planned | 2 of 6 | 8 |
| the most lexically disjoint question found | 4 of 6 | 8 |
**Why, measured rather than guessed.** The pre-pass has a per-concept relevance
rule (`no_lexical_match`) that withholds a concept the question does not reach,
and it works: it fires on 600-odd concepts per run. It does **not** work as a
whole-question gate, because no natural Norwegian question is lexically disjoint
from a 629-concept corpus under a four-character shared-prefix rule. Measured
per token: the interrogative `hvor` reaches 40 concepts, `mange` 17, `brukes` 83,
`etablert` 38. Four candidate questions were probed and none reached zero on
every token. The first control's content words did genuinely match — one of them
reached 17 concepts describing welded seams in floor covering, which is a correct
lexical answer and a useless topical one.
**Consequence for anyone using this, stated in the skill as well as here: an
empty `excerpts` list is evidence of absence; a full one is not evidence of
presence.** When the delivered excerpts do not answer the question, the right
output is `[sourced-not-sufficient]` plus a statement that the cut found nothing
responsive — not eight answers.
**What would fix it, named and not built.** Rarity weighting: a token matching a
large fraction of the corpus carries little information, and requiring a
delivered concept to match at least one *informative* token would make the gate
work. Smoothed IDF is on the plan's own list of mechanisms worth lifting and
this build does not implement it. It is not attempted here because this step's
scope fence freezes the instrument before it is measured — tuning a ranker
against the questions it is being scored on produces a number about the tuning.
## Honesty limits
Six, and the first three matter most.
1. **n = 6.** Six questions is a small sample and the table says so on every
reading. The chance baseline is what keeps it from being meaningless, not
what makes it large.
2. **The question texts were written after the ranker existed.** The plan fixed
the gold documents' *size profile* before any ranker existed — its recorded
per-row baselines sum to 1.35, and the sizes used here reproduce that total
exactly, which is an independent check that this is the gold set the plan
profiled. But the plan recorded no question **texts**, so those were written
during execution. Three of the six gold documents also could not be pinned
uniquely from the recorded sizes and were chosen among equal-sized
candidates. The gold documents themselves came from the corpus's own titles,
per the operator's answer, and not from anything the ranker computes — but
this is not a blind evaluation and must not be quoted as one.
3. **One ranking change was made with a gold answer visible.** The stage-one
document prior originally *summed* lexical overlap across a document's units,
which measures document size: measured on this corpus, a 79-concept document
scored 6.0 (0.076 each) against the price document's 2.0 over 1 concept, and
the price gold fell outside the top 8. The prior is now a density. The fix is
justified by the scoring function's own arithmetic rather than by the answer
— a prior that grows with size is measuring size — but it was found by running
row 1, and the ranker is therefore not blind to that row.
4. **Five of six rows are DOCUMENT granularity**, not the concept granularity
the order asked for. Row 1 is the exception and is labelled throughout.
5. **Two of the three states this contract makes consumers distinguish have
denominator zero in this corpus.** `adjudicated` never occurs (all 618 present
values are `proposed`; 11 concepts carry no key and read as `unknown`), and
`verified` is absent on all 629 — anchored and unanchored — so every excerpt
is `unverified`. `type: verdict` is likewise absent on all 629, which makes
the § 9.1 exclusion **vacuous here**. All three are exercised only against a
synthetic fixture, and the skill states that rather than implying coverage.
6. **The contract checker is a lint, not the acceptance.** Measured: it returns
0 findings on a payload declaring `considered=0, withheld=0, delivered=0`
paired with the *unfilled* template. A green checker means the payload carries
what a claim would have to rest on — not that the claim is sound. The real
gates are the negative controls, the partition assertion, the placeholder scan
and the digest test, and none of those is something the checker performs.
## What the contract's own gates reported
| gate | result |
|---|---|
| SS 5.2, `considered == withheld + delivered` | closes on every run, by construction |
| SS 5.3, every withheld concept names its rule | 621 of 621 on the row-1 run, from a closed set of six |
| SS 7.4, the instrument reproduced a known-positive before reporting | 10 349 expected, 10 349 measured, plus a second route (`wc -c` = 10 060, delta 289) |
| SS 7.2, `spent` is the **delivered set** | 74 838 spent against a 120 000 limit on row 1 — where a whole-payload reading would have put 165 109 against the same limit and refused |
| SS 3.3, the ref is a content identity, not a declared version | `sha256-tree:` over the reachable tree; `--ref` asserts and never overrides |
| SS 9.2, no directory enumerated | the index walk reaches 629 concepts, exactly the set the forbidden method finds |
| contract check on a real payload | `conformant: 14 rules over 8 excerpts and 621 withheld entries, 0 findings`, exit 0 |
| negative control: broken identity | `NOT conformant`, 2 findings, exit 1 |
| negative control: missing payload | exit 2 |
## Reproduce
```sh
# One payload, and the checker on it.
.venv/bin/python tools/okf_consume.py <bundle> \
--question "Hvordan skal prisene fylles ut?" --out payload.json
.venv/bin/python tools/okf_contract_check.py \
--skill skills/okf-consume/SKILL.md --payload payload.json
# The whole table. The gold set is an INPUT and is not in this repository.
.venv/bin/python tools/okf_consume_measure.py <bundle> --questions <gold-set.json>
```
The gold-set file's shape is in `tools/okf_consume_measure.py`'s module
docstring. Determinism is asserted by the suite, not by inspection: two runs of
the same `(bundle, question, k, limit)` produce byte-identical output, and the
module reaches no clock.

2
tests/fixtures/consume-bundle/krav/index.md vendored
View file

@ -1,4 +1,4 @@
- [Prissammenstilling](prissammenstilling.md) — adjudication: proposed
- [Pristabell](pristabell.md) — adjudication: proposed
- [Årlig kontroll av anlegget](aarlig-kontroll.md) — adjudication: adjudicated
- [verdict-lookalike](verdict-lookalike.md) — adjudication: proposed
- [Loggnotat](loggnotat.md) — adjudication: proposed

6
tests/fixtures/consume-bundle/krav/prissammenstilling.md → tests/fixtures/consume-bundle/krav/pristabell.md vendored
View file

@ -1,7 +1,7 @@
---
type: reference
title: Prissammenstilling
source_file: Prisskjema.xlsx
title: Pristabell
source_file: Pristabell.xlsx
source_sha256: 1111111111111111111111111111111111111111111111111111111111111111
ingested_at: 2026-09-01T00:00:00Z
adjudication: proposed
@ -9,7 +9,7 @@ bundle_id: consume-fixture
verified: [{ by: process:okf-check, at: 2026-09-01T00:00:00Z }]
---
## Prissammenstilling
## Pristabell
Prisene fylles ut i dette skjemaet. Summen av alle poster overfoeres til
tilbudsbrevet.

129
tests/test_okf_consume.py
View file

@ -37,6 +37,7 @@ PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
import okf_consume # noqa: E402
import okf_consume_measure # noqa: E402
import okf_contract_check # noqa: E402
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
@ -425,8 +426,8 @@ def test_a_concept_in_a_high_scoring_document_outranks_an_equally_lexical_one()
question = "Hvordan skal prisene fylles ut?"
lifted = okf_consume.concept_scores(concepts, question, {"krav": 10.0, "dyp": 0.0})
dropped = okf_consume.concept_scores(concepts, question, {"krav": 0.0, "dyp": 10.0})
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/prissammenstilling")
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/prissammenstilling")
krav_first = [c.concept_id for c, _, _ in lifted].index("krav/pristabell")
krav_later = [c.concept_id for c, _, _ in dropped].index("krav/pristabell")
assert krav_first < krav_later
@ -442,7 +443,7 @@ def test_the_price_concept_leads_on_the_price_question_in_the_fixture() -> None:
concepts = _fixture_concepts()
scores = okf_consume.document_scores(FIXTURE, "Hvordan skal prisene fylles ut?")
ranked = okf_consume.concept_scores(concepts, "Hvordan skal prisene fylles ut?", scores)
assert ranked[0][0].concept_id == "krav/prissammenstilling"
assert ranked[0][0].concept_id == "krav/pristabell"
# --- Step 7: the cut ----------------------------------------------------------
@ -562,7 +563,7 @@ def test_excerpts_come_back_in_rank_order_and_carry_that_rank() -> None:
# second one.
delivered, _, _ = _cut_fixture()
assert [excerpt["rank"] for excerpt in delivered] == list(range(1, len(delivered) + 1))
assert delivered[0]["concept_id"] == "krav/prissammenstilling"
assert delivered[0]["concept_id"] == "krav/pristabell"
# --- Step 8: the payload ------------------------------------------------------
@ -765,16 +766,33 @@ def test_spent_is_the_delivered_set_where_the_whole_payload_reading_would_refuse
assert int(budget["spent"]) <= int(budget["limit"])
#: The gold set is LOCAL-ONLY: it names corpus documents, which never reach a
#: tracked file here. The test reads it rather than restating it, so this file
#: carries the assertion and not the answer key.
GOLD_SET = PROJECT_ROOT / ".claude/projects/2026-09-07-okf-consume-prepass/hit-at-k-questions.json"
@requires_k2
def test_the_price_form_gold_is_delivered_for_the_price_question() -> None:
# SC6. The one question whose gold is confirmed by a signal from outside
# this repository: a live model reached that directory unprompted in three
# navigation steps on 2026-09-06.
payload = okf_consume.build_payload(K2_BUNDLE, question="Hvordan skal prisene fylles ut?")
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
ids = [str(excerpt["concept_id"]) for excerpt in excerpts]
assert "del-ii-bilag-7-prisskjema/prissammenstilling-sheet-1" in ids
@pytest.mark.skipif(not GOLD_SET.is_file(), reason=f"the local gold set is absent ({GOLD_SET})")
def test_every_gold_document_in_the_local_set_is_reached_or_named_as_a_miss() -> None:
# SC5 and SC6 together, run against the answer key rather than a literal.
# Row 1's gold is the one confirmed by a signal from outside this
# repository -- a live model reached that document unprompted in three
# navigation steps on 2026-09-06 -- and its gold document holds exactly one
# concept, so it is also the one concept-granularity row.
spec = json.loads(GOLD_SET.read_text(encoding="utf-8"))
questions = spec["questions"]
assert len(questions) >= 5, "fewer than five questions is not the measurement"
hits = 0
for entry in questions:
payload = okf_consume.build_payload(K2_BUNDLE, question=entry["question"])
excerpts = payload["excerpts"]
assert isinstance(excerpts, list)
if okf_consume_measure.hit_rank(excerpts, entry["gold_document"]) is not None:
hits += 1
# The published bar, and the published number. A regression that drops a
# row goes red here rather than in a document nobody re-runs.
assert hits == 5, f"hit@8 moved: {hits} of {len(questions)}"
# --- Step 9: the CLI ----------------------------------------------------------
@ -970,22 +988,34 @@ def test_the_shipped_example_payload_is_current_and_regenerates_byte_for_byte()
assert shipped == regenerated
def test_no_k2_concept_path_or_document_title_reaches_the_tracked_skill() -> None:
# CLAUDE.md's public-file rule, with the pattern widened to the bare
# basenames a report is most likely to leak, and shown capable of finding
# against the bundle's own index before its zero here is believed.
leak = re.compile(
r"del-ii-bilag|del-i-vedlegg|del-i-konkurranse|prisskjema|prissammenstilling|stange",
re.IGNORECASE,
@requires_k2
def test_no_corpus_document_name_reaches_any_file_this_work_tracks() -> None:
# CLAUDE.md's public-file rule. The pattern is DERIVED from the corpus's own
# top-level document names at run time rather than hand-picked, so it covers
# every document rather than the six someone thought of -- and so this
# tracked file carries no corpus name of its own.
documents = sorted(
{concept_id.split("/", 1)[0] for concept_id in okf_consume.enumerate_concepts(K2_BUNDLE)}
)
assert len(documents) > 30, "too few documents to be the real corpus"
leak = re.compile("|".join(re.escape(name) for name in documents), re.IGNORECASE)
# The known-positive, first: the pattern must be shown able to find before
# its zero counts as a measurement.
control = (K2_BUNDLE / "index.md").read_text(encoding="utf-8")
assert leak.findall(control), "the pattern cannot find; the zeros below would mean nothing"
tracked = [
SKILL,
SKILL.parent / "references" / "README.md",
SKILL.parent / "references" / "example-payload.json",
PROJECT_ROOT / "tools" / "okf_consume.py",
PROJECT_ROOT / "tools" / "okf_consume_measure.py",
PROJECT_ROOT / "tests" / "test_okf_consume.py",
PROJECT_ROOT / "docs" / "2026-09-07-okf-konsumskill-maaling.md",
PROJECT_ROOT / "README.md",
PROJECT_ROOT / "CLAUDE.md",
]
if K2_BUNDLE.is_dir():
control = (K2_BUNDLE / "index.md").read_text(encoding="utf-8")
assert leak.findall(control), "the pattern cannot find; its zero below would mean nothing"
for path in tracked:
assert leak.findall(path.read_text(encoding="utf-8")) == [], path
@ -997,3 +1027,56 @@ def test_the_readme_consume_section_states_the_rule_count_the_code_emits() -> No
assert len(okf_consume.WITHHOLDING_RULES) == 6
assert "closed set of six" in readme
assert "tools/okf_consume.py" in readme
# --- Step 11: the measurement scorer -----------------------------------------
def test_a_gold_at_rank_three_is_a_hit_at_eight_and_a_miss_at_two() -> None:
excerpts = [
{"concept_id": "other-doc/a", "rank": 1},
{"concept_id": "other-doc/b", "rank": 2},
{"concept_id": "gold-doc/c", "rank": 3},
]
assert okf_consume_measure.hit_rank(excerpts, "gold-doc") == 3
assert okf_consume_measure.hit_rank(excerpts[:2], "gold-doc") is None
def test_the_document_prefix_test_does_not_match_a_longer_document_name() -> None:
# Without the trailing slash, a gold document `bilag-7` would count a
# concept in `bilag-70` as a hit -- a false positive that inflates the
# headline number and looks like a correct answer.
excerpts = [{"concept_id": "bilag-70/a", "rank": 1}]
assert okf_consume_measure.hit_rank(excerpts, "bilag-7") is None
assert okf_consume_measure.hit_rank([{"concept_id": "bilag-7", "rank": 1}], "bilag-7") == 1
def test_the_analytic_chance_baseline_is_exact_for_a_single_gold_concept() -> None:
# One gold concept in a 629-concept corpus, drawn 8: exactly 8/629, which
# is derivable by hand and pins the closed form.
assert okf_consume_measure.chance_analytic(1, 629, 8) == pytest.approx(8 / 629)
assert okf_consume_measure.chance_analytic(629, 629, 8) == 1.0
assert okf_consume_measure.chance_analytic(0, 629, 8) == 0.0
def test_the_empirical_baseline_reproduces_the_analytic_one_on_a_fixed_seed() -> None:
# Two routes to one number. The agreement is about half a percentage point
# at 20 000 trials -- asserted at that tolerance rather than at the three
# decimal places the plan claimed, because the looser figure is the one
# that is true.
for gold in (1, 5, 11, 20, 43, 49):
analytic = okf_consume_measure.chance_analytic(gold, 629, 8)
empirical = okf_consume_measure.chance_empirical(gold, 629, 8)
assert abs(analytic - empirical) < 0.01, gold
def test_document_sizes_counts_root_level_concepts_as_their_own_document() -> None:
sizes = okf_consume_measure.document_sizes(("a/1", "a/2", "b/1", "root-concept"))
assert sizes == {"a": 2, "b": 1, "root-concept": 1}
def test_the_measurement_instrument_names_no_corpus_document() -> None:
# It is tracked and public; the answer key is an input, never a constant.
source = (PROJECT_ROOT / "tools" / "okf_consume_measure.py").read_text(encoding="utf-8")
leak = re.compile(r"del-ii-bilag|del-i-vedlegg|del-i-konkurranse|prisskjema|stange", re.I)
assert leak.findall(source) == []

24
tools/okf_consume.py
View file

@ -491,19 +491,19 @@ MIN_TOKEN_LENGTH = 3
#: How many leading characters two tokens must share to count as a match.
#:
#: THIS INSTRUMENT'S OWN CONSTANT, and a measurement rather than a preference.
#: Token equality fails on Norwegian compounds: the question word `prisene`
#: equals none of a price-form concept's `title`, `source_file` or path tokens.
#: Plain substring containment does not save it either -- neither `prisene` nor
#: `prissammenstilling` contains the other. A shared prefix does: `pris|ene` and
#: `pris|sammenstilling` share 4.
#: Token equality fails on Norwegian compounds: a question's inflected noun
#: equals none of the tokens in a concept's `title`, `source_file` or path when
#: the concept spells the same subject as a compound. Plain substring
#: containment does not save it either -- of `varene` and `varemottak`, neither
#: contains the other. A shared prefix does: `vare|ne` and `vare|mottak` share 4.
#:
#: MEASURED HERE, 2026-09-07, over the 629-concept K2 corpus for the question
#: token `prisene`: a 4-character floor matches **3** concepts -- over the
#: concept id alone AND over title + `source_file` + id together, the same 3 --
#: and the price-form gold is among them. The plan this implements recorded 6
#: for the same measurement; 6 is not reproducible with this rule, and the
#: number that is reproducible is the one carried here. A 3-character floor
#: over-matches Norwegian function words.
#: MEASURED 2026-09-07 over a 629-concept corpus, for one question's subject
#: token: a 4-character floor matches **3** concepts -- over the concept id
#: alone AND over title + `source_file` + id together, the same 3 -- and the
#: gold concept is among them. The plan this implements recorded 6 for the same
#: measurement; 6 is not reproducible with this rule, and the number that is
#: reproducible is the one carried here. A 3-character floor over-matches
#: Norwegian function words.
MIN_SHARED_PREFIX = 4
_TOKEN_SPLIT_RE = re.compile(r"[^0-9a-zà-öø-ÿ]+")

291
tools/okf_consume_measure.py Normal file
View file

@ -0,0 +1,291 @@
"""Measure a consumption pre-pass: hit@k per question, against a chance baseline.
Measure, don't build. This instrument calls `tools/okf_consume.py` exactly as a
consumer would and scores what comes back; it contains no ranking of its own,
because a scorer sharing code with the thing it scores measures its own
agreement rather than the ranker's performance.
**The gold set is an INPUT, never a constant here.** It arrives as a JSON file
naming, per question, the gold document. Two reasons, and the second is the
binding one:
- An instrument with a corpus baked in measures one corpus.
- This file is tracked in a PUBLIC repository. A gold set names documents in a
consumer's corpus, and this repository's standing rule keeps corpus paths and
document titles out of tracked files. The instrument is publishable; the
answer key is not, so it lives beside the run and not beside the code.
**hit@k is scored at DOCUMENT granularity by default**, which is a weaker claim
than concept granularity and is labelled as such wherever the number is used: a
gold document counts as hit at k when any of the top-k excerpts has that
document as its `concept_id` path prefix, and the reported rank is the position
of the first such excerpt. A gold document holding one concept is the only case
where document and concept granularity coincide.
**Every row carries its own chance baseline**, computed two ways, because a
document-prefix hit is easier for a large gold document: a gold set spanning one
to fifty concepts makes a raw hit count uninterpretable without the baseline it
should be read against.
The questions file:
```json
{"questions": [{"question": "...", "gold_document": "...", "note": "optional"}],
"negatives": [{"question": "...", "label": "..."}]}
```
"""
from __future__ import annotations
import argparse
import json
import random
import sys
import time
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
import okf_consume # noqa: E402
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
#: The seed every empirical baseline in a published table is drawn with. Fixed
#: so a reader re-running this reproduces the same figure; the analytic form
#: beside it is what makes the empirical one checkable at all.
DEFAULT_SEED = 20260907
#: Trials for the empirical baseline. Enough that it lands within about half a
#: percentage point of the analytic value, which is the agreement actually
#: observed -- NOT three decimal places, and this instrument does not claim it.
DEFAULT_TRIALS = 20_000
@dataclass(frozen=True)
class Row:
"""One question, measured."""
question: str
gold_document: str
gold_concepts: int
hit: bool
rank: int | None
considered: int
withheld: int
delivered: int
spent: int
payload_bytes: int
seconds: float
chance_analytic: float
chance_empirical: float
def hit_rank(excerpts: Sequence[Mapping[str, object]], gold_document: str) -> int | None:
"""The rank of the first excerpt inside `gold_document`, or `None`.
Prefix on the `concept_id`, which in this library is the concept's
bundle-relative path with the suffix removed -- so the test is a string
operation on a field the payload already carries. The `+ "/"` is not
cosmetic: without it, a gold document `bilag-7` would match a concept in
`bilag-70`.
"""
for excerpt in excerpts:
concept_id = str(excerpt.get("concept_id", ""))
if concept_id == gold_document or concept_id.startswith(f"{gold_document}/"):
rank = excerpt.get("rank")
return int(rank) if isinstance(rank, int) else None
return None
def chance_analytic(gold_concepts: int, corpus: int, k: int) -> float:
"""P(a uniform random k-of-corpus draw touches any of `gold_concepts`).
The complement of drawing k items that all miss, computed as a product of
hypergeometric terms rather than with factorials, so it stays exact for the
sizes a corpus actually reaches.
"""
if gold_concepts <= 0 or k <= 0 or corpus <= 0:
return 0.0
miss = 1.0
for drawn in range(min(k, corpus)):
remaining = corpus - drawn
available = corpus - gold_concepts - drawn
if available <= 0:
return 1.0
miss *= available / remaining
return 1.0 - miss
def chance_empirical(
gold_concepts: int,
corpus: int,
k: int,
*,
trials: int = DEFAULT_TRIALS,
seed: int = DEFAULT_SEED,
) -> float:
"""The same probability, drawn rather than derived.
Kept beside the analytic form because two routes to one number is what makes
either believable: an off-by-one in the closed form and a bug in the sampler
do not agree by accident.
"""
rng = random.Random(seed)
pool = range(corpus)
gold = set(range(gold_concepts))
hits = sum(1 for _ in range(trials) if gold & set(rng.sample(pool, k)))
return hits / trials
def document_sizes(concept_ids: Sequence[str]) -> dict[str, int]:
"""How many concepts each top-level document holds."""
sizes: dict[str, int] = {}
for concept_id in concept_ids:
document = concept_id.split("/", 1)[0]
sizes[document] = sizes.get(document, 0) + 1
return sizes
def measure_question(
bundle: Path, question: str, gold_document: str, *, k: int, sizes: Mapping[str, int]
) -> Row:
started = time.perf_counter()
payload = okf_consume.build_payload(bundle, question=question, k=k)
elapsed = time.perf_counter() - started
text = okf_consume.serialise(payload)
excerpts = payload["excerpts"]
counts = payload["denominators"]
budget = payload["budget"]
assert isinstance(excerpts, list) and isinstance(counts, dict) and isinstance(budget, dict)
corpus = int(counts["considered"])
gold_concepts = sizes.get(gold_document, 0)
rank = hit_rank(excerpts, gold_document)
return Row(
question=question,
gold_document=gold_document,
gold_concepts=gold_concepts,
hit=rank is not None,
rank=rank,
considered=corpus,
withheld=int(counts["withheld"]),
delivered=int(counts["delivered"]),
spent=int(budget["spent"]),
payload_bytes=len(text.encode("utf-8")),
seconds=round(elapsed, 3),
chance_analytic=round(chance_analytic(gold_concepts, corpus, k), 3),
chance_empirical=round(chance_empirical(gold_concepts, corpus, k), 3),
)
def token_reach(bundle: Path, question: str) -> dict[str, int]:
"""How many concepts each of the question's tokens reaches.
The known-negative control's own PRECONDITION, measured before any payload
is built: a question whose tokens all reach real corpus text is not a
question the bundle has no answer to, and scoring it as one would report a
property of the question as a property of the ranker.
"""
root_index = bundle / okf_consume.DEFAULT_PROFILE.index.name
root_bundle_id = parse_frontmatter(root_index).get("bundle_id", "")
corpus = [
set(
okf_consume.normalise(
f"{concept.title} {concept.concept_id.replace('/', ' ')} {concept.body}"
)
)
for concept in (
okf_consume.read_concept(
bundle / f"{concept_id}{okf_consume.CONCEPT_SUFFIX}",
bundle_root=bundle,
root_bundle_id=root_bundle_id,
)
for concept_id in okf_consume.enumerate_concepts(bundle)
)
]
return {
token: sum(
1
for tokens in corpus
if any(okf_consume.tokens_match(token, other) for other in tokens)
)
for token in okf_consume.normalise(question)
}
def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("bundle", type=Path, help="the OKF bundle to measure against")
parser.add_argument(
"--questions", type=Path, required=True, help="JSON gold set (see the module docstring)"
)
parser.add_argument("--k", type=int, default=okf_consume.DEFAULT_K)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
try:
spec = json.loads(args.questions.read_text(encoding="utf-8"))
except OSError as error:
print(f"okf_consume_measure: FAILED - {error}", file=sys.stderr)
return 2
sizes = document_sizes(okf_consume.enumerate_concepts(args.bundle))
rows = [
measure_question(
args.bundle, entry["question"], entry["gold_document"], k=args.k, sizes=sizes
)
for entry in spec.get("questions", [])
]
negatives = []
for entry in spec.get("negatives", []):
reach = token_reach(args.bundle, entry["question"])
started = time.perf_counter()
payload = okf_consume.build_payload(args.bundle, question=entry["question"], k=args.k)
elapsed = time.perf_counter() - started
counts, budget, withheld = (
payload["denominators"],
payload["budget"],
payload["withheld"],
)
assert isinstance(counts, dict) and isinstance(budget, dict) and isinstance(withheld, list)
negatives.append(
{
"question": entry["question"],
"label": entry.get("label", ""),
"token_reach": reach,
"tokens_reaching_nothing": sum(1 for value in reach.values() if value == 0),
"tokens": len(reach),
"delivered": int(counts["delivered"]),
"withheld": int(counts["withheld"]),
"considered": int(counts["considered"]),
"spent": int(budget["spent"]),
"payload_bytes": len(okf_consume.serialise(payload).encode("utf-8")),
"seconds": round(elapsed, 3),
"rules": sorted({str(entry["rule"]) for entry in withheld}),
}
)
print(
json.dumps(
{
"k": args.k,
"seed": DEFAULT_SEED,
"trials": DEFAULT_TRIALS,
"documents": len(sizes),
"rows": [row.__dict__ for row in rows],
"hits": sum(1 for row in rows if row.hit),
"expected_by_chance": round(sum(row.chance_analytic for row in rows), 2),
"negatives": negatives,
},
ensure_ascii=False,
indent=2,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())