hit@8 = 5 of 6, every hit at rank 1, against a chance baseline of 1.35 of 6
over a denominator of 629 concepts per question. Wall time 0.51-0.56 s per
question; spent 17 970 - 74 838 bytes against a 120 000 limit.
Two things this measurement did NOT establish, both in the report:
- BOTH known-negative controls FAILED. A question the bundle has no answer to
still returns eight excerpts, because no natural Norwegian question is
lexically disjoint from a 629-concept corpus under a four-character
shared-prefix rule -- measured per token, the interrogative `hvor` reaches 40
concepts, `brukes` 83. So `no_lexical_match` works per concept and not as a
whole-question gate: an empty excerpt list is evidence of absence, a full one
is not evidence of presence. The fix is named (rarity weighting) and NOT
built, because this step's fence freezes the instrument before it is measured.
- The question texts were written during execution, after the ranker existed.
The plan recorded the gold documents' SIZE profile -- its per-row baselines
sum to 1.35 and the sizes used here reproduce that exactly, which is an
independent check that this is the set the plan profiled -- but it recorded no
question texts, and three of six gold documents could not be pinned uniquely
from the sizes. Not a blind evaluation, and the report says so.
The scorer is a tool, not a script in a document: `tools/okf_consume_measure.py`
takes the gold set as an INPUT because it is tracked in a public repository and
an answer key names a consumer's documents. hit_rank, both chance baselines and
the document-size census are unit-tested; the corpus run is a measurement.
Public-file rule, checked with a pattern DERIVED from the corpus's own 39
document names rather than hand-picked, and shown able to find first (67 hits on
the bundle's own index): zero corpus document names in any tracked file in this
repository. One leak was found and removed on the way -- a corpus concept name
in a code comment and a hardcoded corpus path in a test.
Suite run after git add: 1230 passed, mypy --strict clean on 27 files,
ruff clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 12's README section is brought forward to here because the docs gate is
right: a feat commit that ships a new command needs the command documented.
CLAUDE.md's Commands section gains the pre-pass beside `okf build`. Nothing
else moves.
Contract check against a real payload from the 629-concept bundle:
$ .venv/bin/python tools/okf_consume.py <K2-bundle> \
--question 'Hvordan skal prisene fylles ut?' --out /tmp/k2.json
$ .venv/bin/python tools/okf_contract_check.py \
--skill skills/okf-consume/SKILL.md --payload /tmp/k2.json
conformant: 14 rules over 8 excerpts and 621 withheld entries, 0 findings
exit=0
And the two negative controls, because a green checker proves little on its
own -- measured, it returns 0 findings on an empty payload paired with the
unfilled template:
broken denominator identity -> NOT conformant, 2 findings, exit=1
missing payload file -> exit=2
Placeholder scan, known-positive first: the DOTALL scan reports 20 occurrences
on the template and 0 on this copy. The shipped example payload is generated
from the in-repo golden bundle, not from the corpus, and a test regenerates it
byte for byte. No K2 concept path or document title reaches any tracked file
here, checked with a pattern shown able to find against the bundle's own index.
Suite run after git add: 1224 passed, mypy --strict clean on 26 files,
ruff clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two corrections the plan did not carry, both found by running the instrument:
- A question matching nothing still returned the top eight by tie-break -- a
confident guess wearing a denominator. The closed rule set gains
no_lexical_match, so the known-negative returns a measured empty set with
every considered concept named in withheld.
- The stage-one document prior summed overlap across a document's units, so it
measured document SIZE. Measured on K2 for the price question: the
competition document sums to 6.0 over 79 concepts (0.076 each), the price
document to 2.0 over 1. The prior is now a density; the price-form gold moves
from outside the top eight to rank 1. Found with that answer visible, which
the measurement document states beside the number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
