Kjell Tore Guttormsen
f2c739da75
`eval/soek/` is where the search gate's question sets are placed by hand. The sets themselves are `.gitignore`d rather than committed, and that is this repository's existing rule rather than a new one: `okf_retrieval_gate.py` already states that a gold set names documents in someone's corpus, so it arrives as a path and is never a constant here. Measured before it was decided: the sets carry coordination ids and an absolute path under a home directory, both of which this repository's own frames forbid in a tracked file. A set that is absent is reported `IKKE KJOERT -- fixture mangler` and counted RED, never as 0 hits and never as a quiet pass, so a machine without them cannot read silence as a green row. The README states the SHAPE each set must have and nothing about where any of them came from or what any of them contains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2.9 KiB
Frozen question sets for the search gate
tools/okf_soek_gate.py measures what a reader actually RECEIVES from a
collection at the shipped defaults. It reads its question sets from this
directory and the collection from --bundle; neither is committed here.
The sets are .gitignored and are placed here by hand, by whoever runs the
gate. They are INPUTS, never constants: that is the rule
tools/okf_retrieval_gate.py already states for a gold set, and this
directory inherits it. A set that is absent is reported
IKKE KJOERT -- fixture mangler and counted RED — never as 0 hits, and never
as a quiet pass. So a machine without the sets cannot mistake silence for a
green row, and nothing about anyone's corpus reaches a public mirror.
A set is FROZEN: it is copied verbatim from wherever it was written, nothing here is edited to make a row go green, and a set whose bytes move is a new set with a new name.
The four files and the shape each must have
| File | Shape | Units |
|---|---|---|
fase-sporsmaal.json |
{"questions": [{id, question, fasit: [{doc, quote}], release_only}]} |
the phase set, with a release_only class |
holdout-sporsmaal.json |
the same shape | a blind hold-out |
norske-sporsmaal.json |
{"sporsmaal": {id: text}} — the same ids, asked in plain Norwegian; the fasit is read from the phase set by id |
the phase set's ids |
delsporsmaal.json |
{"delsporsmaal": {id: [...]}, "operator": {question, hit_rule, gold: [{doc, section}]}, "negative": {pass_rule, questions: [{id, question}]}} |
one list per id, the operator's question by two routes, known negatives |
Each set carries its own hit_rule, and the gate implements it verbatim
rather than restating it: an excerpt whose source_file matches a fasit
entry's document and whose text contains that entry's quote (case folded,
whitespace collapsed). Any one fasit entry suffices. The negative set carries
a pass_rule instead, because a question the collection does not cover is
passed by SAYING so, not by returning nothing.
The collection
The collection is not a fixture here either — it is far too large to commit.
Build one with today's code and point the gate at it. It is built with
--gate none and otherwise at the shipped defaults: the sources are the
vendor's own public documentation, and the screen is for material of unknown
origin — at the default tier it refuses some of these pages outright, and every
refused page would read here as a build failure the search never had a chance
at:
okf project <sources> --out <scratch> --gate none
python3 tools/okf_soek_gate.py --bundle <scratch>/.okf/<id>
okf build <sources> --bundle <dir> --bundle-id <id> --okf-version 0.2 --gate none writes the same concept bytes.
A gate measuring a collection nobody builds measures nothing. If the collection is missing the gate exits 2 with its reason rather than reporting 0 hits against nothing.