Kjell Tore Guttormsen
f2c739da75
`eval/soek/` is where the search gate's question sets are placed by hand. The sets themselves are `.gitignore`d rather than committed, and that is this repository's existing rule rather than a new one: `okf_retrieval_gate.py` already states that a gold set names documents in someone's corpus, so it arrives as a path and is never a constant here. Measured before it was decided: the sets carry coordination ids and an absolute path under a home directory, both of which this repository's own frames forbid in a tracked file. A set that is absent is reported `IKKE KJOERT -- fixture mangler` and counted RED, never as 0 hits and never as a quiet pass, so a machine without them cannot read silence as a green row. The README states the SHAPE each set must have and nothing about where any of them came from or what any of them contains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
55 lines
2.9 KiB
Markdown
55 lines
2.9 KiB
Markdown
# Frozen question sets for the search gate
|
|
|
|
`tools/okf_soek_gate.py` measures what a reader actually RECEIVES from a
|
|
collection at the shipped defaults. It reads its question sets from this
|
|
directory and the collection from `--bundle`; **neither is committed here.**
|
|
|
|
**The sets are `.gitignore`d and are placed here by hand, by whoever runs the
|
|
gate.** They are INPUTS, never constants: that is the rule
|
|
`tools/okf_retrieval_gate.py` already states for a gold set, and this
|
|
directory inherits it. A set that is absent is reported
|
|
`IKKE KJOERT -- fixture mangler` and counted RED — never as 0 hits, and never
|
|
as a quiet pass. So a machine without the sets cannot mistake silence for a
|
|
green row, and nothing about anyone's corpus reaches a public mirror.
|
|
|
|
A set is **FROZEN**: it is copied verbatim from wherever it was written,
|
|
nothing here is edited to make a row go green, and a set whose bytes move is a
|
|
new set with a new name.
|
|
|
|
## The four files and the shape each must have
|
|
|
|
| File | Shape | Units |
|
|
|---|---|---|
|
|
| `fase-sporsmaal.json` | `{"questions": [{id, question, fasit: [{doc, quote}], release_only}]}` | the phase set, with a `release_only` class |
|
|
| `holdout-sporsmaal.json` | the same shape | a blind hold-out |
|
|
| `norske-sporsmaal.json` | `{"sporsmaal": {id: text}}` — the same ids, asked in plain Norwegian; the fasit is read from the phase set by id | the phase set's ids |
|
|
| `delsporsmaal.json` | `{"delsporsmaal": {id: [...]}, "operator": {question, hit_rule, gold: [{doc, section}]}, "negative": {pass_rule, questions: [{id, question}]}}` | one list per id, the operator's question by two routes, known negatives |
|
|
|
|
Each set carries its own `hit_rule`, and the gate implements it verbatim
|
|
rather than restating it: an excerpt whose `source_file` matches a fasit
|
|
entry's document **and** whose text contains that entry's quote (case folded,
|
|
whitespace collapsed). Any one fasit entry suffices. The negative set carries
|
|
a `pass_rule` instead, because a question the collection does not cover is
|
|
passed by SAYING so, not by returning nothing.
|
|
|
|
## The collection
|
|
|
|
The collection is not a fixture here either — it is far too large to commit.
|
|
Build one with today's code and point the gate at it. It is built with
|
|
`--gate none` and otherwise at the shipped defaults: the sources are the
|
|
vendor's own public documentation, and the screen is for material of unknown
|
|
origin — at the default tier it refuses some of these pages outright, and every
|
|
refused page would read here as a build failure the search never had a chance
|
|
at:
|
|
|
|
```sh
|
|
okf project <sources> --out <scratch> --gate none
|
|
python3 tools/okf_soek_gate.py --bundle <scratch>/.okf/<id>
|
|
```
|
|
|
|
`okf build <sources> --bundle <dir> --bundle-id <id> --okf-version 0.2
|
|
--gate none` writes the same concept bytes.
|
|
|
|
A gate measuring a collection nobody builds measures nothing. If the
|
|
collection is missing the gate exits **2** with its reason rather than
|
|
reporting 0 hits against nothing.
|