test(eval): a frozen-set directory for the search gate, sets ignored not absent
`eval/soek/` is where the search gate's question sets are placed by hand. The sets themselves are `.gitignore`d rather than committed, and that is this repository's existing rule rather than a new one: `okf_retrieval_gate.py` already states that a gold set names documents in someone's corpus, so it arrives as a path and is never a constant here. Measured before it was decided: the sets carry coordination ids and an absolute path under a home directory, both of which this repository's own frames forbid in a tracked file. A set that is absent is reported `IKKE KJOERT -- fixture mangler` and counted RED, never as 0 hits and never as a quiet pass, so a machine without them cannot read silence as a green row. The README states the SHAPE each set must have and nothing about where any of them came from or what any of them contains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8c1c28a6ac
commit
f2c739da75
2 changed files with 71 additions and 0 deletions
16
eval/soek/.gitignore
vendored
Normal file
16
eval/soek/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# The frozen question sets are INPUTS, never constants in this repository.
|
||||
#
|
||||
# `tools/okf_retrieval_gate.py` states the rule this directory inherits: "a
|
||||
# gold set names documents in a consumer's corpus, so a real set arrives as a
|
||||
# path plus an expected sha256 and is never committed". The sets named here
|
||||
# additionally carry prose quoted verbatim from a consumer whose own status is
|
||||
# unsettled, and CLAUDE.md names that class as not publishable from here.
|
||||
#
|
||||
# So they are ignored, not absent: place them in this directory and the gate
|
||||
# reads them. Nothing here reaches a public mirror by accident, and the gate
|
||||
# reports `IKKE KJOERT -- fixture mangler` (RED) for any set that is not here,
|
||||
# so a machine without them cannot mistake silence for a pass.
|
||||
fase-sporsmaal.json
|
||||
holdout-sporsmaal.json
|
||||
norske-sporsmaal.json
|
||||
delsporsmaal.json
|
||||
Loading…
Add table
Add a link
Reference in a new issue