okf-wiki/gates/ANSWERSET.md
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

182 lines
16 KiB
Markdown

# Answer set for collection 1 (wave 2, order 6)
The frozen question set for `PLAN.md` criterion 4 on collection 1, made from the fresh frozen copy (contract K1) so
that it resembles the collection: questions that need two or more documents seen together, a group of version-history
questions reported on its own, and 20 questions held out outside the repository. It replaces the E2 set
(`MULTIDOC.md`) for the verdict, because 33 of E2's 60 rest on release notes alone. `answerset.py` holds the gate and
`answers.py` the harness the answer measurements use (contract K5). None of them calls a model or starts one.
```
python3 gates/answerset.py check --set <set.json> --sources <K1 root> [--per-question] [--min-paths 3 --min-many 20]
python3 gates/answerset.py inside --set <set.json> --part-root <K1 subset root> [--require-all]
python3 gates/answerset.py split --set <all.json> --sources <K1 root> --seed <n> --held-out-count 20 \
--set-out gates-local/answerset/set-60.json --set-label gates-local/answerset/set-60.json \
--held-out <file OUTSIDE the repository> --part gates-local/answerset/part-del1.json \
--part-sources <part root> --part-label gates-local/answerset/part-del1.json --manifest gates/answerset.manifest.json
python3 gates/answerset.py freeze --set <drafts.json> --sources <K1 root> --seed <n> --count 30 --min-paths 3 \
--min-many 20 --prefix x --set-out <set.json> --set-label <label> --manifest <manifest.json>
python3 gates/answerset.py gate --set <set.json> --manifest gates/answerset.manifest.json --sources <K1 root> \
--arm filsok|bundle --answers <answers.json> [--out <results.json>]
python3 gates/answerset.py stub --set <set.json> --arm filsok|bundle --kind fasit|no-citations --out <answers.json>
```
- **check** reads K1 with its own code (`okfwiki.sources` is not imported). The set names the root it was made from
(`sources.frozen_at` and `sources.manifest_sha256`, the sha256 of the root's `source-manifest.json`), and every cited
file must be listed in K1 with its sha256. It prints the denominators from the set itself: spans that stand byte for
byte in their file (20-400 characters), questions with every span standing, questions with at least two distinct
paths, questions whose `group` is the one K1 and the path count give, questions where every path carries an answer
key inside that path's spans, wording keys (`keyform.py`, as in E2c) and negated keys that stand negated (E2d). PASS
only when every count is N of N with N > 0.
- **The group is set by the sources:** `history` when one path has `main_topic: false` in K1 (a release note or a
weekly What's New page), otherwise `docs-2-3` (2-3 distinct paths) or `docs-4` (4 or more).
- **inside** counts the questions whose every span stands in its file in a part root (a K1 subset root). On the 60 it is
a number for the report; on the part set it is a requirement (`--require-all`).
- **split** checks both the set and the part first (PASS required, and every part question inside the part root),
then holds out per group in the fixed order `docs-2-3`, `docs-4`, `history`, with quotas proportional to the group
sizes (largest remainder) and one `random.Random(seed)` drawing `sample(sorted(ids of the group), quota)`. Both parts
are renumbered in the order of the original ids (`d01..`, `h01..`). The held-out file must lie outside the
repository and is never overwritten. The manifest holds the seed, counts per group, sha256 values and the two roots'
K1 sha256, and no text.
- **gate** scores one arm with the gates' one key test (`multidoc.key_holds`): a question is correct when the answer
holds every key and the citations quote every needed path verbatim (whitespace collapsed; `path` is the K1 path,
and a leading `/` as in a page's `resource` is accepted). The set's sha256 must be one of the manifest's parts, and
the root's K1 sha256 must be that part's root (the copy for the set and the held out, the first part for the part
set). A missing answers file scores 0 of N, RED, exit 1 (fail-closed). `--out` writes the results file of K5 point 5,
sorted by id. GREEN only when every question is correct; exit 0 when every question was answered.
- **stub** writes a known arm: `fasit` (every key as its own sentence, "not <key>" when negated, and every span quoted)
and `no-citations` (the same answer, no citations).
## Frozen 09.10.2026 (wave 2, order 6)
Written from the frozen copy of 08.10.2026 (635 sources, K1 `manifest_sha256` `8c9f87fd…`) by helpers in the user's
session (Opus), each question from a pack of byte-exact source lines that code prepared around rare names shared by an
anchor page and its candidates, spread over the sections of the page overview in proportion to their text. A separate
helper that did not write the question read every question against windows around its spans: the keys (facts a correct
answer must hold) and whether every file is necessary, helped by a code pre-check over all 635 files for another file
that holds the keys of two of them. Questions it marked FIX were repaired and, when a file was not necessary, rebuilt
and read again; the rest were dropped.
| Part | Questions | docs-2-3 | docs-4 | history | Confirmed by a separate reader |
|---|---:|---:|---:|---:|---:|
| all, before the split | 80 | 35 | 34 | 11 | 80 of 80 |
| `set-60.json` | 60 | 26 | 26 | 8 | |
| held out (outside the repository) | 20 | 9 | 8 | 3 | |
| `part-del1.json` (first part only, `p01`-`p18`) | 18 | 9 | 9 | 0 | 18 of 18 |
- **check:** PASS on the 80 (389 of 389 spans, 0 of 391 keys wording, 13 of 13 negated keys negated) and on the part
against the first part's root (91 of 91 spans, 0 of 86 wording, 5 of 5 negated); PASS again on `set-60.json`.
- **History:** 11 of 80 (13.8 %); the release notes and weekly What's New pages are 12.45 % of the copy's characters
(1,149,271 of 9,231,505). The first part holds no history source, so the part set has none.
- **split:** seed 20261009; held out per group docs-2-3 9, docs-4 8, history 3.
- **inside** the first part: 1 of the 60; 18 of 18 of the part set.
- **Proof, no model:** without an answers file 0 of 60, RED, exit 1; the fasit stub 60 of 60, GREEN; right answers
without citations 0 of 60 (correct answer 60 of 60). On the part set: 0 of 18 RED exit 1, 18 of 18 GREEN, 0 of 18.
## The harder set on the first part (wave 4, order 21, frozen 12.10.2026)
Wave 3 measured the 18 part questions near the ceiling (both arms 17 of 18), so order 21 made a harder set from the
first part's 38 files: every question needs at least 3 documents and at least 20 of 30 need 4 or more, every file is
necessary, and at least one key is a condition or relation that only two of the files give together. None of the 18
is reused.
- **check** with `--min-paths N` counts the questions with N or more distinct paths and with `--min-many M` requires
at least M questions with 4 or more (`questions with >= 4 distinct paths: x of n (required >= M)`). The defaults
(2 and 0) print and judge exactly as before, so the order 6 sets keep their verdict.
- **freeze** freezes a set that `split` did not make: it refuses an existing set or manifest, a set that fails the
check with the thresholds, or a set with another count than `--count`; it shuffles the sorted ids with
`random.Random(seed)`, renumbers them `x01..`, and writes a manifest of its own (seed, rules, counts per group, the
set's sha256, the root's and its copy's K1 sha256; no text) that names the set the part of its root, so `gate`
scores it as it scores the part.
- **Made** by helpers in the user's session (Opus) from packs of byte-exact lines that code built around backticked
names standing in 2-6 of the 38 files; one draft helper per question; a separate review helper per question that
did not write it read the question against windows around its spans, a code pre-check over all 38 files, and
confirmed every file necessary and the join; FIX was repaired (a repair that changed files was read again), DROP
dropped. Two questions sharing two keys in one file count as overlap, and the higher id goes. A surplus over 30
keeps every docs-4 and drops docs-2-3 with the highest id.
| `gates-local/answerset/part-del1-hard.json` (`x01`-`x30`) | Questions | docs-2-3 | docs-4 | Confirmed by a separate reader |
|---|---:|---:|---:|---:|
| frozen, seed 20261011 | 30 | 4 | 26 | 30 of 30 |
- **check** against the first part with `--min-paths 3 --min-many 20`: PASS, 30 of 30 with 3 or more paths, 26 of 30
with 4 or more; **inside** the first part 30 of 30.
- **Proof, no model, both arms:** without an answers file 0 of 30, RED; the fasit stub 30 of 30, GREEN; right answers
without citations 0 of 30 (correct answer 30 of 30), RED.
```
python3 gates/answerset.py gate --set gates-local/answerset/part-del1-hard.json \
--manifest gates/answerset-del1-hard.manifest.json --sources <first part root> --arm filsok|bundle --answers <file>
```
## The harness (`answers.py`, K5 points 2-7)
```
python3 gates/answers.py setup --run <dir> --name <run> --arm filsok|bundle --set <set.json> --sources <K1 root> \
[--bundle <collection>] [--part-root <the K1 root the set was made from>]
python3 gates/answers.py collect --run <dir> --arm <arm>
python3 gates/answers.py harvest --run <dir> --name <run> --arm <arm> --transcripts <dir>
python3 gates/answers.py score --results <filsok results> --results <bundle results> [--tokens <file>]... [--out <file>]
```
- **setup** copies every K1 source of the root into `<run>/arms/<arm>/kilde/` (checked against K1; for `bundle` the
collection goes to `<run>/arms/bundle/wiki/` beside it) and writes `<run>/tasks/<arm>/<id>.md`: the question, the
basis and the answer path, never the keys, the spans or the number of documents. It prints one line per task,
`S1-<run>-<arm>-<id>` and the task path: the subagent's description and the file it is given.
The set and `--sources` must have the same K1, unless `--part-root` is given (wave 5, order 31 A3): then the set
names the K1 of the part root, the part's `subset_of.manifest_sha256` is the K1 of `--sources`, and every source of
the part stands in `--sources` with the same path and sha256; otherwise exit 1 with the reason. The arm then searches
the whole of `--sources`, while `answerset.py gate` still judges the set against the part root.
- **collect** gathers `<run>/answers/<arm>/<id>.json` into `<run>/answers/<arm>.json` (K5 point 4; the id comes from the
file name).
- **harvest** reads every `*.jsonl` under the transcripts directory: a subagent is matched by the description in its
`.meta.json`, or, with no description, by the task path in its first prompt. Per question it sums the four usage
fields over all runs (each message id once, at its largest value), adds `output_tokens_derived` (a lower bound for
an output the transcript never recorded, as in `okfwiki/cost.py`), and counts isolation breaches as in
`evals/direction_probe.py`: a tool call that reads or lists a path outside the arm's basis and the task file, a tool
that is not allowed, or a write to another file than the answer. A task without a transcript exits 1.
**`echo` (wave 5, order 31 A2):** in a Bash chain that starts with `cd <basis> &&`, `echo` with literal words is not
a breach (order 13's helpers printed `---` between searches: 21 of its 26 counted breaches, classified call by call
before the change). Its arguments are checked as the search commands' are, so a path outside the basis, `~`, `..`,
a redirection, `$` or a backtick is still a breach. The task text does not offer `echo`. No other rule was made:
order 13 had no backtick or `$` inside single quotes and no `Read` of the helper's own answer file (its three checks
of the answer file were `python3` and `jq` in Bash, which stay breaches).
**Four more kinds (wave 5, order 31 A2b, operator 19.10.2026):** the price measurement B1 counted 9 breaches in 4 of
4 questions, none of which read a source outside the basis. Each kind has a test that failed first, with the
opposite cases that stay breaches:
1. `2>/dev/null` right before a separator (`;`, `|`, `&&`, `||`) is not a breach, as before a space or at the end.
Every other redirection (`> file`, `>/dev/null`, `2>&1`, `2>` anywhere else) still is.
2. The search word of `grep`/`egrep`/`fgrep`/`rg` (the value of `-e`/`--regexp`, or else the first argument that is
not an option) and the script of `sed` (the value of `-e`/`--expression`, or else the first such argument) are not
paths. With `-f`/`--file` the search words come from a file, none is positional, and the file is checked as a path.
A file path after the search word, `~` and `..` are still breaches. **Order 39:** the value of every other option
that takes one (`grep -C`, `--include`, `--exclude-from`, `rg --glob`, `--ignore-file` and the rest, GNU and BSD)
is consumed and checked as a path, so a file option's value outside the basis is a breach, and `rg --pre` or
`--hostname-bin` (which run a program) is one. An option missing from the table in `SEARCH_WORDS` makes its value
the search word, and a path there goes unchecked, so the lists must be whole. `grep --context` with a separate
value is left out, because BSD grep takes its value only after `=` (`--context 1 /x f` searches for 1 in `/x`).
A `sed` script is read command by command instead of checked as a path: `w`, `W` and `e` (also as flags of `s`)
are breaches, the file after `r` or `R` is checked as a path, and a script that cannot be parsed is a breach. A
flag cluster with `i` or `I` (`-ni`, `-Ei`, `-I`) is `sed -i`. An unquoted newline separates commands as `;`
does, and `#` is never read as a comment (bash reads it so only at the start of a word; a real comment is then
checked as arguments, which is stricter).
3. `$` or a backtick inside single quotes is not an expansion. Unquoted or inside double quotes it still is, and so is
an escaped backtick inside double quotes (order 31 A2 classed order 13's one as real).
4. `Read` of the helper's own answer file is not a breach, and neither is a Bash command of the read-only commands
plus `jq` whose only path outside the basis is the answer file: after `cd <basis> &&`, after `cd` into the answer's
folder, or with no `cd` and the answer's absolute path. Outside the basis every operand must be the answer file, so
another file in the answers folder, a glob, `ls` of the folder, a relative path without `cd` and `jq -f` are still
breaches, as are `python3`, a write and `jq` on any other file. Outside the basis a pipe stage that names no
file, as in `jq . <answer> | head`, is counted too (stricter than needed). The task text does not offer `jq`.
Against order 13 the A2b rules move 2 of A2's 5 real calls (one `jq` on the own answer after `cd` into its folder,
one `sed` script starting with `/`) and leave 3; on B1 they leave 1 of 9 (an escaped backtick in double quotes).
- **score** gives per arm and group "x of n", by path count (2-3 and 4 or more, history included), the pairs (both,
only filsok, only bundle, neither) with the two-sided exact McNemar p, the difference bundle - filsok with its paired
95 % interval (Newcombe 1998, Statist. Med. 17:2635-2650, method 10, tested against its Table III), and the smallest
difference the set can show at 80 % power and alpha 0.05 with the measured share of discordant pairs (Connor 1987;
tested against the sample sizes in `PLAN.md § Styrke`). It prints no line per question.
The questions and spans derive from Anthropic documentation, so the set lives in `gates-local/answerset/` and the held
out outside the repository (CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are
tracked. The held-out questions are never opened during development; the manifest's sha256 is the check.
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.