okf-wiki/gates/ANSWERSET.md
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

16 KiB

Answer set for collection 1 (wave 2, order 6)

The frozen question set for PLAN.md criterion 4 on collection 1, made from the fresh frozen copy (contract K1) so that it resembles the collection: questions that need two or more documents seen together, a group of version-history questions reported on its own, and 20 questions held out outside the repository. It replaces the E2 set (MULTIDOC.md) for the verdict, because 33 of E2's 60 rest on release notes alone. answerset.py holds the gate and answers.py the harness the answer measurements use (contract K5). None of them calls a model or starts one.

python3 gates/answerset.py check  --set <set.json> --sources <K1 root> [--per-question] [--min-paths 3 --min-many 20]
python3 gates/answerset.py inside --set <set.json> --part-root <K1 subset root> [--require-all]
python3 gates/answerset.py split  --set <all.json> --sources <K1 root> --seed <n> --held-out-count 20 \
    --set-out gates-local/answerset/set-60.json --set-label gates-local/answerset/set-60.json \
    --held-out <file OUTSIDE the repository> --part gates-local/answerset/part-del1.json \
    --part-sources <part root> --part-label gates-local/answerset/part-del1.json --manifest gates/answerset.manifest.json
python3 gates/answerset.py freeze --set <drafts.json> --sources <K1 root> --seed <n> --count 30 --min-paths 3 \
    --min-many 20 --prefix x --set-out <set.json> --set-label <label> --manifest <manifest.json>
python3 gates/answerset.py gate   --set <set.json> --manifest gates/answerset.manifest.json --sources <K1 root> \
    --arm filsok|bundle --answers <answers.json> [--out <results.json>]
python3 gates/answerset.py stub   --set <set.json> --arm filsok|bundle --kind fasit|no-citations --out <answers.json>
  • check reads K1 with its own code (okfwiki.sources is not imported). The set names the root it was made from (sources.frozen_at and sources.manifest_sha256, the sha256 of the root's source-manifest.json), and every cited file must be listed in K1 with its sha256. It prints the denominators from the set itself: spans that stand byte for byte in their file (20-400 characters), questions with every span standing, questions with at least two distinct paths, questions whose group is the one K1 and the path count give, questions where every path carries an answer key inside that path's spans, wording keys (keyform.py, as in E2c) and negated keys that stand negated (E2d). PASS only when every count is N of N with N > 0.
  • The group is set by the sources: history when one path has main_topic: false in K1 (a release note or a weekly What's New page), otherwise docs-2-3 (2-3 distinct paths) or docs-4 (4 or more).
  • inside counts the questions whose every span stands in its file in a part root (a K1 subset root). On the 60 it is a number for the report; on the part set it is a requirement (--require-all).
  • split checks both the set and the part first (PASS required, and every part question inside the part root), then holds out per group in the fixed order docs-2-3, docs-4, history, with quotas proportional to the group sizes (largest remainder) and one random.Random(seed) drawing sample(sorted(ids of the group), quota). Both parts are renumbered in the order of the original ids (d01.., h01..). The held-out file must lie outside the repository and is never overwritten. The manifest holds the seed, counts per group, sha256 values and the two roots' K1 sha256, and no text.
  • gate scores one arm with the gates' one key test (multidoc.key_holds): a question is correct when the answer holds every key and the citations quote every needed path verbatim (whitespace collapsed; path is the K1 path, and a leading / as in a page's resource is accepted). The set's sha256 must be one of the manifest's parts, and the root's K1 sha256 must be that part's root (the copy for the set and the held out, the first part for the part set). A missing answers file scores 0 of N, RED, exit 1 (fail-closed). --out writes the results file of K5 point 5, sorted by id. GREEN only when every question is correct; exit 0 when every question was answered.
  • stub writes a known arm: fasit (every key as its own sentence, "not " when negated, and every span quoted) and no-citations (the same answer, no citations).

Frozen 09.10.2026 (wave 2, order 6)

Written from the frozen copy of 08.10.2026 (635 sources, K1 manifest_sha256 8c9f87fd…) by helpers in the user's session (Opus), each question from a pack of byte-exact source lines that code prepared around rare names shared by an anchor page and its candidates, spread over the sections of the page overview in proportion to their text. A separate helper that did not write the question read every question against windows around its spans: the keys (facts a correct answer must hold) and whether every file is necessary, helped by a code pre-check over all 635 files for another file that holds the keys of two of them. Questions it marked FIX were repaired and, when a file was not necessary, rebuilt and read again; the rest were dropped.

Part Questions docs-2-3 docs-4 history Confirmed by a separate reader
all, before the split 80 35 34 11 80 of 80
set-60.json 60 26 26 8
held out (outside the repository) 20 9 8 3
part-del1.json (first part only, p01-p18) 18 9 9 0 18 of 18
  • check: PASS on the 80 (389 of 389 spans, 0 of 391 keys wording, 13 of 13 negated keys negated) and on the part against the first part's root (91 of 91 spans, 0 of 86 wording, 5 of 5 negated); PASS again on set-60.json.
  • History: 11 of 80 (13.8 %); the release notes and weekly What's New pages are 12.45 % of the copy's characters (1,149,271 of 9,231,505). The first part holds no history source, so the part set has none.
  • split: seed 20261009; held out per group docs-2-3 9, docs-4 8, history 3.
  • inside the first part: 1 of the 60; 18 of 18 of the part set.
  • Proof, no model: without an answers file 0 of 60, RED, exit 1; the fasit stub 60 of 60, GREEN; right answers without citations 0 of 60 (correct answer 60 of 60). On the part set: 0 of 18 RED exit 1, 18 of 18 GREEN, 0 of 18.

The harder set on the first part (wave 4, order 21, frozen 12.10.2026)

Wave 3 measured the 18 part questions near the ceiling (both arms 17 of 18), so order 21 made a harder set from the first part's 38 files: every question needs at least 3 documents and at least 20 of 30 need 4 or more, every file is necessary, and at least one key is a condition or relation that only two of the files give together. None of the 18 is reused.

  • check with --min-paths N counts the questions with N or more distinct paths and with --min-many M requires at least M questions with 4 or more (questions with >= 4 distinct paths: x of n (required >= M)). The defaults (2 and 0) print and judge exactly as before, so the order 6 sets keep their verdict.
  • freeze freezes a set that split did not make: it refuses an existing set or manifest, a set that fails the check with the thresholds, or a set with another count than --count; it shuffles the sorted ids with random.Random(seed), renumbers them x01.., and writes a manifest of its own (seed, rules, counts per group, the set's sha256, the root's and its copy's K1 sha256; no text) that names the set the part of its root, so gate scores it as it scores the part.
  • Made by helpers in the user's session (Opus) from packs of byte-exact lines that code built around backticked names standing in 2-6 of the 38 files; one draft helper per question; a separate review helper per question that did not write it read the question against windows around its spans, a code pre-check over all 38 files, and confirmed every file necessary and the join; FIX was repaired (a repair that changed files was read again), DROP dropped. Two questions sharing two keys in one file count as overlap, and the higher id goes. A surplus over 30 keeps every docs-4 and drops docs-2-3 with the highest id.
gates-local/answerset/part-del1-hard.json (x01-x30) Questions docs-2-3 docs-4 Confirmed by a separate reader
frozen, seed 20261011 30 4 26 30 of 30
  • check against the first part with --min-paths 3 --min-many 20: PASS, 30 of 30 with 3 or more paths, 26 of 30 with 4 or more; inside the first part 30 of 30.
  • Proof, no model, both arms: without an answers file 0 of 30, RED; the fasit stub 30 of 30, GREEN; right answers without citations 0 of 30 (correct answer 30 of 30), RED.
python3 gates/answerset.py gate --set gates-local/answerset/part-del1-hard.json \
    --manifest gates/answerset-del1-hard.manifest.json --sources <first part root> --arm filsok|bundle --answers <file>

The harness (answers.py, K5 points 2-7)

python3 gates/answers.py setup   --run <dir> --name <run> --arm filsok|bundle --set <set.json> --sources <K1 root> \
    [--bundle <collection>] [--part-root <the K1 root the set was made from>]
python3 gates/answers.py collect --run <dir> --arm <arm>
python3 gates/answers.py harvest --run <dir> --name <run> --arm <arm> --transcripts <dir>
python3 gates/answers.py score   --results <filsok results> --results <bundle results> [--tokens <file>]... [--out <file>]
  • setup copies every K1 source of the root into <run>/arms/<arm>/kilde/ (checked against K1; for bundle the collection goes to <run>/arms/bundle/wiki/ beside it) and writes <run>/tasks/<arm>/<id>.md: the question, the basis and the answer path, never the keys, the spans or the number of documents. It prints one line per task, S1-<run>-<arm>-<id> and the task path: the subagent's description and the file it is given. The set and --sources must have the same K1, unless --part-root is given (wave 5, order 31 A3): then the set names the K1 of the part root, the part's subset_of.manifest_sha256 is the K1 of --sources, and every source of the part stands in --sources with the same path and sha256; otherwise exit 1 with the reason. The arm then searches the whole of --sources, while answerset.py gate still judges the set against the part root.
  • collect gathers <run>/answers/<arm>/<id>.json into <run>/answers/<arm>.json (K5 point 4; the id comes from the file name).
  • harvest reads every *.jsonl under the transcripts directory: a subagent is matched by the description in its .meta.json, or, with no description, by the task path in its first prompt. Per question it sums the four usage fields over all runs (each message id once, at its largest value), adds output_tokens_derived (a lower bound for an output the transcript never recorded, as in okfwiki/cost.py), and counts isolation breaches as in evals/direction_probe.py: a tool call that reads or lists a path outside the arm's basis and the task file, a tool that is not allowed, or a write to another file than the answer. A task without a transcript exits 1. echo (wave 5, order 31 A2): in a Bash chain that starts with cd <basis> &&, echo with literal words is not a breach (order 13's helpers printed --- between searches: 21 of its 26 counted breaches, classified call by call before the change). Its arguments are checked as the search commands' are, so a path outside the basis, ~, .., a redirection, $ or a backtick is still a breach. The task text does not offer echo. No other rule was made: order 13 had no backtick or $ inside single quotes and no Read of the helper's own answer file (its three checks of the answer file were python3 and jq in Bash, which stay breaches). Four more kinds (wave 5, order 31 A2b, operator 19.10.2026): the price measurement B1 counted 9 breaches in 4 of 4 questions, none of which read a source outside the basis. Each kind has a test that failed first, with the opposite cases that stay breaches:
    1. 2>/dev/null right before a separator (;, |, &&, ||) is not a breach, as before a space or at the end. Every other redirection (> file, >/dev/null, 2>&1, 2> anywhere else) still is.
    2. The search word of grep/egrep/fgrep/rg (the value of -e/--regexp, or else the first argument that is not an option) and the script of sed (the value of -e/--expression, or else the first such argument) are not paths. With -f/--file the search words come from a file, none is positional, and the file is checked as a path. A file path after the search word, ~ and .. are still breaches. Order 39: the value of every other option that takes one (grep -C, --include, --exclude-from, rg --glob, --ignore-file and the rest, GNU and BSD) is consumed and checked as a path, so a file option's value outside the basis is a breach, and rg --pre or --hostname-bin (which run a program) is one. An option missing from the table in SEARCH_WORDS makes its value the search word, and a path there goes unchecked, so the lists must be whole. grep --context with a separate value is left out, because BSD grep takes its value only after = (--context 1 /x f searches for 1 in /x). A sed script is read command by command instead of checked as a path: w, W and e (also as flags of s) are breaches, the file after r or R is checked as a path, and a script that cannot be parsed is a breach. A flag cluster with i or I (-ni, -Ei, -I) is sed -i. An unquoted newline separates commands as ; does, and # is never read as a comment (bash reads it so only at the start of a word; a real comment is then checked as arguments, which is stricter).
    3. $ or a backtick inside single quotes is not an expansion. Unquoted or inside double quotes it still is, and so is an escaped backtick inside double quotes (order 31 A2 classed order 13's one as real).
    4. Read of the helper's own answer file is not a breach, and neither is a Bash command of the read-only commands plus jq whose only path outside the basis is the answer file: after cd <basis> &&, after cd into the answer's folder, or with no cd and the answer's absolute path. Outside the basis every operand must be the answer file, so another file in the answers folder, a glob, ls of the folder, a relative path without cd and jq -f are still breaches, as are python3, a write and jq on any other file. Outside the basis a pipe stage that names no file, as in jq . <answer> | head, is counted too (stricter than needed). The task text does not offer jq. Against order 13 the A2b rules move 2 of A2's 5 real calls (one jq on the own answer after cd into its folder, one sed script starting with /) and leave 3; on B1 they leave 1 of 9 (an escaped backtick in double quotes).
  • score gives per arm and group "x of n", by path count (2-3 and 4 or more, history included), the pairs (both, only filsok, only bundle, neither) with the two-sided exact McNemar p, the difference bundle - filsok with its paired 95 % interval (Newcombe 1998, Statist. Med. 17:2635-2650, method 10, tested against its Table III), and the smallest difference the set can show at 80 % power and alpha 0.05 with the measured share of discordant pairs (Connor 1987; tested against the sample sizes in PLAN.md § Styrke). It prints no line per question.

The questions and spans derive from Anthropic documentation, so the set lives in gates-local/answerset/ and the held out outside the repository (CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are tracked. The held-out questions are never opened during development; the manifest's sha256 is the check.

Tests: python3 -m unittest discover -s gates/tests. Types: mypy --strict gates/.