First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
182 lines
16 KiB
Markdown
182 lines
16 KiB
Markdown
# Answer set for collection 1 (wave 2, order 6)
|
|
|
|
The frozen question set for `PLAN.md` criterion 4 on collection 1, made from the fresh frozen copy (contract K1) so
|
|
that it resembles the collection: questions that need two or more documents seen together, a group of version-history
|
|
questions reported on its own, and 20 questions held out outside the repository. It replaces the E2 set
|
|
(`MULTIDOC.md`) for the verdict, because 33 of E2's 60 rest on release notes alone. `answerset.py` holds the gate and
|
|
`answers.py` the harness the answer measurements use (contract K5). None of them calls a model or starts one.
|
|
|
|
```
|
|
python3 gates/answerset.py check --set <set.json> --sources <K1 root> [--per-question] [--min-paths 3 --min-many 20]
|
|
python3 gates/answerset.py inside --set <set.json> --part-root <K1 subset root> [--require-all]
|
|
python3 gates/answerset.py split --set <all.json> --sources <K1 root> --seed <n> --held-out-count 20 \
|
|
--set-out gates-local/answerset/set-60.json --set-label gates-local/answerset/set-60.json \
|
|
--held-out <file OUTSIDE the repository> --part gates-local/answerset/part-del1.json \
|
|
--part-sources <part root> --part-label gates-local/answerset/part-del1.json --manifest gates/answerset.manifest.json
|
|
python3 gates/answerset.py freeze --set <drafts.json> --sources <K1 root> --seed <n> --count 30 --min-paths 3 \
|
|
--min-many 20 --prefix x --set-out <set.json> --set-label <label> --manifest <manifest.json>
|
|
python3 gates/answerset.py gate --set <set.json> --manifest gates/answerset.manifest.json --sources <K1 root> \
|
|
--arm filsok|bundle --answers <answers.json> [--out <results.json>]
|
|
python3 gates/answerset.py stub --set <set.json> --arm filsok|bundle --kind fasit|no-citations --out <answers.json>
|
|
```
|
|
|
|
- **check** reads K1 with its own code (`okfwiki.sources` is not imported). The set names the root it was made from
|
|
(`sources.frozen_at` and `sources.manifest_sha256`, the sha256 of the root's `source-manifest.json`), and every cited
|
|
file must be listed in K1 with its sha256. It prints the denominators from the set itself: spans that stand byte for
|
|
byte in their file (20-400 characters), questions with every span standing, questions with at least two distinct
|
|
paths, questions whose `group` is the one K1 and the path count give, questions where every path carries an answer
|
|
key inside that path's spans, wording keys (`keyform.py`, as in E2c) and negated keys that stand negated (E2d). PASS
|
|
only when every count is N of N with N > 0.
|
|
- **The group is set by the sources:** `history` when one path has `main_topic: false` in K1 (a release note or a
|
|
weekly What's New page), otherwise `docs-2-3` (2-3 distinct paths) or `docs-4` (4 or more).
|
|
- **inside** counts the questions whose every span stands in its file in a part root (a K1 subset root). On the 60 it is
|
|
a number for the report; on the part set it is a requirement (`--require-all`).
|
|
- **split** checks both the set and the part first (PASS required, and every part question inside the part root),
|
|
then holds out per group in the fixed order `docs-2-3`, `docs-4`, `history`, with quotas proportional to the group
|
|
sizes (largest remainder) and one `random.Random(seed)` drawing `sample(sorted(ids of the group), quota)`. Both parts
|
|
are renumbered in the order of the original ids (`d01..`, `h01..`). The held-out file must lie outside the
|
|
repository and is never overwritten. The manifest holds the seed, counts per group, sha256 values and the two roots'
|
|
K1 sha256, and no text.
|
|
- **gate** scores one arm with the gates' one key test (`multidoc.key_holds`): a question is correct when the answer
|
|
holds every key and the citations quote every needed path verbatim (whitespace collapsed; `path` is the K1 path,
|
|
and a leading `/` as in a page's `resource` is accepted). The set's sha256 must be one of the manifest's parts, and
|
|
the root's K1 sha256 must be that part's root (the copy for the set and the held out, the first part for the part
|
|
set). A missing answers file scores 0 of N, RED, exit 1 (fail-closed). `--out` writes the results file of K5 point 5,
|
|
sorted by id. GREEN only when every question is correct; exit 0 when every question was answered.
|
|
- **stub** writes a known arm: `fasit` (every key as its own sentence, "not <key>" when negated, and every span quoted)
|
|
and `no-citations` (the same answer, no citations).
|
|
|
|
## Frozen 09.10.2026 (wave 2, order 6)
|
|
|
|
Written from the frozen copy of 08.10.2026 (635 sources, K1 `manifest_sha256` `8c9f87fd…`) by helpers in the user's
|
|
session (Opus), each question from a pack of byte-exact source lines that code prepared around rare names shared by an
|
|
anchor page and its candidates, spread over the sections of the page overview in proportion to their text. A separate
|
|
helper that did not write the question read every question against windows around its spans: the keys (facts a correct
|
|
answer must hold) and whether every file is necessary, helped by a code pre-check over all 635 files for another file
|
|
that holds the keys of two of them. Questions it marked FIX were repaired and, when a file was not necessary, rebuilt
|
|
and read again; the rest were dropped.
|
|
|
|
| Part | Questions | docs-2-3 | docs-4 | history | Confirmed by a separate reader |
|
|
|---|---:|---:|---:|---:|---:|
|
|
| all, before the split | 80 | 35 | 34 | 11 | 80 of 80 |
|
|
| `set-60.json` | 60 | 26 | 26 | 8 | |
|
|
| held out (outside the repository) | 20 | 9 | 8 | 3 | |
|
|
| `part-del1.json` (first part only, `p01`-`p18`) | 18 | 9 | 9 | 0 | 18 of 18 |
|
|
|
|
- **check:** PASS on the 80 (389 of 389 spans, 0 of 391 keys wording, 13 of 13 negated keys negated) and on the part
|
|
against the first part's root (91 of 91 spans, 0 of 86 wording, 5 of 5 negated); PASS again on `set-60.json`.
|
|
- **History:** 11 of 80 (13.8 %); the release notes and weekly What's New pages are 12.45 % of the copy's characters
|
|
(1,149,271 of 9,231,505). The first part holds no history source, so the part set has none.
|
|
- **split:** seed 20261009; held out per group docs-2-3 9, docs-4 8, history 3.
|
|
- **inside** the first part: 1 of the 60; 18 of 18 of the part set.
|
|
- **Proof, no model:** without an answers file 0 of 60, RED, exit 1; the fasit stub 60 of 60, GREEN; right answers
|
|
without citations 0 of 60 (correct answer 60 of 60). On the part set: 0 of 18 RED exit 1, 18 of 18 GREEN, 0 of 18.
|
|
|
|
## The harder set on the first part (wave 4, order 21, frozen 12.10.2026)
|
|
|
|
Wave 3 measured the 18 part questions near the ceiling (both arms 17 of 18), so order 21 made a harder set from the
|
|
first part's 38 files: every question needs at least 3 documents and at least 20 of 30 need 4 or more, every file is
|
|
necessary, and at least one key is a condition or relation that only two of the files give together. None of the 18
|
|
is reused.
|
|
|
|
- **check** with `--min-paths N` counts the questions with N or more distinct paths and with `--min-many M` requires
|
|
at least M questions with 4 or more (`questions with >= 4 distinct paths: x of n (required >= M)`). The defaults
|
|
(2 and 0) print and judge exactly as before, so the order 6 sets keep their verdict.
|
|
- **freeze** freezes a set that `split` did not make: it refuses an existing set or manifest, a set that fails the
|
|
check with the thresholds, or a set with another count than `--count`; it shuffles the sorted ids with
|
|
`random.Random(seed)`, renumbers them `x01..`, and writes a manifest of its own (seed, rules, counts per group, the
|
|
set's sha256, the root's and its copy's K1 sha256; no text) that names the set the part of its root, so `gate`
|
|
scores it as it scores the part.
|
|
- **Made** by helpers in the user's session (Opus) from packs of byte-exact lines that code built around backticked
|
|
names standing in 2-6 of the 38 files; one draft helper per question; a separate review helper per question that
|
|
did not write it read the question against windows around its spans, a code pre-check over all 38 files, and
|
|
confirmed every file necessary and the join; FIX was repaired (a repair that changed files was read again), DROP
|
|
dropped. Two questions sharing two keys in one file count as overlap, and the higher id goes. A surplus over 30
|
|
keeps every docs-4 and drops docs-2-3 with the highest id.
|
|
|
|
| `gates-local/answerset/part-del1-hard.json` (`x01`-`x30`) | Questions | docs-2-3 | docs-4 | Confirmed by a separate reader |
|
|
|---|---:|---:|---:|---:|
|
|
| frozen, seed 20261011 | 30 | 4 | 26 | 30 of 30 |
|
|
|
|
- **check** against the first part with `--min-paths 3 --min-many 20`: PASS, 30 of 30 with 3 or more paths, 26 of 30
|
|
with 4 or more; **inside** the first part 30 of 30.
|
|
- **Proof, no model, both arms:** without an answers file 0 of 30, RED; the fasit stub 30 of 30, GREEN; right answers
|
|
without citations 0 of 30 (correct answer 30 of 30), RED.
|
|
|
|
```
|
|
python3 gates/answerset.py gate --set gates-local/answerset/part-del1-hard.json \
|
|
--manifest gates/answerset-del1-hard.manifest.json --sources <first part root> --arm filsok|bundle --answers <file>
|
|
```
|
|
|
|
## The harness (`answers.py`, K5 points 2-7)
|
|
|
|
```
|
|
python3 gates/answers.py setup --run <dir> --name <run> --arm filsok|bundle --set <set.json> --sources <K1 root> \
|
|
[--bundle <collection>] [--part-root <the K1 root the set was made from>]
|
|
python3 gates/answers.py collect --run <dir> --arm <arm>
|
|
python3 gates/answers.py harvest --run <dir> --name <run> --arm <arm> --transcripts <dir>
|
|
python3 gates/answers.py score --results <filsok results> --results <bundle results> [--tokens <file>]... [--out <file>]
|
|
```
|
|
|
|
- **setup** copies every K1 source of the root into `<run>/arms/<arm>/kilde/` (checked against K1; for `bundle` the
|
|
collection goes to `<run>/arms/bundle/wiki/` beside it) and writes `<run>/tasks/<arm>/<id>.md`: the question, the
|
|
basis and the answer path, never the keys, the spans or the number of documents. It prints one line per task,
|
|
`S1-<run>-<arm>-<id>` and the task path: the subagent's description and the file it is given.
|
|
The set and `--sources` must have the same K1, unless `--part-root` is given (wave 5, order 31 A3): then the set
|
|
names the K1 of the part root, the part's `subset_of.manifest_sha256` is the K1 of `--sources`, and every source of
|
|
the part stands in `--sources` with the same path and sha256; otherwise exit 1 with the reason. The arm then searches
|
|
the whole of `--sources`, while `answerset.py gate` still judges the set against the part root.
|
|
- **collect** gathers `<run>/answers/<arm>/<id>.json` into `<run>/answers/<arm>.json` (K5 point 4; the id comes from the
|
|
file name).
|
|
- **harvest** reads every `*.jsonl` under the transcripts directory: a subagent is matched by the description in its
|
|
`.meta.json`, or, with no description, by the task path in its first prompt. Per question it sums the four usage
|
|
fields over all runs (each message id once, at its largest value), adds `output_tokens_derived` (a lower bound for
|
|
an output the transcript never recorded, as in `okfwiki/cost.py`), and counts isolation breaches as in
|
|
`evals/direction_probe.py`: a tool call that reads or lists a path outside the arm's basis and the task file, a tool
|
|
that is not allowed, or a write to another file than the answer. A task without a transcript exits 1.
|
|
**`echo` (wave 5, order 31 A2):** in a Bash chain that starts with `cd <basis> &&`, `echo` with literal words is not
|
|
a breach (order 13's helpers printed `---` between searches: 21 of its 26 counted breaches, classified call by call
|
|
before the change). Its arguments are checked as the search commands' are, so a path outside the basis, `~`, `..`,
|
|
a redirection, `$` or a backtick is still a breach. The task text does not offer `echo`. No other rule was made:
|
|
order 13 had no backtick or `$` inside single quotes and no `Read` of the helper's own answer file (its three checks
|
|
of the answer file were `python3` and `jq` in Bash, which stay breaches).
|
|
**Four more kinds (wave 5, order 31 A2b, operator 19.10.2026):** the price measurement B1 counted 9 breaches in 4 of
|
|
4 questions, none of which read a source outside the basis. Each kind has a test that failed first, with the
|
|
opposite cases that stay breaches:
|
|
1. `2>/dev/null` right before a separator (`;`, `|`, `&&`, `||`) is not a breach, as before a space or at the end.
|
|
Every other redirection (`> file`, `>/dev/null`, `2>&1`, `2>` anywhere else) still is.
|
|
2. The search word of `grep`/`egrep`/`fgrep`/`rg` (the value of `-e`/`--regexp`, or else the first argument that is
|
|
not an option) and the script of `sed` (the value of `-e`/`--expression`, or else the first such argument) are not
|
|
paths. With `-f`/`--file` the search words come from a file, none is positional, and the file is checked as a path.
|
|
A file path after the search word, `~` and `..` are still breaches. **Order 39:** the value of every other option
|
|
that takes one (`grep -C`, `--include`, `--exclude-from`, `rg --glob`, `--ignore-file` and the rest, GNU and BSD)
|
|
is consumed and checked as a path, so a file option's value outside the basis is a breach, and `rg --pre` or
|
|
`--hostname-bin` (which run a program) is one. An option missing from the table in `SEARCH_WORDS` makes its value
|
|
the search word, and a path there goes unchecked, so the lists must be whole. `grep --context` with a separate
|
|
value is left out, because BSD grep takes its value only after `=` (`--context 1 /x f` searches for 1 in `/x`).
|
|
A `sed` script is read command by command instead of checked as a path: `w`, `W` and `e` (also as flags of `s`)
|
|
are breaches, the file after `r` or `R` is checked as a path, and a script that cannot be parsed is a breach. A
|
|
flag cluster with `i` or `I` (`-ni`, `-Ei`, `-I`) is `sed -i`. An unquoted newline separates commands as `;`
|
|
does, and `#` is never read as a comment (bash reads it so only at the start of a word; a real comment is then
|
|
checked as arguments, which is stricter).
|
|
3. `$` or a backtick inside single quotes is not an expansion. Unquoted or inside double quotes it still is, and so is
|
|
an escaped backtick inside double quotes (order 31 A2 classed order 13's one as real).
|
|
4. `Read` of the helper's own answer file is not a breach, and neither is a Bash command of the read-only commands
|
|
plus `jq` whose only path outside the basis is the answer file: after `cd <basis> &&`, after `cd` into the answer's
|
|
folder, or with no `cd` and the answer's absolute path. Outside the basis every operand must be the answer file, so
|
|
another file in the answers folder, a glob, `ls` of the folder, a relative path without `cd` and `jq -f` are still
|
|
breaches, as are `python3`, a write and `jq` on any other file. Outside the basis a pipe stage that names no
|
|
file, as in `jq . <answer> | head`, is counted too (stricter than needed). The task text does not offer `jq`.
|
|
Against order 13 the A2b rules move 2 of A2's 5 real calls (one `jq` on the own answer after `cd` into its folder,
|
|
one `sed` script starting with `/`) and leave 3; on B1 they leave 1 of 9 (an escaped backtick in double quotes).
|
|
- **score** gives per arm and group "x of n", by path count (2-3 and 4 or more, history included), the pairs (both,
|
|
only filsok, only bundle, neither) with the two-sided exact McNemar p, the difference bundle - filsok with its paired
|
|
95 % interval (Newcombe 1998, Statist. Med. 17:2635-2650, method 10, tested against its Table III), and the smallest
|
|
difference the set can show at 80 % power and alpha 0.05 with the measured share of discordant pairs (Connor 1987;
|
|
tested against the sample sizes in `PLAN.md § Styrke`). It prints no line per question.
|
|
|
|
The questions and spans derive from Anthropic documentation, so the set lives in `gates-local/answerset/` and the held
|
|
out outside the repository (CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are
|
|
tracked. The held-out questions are never opened during development; the manifest's sha256 is the check.
|
|
|
|
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.
|