First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
79 lines
5.7 KiB
Markdown
79 lines
5.7 KiB
Markdown
# Multi-document set (E2)
|
|
|
|
The frozen question set for `PLAN.md` criterion 4 on collection 1: questions that each need at least four
|
|
different source documents seen together, written before the collection exists. `multidoc.py` holds the
|
|
three tools. None of them calls a model.
|
|
|
|
```
|
|
python3 gates/multidoc.py check --set gates-local/multidoc/set-60.json --sources <source dir> [--per-question]
|
|
python3 gates/multidoc.py gate --set gates-local/multidoc/set-60.json --manifest gates/multidoc.manifest.json \
|
|
--sources <source dir> --arm <name> --answers <answers.json> [--out <results.json>]
|
|
python3 gates/multidoc.py split --set <all.json> --sources <source dir> --seed <n> --held-out-count 20 \
|
|
--dev-out gates-local/multidoc/set-60.json --dev-label gates-local/multidoc/set-60.json \
|
|
--held-out <file OUTSIDE the repository> --manifest gates/multidoc.manifest.json
|
|
```
|
|
|
|
- **check** prints the denominators from the set itself: spans that stand verbatim (byte-exact) in their file,
|
|
questions whose every span does, questions with at least four distinct files, and questions where every file
|
|
carries an answer key that stands inside that file's spans (case-insensitive), the answer keys that are
|
|
wording, and the negated keys that stand negated in their file's spans (below). PASS only when all of them are
|
|
N of N and N > 0, no key is wording, and every negated key stands negated.
|
|
- **gate** scores one arm. A question counts when the answer contains every answer key (case-insensitive,
|
|
backticks and asterisks ignored, whitespace collapsed; a number key matches as a number and a negated key only
|
|
with a negation before it, below) AND the citations quote every needed file verbatim
|
|
(whitespace collapsed). A missing answers file scores 0 of N and exits 1 (fail-closed). The set and the source
|
|
directory must match the manifest's sha256, or the gate refuses to score. `--out` writes per-question results
|
|
for a paired test (McNemar) between two arms.
|
|
- **split** draws the held-out questions with `random.Random(seed).sample(sorted(ids), 20)`, renumbers both parts
|
|
(`d01..`, `h01..`) so the ids tell nothing about the draw, writes the held-out part outside the repository and
|
|
refuses to overwrite it. The manifest holds only the seed, counts and sha256 values.
|
|
|
|
## Answer keys are facts, not wording (E2c)
|
|
|
|
An answer key is the smallest fact the answer must hold: a name, a command, a flag, a setting, a number, a version,
|
|
or one word that carries the meaning. `keyform.py` counts the keys that are wording: two or more words with a
|
|
function word outside backticks (an article, a pronoun or determiner, a conjunction, a preposition, an auxiliary or
|
|
modal verb, a negation, or a focusing particle such as "only"). Such a key is split into the runs of its own words
|
|
without the function words, and each run must still stand in the span: "/vim or /config" becomes `/vim` and
|
|
`/config`, and "pin your model versions" becomes `pin` and `model version`. A key that is a whole number, a
|
|
percentage or an English number word matches as a number: its digits or its word, never inside a longer number,
|
|
version or identifier (`5` does not match `15` or `2.1.5`), and the 3-character minimum does not apply to it.
|
|
|
|
```
|
|
python3 gates/keyform.py --set <set.json> [--set <set.json> ...] [--list]
|
|
```
|
|
|
|
On 27.09.2026 (E2c) the four frozen parts held 104 wording keys of 495; after the rewrite they hold 0 of 523. Only
|
|
the keys changed. The manifests record the sha256 values from before under `keys_rewritten`.
|
|
|
|
## Negated keys (E2d)
|
|
|
|
Where a negation carried the fact ("not synced back" became `synced`), a key without it also accepts the opposite
|
|
answer. Such a key is marked `"negated": true`. It holds only where the answer has a negation at most six words
|
|
before it in the same sentence (a sentence ends at a line break or at `.`, `!`, `?` or `;` ending a word). The
|
|
negations are a fixed list: `not`, `no` (so also "no longer"), `never`, `none`, `neither`, `nor`, `without`, `cannot`,
|
|
and every word ending in `n't` ("isn't", "doesn't", "can't", "don't"; a curly apostrophe counts too). The same
|
|
function, `key_holds` in `multidoc.py`, scores both gates. A key without the mark holds as before, with or without a
|
|
negation. `check` requires every negated key to stand negated in one of its file's spans, so the mark is true to the
|
|
source. The `fasit` answer writes a negated key as "not <key>".
|
|
|
|
`polarity.py` counts the polarity keys that accept an answer of the opposite polarity. The polarity keys are listed
|
|
per set part (`{"schema": "okf-wiki/polarity-keys/1", "keys": [{"id", "atoms"}]}`). For each, the answer from the span is
|
|
the sentence of the key's spans where it stands negated, else the first sentence that holds it; the opposite answer is
|
|
that sentence with the negations within the window taken out, or with "not" put in front of the key when there are none.
|
|
|
|
```
|
|
python3 gates/polarity.py --set <set.json> --keys <keys.json> [--set ... --keys ...] [--list]
|
|
```
|
|
|
|
On 27.09.2026 (E2d) the list was E2c's own 22 polarity keys over the four parts. Before the mark, 22 of 22 accepted the
|
|
opposite answer. The mark went on the 17 that stand negated in their own spans; after it, 5 of 22 accept the opposite
|
|
answer. Those 5 carry a restriction, not a negation ("only" three times, "still", "keeps ... off"): nothing stands
|
|
before them to require, and the rule does not require the absence of a negation. The manifests record the sha256
|
|
values from before under `negated_marked`.
|
|
|
|
The questions and spans derive from Anthropic documentation, so they live in `gates-local/` and outside the
|
|
repository (CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are tracked.
|
|
The held-out questions are never used during development (R54).
|
|
|
|
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.
|