okf-wiki/gates/MULTIDOC.md
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

79 lines
5.7 KiB
Markdown

# Multi-document set (E2)
The frozen question set for `PLAN.md` criterion 4 on collection 1: questions that each need at least four
different source documents seen together, written before the collection exists. `multidoc.py` holds the
three tools. None of them calls a model.
```
python3 gates/multidoc.py check --set gates-local/multidoc/set-60.json --sources <source dir> [--per-question]
python3 gates/multidoc.py gate --set gates-local/multidoc/set-60.json --manifest gates/multidoc.manifest.json \
--sources <source dir> --arm <name> --answers <answers.json> [--out <results.json>]
python3 gates/multidoc.py split --set <all.json> --sources <source dir> --seed <n> --held-out-count 20 \
--dev-out gates-local/multidoc/set-60.json --dev-label gates-local/multidoc/set-60.json \
--held-out <file OUTSIDE the repository> --manifest gates/multidoc.manifest.json
```
- **check** prints the denominators from the set itself: spans that stand verbatim (byte-exact) in their file,
questions whose every span does, questions with at least four distinct files, and questions where every file
carries an answer key that stands inside that file's spans (case-insensitive), the answer keys that are
wording, and the negated keys that stand negated in their file's spans (below). PASS only when all of them are
N of N and N > 0, no key is wording, and every negated key stands negated.
- **gate** scores one arm. A question counts when the answer contains every answer key (case-insensitive,
backticks and asterisks ignored, whitespace collapsed; a number key matches as a number and a negated key only
with a negation before it, below) AND the citations quote every needed file verbatim
(whitespace collapsed). A missing answers file scores 0 of N and exits 1 (fail-closed). The set and the source
directory must match the manifest's sha256, or the gate refuses to score. `--out` writes per-question results
for a paired test (McNemar) between two arms.
- **split** draws the held-out questions with `random.Random(seed).sample(sorted(ids), 20)`, renumbers both parts
(`d01..`, `h01..`) so the ids tell nothing about the draw, writes the held-out part outside the repository and
refuses to overwrite it. The manifest holds only the seed, counts and sha256 values.
## Answer keys are facts, not wording (E2c)
An answer key is the smallest fact the answer must hold: a name, a command, a flag, a setting, a number, a version,
or one word that carries the meaning. `keyform.py` counts the keys that are wording: two or more words with a
function word outside backticks (an article, a pronoun or determiner, a conjunction, a preposition, an auxiliary or
modal verb, a negation, or a focusing particle such as "only"). Such a key is split into the runs of its own words
without the function words, and each run must still stand in the span: "/vim or /config" becomes `/vim` and
`/config`, and "pin your model versions" becomes `pin` and `model version`. A key that is a whole number, a
percentage or an English number word matches as a number: its digits or its word, never inside a longer number,
version or identifier (`5` does not match `15` or `2.1.5`), and the 3-character minimum does not apply to it.
```
python3 gates/keyform.py --set <set.json> [--set <set.json> ...] [--list]
```
On 27.09.2026 (E2c) the four frozen parts held 104 wording keys of 495; after the rewrite they hold 0 of 523. Only
the keys changed. The manifests record the sha256 values from before under `keys_rewritten`.
## Negated keys (E2d)
Where a negation carried the fact ("not synced back" became `synced`), a key without it also accepts the opposite
answer. Such a key is marked `"negated": true`. It holds only where the answer has a negation at most six words
before it in the same sentence (a sentence ends at a line break or at `.`, `!`, `?` or `;` ending a word). The
negations are a fixed list: `not`, `no` (so also "no longer"), `never`, `none`, `neither`, `nor`, `without`, `cannot`,
and every word ending in `n't` ("isn't", "doesn't", "can't", "don't"; a curly apostrophe counts too). The same
function, `key_holds` in `multidoc.py`, scores both gates. A key without the mark holds as before, with or without a
negation. `check` requires every negated key to stand negated in one of its file's spans, so the mark is true to the
source. The `fasit` answer writes a negated key as "not <key>".
`polarity.py` counts the polarity keys that accept an answer of the opposite polarity. The polarity keys are listed
per set part (`{"schema": "okf-wiki/polarity-keys/1", "keys": [{"id", "atoms"}]}`). For each, the answer from the span is
the sentence of the key's spans where it stands negated, else the first sentence that holds it; the opposite answer is
that sentence with the negations within the window taken out, or with "not" put in front of the key when there are none.
```
python3 gates/polarity.py --set <set.json> --keys <keys.json> [--set ... --keys ...] [--list]
```
On 27.09.2026 (E2d) the list was E2c's own 22 polarity keys over the four parts. Before the mark, 22 of 22 accepted the
opposite answer. The mark went on the 17 that stand negated in their own spans; after it, 5 of 22 accept the opposite
answer. Those 5 carry a restriction, not a negation ("only" three times, "still", "keeps ... off"): nothing stands
before them to require, and the rule does not require the absence of a negation. The manifests record the sha256
values from before under `negated_marked`.
The questions and spans derive from Anthropic documentation, so they live in `gates-local/` and outside the
repository (CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are tracked.
The held-out questions are never used during development (R54).
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.