First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
173 lines
15 KiB
Markdown
173 lines
15 KiB
Markdown
# Reader calibration (order 3: 300 of 300, calls of up to ten claims)
|
|
|
|
The frozen calibration set for `PLAN.md` criterion 2: before an independent reader may judge the claims on the wiki's
|
|
pages, it must show that it can tell a claim that follows from its source excerpts from one that does not. From
|
|
06.10.2026 the reader judges up to ten claims per call and gives one verdict per claim (`CLAUDE.md` § The shape; K3 in
|
|
the wave-1 contracts), so the calibration measures the reader in calls, not one claim at a time. `calibration.py` holds
|
|
the tools. None of them calls a model, and the reader pass itself is not here (Step 9 of the v1 plan).
|
|
|
|
```
|
|
python3 gates/calibration.py check --draft <draft.json> --sources <source dir>
|
|
python3 gates/calibration.py check --set gates-local/calibration/pairs-300.json --key gates-local/calibration/key-300.json \
|
|
--calls gates-local/calibration/calls-300.json --manifest gates/calibration.manifest.json --sources <source dir>
|
|
python3 gates/calibration.py build --draft <draft.json> --sources <source dir> --seed <n> --lone 100 --dense 5 \
|
|
--set-out gates-local/calibration/pairs-300.json --key-out gates-local/calibration/key-300.json \
|
|
--calls-out gates-local/calibration/calls-300.json --manifest gates/calibration.manifest.json \
|
|
--set-label <path> --key-label <path> --calls-label <path>
|
|
python3 gates/calibration.py stub --set <set> (--verdict supported|unsupported | --key <key>) \
|
|
[--calls <layout> --accept-at <1-10>] --out <verdicts.json>
|
|
python3 gates/calibration.py gate --set <set> --key <key> --calls <layout> --manifest gates/calibration.manifest.json \
|
|
--sources <source dir> --verdicts <verdicts.json> [--passed <passed.json>]
|
|
```
|
|
|
|
## The three files and the labels
|
|
|
|
- **The set** (`pairs-300.json`) is what the reader sees: per claim an id, the claim and its excerpts (file, line,
|
|
text). The ids follow the call order and say nothing about the label.
|
|
- **The key** (`key-300.json`) gives per claim the label, the kind, the draft id and a one-sentence reason.
|
|
- **The call layout** (`calls-300.json`, schema `okf-wiki/reader-calls/1`): `call_size_max` (10), the calls with their
|
|
claim ids in position order, and `set_sha256` that binds the layout to the set. Every claim stands in exactly one
|
|
call; its position is its index in the call plus one (1-10).
|
|
|
|
Labels: `unsupported` (planted, must be caught), `supported` (counted controls, must be accepted) and
|
|
`filler-supported`. Fillers are supported claims that fill the calls so that most calls look like a real page: one
|
|
wrong claim, if any, among right ones. They are checked like supported claims (verbatim excerpts, not a copy of the
|
|
excerpt) and audited, are reported apart ("filler accepted x of y"), and never count in the thresholds. A filler may
|
|
name the unsupported pair it is the twin of (`twin_of`); the build never puts the two in the same call.
|
|
|
|
## The tools
|
|
|
|
- **check** prints the denominators from the pairs themselves: excerpts that stand verbatim (byte-exact) in their file
|
|
and, for a built set, at their stated line; pairs per label and per kind. It fails when a kind of the two counted
|
|
labels has no pair, when a kind does not fit its label, when a `wrong-number` claim has no number that is absent from
|
|
its excerpts, when a `conflated-sources` pair does not quote two files, when a supported or filler claim is a
|
|
verbatim copy of its excerpt, or when a `twin_of` does not name an unsupported pair. There is no rule of equal
|
|
numbers any more: a built set is checked against the manifest instead, which states the counts per label, per kind
|
|
and per position, the sha256 of set, key, layout and sources, and the layout's own counts. The layout must hold every
|
|
claim once, no call over ten, the right `set_sha256`, and unsupported claims at every position 1-10. Check prints
|
|
the layout as a distribution (unsupported per call, unsupported alone in their call, unsupported per position).
|
|
- **build** checks a labelled draft and lays it out with a fixed seed (the rule is written into the manifest as
|
|
`shuffle`): `--lone` unsupported pairs stand alone in a call, at positions spread evenly over 1-10, the rest stand
|
|
`--dense` to a call; the counted supported pairs go one per call, in call order (so into the lone calls first);
|
|
fillers fill every call to ten, taken in a cycle, never twice in one call and never beside their twin. A filler may
|
|
therefore stand in several calls under different claim ids; the manifest gives the slots and the distinct fillers.
|
|
Nothing is written when the layout breaks a rule. A built set is frozen: build refuses to overwrite the set, the key
|
|
or the layout. An earlier manifest at the same path moves into the new manifest's `history`.
|
|
- **stub** writes verdicts without a reader: the same verdict for every claim (reject-all, accept-all), or the key's
|
|
verdicts (the known-positive), and with `--calls` and `--accept-at P` "supported" for every claim at position P (the
|
|
position-mild stub). A stub names the model `stub`, which no reader transcript names.
|
|
- **gate** scores one reader's verdicts (`{"reader": "<name>", "models": ["<model>"], "verdicts": [{"id", "verdict"}]}`,
|
|
a verdict being `supported` or `unsupported`) and prints the model it scored (`model: <name>`), the totals, the
|
|
caught count per kind, `caught by position: 1 x of y, ..., 10 x of y` and the fillers apart. It never prints or writes a result per call or per pair (K3 point 5; `CLAUDE.md`:
|
|
the reader is improved only from totals and counts per kind). A missing verdict, an invalid value or an id given
|
|
twice counts as wrong (fail-closed); a missing verdicts file scores 0. GREEN only when every unsupported claim is
|
|
caught at every position, at least `accepted_min` counted supported claims are accepted and no verdict is missing,
|
|
fillers included, and the verdicts name exactly one model; verdicts that name no model, or two or more, are RED
|
|
(fail-closed). A RED run ends with one `not met:` line that names what failed, positions included. The set, the
|
|
key, the layout and the sources must match the manifest, or the gate refuses to score.
|
|
|
|
## The gate is bound to the reader's model (PLAN § Rammer 23, 31.10.2026)
|
|
|
|
The reader's agent file names a model family (`model: opus`), and the family alias moves to a new model when one is
|
|
released. A pass of the gate holds only for the model that was scored, so the gate and the reading are tied to it:
|
|
|
|
1. The gate run's readers are collected with `okfwiki.reader collect ... --calibration`. Collect reads the model of
|
|
every reader transcript it uses (`message.model` of the assistant records, as Claude Code writes it) and writes the
|
|
sorted, distinct models into the verdicts file as `models`.
|
|
2. `gate ... --passed <passed.json>` adds a GREEN run to the record of green runs (schema
|
|
`okf-wiki/calibration-passed/1`): `{"schema": ..., "passed": [{"model", "set_sha256", "key_sha256",
|
|
"verdicts_sha256", "reader"}]}`. The record has no time in it, the same entry is not added twice, and a RED run never
|
|
writes it. A record that cannot be read is a FAIL and is left as it was.
|
|
3. A reading of the wiki is collected with `okfwiki.reader collect ... --gate-passed <passed.json>`. When the
|
|
transcripts name no model, or a model that has no entry in the record, collect writes nothing and exits 1. A new
|
|
model therefore needs a new GREEN gate run before its verdicts count.
|
|
|
|
Thresholds (in the manifest): `caught_min` is every unsupported claim (the operator, 06.10: 300 of 300).
|
|
`accepted_min` is 90 % of the counted supported claims, rounded up (45 of 50), chosen by the PM in E3 because a reader
|
|
that rejects more than one supported claim in ten makes the writing step unusable.
|
|
|
|
Kinds of unsupported claim: `overgeneralisation`, `wrong-number`, `swapped-cause`, `not-mentioned`, `reversed-polarity`,
|
|
`conflated-sources` (two excerpts from two files, one subject's property given to the other). Kinds of supported and
|
|
filler claim: `paraphrase`, `inference` (one reasoning step from one excerpt), `synthesis` (needs both excerpts).
|
|
|
|
The pairs derive from Anthropic documentation, so the set, the key, the layout and the drafts live in `gates-local/`
|
|
(CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are tracked. The reader is shown
|
|
the set and the layout only, never the key or the drafts.
|
|
|
|
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.
|
|
|
|
## The set (order 3, 08.10.2026)
|
|
|
|
Built with seed 20261008, `--lone 100 --dense 5`, against the 599 sources of the copy of 21.09 (the same
|
|
`tree_sha256` as E3). 1,400 claims in 140 calls of ten:
|
|
|
|
- **300 unsupported**, 50 of each kind: E3's 50, and 250 new ones from 150 docs pages E3 did not use (277 of the 300
|
|
quote a docs page first, 92.3 %; collection 1 is 92.2 % documentation by text). Each new one was written as one small
|
|
change from a supported reading of its excerpt (its twin).
|
|
- **50 supported** that count: E3's 50 (paraphrase 15, inference 20, synthesis 15), each once, all in lone calls.
|
|
- **1,050 filler slots** from 250 distinct fillers: the twins of the 250 new unsupported pairs, each used 3 to 5 times
|
|
under different claim ids, never in the call of its own unsupported pair. The twins cost almost nothing beyond the
|
|
error itself, and they never count in the thresholds.
|
|
- **Layout:** 100 calls with one unsupported claim among nine supported ones (like a real page), 40 calls with five.
|
|
Unsupported claims per position 1-10: 33, 30, 28, 23, 32, 23, 30, 24, 39, 38.
|
|
|
|
Authoring and audit: 25 batches of ten unsupported pairs and their twins, each written by one helper (Opus) from six
|
|
pages and audited by another with the key visible, judging against the excerpts only. 500 new pairs: 455 OK, 45 FIX,
|
|
0 WRONG; every FIX took the auditor's wording (most were twins too close to the excerpt's wording, with the unsupported
|
|
claim reworded to stay one change from the new twin). `check` on the built set: 1,642 of 1,642 excerpts verbatim and
|
|
at their line.
|
|
|
|
Proofs without a reader (verdicts written before 31.10.2026 name no model and are RED under the model rule; a stub
|
|
written again names the model `stub`): no verdicts 0 of 300, RED; reject-all 300 of 300 and 0 of 50, RED; accept-all 0 of 300, RED;
|
|
the key accepting everything at one position RED at each of the ten positions, with that position named; the key as
|
|
judge GREEN.
|
|
|
|
## The folder set (motor B16, 08.11.2026)
|
|
|
|
The set above is Claude Code documentation and stays out of git. A folder collection (Norwegian and English PDF and
|
|
Office documents) needs its own measurement, so a second set is tracked in full: `gates/calibration-folder/` holds
|
|
the documents (`original/`), the text the engine froze from them (`kilde/`, the reader's sources), the draft, the
|
|
set, the key and the layout; `gates/calibration-folder.manifest.json` is its manifest. The same tools score it:
|
|
|
|
```
|
|
python3 gates/calibration.py gate --set gates/calibration-folder/pairs.json --key gates/calibration-folder/key.json \
|
|
--calls gates/calibration-folder/calls.json --manifest gates/calibration-folder.manifest.json \
|
|
--sources gates/calibration-folder/kilde --verdicts <verdicts.json> [--passed gates/calibration.passed.json]
|
|
```
|
|
|
|
- **The documents:** 32 written for the set, four per invented organisation (a housing cooperative, a sports club, a
|
|
school band, a community farm in Norwegian; a makerspace, an open-source project, an astronomy society, a sailing
|
|
club in English): per language 4 PDF, 4 Word, 4 Excel, 4 PowerPoint. No text from Anthropic or any other source.
|
|
Written by eight helpers (Opus) from one brief, rendered with python-docx, openpyxl and python-pptx, PDF through
|
|
LibreOffice from a Word file, and frozen with `python3 -m okfwiki.sources folder` (32 sources, 0 skipped; `lang`
|
|
16 no and 16 en, all right).
|
|
- **The pairs:** 24 batches, three per organisation, each from its own third of every document (spreadsheet parts keep
|
|
every sheet's title and header row). 300 unsupported (50 per kind), each with its twin as a filler (300), and 50
|
|
counted supported (paraphrase 15, inference 20, synthesis 15). By the document type of the first excerpt: docx 81,
|
|
pdf 78, pptx 71, xlsx 70. Audit 1 (one helper per batch, key visible, excerpts only): 487 OK, 163 FIX, 0 WRONG;
|
|
most fixes took out names or details that stand in the document but not in the excerpt. Audit 2 read the 163 fixed
|
|
pairs and their twins again: 140 OK, 33 FIX, 0 WRONG. One more pair rested on a word only in the file name, fixed.
|
|
- **Built** with seed 20261108, `--lone 100 --dense 5`: 1,400 claims in 140 calls, unsupported per position 1-10:
|
|
32, 33, 25, 27, 30, 32, 28, 31, 33, 29. `check`: 1,666 of 1,666 excerpts verbatim and at their line.
|
|
- **Proofs without a reader:** no verdicts, reject-all, accept-all RED; the key accepting everything at one position
|
|
RED at each of the ten positions, with that position named; the key GREEN.
|
|
- **First reading (run `kalmappe1`, 08.11.2026, claude-opus-5-5): RED.** Unsupported caught 299 of 300 (wrong-number
|
|
49 of 50, position 5 29 of 30, every other kind and position in full), supported accepted 50 of 50, fillers 1,045
|
|
of 1,050, 140 of 140 calls answered. The one miss was checked against the key (operator 08.11, the one exception
|
|
to judging the reader from totals only): the pair was right, the claim gave a difference of two figures that the
|
|
excerpt did not print. The reader's number check now says to work out a derived figure itself before judging.
|
|
- **Second reading, both sets, new instructions (runs `kalmappe2`, `kaldoc2`, 08.11.2026, claude-opus-5-5, as one
|
|
workflow): GREEN on both.** Folder set: caught 300 of 300, accepted 50 of 50, fillers 1,047 of 1,050. Docs set:
|
|
caught 300 of 300, accepted 49 of 50, fillers 1,047 of 1,050. Both runs are in `gates/calibration.passed.json`.
|
|
- **Collect needs both:** `okfwiki.reader collect --gate-passed` checks every manifest given with `--manifest`, by
|
|
default both tracked ones, so a model reads a collection only after a green run on each set.
|
|
|
|
## History: E3 (27.09.2026)
|
|
|
|
The first set had 100 pairs of (claim, excerpt) judged one claim per call: 50 planted unsupported claims
|
|
(overgeneralisation 9, wrong-number 9, swapped-cause 8, not-mentioned 8, reversed-polarity 8, conflated-sources 8) and
|
|
50 supported controls (paraphrase 15, inference 20, synthesis 15), from 90 source files of the copy of 21.09 (42 docs
|
|
pages, 48 release notes). Five authors wrote the pairs and two auditors read every label with the key visible: 93 OK,
|
|
7 FIX, 0 WRONG. The thresholds were 50 of 50 and 45 of 50, and `check` required equal numbers of the two labels. Its
|
|
manifest values stand under `history` in `calibration.manifest.json`; its files (`draft-100.json`, `pairs-100.json`,
|
|
`key-100.json`) stay in `gates-local/calibration/` untouched. Its 100 pairs are reused in the new set.
|