okf-wiki/gates/CALIBRATION.md
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

173 lines
15 KiB
Markdown

# Reader calibration (order 3: 300 of 300, calls of up to ten claims)
The frozen calibration set for `PLAN.md` criterion 2: before an independent reader may judge the claims on the wiki's
pages, it must show that it can tell a claim that follows from its source excerpts from one that does not. From
06.10.2026 the reader judges up to ten claims per call and gives one verdict per claim (`CLAUDE.md` § The shape; K3 in
the wave-1 contracts), so the calibration measures the reader in calls, not one claim at a time. `calibration.py` holds
the tools. None of them calls a model, and the reader pass itself is not here (Step 9 of the v1 plan).
```
python3 gates/calibration.py check --draft <draft.json> --sources <source dir>
python3 gates/calibration.py check --set gates-local/calibration/pairs-300.json --key gates-local/calibration/key-300.json \
--calls gates-local/calibration/calls-300.json --manifest gates/calibration.manifest.json --sources <source dir>
python3 gates/calibration.py build --draft <draft.json> --sources <source dir> --seed <n> --lone 100 --dense 5 \
--set-out gates-local/calibration/pairs-300.json --key-out gates-local/calibration/key-300.json \
--calls-out gates-local/calibration/calls-300.json --manifest gates/calibration.manifest.json \
--set-label <path> --key-label <path> --calls-label <path>
python3 gates/calibration.py stub --set <set> (--verdict supported|unsupported | --key <key>) \
[--calls <layout> --accept-at <1-10>] --out <verdicts.json>
python3 gates/calibration.py gate --set <set> --key <key> --calls <layout> --manifest gates/calibration.manifest.json \
--sources <source dir> --verdicts <verdicts.json> [--passed <passed.json>]
```
## The three files and the labels
- **The set** (`pairs-300.json`) is what the reader sees: per claim an id, the claim and its excerpts (file, line,
text). The ids follow the call order and say nothing about the label.
- **The key** (`key-300.json`) gives per claim the label, the kind, the draft id and a one-sentence reason.
- **The call layout** (`calls-300.json`, schema `okf-wiki/reader-calls/1`): `call_size_max` (10), the calls with their
claim ids in position order, and `set_sha256` that binds the layout to the set. Every claim stands in exactly one
call; its position is its index in the call plus one (1-10).
Labels: `unsupported` (planted, must be caught), `supported` (counted controls, must be accepted) and
`filler-supported`. Fillers are supported claims that fill the calls so that most calls look like a real page: one
wrong claim, if any, among right ones. They are checked like supported claims (verbatim excerpts, not a copy of the
excerpt) and audited, are reported apart ("filler accepted x of y"), and never count in the thresholds. A filler may
name the unsupported pair it is the twin of (`twin_of`); the build never puts the two in the same call.
## The tools
- **check** prints the denominators from the pairs themselves: excerpts that stand verbatim (byte-exact) in their file
and, for a built set, at their stated line; pairs per label and per kind. It fails when a kind of the two counted
labels has no pair, when a kind does not fit its label, when a `wrong-number` claim has no number that is absent from
its excerpts, when a `conflated-sources` pair does not quote two files, when a supported or filler claim is a
verbatim copy of its excerpt, or when a `twin_of` does not name an unsupported pair. There is no rule of equal
numbers any more: a built set is checked against the manifest instead, which states the counts per label, per kind
and per position, the sha256 of set, key, layout and sources, and the layout's own counts. The layout must hold every
claim once, no call over ten, the right `set_sha256`, and unsupported claims at every position 1-10. Check prints
the layout as a distribution (unsupported per call, unsupported alone in their call, unsupported per position).
- **build** checks a labelled draft and lays it out with a fixed seed (the rule is written into the manifest as
`shuffle`): `--lone` unsupported pairs stand alone in a call, at positions spread evenly over 1-10, the rest stand
`--dense` to a call; the counted supported pairs go one per call, in call order (so into the lone calls first);
fillers fill every call to ten, taken in a cycle, never twice in one call and never beside their twin. A filler may
therefore stand in several calls under different claim ids; the manifest gives the slots and the distinct fillers.
Nothing is written when the layout breaks a rule. A built set is frozen: build refuses to overwrite the set, the key
or the layout. An earlier manifest at the same path moves into the new manifest's `history`.
- **stub** writes verdicts without a reader: the same verdict for every claim (reject-all, accept-all), or the key's
verdicts (the known-positive), and with `--calls` and `--accept-at P` "supported" for every claim at position P (the
position-mild stub). A stub names the model `stub`, which no reader transcript names.
- **gate** scores one reader's verdicts (`{"reader": "<name>", "models": ["<model>"], "verdicts": [{"id", "verdict"}]}`,
a verdict being `supported` or `unsupported`) and prints the model it scored (`model: <name>`), the totals, the
caught count per kind, `caught by position: 1 x of y, ..., 10 x of y` and the fillers apart. It never prints or writes a result per call or per pair (K3 point 5; `CLAUDE.md`:
the reader is improved only from totals and counts per kind). A missing verdict, an invalid value or an id given
twice counts as wrong (fail-closed); a missing verdicts file scores 0. GREEN only when every unsupported claim is
caught at every position, at least `accepted_min` counted supported claims are accepted and no verdict is missing,
fillers included, and the verdicts name exactly one model; verdicts that name no model, or two or more, are RED
(fail-closed). A RED run ends with one `not met:` line that names what failed, positions included. The set, the
key, the layout and the sources must match the manifest, or the gate refuses to score.
## The gate is bound to the reader's model (PLAN § Rammer 23, 31.10.2026)
The reader's agent file names a model family (`model: opus`), and the family alias moves to a new model when one is
released. A pass of the gate holds only for the model that was scored, so the gate and the reading are tied to it:
1. The gate run's readers are collected with `okfwiki.reader collect ... --calibration`. Collect reads the model of
every reader transcript it uses (`message.model` of the assistant records, as Claude Code writes it) and writes the
sorted, distinct models into the verdicts file as `models`.
2. `gate ... --passed <passed.json>` adds a GREEN run to the record of green runs (schema
`okf-wiki/calibration-passed/1`): `{"schema": ..., "passed": [{"model", "set_sha256", "key_sha256",
"verdicts_sha256", "reader"}]}`. The record has no time in it, the same entry is not added twice, and a RED run never
writes it. A record that cannot be read is a FAIL and is left as it was.
3. A reading of the wiki is collected with `okfwiki.reader collect ... --gate-passed <passed.json>`. When the
transcripts name no model, or a model that has no entry in the record, collect writes nothing and exits 1. A new
model therefore needs a new GREEN gate run before its verdicts count.
Thresholds (in the manifest): `caught_min` is every unsupported claim (the operator, 06.10: 300 of 300).
`accepted_min` is 90 % of the counted supported claims, rounded up (45 of 50), chosen by the PM in E3 because a reader
that rejects more than one supported claim in ten makes the writing step unusable.
Kinds of unsupported claim: `overgeneralisation`, `wrong-number`, `swapped-cause`, `not-mentioned`, `reversed-polarity`,
`conflated-sources` (two excerpts from two files, one subject's property given to the other). Kinds of supported and
filler claim: `paraphrase`, `inference` (one reasoning step from one excerpt), `synthesis` (needs both excerpts).
The pairs derive from Anthropic documentation, so the set, the key, the layout and the drafts live in `gates-local/`
(CLAUDE.md rule 3). Only the tools, their tests (synthetic fixtures) and the manifest are tracked. The reader is shown
the set and the layout only, never the key or the drafts.
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.
## The set (order 3, 08.10.2026)
Built with seed 20261008, `--lone 100 --dense 5`, against the 599 sources of the copy of 21.09 (the same
`tree_sha256` as E3). 1,400 claims in 140 calls of ten:
- **300 unsupported**, 50 of each kind: E3's 50, and 250 new ones from 150 docs pages E3 did not use (277 of the 300
quote a docs page first, 92.3 %; collection 1 is 92.2 % documentation by text). Each new one was written as one small
change from a supported reading of its excerpt (its twin).
- **50 supported** that count: E3's 50 (paraphrase 15, inference 20, synthesis 15), each once, all in lone calls.
- **1,050 filler slots** from 250 distinct fillers: the twins of the 250 new unsupported pairs, each used 3 to 5 times
under different claim ids, never in the call of its own unsupported pair. The twins cost almost nothing beyond the
error itself, and they never count in the thresholds.
- **Layout:** 100 calls with one unsupported claim among nine supported ones (like a real page), 40 calls with five.
Unsupported claims per position 1-10: 33, 30, 28, 23, 32, 23, 30, 24, 39, 38.
Authoring and audit: 25 batches of ten unsupported pairs and their twins, each written by one helper (Opus) from six
pages and audited by another with the key visible, judging against the excerpts only. 500 new pairs: 455 OK, 45 FIX,
0 WRONG; every FIX took the auditor's wording (most were twins too close to the excerpt's wording, with the unsupported
claim reworded to stay one change from the new twin). `check` on the built set: 1,642 of 1,642 excerpts verbatim and
at their line.
Proofs without a reader (verdicts written before 31.10.2026 name no model and are RED under the model rule; a stub
written again names the model `stub`): no verdicts 0 of 300, RED; reject-all 300 of 300 and 0 of 50, RED; accept-all 0 of 300, RED;
the key accepting everything at one position RED at each of the ten positions, with that position named; the key as
judge GREEN.
## The folder set (motor B16, 08.11.2026)
The set above is Claude Code documentation and stays out of git. A folder collection (Norwegian and English PDF and
Office documents) needs its own measurement, so a second set is tracked in full: `gates/calibration-folder/` holds
the documents (`original/`), the text the engine froze from them (`kilde/`, the reader's sources), the draft, the
set, the key and the layout; `gates/calibration-folder.manifest.json` is its manifest. The same tools score it:
```
python3 gates/calibration.py gate --set gates/calibration-folder/pairs.json --key gates/calibration-folder/key.json \
--calls gates/calibration-folder/calls.json --manifest gates/calibration-folder.manifest.json \
--sources gates/calibration-folder/kilde --verdicts <verdicts.json> [--passed gates/calibration.passed.json]
```
- **The documents:** 32 written for the set, four per invented organisation (a housing cooperative, a sports club, a
school band, a community farm in Norwegian; a makerspace, an open-source project, an astronomy society, a sailing
club in English): per language 4 PDF, 4 Word, 4 Excel, 4 PowerPoint. No text from Anthropic or any other source.
Written by eight helpers (Opus) from one brief, rendered with python-docx, openpyxl and python-pptx, PDF through
LibreOffice from a Word file, and frozen with `python3 -m okfwiki.sources folder` (32 sources, 0 skipped; `lang`
16 no and 16 en, all right).
- **The pairs:** 24 batches, three per organisation, each from its own third of every document (spreadsheet parts keep
every sheet's title and header row). 300 unsupported (50 per kind), each with its twin as a filler (300), and 50
counted supported (paraphrase 15, inference 20, synthesis 15). By the document type of the first excerpt: docx 81,
pdf 78, pptx 71, xlsx 70. Audit 1 (one helper per batch, key visible, excerpts only): 487 OK, 163 FIX, 0 WRONG;
most fixes took out names or details that stand in the document but not in the excerpt. Audit 2 read the 163 fixed
pairs and their twins again: 140 OK, 33 FIX, 0 WRONG. One more pair rested on a word only in the file name, fixed.
- **Built** with seed 20261108, `--lone 100 --dense 5`: 1,400 claims in 140 calls, unsupported per position 1-10:
32, 33, 25, 27, 30, 32, 28, 31, 33, 29. `check`: 1,666 of 1,666 excerpts verbatim and at their line.
- **Proofs without a reader:** no verdicts, reject-all, accept-all RED; the key accepting everything at one position
RED at each of the ten positions, with that position named; the key GREEN.
- **First reading (run `kalmappe1`, 08.11.2026, claude-opus-5-5): RED.** Unsupported caught 299 of 300 (wrong-number
49 of 50, position 5 29 of 30, every other kind and position in full), supported accepted 50 of 50, fillers 1,045
of 1,050, 140 of 140 calls answered. The one miss was checked against the key (operator 08.11, the one exception
to judging the reader from totals only): the pair was right, the claim gave a difference of two figures that the
excerpt did not print. The reader's number check now says to work out a derived figure itself before judging.
- **Second reading, both sets, new instructions (runs `kalmappe2`, `kaldoc2`, 08.11.2026, claude-opus-5-5, as one
workflow): GREEN on both.** Folder set: caught 300 of 300, accepted 50 of 50, fillers 1,047 of 1,050. Docs set:
caught 300 of 300, accepted 49 of 50, fillers 1,047 of 1,050. Both runs are in `gates/calibration.passed.json`.
- **Collect needs both:** `okfwiki.reader collect --gate-passed` checks every manifest given with `--manifest`, by
default both tracked ones, so a model reads a collection only after a green run on each set.
## History: E3 (27.09.2026)
The first set had 100 pairs of (claim, excerpt) judged one claim per call: 50 planted unsupported claims
(overgeneralisation 9, wrong-number 9, swapped-cause 8, not-mentioned 8, reversed-polarity 8, conflated-sources 8) and
50 supported controls (paraphrase 15, inference 20, synthesis 15), from 90 source files of the copy of 21.09 (42 docs
pages, 48 release notes). Five authors wrote the pairs and two auditors read every label with the key visible: 93 OK,
7 FIX, 0 WRONG. The thresholds were 50 of 50 and 45 of 50, and `check` required equal numbers of the two labels. Its
manifest values stand under `history` in `calibration.manifest.json`; its files (`draft-100.json`, `pairs-100.json`,
`key-100.json`) stay in `gates-local/calibration/` untouched. Its 100 pairs are reused in the new set.