First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
238 lines
18 KiB
Markdown
238 lines
18 KiB
Markdown
# gates
|
|
|
|
## Mutant gate (E1)
|
|
|
|
`mutant_gate.py` runs an auditor command on every case in `mutants.manifest.json`: 37 mutants, each carrying a
|
|
defect of a named class, plus one known-positive bundle that must come back clean. It calls no model.
|
|
|
|
E1b added three link classes: `broken-link`, `orphan-page` and `index-coverage`. They are okf-wiki's own checks,
|
|
not SPEC § 11 conformance: SPEC § 6.1 says consumers MUST tolerate broken links. Their exact definitions are
|
|
in the manifest under `provenance.link_rules`, with criterion 6 as it stands (see below).
|
|
|
|
```
|
|
python3 gates/mutant_gate.py --manifest gates/mutants.manifest.json --cases gates-local/mutants \
|
|
--sources gates-local/mutants-kilder -- <auditor> {bundle} {sources}
|
|
```
|
|
|
|
- `{bundle}` becomes a fresh copy of the case bundle, and `{sources}` becomes the raw-source root.
|
|
- The auditor prints `{"findings": [{"check": "<class>", ...}]}` and exits 0 when clean or 1 when it raises an alarm.
|
|
Anything else counts as an error and catches nothing: a missing auditor, a crash, a timeout, output that is not
|
|
JSON, or an exit status that disagrees with the findings.
|
|
- The gate is GREEN only when every mutant is caught **with its own class**, the known-positive is clean and
|
|
nothing errored. Rejecting everything catches every mutant, but it raises a false alarm and matches no class.
|
|
- Before anything runs, every case file and every raw source listed in the manifest is checked against its
|
|
sha256. A mismatch stops the gate with exit 2.
|
|
|
|
The sources root is `gates-local/mutants-kilder/` (wave 1 order 4): byte-identical copies of the three documentation
|
|
files the pages cite and one release note, under `kilde/`, with `source-manifest.json` in the form of contract K1
|
|
(`docs/bolge1/0-felles.local.md` § 4, local). The manifest lists all five files with their sha256. Until 08.10.2026
|
|
the root was the copy of 21.09 (`~/okf/benk/wiki`); that root has no K1 file, so the gate now stops on it with exit 2.
|
|
|
|
The auditor is `okfwiki/audit.py` (L1a). With it the gate was GREEN at 23 of 23 class matched and 0 of 1 false alarms
|
|
before V1 added the link-rule mutants, and GREEN at 29 of 29 and 0 of 1 on 08.10.2026 with the manifest of that day and
|
|
`--sources ~/okf/benk/wiki`, before the main-page rule below. The command today (GREEN since wave 2 order 7, see below):
|
|
|
|
```
|
|
python3 gates/mutant_gate.py --manifest gates/mutants.manifest.json --cases gates-local/mutants \
|
|
--sources gates-local/mutants-kilder -- python3 -m okfwiki.audit '{bundle}' '{sources}'
|
|
```
|
|
|
|
In zsh, quote the placeholders as above when the output is redirected: `{sources} > file` opens a named file descriptor,
|
|
the auditor gets only `{bundle}`, and every case errors.
|
|
|
|
### Criterion 6 under the main-page rule (wave 1 order 4, RED)
|
|
|
|
Criterion 6 in `PLAN.md § Ferdig-kriteriet` as the operator set it on 06.10.2026 (choice 2): Google's rule as
|
|
written, applied to documents. Every documentation page has its main page, and every other concept page has a link from
|
|
at least one main page. Links are made by the model from meaning; the auditor checks structure only, without a model.
|
|
The exact definitions are in the manifest under `provenance.link_rules`, and the contract is K2:
|
|
|
|
- A concept page lists the sources it is the main page for in its frontmatter, as a YAML list under `x-main-for`. A
|
|
valid value is `/` + a path with `main_topic: true` in the K1 file at the sources root, and stands in the page's own
|
|
`sources[].resource`. Without a valid K1 file the rule cannot run (exit 2, never 0).
|
|
- `main-page-missing`: a main-topic source with no main page. `main-page-duplicate`: two main pages for one source.
|
|
`main-page-invalid`: not a list, a value not among the page's own sources, or a value that is not a main-topic source.
|
|
- `no-main-page-link`: a page that is not a main page has no link from a main page's body, in a sentence or in a list
|
|
of related concepts. `index.md`, `log.md` and the page itself do not count. A main page needs no inbound link.
|
|
- Unchanged: (a) `broken-link`, (b) `link-misplaced` (a link in code, a heading or a footnote definition),
|
|
`orphan-page`, `index-coverage`. `link-rule` (an inbound link from any other concept page, the rule of 04.10.2026)
|
|
is retired in Step 7; no mutant carries it any more.
|
|
|
|
The known-positive (normalisation N7): `dontask-modus.md` is the main page of all three documentation sources, and
|
|
has no inbound link from another concept page (only from `index.md`). Two non-main pages: `bypasspermissions-modus.md`
|
|
(linked from a sentence on the main page) and `acceptedits-modus.md` (linked from the main page's related-concepts list,
|
|
and from a sentence on the bypass page; it cites the release note). The main page has exactly one link to each, so a
|
|
mutant moves or removes "the one link". Every case is rebuilt from this known-positive, so every sha256 changed.
|
|
|
|
| Mutant | Class | Defect | Expected extra findings |
|
|
|---|---|---|---|
|
|
| m21 | `orphan-page` | acceptEdits no longer linked from any page or `index.md` (its `log.md` entry stays) | `index-coverage`, `no-main-page-link` |
|
|
| m24 | `no-main-page-link` | bypass linked only from `index.md` | none |
|
|
| m25 | `link-misplaced` | the main page's one link to bypass inside inline code | `no-main-page-link` |
|
|
| m26 | `link-misplaced` | the main page's one link to bypass moved into a heading | `no-main-page-link` |
|
|
| m27 | `link-misplaced` | the main page's one link to bypass moved into a footnote definition | `no-main-page-link` |
|
|
| m28 | `no-main-page-link` | the only concept-body link to bypass is a self-link (`index.md` and `log.md` still link it) | none |
|
|
| m29 | `no-main-page-link` | bypass linked only from `log.md` | `index-coverage`, `orphan-page` |
|
|
| m30 | `no-main-page-link` | acceptEdits linked only from the other non-main page (and `index.md`, `log.md`) | none |
|
|
| m31 | `main-page-missing` | the main page loses its mark for one documentation source | none |
|
|
| m32 | `main-page-duplicate` | bypass marks a source the main page marks (and bypass cites) | none |
|
|
| m33 | `main-page-invalid` | bypass's `x-main-for` is a string, not a list | none |
|
|
| m34 | `main-page-invalid` | bypass marks a main-topic source it does not cite | none |
|
|
| m35 | `main-page-invalid` | acceptEdits marks the release note it cites (in K1, not a main topic) | none |
|
|
| m36 | `main-page-duplicate` | the main page adds an invalid value, and bypass marks a source the main page marks | `main-page-invalid` |
|
|
| m37 | `no-main-page-link` | bypass marks only a source it does not cite, and the main page's one link to it is removed | `main-page-invalid` |
|
|
|
|
m36 and m37 test K2 point 2's last sentence (a page is a main page when at least one value is valid; invalid values do
|
|
not take that role away). Since the gate counts a mutant matched when its own class is among the findings, each carries a
|
|
class that only a right reading of the role gives, so each is two edits by design. Measured with two wrong readings of
|
|
the rule (scratch only): one that makes every page with the key a main page misses m37's class, and one that takes the
|
|
role from a page with an invalid value misses m36's; on the set without m36 and m37 both missed nothing. Under K2 only
|
|
links from main pages count, so a self-link on a non-main page can never count: m28 is kept because the order names the
|
|
case, but it tells K2 auditors apart no better than m24.
|
|
|
|
The extras are the classes the builder measures beside the case's own class, as the auditor of Step 7 should report them;
|
|
they are documentation (`expected_extra` per case), and the gate counts a mutant matched when its own class is among the
|
|
findings. Two older cases also gain an extra under K2: m11 (`main-page-missing`, `no-main-page-link`: the main page's
|
|
frontmatter no longer parses) and m12 (`no-main-page-link`: its extra page is linked from `index.md` alone). The
|
|
manifest maps every K2 case to its mutant under `provenance.k2.coverage`.
|
|
|
|
Recorded RED on 08.10.2026 with today's auditor (`okfwiki/audit.py` unchanged), with the command above: **class matched
|
|
26 of 37**. The old classes are matched 26 of 26 (m01-m23 and m25-m27). The new classes are matched **0 of 11**: m24 and
|
|
m28-m37 are caught only with the retired `link-rule` (m29 also with `index-coverage` and `orphan-page`). **False alarms
|
|
1 of 1, by design:** today's auditor reports `link-rule` on the main page of the known-positive, which has no inbound
|
|
concept link because K2 point 3 does not ask for one; Step 7 retires `link-rule`. Errors 0 of 38, exit 1. Step 7 must
|
|
turn this into 37 of 37 with 0 false alarms.
|
|
|
|
Wave 2 order 7 did (merged into `v1` 09.10.2026, measured in the main copy after the merge): caught 37 of 37 · class
|
|
matched 37 of 37 · false alarms 0 of 1 · errors 0 of 38, GREEN, exit 0. Since K2 point 15 (a link with `/` does not
|
|
count), m09 also gives `no-main-page-link` on `acceptedits-modus.md`; the gate counts only each case's own class, and
|
|
`expected_extra` for m09 in the frozen manifest still says `[]`.
|
|
|
|
History: V1 session 2 (05.10.2026) recorded RED at 23 of 29 for the rule of 04.10.2026 (an inbound link from any other
|
|
concept page, class `link-rule` on m24, m28 and m29), and Step 6 of the v1 plan turned it GREEN at 29 of 29.
|
|
|
|
The cases and their sources derive from Anthropic documentation, so they live in `gates-local/` (gitignored, CLAUDE.md
|
|
rule 3). Only the runner, its tests, the reject-all stub and the manifest (names, classes and sha256) are tracked.
|
|
|
|
## File-type gate (E4)
|
|
|
|
`filetype_gate.py` asks a reader where each frozen quote occurs, one run per source file. It measures the L1 number
|
|
"file types read with a locator, of 7". It calls no model.
|
|
|
|
```
|
|
python3 gates/filetype_gate.py --manifest gates/filetypes.manifest.json --pairs gates-local/filetypes/pairs.json \
|
|
--sources <folder holding blandet/ and wiki/> --cases gates-local/filetypes/files -- <reader> {file} {queries}
|
|
```
|
|
|
|
- `{file}` becomes a fresh copy of one source file, and `{queries}` becomes a JSON file of `{"id", "quote"}` objects.
|
|
The reader never sees the expected locator. It prints `{"answers": [{"id", "found": [{"locator", "hidden"}]}]}`:
|
|
every place where the quote occurs, after runs of whitespace are collapsed to one space.
|
|
- A pair is located when the answer names exactly one place, that place is the frozen locator, and `hidden` is right.
|
|
Hidden Office content (a hidden sheet, rows or columns, a hidden slide, hidden text, or a tracked deletion) must be
|
|
marked `hidden: true`, not flattened into ordinary text. Each quote occurs once in its file, so a second place is wrong.
|
|
- Locators: pdf `{"page": N}`, docx `{"paragraph": N}`, xlsx `{"sheet", "cell"}`, pptx `{"slide": N, "notes": bool}`,
|
|
and md, txt and html `{"line": N}`. The manifest defines each one and says why it was chosen.
|
|
- A file type counts as read when every one of its pairs is located. The gate prints this as `file types with every pair
|
|
located: X of 7`. The hidden classes are reported separately. The gate is GREEN only when every pair is located and no
|
|
reader run errs.
|
|
- Before anything runs, the pairs file and every source it names are checked against the manifest's sha256.
|
|
|
|
Working set: 112 pairs (16 per type) plus 16 in the hidden cases, from 64 files. A held-out set of 28 pairs (4 per type)
|
|
lies outside the repository, with only its sha256 here (same form as E2). How the fasit was made is in the manifest under
|
|
`fasit_method`. The stubs are `stubs/first_locator.py` (always the first page, paragraph, cell, slide or line) and
|
|
`stubs/fasit_reader.py` (the known-positive, which answers from the pairs file).
|
|
|
|
The engine's reader is `python3 -m okfwiki.readers locate {file} {queries}`. On 07.11.2026 it located 128 of 128 in the
|
|
working set (16 of 16 per type, every hidden class, 4 of 4 controls, 0 reader errors); `first_locator.py` located 7.
|
|
|
|
Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`.
|
|
|
|
## Answer set for collection 1 (wave 2 order 6, K5)
|
|
|
|
`answerset.py` and `answers.py` hold the answer set for collection 1 and the harness for the answer measurements
|
|
(contract K5); see `ANSWERSET.md`. It replaces E2 for the verdict.
|
|
|
|
## Whole-document and counting group (E2b)
|
|
|
|
`wholedoc.py` holds a small group of questions for the class the E2 set lacks: an answer that needs one long file read
|
|
as a whole, or a count over many places in one file. It is reported on its own and calls no model. It is a separate
|
|
gate, not a group inside `multidoc.py`, because that checker requires four or more files per question and E2 is frozen.
|
|
|
|
```
|
|
python3 gates/wholedoc.py cut --e2-set gates-local/multidoc/set-60.json --sources <source dir> --manifest gates/wholedoc.manifest.json
|
|
python3 gates/wholedoc.py check --set <set> --sources <source dir> --manifest gates/wholedoc.manifest.json --e2-set gates-local/multidoc/set-60.json
|
|
python3 gates/wholedoc.py split --set <all.json> ... --seed <n> --held-out-per-type 2 --held-out <file OUTSIDE the repository>
|
|
python3 gates/wholedoc.py gate --set gates-local/wholedoc/set-12.json --manifest gates/wholedoc.manifest.json \
|
|
--sources <source dir> --arm <name> --answers <answers.json> [--out <results.json>]
|
|
python3 gates/stubs/wholedoc_arm.py <one-span|fasit> <set> <answers.json>
|
|
```
|
|
|
|
- **cut** takes the sources from the E2 dev set: sorted by id, repeatedly the question needing the fewest new files, while
|
|
the union stays at 20 files or fewer. It wrote the manifest with the thresholds before any question existed.
|
|
- **check** measures every property in the source files. Every span stands verbatim and exactly once in its file. A
|
|
`whole` question sits on a file of 150 lines or more, with at least three keys, spans in at least three parts
|
|
(a part begins where two span starts lie 15 % of the file or more apart), the first span in the first third and the
|
|
last one in the last third, no single span holding every key, no key that is wording, and every key marked
|
|
`negated` standing negated in a span (see `MULTIDOC.md`, "Answer keys are facts, not wording" and "Negated keys").
|
|
A `count` question has a count of 5 or more that
|
|
equals its spans and the matches of its `item_pattern` in the file, and the count stands nowhere in the file as a
|
|
number or an English number word.
|
|
- **gate**: `whole` counts when the answer holds every key (a number key matches as a number, a negated key only with a
|
|
negation before it); `count` counts when the answer's `count` field is that exact
|
|
integer. Both need a citation that quotes the file verbatim. It prints `whole` and `count` separately, and GREEN
|
|
only when every question is correct. The `one-span` arm must score 0 (RED), and the `fasit` arm is the known-positive
|
|
(it writes a negated key as "not <key>").
|
|
|
|
Frozen 27.09.2026: 16 questions (8 `whole` on the three documentation files of 150 lines or more in the cut, 8 `count`
|
|
on five files of the cut) were written from the sources after the thresholds were committed. Seed 20260927 held out 2 per
|
|
type (4 questions, outside the repository, sha256 only here), which leaves 12 for development (6 `whole`, 6 `count`).
|
|
`check` is PASS on both parts. On the development part the gate is RED at 0 of 12 with the `one-span` arm and GREEN at
|
|
12 of 12 with the `fasit` arm. Without an answers file it is fail-closed at 0 of 12.
|
|
|
|
## Quote rule for K's fragments (V1 Step 4, brief SC3)
|
|
|
|
`quote_rule.py` measures two rules that close the path P1 saw (the compiler wrote numbers as words to slip past a rule
|
|
that asked for a quote only for a digit or a backticked name). It calls no model and prints counts only.
|
|
|
|
```
|
|
python3 -m gates.quote_rule measure ~/okf/benk/wiki/prove-p1/K-fragments ~/okf/benk/wiki/prove-p1/K-work/index.json
|
|
```
|
|
|
|
- **Rule E:** every fragment carries a quote. **Rule L:** a quote is required for a digit, a backticked name or a word
|
|
from the narrow list (zero to twenty, hundred, thousand, million).
|
|
- **Answer key**, made apart from rule L: a quoteless fragment whose text holds a word from the broad list of brief SC3
|
|
(the narrow list plus thirty to ninety, dozen, dozens, once, twice, first to tenth, half, double, pair, couple,
|
|
several, single), as a whole word. The denominator is the number of hits.
|
|
- **Choice:** E, unless L lets 0 key fragments through and E rejects at least 33 of 650 (5 %) more than L.
|
|
|
|
Measured on 04.10.2026 on P1's 650 fragments (all 650 accepted in P1; 160 have no quote):
|
|
|
|
| Count | Value |
|
|
|---|---|
|
|
| E rejects | 160 of 650 |
|
|
| L rejects | 10 of 650 |
|
|
| Answer-key hits (denominator) | 26 (10 hold a narrow-list word) |
|
|
| Key hits per file | pack-1.json 19, pack-4.json 7 |
|
|
| Key fragments E lets through | 0 of 26 |
|
|
| Key fragments L lets through | 16 of 26 |
|
|
| E rejects minus L rejects | 150 (threshold 33) |
|
|
|
|
**Chose E because L lets 16 of the 26 key fragments through** (the rule needs 0); E rejects 150 more fragments than
|
|
L, so the cost condition alone would also have allowed L, but the slip condition rules it out. The known-positive holds
|
|
(26 hits, at least 1). The P1 README says the compiler wrote numbers as words "in one pack" without naming it; the key
|
|
finds hits in packs 1 and 4, and the four compiler transcripts end in a tool call with no report text, so which pack the
|
|
README meant is not measured.
|
|
|
|
New reject rules, adopted in Step 6 only if each rejects at most 5 % of the 650 (32):
|
|
|
|
| Rule | Rejects | Adopt |
|
|
|---|---|---|
|
|
| short quote (fewer than 3 words) | 34 of 650 (5.2 %) | no: over 5 %, reported to the PM |
|
|
| id mismatch (an `existing` id whose entry matches the concept name neither by normalised form nor by Jaccard >= 0.6) | 0 of 650 | yes |
|
|
|
|
The 0 id mismatches were checked against a known-positive: with every fragment given its neighbour's id, the check
|
|
flags 454 of 650, which is every fragment whose neighbour has a different id (196 neighbours share one).
|
|
|
|
`NEEDS_QUOTE`, `MERGE_JACCARD` and `norm` are copies of the trial's, kept equal by `TrialConstantsTest`, which reads the
|
|
trial at run time. Chose copies over an import, because `mypy --strict okfwiki gates tests` does not see the trial
|
|
package (it is found only when `evals` is listed), so an import would turn the v1 mypy line red once `gates/` is frozen.
|