# gates ## Mutant gate (E1) `mutant_gate.py` runs an auditor command on every case in `mutants.manifest.json`: 37 mutants, each carrying a defect of a named class, plus one known-positive bundle that must come back clean. It calls no model. E1b added three link classes: `broken-link`, `orphan-page` and `index-coverage`. They are okf-wiki's own checks, not SPEC § 11 conformance: SPEC § 6.1 says consumers MUST tolerate broken links. Their exact definitions are in the manifest under `provenance.link_rules`, with criterion 6 as it stands (see below). ``` python3 gates/mutant_gate.py --manifest gates/mutants.manifest.json --cases gates-local/mutants \ --sources gates-local/mutants-kilder -- {bundle} {sources} ``` - `{bundle}` becomes a fresh copy of the case bundle, and `{sources}` becomes the raw-source root. - The auditor prints `{"findings": [{"check": "", ...}]}` and exits 0 when clean or 1 when it raises an alarm. Anything else counts as an error and catches nothing: a missing auditor, a crash, a timeout, output that is not JSON, or an exit status that disagrees with the findings. - The gate is GREEN only when every mutant is caught **with its own class**, the known-positive is clean and nothing errored. Rejecting everything catches every mutant, but it raises a false alarm and matches no class. - Before anything runs, every case file and every raw source listed in the manifest is checked against its sha256. A mismatch stops the gate with exit 2. The sources root is `gates-local/mutants-kilder/` (wave 1 order 4): byte-identical copies of the three documentation files the pages cite and one release note, under `kilde/`, with `source-manifest.json` in the form of contract K1 (`docs/bolge1/0-felles.local.md` § 4, local). The manifest lists all five files with their sha256. Until 08.10.2026 the root was the copy of 21.09 (`~/okf/benk/wiki`); that root has no K1 file, so the gate now stops on it with exit 2. The auditor is `okfwiki/audit.py` (L1a). With it the gate was GREEN at 23 of 23 class matched and 0 of 1 false alarms before V1 added the link-rule mutants, and GREEN at 29 of 29 and 0 of 1 on 08.10.2026 with the manifest of that day and `--sources ~/okf/benk/wiki`, before the main-page rule below. The command today (GREEN since wave 2 order 7, see below): ``` python3 gates/mutant_gate.py --manifest gates/mutants.manifest.json --cases gates-local/mutants \ --sources gates-local/mutants-kilder -- python3 -m okfwiki.audit '{bundle}' '{sources}' ``` In zsh, quote the placeholders as above when the output is redirected: `{sources} > file` opens a named file descriptor, the auditor gets only `{bundle}`, and every case errors. ### Criterion 6 under the main-page rule (wave 1 order 4, RED) Criterion 6 in `PLAN.md § Ferdig-kriteriet` as the operator set it on 06.10.2026 (choice 2): Google's rule as written, applied to documents. Every documentation page has its main page, and every other concept page has a link from at least one main page. Links are made by the model from meaning; the auditor checks structure only, without a model. The exact definitions are in the manifest under `provenance.link_rules`, and the contract is K2: - A concept page lists the sources it is the main page for in its frontmatter, as a YAML list under `x-main-for`. A valid value is `/` + a path with `main_topic: true` in the K1 file at the sources root, and stands in the page's own `sources[].resource`. Without a valid K1 file the rule cannot run (exit 2, never 0). - `main-page-missing`: a main-topic source with no main page. `main-page-duplicate`: two main pages for one source. `main-page-invalid`: not a list, a value not among the page's own sources, or a value that is not a main-topic source. - `no-main-page-link`: a page that is not a main page has no link from a main page's body, in a sentence or in a list of related concepts. `index.md`, `log.md` and the page itself do not count. A main page needs no inbound link. - Unchanged: (a) `broken-link`, (b) `link-misplaced` (a link in code, a heading or a footnote definition), `orphan-page`, `index-coverage`. `link-rule` (an inbound link from any other concept page, the rule of 04.10.2026) is retired in Step 7; no mutant carries it any more. The known-positive (normalisation N7): `dontask-modus.md` is the main page of all three documentation sources, and has no inbound link from another concept page (only from `index.md`). Two non-main pages: `bypasspermissions-modus.md` (linked from a sentence on the main page) and `acceptedits-modus.md` (linked from the main page's related-concepts list, and from a sentence on the bypass page; it cites the release note). The main page has exactly one link to each, so a mutant moves or removes "the one link". Every case is rebuilt from this known-positive, so every sha256 changed. | Mutant | Class | Defect | Expected extra findings | |---|---|---|---| | m21 | `orphan-page` | acceptEdits no longer linked from any page or `index.md` (its `log.md` entry stays) | `index-coverage`, `no-main-page-link` | | m24 | `no-main-page-link` | bypass linked only from `index.md` | none | | m25 | `link-misplaced` | the main page's one link to bypass inside inline code | `no-main-page-link` | | m26 | `link-misplaced` | the main page's one link to bypass moved into a heading | `no-main-page-link` | | m27 | `link-misplaced` | the main page's one link to bypass moved into a footnote definition | `no-main-page-link` | | m28 | `no-main-page-link` | the only concept-body link to bypass is a self-link (`index.md` and `log.md` still link it) | none | | m29 | `no-main-page-link` | bypass linked only from `log.md` | `index-coverage`, `orphan-page` | | m30 | `no-main-page-link` | acceptEdits linked only from the other non-main page (and `index.md`, `log.md`) | none | | m31 | `main-page-missing` | the main page loses its mark for one documentation source | none | | m32 | `main-page-duplicate` | bypass marks a source the main page marks (and bypass cites) | none | | m33 | `main-page-invalid` | bypass's `x-main-for` is a string, not a list | none | | m34 | `main-page-invalid` | bypass marks a main-topic source it does not cite | none | | m35 | `main-page-invalid` | acceptEdits marks the release note it cites (in K1, not a main topic) | none | | m36 | `main-page-duplicate` | the main page adds an invalid value, and bypass marks a source the main page marks | `main-page-invalid` | | m37 | `no-main-page-link` | bypass marks only a source it does not cite, and the main page's one link to it is removed | `main-page-invalid` | m36 and m37 test K2 point 2's last sentence (a page is a main page when at least one value is valid; invalid values do not take that role away). Since the gate counts a mutant matched when its own class is among the findings, each carries a class that only a right reading of the role gives, so each is two edits by design. Measured with two wrong readings of the rule (scratch only): one that makes every page with the key a main page misses m37's class, and one that takes the role from a page with an invalid value misses m36's; on the set without m36 and m37 both missed nothing. Under K2 only links from main pages count, so a self-link on a non-main page can never count: m28 is kept because the order names the case, but it tells K2 auditors apart no better than m24. The extras are the classes the builder measures beside the case's own class, as the auditor of Step 7 should report them; they are documentation (`expected_extra` per case), and the gate counts a mutant matched when its own class is among the findings. Two older cases also gain an extra under K2: m11 (`main-page-missing`, `no-main-page-link`: the main page's frontmatter no longer parses) and m12 (`no-main-page-link`: its extra page is linked from `index.md` alone). The manifest maps every K2 case to its mutant under `provenance.k2.coverage`. Recorded RED on 08.10.2026 with today's auditor (`okfwiki/audit.py` unchanged), with the command above: **class matched 26 of 37**. The old classes are matched 26 of 26 (m01-m23 and m25-m27). The new classes are matched **0 of 11**: m24 and m28-m37 are caught only with the retired `link-rule` (m29 also with `index-coverage` and `orphan-page`). **False alarms 1 of 1, by design:** today's auditor reports `link-rule` on the main page of the known-positive, which has no inbound concept link because K2 point 3 does not ask for one; Step 7 retires `link-rule`. Errors 0 of 38, exit 1. Step 7 must turn this into 37 of 37 with 0 false alarms. Wave 2 order 7 did (merged into `v1` 09.10.2026, measured in the main copy after the merge): caught 37 of 37 · class matched 37 of 37 · false alarms 0 of 1 · errors 0 of 38, GREEN, exit 0. Since K2 point 15 (a link with `/` does not count), m09 also gives `no-main-page-link` on `acceptedits-modus.md`; the gate counts only each case's own class, and `expected_extra` for m09 in the frozen manifest still says `[]`. History: V1 session 2 (05.10.2026) recorded RED at 23 of 29 for the rule of 04.10.2026 (an inbound link from any other concept page, class `link-rule` on m24, m28 and m29), and Step 6 of the v1 plan turned it GREEN at 29 of 29. The cases and their sources derive from Anthropic documentation, so they live in `gates-local/` (gitignored, CLAUDE.md rule 3). Only the runner, its tests, the reject-all stub and the manifest (names, classes and sha256) are tracked. ## File-type gate (E4) `filetype_gate.py` asks a reader where each frozen quote occurs, one run per source file. It measures the L1 number "file types read with a locator, of 7". It calls no model. ``` python3 gates/filetype_gate.py --manifest gates/filetypes.manifest.json --pairs gates-local/filetypes/pairs.json \ --sources --cases gates-local/filetypes/files -- {file} {queries} ``` - `{file}` becomes a fresh copy of one source file, and `{queries}` becomes a JSON file of `{"id", "quote"}` objects. The reader never sees the expected locator. It prints `{"answers": [{"id", "found": [{"locator", "hidden"}]}]}`: every place where the quote occurs, after runs of whitespace are collapsed to one space. - A pair is located when the answer names exactly one place, that place is the frozen locator, and `hidden` is right. Hidden Office content (a hidden sheet, rows or columns, a hidden slide, hidden text, or a tracked deletion) must be marked `hidden: true`, not flattened into ordinary text. Each quote occurs once in its file, so a second place is wrong. - Locators: pdf `{"page": N}`, docx `{"paragraph": N}`, xlsx `{"sheet", "cell"}`, pptx `{"slide": N, "notes": bool}`, and md, txt and html `{"line": N}`. The manifest defines each one and says why it was chosen. - A file type counts as read when every one of its pairs is located. The gate prints this as `file types with every pair located: X of 7`. The hidden classes are reported separately. The gate is GREEN only when every pair is located and no reader run errs. - Before anything runs, the pairs file and every source it names are checked against the manifest's sha256. Working set: 112 pairs (16 per type) plus 16 in the hidden cases, from 64 files. A held-out set of 28 pairs (4 per type) lies outside the repository, with only its sha256 here (same form as E2). How the fasit was made is in the manifest under `fasit_method`. The stubs are `stubs/first_locator.py` (always the first page, paragraph, cell, slide or line) and `stubs/fasit_reader.py` (the known-positive, which answers from the pairs file). The engine's reader is `python3 -m okfwiki.readers locate {file} {queries}`. On 07.11.2026 it located 128 of 128 in the working set (16 of 16 per type, every hidden class, 4 of 4 controls, 0 reader errors); `first_locator.py` located 7. Tests: `python3 -m unittest discover -s gates/tests`. Types: `mypy --strict gates/`. ## Answer set for collection 1 (wave 2 order 6, K5) `answerset.py` and `answers.py` hold the answer set for collection 1 and the harness for the answer measurements (contract K5); see `ANSWERSET.md`. It replaces E2 for the verdict. ## Whole-document and counting group (E2b) `wholedoc.py` holds a small group of questions for the class the E2 set lacks: an answer that needs one long file read as a whole, or a count over many places in one file. It is reported on its own and calls no model. It is a separate gate, not a group inside `multidoc.py`, because that checker requires four or more files per question and E2 is frozen. ``` python3 gates/wholedoc.py cut --e2-set gates-local/multidoc/set-60.json --sources --manifest gates/wholedoc.manifest.json python3 gates/wholedoc.py check --set --sources --manifest gates/wholedoc.manifest.json --e2-set gates-local/multidoc/set-60.json python3 gates/wholedoc.py split --set ... --seed --held-out-per-type 2 --held-out python3 gates/wholedoc.py gate --set gates-local/wholedoc/set-12.json --manifest gates/wholedoc.manifest.json \ --sources --arm --answers [--out ] python3 gates/stubs/wholedoc_arm.py ``` - **cut** takes the sources from the E2 dev set: sorted by id, repeatedly the question needing the fewest new files, while the union stays at 20 files or fewer. It wrote the manifest with the thresholds before any question existed. - **check** measures every property in the source files. Every span stands verbatim and exactly once in its file. A `whole` question sits on a file of 150 lines or more, with at least three keys, spans in at least three parts (a part begins where two span starts lie 15 % of the file or more apart), the first span in the first third and the last one in the last third, no single span holding every key, no key that is wording, and every key marked `negated` standing negated in a span (see `MULTIDOC.md`, "Answer keys are facts, not wording" and "Negated keys"). A `count` question has a count of 5 or more that equals its spans and the matches of its `item_pattern` in the file, and the count stands nowhere in the file as a number or an English number word. - **gate**: `whole` counts when the answer holds every key (a number key matches as a number, a negated key only with a negation before it); `count` counts when the answer's `count` field is that exact integer. Both need a citation that quotes the file verbatim. It prints `whole` and `count` separately, and GREEN only when every question is correct. The `one-span` arm must score 0 (RED), and the `fasit` arm is the known-positive (it writes a negated key as "not "). Frozen 27.09.2026: 16 questions (8 `whole` on the three documentation files of 150 lines or more in the cut, 8 `count` on five files of the cut) were written from the sources after the thresholds were committed. Seed 20260927 held out 2 per type (4 questions, outside the repository, sha256 only here), which leaves 12 for development (6 `whole`, 6 `count`). `check` is PASS on both parts. On the development part the gate is RED at 0 of 12 with the `one-span` arm and GREEN at 12 of 12 with the `fasit` arm. Without an answers file it is fail-closed at 0 of 12. ## Quote rule for K's fragments (V1 Step 4, brief SC3) `quote_rule.py` measures two rules that close the path P1 saw (the compiler wrote numbers as words to slip past a rule that asked for a quote only for a digit or a backticked name). It calls no model and prints counts only. ``` python3 -m gates.quote_rule measure ~/okf/benk/wiki/prove-p1/K-fragments ~/okf/benk/wiki/prove-p1/K-work/index.json ``` - **Rule E:** every fragment carries a quote. **Rule L:** a quote is required for a digit, a backticked name or a word from the narrow list (zero to twenty, hundred, thousand, million). - **Answer key**, made apart from rule L: a quoteless fragment whose text holds a word from the broad list of brief SC3 (the narrow list plus thirty to ninety, dozen, dozens, once, twice, first to tenth, half, double, pair, couple, several, single), as a whole word. The denominator is the number of hits. - **Choice:** E, unless L lets 0 key fragments through and E rejects at least 33 of 650 (5 %) more than L. Measured on 04.10.2026 on P1's 650 fragments (all 650 accepted in P1; 160 have no quote): | Count | Value | |---|---| | E rejects | 160 of 650 | | L rejects | 10 of 650 | | Answer-key hits (denominator) | 26 (10 hold a narrow-list word) | | Key hits per file | pack-1.json 19, pack-4.json 7 | | Key fragments E lets through | 0 of 26 | | Key fragments L lets through | 16 of 26 | | E rejects minus L rejects | 150 (threshold 33) | **Chose E because L lets 16 of the 26 key fragments through** (the rule needs 0); E rejects 150 more fragments than L, so the cost condition alone would also have allowed L, but the slip condition rules it out. The known-positive holds (26 hits, at least 1). The P1 README says the compiler wrote numbers as words "in one pack" without naming it; the key finds hits in packs 1 and 4, and the four compiler transcripts end in a tool call with no report text, so which pack the README meant is not measured. New reject rules, adopted in Step 6 only if each rejects at most 5 % of the 650 (32): | Rule | Rejects | Adopt | |---|---|---| | short quote (fewer than 3 words) | 34 of 650 (5.2 %) | no: over 5 %, reported to the PM | | id mismatch (an `existing` id whose entry matches the concept name neither by normalised form nor by Jaccard >= 0.6) | 0 of 650 | yes | The 0 id mismatches were checked against a known-positive: with every fragment given its neighbour's id, the check flags 454 of 650, which is every fragment whose neighbour has a different id (196 neighbours share one). `NEEDS_QUOTE`, `MERGE_JACCARD` and `norm` are copies of the trial's, kept equal by `TrialConstantsTest`, which reads the trial at run time. Chose copies over an import, because `mypy --strict okfwiki gates tests` does not see the trial package (it is found only when `evals` is listed), so an import would turn the v1 mypy line red once `gates/` is frozen.