okf-wiki/gates
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00
..
calibration-folder okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
repair okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
stubs okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
tests okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
answers.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
answerset-del1-hard.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
answerset.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
ANSWERSET.md okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
answerset.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
calibration-folder.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
calibration.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
CALIBRATION.md okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
calibration.passed.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
calibration.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
filetype_gate.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
filetypes.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
keyform.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
multidoc.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
MULTIDOC.md okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
multidoc.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
mutant_gate.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
mutants.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
polarity.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
quote_rule.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
README.md okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
repair.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
REPAIR.md okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
repair_gate.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
wholedoc.manifest.json okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
wholedoc.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00

gates

Mutant gate (E1)

mutant_gate.py runs an auditor command on every case in mutants.manifest.json: 37 mutants, each carrying a defect of a named class, plus one known-positive bundle that must come back clean. It calls no model.

E1b added three link classes: broken-link, orphan-page and index-coverage. They are okf-wiki's own checks, not SPEC § 11 conformance: SPEC § 6.1 says consumers MUST tolerate broken links. Their exact definitions are in the manifest under provenance.link_rules, with criterion 6 as it stands (see below).

python3 gates/mutant_gate.py --manifest gates/mutants.manifest.json --cases gates-local/mutants \
    --sources gates-local/mutants-kilder -- <auditor> {bundle} {sources}
  • {bundle} becomes a fresh copy of the case bundle, and {sources} becomes the raw-source root.
  • The auditor prints {"findings": [{"check": "<class>", ...}]} and exits 0 when clean or 1 when it raises an alarm. Anything else counts as an error and catches nothing: a missing auditor, a crash, a timeout, output that is not JSON, or an exit status that disagrees with the findings.
  • The gate is GREEN only when every mutant is caught with its own class, the known-positive is clean and nothing errored. Rejecting everything catches every mutant, but it raises a false alarm and matches no class.
  • Before anything runs, every case file and every raw source listed in the manifest is checked against its sha256. A mismatch stops the gate with exit 2.

The sources root is gates-local/mutants-kilder/ (wave 1 order 4): byte-identical copies of the three documentation files the pages cite and one release note, under kilde/, with source-manifest.json in the form of contract K1 (docs/bolge1/0-felles.local.md § 4, local). The manifest lists all five files with their sha256. Until 08.10.2026 the root was the copy of 21.09 (~/okf/benk/wiki); that root has no K1 file, so the gate now stops on it with exit 2.

The auditor is okfwiki/audit.py (L1a). With it the gate was GREEN at 23 of 23 class matched and 0 of 1 false alarms before V1 added the link-rule mutants, and GREEN at 29 of 29 and 0 of 1 on 08.10.2026 with the manifest of that day and --sources ~/okf/benk/wiki, before the main-page rule below. The command today (GREEN since wave 2 order 7, see below):

python3 gates/mutant_gate.py --manifest gates/mutants.manifest.json --cases gates-local/mutants \
    --sources gates-local/mutants-kilder -- python3 -m okfwiki.audit '{bundle}' '{sources}'

In zsh, quote the placeholders as above when the output is redirected: {sources} > file opens a named file descriptor, the auditor gets only {bundle}, and every case errors.

Criterion 6 under the main-page rule (wave 1 order 4, RED)

Criterion 6 in PLAN.md § Ferdig-kriteriet as the operator set it on 06.10.2026 (choice 2): Google's rule as written, applied to documents. Every documentation page has its main page, and every other concept page has a link from at least one main page. Links are made by the model from meaning; the auditor checks structure only, without a model. The exact definitions are in the manifest under provenance.link_rules, and the contract is K2:

  • A concept page lists the sources it is the main page for in its frontmatter, as a YAML list under x-main-for. A valid value is / + a path with main_topic: true in the K1 file at the sources root, and stands in the page's own sources[].resource. Without a valid K1 file the rule cannot run (exit 2, never 0).
  • main-page-missing: a main-topic source with no main page. main-page-duplicate: two main pages for one source. main-page-invalid: not a list, a value not among the page's own sources, or a value that is not a main-topic source.
  • no-main-page-link: a page that is not a main page has no link from a main page's body, in a sentence or in a list of related concepts. index.md, log.md and the page itself do not count. A main page needs no inbound link.
  • Unchanged: (a) broken-link, (b) link-misplaced (a link in code, a heading or a footnote definition), orphan-page, index-coverage. link-rule (an inbound link from any other concept page, the rule of 04.10.2026) is retired in Step 7; no mutant carries it any more.

The known-positive (normalisation N7): dontask-modus.md is the main page of all three documentation sources, and has no inbound link from another concept page (only from index.md). Two non-main pages: bypasspermissions-modus.md (linked from a sentence on the main page) and acceptedits-modus.md (linked from the main page's related-concepts list, and from a sentence on the bypass page; it cites the release note). The main page has exactly one link to each, so a mutant moves or removes "the one link". Every case is rebuilt from this known-positive, so every sha256 changed.

Mutant Class Defect Expected extra findings
m21 orphan-page acceptEdits no longer linked from any page or index.md (its log.md entry stays) index-coverage, no-main-page-link
m24 no-main-page-link bypass linked only from index.md none
m25 link-misplaced the main page's one link to bypass inside inline code no-main-page-link
m26 link-misplaced the main page's one link to bypass moved into a heading no-main-page-link
m27 link-misplaced the main page's one link to bypass moved into a footnote definition no-main-page-link
m28 no-main-page-link the only concept-body link to bypass is a self-link (index.md and log.md still link it) none
m29 no-main-page-link bypass linked only from log.md index-coverage, orphan-page
m30 no-main-page-link acceptEdits linked only from the other non-main page (and index.md, log.md) none
m31 main-page-missing the main page loses its mark for one documentation source none
m32 main-page-duplicate bypass marks a source the main page marks (and bypass cites) none
m33 main-page-invalid bypass's x-main-for is a string, not a list none
m34 main-page-invalid bypass marks a main-topic source it does not cite none
m35 main-page-invalid acceptEdits marks the release note it cites (in K1, not a main topic) none
m36 main-page-duplicate the main page adds an invalid value, and bypass marks a source the main page marks main-page-invalid
m37 no-main-page-link bypass marks only a source it does not cite, and the main page's one link to it is removed main-page-invalid

m36 and m37 test K2 point 2's last sentence (a page is a main page when at least one value is valid; invalid values do not take that role away). Since the gate counts a mutant matched when its own class is among the findings, each carries a class that only a right reading of the role gives, so each is two edits by design. Measured with two wrong readings of the rule (scratch only): one that makes every page with the key a main page misses m37's class, and one that takes the role from a page with an invalid value misses m36's; on the set without m36 and m37 both missed nothing. Under K2 only links from main pages count, so a self-link on a non-main page can never count: m28 is kept because the order names the case, but it tells K2 auditors apart no better than m24.

The extras are the classes the builder measures beside the case's own class, as the auditor of Step 7 should report them; they are documentation (expected_extra per case), and the gate counts a mutant matched when its own class is among the findings. Two older cases also gain an extra under K2: m11 (main-page-missing, no-main-page-link: the main page's frontmatter no longer parses) and m12 (no-main-page-link: its extra page is linked from index.md alone). The manifest maps every K2 case to its mutant under provenance.k2.coverage.

Recorded RED on 08.10.2026 with today's auditor (okfwiki/audit.py unchanged), with the command above: class matched 26 of 37. The old classes are matched 26 of 26 (m01-m23 and m25-m27). The new classes are matched 0 of 11: m24 and m28-m37 are caught only with the retired link-rule (m29 also with index-coverage and orphan-page). False alarms 1 of 1, by design: today's auditor reports link-rule on the main page of the known-positive, which has no inbound concept link because K2 point 3 does not ask for one; Step 7 retires link-rule. Errors 0 of 38, exit 1. Step 7 must turn this into 37 of 37 with 0 false alarms.

Wave 2 order 7 did (merged into v1 09.10.2026, measured in the main copy after the merge): caught 37 of 37 · class matched 37 of 37 · false alarms 0 of 1 · errors 0 of 38, GREEN, exit 0. Since K2 point 15 (a link with / does not count), m09 also gives no-main-page-link on acceptedits-modus.md; the gate counts only each case's own class, and expected_extra for m09 in the frozen manifest still says [].

History: V1 session 2 (05.10.2026) recorded RED at 23 of 29 for the rule of 04.10.2026 (an inbound link from any other concept page, class link-rule on m24, m28 and m29), and Step 6 of the v1 plan turned it GREEN at 29 of 29.

The cases and their sources derive from Anthropic documentation, so they live in gates-local/ (gitignored, CLAUDE.md rule 3). Only the runner, its tests, the reject-all stub and the manifest (names, classes and sha256) are tracked.

File-type gate (E4)

filetype_gate.py asks a reader where each frozen quote occurs, one run per source file. It measures the L1 number "file types read with a locator, of 7". It calls no model.

python3 gates/filetype_gate.py --manifest gates/filetypes.manifest.json --pairs gates-local/filetypes/pairs.json \
    --sources <folder holding blandet/ and wiki/> --cases gates-local/filetypes/files -- <reader> {file} {queries}
  • {file} becomes a fresh copy of one source file, and {queries} becomes a JSON file of {"id", "quote"} objects. The reader never sees the expected locator. It prints {"answers": [{"id", "found": [{"locator", "hidden"}]}]}: every place where the quote occurs, after runs of whitespace are collapsed to one space.
  • A pair is located when the answer names exactly one place, that place is the frozen locator, and hidden is right. Hidden Office content (a hidden sheet, rows or columns, a hidden slide, hidden text, or a tracked deletion) must be marked hidden: true, not flattened into ordinary text. Each quote occurs once in its file, so a second place is wrong.
  • Locators: pdf {"page": N}, docx {"paragraph": N}, xlsx {"sheet", "cell"}, pptx {"slide": N, "notes": bool}, and md, txt and html {"line": N}. The manifest defines each one and says why it was chosen.
  • A file type counts as read when every one of its pairs is located. The gate prints this as file types with every pair located: X of 7. The hidden classes are reported separately. The gate is GREEN only when every pair is located and no reader run errs.
  • Before anything runs, the pairs file and every source it names are checked against the manifest's sha256.

Working set: 112 pairs (16 per type) plus 16 in the hidden cases, from 64 files. A held-out set of 28 pairs (4 per type) lies outside the repository, with only its sha256 here (same form as E2). How the fasit was made is in the manifest under fasit_method. The stubs are stubs/first_locator.py (always the first page, paragraph, cell, slide or line) and stubs/fasit_reader.py (the known-positive, which answers from the pairs file).

The engine's reader is python3 -m okfwiki.readers locate {file} {queries}. On 07.11.2026 it located 128 of 128 in the working set (16 of 16 per type, every hidden class, 4 of 4 controls, 0 reader errors); first_locator.py located 7.

Tests: python3 -m unittest discover -s gates/tests. Types: mypy --strict gates/.

Answer set for collection 1 (wave 2 order 6, K5)

answerset.py and answers.py hold the answer set for collection 1 and the harness for the answer measurements (contract K5); see ANSWERSET.md. It replaces E2 for the verdict.

Whole-document and counting group (E2b)

wholedoc.py holds a small group of questions for the class the E2 set lacks: an answer that needs one long file read as a whole, or a count over many places in one file. It is reported on its own and calls no model. It is a separate gate, not a group inside multidoc.py, because that checker requires four or more files per question and E2 is frozen.

python3 gates/wholedoc.py cut   --e2-set gates-local/multidoc/set-60.json --sources <source dir> --manifest gates/wholedoc.manifest.json
python3 gates/wholedoc.py check --set <set> --sources <source dir> --manifest gates/wholedoc.manifest.json --e2-set gates-local/multidoc/set-60.json
python3 gates/wholedoc.py split --set <all.json> ... --seed <n> --held-out-per-type 2 --held-out <file OUTSIDE the repository>
python3 gates/wholedoc.py gate  --set gates-local/wholedoc/set-12.json --manifest gates/wholedoc.manifest.json \
    --sources <source dir> --arm <name> --answers <answers.json> [--out <results.json>]
python3 gates/stubs/wholedoc_arm.py <one-span|fasit> <set> <answers.json>
  • cut takes the sources from the E2 dev set: sorted by id, repeatedly the question needing the fewest new files, while the union stays at 20 files or fewer. It wrote the manifest with the thresholds before any question existed.
  • check measures every property in the source files. Every span stands verbatim and exactly once in its file. A whole question sits on a file of 150 lines or more, with at least three keys, spans in at least three parts (a part begins where two span starts lie 15 % of the file or more apart), the first span in the first third and the last one in the last third, no single span holding every key, no key that is wording, and every key marked negated standing negated in a span (see MULTIDOC.md, "Answer keys are facts, not wording" and "Negated keys"). A count question has a count of 5 or more that equals its spans and the matches of its item_pattern in the file, and the count stands nowhere in the file as a number or an English number word.
  • gate: whole counts when the answer holds every key (a number key matches as a number, a negated key only with a negation before it); count counts when the answer's count field is that exact integer. Both need a citation that quotes the file verbatim. It prints whole and count separately, and GREEN only when every question is correct. The one-span arm must score 0 (RED), and the fasit arm is the known-positive (it writes a negated key as "not ").

Frozen 27.09.2026: 16 questions (8 whole on the three documentation files of 150 lines or more in the cut, 8 count on five files of the cut) were written from the sources after the thresholds were committed. Seed 20260927 held out 2 per type (4 questions, outside the repository, sha256 only here), which leaves 12 for development (6 whole, 6 count). check is PASS on both parts. On the development part the gate is RED at 0 of 12 with the one-span arm and GREEN at 12 of 12 with the fasit arm. Without an answers file it is fail-closed at 0 of 12.

Quote rule for K's fragments (V1 Step 4, brief SC3)

quote_rule.py measures two rules that close the path P1 saw (the compiler wrote numbers as words to slip past a rule that asked for a quote only for a digit or a backticked name). It calls no model and prints counts only.

python3 -m gates.quote_rule measure ~/okf/benk/wiki/prove-p1/K-fragments ~/okf/benk/wiki/prove-p1/K-work/index.json
  • Rule E: every fragment carries a quote. Rule L: a quote is required for a digit, a backticked name or a word from the narrow list (zero to twenty, hundred, thousand, million).
  • Answer key, made apart from rule L: a quoteless fragment whose text holds a word from the broad list of brief SC3 (the narrow list plus thirty to ninety, dozen, dozens, once, twice, first to tenth, half, double, pair, couple, several, single), as a whole word. The denominator is the number of hits.
  • Choice: E, unless L lets 0 key fragments through and E rejects at least 33 of 650 (5 %) more than L.

Measured on 04.10.2026 on P1's 650 fragments (all 650 accepted in P1; 160 have no quote):

Count Value
E rejects 160 of 650
L rejects 10 of 650
Answer-key hits (denominator) 26 (10 hold a narrow-list word)
Key hits per file pack-1.json 19, pack-4.json 7
Key fragments E lets through 0 of 26
Key fragments L lets through 16 of 26
E rejects minus L rejects 150 (threshold 33)

Chose E because L lets 16 of the 26 key fragments through (the rule needs 0); E rejects 150 more fragments than L, so the cost condition alone would also have allowed L, but the slip condition rules it out. The known-positive holds (26 hits, at least 1). The P1 README says the compiler wrote numbers as words "in one pack" without naming it; the key finds hits in packs 1 and 4, and the four compiler transcripts end in a tool call with no report text, so which pack the README meant is not measured.

New reject rules, adopted in Step 6 only if each rejects at most 5 % of the 650 (32):

Rule Rejects Adopt
short quote (fewer than 3 words) 34 of 650 (5.2 %) no: over 5 %, reported to the PM
id mismatch (an existing id whose entry matches the concept name neither by normalised form nor by Jaccard >= 0.6) 0 of 650 yes

The 0 id mismatches were checked against a known-positive: with every fragment given its neighbour's id, the check flags 454 of 650, which is every fragment whose neighbour has a different id (196 neighbours share one).

NEEDS_QUOTE, MERGE_JACCARD and norm are copies of the trial's, kept equal by TrialConstantsTest, which reads the trial at run time. Chose copies over an import, because mypy --strict okfwiki gates tests does not see the trial package (it is found only when evals is listed), so an import would turn the v1 mypy line red once gates/ is frozen.