okf-wiki/evals
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00
..
p1 okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
tests okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
direction_probe.py okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00
README.md okf-wiki 1.1.1 2026-11-14 21:03:20 +01:00

evals

Direction probe (R)

direction_probe.py sets up and scores the direction probe: three arms answer the same questions from the same sources. It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm.

  • filsok: file search alone over the sources in the cut.
  • wiki: the okf-wiki plus file search, following the skill's router (index.md, the pages, the raw source at x-locator).
  • astro: a wiki built with Astro-Han's karpathy-llm-wiki skill (MIT) on the same sources, following its Query steps.
python3 evals/direction_probe.py fetch-astro --sources <source dir> --cut-manifest gates/wholedoc.manifest.json \
    --project <astro project OUTSIDE the repository> --topic <topic> --collected <YYYY-MM-DD>
python3 evals/direction_probe.py setup --run <run dir> --arm <filsok|wiki|astro> --sources <source dir> \
    --cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \
    --wd-set gates-local/wholedoc/set-12.json [--wiki <okf-wiki bundle>] [--astro <astro project>]
python3 evals/direction_probe.py collect --run <run dir> --arm <arm>
python3 gates/multidoc.py gate ... --answers <run>/answers/<arm>.md.json --out <run>/results/<arm>.md.json
python3 gates/wholedoc.py gate ... --answers <run>/answers/<arm>.wd.json --out <run>/results/<arm>.wd.json
python3 evals/direction_probe.py harvest --run <run dir> --transcripts <subagent transcript folder>
python3 evals/direction_probe.py score --run <run dir> --arms filsok,wiki,astro --md-ids d01,d02,...
  • setup copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per question with the question, the search rules and the answer format, and never the key, the evidence or the count. Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with cd <basis> && and uses a fixed list of read-only tools.
  • harvest sums every usage field in each task's transcript, counting each message id once, and names every tool call that breaks the task's rules. A breach is a fact to report, not a verdict: echo between two greps breaks the rule without reading anything outside the basis.
  • score filters the unchanged gates' per-question results to the probe questions. It reports per arm and group (multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing result file is a failure, not a zero.

Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository (CLAUDE.md rule 3). Tests: python3 -m unittest discover -s evals/tests. Types: mypy --strict evals.

The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for link + orchestration-build; the stop check runs after every pass. Output stays outside the repository (~/okf/benk/wiki/v1/lenkemaaling/, rule 3).

Result: STOP after pass 1 of 5. Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029 of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun.

Number Value
Pages with an inbound link (link rule) 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run)
Misplaced / broken / not to a concept page 0 / 0 / 0
Max out-degree 10
Links accepted / rejected 304 / 0 (no reason hit)
"none" answers 9 of 104 pages
Link agents 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each
link 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked)
orchestration-build at the result 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn
Catalogue share of link input 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted)

Projection to collection 1 (~120 packs), orchestration at the measured a + b × turn restarted every session:

Turns per pack 20 turns per session 40 80
4 (480 turns) 50.0 M 60.9 M 82.5 M
5 (600 turns) 62.5 M 76.1 M 101.3 M

Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M, plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent × ≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so klink task cannot run on it without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M.

Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent with the linker's instructions verbatim (the session started in the main checkout, where okf-wiki-linker is not defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a fully stripped K) is not in it.

Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (klink pass-before --pack 2 --to 4 without --catalogue, three linkers spawned in one turn, one klink pass-after), cost line 7 M tokens for link + orchestration-build counted on this step only: the stop check read a copy of the session's main transcript from the step's start marker on, because the session had first carried another order.

Result: STOP before the closing pass (pass 5). After packs 2–4 the stop line said continue (measured 3,396,621; flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters (5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned.

Number Value
Pages with an inbound link (link rule) 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run)
Misplaced / broken / not to a concept page 0 / 0 / 0
Max out-degree 11
Links accepted / rejected 270 / 0 in packs 2–4 (574 / 0 in all)
"none" answers 38 in packs 2–4 (47 in all)
Link agents 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each
link 658,909 (2,098 per sentence; 214,054–228,310 per agent)
orchestration-build at the result 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn
Catalogue filter (step 9.2) on P1 pools 146 / 174 / 212, nothing cut (largest 8,437 characters)
Price of the filter at collection 1's budget 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited)

Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session) and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns × (47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M:

k packs per interval 20 packs per session 40 packs per session
5 68.3–99.2 M (× 1.10: 75.2–109.2) 69.4–100.3 M (76.4–110.4)
10 61.4–92.3 M (67.6–101.6) 61.0–91.9 M (67.1–101.0)
20 58.4–89.3 M (64.2–98.2) 57.5–88.4 M (63.3–97.2)

The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file) already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model (52.8 M, 4 points, not validated).

Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with the linker's instructions verbatim (start-up 47,705; the okf-wiki-linker type is not defined in the main checkout where the session started), and at least one used a script to check its own anchors; a was measured in a session that was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.