# evals ## Direction probe (R) `direction_probe.py` sets up and scores the direction probe: three arms answer the same questions from the same sources. It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm. - `filsok`: file search alone over the sources in the cut. - `wiki`: the okf-wiki plus file search, following the skill's router (`index.md`, the pages, the raw source at `x-locator`). - `astro`: a wiki built with Astro-Han's `karpathy-llm-wiki` skill (MIT) on the same sources, following its Query steps. ``` python3 evals/direction_probe.py fetch-astro --sources --cut-manifest gates/wholedoc.manifest.json \ --project --topic --collected python3 evals/direction_probe.py setup --run --arm --sources \ --cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \ --wd-set gates-local/wholedoc/set-12.json [--wiki ] [--astro ] python3 evals/direction_probe.py collect --run --arm python3 gates/multidoc.py gate ... --answers /answers/.md.json --out /results/.md.json python3 gates/wholedoc.py gate ... --answers /answers/.wd.json --out /results/.wd.json python3 evals/direction_probe.py harvest --run --transcripts python3 evals/direction_probe.py score --run --arms filsok,wiki,astro --md-ids d01,d02,... ``` - **setup** copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per question with the question, the search rules and the answer format, and never the key, the evidence or the count. Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with `cd &&` and uses a fixed list of read-only tools. - **harvest** sums every usage field in each task's transcript, counting each message id once, and names every tool call that breaks the task's rules. A breach is a fact to report, not a verdict: `echo` between two greps breaks the rule without reading anything outside the basis. - **score** filters the unchanged gates' per-question results to the probe questions. It reports per arm and group (multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing result file is a failure, not a zero. Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository (CLAUDE.md rule 3). Tests: `python3 -m unittest discover -s evals/tests`. Types: `mypy --strict evals`. ## v1-k step 9: semantic links measured on P1's pages (SC13) The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for `link` + `orchestration-build`; the stop check runs after every pass. Output stays outside the repository (`~/okf/benk/wiki/v1/lenkemaaling/`, rule 3). **Result: STOP after pass 1 of 5.** Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029 of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun. | Number | Value | |---|---| | Pages with an inbound link (link rule) | 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run) | | Misplaced / broken / not to a concept page | 0 / 0 / 0 | | Max out-degree | 10 | | Links accepted / rejected | 304 / 0 (no reason hit) | | "none" answers | 9 of 104 pages | | Link agents | 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each | | `link` | 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked) | | `orchestration-build` at the result | 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn | | Catalogue share of link input | 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted) | Projection to collection 1 (~120 packs), orchestration at the measured `a + b × turn` restarted every session: | Turns per pack | 20 turns per session | 40 | 80 | |---|---|---|---| | 4 (480 turns) | 50.0 M | 60.9 M | 82.5 M | | 5 (600 turns) | 62.5 M | 76.1 M | 101.3 M | Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M, plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent × ≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so `klink task` cannot run on it without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M. Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent with the linker's instructions verbatim (the session started in the main checkout, where `okf-wiki-linker` is not defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a fully stripped K) is not in it. ## v1-k step 9b: semantic links measured with the cost cuts, collection 1 re-projected Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (`klink pass-before --pack 2 --to 4` without `--catalogue`, three linkers spawned in one turn, one `klink pass-after`), cost line 7 M tokens for `link` + `orchestration-build` counted on this step only: the stop check read a copy of the session's main transcript from the step's start marker on, because the session had first carried another order. **Result: STOP before the closing pass (pass 5).** After packs 2–4 the stop line said continue (measured 3,396,621; flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters (5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned. | Number | Value | |---|---| | Pages with an inbound link (link rule) | 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run) | | Misplaced / broken / not to a concept page | 0 / 0 / 0 | | Max out-degree | 11 | | Links accepted / rejected | 270 / 0 in packs 2–4 (574 / 0 in all) | | "none" answers | 38 in packs 2–4 (47 in all) | | Link agents | 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each | | `link` | 658,909 (2,098 per sentence; 214,054–228,310 per agent) | | `orchestration-build` at the result | 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn | | Catalogue filter (step 9.2) on P1 | pools 146 / 174 / 212, nothing cut (largest 8,437 characters) | | Price of the filter at collection 1's budget | 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited) | Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session) and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns × (47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M: | k packs per interval | 20 packs per session | 40 packs per session | |---|---|---| | 5 | 68.3–99.2 M (× 1.10: 75.2–109.2) | 69.4–100.3 M (76.4–110.4) | | 10 | 61.4–92.3 M (67.6–101.6) | 61.0–91.9 M (67.1–101.0) | | 20 | 58.4–89.3 M (64.2–98.2) | 57.5–88.4 M (63.3–97.2) | The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file) already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model (52.8 M, 4 points, not validated). Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with the linker's instructions verbatim (start-up 47,705; the `okf-wiki-linker` type is not defined in the main checkout where the session started), and at least one used a script to check its own anchors; a was measured in a session that was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.