okf-wiki/evals/README.md
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

120 lines
8.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# evals
## Direction probe (R)
`direction_probe.py` sets up and scores the direction probe: three arms answer the same questions from the same sources.
It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm.
- `filsok`: file search alone over the sources in the cut.
- `wiki`: the okf-wiki plus file search, following the skill's router (`index.md`, the pages, the raw source at `x-locator`).
- `astro`: a wiki built with Astro-Han's `karpathy-llm-wiki` skill (MIT) on the same sources, following its Query steps.
```
python3 evals/direction_probe.py fetch-astro --sources <source dir> --cut-manifest gates/wholedoc.manifest.json \
--project <astro project OUTSIDE the repository> --topic <topic> --collected <YYYY-MM-DD>
python3 evals/direction_probe.py setup --run <run dir> --arm <filsok|wiki|astro> --sources <source dir> \
--cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \
--wd-set gates-local/wholedoc/set-12.json [--wiki <okf-wiki bundle>] [--astro <astro project>]
python3 evals/direction_probe.py collect --run <run dir> --arm <arm>
python3 gates/multidoc.py gate ... --answers <run>/answers/<arm>.md.json --out <run>/results/<arm>.md.json
python3 gates/wholedoc.py gate ... --answers <run>/answers/<arm>.wd.json --out <run>/results/<arm>.wd.json
python3 evals/direction_probe.py harvest --run <run dir> --transcripts <subagent transcript folder>
python3 evals/direction_probe.py score --run <run dir> --arms filsok,wiki,astro --md-ids d01,d02,...
```
- **setup** copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per
question with the question, the search rules and the answer format, and never the key, the evidence or the count.
Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with
`cd <basis> &&` and uses a fixed list of read-only tools.
- **harvest** sums every usage field in each task's transcript, counting each message id once, and names every tool
call that breaks the task's rules. A breach is a fact to report, not a verdict: `echo` between two greps breaks the
rule without reading anything outside the basis.
- **score** filters the unchanged gates' per-question results to the probe questions. It reports per arm and group
(multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing
result file is a failure, not a zero.
Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository
(CLAUDE.md rule 3). Tests: `python3 -m unittest discover -s evals/tests`. Types: `mypy --strict evals`.
## v1-k step 9: semantic links measured on P1's pages (SC13)
The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to
their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for `link` + `orchestration-build`; the stop
check runs after every pass. Output stays outside the repository (`~/okf/benk/wiki/v1/lenkemaaling/`, rule 3).
**Result: STOP after pass 1 of 5.** Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029
of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun.
| Number | Value |
|---|---|
| Pages with an inbound link (link rule) | 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run) |
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
| Max out-degree | 10 |
| Links accepted / rejected | 304 / 0 (no reason hit) |
| "none" answers | 9 of 104 pages |
| Link agents | 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each |
| `link` | 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked) |
| `orchestration-build` at the result | 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn |
| Catalogue share of link input | 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted) |
Projection to collection 1 (~120 packs), orchestration at the measured `a + b × turn` restarted every session:
| Turns per pack | 20 turns per session | 40 | 80 |
|---|---|---|---|
| 4 (480 turns) | 50.0 M | 60.9 M | 82.5 M |
| 5 (600 turns) | 62.5 M | 76.1 M | 101.3 M |
Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M,
plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent ×
≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so `klink task` cannot run on it
without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M.
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent
with the linker's instructions verbatim (the session started in the main checkout, where `okf-wiki-linker` is not
defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's
start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a
fully stripped K) is not in it.
## v1-k step 9b: semantic links measured with the cost cuts, collection 1 re-projected
Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (`klink pass-before --pack 2 --to 4`
without `--catalogue`, three linkers spawned in one turn, one `klink pass-after`), cost line 7 M tokens for `link` +
`orchestration-build` counted on this step only: the stop check read a copy of the session's main transcript from the
step's start marker on, because the session had first carried another order.
**Result: STOP before the closing pass (pass 5).** After packs 2–4 the stop line said continue (measured 3,396,621;
flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters
(5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per
orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned.
| Number | Value |
|---|---|
| Pages with an inbound link (link rule) | 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run) |
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
| Max out-degree | 11 |
| Links accepted / rejected | 270 / 0 in packs 2–4 (574 / 0 in all) |
| "none" answers | 38 in packs 2–4 (47 in all) |
| Link agents | 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each |
| `link` | 658,909 (2,098 per sentence; 214,054–228,310 per agent) |
| `orchestration-build` at the result | 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn |
| Catalogue filter (step 9.2) on P1 | pools 146 / 174 / 212, nothing cut (largest 8,437 characters) |
| Price of the filter at collection 1's budget | 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited) |
Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session)
and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns ×
(47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M:
| k packs per interval | 20 packs per session | 40 packs per session |
|---|---|---|
| 5 | 68.3–99.2 M (× 1.10: 75.2–109.2) | 69.4–100.3 M (76.4–110.4) |
| 10 | 61.4–92.3 M (67.6–101.6) | 61.0–91.9 M (67.1–101.0) |
| 20 | 58.4–89.3 M (64.2–98.2) | 57.5–88.4 M (63.3–97.2) |
The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file)
already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model
(52.8 M, 4 points, not validated).
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with
the linker's instructions verbatim (start-up 47,705; the `okf-wiki-linker` type is not defined in the main checkout
where the session started), and at least one used a script to check its own anchors; a was measured in a session that
was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.