First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
120 lines
8.4 KiB
Markdown
120 lines
8.4 KiB
Markdown
# evals
|
||
|
||
## Direction probe (R)
|
||
|
||
`direction_probe.py` sets up and scores the direction probe: three arms answer the same questions from the same sources.
|
||
It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm.
|
||
|
||
- `filsok`: file search alone over the sources in the cut.
|
||
- `wiki`: the okf-wiki plus file search, following the skill's router (`index.md`, the pages, the raw source at `x-locator`).
|
||
- `astro`: a wiki built with Astro-Han's `karpathy-llm-wiki` skill (MIT) on the same sources, following its Query steps.
|
||
|
||
```
|
||
python3 evals/direction_probe.py fetch-astro --sources <source dir> --cut-manifest gates/wholedoc.manifest.json \
|
||
--project <astro project OUTSIDE the repository> --topic <topic> --collected <YYYY-MM-DD>
|
||
python3 evals/direction_probe.py setup --run <run dir> --arm <filsok|wiki|astro> --sources <source dir> \
|
||
--cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \
|
||
--wd-set gates-local/wholedoc/set-12.json [--wiki <okf-wiki bundle>] [--astro <astro project>]
|
||
python3 evals/direction_probe.py collect --run <run dir> --arm <arm>
|
||
python3 gates/multidoc.py gate ... --answers <run>/answers/<arm>.md.json --out <run>/results/<arm>.md.json
|
||
python3 gates/wholedoc.py gate ... --answers <run>/answers/<arm>.wd.json --out <run>/results/<arm>.wd.json
|
||
python3 evals/direction_probe.py harvest --run <run dir> --transcripts <subagent transcript folder>
|
||
python3 evals/direction_probe.py score --run <run dir> --arms filsok,wiki,astro --md-ids d01,d02,...
|
||
```
|
||
|
||
- **setup** copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per
|
||
question with the question, the search rules and the answer format, and never the key, the evidence or the count.
|
||
Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with
|
||
`cd <basis> &&` and uses a fixed list of read-only tools.
|
||
- **harvest** sums every usage field in each task's transcript, counting each message id once, and names every tool
|
||
call that breaks the task's rules. A breach is a fact to report, not a verdict: `echo` between two greps breaks the
|
||
rule without reading anything outside the basis.
|
||
- **score** filters the unchanged gates' per-question results to the probe questions. It reports per arm and group
|
||
(multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing
|
||
result file is a failure, not a zero.
|
||
|
||
Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository
|
||
(CLAUDE.md rule 3). Tests: `python3 -m unittest discover -s evals/tests`. Types: `mypy --strict evals`.
|
||
|
||
## v1-k step 9: semantic links measured on P1's pages (SC13)
|
||
|
||
The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to
|
||
their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for `link` + `orchestration-build`; the stop
|
||
check runs after every pass. Output stays outside the repository (`~/okf/benk/wiki/v1/lenkemaaling/`, rule 3).
|
||
|
||
**Result: STOP after pass 1 of 5.** Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029
|
||
of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun.
|
||
|
||
| Number | Value |
|
||
|---|---|
|
||
| Pages with an inbound link (link rule) | 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run) |
|
||
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
|
||
| Max out-degree | 10 |
|
||
| Links accepted / rejected | 304 / 0 (no reason hit) |
|
||
| "none" answers | 9 of 104 pages |
|
||
| Link agents | 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each |
|
||
| `link` | 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked) |
|
||
| `orchestration-build` at the result | 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn |
|
||
| Catalogue share of link input | 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted) |
|
||
|
||
Projection to collection 1 (~120 packs), orchestration at the measured `a + b × turn` restarted every session:
|
||
|
||
| Turns per pack | 20 turns per session | 40 | 80 |
|
||
|---|---|---|---|
|
||
| 4 (480 turns) | 50.0 M | 60.9 M | 82.5 M |
|
||
| 5 (600 turns) | 62.5 M | 76.1 M | 101.3 M |
|
||
|
||
Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M,
|
||
plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent ×
|
||
≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so `klink task` cannot run on it
|
||
without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M.
|
||
|
||
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent
|
||
with the linker's instructions verbatim (the session started in the main checkout, where `okf-wiki-linker` is not
|
||
defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's
|
||
start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a
|
||
fully stripped K) is not in it.
|
||
|
||
## v1-k step 9b: semantic links measured with the cost cuts, collection 1 re-projected
|
||
|
||
Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (`klink pass-before --pack 2 --to 4`
|
||
without `--catalogue`, three linkers spawned in one turn, one `klink pass-after`), cost line 7 M tokens for `link` +
|
||
`orchestration-build` counted on this step only: the stop check read a copy of the session's main transcript from the
|
||
step's start marker on, because the session had first carried another order.
|
||
|
||
**Result: STOP before the closing pass (pass 5).** After packs 2–4 the stop line said continue (measured 3,396,621;
|
||
flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters
|
||
(5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per
|
||
orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned.
|
||
|
||
| Number | Value |
|
||
|---|---|
|
||
| Pages with an inbound link (link rule) | 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run) |
|
||
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
|
||
| Max out-degree | 11 |
|
||
| Links accepted / rejected | 270 / 0 in packs 2–4 (574 / 0 in all) |
|
||
| "none" answers | 38 in packs 2–4 (47 in all) |
|
||
| Link agents | 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each |
|
||
| `link` | 658,909 (2,098 per sentence; 214,054–228,310 per agent) |
|
||
| `orchestration-build` at the result | 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn |
|
||
| Catalogue filter (step 9.2) on P1 | pools 146 / 174 / 212, nothing cut (largest 8,437 characters) |
|
||
| Price of the filter at collection 1's budget | 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited) |
|
||
|
||
Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session)
|
||
and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns ×
|
||
(47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M:
|
||
|
||
| k packs per interval | 20 packs per session | 40 packs per session |
|
||
|---|---|---|
|
||
| 5 | 68.3–99.2 M (× 1.10: 75.2–109.2) | 69.4–100.3 M (76.4–110.4) |
|
||
| 10 | 61.4–92.3 M (67.6–101.6) | 61.0–91.9 M (67.1–101.0) |
|
||
| 20 | 58.4–89.3 M (64.2–98.2) | 57.5–88.4 M (63.3–97.2) |
|
||
|
||
The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file)
|
||
already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model
|
||
(52.8 M, 4 points, not validated).
|
||
|
||
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with
|
||
the linker's instructions verbatim (start-up 47,705; the `okf-wiki-linker` type is not defined in the main checkout
|
||
where the session started), and at least one used a script to check its own anchors; a was measured in a session that
|
||
was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.
|