okf-wiki 1.1.1
First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
commit
0fd86a23a3
227 changed files with 96077 additions and 0 deletions
120
evals/README.md
Normal file
120
evals/README.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
# evals
|
||||
|
||||
## Direction probe (R)
|
||||
|
||||
`direction_probe.py` sets up and scores the direction probe: three arms answer the same questions from the same sources.
|
||||
It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm.
|
||||
|
||||
- `filsok`: file search alone over the sources in the cut.
|
||||
- `wiki`: the okf-wiki plus file search, following the skill's router (`index.md`, the pages, the raw source at `x-locator`).
|
||||
- `astro`: a wiki built with Astro-Han's `karpathy-llm-wiki` skill (MIT) on the same sources, following its Query steps.
|
||||
|
||||
```
|
||||
python3 evals/direction_probe.py fetch-astro --sources <source dir> --cut-manifest gates/wholedoc.manifest.json \
|
||||
--project <astro project OUTSIDE the repository> --topic <topic> --collected <YYYY-MM-DD>
|
||||
python3 evals/direction_probe.py setup --run <run dir> --arm <filsok|wiki|astro> --sources <source dir> \
|
||||
--cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \
|
||||
--wd-set gates-local/wholedoc/set-12.json [--wiki <okf-wiki bundle>] [--astro <astro project>]
|
||||
python3 evals/direction_probe.py collect --run <run dir> --arm <arm>
|
||||
python3 gates/multidoc.py gate ... --answers <run>/answers/<arm>.md.json --out <run>/results/<arm>.md.json
|
||||
python3 gates/wholedoc.py gate ... --answers <run>/answers/<arm>.wd.json --out <run>/results/<arm>.wd.json
|
||||
python3 evals/direction_probe.py harvest --run <run dir> --transcripts <subagent transcript folder>
|
||||
python3 evals/direction_probe.py score --run <run dir> --arms filsok,wiki,astro --md-ids d01,d02,...
|
||||
```
|
||||
|
||||
- **setup** copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per
|
||||
question with the question, the search rules and the answer format, and never the key, the evidence or the count.
|
||||
Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with
|
||||
`cd <basis> &&` and uses a fixed list of read-only tools.
|
||||
- **harvest** sums every usage field in each task's transcript, counting each message id once, and names every tool
|
||||
call that breaks the task's rules. A breach is a fact to report, not a verdict: `echo` between two greps breaks the
|
||||
rule without reading anything outside the basis.
|
||||
- **score** filters the unchanged gates' per-question results to the probe questions. It reports per arm and group
|
||||
(multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing
|
||||
result file is a failure, not a zero.
|
||||
|
||||
Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository
|
||||
(CLAUDE.md rule 3). Tests: `python3 -m unittest discover -s evals/tests`. Types: `mypy --strict evals`.
|
||||
|
||||
## v1-k step 9: semantic links measured on P1's pages (SC13)
|
||||
|
||||
The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to
|
||||
their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for `link` + `orchestration-build`; the stop
|
||||
check runs after every pass. Output stays outside the repository (`~/okf/benk/wiki/v1/lenkemaaling/`, rule 3).
|
||||
|
||||
**Result: STOP after pass 1 of 5.** Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029
|
||||
of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun.
|
||||
|
||||
| Number | Value |
|
||||
|---|---|
|
||||
| Pages with an inbound link (link rule) | 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run) |
|
||||
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
|
||||
| Max out-degree | 10 |
|
||||
| Links accepted / rejected | 304 / 0 (no reason hit) |
|
||||
| "none" answers | 9 of 104 pages |
|
||||
| Link agents | 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each |
|
||||
| `link` | 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked) |
|
||||
| `orchestration-build` at the result | 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn |
|
||||
| Catalogue share of link input | 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted) |
|
||||
|
||||
Projection to collection 1 (~120 packs), orchestration at the measured `a + b × turn` restarted every session:
|
||||
|
||||
| Turns per pack | 20 turns per session | 40 | 80 |
|
||||
|---|---|---|---|
|
||||
| 4 (480 turns) | 50.0 M | 60.9 M | 82.5 M |
|
||||
| 5 (600 turns) | 62.5 M | 76.1 M | 101.3 M |
|
||||
|
||||
Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M,
|
||||
plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent ×
|
||||
≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so `klink task` cannot run on it
|
||||
without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M.
|
||||
|
||||
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent
|
||||
with the linker's instructions verbatim (the session started in the main checkout, where `okf-wiki-linker` is not
|
||||
defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's
|
||||
start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a
|
||||
fully stripped K) is not in it.
|
||||
|
||||
## v1-k step 9b: semantic links measured with the cost cuts, collection 1 re-projected
|
||||
|
||||
Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (`klink pass-before --pack 2 --to 4`
|
||||
without `--catalogue`, three linkers spawned in one turn, one `klink pass-after`), cost line 7 M tokens for `link` +
|
||||
`orchestration-build` counted on this step only: the stop check read a copy of the session's main transcript from the
|
||||
step's start marker on, because the session had first carried another order.
|
||||
|
||||
**Result: STOP before the closing pass (pass 5).** After packs 2–4 the stop line said continue (measured 3,396,621;
|
||||
flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters
|
||||
(5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per
|
||||
orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned.
|
||||
|
||||
| Number | Value |
|
||||
|---|---|
|
||||
| Pages with an inbound link (link rule) | 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run) |
|
||||
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
|
||||
| Max out-degree | 11 |
|
||||
| Links accepted / rejected | 270 / 0 in packs 2–4 (574 / 0 in all) |
|
||||
| "none" answers | 38 in packs 2–4 (47 in all) |
|
||||
| Link agents | 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each |
|
||||
| `link` | 658,909 (2,098 per sentence; 214,054–228,310 per agent) |
|
||||
| `orchestration-build` at the result | 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn |
|
||||
| Catalogue filter (step 9.2) on P1 | pools 146 / 174 / 212, nothing cut (largest 8,437 characters) |
|
||||
| Price of the filter at collection 1's budget | 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited) |
|
||||
|
||||
Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session)
|
||||
and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns ×
|
||||
(47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M:
|
||||
|
||||
| k packs per interval | 20 packs per session | 40 packs per session |
|
||||
|---|---|---|
|
||||
| 5 | 68.3–99.2 M (× 1.10: 75.2–109.2) | 69.4–100.3 M (76.4–110.4) |
|
||||
| 10 | 61.4–92.3 M (67.6–101.6) | 61.0–91.9 M (67.1–101.0) |
|
||||
| 20 | 58.4–89.3 M (64.2–98.2) | 57.5–88.4 M (63.3–97.2) |
|
||||
|
||||
The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file)
|
||||
already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model
|
||||
(52.8 M, 4 points, not validated).
|
||||
|
||||
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with
|
||||
the linker's instructions verbatim (start-up 47,705; the `okf-wiki-linker` type is not defined in the main checkout
|
||||
where the session started), and at least one used a script to check its own anchors; a was measured in a session that
|
||||
was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.
|
||||
Loading…
Add table
Add a link
Reference in a new issue