okf-wiki 1.1.1

First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-11-14 21:03:20 +01:00
commit 0fd86a23a3
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
227 changed files with 96077 additions and 0 deletions

120
evals/README.md Normal file
View file

@ -0,0 +1,120 @@
# evals
## Direction probe (R)
`direction_probe.py` sets up and scores the direction probe: three arms answer the same questions from the same sources.
It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm.
- `filsok`: file search alone over the sources in the cut.
- `wiki`: the okf-wiki plus file search, following the skill's router (`index.md`, the pages, the raw source at `x-locator`).
- `astro`: a wiki built with Astro-Han's `karpathy-llm-wiki` skill (MIT) on the same sources, following its Query steps.
```
python3 evals/direction_probe.py fetch-astro --sources <source dir> --cut-manifest gates/wholedoc.manifest.json \
--project <astro project OUTSIDE the repository> --topic <topic> --collected <YYYY-MM-DD>
python3 evals/direction_probe.py setup --run <run dir> --arm <filsok|wiki|astro> --sources <source dir> \
--cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \
--wd-set gates-local/wholedoc/set-12.json [--wiki <okf-wiki bundle>] [--astro <astro project>]
python3 evals/direction_probe.py collect --run <run dir> --arm <arm>
python3 gates/multidoc.py gate ... --answers <run>/answers/<arm>.md.json --out <run>/results/<arm>.md.json
python3 gates/wholedoc.py gate ... --answers <run>/answers/<arm>.wd.json --out <run>/results/<arm>.wd.json
python3 evals/direction_probe.py harvest --run <run dir> --transcripts <subagent transcript folder>
python3 evals/direction_probe.py score --run <run dir> --arms filsok,wiki,astro --md-ids d01,d02,...
```
- **setup** copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per
question with the question, the search rules and the answer format, and never the key, the evidence or the count.
Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with
`cd <basis> &&` and uses a fixed list of read-only tools.
- **harvest** sums every usage field in each task's transcript, counting each message id once, and names every tool
call that breaks the task's rules. A breach is a fact to report, not a verdict: `echo` between two greps breaks the
rule without reading anything outside the basis.
- **score** filters the unchanged gates' per-question results to the probe questions. It reports per arm and group
(multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing
result file is a failure, not a zero.
Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository
(CLAUDE.md rule 3). Tests: `python3 -m unittest discover -s evals/tests`. Types: `mypy --strict evals`.
## v1-k step 9: semantic links measured on P1's pages (SC13)
The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to
their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for `link` + `orchestration-build`; the stop
check runs after every pass. Output stays outside the repository (`~/okf/benk/wiki/v1/lenkemaaling/`, rule 3).
**Result: STOP after pass 1 of 5.** Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029
of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun.
| Number | Value |
|---|---|
| Pages with an inbound link (link rule) | 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run) |
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
| Max out-degree | 10 |
| Links accepted / rejected | 304 / 0 (no reason hit) |
| "none" answers | 9 of 104 pages |
| Link agents | 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each |
| `link` | 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked) |
| `orchestration-build` at the result | 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn |
| Catalogue share of link input | 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted) |
Projection to collection 1 (~120 packs), orchestration at the measured `a + b × turn` restarted every session:
| Turns per pack | 20 turns per session | 40 | 80 |
|---|---|---|---|
| 4 (480 turns) | 50.0 M | 60.9 M | 82.5 M |
| 5 (600 turns) | 62.5 M | 76.1 M | 101.3 M |
Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M,
plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent ×
≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so `klink task` cannot run on it
without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M.
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent
with the linker's instructions verbatim (the session started in the main checkout, where `okf-wiki-linker` is not
defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's
start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a
fully stripped K) is not in it.
## v1-k step 9b: semantic links measured with the cost cuts, collection 1 re-projected
Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (`klink pass-before --pack 2 --to 4`
without `--catalogue`, three linkers spawned in one turn, one `klink pass-after`), cost line 7 M tokens for `link` +
`orchestration-build` counted on this step only: the stop check read a copy of the session's main transcript from the
step's start marker on, because the session had first carried another order.
**Result: STOP before the closing pass (pass 5).** After packs 2–4 the stop line said continue (measured 3,396,621;
flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters
(5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per
orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned.
| Number | Value |
|---|---|
| Pages with an inbound link (link rule) | 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run) |
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
| Max out-degree | 11 |
| Links accepted / rejected | 270 / 0 in packs 2–4 (574 / 0 in all) |
| "none" answers | 38 in packs 2–4 (47 in all) |
| Link agents | 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each |
| `link` | 658,909 (2,098 per sentence; 214,054–228,310 per agent) |
| `orchestration-build` at the result | 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn |
| Catalogue filter (step 9.2) on P1 | pools 146 / 174 / 212, nothing cut (largest 8,437 characters) |
| Price of the filter at collection 1's budget | 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited) |
Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session)
and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns ×
(47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M:
| k packs per interval | 20 packs per session | 40 packs per session |
|---|---|---|
| 5 | 68.3–99.2 M (× 1.10: 75.2–109.2) | 69.4–100.3 M (76.4–110.4) |
| 10 | 61.4–92.3 M (67.6–101.6) | 61.0–91.9 M (67.1–101.0) |
| 20 | 58.4–89.3 M (64.2–98.2) | 57.5–88.4 M (63.3–97.2) |
The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file)
already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model
(52.8 M, 4 points, not validated).
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with
the linker's instructions verbatim (start-up 47,705; the `okf-wiki-linker` type is not defined in the main checkout
where the session started), and at least one used a script to check its own anchors; a was measured in a session that
was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.

540
evals/direction_probe.py Normal file
View file

@ -0,0 +1,540 @@
"""The direction probe R (PLAN.md § Leveranser, row R): three arms answer the same questions from the same sources.
filsok file search alone over the sources in the cut
wiki the okf-wiki plus file search, following the skill's router (index.md, the pages, the raw source)
astro a wiki built with Astro-Han's karpathy-llm-wiki skill on the same sources, following its Query steps
Each question is answered by one isolated subagent per arm in the user's session. This module never calls a model:
fetch-astro the skill's Fetch step done mechanically: each source copied byte-exact under the raw template's
header into <project>/raw/<topic>/<name> (the sources are local, clean Markdown already)
setup one arm's basis (a copy, so no arm reads another's folder) and one task file per question, which
carries the question and never the answer key, the evidence or the count
collect the per-question answer files into the answers files the gates read
harvest tokens (every usage field, each message id once) and isolation breaches from the transcripts
score the gates' per-question results for the probe questions only: per arm and group, and in pairs
The gates are run unchanged (`gates/multidoc.py gate`, `gates/wholedoc.py gate`, each with --out).
"""
from __future__ import annotations
import argparse
import json
import os
import re
import shlex
import shutil
import sys
from itertools import combinations
from math import comb
from pathlib import Path
from typing import Any
ARMS = ("filsok", "wiki", "astro")
GROUPS = ("flerdokument", "helt dokument", "telling")
MD_SCHEMA = "okf-wiki/multidoc-set/1"
WD_SCHEMA = "okf-wiki/wholedoc-set/1"
ALLOWED_TOOLS = ("Read", "Grep", "Glob", "Bash", "Write", "SubagentHandback")
READ_ONLY = ("grep", "egrep", "fgrep", "rg", "find", "ls", "wc", "head", "tail", "sed", "cat", "sort", "uniq", "cut",
"tr", "nl")
SEPARATORS = ("|", "&&", ";", "||")
FIND_ACTIONS = ("-exec", "-execdir", "-ok", "-okdir", "-delete", "-fprint", "-fprint0", "-fprintf", "-fls")
USAGE_FIELDS = ("input_tokens", "cache_creation_input_tokens", "cache_read_input_tokens", "output_tokens")
class ProbeError(Exception):
"""The command cannot run; reported as FAIL with exit 1."""
def load_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
raise ProbeError(f"cannot read {path}: {exc}") from exc
def write_json(path: Path, data: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
def copy_names(sources: Path, names: list[str], target: Path) -> None:
target.mkdir(parents=True, exist_ok=True)
for name in names:
src = sources / name
if not src.is_file():
raise ProbeError(f"source {src} is missing")
shutil.copyfile(src, target / name)
# ---------------------------------------------------------------------------------------------- fetch-astro
def fetch_astro(sources: Path, names: list[str], project: Path, topic: str, collected: str) -> list[Path]:
"""The Fetch step of karpathy-llm-wiki, per its raw template, with the source text preserved byte-exact."""
folder = project / "raw" / topic
targets = [folder / name for name in names]
existing = [str(t) for t in targets if t.exists()]
if existing:
raise ProbeError(f"raw files exist already and are never overwritten: {', '.join(existing)}")
folder.mkdir(parents=True, exist_ok=True)
for name, target in zip(names, targets):
src = sources / name
if not src.is_file():
raise ProbeError(f"source {src} is missing")
header = (f"# {Path(name).stem}\n\n> Source: kilde/{name}\n> Collected: {collected}\n"
"> Published: Unknown\n\n")
target.write_bytes(header.encode("utf-8") + src.read_bytes())
return targets
# ---------------------------------------------------------------------------------------------- setup
SEARCH = {
"filsok": """BASIS holds the source files and nothing else. Search them with grep and find (Bash, as above), and read
what you find with Read.""",
"wiki": """BASIS holds `wiki/`, a knowledge wiki compiled from the sources, and `kilde/`, the source files. A page
names its source as `/kilde/<name>`, which is the file `BASIS/kilde/<name>`. Follow this order:
1. Read `BASIS/wiki/index.md` first.
2. Read the pages the index points to.
3. Open the raw source behind a footnote (`resource` at `x-locator`) when the quote has to be checked or when the
page does not say enough.
You may also search `wiki/` and `kilde/` with grep and find (Bash, as above) at any point.""",
"astro": """BASIS holds `wiki/`, a knowledge wiki compiled from the sources with the karpathy-llm-wiki skill, and
`raw/`, the source files it was compiled from (each raw file starts with a short metadata header; the source text
follows it unchanged). Follow the skill's Query steps:
1. Read `BASIS/wiki/index.md` to locate candidate articles, then full-text search `wiki/` with the topic's key terms
and their synonyms (grep through Bash, as above).
2. Read the articles you found and synthesize an answer.
3. Prefer wiki content over your own training knowledge.
Each article links the raw files it was compiled from in its `Raw` field. Open those raw files when you need an exact
value or a verbatim quote.""",
}
CITE_NAME = {
"filsok": "the bare name of a source file in BASIS",
"wiki": "the bare name of a source file in `BASIS/kilde`",
"astro": "the bare name of a raw file in `BASIS/raw/<topic>/`",
}
def render_task(arm: str, basis: Path, qid: str, question: str, is_count: bool, answer_path: Path) -> str:
count_line = '"count": <integer>, ' if is_count else ""
count_rule = ("\n- `count`: the number the question asks for, as a JSON integer. Say in `answer` how you counted."
if is_count else "")
return f"""# Answer one question
BASIS: `{basis}`
You answer ONE question from the files in BASIS. Read nothing outside BASIS apart from this task file: no other
folder, no web, no earlier work. Answer only from what the files say, and say plainly when they do not say something.
Tools: Read (a file inside BASIS), Bash for read-only search, and Write once, for the answer file. No other tool.
Every Bash command starts with `cd {basis} && ` and then uses only {', '.join(READ_ONLY)}, joined by `|` or `&&`,
with paths relative to BASIS: never `..`, `~` or a path outside BASIS, no `$` expansion, no output redirection except
`2>/dev/null`, no `find -exec` or `-delete`, no `sed -i`. For example: `cd {basis} && grep -rn "term" .`
## How to search
{SEARCH[arm]}
## The question
{question}
## The answer
Write one JSON file to `{answer_path}`:
```json
{{"id": "{qid}", "answer": "<your answer in prose>", {count_line}"citations": [{{"file": "<name>", "quote": "<text>"}}]}}
```
- `answer`: answer every part of the question.{count_rule}
- `citations`: one or more per source file your answer relies on, and cite every such file.
- `file`: {CITE_NAME[arm]}, for example `notes.md`: never a path and never a wiki page.
- `quote`: text copied character for character from that file (only whitespace may differ), long enough to carry
the fact, for example the whole line or sentence.
When the file is written, reply with the one word `done`.
"""
def probe_questions(md_set: Path, md_ids: list[str], wd_set: Path) -> list[tuple[str, dict[str, Any]]]:
md, wd = load_json(md_set), load_json(wd_set)
if not isinstance(md, dict) or md.get("schema") != MD_SCHEMA:
raise ProbeError(f"{md_set} is not a set with schema {MD_SCHEMA}")
if not isinstance(wd, dict) or wd.get("schema") != WD_SCHEMA:
raise ProbeError(f"{wd_set} is not a set with schema {WD_SCHEMA}")
by_id = {q["id"]: q for q in md["questions"]}
missing = [i for i in md_ids if i not in by_id]
if missing:
raise ProbeError(f"probe ids missing from {md_set}: {', '.join(missing)}")
return [("md", by_id[i]) for i in md_ids] + [("wd", q) for q in wd["questions"]]
def setup_arm(run: Path, arm: str, sources: Path, names: list[str], md_set: Path, md_ids: list[str], wd_set: Path,
wiki: Path | None = None, astro: Path | None = None) -> list[Path]:
if arm not in ARMS:
raise ProbeError(f"unknown arm {arm!r}; the arms are {', '.join(ARMS)}")
if arm == "wiki" and wiki is None:
raise ProbeError("the wiki arm needs --wiki")
if arm == "astro" and astro is None:
raise ProbeError("the astro arm needs --astro")
questions = probe_questions(md_set, md_ids, wd_set)
basis = run / "arms" / arm
if basis.exists():
raise ProbeError(f"{basis} exists already; an arm is set up once, from fresh")
if arm == "filsok":
copy_names(sources, names, basis)
elif arm == "wiki" and wiki is not None:
shutil.copytree(wiki, basis / "wiki")
copy_names(sources, names, basis / "kilde")
elif astro is not None:
shutil.copytree(astro / "raw", basis / "raw")
shutil.copytree(astro / "wiki", basis / "wiki")
tasks = []
for group, q in questions:
key = f"{group}-{q['id']}"
task = run / "tasks" / arm / f"{key}.md"
answer = run / "answers" / arm / f"{key}.json"
task.parent.mkdir(parents=True, exist_ok=True)
task.write_text(render_task(arm, basis, q["id"], q["question"], q.get("type") == "count", answer),
encoding="utf-8")
tasks.append(task)
(run / "answers" / arm).mkdir(parents=True, exist_ok=True)
return tasks
# ---------------------------------------------------------------------------------------------- collect
def collect(run: Path, arm: str) -> tuple[Path, Path, list[str]]:
"""The per-question answers into one answers file per gate. The id comes from the file name."""
folder = run / "answers" / arm
gathered: dict[str, list[dict[str, Any]]] = {"md": [], "wd": []}
problems = []
for path in sorted(folder.glob("*.json")):
group, _, qid = path.stem.partition("-")
if group not in gathered or not qid:
problems.append(f"{path.name}: not a probe answer name")
continue
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
data = None
if not isinstance(data, dict):
problems.append(f"{path.name}: not a JSON object")
continue
gathered[group].append({**data, "id": qid})
md_path, wd_path = run / "answers" / f"{arm}.md.json", run / "answers" / f"{arm}.wd.json"
write_json(md_path, {"arm": arm, "answers": gathered["md"]})
write_json(wd_path, {"arm": arm, "answers": gathered["wd"]})
return md_path, wd_path, problems
# ---------------------------------------------------------------------------------------------- harvest
def first_prompt(lines: list[dict[str, Any]]) -> tuple[str, str]:
for line in lines:
if line.get("type") == "user":
content = line.get("message", {}).get("content")
if isinstance(content, list):
content = " ".join(b.get("text", "") for b in content if isinstance(b, dict))
return (content if isinstance(content, str) else ""), str(line.get("timestamp", ""))
return "", ""
def inside(path: object, root: Path) -> bool:
if not isinstance(path, str) or not os.path.isabs(path):
return False
norm = Path(os.path.normpath(path))
return norm == root or root in norm.parents
def bash_breach(command: str, basis: Path) -> str | None:
"""None when the command is read-only search that starts with `cd <basis> &&` and stays inside the basis."""
if "`" in command or re.search(r"\$[A-Za-z_{(]", command):
return "Bash with an expansion"
try:
lexer = shlex.shlex(re.sub(r"\s2>/dev/null(?=\s|$)", "", command), posix=True, punctuation_chars="|&;<>()")
lexer.whitespace_split = True
tokens = list(lexer)
except ValueError:
return "Bash that cannot be parsed"
segments: list[list[str]] = [[]]
for token in tokens:
if token in SEPARATORS:
segments.append([])
elif token and set(token) <= set("&<>()"):
return f"Bash with {token!r}"
else:
segments[-1].append(token)
first = segments[0]
if len(first) != 2 or first[0] != "cd" or not inside(first[1], basis):
return "Bash that does not start with cd into the basis"
for segment in segments[1:]:
if not segment or segment[0] not in READ_ONLY:
return f"Bash command not allowed: {segment[0] if segment else '(empty)'}"
for token in segment[1:]:
value = token.split("=", 1)[1] if token.startswith("-") and "=" in token else token
if segment[0] == "find" and token in FIND_ACTIONS:
return f"Bash find action {token}"
if segment[0] == "sed" and (token.startswith("-i") or token.startswith("--in-place")):
return "Bash sed -i"
if value.startswith("~") or ".." in value.split("/"):
return f"Bash path outside the basis: {token}"
if value.startswith("/") and value != "/dev/null" and not inside(value, basis):
return f"Bash path outside the basis: {token}"
return None
def breaches_of(tools: list[tuple[str, dict[str, Any]]], basis: Path, task: Path, answer: Path) -> list[str]:
found = []
for name, inp in tools:
if name not in ALLOWED_TOOLS:
found.append(f"{name}: tool not allowed")
elif name == "Read":
path = inp.get("file_path")
if not (inside(path, basis) or (isinstance(path, str) and os.path.normpath(path) == str(task))):
found.append(f"Read outside the basis: {path}")
elif name in ("Grep", "Glob"):
path = inp.get("path")
pattern = inp.get("pattern", "")
if path is None:
found.append(f"{name} without a path (searches the working directory)")
elif not inside(path, basis):
found.append(f"{name} outside the basis: {path}")
elif name == "Glob" and isinstance(pattern, str) and os.path.isabs(pattern) and not inside(pattern, basis):
found.append(f"Glob pattern outside the basis: {pattern}")
elif name == "Bash":
breach = bash_breach(str(inp.get("command", "")), basis)
if breach:
found.append(breach)
elif name == "Write" and inp.get("file_path") != str(answer):
found.append(f"Write to another file: {inp.get('file_path')}")
return found
def read_transcript(path: Path) -> list[dict[str, Any]]:
lines = []
for raw in path.read_text(encoding="utf-8").splitlines():
try:
line = json.loads(raw)
except json.JSONDecodeError:
continue
if isinstance(line, dict):
lines.append(line)
return lines
def harvest(transcripts: Path, run: Path) -> dict[str, dict[str, Any]]:
"""Per task: all tokens its transcript records (each message id once), and every isolation breach."""
pattern = re.compile(re.escape(str(run / "tasks")) + r"/(" + "|".join(ARMS) + r")/((?:md|wd)-[A-Za-z0-9]+)\.md")
rows: dict[str, dict[str, Any]] = {}
for path in sorted(transcripts.glob("*.jsonl")):
lines = read_transcript(path)
prompt, started = first_prompt(lines)
match = pattern.search(prompt)
if match is None:
continue
arm, key = match.group(1), match.group(2)
usage: dict[str, dict[str, int]] = {}
tools: list[tuple[str, dict[str, Any]]] = []
for line in lines:
message = line.get("message", {})
if line.get("type") != "assistant" or not isinstance(message, dict):
continue
mid = str(message.get("id", ""))
fields = message.get("usage") or {}
seen = usage.setdefault(mid, {})
for f in USAGE_FIELDS:
value = fields.get(f)
if isinstance(value, int):
seen[f] = max(seen.get(f, 0), value)
for block in message.get("content") or []:
if isinstance(block, dict) and block.get("type") == "tool_use":
tools.append((str(block.get("name")), block.get("input") or {}))
row = {
"tokens": sum(sum(v.values()) for v in usage.values()),
"breaches": breaches_of(tools, run / "arms" / arm, run / "tasks" / arm / f"{key}.md",
run / "answers" / arm / f"{key}.json"),
"transcript": path.name,
"started": started,
}
name = f"{arm}/{key}"
previous = rows.get(name)
runs = 1 + (previous["runs"] if previous else 0)
if previous is None or started >= previous["started"]:
rows[name] = {**row, "runs": runs}
else:
previous["runs"] = runs
for row in rows.values():
row.pop("started")
return rows
# ---------------------------------------------------------------------------------------------- score
def gate_results(path: Path) -> dict[str, dict[str, Any]]:
if not path.is_file():
raise ProbeError(f"no gate results at {path}; an arm that was not scored is a failure, not a zero")
data = load_json(path)
return {r["id"]: r for r in data.get("results", [])}
def mcnemar_exact(only_a: int, only_b: int) -> float:
"""Two-sided exact McNemar: the binomial test over the discordant pairs, p = 0.5."""
n, k = only_a + only_b, min(only_a, only_b)
if n == 0:
return 1.0
return min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / float(2 ** n))
def score(run: Path, arms: list[str], md_ids: list[str]) -> dict[str, Any]:
tokens_path = run / "tokens.json"
tokens = load_json(tokens_path) if tokens_path.is_file() else {}
correct: dict[str, dict[str, dict[str, bool]]] = {}
table: dict[str, Any] = {"arms": {}, "pairs": {}, "mcnemar": {}}
for arm in arms:
md = gate_results(run / "results" / f"{arm}.md.json")
wd = gate_results(run / "results" / f"{arm}.wd.json")
by_group: dict[str, dict[str, bool]] = {g: {} for g in GROUPS}
answer_ok = source_ok = 0
for qid in md_ids:
r = md.get(qid, {})
by_group["flerdokument"][f"md-{qid}"] = bool(r.get("correct"))
answer_ok += bool(r.get("answer_ok"))
source_ok += bool(r.get("source_ok"))
for qid, r in wd.items():
group = "telling" if r.get("type") == "count" else "helt dokument"
by_group[group][f"wd-{qid}"] = bool(r.get("correct"))
answer_ok += bool(r.get("answer_ok"))
source_ok += bool(r.get("source_ok"))
correct[arm] = by_group
n = sum(len(v) for v in by_group.values())
own = [v for k, v in tokens.items() if k.startswith(f"{arm}/")]
table["arms"][arm] = {
"groups": {g: [sum(v.values()), len(v)] for g, v in by_group.items()},
"total": [sum(sum(v.values()) for v in by_group.values()), n],
"answer_ok": answer_ok,
"source_ok": source_ok,
"tokens": {"sum": sum(int(v["tokens"]) for v in own), "questions": len(own)},
"breaches": sum(len(v["breaches"]) for v in own),
}
for a, b in combinations(arms, 2):
a, b = (b, a) if b == "wiki" else (a, b)
pair: dict[str, dict[str, int]] = {}
for g in GROUPS:
counts = {"both": 0, f"only {a}": 0, f"only {b}": 0, "neither": 0}
for key, ok_a in correct[a][g].items():
ok_b = correct[b][g].get(key, False)
label = "both" if ok_a and ok_b else f"only {a}" if ok_a else f"only {b}" if ok_b else "neither"
counts[label] += 1
pair[g] = counts
table["pairs"][f"{a} vs {b}"] = pair
only_a, only_b = sum(c[f"only {a}"] for c in pair.values()), sum(c[f"only {b}"] for c in pair.values())
table["mcnemar"][f"{a} vs {b}"] = {f"only {a}": only_a, f"only {b}": only_b, "p": mcnemar_exact(only_a, only_b)}
table["per_question"] = {arm: {k: v for g in GROUPS for k, v in correct[arm][g].items()} for arm in arms}
return table
def render_table(table: dict[str, Any]) -> str:
head = "| arm | " + " | ".join(f"{g} (N)" for g in GROUPS) + " | total | answer ok | source ok | tokens/question |"
lines = [head, "|" + "---|" * (len(GROUPS) + 5)]
for arm, row in table["arms"].items():
t = row["tokens"]
per_q = f"{t['sum'] / t['questions'] / 1e6:.2f} M" if t["questions"] else "not harvested"
cells = [f"{c} of {n}" for c, n in (row["groups"][g] for g in GROUPS)]
total = row["total"]
lines.append(f"| {arm} | " + " | ".join(cells) + f" | {total[0]} of {total[1]} | {row['answer_ok']} | "
f"{row['source_ok']} | {per_q} |")
lines.append("")
for name, pair in table["pairs"].items():
lines.append(f"{name}: " + "; ".join(f"{g}: " + ", ".join(f"{k} {v}" for k, v in c.items())
for g, c in pair.items()))
m = table["mcnemar"][name]
lines.append(f"{name}, all groups: " + ", ".join(f"{k} {v}" for k, v in m.items() if k != "p")
+ f"; exact McNemar p = {m['p']:.3g}")
return "\n".join(lines)
# ---------------------------------------------------------------------------------------------- main
def cut_names(manifest: Path) -> list[str]:
names = load_json(manifest).get("cut", {}).get("names")
if not isinstance(names, list) or not names:
raise ProbeError(f"{manifest} records no cut names")
return [str(n) for n in names]
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
f = sub.add_parser("fetch-astro")
f.add_argument("--sources", required=True)
f.add_argument("--cut-manifest", required=True)
f.add_argument("--project", required=True)
f.add_argument("--topic", required=True)
f.add_argument("--collected", required=True)
s = sub.add_parser("setup")
s.add_argument("--run", required=True)
s.add_argument("--arm", required=True)
s.add_argument("--sources", required=True)
s.add_argument("--cut-manifest", required=True)
s.add_argument("--md-set", required=True)
s.add_argument("--md-ids", required=True, help="comma-separated")
s.add_argument("--wd-set", required=True)
s.add_argument("--wiki")
s.add_argument("--astro")
c = sub.add_parser("collect")
c.add_argument("--run", required=True)
c.add_argument("--arm", required=True)
h = sub.add_parser("harvest")
h.add_argument("--run", required=True)
h.add_argument("--transcripts", required=True)
r = sub.add_parser("score")
r.add_argument("--run", required=True)
r.add_argument("--arms", required=True, help="comma-separated")
r.add_argument("--md-ids", required=True, help="comma-separated")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "fetch-astro":
written = fetch_astro(Path(args.sources), cut_names(Path(args.cut_manifest)), Path(args.project),
args.topic, args.collected)
print(f"raw files: {len(written)}")
elif args.command == "setup":
tasks = setup_arm(Path(args.run), args.arm, Path(args.sources), cut_names(Path(args.cut_manifest)),
Path(args.md_set), args.md_ids.split(","), Path(args.wd_set),
Path(args.wiki) if args.wiki else None, Path(args.astro) if args.astro else None)
print("\n".join(str(t) for t in tasks))
elif args.command == "collect":
md_path, wd_path, problems = collect(Path(args.run), args.arm)
print(f"{md_path}\n{wd_path}")
for problem in problems:
print(f"problem: {problem}")
elif args.command == "harvest":
rows = harvest(Path(args.transcripts), Path(args.run))
write_json(Path(args.run) / "tokens.json", rows)
print(f"tasks harvested: {len(rows)}; breaches: {sum(len(r['breaches']) for r in rows.values())}")
else:
table = score(Path(args.run), args.arms.split(","), args.md_ids.split(","))
write_json(Path(args.run) / "table.json", table)
print(render_table(table))
except ProbeError as exc:
print(f"FAIL: {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())

125
evals/p1/README.md Normal file
View file

@ -0,0 +1,125 @@
# P1 method trial (TRIAL, branch `p1-metodeprove`, not v1 code)
The P1 method trial builds the 19 sources of the cut (the sources behind d01 d02 d03 d04 d37 d46 d53) three ways, in one
fixed order, and measures cost and quality side by side, so the operator can choose the build method for v1 by seeing
small real examples (PLAN.md § Rammer, order `20261002T094056Z-495685981`, plan v3).
- **K**: the fragment compiler. Code splits the sources into blocks, a narrow trial agent (`p1-compiler`) writes short
fragments per pack of five sources, and code resolves concept identity and assembles the pages.
- **K+**: a copy of K plus one consolidation pass (`p1-consolidator`) that rewrites pages with three or more fragments
into flowing text, keeping every footnote label.
- **D**: no model. A generated map of each source and a code-built concept register.
No module here calls a model. The model work is done in the operator's session by the two trial agent types under
`.claude/agents/`. Every output lives outside the repository, under `<wiki>/prove-p1/` (CLAUDE.md rule 3); this README
records numbers only.
## Modules
Run each as `PYTHONPATH=evals:. python3 -m p1.<module> ...`; each module's docstring is its usage.
| Module | What it does |
|---|---|
| `split.py` | the fixed source order, blocks with line spans, the block total, the three side-by-side concepts |
| `dmap.py` | D: map pages and register pages |
| `kpack.py` | K: the skeleton from every docs page, pack input parts, fragment validation, concept identity |
| `assemble.py` | K's pages from accepted fragments; K+'s copy, tasks and apply |
| `measure.py` | the audit record, coverage of source blocks, structure counts |
| `cost.py` | tokens from the run sessions' transcripts, tokens per source, projection, the stop rule |
| `sidebyside.py` | the side-by-side page and its link check |
Output files under `prove-p1/`: `rekkefolge.txt`, `blocks.json`, `begreper.txt`, `okter.txt`, `D/`, `K/`, `K+/`,
`resultat.json`, `struktur-<V>.json`, `dekning-<V>.json`, `kost.json`, `side-om-side.html`, `rapport.md`.
Tests: `python3 -m unittest discover -s evals/tests -p 'test_p1_*.py'`. Checks: `bash tests/p1_repo.sh` (repository)
and `bash tests/p1_trial.sh` (the trial's outputs; red until the runs are done).
## Results
Numbers only, filled in by the run steps (plan steps 8-11).
### Run 0 (step 8, 02.10.2026)
- Order: 19 sources (3 docs pages, then 16 release notes by version), equal to the files cited by the L4 cut.
- Blocks: 1221 in the 19 sources (docs pages 599, release notes 622; pack 1 = sources 1-5 = 602 blocks); 33727 blocks
in all 599 files of `kilde/` (the D12 projection base).
- Blocks per source in order: 59, 308, 232, 1, 2, 22, 35, 76, 21, 25, 57, 35, 34, 52, 33, 22, 46, 53, 108.
- Load test (the empty baseline run, not counted toward any cap): both trial types spawned and wrote under `prove-p1/`,
so the agents' work directory is `prove-p1/` (no `runtime/p1` fallback). `p1-compiler`: startup 18604, 2 turns,
39142 tokens in all; `p1-consolidator`: startup 18167, 2 turns, 38270 tokens in all.
- D: 254 pages (19 map, 235 register), 0 model tokens. Auditor exit 0, 0 findings. Criterion 1: 0 of 0 (bodies are
headings and link lists only, D8, by construction). Coverage 0 of 1221 blocks: by construction, because D cites no
source in a body footnote (D21 counts body-cited entries only; D's locators live in `sources[]`). Duplicate pairs 9
(5 by name, 4 by title Jaccard), merge candidates 0, islands 0, concept test 2 held by 254 pages, test 4 by 19.
### Run K (step 9, 02.10.2026)
- Skeleton: 3493 index entries from the titles and level 2-3 headings of all 198 docs pages (0 tokens).
- The cost harvester was fixed twice during the run, each with a failing test first (`77b16ab`, `0a50286`): a long
streamed message keeps only its first usage record in the transcript (output 2, no stop reason), so output was
undercounted (373 recorded for pack 1's 336 fragments). Output is now derived from the next message's input growth,
less the injected text counted high (2.5 characters per token), so every K output below is a lower bound. Checked on
the load test's known short call, which keeps its recorded value. The recorded value stays beside it in `kost.json`.
- K: per pack (blocks / index lines sent of 3493 / input parts / fragments accepted of proposed / rejected / accepted
with a concept not among the lines sent / turns / tokens):
- pack 1: 602 / 230 / 4 / 336 of 336 / 0 / 0 / 3 / 372996
- pack 2: 179 / 117 / 1 / 87 of 87 / 0 / 0 / 3 / 125716
- pack 3: 211 / 125 / 1 / 108 of 108 / 0 / 0 / 3 / 141320
- pack 4: 229 / 127 / 1 / 119 of 119 / 0 / 0 / 3 / 153494
- No fragment was rejected for any reason (incl. `release-note-new` 0); no new concept was proposed, so all 650
accepted fragments enrich skeleton entries. Merge candidates 0.
- Stop line after pack 1: projected per block 756525 (naive per source 1417385), cap 1000000: continue.
- K total: 793526 tokens in 4 runs and 12 turns (input 24, cache read 228534, cache write 429550, output 135418);
startup 74596 (4 x about 18.6 k). Tokens per source, mean of sources 1-5: 74599; of sources 15-19: 35119.
- Projected to all 599 sources (33727 blocks): fit 52842866 (4 points, growth term b = 0.0554, not validated);
flat 21919125.
- Auditor exit 0, 0 findings. Criterion 1: 664 of 664. Quotes: 650 entries, 160 supplied by code, 0 on a structural
line. Precision of body-cited entries: line 591, block 59, section 0, document 0.
- Coverage: 623 of 1221 blocks. Per position in the pack (covered of blocks, n packs): 1: 85 of 160 (4); 2: 241 of 424
(4); 3: 181 of 395 (4); 4: 83 of 182 (4); 5: 33 of 60 (3). A direction signal, n = 3-4.
- Structure: 212 concept pages, 3281 skeleton entries unused. Concepts per source in order: 11, 61, 31, 1, 0, 5, 13,
15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11. Duplicate pairs 3 (all title Jaccard), islands 63, documents per page mean
1.65 (max 8), version numbers in running text 5 of 664 claims, concept test 2 held by 212, test 4 by 73.
- Seen in the compiler's own reports: in one pack it wrote numbers as words to avoid the quote rule (D5 requires a
quote when the text holds a digit); the compilers' own fragment counts differed from the files (the files count).
### Run K+ (step 10, 02.10.2026)
- K+: a copy of K; 76 pages carry 3 or more fragments, which gives 10 consolidation runs of up to 8 pages.
- Run 1: 8 pages, applied 8, refused 0. 109651 tokens (input 8, cache read 72768, cache write 29909, output 6966, a lower
bound), 4 turns, startup 18176. The consolidator took one more turn than its instructions allow (it read its
output back).
- Stop line after run 1: 109651 x 76 / 8 = 1041685 > cap 1000000: STOP. Runs 2-10 were not run; K+ stands as K with
8 of 76 pages consolidated.
- K+ total = K + consolidation = 793526 + 109651 = 903177 tokens.
- Auditor exit 0, 0 findings. Criterion 1: 662 of 662 (2 fewer claims than K: sentences joined). Coverage 623 of 1221,
precision and quotes as K (line 591, block 59; 160 code quotes, 0 structural). 212 pages, 3 duplicate pairs,
63 islands, version numbers in running text 5 of 662.
### Step 11: the side-by-side table (02.10.2026)
Corrections after run 0, each its own commit with a failing test first: the backticked-term length is checked after
trimming (`26f521f`) and the concept pick counts a document only where the term stands as a term (`09a50b2`). The
first pick and D built before `26f521f` are kept in `prove-p1/forkastet/`; D was rebuilt (252 pages: 19 map, 233
register; 9 duplicate pairs; auditor exit 0) and the concepts were picked again by the same rule.
| variant | built to source | concepts (total / per source 1-5, 15-19 / test 2 / test 4) | skeleton unused | merge candidates | coverage | tokens per source (input / cache read / cache write / output / startup) | 1-5 | 15-19 | turns / subagents | projected 599 (fit / flat) | criterion 1 | fragments accepted / code quotes / structural quotes | precision (line / block / section / document) | pages / islands / duplicates |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| K | 19 of 19 | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | 1 / 12028 / 22608 / 7127 / 3926 | 74599 | 35119 | 12 / 4 | 52842866 / 21919125 | 664 of 664 | 650 of 650 / 160 / 0 | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
| K+ | 19 of 19, consolidation stopped after run 1 (8 of 76 pages) | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | consolidation 109651 for 8 pages; K+ total = K + consolidation = 903177 | - | - | 4 / 1 | not computed: stopped after 1 of 10 runs, and the pages to consolidate at 599 sources are not measured | 662 of 662 | as K | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
| D | 19 of 19 | 0 concept pages (252 map and register) / 0, 0 by construction / 252 / 19 | - | 0 | 0 of 1221 (by construction) | 0 | 0 | 0 | 0 / 0 | 0 | 0 of 0 (by construction) | - | - | 252 / 0 / 9 |
| L4 (structure counted today) | 19 of 19 | 151 / 33, 33 / 151 / 68 | - | 0 | 685 of 1221 | see the cost row | - | - | - | - | 1048 of 1048 | - / 0 / 41 of 832 | 832 / 0 / 0 / 0 | 151 / 54 / 2 |
| L2b / L4 cost | measured 27-28.09, not measured again | | | | | L4 105.8 M for 19 sources (84 % cache read); L2b writing 30.93 M for 19 sources | | | | L2b about 1.6-1.8 bn writing + 8.6 bn reading | | | | |
- Every K and K+ output number is a lower bound (the harvester fix in run K). Tokens per source split each pack's cost
by block share. The projection is not validated (4 points).
- Side-by-side page: 3 concepts x 4 columns; 454 of 454 source links resolve. Main session (orchestration, apart):
10.9 M tokens up to step 11, almost all cache read.
- Concepts per source in order 1-19 (a page counts at the earliest source its body cites, D10): K and K+ 11, 61, 31,
1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11; L4 2, 9, 19, 1, 2, 10, 14, 17, 5, 6, 14, 10, 7, 2, 9, 6, 3, 6, 9;
D 0 for every source (its register pages cite nothing in a body).
- K+ consolidation cost 13706 per page (lower bound) and is not split per source, because a page joins fragments from
several sources.
- Seen in the review (BLOCK, not fixed in this run): 45 of 1221 blocks are only MDX tag lines and cannot be covered
(K 623 of 1176 without them); the index-line filter sends a line when the whole name occurs as a phrase, not per
token as D7 is written; `cost.summarize` and the stop command have no test.

5
evals/p1/__init__.py Normal file
View file

@ -0,0 +1,5 @@
"""TRIAL, branch p1-metodeprove, not v1 code. Nothing outside evals/ imports it.
The P1 method trial (PLAN.md § Rammer, order 20261002T094056Z-495685981): three build methods for a concept wiki (K, K+,
D) on the 19 sources of the cut, measured side by side for cost and quality. No module here calls a model.
"""

287
evals/p1/assemble.py Normal file
View file

@ -0,0 +1,287 @@
"""K's assembler and K+'s apply step (TRIAL, P1): pages from accepted fragments, by code. It calls no model.
build --work DIR --kilde-root DIR --out DIR [--at ISO]
One page per concept that at least one accepted fragment enriches (unused skeleton entries get none). Docs
fragments form the body; release-note fragments go under "## History" in version order. Every sentence ends
in a content footnote and, for a release note, a version footnote whose quote is the bare version on the
note's heading line. A fragment without a model quote gets a code quote: the first line of its span that is
not structural, marked x-quote-by: code. A fragment that names another concept's title links to it once.
copy --src DIR --out DIR
K+ starts as a copy of K.
tasks --bundle DIR --work DIR [--min 3] [--per-run 8]
One task file per consolidation run (pages with at least --min fragments, --per-run pages each) in
<work>/Kplus-oppgaver/, output path <work>/Kplus-bodies/run-N.json.
apply --bundle DIR --bodies FILE
Replaces a page body only when the new body keeps every old footnote label and adds none; prints
"applied X, refused Y".
--kilde-root is the folder that contains kilde/. Exit 0 = done, 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import shutil
import sys
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from okfwiki.audit import FOOTNOTE_DEF, FOOTNOTE_REF, HEADING, read_document, sentence_spans
from p1.dmap import frontmatter
from p1.split import FENCE, headings_of, version_key
GENERATOR = "okf-wiki/p1-assemble"
QUOTE_MAX = 200
HTML_TAG_ONLY = re.compile(r"^\s*</?[A-Za-z][^<>]*/?>\s*$")
TABLE_RULE = re.compile(r"^\s*\|?\s*:?-+:?\s*(\|\s*:?-+:?\s*)*\|?\s*$")
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
@dataclass
class Run:
task: Path
output: Path
pages: list[str]
def structural(line: str) -> bool:
return (not line.strip() or HEADING.match(line) is not None or FENCE.match(line) is not None
or HTML_TAG_ONLY.match(line) is not None or TABLE_RULE.match(line) is not None)
def code_quote(span: list[str], start: int) -> tuple[str, int]:
"""The first non-structural line of a span (stripped, capped), and its file line."""
for i, line in enumerate(span):
if not structural(line):
return line.strip()[:QUOTE_MAX], start + i
return span[0].strip()[:QUOTE_MAX] or span[0], start
def footnoted(text: str, labels: list[str]) -> str:
text = text.strip()
if text and text[-1] not in ".!?":
text += "."
marks = "".join(f"[^{x}]" for x in labels)
out, last = [], 0
for _a, b in sentence_spans(text):
out.append(text[last:b] + marks)
last = b
out.append(text[last:])
return "".join(out)
def _link_titles(text: str, titles: dict[str, str], self_id: str, linked: set[str]) -> str:
for cid, title in sorted(titles.items(), key=lambda kv: -len(kv[1])):
if cid == self_id or cid in linked or len(title) < 4:
continue
pattern = re.compile(r"(?<![\w`\[])" + re.escape(title) + r"(?![\w`\]])", re.IGNORECASE)
m = pattern.search(text)
if m is None:
continue
text = text[:m.start()] + f"[{m.group(0)}]({cid}.md)" + text[m.end():]
linked.add(cid)
return text
def _version_line(lines: list[str], version: str) -> int:
for n, _level, text in headings_of(lines):
m = VERSION_HEADING.match(text)
if m and m.group(1) == version:
return n
raise RunError(f"no heading for version {version}")
def assemble(work: Path, kilde_root: Path, out: Path, at: str | None = None) -> int:
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
index = json.loads((work / "K-work" / "index.json").read_text(encoding="utf-8"))
frags = json.loads((work / "K-work" / "accepted.json").read_text(encoding="utf-8"))["fragments"]
entries = {e["id"]: e for e in index["entries"]}
kilde = kilde_root / "kilde"
cache: dict[str, list[str]] = {}
def lines_of(name: str) -> list[str]:
if name not in cache:
cache[name] = (kilde / name).read_text(encoding="utf-8").split("\n")
return cache[name]
by_concept: dict[str, list[dict[str, Any]]] = {}
for f in frags:
by_concept.setdefault(f["concept_id"], []).append(f)
titles = {cid: str(entries[cid]["title"]) for cid in by_concept}
pages: dict[str, str] = {}
for cid, items in by_concept.items():
e = entries[cid]
sources: list[dict[str, Any]] = []
versions: dict[str, str] = {}
body_parts: list[str] = []
history: list[tuple[tuple[int, ...], int, str]] = []
linked: set[str] = set()
for k, f in enumerate(items, 1):
lines = lines_of(f["file"])
src: dict[str, Any] = {"id": f"f{k}", "resource": f"/kilde/{f['file']}", "title": f["section"]}
if f.get("quote"):
src["x-locator"] = {"line": f["quote_line"]}
src["x-quote"] = f["quote"]
else:
quote, line = code_quote(lines[f["start"] - 1:f["end"]], f["start"])
src["x-locator"] = {"line": line}
src["x-quote"] = quote
src["x-quote-by"] = "code"
src["x-span"] = [f["start"], f["end"]]
sources.append(src)
labels = [f"f{k}"]
if f.get("version"):
v = str(f["version"])
if v not in versions:
versions[v] = "v" + re.sub(r"\D", "", v)
sources.append({"id": versions[v], "resource": f"/kilde/{f['file']}", "title": v,
"x-locator": {"line": _version_line(lines, v)}, "x-quote": v})
labels.append(versions[v])
text = _link_titles(footnoted(f["text"], labels), titles, cid, linked)
history.append((version_key(f"release-v{v}.md"), k, text))
else:
body_parts.append(_link_titles(footnoted(f["text"], labels), titles, cid, linked))
body = [f"# {e['title']}", ""]
for p in body_parts:
body += [p, ""]
if history:
body += ["## History", ""]
body += [f"- {t}" for _, _, t in sorted(history)]
body.append("")
body += [f"[^{s['id']}]: {Path(s['resource']).name}, line {s['x-locator']['line']}" for s in sources]
desc = FOOTNOTE_REF.sub("", items[0]["text"]).strip()[:200]
fm: dict[str, Any] = {"type": "Concept", "title": str(e["title"]), "description": desc, "sources": sources,
"x-aliases": [str(a) for a in e.get("aliases", [])], "x-fragments": len(items)}
if e.get("kind") == "merge-candidate" and e.get("merge_of"):
fm["x-merge-candidate"] = str(e["merge_of"])
fm["generated"] = {"by": GENERATOR, "at": at}
pages[f"{cid}.md"] = frontmatter(fm) + "\n" + "\n".join(body) + "\n"
out.mkdir(parents=True)
for name, text in pages.items():
(out / name).write_text(text, encoding="utf-8")
index_md = ["---", 'okf_version: "0.2"', "---", "", "# Concepts", ""]
index_md += [f"- [{titles[cid]}]({cid}.md)" for cid in sorted(by_concept)]
(out / "index.md").write_text("\n".join(index_md) + "\n", encoding="utf-8")
(out / "log.md").write_text(f"# Log\n\n## {at[:10]}\n* **Build**: K assembled by {GENERATOR}: {len(pages)} "
f"pages from {len(frags)} accepted fragments.\n", encoding="utf-8")
return len(pages)
def copy(src: Path, dst: Path) -> None:
if dst.exists():
raise RunError(f"{dst} exists; refusing to overwrite")
shutil.copytree(src, dst)
def _split_page(text: str) -> tuple[str, str, list[str], list[str]]:
"""(frontmatter block, title line, body lines without title and footnote definitions, definition lines)."""
end = text.index("\n---\n", 4) + 5
head, rest = text[:end], text[end:].split("\n")
title = next(line for line in rest if line.startswith("# "))
body = [line for line in rest if line != title and not FOOTNOTE_DEF.match(line.strip())]
defs = [line for line in rest if FOOTNOTE_DEF.match(line.strip())]
return head, title, body, defs
def consolidation_tasks(bundle: Path, work: Path, min_fragments: int = 3, per_run: int = 8) -> list[Run]:
chosen: list[str] = []
for p in sorted(bundle.glob("*.md")):
if p.name in ("index.md", "log.md"):
continue
fm = read_document(p, p.name).frontmatter or {}
if int(fm.get("x-fragments", 0)) >= min_fragments:
chosen.append(p.stem)
runs: list[Run] = []
for i in range(0, len(chosen), per_run):
n = len(runs) + 1
task = work / "Kplus-oppgaver" / f"run-{n}.md"
output = work / "Kplus-bodies" / f"run-{n}.json"
if task.exists():
raise RunError(f"{task} exists; refusing to overwrite")
parts = [f"# K+ consolidation run {n}\n\nRewrite each page body below into flowing text. Output: {output}\n"]
for cid in chosen[i:i + per_run]:
text = (bundle / f"{cid}.md").read_text(encoding="utf-8")
fm = read_document(bundle / f"{cid}.md", f"{cid}.md").frontmatter or {}
_head, title, body, _defs = _split_page(text)
quotes = [f"- [^{s['id']}]: {s.get('x-quote', '')}" for s in fm.get("sources", [])]
parts.append(f"## Page {cid} ({title[2:]})\n\nBody:\n\n" + "\n".join(body).strip()
+ "\n\nWhat each label quotes:\n\n" + "\n".join(quotes) + "\n")
task.parent.mkdir(parents=True, exist_ok=True)
task.write_text("\n".join(parts), encoding="utf-8")
runs.append(Run(task, output, chosen[i:i + per_run]))
return runs
def apply(bundle: Path, bodies_path: Path) -> tuple[int, int]:
bodies = json.loads(bodies_path.read_text(encoding="utf-8"))
applied = refused = 0
for cid, new in bodies.items():
page = bundle / f"{cid}.md"
if not page.is_file() or not isinstance(new, str):
refused += 1
continue
head, title, body, defs = _split_page(page.read_text(encoding="utf-8"))
old_labels = set(FOOTNOTE_REF.findall("\n".join(body)))
if set(FOOTNOTE_REF.findall(new)) != old_labels or FOOTNOTE_DEF.search(new):
refused += 1
continue
page.write_text(head + "\n" + title + "\n\n" + new.strip() + "\n\n" + "\n".join(defs) + "\n", encoding="utf-8")
applied += 1
return applied, refused
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
b = sub.add_parser("build")
b.add_argument("--work", required=True)
b.add_argument("--kilde-root", required=True)
b.add_argument("--out", required=True)
b.add_argument("--at")
c = sub.add_parser("copy")
c.add_argument("--src", required=True)
c.add_argument("--out", required=True)
t = sub.add_parser("tasks")
t.add_argument("--bundle", required=True)
t.add_argument("--work", required=True)
t.add_argument("--min", type=int, default=3)
t.add_argument("--per-run", type=int, default=8)
a = sub.add_parser("apply")
a.add_argument("--bundle", required=True)
a.add_argument("--bodies", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "build":
n = assemble(Path(args.work), Path(args.kilde_root), Path(args.out), args.at)
print(f"K: {n} pages -> {args.out}")
elif args.command == "copy":
copy(Path(args.src), Path(args.out))
print(f"copied {args.src} -> {args.out}")
elif args.command == "tasks":
for r in consolidation_tasks(Path(args.bundle), Path(args.work), args.min, args.per_run):
print(f"task: {r.task} output: {r.output} pages: {len(r.pages)}")
else:
applied, refused = apply(Path(args.bundle), Path(args.bodies))
print(f"applied {applied}, refused {refused}")
except (RunError, OSError, KeyError, ValueError) as exc:
print(f"assemble cannot run: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())

329
evals/p1/cost.py Normal file
View file

@ -0,0 +1,329 @@
"""The trial's cost from the run sessions' transcripts (TRIAL, P1): harvest, allocation, projection, stop. No model.
harvest --project-dir D --okter FILE --blocks F --total N [--packs FILE] --out FILE
Reads only the sessions listed in okter.txt ("<session-id> <step>" per line): each session's main
transcript (orchestration, reported apart) and every subagent transcript under <session>/subagents/. Per
subagent: input, cache read, cache write and output tokens kept apart (each message id counted once, at
its largest value, because a streamed message repeats its id with growing output), turns (distinct
message ids), startup (the first message's input side), agentType and description from its .meta.json.
A message whose final usage was never written (no stop reason on any record) gets its output from the
growth of the next message's input side less the injected text counted high (a lower bound); the
recorded output stays beside it (output_recorded).
A description that starts "P1-<variant>-<unit>" attributes the run; any other is listed as unattributed,
never dropped. Per variant: totals, tokens per source (a pack's cost split by block share), the means of
sources 1-5 and 15-19, and the projection to all sources (N blocks). The file is rewritten from the
transcripts on every harvest; stop rows already in it are kept.
stop VARIANT --kost FILE --blocks F [--pages-total N --pages-first M] [--record]
The stop rule (cap 1 M tokens, all four categories): K after pack 1, projected per block processed and,
beside it, naively per source; K+ after run 1, projected over the pages to consolidate. Exit 1 = stop.
With --record the stop row is written into the kost file.
Projection: cost per block c = a + b*R by least squares over the measured units, R = blocks read before the unit;
the projected total for T blocks is a*T + b*T*T/2 (the sum as R grows from 0 to T). One unit: b = 0. The flat
projection (mean cost per block times T) stands beside it. Exit 0 = done, 1 = stop (stop only), 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections.abc import Mapping
from pathlib import Path
from typing import Any
from direction_probe import USAGE_FIELDS, read_transcript
SCHEMA = "okf-wiki/p1-cost/1"
CAP = 1_000_000
CATEGORIES = {"input_tokens": "input", "cache_read_input_tokens": "cache_read",
"cache_creation_input_tokens": "cache_write", "output_tokens": "output"}
PREFIX = re.compile(r"^P1-([A-Za-z0-9+]+)-(\S+)")
PACK_UNIT = re.compile(r"^pack(\d+)")
CLASS = {"LOAD": "load", "K": "write", "K+": "write"}
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
def sessions(okter: Path) -> list[str]:
ids: list[str] = []
for line in okter.read_text(encoding="utf-8").splitlines():
parts = line.split()
if parts and parts[0] not in ids:
ids.append(parts[0])
return ids
INPUT_SIDE = ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")
CHARS_PER_TOKEN_LOW = 2.5 # injected text counted high, so a derived output is a lower bound
def _result_chars(message: dict[str, Any]) -> int:
content = message.get("content")
if not isinstance(content, list):
return 0
total = 0
for item in content:
if not (isinstance(item, dict) and item.get("type") == "tool_result"):
continue
inner = item.get("content")
if isinstance(inner, str):
total += len(inner)
elif isinstance(inner, list):
total += sum(len(str(x.get("text", ""))) for x in inner if isinstance(x, dict))
return total
def usage(path: Path) -> dict[str, Any]:
"""Tokens of one transcript. A message whose records all lack a stop reason never got its final usage written
(seen for a long streamed message: output 2); when another message follows, its output is the growth of the
next message's input side less the text injected between them (tool results and rendered attachments, counted
high at CHARS_PER_TOKEN_LOW), a lower bound, and never below what was recorded."""
per: dict[str, dict[str, int]] = {}
order: list[str] = []
complete: set[str] = set()
between: dict[str, int] = {}
last_stop: str | None = None
for line in read_transcript(path):
message = line.get("message")
if line.get("type") == "attachment" and order:
between[order[-1]] = between.get(order[-1], 0) + len(str(line.get("rendered") or ""))
continue
if not isinstance(message, dict):
continue
if line.get("type") == "user" and order:
between[order[-1]] = between.get(order[-1], 0) + _result_chars(message)
continue
if line.get("type") != "assistant":
continue
mid = str(message.get("id", ""))
if mid not in per:
order.append(mid)
seen = per.setdefault(mid, {})
fields = message.get("usage") or {}
for f in USAGE_FIELDS:
value = fields.get(f)
if isinstance(value, int):
seen[f] = max(seen.get(f, 0), value)
stop = message.get("stop_reason")
last_stop = stop if isinstance(stop, str) else None
if isinstance(stop, str):
complete.add(mid)
out: dict[str, Any] = {CATEGORIES[f]: sum(v.get(f, 0) for v in per.values()) for f in USAGE_FIELDS}
out["output_recorded"] = out["output"]
derived = unrecorded = 0
for i, mid in enumerate(order):
if mid in complete:
continue
if i + 1 == len(order):
unrecorded += 1
continue
growth = (sum(per[order[i + 1]].get(f, 0) for f in INPUT_SIDE) - sum(per[mid].get(f, 0) for f in INPUT_SIDE)
- int(between.get(mid, 0) / CHARS_PER_TOKEN_LOW))
recorded = per[mid].get("output_tokens", 0)
if growth > recorded:
out["output"] += growth - recorded
derived += 1
out["output_derived_messages"] = derived
out["output_unrecorded"] = unrecorded
out["total"] = sum(out[c] for c in CATEGORIES.values())
out["turns"] = len(order)
first = per[order[0]] if order else {}
out["startup"] = sum(first.get(f, 0) for f in INPUT_SIDE)
out["stop_reason"] = last_stop
return out
def harvest(project_dir: Path, ids: list[str]) -> dict[str, Any]:
runs: list[dict[str, Any]] = []
unattributed: list[dict[str, Any]] = []
main: dict[str, Any] = {}
for sid in ids:
top = project_dir / f"{sid}.jsonl"
if top.is_file():
main[sid] = usage(top)
for path in sorted((project_dir / sid / "subagents").glob("agent-*.jsonl")):
meta_path = path.with_name(path.stem + ".meta.json")
meta = json.loads(meta_path.read_text(encoding="utf-8")) if meta_path.is_file() else {}
row = {"session": sid, "agent": path.stem, "agentType": meta.get("agentType"),
"description": str(meta.get("description", "")), **usage(path)}
m = PREFIX.match(row["description"])
if m is None:
unattributed.append(row)
continue
row["variant"], row["unit"] = m.group(1), m.group(2)
row["class"] = CLASS.get(m.group(1), "other")
runs.append(row)
return {"sessions_read": ids, "runs": runs, "unattributed": unattributed, "main": main}
def allocate(pack_costs: Mapping[int, float], packs: list[list[str]], blocks_doc: dict[str, Any]) -> dict[str, float]:
"""Tokens per source: each pack's cost split over its sources by their share of the pack's blocks."""
count: dict[str, int] = {}
for b in blocks_doc["blocks"]:
count[b["file"]] = count.get(b["file"], 0) + 1
per: dict[str, float] = {}
for n, names in enumerate(packs, 1):
if n not in pack_costs:
continue
total = sum(count.get(x, 0) for x in names) or 1
for x in names:
per[x] = pack_costs[n] * count.get(x, 0) / total
return per
def means(per: dict[str, float], order: list[str]) -> dict[str, float | None]:
def mean(names: list[str]) -> float | None:
vals = [per[x] for x in names if x in per]
return sum(vals) / len(vals) if vals else None
return {"1-5": mean(order[0:5]), "15-19": mean(order[14:19])}
def project(points: list[tuple[float, int, int]], total_blocks: int) -> dict[str, Any]:
"""points: (cost, blocks in the unit, blocks read before it)."""
if not points:
return {"fit": None, "flat": None, "T": total_blocks}
xs = [float(r) for _, _, r in points]
ys = [c / b for c, b, _ in points]
fit: dict[str, Any]
if len(points) == 1 or len(set(xs)) == 1:
a, b = sum(ys) / len(ys), 0.0
fit = {"a": a, "b": 0, "note": "one point, no growth term"}
else:
mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
a = my - b * mx
fit = {"a": a, "b": b, "note": f"{len(points)} points, not validated"}
fit["value"] = a * total_blocks + b * total_blocks * total_blocks / 2
flat = sum(c for c, _, _ in points) / sum(bl for _, bl, _ in points) * total_blocks
return {"fit": fit, "flat": flat, "T": total_blocks}
def stop_k(pack1_cost: int, pack1_blocks: int, cut_blocks: int, pack1_sources: int, cut_sources: int
) -> dict[str, Any]:
per_block = pack1_cost * cut_blocks / pack1_blocks
per_source = pack1_cost * cut_sources / pack1_sources
return {"per_block": per_block, "per_source": per_source, "cap": CAP, "stop": per_block > CAP}
def stop_kplus(run1_cost: int, run1_pages: int, pages_total: int) -> dict[str, Any]:
projected = run1_cost * pages_total / run1_pages
return {"projected": projected, "cap": CAP, "stop": projected > CAP}
def _pack_number(unit: str) -> int | None:
m = PACK_UNIT.match(unit)
return int(m.group(1)) if m else None
def summarize(h: dict[str, Any], blocks_doc: dict[str, Any], packs_log: dict[str, Any] | None,
total_blocks: int) -> dict[str, Any]:
order = [str(x) for x in blocks_doc["order"]]
count: dict[str, int] = {}
for b in blocks_doc["blocks"]:
count[b["file"]] = count.get(b["file"], 0) + 1
variants: dict[str, Any] = {}
for v in sorted({r["variant"] for r in h["runs"]}):
mine = [r for r in h["runs"] if r["variant"] == v]
row: dict[str, Any] = {c: sum(r[c] for r in mine) for c in (*CATEGORIES.values(), "total", "turns", "startup")}
row["subagents"] = len(mine)
row["class"] = mine[0]["class"]
row["counts_toward_cap"] = v in ("K", "K+")
if v == "K" and packs_log:
packs = [list(p["sources"]) for p in sorted(packs_log["packs"], key=lambda p: int(p["pack"]))]
pack_costs: dict[int, float] = {}
per_cat: dict[str, dict[int, float]] = {c: {} for c in CATEGORIES.values()}
for r in mine:
n = _pack_number(r["unit"])
if n is not None:
pack_costs[n] = pack_costs.get(n, 0) + r["total"]
for c in CATEGORIES.values():
per_cat[c][n] = per_cat[c].get(n, 0) + r[c]
row["per_pack"] = {str(n): c for n, c in sorted(pack_costs.items())}
row["per_source"] = allocate(pack_costs, packs, blocks_doc)
row["per_source_by_category"] = {c: allocate(per_cat[c], packs, blocks_doc) for c in CATEGORIES.values()}
row["means"] = means(row["per_source"], order)
points, read = [], 0
for n, names in enumerate(packs, 1):
blocks_n = sum(count.get(x, 0) for x in names)
if n in pack_costs:
points.append((pack_costs[n], blocks_n, read))
read += blocks_n
row["projection"] = project(points, total_blocks)
variants[v] = row
return variants
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
h = sub.add_parser("harvest")
h.add_argument("--project-dir", required=True)
h.add_argument("--okter", required=True)
h.add_argument("--blocks", required=True)
h.add_argument("--total", type=int, required=True)
h.add_argument("--packs")
h.add_argument("--out", required=True)
s = sub.add_parser("stop")
s.add_argument("variant", choices=("K", "K+"))
s.add_argument("--kost", required=True)
s.add_argument("--blocks", required=True)
s.add_argument("--packs")
s.add_argument("--pages-total", type=int)
s.add_argument("--pages-first", type=int)
s.add_argument("--record", action="store_true")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
if args.command == "harvest":
out = Path(args.out)
old = json.loads(out.read_text(encoding="utf-8")) if out.is_file() else {}
h = harvest(Path(args.project_dir), sessions(Path(args.okter)))
packs_log = json.loads(Path(args.packs).read_text(encoding="utf-8")) if args.packs else None
doc = {"schema": SCHEMA, **h, "variants": summarize(h, blocks_doc, packs_log, args.total),
"stops": old.get("stops", {})}
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
print(f"harvest: {len(h['runs'])} attributed runs, {len(h['unattributed'])} unattributed, sessions read: "
f"{', '.join(h['sessions_read'])} -> {out}")
return 0
kost_path = Path(args.kost)
kost = json.loads(kost_path.read_text(encoding="utf-8"))
runs = [r for r in kost["runs"] if r["variant"] == args.variant]
if args.variant == "K":
first = [r for r in runs if _pack_number(r["unit"]) == 1]
if not first:
raise RunError("no pack 1 run in the kost file; harvest after pack 1")
if not args.packs:
raise RunError("--packs is needed for K")
log = json.loads(Path(args.packs).read_text(encoding="utf-8"))
p1 = next(p for p in log["packs"] if int(p["pack"]) == 1)
row = stop_k(sum(r["total"] for r in first), int(p1["blocks"]), len(blocks_doc["blocks"]),
len(p1["sources"]), len(blocks_doc["order"]))
else:
first = [r for r in runs if r["unit"].startswith("run1")]
if not first or not args.pages_total or not args.pages_first:
raise RunError("K+ needs a run 1 in the kost file, --pages-total and --pages-first")
row = stop_kplus(sum(r["total"] for r in first), args.pages_first, args.pages_total)
print(f"{args.variant}: projected {row.get('per_block', row.get('projected'))} tokens"
+ (f" (naive per source {row['per_source']})" if "per_source" in row else "")
+ f", cap {CAP}: {'STOP' if row['stop'] else 'continue'}")
if args.record:
kost.setdefault("stops", {})[args.variant] = row
kost_path.write_text(json.dumps(kost, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
return 1 if row["stop"] else 0
except (RunError, OSError, json.JSONDecodeError, KeyError, StopIteration) as exc:
print(f"cost cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

178
evals/p1/dmap.py Normal file
View file

@ -0,0 +1,178 @@
"""D, the trial's no-model baseline (TRIAL, P1): a generated map and a code-built concept register. It calls no model.
build --blocks FILE --kilde-root DIR --out DIR [--at ISO]
One map page per source (k-<slug>.md) whose body mirrors the source's headings, with one sources[] entry per
section; one register page per kept term (r-<slug>.md) whose body is a pure link list to the map sections
where the term occurs, with one sources[] entry per occurrence. A term is kept when it is a docs-page heading
that passes concept test 2, or when it occurs in at least 2 of the documents. Bodies hold only headings and
pure link lists, so the claim count is 0 by construction; the exact locators live in sources[].
--kilde-root is the folder that contains kilde/ (the auditor's SOURCES argument). Exit 0 = written, 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import yaml
from p1.split import headings_of, is_meta, occurs, terms
MIN_DOCUMENTS = 2
GENERATOR = "okf-wiki/p1-dmap"
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
@dataclass
class DStats:
maps: int
registers: int
def slug(text: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
def frontmatter(data: dict[str, Any]) -> str:
"""A frontmatter block; PyYAML quotes any string that would load as another type (no, 1.10)."""
return "---\n" + yaml.safe_dump(data, sort_keys=False, allow_unicode=True, width=10_000) + "---\n"
def _anchors(headings: list[tuple[int, int, str]]) -> dict[int, str]:
seen: dict[str, int] = {}
out: dict[int, str] = {}
for line, _level, text in headings:
base = slug(text)
n = seen.get(base, 0)
seen[base] = n + 1
out[line] = base if n == 0 else f"{base}-{n}"
return out
def _title(name: str, headings: list[tuple[int, int, str]]) -> str:
first = next((t for _, level, t in headings if level == 1), None)
if first:
return first
return f"Release notes {headings[0][2]}" if name.startswith("release-") and headings else name
def build(blocks_doc: dict[str, Any], kilde_root: Path, out: Path, at: str | None = None) -> DStats:
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
names = [str(n) for n in blocks_doc["order"]]
kilde = kilde_root / "kilde"
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
lines = {n: t.split("\n") for n, t in texts.items()}
heads = {n: headings_of(lines[n]) for n in names}
anchors = {n: _anchors(heads[n]) for n in names}
titles = {n: _title(n, heads[n]) for n in names}
pages: dict[str, str] = {}
gen = {"by": GENERATOR, "at": at}
for n in names:
sources = []
body = [f"# {titles[n]}", ""]
for i, (line, level, text) in enumerate(heads[n], 1):
sources.append({"id": f"s{i}", "resource": f"/kilde/{n}", "title": text, "x-locator": {"line": line},
"x-quote": text})
if level > 1 or text != titles[n]:
body += [f"{'#' * max(level, 2)} {text}", ""]
fm = {"type": "Document map", "title": titles[n], "description": f"The sections of {n}.", "sources": sources,
"generated": gen}
pages[f"k-{slug(n.removesuffix('.md'))}.md"] = frontmatter(fm) + "\n" + "\n".join(body).rstrip() + "\n"
candidates = [t for n in names if n.startswith("docs-") for _, level, t in heads[n] if level in (2, 3)]
from_headings = {t for t in candidates if not is_meta(t)}
pool = list(dict.fromkeys(list(from_headings) + [t for n in names for t in terms(texts[n])]))
registers: dict[str, str] = {}
for term in pool:
if is_meta(term):
continue
holders = [n for n in names if occurs(term, texts[n])]
if term not in from_headings and len(holders) < MIN_DOCUMENTS:
continue
pattern = re.compile(r"(?<!\w)" + re.escape(term) + r"(?!\w)", re.IGNORECASE)
sources, links, seen = [], [], set()
for n in holders:
for i, raw in enumerate(lines[n], 1):
m = pattern.search(raw)
if m is None:
continue
section = max((h for h in heads[n] if h[0] <= i), default=None)
key = (n, section[0] if section else 0)
if key in seen:
continue
seen.add(key)
sources.append({"id": f"o{len(sources) + 1}", "resource": f"/kilde/{n}", "title": titles[n],
"x-locator": {"line": i}, "x-quote": m.group(0)})
target = f"k-{slug(n.removesuffix('.md'))}.md"
if section is not None:
target += f"#{anchors[n][section[0]]}"
where = titles[n] + (f" › {section[2]}" if section and section[2] != titles[n] else "")
links.append(f"- [{where.replace('[', '(').replace(']', ')')}]({target})")
if not sources:
continue
name = f"r-{slug(term)}.md"
k = 2
while name in registers:
name, k = f"r-{slug(term)}-{k}.md", k + 1
fm = {"type": "Register entry", "title": term, "description": f"Where {term} occurs in the sources.",
"sources": sources, "generated": gen}
registers[name] = frontmatter(fm) + "\n" + f"# {term}\n\n" + "\n".join(links) + "\n"
out.mkdir(parents=True)
for name, text in {**pages, **registers}.items():
(out / name).write_text(text, encoding="utf-8")
index = ["---", 'okf_version: "0.2"', "---", "", "# Document maps", ""]
index += [f"- [{titles[n]}](k-{slug(n.removesuffix('.md'))}.md)" for n in names]
index += ["", "# Register", ""]
index += [f"- [{name[2:-3]}]({name})" for name in sorted(registers)]
(out / "index.md").write_text("\n".join(index) + "\n", encoding="utf-8")
day = at[:10]
(out / "log.md").write_text(
f"# Log\n\n## {day}\n"
f"* **Build**: D by {GENERATOR}, {len(pages)} map pages and {len(registers)} register pages from {len(names)} "
f"sources. Thresholds, recorded before any count: a term is kept when it is a docs-page heading of level 2 or "
f"3 that passes concept test 2, or when it occurs in at least {MIN_DOCUMENTS} of the {len(names)} documents "
f"(case-insensitive, whole phrase). Terms: backticked spans of 3-40 characters and capitalised phrases of 2-4 "
f"words, a sentence-initial word dropped.\n",
encoding="utf-8")
return DStats(len(pages), len(registers))
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
b = sub.add_parser("build")
b.add_argument("--blocks", required=True)
b.add_argument("--kilde-root", required=True)
b.add_argument("--out", required=True)
b.add_argument("--at")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
s = build(doc, Path(args.kilde_root), Path(args.out), args.at)
except (RunError, OSError, json.JSONDecodeError) as exc:
print(f"dmap cannot run: {exc}", file=sys.stderr)
return 2
print(f"D: {s.maps} map pages, {s.registers} register pages -> {args.out}")
return 0
if __name__ == "__main__":
sys.exit(main())

374
evals/p1/kpack.py Normal file
View file

@ -0,0 +1,374 @@
"""K, the fragment compiler's code side (TRIAL, P1): skeleton, packs, input parts, validation, identity. No model.
skeleton --kilde DIR --out FILE
The concept index from the titles and level 2-3 headings of every docs page under DIR that pass concept
test 2. One entry per normalised name (the first page in name order wins); an index, not pages.
pack N --blocks FILE --index FILE --kilde-root DIR --work DIR [--size 5]
Pack N's input for the compiler: every block of its sources with its anchor (file › section › span) and
numbered lines, then the index lines whose name occurs in those blocks, then the output contract, in parts
of at most PART_CHARS characters (the Read tool's cap). Prints the part paths and the output path.
accept N --work DIR --blocks FILE --kilde-root DIR
Validates pack N's fragments, resolves each to a concept by code (existing, merge candidate, new or
rejected), updates the index and appends the accepted fragments. Prints "accepted X of N" and the
rejections by reason, and how many accepted fragments resolved to a concept whose index line was not
sent with the pack (the pack log keeps the ids sent; the risk of the D7 phrase filter).
The work directory holds K-oppgaver/ (task parts), K-fragments/ (the compiler's JSON) and K-work/ (index.json,
accepted.json, packs.json). Exit 0 = done, 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import Counter
from pathlib import Path
from typing import Any
from okfwiki.audit import sentence_spans
from p1.split import docs_headings, is_meta
PART_CHARS = 60_000
MERGE_JACCARD = 0.6
NEEDS_QUOTE = re.compile(r"\d|`")
REASONS = ("malformed", "anchor-outside-pack", "anchor-crosses-blocks", "sentence-count", "quote-missing",
"quote-not-in-span", "tests-missing", "release-note-new", "meta")
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
def norm(name: str) -> str:
text = name.replace("`", "").lower()
text = re.sub(r"[^\w\s/.\-]", " ", text)
words = text.split()
if words and words[0] == "the":
words = words[1:]
words = [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
return " ".join(w.strip(".") or w for w in words)
def slug(text: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
def _unique_id(index: dict[str, Any], title: str) -> str:
taken = {e["id"] for e in index["entries"]}
base = slug(title)
eid, k = base, 2
while eid in taken:
eid, k = f"{base}-{k}", k + 1
return eid
def skeleton(kilde: Path) -> dict[str, Any]:
index: dict[str, Any] = {"schema": "okf-wiki/p1-index/1", "entries": []}
seen: dict[str, dict[str, Any]] = {}
for h in docs_headings(kilde):
if is_meta(h.text):
continue
n = norm(h.text)
if not n:
continue
if n in seen:
if h.text not in seen[n]["aliases"]:
seen[n]["aliases"].append(h.text)
continue
e = {"id": _unique_id(index, h.text), "title": h.text, "aliases": [h.text], "kind": "skeleton",
"merge_of": None, "origin": {"file": h.file, "line": h.line}}
index["entries"].append(e)
seen[n] = e
return index
def _names(e: dict[str, Any]) -> set[str]:
return {norm(a) for a in [e["title"], *e.get("aliases", [])]} - {""}
def _jaccard(a: str, b: str) -> float:
x, y = set(a.split()), set(b.split())
return len(x & y) / len(x | y) if x | y else 0.0
def resolve(fragment: dict[str, Any], index: dict[str, Any]) -> tuple[str, str | None]:
"""existing <id> | merge-candidate <id> | new | rejected <reason> (D6, D3b)."""
by_id = {e["id"]: e for e in index["entries"]}
if fragment.get("status") == "existing" and fragment.get("id") in by_id:
return "existing", str(fragment["id"])
n = norm(str(fragment.get("concept", "")))
for e in index["entries"]:
if n in _names(e):
return "existing", str(e["id"])
if str(fragment.get("anchor", {}).get("file", "")).startswith("release-"):
return "rejected", "release-note-new"
if is_meta(str(fragment.get("concept", ""))):
return "rejected", "meta"
best = max(((max(_jaccard(n, m) for m in _names(e)), str(e["id"])) for e in index["entries"] if _names(e)),
default=(0.0, ""))
if best[0] >= MERGE_JACCARD:
return "merge-candidate", best[1]
return "new", None
def relevant_lines(index: dict[str, Any], text: str) -> list[dict[str, Any]]:
"""Index entries whose normalised name or alias occurs as a phrase in the normalised text (D7)."""
hay = f" {norm(text)} "
return [e for e in index["entries"] if any(f" {m} " in hay for m in _names(e))]
def packs(names: list[str], size: int = 5) -> list[list[str]]:
return [names[i:i + size] for i in range(0, len(names), size)]
CONTRACT = """## Output contract
Write one JSON object `{"fragments": [...]}` to the output path named in your prompt. Each fragment:
`concept`, `status` ("existing" with the index `id`, or "new" with `tests`: {"topic_shape", "citation"}),
`text` (1-3 sentences), `anchor` {"file", "start", "end"} inside one block above, and `quote` (one piece copied from
one line inside the anchor; required when the text holds a digit or a backticked name). A new concept only from a docs
page, never from a release note.
"""
def render_input(pack_no: int, names: list[str], blocks_doc: dict[str, Any], index_lines: list[dict[str, Any]],
kilde: Path) -> list[str]:
lines = {n: (kilde / n).read_text(encoding="utf-8").split("\n") for n in names}
pieces: list[str] = [f"# K pack {pack_no}: {len(names)} sources\n\nSources in order: {', '.join(names)}\n\n"
+ CONTRACT + "\n## Blocks\n\n"]
for b in blocks_doc["blocks"]:
if b["file"] not in names:
continue
head = f"### {b['file']} › {b['section'] or '-'} › {b['start']}–{b['end']}"
if b.get("version"):
head += f" (version {b['version']})"
body = [f"{n}: {lines[b['file']][n - 1]}" for n in range(b["start"], b["end"] + 1)]
chunk = head + "\n"
for row in body:
if len(chunk) + len(row) + 1 > PART_CHARS // 2:
pieces.append(chunk + "\n")
chunk = head + " (continued)\n"
chunk += row + "\n"
pieces.append(chunk + "\n")
pieces.append("## Index (concepts named in these blocks)\n\n")
for e in index_lines:
aliases = "; ".join(a for a in e["aliases"] if a != e["title"])
pieces.append(f"- id: {e['id']} | {e['title']}" + (f" | aliases: {aliases}" if aliases else "") + "\n")
room = PART_CHARS - len("<!-- part 99 of 99 -->\n")
parts: list[str] = []
current = ""
for piece in pieces:
if current and len(current) + len(piece) > room:
parts.append(current)
current = ""
current += piece
if current:
parts.append(current)
total = len(parts)
return [f"<!-- part {i} of {total} -->\n" + p for i, p in enumerate(parts, 1)]
def validate(fragments: list[Any], names: list[str], blocks_doc: dict[str, Any], kilde: Path
) -> tuple[list[dict[str, Any]], Counter[str]]:
by_file: dict[str, list[dict[str, Any]]] = {}
for b in blocks_doc["blocks"]:
by_file.setdefault(b["file"], []).append(b)
texts: dict[str, list[str]] = {}
accepted: list[dict[str, Any]] = []
rejected: Counter[str] = Counter()
for f in fragments:
a = f.get("anchor") if isinstance(f, dict) else None
if not (isinstance(f, dict) and isinstance(f.get("concept"), str) and f["concept"].strip()
and f.get("status") in ("new", "existing") and isinstance(f.get("text"), str)
and isinstance(a, dict) and isinstance(a.get("file"), str)
and type(a.get("start")) is int and type(a.get("end")) is int and a["start"] <= a["end"]):
rejected["malformed"] += 1
continue
if a["file"] not in names:
rejected["anchor-outside-pack"] += 1
continue
block = next((b for b in by_file.get(a["file"], []) if b["start"] <= a["start"] and a["end"] <= b["end"]), None)
if block is None:
rejected["anchor-crosses-blocks"] += 1
continue
if not 1 <= len(sentence_spans(f["text"].strip())) <= 3:
rejected["sentence-count"] += 1
continue
if a["file"] not in texts:
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
span = texts[a["file"]][a["start"] - 1:a["end"]]
quote = f.get("quote")
if not (isinstance(quote, str) and quote.strip()):
if NEEDS_QUOTE.search(f["text"]):
rejected["quote-missing"] += 1
continue
quote, quote_line = None, None
else:
hit = next((a["start"] + i for i, line in enumerate(span) if quote in line), None)
if hit is None:
rejected["quote-not-in-span"] += 1
continue
quote_line = hit
tests = f.get("tests")
if f["status"] == "new" and not (isinstance(tests, dict) and str(tests.get("topic_shape", "")).strip()
and str(tests.get("citation", "")).strip()):
rejected["tests-missing"] += 1
continue
accepted.append({**f, "quote": quote, "quote_line": quote_line, "version": block.get("version"),
"section": block["section"], "position": names.index(a["file"]) + 1})
return accepted, rejected
def _read_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise RunError(f"cannot read {path}: {exc}") from exc
def _write_json(path: Path, data: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
def _pack_log(work: Path) -> dict[str, Any]:
path = work / "K-work" / "packs.json"
return _read_json(path) if path.is_file() else {"schema": "okf-wiki/p1-packs/1", "packs": []}
def _record(log: dict[str, Any], n: int) -> dict[str, Any]:
for r in log["packs"]:
if r["pack"] == n:
return r # type: ignore[no-any-return]
r = {"pack": n}
log["packs"].append(r)
return r
def cmd_pack(args: argparse.Namespace) -> int:
work, n = Path(args.work), args.n
doc = _read_json(Path(args.blocks))
all_packs = packs([str(x) for x in doc["order"]], args.size)
if not 1 <= n <= len(all_packs):
raise RunError(f"pack {n} does not exist; there are {len(all_packs)}")
names = all_packs[n - 1]
kilde = Path(args.kilde_root) / "kilde"
index = _read_json(Path(args.index))
text = "\n".join((kilde / name).read_text(encoding="utf-8") for name in names)
lines = relevant_lines(index, text)
parts = render_input(n, names, doc, lines, kilde)
out_dir = work / "K-oppgaver"
paths = [out_dir / f"pack-{n}-part-{i}.md" for i in range(1, len(parts) + 1)]
if any(p.exists() for p in paths):
raise RunError(f"pack {n}'s task parts exist in {out_dir}; refusing to overwrite")
out_dir.mkdir(parents=True, exist_ok=True)
for p, text_part in zip(paths, parts):
p.write_text(text_part, encoding="utf-8")
log = _pack_log(work)
rec = _record(log, n)
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]), index_lines=len(lines),
index_total=len(index["entries"]), index_ids=[str(e["id"]) for e in lines], parts=len(parts),
chars=sum(len(p) for p in parts))
_write_json(work / "K-work" / "packs.json", log)
for p in paths:
print(f"part: {p}")
print(f"output: {work / 'K-fragments' / f'pack-{n}.json'}")
print(f"index lines sent: {len(lines)} of {len(index['entries'])}")
return 0
def cmd_accept(args: argparse.Namespace) -> int:
work, n = Path(args.work), args.n
doc = _read_json(Path(args.blocks))
names = packs([str(x) for x in doc["order"]], args.size)[n - 1]
kilde = Path(args.kilde_root) / "kilde"
raw = _read_json(work / "K-fragments" / f"pack-{n}.json")
fragments = raw.get("fragments") if isinstance(raw, dict) else None
if not isinstance(fragments, list):
raise RunError(f"pack {n}'s output has no fragments list")
index_path = work / "K-work" / "index.json"
index = _read_json(index_path)
valid, rejected = validate(fragments, names, doc, kilde)
acc_path = work / "K-work" / "accepted.json"
acc: dict[str, Any] = (_read_json(acc_path) if acc_path.is_file()
else {"schema": "okf-wiki/p1-accepted/1", "fragments": []})
if any(a["pack"] == n for a in acc["fragments"]):
raise RunError(f"pack {n} is already accepted")
sent = set(_record(_pack_log(work), n).get("index_ids", []))
not_sent: Counter[str] = Counter()
accepted = 0
for f in valid:
outcome, ref = resolve(f, index)
if outcome == "rejected":
rejected[str(ref)] += 1
continue
if outcome == "existing":
cid = str(ref)
else:
cid = _unique_id(index, f["concept"])
index["entries"].append({"id": cid, "title": f["concept"], "aliases": [f["concept"]], "kind": outcome,
"merge_of": ref, "origin": {"file": f["anchor"]["file"], "line": f["anchor"]["start"]},
"tests": f.get("tests")})
acc["fragments"].append({"pack": n, "position": f["position"], "file": f["anchor"]["file"],
"start": f["anchor"]["start"], "end": f["anchor"]["end"], "section": f["section"],
"text": f["text"].strip(), "quote": f["quote"], "quote_line": f["quote_line"],
"version": f["version"], "concept_id": cid, "outcome": outcome})
if cid not in sent:
not_sent[outcome] += 1
accepted += 1
_write_json(index_path, index)
_write_json(acc_path, acc)
log = _pack_log(work)
rec = _record(log, n)
rec.update(proposed=len(fragments), accepted=accepted, rejected=dict(sorted(rejected.items())),
not_sent={"total": sum(not_sent.values()), "by_outcome": dict(sorted(not_sent.items()))})
_write_json(work / "K-work" / "packs.json", log)
print(f"pack {n}: accepted {accepted} of {len(fragments)}")
print(f" accepted with a concept not among the index lines sent: {sum(not_sent.values())}")
for reason, count in sorted(rejected.items()):
print(f" rejected {reason}: {count}")
return 0
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
s = sub.add_parser("skeleton")
s.add_argument("--kilde", required=True)
s.add_argument("--out", required=True)
for name in ("pack", "accept"):
c = sub.add_parser(name)
c.add_argument("n", type=int)
c.add_argument("--blocks", required=True)
c.add_argument("--kilde-root", required=True)
c.add_argument("--work", required=True)
c.add_argument("--size", type=int, default=5)
if name == "pack":
c.add_argument("--index", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "skeleton":
out = Path(args.out)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
index = skeleton(Path(args.kilde))
_write_json(out, index)
print(f"skeleton: {len(index['entries'])} entries -> {out}")
return 0
if args.command == "pack":
return cmd_pack(args)
return cmd_accept(args)
except (RunError, OSError) as exc:
print(f"kpack cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

361
evals/p1/measure.py Normal file
View file

@ -0,0 +1,361 @@
"""The trial's measures without a reader (TRIAL, P1): audit record, coverage, structure. It calls no model.
audit --bundle B --root R --variant V --out FILE
Runs the auditor in process and records, per variant, the exit status, findings per check, uncited
sentences by cause (fence line, table row, link lead-in, prose), "yaml form" findings (a quote that YAML
loaded as another type, a timestamp in the wrong form) and criterion 1 (N of N, claims.py's count).
coverage --bundle B --root R --blocks F --built N [--pack-size K] --out FILE
Blocks of the first N sources in the order with at least one body-cited source entry whose locator line
lies inside the block (spans are not used), X of N blocks; per pack position with --pack-size.
structure --bundle B --root R --blocks F [--skeleton I] --out FILE
Pages, concepts per source (the earliest source in the order among a page's body-cited sources),
skeleton entries without a page, duplicates (same name, alias or title Jaccard >= 0.6), merge
candidates, islands, sources per page, version numbers in running text, concept tests 2 and 4,
the share of quotes on a structural line and of code-supplied quotes (version sources left out), and the
precision of each body-cited entry (line, block, section, document).
--root is the folder that contains kilde/. Before reporting, structure and coverage check that at least one quote
stands at its locator under --root, because a wrong root makes every quote unreadable. Exit 0 = written, 2 = could
not run (an existing output, or a variant already in the audit record, is never overwritten).
"""
from __future__ import annotations
import argparse
import json
import sys
from collections import Counter
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from okfwiki import claims as claims_mod
from okfwiki.audit import (
FOOTNOTE_REF,
LINK,
LINK_ONLY,
Document,
audit,
body_links,
is_bundle_link,
load_bundle,
mask_code,
prose_blocks,
resolve,
sentences,
)
from okfwiki.view import is_version_heading, related, versions
from p1.assemble import structural
from p1.kpack import MERGE_JACCARD, norm
from p1.split import FENCE, is_meta
STRUCTURE_SCHEMA = "okf-wiki/p1-structure/1"
RESULT_SCHEMA = "okf-wiki/p1-result/1"
class RunError(Exception):
"""The measure cannot run as asked; nothing was written."""
@dataclass
class Coverage:
covered: int
total: int
per_position: dict[int, tuple[int, int, int]] = field(default_factory=dict)
def _entries(doc: Document) -> dict[str, dict[str, Any]]:
srcs = (doc.frontmatter or {}).get("sources")
return {str(s["id"]): s for s in srcs if isinstance(s, dict) and "id" in s} if isinstance(srcs, list) else {}
def cited(doc: Document) -> list[dict[str, Any]]:
"""The source entries a body footnote uses, in label order of first use."""
entries = _entries(doc)
labels: list[str] = []
for block in prose_blocks(doc):
for label in FOOTNOTE_REF.findall(mask_code(block.text)):
if label not in labels:
labels.append(label)
return [entries[x] for x in labels if x in entries]
def _file(s: dict[str, Any]) -> str:
return Path(str(s.get("resource", ""))).name
def _line(s: dict[str, Any]) -> int | None:
loc = s.get("x-locator")
return loc["line"] if isinstance(loc, dict) and type(loc.get("line")) is int else None
class _Lines:
def __init__(self, root: Path) -> None:
self.root = root
self.cache: dict[str, list[str] | None] = {}
def get(self, name: str) -> list[str] | None:
if name not in self.cache:
p = self.root / "kilde" / name
self.cache[name] = p.read_text(encoding="utf-8").split("\n") if p.is_file() else None
return self.cache[name]
def _check_root(concepts: list[Document], root: Path) -> None:
lines = _Lines(root)
seen = 0
for d in concepts:
for s in _entries(d).values():
n, quote, text = _line(s), s.get("x-quote"), lines.get(_file(s))
seen += 1
if n and isinstance(quote, str) and text and n <= len(text) and quote in text[n - 1]:
return
if seen:
raise RunError(f"no quote stands at its locator under {root}; is it the folder that contains kilde/?")
def _block_of(blocks: list[dict[str, Any]], file: str, line: int) -> dict[str, Any] | None:
return next((b for b in blocks if b["file"] == file and b["start"] <= line <= b["end"]), None)
def coverage(bundle: Path, root: Path, blocks_doc: dict[str, Any], built: list[str],
pack_size: int | None = None) -> Coverage:
concepts = load_bundle(bundle, root).concepts
_check_root(concepts, root)
blocks = [b for b in blocks_doc["blocks"] if b["file"] in built]
hit: set[tuple[str, int]] = set()
for d in concepts:
for s in cited(d):
n = _line(s)
b = _block_of(blocks, _file(s), n) if n else None
if b is not None:
hit.add((b["file"], b["start"]))
result = Coverage(len(hit), len(blocks))
if pack_size:
names = [str(x) for x in blocks_doc["order"] if x in built]
for pos in range(1, pack_size + 1):
files = {n for i, n in enumerate(names) if i % pack_size == pos - 1}
if files:
mine = [b for b in blocks if b["file"] in files]
result.per_position[pos] = (sum((b["file"], b["start"]) in hit for b in mine), len(mine), len(files))
return result
def _jaccard(a: str, b: str) -> float:
x, y = set(a.split()), set(b.split())
return len(x & y) / len(x | y) if x | y else 0.0
def duplicates(concepts: list[Document]) -> list[tuple[str, str, str]]:
info = []
for d in concepts:
fm = d.frontmatter or {}
aliases = fm.get("x-aliases")
info.append((d.rel, norm(str(fm.get("title", ""))),
{norm(str(a)) for a in aliases} if isinstance(aliases, list) else set()))
pairs: list[tuple[str, str, str]] = []
for i, (a, na, aa) in enumerate(info):
for b, nb, ab in info[i + 1:]:
if na and na == nb:
pairs.append((a, b, "norm"))
elif na in ab or nb in aa:
pairs.append((a, b, "alias"))
elif _jaccard(na, nb) >= MERGE_JACCARD:
pairs.append((a, b, "jaccard"))
return pairs
def _kind(d: Document) -> str:
fm = d.frontmatter or {}
if fm.get("x-merge-candidate"):
return "merge-candidate"
return {"Document map": "map", "Register entry": "register"}.get(str(fm.get("type", "")), "concept")
def _precision(s: dict[str, Any], blocks: list[dict[str, Any]]) -> str:
span = s.get("x-span")
n = _line(s) or 0
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (n, n)
if a == z:
return "line"
inside = [b for b in blocks if b["file"] == _file(s) and b["end"] >= a and b["start"] <= z]
if len(inside) == 1 and inside[0]["start"] <= a and z <= inside[0]["end"]:
return "block"
if inside and len({b["section"] for b in inside}) == 1:
return "section"
return "document"
def structure(bundle: Path, root: Path, blocks_doc: dict[str, Any], order: list[str],
skeleton: dict[str, Any] | None = None) -> dict[str, Any]:
concepts = load_bundle(bundle, root).concepts
_check_root(concepts, root)
lines = _Lines(root)
blocks = list(blocks_doc["blocks"])
rank = {n: i for i, n in enumerate(order)}
per_source: Counter[str] = Counter()
incoming: Counter[str] = Counter()
rels = {d.rel for d in concepts}
for d in concepts:
for _line_no, target in body_links(d):
other = resolve(d.rel, target) if is_bundle_link(target) else None
if other in rels and other != d.rel:
incoming[str(other)] += 1
quotes: Counter[str] = Counter()
precision: Counter[str] = Counter()
docs_per_page: list[int] = []
test2 = test4 = 0
for d in concepts:
cs = cited(d)
files = [_file(s) for s in cs if _file(s) in rank]
if files:
per_source[min(files, key=lambda f: rank[f])] += 1
docs_per_page.append(len(set(_file(s) for s in cs)))
title = str((d.frontmatter or {}).get("title", ""))
test2 += not is_meta(title)
test4 += incoming[d.rel] >= 2 or len(set(files)) >= 2
for s in cs:
if is_version_heading(s):
continue
quotes["entries"] += 1
quotes["code"] += s.get("x-quote-by") == "code"
text, n = lines.get(_file(s)), _line(s)
quotes["structural"] += bool(text and n and n <= len(text) and structural(text[n - 1]))
precision[_precision(s, blocks)] += 1
rel_map = related(concepts)
kinds = {d.rel: _kind(d) for d in concepts}
islands: dict[str, Any] = {"total": sum(not v for v in rel_map.values())}
for k in sorted(set(kinds.values())):
islands[k] = sum(not rel_map[r] for r, kk in kinds.items() if kk == k)
pages = {d.rel for d in concepts}
unused = None
if skeleton is not None:
unused = sum(e.get("kind") == "skeleton" and f"{e['id']}.md" not in pages for e in skeleton["entries"])
tests13 = None
if skeleton is not None:
new_ids = {e["id"] for e in skeleton["entries"] if e.get("kind") == "new" and e.get("tests")}
tests13 = sum(f"{i}.md" in pages for i in new_ids)
v = versions(bundle, root)
dups = duplicates(concepts)
return {
"schema": STRUCTURE_SCHEMA,
"pages": len(concepts),
"page_kinds": dict(Counter(kinds.values())),
"concepts_per_source": {n: per_source[n] for n in order if per_source[n]},
"skeleton_unused": unused,
"duplicates": {"pairs": len(dups), "by_rule": dict(Counter(r for _, _, r in dups)),
"list": [list(p) for p in dups]},
"merge_candidates": sum(k == "merge-candidate" for k in kinds.values()),
"islands": islands,
"documents_per_page": {"mean": round(sum(docs_per_page) / len(docs_per_page), 2) if docs_per_page else 0,
"max": max(docs_per_page, default=0)},
"version_numbers_in_text": {"flagged": len(v.flagged), "claims": v.total},
"concept_tests": {"test2": test2, "test4": test4, "tests1and3_self_declared": tests13},
"quotes": {"entries": quotes["entries"], "structural": quotes["structural"], "code": quotes["code"]},
"precision": {k: precision[k] for k in ("line", "block", "section", "document")},
}
def _uncited_cause(sentence: str, block_text: str) -> str:
s = sentence.strip()
if FENCE.match(block_text): # the auditor's sentence text masks code spans, so a fence is read on the block
return "fence line"
if s.startswith("|"):
return "table row"
if LINK.search(block_text) and not LINK_ONLY.match(mask_code(block_text)):
return "link lead-in"
return "prose"
def audit_record(bundle: Path, root: Path, variant: str, out: Path) -> dict[str, Any]:
record_all: dict[str, Any] = (json.loads(out.read_text(encoding="utf-8")) if out.is_file()
else {"schema": RESULT_SCHEMA, "variants": {}})
if variant in record_all["variants"]:
raise RunError(f"{out} already records variant {variant}")
b = load_bundle(bundle, root)
found = audit(bundle, root)
uncited: Counter[str] = Counter()
for d in b.concepts:
for block in prose_blocks(d):
if LINK_ONLY.match(mask_code(block.text)):
continue
for s in sentences(block.text):
if not FOOTNOTE_REF.search(s):
uncited[_uncited_cause(s, block.text)] += 1
yaml_form = 0
for d in b.concepts:
for entry in _entries(d).values():
q = entry.get("x-quote")
yaml_form += q is not None and not isinstance(q, str)
yaml_form += sum(f.check == "timestamp-form" for f in found)
listed = claims_mod.extract(bundle, root)
ok, total = claims_mod.criterion_1(listed)["total"]
rec = {"exit": 1 if found else 0, "findings": dict(sorted(Counter(f.check for f in found).items())),
"uncited": dict(sorted(uncited.items())), "yaml_form": yaml_form, "criterion_1": [ok, total]}
record_all["variants"][variant] = rec
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(record_all, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
return rec
def _write_new(path: Path, data: Any) -> None:
if path.exists():
raise RunError(f"{path} exists; refusing to overwrite")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
a = sub.add_parser("audit")
a.add_argument("--bundle", required=True)
a.add_argument("--root", required=True)
a.add_argument("--variant", required=True)
a.add_argument("--out", required=True)
c = sub.add_parser("coverage")
s = sub.add_parser("structure")
for x in (c, s):
x.add_argument("--bundle", required=True)
x.add_argument("--root", required=True)
x.add_argument("--blocks", required=True)
x.add_argument("--out", required=True)
c.add_argument("--built", type=int, required=True)
c.add_argument("--pack-size", type=int)
s.add_argument("--skeleton")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
bundle, root = Path(args.bundle), Path(args.root)
if not bundle.is_dir():
raise RunError(f"the bundle {bundle} is not a directory")
if args.command == "audit":
rec = audit_record(bundle, root, args.variant, Path(args.out))
print(f"{args.variant}: audit exit {rec['exit']}, criterion 1 {rec['criterion_1'][0]} of "
f"{rec['criterion_1'][1]}, findings {rec['findings']}")
return 0
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
order = [str(x) for x in blocks_doc["order"]]
if args.command == "coverage":
cov = coverage(bundle, root, blocks_doc, order[:args.built], args.pack_size)
_write_new(Path(args.out), {"schema": "okf-wiki/p1-coverage/1", "covered": cov.covered, "total": cov.total,
"built": args.built,
"per_position": {str(k): list(v) for k, v in cov.per_position.items()}})
print(f"coverage: {cov.covered} of {cov.total} blocks")
return 0
skeleton = json.loads(Path(args.skeleton).read_text(encoding="utf-8")) if args.skeleton else None
data = structure(bundle, root, blocks_doc, order, skeleton)
_write_new(Path(args.out), data)
print(f"structure: {data['pages']} pages, {data['duplicates']['pairs']} duplicate pairs, "
f"{data['islands']['total']} islands")
return 0
except (RunError, OSError, json.JSONDecodeError, KeyError) as exc:
print(f"measure cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

185
evals/p1/sidebyside.py Normal file
View file

@ -0,0 +1,185 @@
"""The side-by-side page (TRIAL, P1): the same concepts in every build, every source link checked. It calls no model.
render --begreper FILE --col LABEL=DIR ... [--col LABEL=-] --kilde-root DIR --out FILE
One self-contained HTML page: one row per concept in begreper.txt (name, a tab, aliases joined by ;), one
column per build. A cell shows the pages whose normalised title or alias equals the concept's (for the L4
column: whose normalised title or description contains it), each with its body and its sources
(file › line or span). An empty cell says why: "ikke bygget" (a build given as -, stopped), "ikke i
registeret" (D has no register page for it), "ingen side" (built, no page).
check --html FILE --kilde-root DIR
A source link resolves when its file exists under kilde/ and its line or span lies inside the file.
Prints "resolved X of N"; exit 1 when X < N.
Exit 0 = done, 1 = an unresolved link (check), 2 = could not run.
"""
from __future__ import annotations
import argparse
import os
import re
import sys
from pathlib import Path
from typing import Any
from okfwiki.audit import load_bundle
from okfwiki.view import esc
from p1.kpack import norm
from p1.measure import cited
Concept = tuple[str, tuple[str, ...]]
SRC = re.compile(r'<a class="src" href="[^"]*" data-file="([^"]+)" data-start="(\d+)" data-end="(\d+)">')
STYLE = """
:root { --bg: #fdfdfb; --fg: #1d1d1b; --muted: #6b6b66; --line: #ddd9cf; --cell: #ffffff; }
@media (prefers-color-scheme: dark) { :root { --bg: #1b1b1a; --fg: #ecebe6; --muted: #a3a29b; --line: #3a3934;
--cell: #242422; } }
body { background: var(--bg); color: var(--fg); font: 15px/1.5 system-ui, sans-serif; margin: 16px; }
table { border-collapse: collapse; width: 100%; table-layout: fixed; }
th, td { border: 1px solid var(--line); vertical-align: top; padding: 8px; background: var(--cell); }
th.concept-name { width: 10em; text-align: left; }
.empty { color: var(--muted); font-style: italic; }
.page h3 { margin: 0 0 4px; font-size: 15px; }
.sources { color: var(--muted); font-size: 13px; }
.wrap { overflow-x: auto; }
"""
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
def read_begreper(path: Path) -> list[Concept]:
out: list[Concept] = []
for line in path.read_text(encoding="utf-8").splitlines():
if line.strip():
name, _, aliases = line.partition("\t")
out.append((name.strip(), tuple(a.strip() for a in aliases.split(";") if a.strip()) or (name.strip(),)))
return out
def _l4_norm(text: str) -> str:
return f" {norm(text.replace('-', ' '))} "
def find(bundle: Path, concept: Concept, l4: bool = False) -> list[str]:
wanted = {norm(n) for n in (concept[0], *concept[1])} - {""}
found: list[str] = []
for d in load_bundle(bundle, bundle).concepts:
fm = d.frontmatter or {}
if l4:
hay = _l4_norm(str(fm.get("title", ""))) + _l4_norm(str(fm.get("description", "")))
if any(f" {norm(w.replace('-', ' '))} " in hay for w in wanted):
found.append(d.rel)
continue
aliases = fm.get("x-aliases")
names = {norm(str(fm.get("title", "")))} | ({norm(str(a)) for a in aliases} if isinstance(aliases, list)
else set())
if names & wanted:
found.append(d.rel)
return found
def _sources_html(entries: list[dict[str, Any]], kilde: Path, out_dir: Path) -> str:
items = []
for s in entries:
file = Path(str(s.get("resource", ""))).name
loc = s.get("x-locator")
line = loc.get("line") if isinstance(loc, dict) else None
span = s.get("x-span")
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (line, line)
if not isinstance(a, int) or not isinstance(z, int):
continue
href = os.path.relpath(kilde / file, out_dir).replace(os.sep, "/") + f"#L{a}"
where = f"{a}" if a == z else f"{a}–{z}"
items.append(f'<li><a class="src" href="{esc(href)}" data-file="{esc(file)}" data-start="{a}" '
f'data-end="{z}">{esc(file)} › {where}</a></li>')
return '<ul class="sources">' + "".join(items) + "</ul>"
def _cell(bundle: Path | None, label: str, concept: Concept, kilde_root: Path, out_dir: Path) -> str:
if bundle is None:
return '<td class="cell empty">ikke bygget</td>'
rels = find(bundle, concept, l4=label == "L4")
if not rels:
return f'<td class="cell empty">{"ikke i registeret" if label == "D" else "ingen side"}</td>'
docs = {d.rel: d for d in load_bundle(bundle, kilde_root).concepts}
parts = []
for rel in rels:
d = docs[rel]
fm = d.frontmatter or {}
srcs = fm.get("sources")
entries = cited(d) or ([s for s in srcs if isinstance(s, dict)] if isinstance(srcs, list) else [])
paras = [p.strip() for p in re.split(r"\n\s*\n", d.body) if p.strip() and not p.strip().startswith("# ")]
body = "".join(f"<p>{esc(p)}</p>" for p in paras if not p.startswith("[^"))
parts.append(f'<div class="page"><h3>{esc(str(fm.get("title", rel)))} <small>({esc(rel)})</small></h3>'
f"{body}{_sources_html(entries, kilde_root / 'kilde', out_dir)}</div>")
return '<td class="cell">' + "".join(parts) + "</td>"
def render(begreper: list[Concept], columns: list[tuple[str, Path | None]], kilde_root: Path, out: Path) -> None:
out_dir = out.parent
head = "".join(f"<th>{esc(label)}</th>" for label, _ in columns)
rows = []
for concept in begreper:
cells = "".join(_cell(b, label, concept, kilde_root, out_dir) for label, b in columns)
rows.append(f'<tr class="concept"><th class="concept-name">{esc(concept[0])}</th>{cells}</tr>')
html = (f'<!doctype html><html lang="no"><head><meta charset="utf-8">'
f'<meta name="viewport" content="width=device-width, initial-scale=1"><title>P1 side om side</title>'
f"<style>{STYLE}</style></head><body><h1>P1: tre begreper, side om side</h1>"
f'<div class="wrap"><table><tr><th class="concept-name">Begrep</th>{head}</tr>{"".join(rows)}'
f"</table></div></body></html>\n")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(html, encoding="utf-8")
def check(html_path: Path, kilde_root: Path) -> tuple[int, int]:
html = html_path.read_text(encoding="utf-8")
resolved = total = 0
sizes: dict[str, int | None] = {}
for file, a, z in SRC.findall(html):
total += 1
if file not in sizes:
p = kilde_root / "kilde" / file
sizes[file] = len(p.read_text(encoding="utf-8").splitlines()) if p.is_file() else None
n = sizes[file]
resolved += n is not None and 1 <= int(a) <= int(z) <= n
return resolved, total
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
r = sub.add_parser("render")
r.add_argument("--begreper", required=True)
r.add_argument("--col", action="append", required=True, help="LABEL=DIR, or LABEL=- for a stopped build")
r.add_argument("--kilde-root", required=True)
r.add_argument("--out", required=True)
c = sub.add_parser("check")
c.add_argument("--html", required=True)
c.add_argument("--kilde-root", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "render":
columns: list[tuple[str, Path | None]] = []
for col in args.col:
label, _, path = col.partition("=")
if not label or not path:
raise RunError(f"--col {col!r} is not LABEL=DIR")
columns.append((label, None if path == "-" else Path(path)))
render(read_begreper(Path(args.begreper)), columns, Path(args.kilde_root), Path(args.out))
print(f"side by side: {len(columns)} columns -> {args.out}")
return 0
resolved, total = check(Path(args.html), Path(args.kilde_root))
print(f"resolved {resolved} of {total}")
return 0 if total and resolved == total else 1
except (RunError, OSError) as exc:
print(f"sidebyside cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

328
evals/p1/split.py Normal file
View file

@ -0,0 +1,328 @@
"""Split the trial's sources into a fixed order and into blocks with line spans (TRIAL, P1). It calls no model.
order --manifest M --l4 L4DIR --out FILE
The 19 names of the cut (gates/wholedoc.manifest.json cut.names) in the trial's fixed order: docs pages by
name, then release notes by version. Refuses when the names differ from the files the L4 cut cites.
blocks --kilde DIR --order FILE --out FILE
Every block of every source in the order: a paragraph, one list item with its continuation lines, one
table or one fenced code block, with its 1-based inclusive line span, its section (the heading path) and,
for a release note, its version. Heading lines are section markers, not blocks.
total --kilde DIR
The number of blocks in every file under DIR (the denominator of the 599-source projection).
pick --blocks FILE --kilde DIR [--focus NAME] --out FILE
The three concepts of the side-by-side page, picked from the source text before any build: the focus page's
heading with the most blocks, the term in the most release notes, and the term in at least two docs pages and
two release notes with the highest document count; a document counts for a term only where the term stands
as a term (backticked or a capitalised phrase), never as a word in prose. One line per concept: name, a tab,
aliases joined by ;.
Exit 0 = written, 2 = could not run (an existing output is never overwritten).
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any
from okfwiki.audit import HEADING, LIST_ITEM, read_document
SCHEMA = "okf-wiki/p1-blocks/1"
RELEASE = re.compile(r"^release-v(\d+(?:\.\d+)*)\.md$")
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
FENCE = re.compile(r"^\s*(```|~~~)")
# Concept test 2 of the brief (Google's reference agent), plus the generic section names of docs pages.
META_STOP = frozenset({
"overview", "introduction", "getting started", "quickstart", "tutorial", "walkthrough", "release notes",
"changelog", "roadmap", "faq", "prerequisites", "next steps", "before you begin", "troubleshooting", "see also",
"related resources", "learn more",
})
CODE_TERM = re.compile(r"`([^`\n]{3,40})`")
CAPITAL_PHRASE = re.compile(r"\b[A-Z][\w-]*(?:[ \t]+[A-Z][\w-]*){1,3}\b")
SENTENCE_START = re.compile(r"(?:^|[.!?:]\s+|^\s*(?:[-*+]|\d+[.)])\s+)$")
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
@dataclass(frozen=True)
class SBlock:
file: str
start: int
end: int
kind: str # paragraph | item | table | code
section: str
version: str | None
@dataclass(frozen=True)
class Heading:
file: str
line: int
level: int
text: str
@dataclass(frozen=True)
class Picked:
name: str
aliases: tuple[str, ...]
def is_meta(title: str) -> bool:
return title.strip().lower() in META_STOP
def version_key(name: str) -> tuple[int, ...]:
m = RELEASE.match(name)
if m is None:
raise RunError(f"{name} is not a release note name")
return tuple(int(p) for p in m.group(1).split("."))
def order(names: list[str]) -> list[str]:
docs = sorted(n for n in names if n.startswith("docs-"))
releases = [n for n in names if n.startswith("release-")]
other = [n for n in names if n not in docs and n not in releases]
if other:
raise RunError(f"names that are neither docs pages nor release notes: {other}")
return docs + sorted(releases, key=version_key)
def headings_of(lines: list[str]) -> list[tuple[int, int, str]]:
"""(line, level, text) of every heading outside fenced code."""
out: list[tuple[int, int, str]] = []
fence: str | None = None
for n, raw in enumerate(lines, 1):
f = FENCE.match(raw)
if f:
fence = None if fence == f.group(1) else (fence or f.group(1))
continue
if fence is None:
m = HEADING.match(raw)
if m:
out.append((n, len(m.group(1)), (m.group(2) or "").strip()))
return out
def blocks(path: Path) -> list[SBlock]:
lines = path.read_text(encoding="utf-8").split("\n")
name = path.name
version: str | None = None
if name.startswith("release-"):
for _n, _level, text in headings_of(lines):
m = VERSION_HEADING.match(text)
if m:
version = m.group(1)
break
out: list[SBlock] = []
stack: list[tuple[int, str]] = []
kind: str | None = None
start = 0
fence: str | None = None
def close(end: int) -> None:
nonlocal kind
if kind is not None:
out.append(SBlock(name, start, end, kind, " › ".join(t for _, t in stack), version))
kind = None
for n, raw in enumerate(lines, 1):
f = FENCE.match(raw)
if fence is not None:
if f and f.group(1) == fence:
fence = None
close(n)
continue
if f:
close(n - 1)
fence, kind, start = f.group(1), "code", n
continue
if not raw.strip():
close(n - 1)
continue
m = HEADING.match(raw)
if m:
close(n - 1)
level = len(m.group(1))
stack = [(lv, t) for lv, t in stack if lv < level] + [(level, (m.group(2) or "").strip())]
continue
if raw.lstrip().startswith("|"):
if kind != "table":
close(n - 1)
kind, start = "table", n
continue
if LIST_ITEM.match(raw):
close(n - 1)
kind, start = "item", n
continue
if kind in (None, "table"):
close(n - 1)
kind, start = "paragraph", n
if fence is not None:
close(len(lines))
else:
last = len(lines)
while last > 0 and not lines[last - 1].strip():
last -= 1
close(last)
return out
def docs_headings(kilde: Path) -> list[Heading]:
out: list[Heading] = []
for path in sorted(kilde.glob("docs-*.md")):
for n, level, text in headings_of(path.read_text(encoding="utf-8").split("\n")):
if level <= 3 and text:
out.append(Heading(path.name, n, level, text))
return out
def blocks_document(kilde: Path, names: list[str]) -> dict[str, Any]:
found: list[dict[str, Any]] = []
for name in names:
path = kilde / name
if not path.is_file():
raise RunError(f"{path} is not a file")
found.extend(asdict(b) for b in blocks(path))
return {"schema": SCHEMA, "order": names, "blocks": found}
def blocks_total(kilde: Path) -> int:
return sum(len(blocks(p)) for p in sorted(kilde.glob("*.md")))
def terms(text: str) -> list[str]:
"""Backticked spans of 3-40 characters and capitalised phrases of 2-4 words, a sentence-initial word dropped."""
found: list[str] = [t for t in (m.group(1).strip() for m in CODE_TERM.finditer(text)) if 3 <= len(t) <= 40]
masked = CODE_TERM.sub(lambda m: " " * len(m.group(0)), text)
for raw in masked.split("\n"):
for m in CAPITAL_PHRASE.finditer(raw):
words = m.group(0).split()
if SENTENCE_START.search(raw[:m.start()]):
words = words[1:]
if len(words) >= 2:
found.append(" ".join(words))
return list(dict.fromkeys(t for t in found if t))
def occurs(term: str, text: str) -> bool:
return re.search(r"(?<!\w)" + re.escape(term) + r"(?!\w)", text, re.IGNORECASE) is not None
def pick(blocks_doc: dict[str, Any], kilde: Path, focus: str = "docs-en-permission-modes.md") -> list[Picked]:
names = [str(n) for n in blocks_doc["order"]]
if focus not in names:
raise RunError(f"the focus page {focus} is not in the order")
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
per_section: dict[str, int] = {}
for b in blocks_doc["blocks"]:
if b["file"] == focus and b["section"].count(" › ") >= 1:
per_section[b["section"]] = per_section.get(b["section"], 0) + 1
candidates = [(count, section.split(" › ")[-1]) for section, count in per_section.items()
if not is_meta(section.split(" › ")[-1])]
if not candidates:
raise RunError(f"{focus} has no heading below its title that passes test 2")
first = sorted(candidates, key=lambda c: (-c[0], c[1]))[0][1]
pool = list(dict.fromkeys(t for n in names for t in terms(texts[n])))
docs = [n for n in names if n.startswith("docs-")]
releases = [n for n in names if n.startswith("release-")]
as_terms = {n: set(terms(texts[n])) for n in names}
def count(term: str, files: list[str]) -> int:
return sum(term in as_terms[f] for f in files) # where it stands as a term, not as a word in prose
taken = {first.lower()}
second_pool = [t for t in pool if t.lower() not in taken]
if not second_pool:
raise RunError("no term for the second concept")
second = sorted(second_pool, key=lambda t: (-count(t, releases), t))[0]
taken.add(second.lower())
third_pool = [t for t in pool if t.lower() not in taken and count(t, docs) >= 2 and count(t, releases) >= 2]
if not third_pool:
raise RunError("no term in two docs pages and two release notes for the third concept")
third = sorted(third_pool, key=lambda t: (-count(t, names), t))[0]
return [Picked(first, (first,)), Picked(second, (second,)), Picked(third, (third,))]
def _write_new(path: Path, text: str) -> None:
if path.exists():
raise RunError(f"{path} exists; refusing to overwrite")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
def l4_cited(l4: Path) -> set[str]:
cited: set[str] = set()
for page in sorted(l4.glob("*.md")):
fm = read_document(page, page.name).frontmatter or {}
srcs = fm.get("sources")
for s in srcs if isinstance(srcs, list) else []:
res = s.get("resource") if isinstance(s, dict) else None
if isinstance(res, str) and res:
cited.add(Path(res).name)
return cited
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
o = sub.add_parser("order")
o.add_argument("--manifest", required=True)
o.add_argument("--l4", required=True)
o.add_argument("--out", required=True)
b = sub.add_parser("blocks")
b.add_argument("--kilde", required=True)
b.add_argument("--order", required=True)
b.add_argument("--out", required=True)
t = sub.add_parser("total")
t.add_argument("--kilde", required=True)
k = sub.add_parser("pick")
k.add_argument("--blocks", required=True)
k.add_argument("--kilde", required=True)
k.add_argument("--focus", default="docs-en-permission-modes.md")
k.add_argument("--out", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "order":
manifest = json.loads(Path(args.manifest).read_text(encoding="utf-8"))
names = manifest.get("cut", {}).get("names")
if not isinstance(names, list) or not names:
raise RunError(f"{args.manifest} records no cut names")
cited = l4_cited(Path(args.l4))
if set(map(str, names)) != cited:
raise RunError(f"the manifest's {len(names)} names differ from the {len(cited)} files the L4 cut cites: "
f"only in manifest {sorted(set(names) - cited)}, only in L4 {sorted(cited - set(names))}")
fixed = order([str(n) for n in names])
_write_new(Path(args.out), "\n".join(fixed) + "\n")
print(f"order: {len(fixed)} sources -> {args.out}")
elif args.command == "blocks":
names = Path(args.order).read_text(encoding="utf-8").split()
doc = blocks_document(Path(args.kilde), names)
_write_new(Path(args.out), json.dumps(doc, ensure_ascii=False, indent=1) + "\n")
print(f"blocks: {len(doc['blocks'])} in {len(names)} sources -> {args.out}")
elif args.command == "total":
print(f"blocks in every file of {args.kilde}: {blocks_total(Path(args.kilde))}")
else:
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
picked = pick(doc, Path(args.kilde), args.focus)
_write_new(Path(args.out), "".join(f"{p.name}\t{';'.join(p.aliases)}\n" for p in picked))
print(f"pick: {len(picked)} concepts -> {args.out}")
except (RunError, OSError, json.JSONDecodeError) as exc:
print(f"split cannot run: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,355 @@
"""Tests for evals/direction_probe.py: the direction probe R (PLAN.md § Leveranser, row R).
Three arms answer the same questions from the same sources: file search alone, the okf-wiki plus file search, and a
wiki built with Astro-Han's karpathy-llm-wiki skill. The module sets up each arm's basis and task files, collects the
answers into the gates' format, harvests tokens and isolation breaches from the subagent transcripts, and prints the
table. It never calls a model. The fixtures are synthetic text about gardening (CLAUDE.md rule 3).
Run: python3 -m unittest discover -s evals/tests
"""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / "evals"))
import direction_probe as dp # noqa: E402
TOMATO = "# Tomatoes\nTomatoes need full sun.\nWater them every morning.\n"
BASIL = "# Basil\nBasil likes warm soil.\n"
MINT = "# Mint\nMint spreads through runners.\n"
NOT_IN_CUT = "# Cactus\nCacti need little water.\n"
MD_SET: dict[str, Any] = {
"schema": "okf-wiki/multidoc-set/1",
"questions": [
{"id": "d01", "question": "What do tomatoes and basil need?",
"answer_keys": [{"file": "tomato.md", "text": "SECRETKEYSUN"}],
"evidence": [{"file": "tomato.md", "span": "SECRETSPANTOMATO"}]},
{"id": "d02", "question": "A question outside the probe.",
"answer_keys": [{"file": "mint.md", "text": "OTHERKEY"}],
"evidence": [{"file": "mint.md", "span": "OTHERSPAN"}]},
],
}
WD_SET: dict[str, Any] = {
"schema": "okf-wiki/wholedoc-set/1",
"questions": [
{"id": "d01", "type": "whole", "file": "tomato.md", "question": "What does the tomato page say overall?",
"answer_keys": [{"text": "SECRETWHOLEKEY"}], "evidence": [{"span": "SECRETWHOLESPAN"}]},
{"id": "d02", "type": "count", "file": "basil.md", "question": "How many basil lines are there?",
"count": 424242, "item_pattern": "SECRETPATTERN", "evidence": [{"span": "SECRETCOUNTSPAN"}]},
],
}
SECRETS = ("SECRETKEYSUN", "SECRETSPANTOMATO", "SECRETWHOLEKEY", "SECRETWHOLESPAN", "424242", "SECRETPATTERN",
"SECRETCOUNTSPAN", "OTHERKEY", "A question outside the probe")
class Fixture:
def __init__(self, root: Path) -> None:
self.root = root
self.sources = root / "kilde"
self.sources.mkdir()
for name, text in (("tomato.md", TOMATO), ("basil.md", BASIL), ("mint.md", MINT), ("cactus.md", NOT_IN_CUT)):
(self.sources / name).write_text(text, encoding="utf-8")
self.names = ["basil.md", "mint.md", "tomato.md"]
self.md_set = root / "md.json"
self.md_set.write_text(json.dumps(MD_SET), encoding="utf-8")
self.wd_set = root / "wd.json"
self.wd_set.write_text(json.dumps(WD_SET), encoding="utf-8")
self.wiki = root / "wiki-src"
self.wiki.mkdir()
(self.wiki / "index.md").write_text("# Index\n* [Tomatoes](tomatoes.md)\n", encoding="utf-8")
(self.wiki / "tomatoes.md").write_text("Tomatoes need full sun.\n", encoding="utf-8")
self.run = root / "run"
def setup(self, arm: str, **extra: Path) -> list[Path]:
return dp.setup_arm(run=self.run, arm=arm, sources=self.sources, names=self.names, md_set=self.md_set,
md_ids=["d01"], wd_set=self.wd_set, **extra)
class FetchAstroTest(unittest.TestCase):
def test_raw_file_keeps_the_name_and_the_body_byte_exact_under_the_template_header(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
project = Path(tmp) / "astro"
written = dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden",
collected="2026-09-27")
self.assertEqual([p.name for p in written], fx.names)
raw = (project / "raw" / "garden" / "tomato.md").read_text(encoding="utf-8")
self.assertTrue(raw.startswith("# tomato\n\n> Source: kilde/tomato.md\n> Collected: 2026-09-27\n"
"> Published: Unknown\n\n"))
self.assertTrue(raw.endswith(TOMATO))
self.assertFalse((project / "raw" / "garden" / "cactus.md").exists())
def test_an_existing_raw_file_is_never_overwritten(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
project = Path(tmp) / "astro"
dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden", collected="2026-09-27")
with self.assertRaises(dp.ProbeError):
dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden",
collected="2026-09-27")
class SetupTest(unittest.TestCase):
def test_file_search_basis_holds_exactly_the_cut(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
fx.setup("filsok")
basis = fx.run / "arms" / "filsok"
self.assertEqual(sorted(p.name for p in basis.iterdir()), fx.names)
self.assertEqual((basis / "tomato.md").read_text(encoding="utf-8"), TOMATO)
def test_one_task_per_probe_question_and_none_for_the_rest(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
tasks = fx.setup("filsok")
self.assertEqual(sorted(p.name for p in tasks), ["md-d01.md", "wd-d01.md", "wd-d02.md"])
self.assertTrue(all(p.parent == fx.run / "tasks" / "filsok" for p in tasks))
def test_a_task_carries_the_question_the_basis_and_the_answer_path_but_nothing_from_the_key(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
tasks = {p.name: p.read_text(encoding="utf-8") for p in fx.setup("filsok")}
md = tasks["md-d01.md"]
self.assertIn("What do tomatoes and basil need?", md)
self.assertIn(str(fx.run / "arms" / "filsok"), md)
self.assertIn(str(fx.run / "answers" / "filsok" / "md-d01.json"), md)
self.assertIn('"count"', tasks["wd-d02.md"])
self.assertNotIn('"count"', md)
for text in tasks.values():
for secret in SECRETS:
self.assertNotIn(secret, text)
def test_the_wiki_arm_gets_the_wiki_and_the_cut_and_the_router_order(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
tasks = fx.setup("wiki", wiki=fx.wiki)
basis = fx.run / "arms" / "wiki"
self.assertEqual(sorted(p.name for p in (basis / "kilde").iterdir()), fx.names)
self.assertTrue((basis / "wiki" / "index.md").is_file())
text = tasks[0].read_text(encoding="utf-8")
self.assertLess(text.index("index.md"), text.index("x-locator"))
def test_the_astro_arm_gets_raw_and_wiki_from_the_built_project(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
project = Path(tmp) / "astro"
dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden", collected="2026-09-27")
(project / "wiki").mkdir()
(project / "wiki" / "index.md").write_text("# Knowledge Base Index\n", encoding="utf-8")
tasks = fx.setup("astro", astro=project)
basis = fx.run / "arms" / "astro"
self.assertTrue((basis / "raw" / "garden" / "basil.md").is_file())
self.assertTrue((basis / "wiki" / "index.md").is_file())
self.assertIn("Raw", tasks[0].read_text(encoding="utf-8"))
def test_an_arm_without_its_basis_or_an_existing_arm_is_refused(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
with self.assertRaises(dp.ProbeError):
fx.setup("wiki")
fx.setup("filsok")
with self.assertRaises(dp.ProbeError):
fx.setup("filsok")
with self.assertRaises(dp.ProbeError):
fx.setup("grep-only")
def test_a_probe_id_missing_from_the_set_is_refused(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
with self.assertRaises(dp.ProbeError):
dp.setup_arm(run=fx.run, arm="filsok", sources=fx.sources, names=fx.names, md_set=fx.md_set,
md_ids=["d09"], wd_set=fx.wd_set)
class CollectTest(unittest.TestCase):
def test_answers_are_gathered_per_gate_with_ids_from_the_file_names(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = Path(tmp)
out = run / "answers" / "wiki"
out.mkdir(parents=True)
(out / "md-d01.json").write_text(json.dumps({"id": "wrong", "answer": "a", "citations": []}))
(out / "wd-d02.json").write_text(json.dumps({"answer": "7", "count": 7, "citations": []}))
(out / "wd-d03.json").write_text("not json")
md_path, wd_path, problems = dp.collect(run, "wiki")
md = json.loads(md_path.read_text(encoding="utf-8"))
wd = json.loads(wd_path.read_text(encoding="utf-8"))
self.assertEqual(md, {"arm": "wiki", "answers": [{"id": "d01", "answer": "a", "citations": []}]})
self.assertEqual(wd["answers"], [{"id": "d02", "answer": "7", "count": 7, "citations": []}])
self.assertEqual(problems, ["wd-d03.json: not a JSON object"])
def transcript(lines: list[dict[str, Any]]) -> str:
return "".join(json.dumps(line) + "\n" for line in lines)
def assistant(msg_id: str, tokens: tuple[int, int, int, int], *tools: tuple[str, dict[str, Any]]) -> dict[str, Any]:
i, cc, cr, o = tokens
content = [{"type": "tool_use", "name": name, "input": inp} for name, inp in tools]
return {"type": "assistant", "message": {"id": msg_id, "content": content, "usage": {
"input_tokens": i, "cache_creation_input_tokens": cc, "cache_read_input_tokens": cr, "output_tokens": o}}}
class HarvestTest(unittest.TestCase):
def write(self, folder: Path, name: str, lines: list[dict[str, Any]]) -> None:
(folder / name).write_text(transcript(lines), encoding="utf-8")
def test_tokens_count_each_message_once_and_breaches_are_named(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run, folder = Path(tmp) / "run", Path(tmp) / "subagents"
folder.mkdir()
task = run / "tasks" / "wiki" / "md-d01.md"
basis = run / "arms" / "wiki"
answer = run / "answers" / "wiki" / "md-d01.json"
first = {"type": "user", "message": {"content": f"Read {task} and follow it."}}
self.write(folder, "agent-a.jsonl", [
first,
assistant("m1", (2, 100, 1000, 5), ("Read", {"file_path": str(task)})),
assistant("m1", (2, 100, 1000, 5)),
assistant("m2", (1, 10, 2000, 50), ("Grep", {"pattern": "sun", "path": str(basis / "kilde")}),
("Glob", {"pattern": "*.md", "path": str(basis / "wiki")})),
assistant("m3", (1, 0, 3000, 80), ("Write", {"file_path": str(answer), "content": "{}"}),
("SubagentHandback", {"message": "done"})),
])
self.write(folder, "agent-b.jsonl", [
{"type": "user", "message": {"content": [{"type": "text", "text": f"Read {run}/tasks/filsok/wd-d02.md"}]}},
assistant("n1", (1, 1, 1, 1), ("Read", {"file_path": str(run / "arms" / "wiki" / "wiki" / "x.md")}),
("Grep", {"pattern": "sun"}), ("Bash", {"command": "ls"}),
("Read", {"file_path": str(run / "arms" / "filsok" / ".." / "wiki" / "y.md")})),
])
self.write(folder, "agent-c.jsonl", [{"type": "user", "message": {"content": "Something else."}}])
rows = dp.harvest(folder, run)
self.assertEqual(sorted(rows), ["filsok/wd-d02", "wiki/md-d01"])
wiki = rows["wiki/md-d01"]
self.assertEqual(wiki["tokens"], 1107 + 2061 + 3081)
self.assertEqual(wiki["breaches"], [])
self.assertEqual(wiki["transcript"], "agent-a.jsonl")
breaches = rows["filsok/wd-d02"]["breaches"]
self.assertEqual(len(breaches), 4)
self.assertTrue(any(b.startswith("Bash") for b in breaches))
self.assertTrue(any(b.startswith("Grep without a path") for b in breaches))
def test_bash_is_read_only_search_that_starts_in_the_basis(self) -> None:
basis = Path("/runs/r/arms/filsok")
task, answer = Path("/runs/r/tasks/filsok/md-d01.md"), Path("/runs/r/answers/filsok/md-d01.json")
allowed = [
f"cd {basis} && grep -rn 'full sun' .",
f"cd {basis} && grep -n sun tomato.md | head -5",
f"cd {basis} && find . -name '*.md' | sort && wc -l tomato.md 2>/dev/null",
f"cd '{basis}' && sed -n '1,40p' tomato.md",
f"cd {basis}/kilde && ls",
]
refused = [
"grep -rn sun /runs/r/arms/filsok",
f"cd {basis} && cat ../wiki/x.md",
f"cd {basis} && grep sun /runs/r/tasks/filsok/other.md",
f"cd {basis} && rm tomato.md",
f"cd {basis} && grep sun tomato.md > copy.txt",
f"cd {basis} && find . -name '*.md' -exec cat {{}} ;",
f"cd {basis} && sed -i 's/a/b/' tomato.md",
f"cd {basis} && cat $(echo /etc/hosts)",
f"cd {basis} && cd / && ls",
f"cd {basis} && ls ~",
"cd /runs/r/arms/wiki && ls",
]
for command in allowed:
self.assertEqual(dp.breaches_of([("Bash", {"command": command})], basis, task, answer), [], command)
for command in refused:
self.assertEqual(len(dp.breaches_of([("Bash", {"command": command})], basis, task, answer)), 1, command)
def test_the_task_names_bash_as_the_search_tool_and_its_rule(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
fx = Fixture(Path(tmp))
text = fx.setup("filsok")[0].read_text(encoding="utf-8")
self.assertIn(f"cd {fx.run / 'arms' / 'filsok'} && ", text)
self.assertIn("grep", text)
def test_two_transcripts_for_one_task_keep_the_latest_and_report_the_rerun(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run, folder = Path(tmp) / "run", Path(tmp) / "subagents"
folder.mkdir()
task = run / "tasks" / "filsok" / "md-d01.md"
first = {"type": "user", "timestamp": "2026-09-27T10:00:00Z", "message": {"content": f"Read {task}"}}
later = {"type": "user", "timestamp": "2026-09-27T11:00:00Z", "message": {"content": f"Read {task}"}}
self.write(folder, "agent-old.jsonl", [first, assistant("o", (1, 1, 1, 1))])
self.write(folder, "agent-new.jsonl", [later, assistant("n", (2, 2, 2, 2))])
rows = dp.harvest(folder, run)
self.assertEqual(rows["filsok/md-d01"]["transcript"], "agent-new.jsonl")
self.assertEqual(rows["filsok/md-d01"]["tokens"], 8)
self.assertEqual(rows["filsok/md-d01"]["runs"], 2)
def results(arm: str, correct: dict[str, bool], kind: str) -> dict[str, Any]:
rows = []
for qid, ok in correct.items():
row: dict[str, Any] = {"id": qid, "answered": True, "answer_ok": ok, "source_ok": True, "correct": ok}
if kind == "wd":
row["type"] = "whole" if qid in ("d01", "d02") else "count"
rows.append(row)
return {"arm": arm, "set_sha256": "x", "results": rows}
class ScoreTest(unittest.TestCase):
def test_groups_are_counted_from_the_gate_results_and_pairs_compared(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = Path(tmp)
res = run / "results"
res.mkdir()
md = {"filsok": {"d01": True, "d02": False, "d99": True}, "wiki": {"d01": True, "d02": True, "d99": False}}
wd = {"filsok": {"d01": True, "d02": False, "d03": True}, "wiki": {"d01": False, "d02": True, "d03": True}}
for arm in ("filsok", "wiki"):
(res / f"{arm}.md.json").write_text(json.dumps(results(arm, md[arm], "md")))
(res / f"{arm}.wd.json").write_text(json.dumps(results(arm, wd[arm], "wd")))
(run / "tokens.json").write_text(json.dumps({
"filsok/md-d01": {"tokens": 100, "breaches": [], "transcript": "a", "runs": 1},
"wiki/md-d01": {"tokens": 300, "breaches": [], "transcript": "b", "runs": 1}}))
table = dp.score(run, ["filsok", "wiki"], md_ids=["d01", "d02"])
self.assertEqual(table["arms"]["filsok"]["groups"]["flerdokument"], [1, 2])
self.assertEqual(table["arms"]["wiki"]["groups"]["flerdokument"], [2, 2])
self.assertEqual(table["arms"]["wiki"]["groups"]["helt dokument"], [1, 2])
self.assertEqual(table["arms"]["filsok"]["groups"]["telling"], [1, 1])
self.assertEqual(table["arms"]["wiki"]["total"], [4, 5])
self.assertEqual(table["arms"]["wiki"]["tokens"], {"sum": 300, "questions": 1})
pair = table["pairs"]["wiki vs filsok"]["flerdokument"]
self.assertEqual(pair, {"both": 1, "only wiki": 1, "only filsok": 0, "neither": 0})
self.assertEqual(table["pairs"]["wiki vs filsok"]["helt dokument"]["only filsok"], 1)
self.assertIn("| wiki |", dp.render_table(table))
def test_exact_mcnemar_is_the_two_sided_binomial_over_the_discordant_pairs(self) -> None:
self.assertEqual(dp.mcnemar_exact(0, 0), 1.0)
self.assertEqual(dp.mcnemar_exact(2, 0), 0.5)
self.assertAlmostEqual(dp.mcnemar_exact(6, 0), 0.03125)
self.assertAlmostEqual(dp.mcnemar_exact(1, 5), 0.21875)
self.assertEqual(dp.mcnemar_exact(3, 3), 1.0)
def test_every_pair_carries_its_mcnemar_p_over_all_groups(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = Path(tmp)
res = run / "results"
res.mkdir()
md = {"filsok": {"d01": False, "d02": False}, "wiki": {"d01": True, "d02": True}}
for arm in ("filsok", "wiki"):
(res / f"{arm}.md.json").write_text(json.dumps(results(arm, md[arm], "md")))
(res / f"{arm}.wd.json").write_text(json.dumps(results(arm, {}, "wd")))
table = dp.score(run, ["filsok", "wiki"], md_ids=["d01", "d02"])
self.assertEqual(table["mcnemar"]["wiki vs filsok"], {"only wiki": 2, "only filsok": 0, "p": 0.5})
def test_a_missing_result_file_is_a_failure_not_a_zero(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = Path(tmp)
(run / "results").mkdir()
with self.assertRaises(dp.ProbeError):
dp.score(run, ["filsok"], md_ids=["d01"])
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,609 @@
"""Tests for the P1 method trial's build side (TRIAL, branch p1-metodeprove, not v1 code).
Covers the two trial agent types, the splitter (evals/p1/split.py), the D map and register (dmap.py), K packing
(kpack.py) and assembly with K+ (assemble.py). None of these call a model. The fixtures are synthetic text about
gardening and bread (CLAUDE.md rule 3: no text from the Claude Code documentation is committed).
Run: python3 -m unittest discover -s evals/tests -p 'test_p1_build.py'
"""
from __future__ import annotations
import hashlib
import json
import re
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "evals"))
from okfwiki import audit, claims, view # noqa: E402
from p1 import assemble, dmap, kpack, split # noqa: E402
READER_SHA256 = "13a93bc5a9e87a7282752316ac670664cbdca321b0dac9cdaf92d9e74c383445"
BANNED = ("anthropic", "openai", "urllib", "http.client", "requests", "subprocess", "socket")
DOC_TOMATO = """# Growing tomatoes
Source: https://example.org/garden/tomatoes
How to grow tomatoes in a small garden.
## Soil
Tomatoes need deep, loose soil with `compost` mixed in.
Water the soil every morning.
* **Sun**: full sun for eight hours.
* **Space**: forty centimetres between plants,
more in a wet summer.
```bash
# not a heading
mix compost
```
| Month | Task |
|---|---|
| May | plant |
## Overview
The Kitchen Garden keeps tomatoes and basil together.
### Pruning
Remove side shoots once a week.
"""
DOC_BASIL = """# Growing basil
Source: https://example.org/garden/basil
## Soil
Basil likes warm soil and the Kitchen Garden.
Plant Sweet Genovese early.
## Watering
Water basil with `compost` tea when the leaves droop.
"""
REL_1 = """## 1.2.3
- Added the `compost` command to the Kitchen Garden planner.
- Fixed watering reminders.
"""
REL_2 = """## 1.10.0
- The Kitchen Garden planner now shows `compost` levels.
"""
REL_3 = """## 1.9.2
- Basil seedlings get their own tray.
"""
def runs8(text: str) -> set[tuple[str, ...]]:
words = re.findall(r"[a-z0-9]+", text.lower())
return {tuple(words[i:i + 8]) for i in range(len(words) - 7)}
def before_link_rule(found: list[audit.Finding]) -> list[audit.Finding]:
"""The P1 trial predates criterion 6 as structure (operator 04.10.2026), so its link-rule findings are left out."""
return [f for f in found if f.check != "link-rule"]
class Fixture(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.tmp = Path(self._tmp.name)
self.kilde = self.tmp / "wiki" / "kilde"
self.kilde.mkdir(parents=True)
files = {
"docs-en-tomatoes.md": DOC_TOMATO,
"docs-en-basil.md": DOC_BASIL,
"release-v1.2.3.md": REL_1,
"release-v1.10.0.md": REL_2,
"release-v1.9.2.md": REL_3,
}
for name, text in files.items():
(self.kilde / name).write_text(text, encoding="utf-8")
def tearDown(self) -> None:
self._tmp.cleanup()
# ---------------------------------------------------------------------------------------------- step 1: agents
class AgentFilesTest(unittest.TestCase):
def frontmatter(self, text: str) -> dict[str, str]:
self.assertTrue(text.startswith("---\n"))
head = text[4:text.index("\n---\n", 4)]
return dict(line.split(": ", 1) for line in head.splitlines())
def test_two_trial_agents_have_narrow_tools_and_a_trial_marker(self) -> None:
for name in ("p1-compiler", "p1-consolidator"):
text = (ROOT / ".claude" / "agents" / f"{name}.md").read_text(encoding="utf-8")
fm = self.frontmatter(text)
self.assertEqual(fm["name"], name)
self.assertEqual(fm["tools"], "Read, Write")
self.assertEqual(fm["model"], "opus")
body = text[text.index("\n---\n", 4) + 5:].strip()
self.assertTrue(body.startswith("TRIAL"), name)
fixture = DOC_TOMATO + DOC_BASIL + REL_1 + REL_2 + REL_3
self.assertEqual(runs8(body) & runs8(fixture), set(), name)
def test_the_reader_is_unchanged(self) -> None:
shipped = (ROOT / "agents" / "okf-wiki-reader.md").read_bytes()
self.assertEqual(hashlib.sha256(shipped).hexdigest(), READER_SHA256)
self.assertEqual(shipped, (ROOT / ".claude" / "agents" / "okf-wiki-reader.md").read_bytes())
def test_compiler_names_the_rules_it_must_keep(self) -> None:
body = (ROOT / ".claude" / "agents" / "p1-compiler.md").read_text(encoding="utf-8").lower()
for idea in ("first turn", "never read", "release note", "four tests", "json"):
self.assertIn(idea, body, idea)
def test_consolidator_keeps_every_label(self) -> None:
body = (ROOT / ".claude" / "agents" / "p1-consolidator.md").read_text(encoding="utf-8").lower()
for idea in ("every footnote label", "no new label", "json"):
self.assertIn(idea, body, idea)
# ---------------------------------------------------------------------------------------------- step 1: split
class OrderTest(unittest.TestCase):
def test_docs_first_then_release_notes_by_version_tuple(self) -> None:
names = ["release-v0.2.34.md", "docs-en-b.md", "release-v0.2.9.md", "docs-en-a.md", "release-v1.0.0.md"]
self.assertEqual(split.order(names), ["docs-en-a.md", "docs-en-b.md", "release-v0.2.9.md",
"release-v0.2.34.md", "release-v1.0.0.md"])
def test_an_unknown_name_is_refused(self) -> None:
with self.assertRaises(split.RunError):
split.order(["notes.md"])
class BlocksTest(Fixture):
def test_fence_aware_blocks_with_spans(self) -> None:
got = split.blocks(self.kilde / "docs-en-tomatoes.md")
kinds = [(b.kind, b.start, b.end) for b in got]
# line 2 "Source:" and line 4 are paragraphs; headings are not blocks
self.assertIn(("paragraph", 2, 2), kinds)
self.assertIn(("paragraph", 8, 9), kinds)
self.assertIn(("item", 11, 11), kinds)
self.assertIn(("item", 12, 13), kinds) # a continuation line belongs to its item
self.assertIn(("code", 15, 18), kinds) # the "#" inside the fence is not a heading
self.assertIn(("table", 20, 22), kinds)
heads = {b.section for b in got}
self.assertIn("Growing tomatoes › Soil", heads)
self.assertIn("Growing tomatoes › Overview › Pruning", heads)
self.assertNotIn("not a heading", " ".join(heads))
def test_every_non_blank_non_heading_line_is_in_exactly_one_block(self) -> None:
path = self.kilde / "docs-en-tomatoes.md"
lines = path.read_text(encoding="utf-8").split("\n")
got = split.blocks(path)
covered: dict[int, int] = {}
for b in got:
for n in range(b.start, b.end + 1):
covered[n] = covered.get(n, 0) + 1
for n, raw in enumerate(lines, 1):
heading = raw.startswith("#") and not (15 <= n <= 18)
if raw.strip() and not heading:
self.assertEqual(covered.get(n), 1, (n, raw))
if heading:
self.assertNotIn(n, covered, (n, raw))
def test_release_note_blocks_carry_the_version(self) -> None:
got = split.blocks(self.kilde / "release-v1.10.0.md")
self.assertEqual({b.version for b in got}, {"1.10.0"})
self.assertEqual(split.blocks(self.kilde / "docs-en-basil.md")[0].version, None)
class DocsHeadingsTest(Fixture):
def test_titles_and_second_and_third_level_headings_of_docs_pages_only(self) -> None:
got = split.docs_headings(self.kilde)
texts = [(h.file, h.level, h.text) for h in got]
self.assertIn(("docs-en-tomatoes.md", 1, "Growing tomatoes"), texts)
self.assertIn(("docs-en-tomatoes.md", 3, "Pruning"), texts)
self.assertFalse(any(f.startswith("release-") for f, _, _ in texts))
self.assertFalse(any(t == "not a heading" for _, _, t in texts))
class PickTest(Fixture):
def blocks_doc(self, order: list[str]) -> dict[str, object]:
return split.blocks_document(self.kilde, order)
def test_three_concepts_by_the_stated_rules(self) -> None:
order = split.order([p.name for p in self.kilde.iterdir()])
picked = split.pick(self.blocks_doc(order), self.kilde, focus="docs-en-tomatoes.md")
names = [p.name for p in picked]
self.assertEqual(len(names), 3)
self.assertEqual(len(set(names)), 3)
self.assertEqual(names[0], "Soil") # the focus page's heading with the most blocks
# "Kitchen Garden" and "compost" are both in 2 release notes; the tie is broken by name
self.assertEqual(names[1], "Kitchen Garden")
self.assertEqual(names[2], "compost") # in 2 docs pages and 2 release notes, distinct from 1 and 2
def test_a_document_counts_for_a_term_only_where_it_stands_as_a_term(self) -> None:
# Found in session 2: the word "and", backticked once, was counted in every document where it is prose.
(self.kilde / "docs-en-zz.md").write_text("# Joining\n\n## Words\n\nUse `and` to join two beds.\n",
encoding="utf-8")
for v in ("2.0.0", "2.0.1", "2.0.2"):
(self.kilde / f"release-v{v}.md").write_text(f"## {v}\n\n- Tomatoes and basil and mint.\n",
encoding="utf-8")
order = split.order([p.name for p in self.kilde.iterdir()])
names = [p.name for p in split.pick(self.blocks_doc(order), self.kilde, focus="docs-en-tomatoes.md")]
self.assertNotIn("and", names)
self.assertEqual(names[1:], ["Kitchen Garden", "compost"])
def test_a_backticked_term_counts_its_length_after_trimming(self) -> None:
# Found in session 2: "` or `" is four characters with its spaces, and became the two-letter term "or".
got = split.terms("Use `compost` ` or ` ` ab ` ` mulch ` here.")
self.assertEqual(got, ["compost", "mulch"])
def test_the_focus_page_must_be_in_the_order(self) -> None:
with self.assertRaises(split.RunError):
split.pick(self.blocks_doc(["docs-en-basil.md"]), self.kilde, focus="docs-en-tomatoes.md")
class CliTest(Fixture):
def test_order_refuses_names_that_differ_from_the_l4_cut(self) -> None:
manifest = self.tmp / "m.json"
manifest.write_text(json.dumps({"cut": {"names": ["docs-en-basil.md", "release-v1.2.3.md"]}}), encoding="utf-8")
l4 = self.tmp / "l4"
l4.mkdir()
(l4 / "a.md").write_text("---\ntype: Concept\nsources:\n - id: a\n resource: /kilde/docs-en-basil.md\n"
"---\n\nx\n", encoding="utf-8")
out = self.tmp / "rekkefolge.txt"
self.assertEqual(split.main(["order", "--manifest", str(manifest), "--l4", str(l4), "--out", str(out)]), 2)
self.assertFalse(out.exists())
(l4 / "b.md").write_text("---\ntype: Concept\nsources:\n - id: b\n resource: /kilde/release-v1.2.3.md\n"
"---\n\nx\n", encoding="utf-8")
self.assertEqual(split.main(["order", "--manifest", str(manifest), "--l4", str(l4), "--out", str(out)]), 0)
self.assertEqual(out.read_text(encoding="utf-8").split(), ["docs-en-basil.md", "release-v1.2.3.md"])
def test_blocks_and_pick_refuse_to_overwrite(self) -> None:
order = self.tmp / "order.txt"
order.write_text("\n".join(split.order([p.name for p in self.kilde.iterdir()])) + "\n", encoding="utf-8")
out = self.tmp / "blocks.json"
self.assertEqual(split.main(["blocks", "--kilde", str(self.kilde), "--order", str(order), "--out", str(out)]), 0)
data = json.loads(out.read_text(encoding="utf-8"))
self.assertEqual(data["schema"], "okf-wiki/p1-blocks/1")
self.assertEqual(split.main(["blocks", "--kilde", str(self.kilde), "--order", str(order), "--out", str(out)]), 2)
picked = self.tmp / "begreper.txt"
args = ["pick", "--blocks", str(out), "--kilde", str(self.kilde), "--focus", "docs-en-tomatoes.md",
"--out", str(picked)]
self.assertEqual(split.main(args), 0)
self.assertEqual(len(picked.read_text(encoding="utf-8").strip().splitlines()), 3)
self.assertEqual(split.main(args), 2)
def test_total_counts_blocks_in_every_file(self) -> None:
expected = sum(len(split.blocks(p)) for p in self.kilde.iterdir())
self.assertEqual(split.blocks_total(self.kilde), expected)
# ---------------------------------------------------------------------------------------------- step 2: D
class DMapTest(Fixture):
def build(self) -> Path:
order = split.order([p.name for p in self.kilde.iterdir()])
out = self.tmp / "D"
dmap.build(split.blocks_document(self.kilde, order), self.kilde.parent, out, at="2026-10-02T12:00:00Z")
return out
def test_d_passes_the_auditor_with_no_claims(self) -> None:
out = self.build()
self.assertEqual(before_link_rule(audit.audit(out, self.kilde.parent)), [])
found = claims.extract(out, self.kilde.parent)
self.assertEqual(found.claims, [])
self.assertTrue(found.exceptions)
def test_register_links_point_to_map_sections(self) -> None:
out = self.build()
page = (out / "r-kitchen-garden.md").read_text(encoding="utf-8")
self.assertIn("(k-docs-en-basil.md#soil)", page)
self.assertIn("(k-release-v1-2-3.md#1-2-3)", page)
def test_register_keeps_docs_headings_and_terms_in_two_documents_only(self) -> None:
out = self.build()
names = {p.name for p in out.glob("r-*.md")}
self.assertIn("r-pruning.md", names) # a docs heading (i), in one document
self.assertIn("r-compost.md", names) # a backticked term in several documents
self.assertNotIn("r-sweet-genovese.md", names) # a capitalised phrase in one document only
self.assertNotIn("r-overview.md", names) # a stop-listed heading fails test 2
def test_quotes_stand_on_their_lines_and_stay_strings(self) -> None:
out = self.build()
doc = audit.read_document(out / "r-compost.md", "r-compost.md")
assert doc.frontmatter is not None
for s in doc.frontmatter["sources"]:
self.assertIsInstance(s["x-quote"], str)
line = (self.kilde / Path(s["resource"]).name).read_text(encoding="utf-8").split("\n")[s["x-locator"]["line"] - 1]
self.assertIn(s["x-quote"], line)
self.assertIsInstance(doc.frontmatter["generated"]["at"], str)
def test_a_quote_like_no_survives_a_yaml_round_trip(self) -> None:
text = dmap.frontmatter({"type": "Concept", "sources": [{"id": "a", "x-quote": "no"}, {"id": "b", "x-quote": "1.10"}]})
path = self.tmp / "x.md"
path.write_text(text + "\n# x\n", encoding="utf-8")
fm = audit.read_document(path, "x.md").frontmatter
assert fm is not None
self.assertEqual([s["x-quote"] for s in fm["sources"]], ["no", "1.10"])
def test_the_log_records_the_thresholds_and_build_refuses_an_existing_target(self) -> None:
out = self.build()
log = (out / "log.md").read_text(encoding="utf-8")
self.assertIn("2 of the", log)
with self.assertRaises(dmap.RunError):
dmap.build({"order": [], "blocks": []}, self.kilde.parent, out, at="2026-10-02T12:00:00Z")
# ---------------------------------------------------------------------------------------------- step 3: K packing
def entry(eid: str, title: str, aliases: tuple[str, ...] = ()) -> dict[str, object]:
return {"id": eid, "title": title, "aliases": [title, *aliases], "kind": "skeleton", "merge_of": None,
"origin": None}
class KSkeletonTest(Fixture):
def test_the_skeleton_is_an_index_of_docs_headings_that_pass_test_2(self) -> None:
index = kpack.skeleton(self.kilde)
titles = {e["title"] for e in index["entries"]}
self.assertIn("Growing tomatoes", titles)
self.assertIn("Pruning", titles)
self.assertIn("Soil", titles)
self.assertNotIn("Overview", titles)
soil = [e for e in index["entries"] if e["title"] == "Soil"]
self.assertEqual(len(soil), 1) # the same heading in two pages is one entry
self.assertEqual(soil[0]["origin"]["file"], "docs-en-basil.md") # pages in name order, first one wins
self.assertFalse(any(p.suffix == ".md" for p in self.tmp.rglob("*") if p.parent.name != "kilde"))
def test_norm(self) -> None:
self.assertEqual(kpack.norm("The `Compost` bins"), "compost bin")
self.assertEqual(kpack.norm("settings.json"), "settings.json")
self.assertEqual(kpack.norm(" Kitchen Garden! "), "kitchen garden")
self.assertEqual(kpack.norm("CLI flags"), "cli flag")
class KIdentityTest(unittest.TestCase):
def setUp(self) -> None:
self.index = {"entries": [entry("kitchen-garden-planner", "Kitchen Garden planner"),
entry("compost", "compost", ("compost heap",))]}
def frag(self, concept: str, status: str = "new", file: str = "docs-en-tomatoes.md", eid: str | None = None
) -> dict[str, object]:
f: dict[str, object] = {"concept": concept, "status": status, "anchor": {"file": file, "start": 8, "end": 9}}
if eid:
f["id"] = eid
return f
def test_outcomes(self) -> None:
r = kpack.resolve
self.assertEqual(r(self.frag("x", "existing", eid="compost"), self.index), ("existing", "compost"))
self.assertEqual(r(self.frag("Kitchen Garden planners"), self.index), ("existing", "kitchen-garden-planner"))
self.assertEqual(r(self.frag("Compost heap"), self.index), ("existing", "compost"))
self.assertEqual(r(self.frag("Kitchen Garden planner tool"), self.index),
("merge-candidate", "kitchen-garden-planner"))
self.assertEqual(r(self.frag("Garden hose"), self.index), ("new", None))
self.assertEqual(r(self.frag("Garden hose", file="release-v1.2.3.md"), self.index), ("rejected", "release-note-new"))
self.assertEqual(r(self.frag("Kitchen Garden planner tool", file="release-v1.2.3.md"), self.index),
("rejected", "release-note-new"))
self.assertEqual(r(self.frag("Troubleshooting"), self.index), ("rejected", "meta"))
def test_relevant_lines_by_phrase(self) -> None:
got = kpack.relevant_lines(self.index, "We keep the compost heaps behind the shed.")
self.assertEqual([e["id"] for e in got], ["compost"])
self.assertEqual(kpack.relevant_lines(self.index, "Nothing about gardens here."), [])
class KPackTest(Fixture):
def blocks_doc(self) -> dict[str, object]:
return split.blocks_document(self.kilde, split.order([p.name for p in self.kilde.iterdir()]))
def test_packs_of_five(self) -> None:
self.assertEqual([len(p) for p in kpack.packs([f"n{i}" for i in range(19)], 5)], [5, 5, 5, 4])
def test_a_large_pack_is_split_into_parts_under_the_read_cap(self) -> None:
big = "".join(f"Line {i} says the tomatoes are fine today and tomorrow, and the basil too.\n\n"
for i in range(1110))
(self.kilde / "docs-en-big.md").write_text("# Big\n\n" + big, encoding="utf-8")
doc = split.blocks_document(self.kilde, ["docs-en-big.md"])
parts = kpack.render_input(1, ["docs-en-big.md"], doc, [], self.kilde)
self.assertGreater(sum(len(p) for p in parts), 130_000)
self.assertEqual(len(parts), 3)
self.assertTrue(all(len(p) <= kpack.PART_CHARS for p in parts))
self.assertIn("docs-en-big.md › Big › 3–3", parts[0])
def test_validate(self) -> None:
doc = self.blocks_doc()
pack = ["docs-en-tomatoes.md", "release-v1.2.3.md"]
good = {"concept": "Soil", "status": "existing", "id": "soil", "text": "Tomatoes need deep, loose soil.",
"anchor": {"file": "docs-en-tomatoes.md", "start": 8, "end": 9}}
cases = {
"anchor-crosses-blocks": {**good, "anchor": {"file": "docs-en-tomatoes.md", "start": 8, "end": 11}},
"quote-missing": {**good, "text": "Plants stand 40 cm apart."},
"quote-not-in-span": {**good, "text": "Keep 40 cm.", "quote": "forty centimetres"},
"sentence-count": {**good, "text": "One. Two. Three. Four."},
"tests-missing": {**good, "status": "new", "concept": "Loose soil"},
"anchor-outside-pack": {**good, "anchor": {"file": "docs-en-basil.md", "start": 7, "end": 7}},
"malformed": {"concept": "Soil"},
}
no_text = {**good, "text": ""}
accepted, rejected = kpack.validate([good, no_text, *cases.values()], pack, doc, self.kilde)
self.assertEqual(len(accepted), 1)
self.assertEqual(rejected["sentence-count"], 2) # the empty text and the four sentences
for reason in cases:
self.assertGreaterEqual(rejected[reason], 1, reason)
ok = {**good, "text": "Plants stand forty centimetres apart.", "anchor": {"file": "docs-en-tomatoes.md",
"start": 12, "end": 13}, "quote": "forty centimetres"}
accepted, rejected = kpack.validate([ok], pack, doc, self.kilde)
self.assertEqual((len(accepted), sum(rejected.values())), (1, 0))
class KCliTest(Fixture):
def test_skeleton_pack_and_accept(self) -> None:
work = self.tmp / "work"
blocks = self.tmp / "blocks.json"
order = split.order([p.name for p in self.kilde.iterdir()])
blocks.write_text(json.dumps(split.blocks_document(self.kilde, order)), encoding="utf-8")
index = work / "K-work" / "index.json"
self.assertEqual(kpack.main(["skeleton", "--kilde", str(self.kilde), "--out", str(index)]), 0)
self.assertEqual(kpack.main(["skeleton", "--kilde", str(self.kilde), "--out", str(index)]), 2)
root = str(self.kilde.parent)
self.assertEqual(kpack.main(["pack", "1", "--blocks", str(blocks), "--index", str(index), "--kilde-root", root,
"--work", str(work)]), 0)
self.assertTrue((work / "K-oppgaver" / "pack-1-part-1.md").is_file())
fragments = {"fragments": [
{"concept": "Soil", "status": "existing", "id": "soil", "text": "Tomatoes need deep, loose soil.",
"anchor": {"file": "docs-en-tomatoes.md", "start": 8, "end": 9}},
{"concept": "Kitchen Garden", "status": "new", "text": "The Kitchen Garden keeps tomatoes and basil.",
"anchor": {"file": "docs-en-tomatoes.md", "start": 26, "end": 26},
"tests": {"topic_shape": "a named planner", "citation": "See the Kitchen Garden reference"}},
{"concept": "Kitchen Garden", "status": "existing", "text": "The planner gained a command.",
"anchor": {"file": "release-v1.2.3.md", "start": 3, "end": 3}},
{"concept": "Watering reminders", "status": "new", "text": "Reminders were fixed.",
"anchor": {"file": "release-v1.2.3.md", "start": 4, "end": 4},
"tests": {"topic_shape": "x", "citation": "y"}},
]}
out = work / "K-fragments" / "pack-1.json"
out.parent.mkdir(parents=True)
out.write_text(json.dumps(fragments), encoding="utf-8")
self.assertEqual(kpack.main(["accept", "1", "--work", str(work), "--blocks", str(blocks), "--kilde-root", root]), 0)
acc = json.loads((work / "K-work" / "accepted.json").read_text(encoding="utf-8"))["fragments"]
self.assertEqual([a["outcome"] for a in acc], ["existing", "new", "existing"])
self.assertEqual(acc[2]["concept_id"], acc[1]["concept_id"]) # the release note enriches the new concept
self.assertEqual(acc[2]["version"], "1.2.3")
log = json.loads((work / "K-work" / "packs.json").read_text(encoding="utf-8"))
self.assertEqual(log["packs"][0]["rejected"], {"release-note-new": 1})
self.assertEqual(log["packs"][0]["proposed"], 4)
# PM 02.10: the ids of the index lines sent, and the accepted fragments whose concept was not among them
sent = log["packs"][0]["index_ids"]
self.assertIn("soil", sent)
self.assertNotIn(acc[1]["concept_id"], sent)
self.assertEqual(log["packs"][0]["not_sent"], {"total": 2, "by_outcome": {"existing": 1, "new": 1}})
# ---------------------------------------------------------------------------------------------- step 4: assembly
def accepted(cid: str, file: str, start: int, end: int, text: str, pack: int, version: str | None = None,
quote: str | None = None, quote_line: int | None = None) -> dict[str, object]:
return {"pack": pack, "position": 1, "file": file, "start": start, "end": end, "section": "-", "text": text,
"quote": quote, "quote_line": quote_line, "version": version, "concept_id": cid,
"outcome": "existing"}
class AssembleTest(Fixture):
def setUp(self) -> None:
super().setUp()
self.work = self.tmp / "work"
(self.work / "K-work").mkdir(parents=True)
index = {"entries": [
{**entry("soil", "Soil"), "origin": {"file": "docs-en-basil.md", "line": 4}},
{**entry("pruning", "Pruning"), "origin": {"file": "docs-en-tomatoes.md", "line": 28}},
{**entry("kitchen-garden", "Kitchen Garden"), "kind": "new"},
{**entry("kitchen-garden-planner", "Kitchen Garden planner"), "kind": "merge-candidate",
"merge_of": "kitchen-garden"},
]}
frags = [
accepted("soil", "docs-en-tomatoes.md", 8, 9, "Tomatoes need deep, loose soil.", 1),
accepted("kitchen-garden", "docs-en-tomatoes.md", 26, 26,
"The Kitchen Garden keeps tomatoes and basil together.", 1, quote="Kitchen Garden", quote_line=26),
accepted("kitchen-garden", "release-v1.2.3.md", 3, 3, "The planner gained a compost command.", 1, "1.2.3"),
accepted("soil", "docs-en-basil.md", 6, 7, "Basil likes warm soil near the Kitchen Garden.", 1),
accepted("kitchen-garden", "release-v1.10.0.md", 3, 3, "The planner shows compost levels.", 2, "1.10.0"),
accepted("kitchen-garden-planner", "release-v1.9.2.md", 3, 3, "Seedlings get their own tray.", 2, "1.9.2"),
]
(self.work / "K-work" / "index.json").write_text(json.dumps(index), encoding="utf-8")
(self.work / "K-work" / "accepted.json").write_text(json.dumps({"fragments": frags}), encoding="utf-8")
self.out = self.tmp / "K"
assemble.assemble(self.work, self.kilde.parent, self.out, at="2026-10-02T12:00:00Z")
def page(self, name: str) -> audit.Document:
return audit.read_document(self.out / name, name)
def test_the_bundle_passes_the_auditor(self) -> None:
self.assertEqual(before_link_rule(audit.audit(self.out, self.kilde.parent)), [])
found = claims.extract(self.out, self.kilde.parent)
self.assertEqual(claims.criterion_1(found)["total"], (6, 6))
def test_only_enriched_concepts_get_pages(self) -> None:
names = {p.name for p in self.out.glob("*.md")} - {"index.md", "log.md"}
self.assertEqual(names, {"soil.md", "kitchen-garden.md", "kitchen-garden-planner.md"})
fm = self.page("kitchen-garden-planner.md").frontmatter
assert fm is not None
self.assertEqual(fm["x-merge-candidate"], "kitchen-garden")
def test_history_in_version_order_without_version_numbers_in_the_text(self) -> None:
doc = self.page("kitchen-garden.md")
body = doc.body
self.assertLess(body.index("gained a compost command"), body.index("shows compost levels"))
prose = " ".join(b.text for b in audit.prose_blocks(doc))
self.assertEqual(view.version_numbers(audit.FOOTNOTE_REF.sub("", prose)), [])
fm = doc.frontmatter
assert fm is not None
versions = [s for s in fm["sources"] if view.is_version_heading(s)]
self.assertEqual(sorted(s["x-quote"] for s in versions), ["1.10.0", "1.2.3"])
self.assertEqual(fm["x-fragments"], 3)
def test_code_quotes_skip_structural_lines_and_are_marked(self) -> None:
fm = self.page("soil.md").frontmatter
assert fm is not None
first = fm["sources"][0]
self.assertEqual(first["x-locator"], {"line": 8})
self.assertEqual(first["x-quote-by"], "code")
self.assertEqual(first["x-span"], [8, 9])
kg = self.page("kitchen-garden.md").frontmatter
assert kg is not None
self.assertNotIn("x-quote-by", kg["sources"][0])
self.assertEqual(assemble.code_quote(["```bash", "<Note>", "|---|---|", "", " real line "], 10),
("real line", 14))
def test_a_concept_title_in_another_page_becomes_one_link(self) -> None:
self.assertIn("[Kitchen Garden](kitchen-garden.md)", self.page("soil.md").body)
class KPlusTest(AssembleTest):
def test_copy_tasks_and_apply(self) -> None:
kplus = self.tmp / "K+"
assemble.copy(self.out, kplus)
with self.assertRaises(assemble.RunError):
assemble.copy(self.out, kplus)
runs = assemble.consolidation_tasks(kplus, self.work, min_fragments=3, per_run=8)
self.assertEqual([r.pages for r in runs], [["kitchen-garden"]])
self.assertTrue(runs[0].task.is_file())
old = self.page("kitchen-garden.md").body
labels = sorted(set(audit.FOOTNOTE_REF.findall(old)))
good = "The Kitchen Garden keeps tomatoes and basil together, and its planner tracks compost " + "".join(
f"[^{x}]" for x in labels) + "."
dropped = "The Kitchen Garden keeps tomatoes and basil together[^" + labels[0] + "]."
added = good.replace(".", "[^zz].", 1)
bodies = self.tmp / "bodies.json"
bodies.write_text(json.dumps({"kitchen-garden": good, "soil": dropped}), encoding="utf-8")
self.assertEqual(assemble.apply(kplus, bodies), (1, 1))
self.assertIn("planner tracks compost", (kplus / "kitchen-garden.md").read_text(encoding="utf-8"))
self.assertEqual(before_link_rule(audit.audit(kplus, self.kilde.parent)), [])
bodies.write_text(json.dumps({"kitchen-garden": added}), encoding="utf-8")
self.assertEqual(assemble.apply(kplus, bodies), (0, 1))
class NoModelTest(unittest.TestCase):
def test_trial_modules_import_no_network_or_model_client(self) -> None:
for path in sorted((ROOT / "evals" / "p1").glob("*.py")):
source = path.read_text(encoding="utf-8")
for banned in BANNED:
self.assertNotIn(f"import {banned}", source, path.name)
self.assertNotIn(f"from {banned}", source, path.name)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,366 @@
"""Tests for the P1 method trial's report side (TRIAL, branch p1-metodeprove, not v1 code).
Covers the measures (evals/p1/measure.py), the cost harvester (cost.py), the side-by-side page (sidebyside.py) and the
two verification scripts. None of these call a model. The fixtures are synthetic text about gardening (CLAUDE.md
rule 3: no text from the Claude Code documentation is committed).
Run: python3 -m unittest discover -s evals/tests -p 'test_p1_report.py'
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
from typing import Any
import yaml
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "evals"))
from okfwiki import audit, view # noqa: E402
from p1 import cost, measure, sidebyside, split # noqa: E402
BANNED = ("anthropic", "openai", "urllib", "http.client", "requests", "subprocess", "socket")
DOC_A = """# Alpha
## Beds
Raised beds drain well.
They warm early in spring.
- Line one item.
- Line two item.
## Paths
Gravel paths stay dry.
"""
REL = """## 2.0.0
- Beds got labels.
"""
def src(sid: str, file: str, line: int, quote: str, span: tuple[int, int] | None = None) -> dict[str, Any]:
s: dict[str, Any] = {"id": sid, "resource": f"/kilde/{file}", "title": file, "x-locator": {"line": line},
"x-quote": quote}
if span:
s["x-span"] = list(span)
return s
def write_page(folder: Path, name: str, title: str, body: str, sources: list[dict[str, Any]],
**extra: Any) -> None:
fm = {"type": "Concept", "title": title, "description": title, "sources": sources, **extra,
"generated": {"by": "test", "at": "2026-10-02T12:00:00Z"}}
text = "---\n" + yaml.safe_dump(fm, sort_keys=False) + "---\n\n# " + title + "\n\n" + body + "\n"
folder.mkdir(parents=True, exist_ok=True)
(folder / name).write_text(text, encoding="utf-8")
class Fixture(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.tmp = Path(self._tmp.name)
self.root = self.tmp / "wiki"
self.kilde = self.root / "kilde"
self.kilde.mkdir(parents=True)
(self.kilde / "docs-en-a.md").write_text(DOC_A, encoding="utf-8")
(self.kilde / "release-v2.0.0.md").write_text(REL, encoding="utf-8")
self.order = ["docs-en-a.md", "release-v2.0.0.md"]
self.blocks = split.blocks_document(self.kilde, self.order)
self.bundle = self.tmp / "B"
write_page(self.bundle, "beds.md", "Beds",
"Raised beds drain well.[^b1] The beds have a heading.[^b2] Items begin.[^b3] Two items.[^b4]\n\n"
"## History\n\n- Beds got labels.[^r1][^v1]\n\n| a | b |",
[src("b1", "docs-en-a.md", 5, "Raised beds drain well.", (5, 6)),
src("b2", "docs-en-a.md", 3, "Beds", (3, 3)),
src("b3", "docs-en-a.md", 8, "Line one item.", (8, 13)),
src("b4", "docs-en-a.md", 9, "Line two item.", (8, 9)),
src("v1", "release-v2.0.0.md", 1, "2.0.0"),
src("r1", "release-v2.0.0.md", 3, "Beds got labels.", (3, 3)),
src("u1", "docs-en-a.md", 13, "Gravel paths stay dry.", (13, 13))],
**{"x-aliases": ["Beds", "Raised beds"]})
write_page(self.bundle, "labels.md", "Labels", "Beds got labels.[^r1][^v1] See also: [Beds](beds.md)",
[src("v1", "release-v2.0.0.md", 1, "2.0.0"), src("r1", "release-v2.0.0.md", 3, "Beds got labels.")])
(self.bundle / "index.md").write_text('---\nokf_version: "0.2"\n---\n\n# Concepts\n\n- [Beds](beds.md)\n'
'- [Labels](labels.md)\n', encoding="utf-8")
(self.bundle / "log.md").write_text("# Log\n\n## 2026-10-02\n* built\n", encoding="utf-8")
def tearDown(self) -> None:
self._tmp.cleanup()
# ---------------------------------------------------------------------------------------------- step 5: measures
class CoverageTest(Fixture):
def test_cited_entries_by_locator_line_only(self) -> None:
got = measure.coverage(self.bundle, self.root, self.blocks, self.order)
# covered: 5-6 (b1), 8 (b3's locator; its wide span is ignored), 9 (b4), release 3 (r1); not 13 (u1 uncited)
self.assertEqual((got.covered, got.total), (4, 5))
def test_a_stopped_variant_counts_only_the_sources_it_built(self) -> None:
got = measure.coverage(self.bundle, self.root, self.blocks, self.order[:1])
self.assertEqual((got.covered, got.total), (3, 4))
def test_per_pack_position(self) -> None:
got = measure.coverage(self.bundle, self.root, self.blocks, self.order, pack_size=5)
self.assertEqual(got.per_position, {1: (3, 4, 1), 2: (1, 1, 1)})
class StructureTest(Fixture):
def test_shares_precision_and_concepts_per_source(self) -> None:
s = measure.structure(self.bundle, self.root, self.blocks, self.order)
self.assertEqual(s["pages"], 2)
self.assertEqual(s["concepts_per_source"], {"docs-en-a.md": 1, "release-v2.0.0.md": 1})
# version sources are left out of the quote and precision counts
self.assertEqual(s["quotes"]["entries"], 6)
self.assertEqual(s["quotes"]["structural"], 1) # b2 quotes a heading line
self.assertEqual(s["precision"], {"line": 3, "block": 1, "section": 1, "document": 1})
self.assertEqual(s["islands"]["total"], sum(not v for v in view.related(
audit.load_bundle(self.bundle, self.root).concepts).values()))
def test_duplicates_by_norm_alias_and_jaccard_in_any_bundle(self) -> None:
b = self.tmp / "dups"
for name, title in (("a.md", "Beds"), ("b.md", "Bed"), ("c.md", "Raised beds"), ("d.md", "Gravel path"),
("e.md", "Gravel path rules"), ("f.md", "Tools")):
write_page(b, name, title, "x[^s].", [src("s", "docs-en-a.md", 5, "Raised")],
**({"x-aliases": ["Beds", "Raised beds"]} if name == "a.md" else {}))
pairs = measure.duplicates(audit.load_bundle(b, self.root).concepts)
self.assertIn(("a.md", "b.md", "norm"), pairs)
self.assertIn(("a.md", "c.md", "alias"), pairs)
self.assertIn(("d.md", "e.md", "jaccard"), pairs)
self.assertFalse(any("f.md" in p for p in pairs))
def test_a_wrong_root_raises(self) -> None:
with self.assertRaises(measure.RunError):
measure.structure(self.bundle, self.tmp / "nowhere", self.blocks, self.order)
class AuditRecordTest(Fixture):
def test_uncited_sentences_are_classed_by_cause_and_yaml_form_apart(self) -> None:
write_page(self.bundle, "odd.md", "Odd", "```bash\n\nPlain words here.\n\nSee also: [Beds](beds.md)",
[{"id": "y", "resource": "/kilde/docs-en-a.md", "x-locator": {"line": 5}, "x-quote": False}])
out = self.tmp / "resultat.json"
rec = measure.audit_record(self.bundle, self.root, "K", out)
self.assertEqual(rec["exit"], 1)
# labels.md in the shared fixture holds one more link lead-in ("See also: [Beds](beds.md)")
self.assertEqual(rec["uncited"], {"fence line": 1, "link lead-in": 2, "prose": 1, "table row": 1})
self.assertEqual(rec["yaml_form"], 1)
self.assertEqual(json.loads(out.read_text(encoding="utf-8"))["variants"]["K"]["exit"], 1)
with self.assertRaises(measure.RunError):
measure.audit_record(self.bundle, self.root, "K", out) # one record per variant
# ---------------------------------------------------------------------------------------------- step 6: cost
def msg(mid: str, inp: int, cr: int, cw: int, out: int, stop: str | None = "end_turn") -> dict[str, Any]:
return {"type": "assistant", "message": {"id": mid, "stop_reason": stop, "usage": {
"input_tokens": inp, "cache_read_input_tokens": cr, "cache_creation_input_tokens": cw, "output_tokens": out}}}
def jsonl(path: Path, lines: list[dict[str, Any]]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("\n".join(json.dumps(x) for x in lines) + "\n", encoding="utf-8")
class HarvestTest(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.proj = Path(self._tmp.name)
user = {"type": "user", "message": {"content": "task"}}
jsonl(self.proj / "s1.jsonl", [user, msg("m0", 5, 100, 10, 7)])
sub = self.proj / "s1" / "subagents"
# a streamed message id appears twice with growing output: counted once, at its max
jsonl(sub / "agent-a1.jsonl", [user, msg("x1", 3, 20_000, 2_000, 10, None), msg("x1", 3, 20_000, 2_000, 50),
msg("x2", 1, 22_000, 500, 900)])
(sub / "agent-a1.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "P1-K-pack1"}))
jsonl(sub / "agent-a2.jsonl", [user, msg("y1", 2, 19_000, 1_000, 5)])
(sub / "agent-a2.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "load test"}))
jsonl(sub / "agent-a3.jsonl", [user, msg("z1", 1, 1, 1, 1, None)])
(sub / "agent-a3.meta.json").write_text(json.dumps({"agentType": "p1-consolidator",
"description": "P1-K+-run1"}))
# a session not listed in okter.txt: its transcript is never opened
jsonl(self.proj / "s9" / "subagents" / "agent-b1.jsonl", [user, msg("q", 1, 1, 1, 1)])
(self.proj / "s9" / "subagents" / "agent-b1.meta.json").write_text(
json.dumps({"agentType": "p1-compiler", "description": "P1-K-pack2"}))
self.okter = self.proj / "okter.txt"
self.okter.write_text("s1 step8\ns1 step9\n", encoding="utf-8")
def tearDown(self) -> None:
self._tmp.cleanup()
def test_four_categories_turns_startup_and_attribution(self) -> None:
got = cost.harvest(self.proj, cost.sessions(self.okter))
self.assertEqual(got["sessions_read"], ["s1"])
runs = {r["unit"]: r for r in got["runs"]}
k = runs["pack1"]
self.assertEqual((k["variant"], k["input"], k["cache_read"], k["cache_write"], k["output"]),
("K", 4, 42_000, 2_500, 950))
self.assertEqual(k["turns"], 2)
self.assertEqual(k["startup"], 3 + 20_000 + 2_000)
self.assertEqual(runs["run1"]["variant"], "K+")
self.assertEqual(runs["run1"]["stop_reason"], None) # a stopped agent is listed, not dropped
self.assertEqual([u["description"] for u in got["unattributed"]], ["load test"])
self.assertEqual(got["main"]["s1"]["cache_read"], 100)
self.assertNotIn("pack2", runs)
def test_output_never_recorded_is_derived_from_the_next_messages_input_growth(self) -> None:
# Found in session 2: a long streamed message keeps only its first usage record (output 2, no stop
# reason). The next message's input side grew by that output plus the text injected between them (tool
# results and rendered attachments), counted high at 2.5 characters per token so the output is a lower bound.
def result(text: str) -> dict[str, Any]:
return {"type": "user", "message": {"content": [{"type": "tool_result", "content": text}]}}
sub = self.proj / "s1" / "subagents"
jsonl(sub / "agent-a4.jsonl", [{"type": "user", "message": {"content": "task"}},
msg("w1", 2, 0, 10_000, 300, "tool_use"), result("x" * 400),
msg("w2", 2, 10_000, 50_000, 2, None), msg("w2", 2, 10_000, 50_000, 2, None),
result("y" * 40), msg("w3", 2, 60_000, 5_010, 20)])
(sub / "agent-a4.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "P1-K-pack3"}))
runs = {r["unit"]: r for r in cost.harvest(self.proj, cost.sessions(self.okter))["runs"]}
w = runs["pack3"]
self.assertEqual(w["output_recorded"], 300 + 2 + 20)
self.assertEqual(w["output"], 300 + (5_010 - 40 * 2 // 5) + 20)
self.assertEqual((w["output_derived_messages"], w["output_unrecorded"]), (1, 0))
self.assertEqual(w["total"], 2 + 2 + 2 + 70_000 + 65_010 + w["output"])
# the last message of a run has no next message: its output stays as recorded and is flagged
self.assertEqual((runs["run1"]["output"], runs["run1"]["output_unrecorded"]), (1, 1))
self.assertEqual(runs["pack1"]["output_derived_messages"], 0)
def test_growth_from_injected_attachments_is_not_counted_as_output(self) -> None:
# The load test's pattern: a short call, then 5133 characters of rendered attachments before the next turn.
sub = self.proj / "s1" / "subagents"
jsonl(sub / "agent-a5.jsonl", [{"type": "user", "message": {"content": "task"}},
msg("v1", 2, 0, 18_602, 16, None),
{"type": "attachment", "rendered": "z" * 5_133},
{"type": "user", "message": {"content": [{"type": "tool_result",
"content": "w" * 151}]}},
msg("v2", 2, 18_602, 1_894, 24, None)])
(sub / "agent-a5.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "P1-LOAD-x"}))
runs = {r["unit"]: r for r in cost.harvest(self.proj, cost.sessions(self.okter))["runs"]}
self.assertEqual((runs["x"]["output"], runs["x"]["output_derived_messages"]), (16 + 24, 0))
class ProjectionTest(unittest.TestCase):
def test_least_squares_on_cost_per_block_against_blocks_read(self) -> None:
p = cost.project([(600_000, 600, 0), (220_000, 200, 600)], total_blocks=3000)
self.assertAlmostEqual(p["fit"]["a"], 1000)
self.assertAlmostEqual(p["fit"]["b"], 1 / 6)
self.assertAlmostEqual(p["fit"]["value"], 3_750_000)
self.assertAlmostEqual(p["flat"], 3_075_000)
def test_one_point_has_no_growth_term(self) -> None:
p = cost.project([(600_000, 600, 0)], total_blocks=3000)
self.assertEqual(p["fit"]["b"], 0)
self.assertEqual(p["fit"]["note"], "one point, no growth term")
self.assertAlmostEqual(p["fit"]["value"], 3_000_000)
def test_stop_per_block_against_naive_per_source(self) -> None:
s = cost.stop_k(pack1_cost=600_000, pack1_blocks=600, cut_blocks=1200, pack1_sources=5, cut_sources=19)
self.assertEqual(s["per_block"], 1_200_000)
self.assertEqual(s["per_source"], 2_280_000)
self.assertTrue(s["stop"])
self.assertFalse(cost.stop_k(500_000, 600, 1200, 5, 19)["stop"]) # exactly 1 M does not stop
self.assertTrue(cost.stop_k(500_001, 600, 1200, 5, 19)["stop"])
def test_kplus_stop(self) -> None:
self.assertFalse(cost.stop_kplus(run1_cost=200_000, run1_pages=8, pages_total=40)["stop"])
self.assertTrue(cost.stop_kplus(run1_cost=200_001, run1_pages=8, pages_total=40)["stop"])
class AllocationTest(unittest.TestCase):
def test_tokens_per_source_by_block_share_and_the_two_means(self) -> None:
names = [f"s{i:02d}.md" for i in range(1, 20)]
blocks_doc = {"order": names, "blocks": [{"file": n, "start": k, "end": k} for n in names for k in (1, 2)]}
packs = [names[i:i + 5] for i in range(0, 19, 5)]
costs = {1: 1000, 2: 500, 3: 500, 4: 800}
per = cost.allocate(costs, packs, blocks_doc)
self.assertEqual(per["s01.md"], 200)
self.assertEqual(per["s16.md"], 200)
means = cost.means(per, names)
self.assertEqual(means["1-5"], 200)
self.assertEqual(means["15-19"], 180) # source 15 in pack 3 (100), 16-19 in pack 4 (200 each)
# ---------------------------------------------------------------------------------------------- step 7: side by side
class SideBySideTest(Fixture):
def setUp(self) -> None:
super().setUp()
self.begreper = self.tmp / "begreper.txt"
self.begreper.write_text("Beds\tBeds;Raised beds\nLabels\tLabels\nCompost\tCompost\n", encoding="utf-8")
self.d = self.tmp / "D"
write_page(self.d, "r-beds.md", "Raised beds", "- [Alpha](k.md)", [src("o1", "docs-en-a.md", 5, "Raised beds")])
self.l4 = self.tmp / "L4"
write_page(self.l4, "x.md", "THE raised-beds, explained!", "Beds drain.[^b1]",
[src("b1", "docs-en-a.md", 5, "Raised beds drain well.")])
write_page(self.l4, "y.md", "Compost", "Compost helps.[^b1]",
[src("b1", "docs-en-a.md", 99, "Compost")]) # a link past the end of the file
def columns(self) -> list[tuple[str, Path | None]]:
return [("K", self.bundle), ("K+", None), ("D", self.d), ("L4", self.l4)]
def test_three_concepts_in_four_columns_with_the_empty_states(self) -> None:
out = self.tmp / "side.html"
sidebyside.render(sidebyside.read_begreper(self.begreper), self.columns(), self.root, out)
html = out.read_text(encoding="utf-8")
self.assertEqual(html.count('<tr class="concept">'), 3)
self.assertEqual(html.count('<td class="cell'), 12)
self.assertIn("ikke bygget", html) # K+ stopped
self.assertIn("ikke i registeret", html) # D has no Labels register page
self.assertIn("ingen side", html) # K is built and has no Compost page
self.assertNotIn("http", html)
def test_l4_finds_a_title_with_other_case_and_punctuation(self) -> None:
found = sidebyside.find(self.l4, ("Raised beds", ("Raised beds",)), l4=True)
self.assertEqual(found, ["x.md"])
def test_the_link_check_flags_a_line_past_the_end(self) -> None:
out = self.tmp / "side.html"
sidebyside.render(sidebyside.read_begreper(self.begreper), self.columns(), self.root, out)
resolved, total = sidebyside.check(out, self.root)
self.assertEqual(total - resolved, 1)
self.assertEqual(sidebyside.main(["check", "--html", str(out), "--kilde-root", str(self.root)]), 1)
class ScriptsTest(unittest.TestCase):
SCRIPTS = (ROOT / "tests" / "p1_repo.sh", ROOT / "tests" / "p1_trial.sh")
def test_bash_syntax_and_ascii(self) -> None:
for script in self.SCRIPTS:
self.assertEqual(subprocess.run(["bash", "-n", str(script)]).returncode, 0, script.name)
script.read_bytes().decode("ascii")
def test_the_trial_check_fails_every_check_when_nothing_was_built(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
env = {**os.environ, "P1_ROOT": tmp}
res = subprocess.run(["bash", str(self.SCRIPTS[1])], env=env, capture_output=True, text=True, cwd=ROOT)
self.assertEqual(res.returncode, 1)
lines = [x for x in res.stdout.splitlines() if x.strip()]
self.assertGreaterEqual(len(lines), 8)
self.assertTrue(all(x.startswith("FAIL") for x in lines), res.stdout)
class NoModelTest(unittest.TestCase):
def test_trial_modules_import_no_network_or_model_client(self) -> None:
for path in sorted((ROOT / "evals" / "p1").glob("*.py")):
source = path.read_text(encoding="utf-8")
for banned in BANNED:
self.assertNotIn(f"import {banned}", source, path.name)
self.assertNotIn(f"from {banned}", source, path.name)
if __name__ == "__main__":
unittest.main()