okf-wiki 1.1.1
First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
commit
0fd86a23a3
227 changed files with 96077 additions and 0 deletions
120
evals/README.md
Normal file
120
evals/README.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
# evals
|
||||
|
||||
## Direction probe (R)
|
||||
|
||||
`direction_probe.py` sets up and scores the direction probe: three arms answer the same questions from the same sources.
|
||||
It calls no model. The answering is done in the user's own session, by one isolated subagent per question per arm.
|
||||
|
||||
- `filsok`: file search alone over the sources in the cut.
|
||||
- `wiki`: the okf-wiki plus file search, following the skill's router (`index.md`, the pages, the raw source at `x-locator`).
|
||||
- `astro`: a wiki built with Astro-Han's `karpathy-llm-wiki` skill (MIT) on the same sources, following its Query steps.
|
||||
|
||||
```
|
||||
python3 evals/direction_probe.py fetch-astro --sources <source dir> --cut-manifest gates/wholedoc.manifest.json \
|
||||
--project <astro project OUTSIDE the repository> --topic <topic> --collected <YYYY-MM-DD>
|
||||
python3 evals/direction_probe.py setup --run <run dir> --arm <filsok|wiki|astro> --sources <source dir> \
|
||||
--cut-manifest gates/wholedoc.manifest.json --md-set gates-local/multidoc/set-60.json --md-ids d01,d02,... \
|
||||
--wd-set gates-local/wholedoc/set-12.json [--wiki <okf-wiki bundle>] [--astro <astro project>]
|
||||
python3 evals/direction_probe.py collect --run <run dir> --arm <arm>
|
||||
python3 gates/multidoc.py gate ... --answers <run>/answers/<arm>.md.json --out <run>/results/<arm>.md.json
|
||||
python3 gates/wholedoc.py gate ... --answers <run>/answers/<arm>.wd.json --out <run>/results/<arm>.wd.json
|
||||
python3 evals/direction_probe.py harvest --run <run dir> --transcripts <subagent transcript folder>
|
||||
python3 evals/direction_probe.py score --run <run dir> --arms filsok,wiki,astro --md-ids d01,d02,...
|
||||
```
|
||||
|
||||
- **setup** copies each arm's basis into its own folder, so no arm can read another's. It writes one task file per
|
||||
question with the question, the search rules and the answer format, and never the key, the evidence or the count.
|
||||
Search runs through read-only Bash, because the subagents have no Grep or Glob tool. Every command starts with
|
||||
`cd <basis> &&` and uses a fixed list of read-only tools.
|
||||
- **harvest** sums every usage field in each task's transcript, counting each message id once, and names every tool
|
||||
call that breaks the task's rules. A breach is a fact to report, not a verdict: `echo` between two greps breaks the
|
||||
rule without reading anything outside the basis.
|
||||
- **score** filters the unchanged gates' per-question results to the probe questions. It reports per arm and group
|
||||
(multi-document, whole document, counting), compares every pair of arms, and gives the exact McNemar p. A missing
|
||||
result file is a failure, not a zero.
|
||||
|
||||
Everything the run produces (bases, tasks, answers, results) derives from the corpus and stays outside the repository
|
||||
(CLAUDE.md rule 3). Tests: `python3 -m unittest discover -s evals/tests`. Types: `mypy --strict evals`.
|
||||
|
||||
## v1-k step 9: semantic links measured on P1's pages (SC13)
|
||||
|
||||
The link pass of the v1 plan run on a stripped copy of the P1 trial's K bundle (212 concept pages, 307 links reduced to
|
||||
their anchor text, link rule 0 of 212 before the run). Cost line 7 M tokens for `link` + `orchestration-build`; the stop
|
||||
check runs after every pass. Output stays outside the repository (`~/okf/benk/wiki/v1/lenkemaaling/`, rule 3).
|
||||
|
||||
**Result: STOP after pass 1 of 5.** Measured 1,754,187 tokens at the check; flat projection 8,770,935 × 1.10 = 9,648,029
|
||||
of 7,000,000. Passes 2–4 and the closing pass were not run, and nothing was rerun.
|
||||
|
||||
| Number | Value |
|
||||
|---|---|
|
||||
| Pages with an inbound link (link rule) | 71 of 212 after pass 1 (pass 1: 71; passes 2–4 and closing: not run) |
|
||||
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
|
||||
| Max out-degree | 10 |
|
||||
| Links accepted / rejected | 304 / 0 (no reason hit) |
|
||||
| "none" answers | 9 of 104 pages |
|
||||
| Link agents | 2 (311 + 39 sentences, 91 + 13 pages), 3 turns each, start-up 47,303 each |
|
||||
| `link` | 486,256 + margin 0.10 = 534,882 (1,389 per sentence, 4,676 per page tasked) |
|
||||
| `orchestration-build` at the result | 1,849,940 over 18 turns (102,774 per turn); input per turn 80,564 + 2,255 × turn |
|
||||
| Catalogue share of link input | 5,063 characters ÷ 2.5 × 3 turns ≈ 6,076 per agent (computed from size, not fitted) |
|
||||
|
||||
Projection to collection 1 (~120 packs), orchestration at the measured `a + b × turn` restarted every session:
|
||||
|
||||
| Turns per pack | 20 turns per session | 40 | 80 |
|
||||
|---|---|---|---|
|
||||
| 4 (480 turns) | 50.0 M | 60.9 M | 82.5 M |
|
||||
| 5 (600 turns) | 62.5 M | 76.1 M | 101.3 M |
|
||||
|
||||
Link for collection 1: ~20,900 sentences (664 per 19 sources × 599) × 1,355 per sentence without the catalogue ≈ 28.4 M,
|
||||
plus a ~3,500-line catalogue (~170,000 characters at P1's 48.7 per line, ≈ 68,000 tokens) × 3 turns per agent ×
|
||||
≥ 120 agents ≈ 24.5 M. That catalogue alone is over the 60,000-character part limit, so `klink task` cannot run on it
|
||||
without a split or filter. Build 21.9–52.8 M + link ≈ 53 M + orchestration 50.0–101.3 M = 125–207 M, above 60 M.
|
||||
|
||||
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linker ran as a general-purpose agent
|
||||
with the linker's instructions verbatim (the session started in the main checkout, where `okf-wiki-linker` is not
|
||||
defined), so its start-up of 47,303 is above the ~18.6 k assumed for the linker type; pass 1 carries the session's
|
||||
start-up turns, which makes the flat projection high for passes 2–5, while the closing pass (4.36 M characters on a
|
||||
fully stripped K) is not in it.
|
||||
|
||||
## v1-k step 9b: semantic links measured with the cost cuts, collection 1 re-projected
|
||||
|
||||
Step 9 continued on the same output (pass 1 kept): packs 2–4 as ONE interval (`klink pass-before --pack 2 --to 4`
|
||||
without `--catalogue`, three linkers spawned in one turn, one `klink pass-after`), cost line 7 M tokens for `link` +
|
||||
`orchestration-build` counted on this step only: the stop check read a copy of the session's main transcript from the
|
||||
step's start marker on, because the session had first carried another order.
|
||||
|
||||
**Result: STOP before the closing pass (pass 5).** After packs 2–4 the stop line said continue (measured 3,396,621;
|
||||
flat 4,528,828 × 1.10 = 4,981,711 of 7,000,000), but the closing pass's tasks are 19 parts of 1,114,516 characters
|
||||
(5,582 candidate sentences for 87 pages without an inbound link). At this step's measured cost per linker and per
|
||||
orchestrating turn that is ~5 M + ~1 M, over the line, so it was not spawned.
|
||||
|
||||
| Number | Value |
|
||||
|---|---|
|
||||
| Pages with an inbound link (link rule) | 125 of 212 (pass 1: 71; packs 2–4: +54; closing: not run) |
|
||||
| Misplaced / broken / not to a concept page | 0 / 0 / 0 |
|
||||
| Max out-degree | 11 |
|
||||
| Links accepted / rejected | 270 / 0 in packs 2–4 (574 / 0 in all) |
|
||||
| "none" answers | 38 in packs 2–4 (47 in all) |
|
||||
| Link agents | 3 (87, 108, 119 sentences; 21,511–31,725 characters), 3 turns each, start-up 47,705 each |
|
||||
| `link` | 658,909 (2,098 per sentence; 214,054–228,310 per agent) |
|
||||
| `orchestration-build` at the result | 2,737,712 over 11 turns; input per turn 237,548 + 1,768 × turn |
|
||||
| Catalogue filter (step 9.2) on P1 | pools 146 / 174 / 212, nothing cut (largest 8,437 characters) |
|
||||
| Price of the filter at collection 1's budget | 568 of 574 accepted links stay possible (6 cut: targets far back and not among the most cited) |
|
||||
|
||||
Projection to collection 1 (~120 packs, ~20,900 sentences), orchestration at a = 80,564 (step 9, a fresh session)
|
||||
and b = 1,768 (this step), 5 turns per interval of k packs plus 4 per session; link = 105 parts × 3 turns ×
|
||||
(47,705 + 60,000 / 2.5) + 434 per sentence (measured residual) ≈ 31.7 M; build 21.9–52.8 M:
|
||||
|
||||
| k packs per interval | 20 packs per session | 40 packs per session |
|
||||
|---|---|---|
|
||||
| 5 | 68.3–99.2 M (× 1.10: 75.2–109.2) | 69.4–100.3 M (76.4–110.4) |
|
||||
| 10 | 61.4–92.3 M (67.6–101.6) | 61.0–91.9 M (67.1–101.0) |
|
||||
| 20 | 58.4–89.3 M (64.2–98.2) | 57.5–88.4 M (63.3–97.2) |
|
||||
|
||||
The closing pass is not in these sums: on P1 its candidate rule (every sentence citing an orphan's source file)
|
||||
already gives 1.1 M characters, and it grows with the collection. The upper end rests on the build's growth model
|
||||
(52.8 M, 4 points, not validated).
|
||||
|
||||
Bias: P1's pages are final versions, not per-pack states; one run (n = 1); the linkers ran as general-purpose agents with
|
||||
the linker's instructions verbatim (start-up 47,705; the `okf-wiki-linker` type is not defined in the main checkout
|
||||
where the session started), and at least one used a script to check its own anchors; a was measured in a session that
|
||||
was not fresh (238 k at its first turn), so a from step 9 is used; b is measured with an interval of 3 packs.
|
||||
540
evals/direction_probe.py
Normal file
540
evals/direction_probe.py
Normal file
|
|
@ -0,0 +1,540 @@
|
|||
"""The direction probe R (PLAN.md § Leveranser, row R): three arms answer the same questions from the same sources.
|
||||
|
||||
filsok file search alone over the sources in the cut
|
||||
wiki the okf-wiki plus file search, following the skill's router (index.md, the pages, the raw source)
|
||||
astro a wiki built with Astro-Han's karpathy-llm-wiki skill on the same sources, following its Query steps
|
||||
|
||||
Each question is answered by one isolated subagent per arm in the user's session. This module never calls a model:
|
||||
|
||||
fetch-astro the skill's Fetch step done mechanically: each source copied byte-exact under the raw template's
|
||||
header into <project>/raw/<topic>/<name> (the sources are local, clean Markdown already)
|
||||
setup one arm's basis (a copy, so no arm reads another's folder) and one task file per question, which
|
||||
carries the question and never the answer key, the evidence or the count
|
||||
collect the per-question answer files into the answers files the gates read
|
||||
harvest tokens (every usage field, each message id once) and isolation breaches from the transcripts
|
||||
score the gates' per-question results for the probe questions only: per arm and group, and in pairs
|
||||
|
||||
The gates are run unchanged (`gates/multidoc.py gate`, `gates/wholedoc.py gate`, each with --out).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import sys
|
||||
from itertools import combinations
|
||||
from math import comb
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
ARMS = ("filsok", "wiki", "astro")
|
||||
GROUPS = ("flerdokument", "helt dokument", "telling")
|
||||
MD_SCHEMA = "okf-wiki/multidoc-set/1"
|
||||
WD_SCHEMA = "okf-wiki/wholedoc-set/1"
|
||||
ALLOWED_TOOLS = ("Read", "Grep", "Glob", "Bash", "Write", "SubagentHandback")
|
||||
READ_ONLY = ("grep", "egrep", "fgrep", "rg", "find", "ls", "wc", "head", "tail", "sed", "cat", "sort", "uniq", "cut",
|
||||
"tr", "nl")
|
||||
SEPARATORS = ("|", "&&", ";", "||")
|
||||
FIND_ACTIONS = ("-exec", "-execdir", "-ok", "-okdir", "-delete", "-fprint", "-fprint0", "-fprintf", "-fls")
|
||||
USAGE_FIELDS = ("input_tokens", "cache_creation_input_tokens", "cache_read_input_tokens", "output_tokens")
|
||||
|
||||
|
||||
class ProbeError(Exception):
|
||||
"""The command cannot run; reported as FAIL with exit 1."""
|
||||
|
||||
|
||||
def load_json(path: Path) -> Any:
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise ProbeError(f"cannot read {path}: {exc}") from exc
|
||||
|
||||
|
||||
def write_json(path: Path, data: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def copy_names(sources: Path, names: list[str], target: Path) -> None:
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
for name in names:
|
||||
src = sources / name
|
||||
if not src.is_file():
|
||||
raise ProbeError(f"source {src} is missing")
|
||||
shutil.copyfile(src, target / name)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- fetch-astro
|
||||
|
||||
|
||||
def fetch_astro(sources: Path, names: list[str], project: Path, topic: str, collected: str) -> list[Path]:
|
||||
"""The Fetch step of karpathy-llm-wiki, per its raw template, with the source text preserved byte-exact."""
|
||||
folder = project / "raw" / topic
|
||||
targets = [folder / name for name in names]
|
||||
existing = [str(t) for t in targets if t.exists()]
|
||||
if existing:
|
||||
raise ProbeError(f"raw files exist already and are never overwritten: {', '.join(existing)}")
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
for name, target in zip(names, targets):
|
||||
src = sources / name
|
||||
if not src.is_file():
|
||||
raise ProbeError(f"source {src} is missing")
|
||||
header = (f"# {Path(name).stem}\n\n> Source: kilde/{name}\n> Collected: {collected}\n"
|
||||
"> Published: Unknown\n\n")
|
||||
target.write_bytes(header.encode("utf-8") + src.read_bytes())
|
||||
return targets
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- setup
|
||||
|
||||
|
||||
SEARCH = {
|
||||
"filsok": """BASIS holds the source files and nothing else. Search them with grep and find (Bash, as above), and read
|
||||
what you find with Read.""",
|
||||
"wiki": """BASIS holds `wiki/`, a knowledge wiki compiled from the sources, and `kilde/`, the source files. A page
|
||||
names its source as `/kilde/<name>`, which is the file `BASIS/kilde/<name>`. Follow this order:
|
||||
1. Read `BASIS/wiki/index.md` first.
|
||||
2. Read the pages the index points to.
|
||||
3. Open the raw source behind a footnote (`resource` at `x-locator`) when the quote has to be checked or when the
|
||||
page does not say enough.
|
||||
You may also search `wiki/` and `kilde/` with grep and find (Bash, as above) at any point.""",
|
||||
"astro": """BASIS holds `wiki/`, a knowledge wiki compiled from the sources with the karpathy-llm-wiki skill, and
|
||||
`raw/`, the source files it was compiled from (each raw file starts with a short metadata header; the source text
|
||||
follows it unchanged). Follow the skill's Query steps:
|
||||
1. Read `BASIS/wiki/index.md` to locate candidate articles, then full-text search `wiki/` with the topic's key terms
|
||||
and their synonyms (grep through Bash, as above).
|
||||
2. Read the articles you found and synthesize an answer.
|
||||
3. Prefer wiki content over your own training knowledge.
|
||||
Each article links the raw files it was compiled from in its `Raw` field. Open those raw files when you need an exact
|
||||
value or a verbatim quote.""",
|
||||
}
|
||||
|
||||
CITE_NAME = {
|
||||
"filsok": "the bare name of a source file in BASIS",
|
||||
"wiki": "the bare name of a source file in `BASIS/kilde`",
|
||||
"astro": "the bare name of a raw file in `BASIS/raw/<topic>/`",
|
||||
}
|
||||
|
||||
|
||||
def render_task(arm: str, basis: Path, qid: str, question: str, is_count: bool, answer_path: Path) -> str:
|
||||
count_line = '"count": <integer>, ' if is_count else ""
|
||||
count_rule = ("\n- `count`: the number the question asks for, as a JSON integer. Say in `answer` how you counted."
|
||||
if is_count else "")
|
||||
return f"""# Answer one question
|
||||
|
||||
BASIS: `{basis}`
|
||||
|
||||
You answer ONE question from the files in BASIS. Read nothing outside BASIS apart from this task file: no other
|
||||
folder, no web, no earlier work. Answer only from what the files say, and say plainly when they do not say something.
|
||||
|
||||
Tools: Read (a file inside BASIS), Bash for read-only search, and Write once, for the answer file. No other tool.
|
||||
Every Bash command starts with `cd {basis} && ` and then uses only {', '.join(READ_ONLY)}, joined by `|` or `&&`,
|
||||
with paths relative to BASIS: never `..`, `~` or a path outside BASIS, no `$` expansion, no output redirection except
|
||||
`2>/dev/null`, no `find -exec` or `-delete`, no `sed -i`. For example: `cd {basis} && grep -rn "term" .`
|
||||
|
||||
## How to search
|
||||
|
||||
{SEARCH[arm]}
|
||||
|
||||
## The question
|
||||
|
||||
{question}
|
||||
|
||||
## The answer
|
||||
|
||||
Write one JSON file to `{answer_path}`:
|
||||
|
||||
```json
|
||||
{{"id": "{qid}", "answer": "<your answer in prose>", {count_line}"citations": [{{"file": "<name>", "quote": "<text>"}}]}}
|
||||
```
|
||||
|
||||
- `answer`: answer every part of the question.{count_rule}
|
||||
- `citations`: one or more per source file your answer relies on, and cite every such file.
|
||||
- `file`: {CITE_NAME[arm]}, for example `notes.md`: never a path and never a wiki page.
|
||||
- `quote`: text copied character for character from that file (only whitespace may differ), long enough to carry
|
||||
the fact, for example the whole line or sentence.
|
||||
|
||||
When the file is written, reply with the one word `done`.
|
||||
"""
|
||||
|
||||
|
||||
def probe_questions(md_set: Path, md_ids: list[str], wd_set: Path) -> list[tuple[str, dict[str, Any]]]:
|
||||
md, wd = load_json(md_set), load_json(wd_set)
|
||||
if not isinstance(md, dict) or md.get("schema") != MD_SCHEMA:
|
||||
raise ProbeError(f"{md_set} is not a set with schema {MD_SCHEMA}")
|
||||
if not isinstance(wd, dict) or wd.get("schema") != WD_SCHEMA:
|
||||
raise ProbeError(f"{wd_set} is not a set with schema {WD_SCHEMA}")
|
||||
by_id = {q["id"]: q for q in md["questions"]}
|
||||
missing = [i for i in md_ids if i not in by_id]
|
||||
if missing:
|
||||
raise ProbeError(f"probe ids missing from {md_set}: {', '.join(missing)}")
|
||||
return [("md", by_id[i]) for i in md_ids] + [("wd", q) for q in wd["questions"]]
|
||||
|
||||
|
||||
def setup_arm(run: Path, arm: str, sources: Path, names: list[str], md_set: Path, md_ids: list[str], wd_set: Path,
|
||||
wiki: Path | None = None, astro: Path | None = None) -> list[Path]:
|
||||
if arm not in ARMS:
|
||||
raise ProbeError(f"unknown arm {arm!r}; the arms are {', '.join(ARMS)}")
|
||||
if arm == "wiki" and wiki is None:
|
||||
raise ProbeError("the wiki arm needs --wiki")
|
||||
if arm == "astro" and astro is None:
|
||||
raise ProbeError("the astro arm needs --astro")
|
||||
questions = probe_questions(md_set, md_ids, wd_set)
|
||||
basis = run / "arms" / arm
|
||||
if basis.exists():
|
||||
raise ProbeError(f"{basis} exists already; an arm is set up once, from fresh")
|
||||
if arm == "filsok":
|
||||
copy_names(sources, names, basis)
|
||||
elif arm == "wiki" and wiki is not None:
|
||||
shutil.copytree(wiki, basis / "wiki")
|
||||
copy_names(sources, names, basis / "kilde")
|
||||
elif astro is not None:
|
||||
shutil.copytree(astro / "raw", basis / "raw")
|
||||
shutil.copytree(astro / "wiki", basis / "wiki")
|
||||
tasks = []
|
||||
for group, q in questions:
|
||||
key = f"{group}-{q['id']}"
|
||||
task = run / "tasks" / arm / f"{key}.md"
|
||||
answer = run / "answers" / arm / f"{key}.json"
|
||||
task.parent.mkdir(parents=True, exist_ok=True)
|
||||
task.write_text(render_task(arm, basis, q["id"], q["question"], q.get("type") == "count", answer),
|
||||
encoding="utf-8")
|
||||
tasks.append(task)
|
||||
(run / "answers" / arm).mkdir(parents=True, exist_ok=True)
|
||||
return tasks
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- collect
|
||||
|
||||
|
||||
def collect(run: Path, arm: str) -> tuple[Path, Path, list[str]]:
|
||||
"""The per-question answers into one answers file per gate. The id comes from the file name."""
|
||||
folder = run / "answers" / arm
|
||||
gathered: dict[str, list[dict[str, Any]]] = {"md": [], "wd": []}
|
||||
problems = []
|
||||
for path in sorted(folder.glob("*.json")):
|
||||
group, _, qid = path.stem.partition("-")
|
||||
if group not in gathered or not qid:
|
||||
problems.append(f"{path.name}: not a probe answer name")
|
||||
continue
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
|
||||
data = None
|
||||
if not isinstance(data, dict):
|
||||
problems.append(f"{path.name}: not a JSON object")
|
||||
continue
|
||||
gathered[group].append({**data, "id": qid})
|
||||
md_path, wd_path = run / "answers" / f"{arm}.md.json", run / "answers" / f"{arm}.wd.json"
|
||||
write_json(md_path, {"arm": arm, "answers": gathered["md"]})
|
||||
write_json(wd_path, {"arm": arm, "answers": gathered["wd"]})
|
||||
return md_path, wd_path, problems
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- harvest
|
||||
|
||||
|
||||
def first_prompt(lines: list[dict[str, Any]]) -> tuple[str, str]:
|
||||
for line in lines:
|
||||
if line.get("type") == "user":
|
||||
content = line.get("message", {}).get("content")
|
||||
if isinstance(content, list):
|
||||
content = " ".join(b.get("text", "") for b in content if isinstance(b, dict))
|
||||
return (content if isinstance(content, str) else ""), str(line.get("timestamp", ""))
|
||||
return "", ""
|
||||
|
||||
|
||||
def inside(path: object, root: Path) -> bool:
|
||||
if not isinstance(path, str) or not os.path.isabs(path):
|
||||
return False
|
||||
norm = Path(os.path.normpath(path))
|
||||
return norm == root or root in norm.parents
|
||||
|
||||
|
||||
def bash_breach(command: str, basis: Path) -> str | None:
|
||||
"""None when the command is read-only search that starts with `cd <basis> &&` and stays inside the basis."""
|
||||
if "`" in command or re.search(r"\$[A-Za-z_{(]", command):
|
||||
return "Bash with an expansion"
|
||||
try:
|
||||
lexer = shlex.shlex(re.sub(r"\s2>/dev/null(?=\s|$)", "", command), posix=True, punctuation_chars="|&;<>()")
|
||||
lexer.whitespace_split = True
|
||||
tokens = list(lexer)
|
||||
except ValueError:
|
||||
return "Bash that cannot be parsed"
|
||||
segments: list[list[str]] = [[]]
|
||||
for token in tokens:
|
||||
if token in SEPARATORS:
|
||||
segments.append([])
|
||||
elif token and set(token) <= set("&<>()"):
|
||||
return f"Bash with {token!r}"
|
||||
else:
|
||||
segments[-1].append(token)
|
||||
first = segments[0]
|
||||
if len(first) != 2 or first[0] != "cd" or not inside(first[1], basis):
|
||||
return "Bash that does not start with cd into the basis"
|
||||
for segment in segments[1:]:
|
||||
if not segment or segment[0] not in READ_ONLY:
|
||||
return f"Bash command not allowed: {segment[0] if segment else '(empty)'}"
|
||||
for token in segment[1:]:
|
||||
value = token.split("=", 1)[1] if token.startswith("-") and "=" in token else token
|
||||
if segment[0] == "find" and token in FIND_ACTIONS:
|
||||
return f"Bash find action {token}"
|
||||
if segment[0] == "sed" and (token.startswith("-i") or token.startswith("--in-place")):
|
||||
return "Bash sed -i"
|
||||
if value.startswith("~") or ".." in value.split("/"):
|
||||
return f"Bash path outside the basis: {token}"
|
||||
if value.startswith("/") and value != "/dev/null" and not inside(value, basis):
|
||||
return f"Bash path outside the basis: {token}"
|
||||
return None
|
||||
|
||||
|
||||
def breaches_of(tools: list[tuple[str, dict[str, Any]]], basis: Path, task: Path, answer: Path) -> list[str]:
|
||||
found = []
|
||||
for name, inp in tools:
|
||||
if name not in ALLOWED_TOOLS:
|
||||
found.append(f"{name}: tool not allowed")
|
||||
elif name == "Read":
|
||||
path = inp.get("file_path")
|
||||
if not (inside(path, basis) or (isinstance(path, str) and os.path.normpath(path) == str(task))):
|
||||
found.append(f"Read outside the basis: {path}")
|
||||
elif name in ("Grep", "Glob"):
|
||||
path = inp.get("path")
|
||||
pattern = inp.get("pattern", "")
|
||||
if path is None:
|
||||
found.append(f"{name} without a path (searches the working directory)")
|
||||
elif not inside(path, basis):
|
||||
found.append(f"{name} outside the basis: {path}")
|
||||
elif name == "Glob" and isinstance(pattern, str) and os.path.isabs(pattern) and not inside(pattern, basis):
|
||||
found.append(f"Glob pattern outside the basis: {pattern}")
|
||||
elif name == "Bash":
|
||||
breach = bash_breach(str(inp.get("command", "")), basis)
|
||||
if breach:
|
||||
found.append(breach)
|
||||
elif name == "Write" and inp.get("file_path") != str(answer):
|
||||
found.append(f"Write to another file: {inp.get('file_path')}")
|
||||
return found
|
||||
|
||||
|
||||
def read_transcript(path: Path) -> list[dict[str, Any]]:
|
||||
lines = []
|
||||
for raw in path.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
line = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(line, dict):
|
||||
lines.append(line)
|
||||
return lines
|
||||
|
||||
|
||||
def harvest(transcripts: Path, run: Path) -> dict[str, dict[str, Any]]:
|
||||
"""Per task: all tokens its transcript records (each message id once), and every isolation breach."""
|
||||
pattern = re.compile(re.escape(str(run / "tasks")) + r"/(" + "|".join(ARMS) + r")/((?:md|wd)-[A-Za-z0-9]+)\.md")
|
||||
rows: dict[str, dict[str, Any]] = {}
|
||||
for path in sorted(transcripts.glob("*.jsonl")):
|
||||
lines = read_transcript(path)
|
||||
prompt, started = first_prompt(lines)
|
||||
match = pattern.search(prompt)
|
||||
if match is None:
|
||||
continue
|
||||
arm, key = match.group(1), match.group(2)
|
||||
usage: dict[str, dict[str, int]] = {}
|
||||
tools: list[tuple[str, dict[str, Any]]] = []
|
||||
for line in lines:
|
||||
message = line.get("message", {})
|
||||
if line.get("type") != "assistant" or not isinstance(message, dict):
|
||||
continue
|
||||
mid = str(message.get("id", ""))
|
||||
fields = message.get("usage") or {}
|
||||
seen = usage.setdefault(mid, {})
|
||||
for f in USAGE_FIELDS:
|
||||
value = fields.get(f)
|
||||
if isinstance(value, int):
|
||||
seen[f] = max(seen.get(f, 0), value)
|
||||
for block in message.get("content") or []:
|
||||
if isinstance(block, dict) and block.get("type") == "tool_use":
|
||||
tools.append((str(block.get("name")), block.get("input") or {}))
|
||||
row = {
|
||||
"tokens": sum(sum(v.values()) for v in usage.values()),
|
||||
"breaches": breaches_of(tools, run / "arms" / arm, run / "tasks" / arm / f"{key}.md",
|
||||
run / "answers" / arm / f"{key}.json"),
|
||||
"transcript": path.name,
|
||||
"started": started,
|
||||
}
|
||||
name = f"{arm}/{key}"
|
||||
previous = rows.get(name)
|
||||
runs = 1 + (previous["runs"] if previous else 0)
|
||||
if previous is None or started >= previous["started"]:
|
||||
rows[name] = {**row, "runs": runs}
|
||||
else:
|
||||
previous["runs"] = runs
|
||||
for row in rows.values():
|
||||
row.pop("started")
|
||||
return rows
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- score
|
||||
|
||||
|
||||
def gate_results(path: Path) -> dict[str, dict[str, Any]]:
|
||||
if not path.is_file():
|
||||
raise ProbeError(f"no gate results at {path}; an arm that was not scored is a failure, not a zero")
|
||||
data = load_json(path)
|
||||
return {r["id"]: r for r in data.get("results", [])}
|
||||
|
||||
|
||||
def mcnemar_exact(only_a: int, only_b: int) -> float:
|
||||
"""Two-sided exact McNemar: the binomial test over the discordant pairs, p = 0.5."""
|
||||
n, k = only_a + only_b, min(only_a, only_b)
|
||||
if n == 0:
|
||||
return 1.0
|
||||
return min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / float(2 ** n))
|
||||
|
||||
|
||||
def score(run: Path, arms: list[str], md_ids: list[str]) -> dict[str, Any]:
|
||||
tokens_path = run / "tokens.json"
|
||||
tokens = load_json(tokens_path) if tokens_path.is_file() else {}
|
||||
correct: dict[str, dict[str, dict[str, bool]]] = {}
|
||||
table: dict[str, Any] = {"arms": {}, "pairs": {}, "mcnemar": {}}
|
||||
for arm in arms:
|
||||
md = gate_results(run / "results" / f"{arm}.md.json")
|
||||
wd = gate_results(run / "results" / f"{arm}.wd.json")
|
||||
by_group: dict[str, dict[str, bool]] = {g: {} for g in GROUPS}
|
||||
answer_ok = source_ok = 0
|
||||
for qid in md_ids:
|
||||
r = md.get(qid, {})
|
||||
by_group["flerdokument"][f"md-{qid}"] = bool(r.get("correct"))
|
||||
answer_ok += bool(r.get("answer_ok"))
|
||||
source_ok += bool(r.get("source_ok"))
|
||||
for qid, r in wd.items():
|
||||
group = "telling" if r.get("type") == "count" else "helt dokument"
|
||||
by_group[group][f"wd-{qid}"] = bool(r.get("correct"))
|
||||
answer_ok += bool(r.get("answer_ok"))
|
||||
source_ok += bool(r.get("source_ok"))
|
||||
correct[arm] = by_group
|
||||
n = sum(len(v) for v in by_group.values())
|
||||
own = [v for k, v in tokens.items() if k.startswith(f"{arm}/")]
|
||||
table["arms"][arm] = {
|
||||
"groups": {g: [sum(v.values()), len(v)] for g, v in by_group.items()},
|
||||
"total": [sum(sum(v.values()) for v in by_group.values()), n],
|
||||
"answer_ok": answer_ok,
|
||||
"source_ok": source_ok,
|
||||
"tokens": {"sum": sum(int(v["tokens"]) for v in own), "questions": len(own)},
|
||||
"breaches": sum(len(v["breaches"]) for v in own),
|
||||
}
|
||||
for a, b in combinations(arms, 2):
|
||||
a, b = (b, a) if b == "wiki" else (a, b)
|
||||
pair: dict[str, dict[str, int]] = {}
|
||||
for g in GROUPS:
|
||||
counts = {"both": 0, f"only {a}": 0, f"only {b}": 0, "neither": 0}
|
||||
for key, ok_a in correct[a][g].items():
|
||||
ok_b = correct[b][g].get(key, False)
|
||||
label = "both" if ok_a and ok_b else f"only {a}" if ok_a else f"only {b}" if ok_b else "neither"
|
||||
counts[label] += 1
|
||||
pair[g] = counts
|
||||
table["pairs"][f"{a} vs {b}"] = pair
|
||||
only_a, only_b = sum(c[f"only {a}"] for c in pair.values()), sum(c[f"only {b}"] for c in pair.values())
|
||||
table["mcnemar"][f"{a} vs {b}"] = {f"only {a}": only_a, f"only {b}": only_b, "p": mcnemar_exact(only_a, only_b)}
|
||||
table["per_question"] = {arm: {k: v for g in GROUPS for k, v in correct[arm][g].items()} for arm in arms}
|
||||
return table
|
||||
|
||||
|
||||
def render_table(table: dict[str, Any]) -> str:
|
||||
head = "| arm | " + " | ".join(f"{g} (N)" for g in GROUPS) + " | total | answer ok | source ok | tokens/question |"
|
||||
lines = [head, "|" + "---|" * (len(GROUPS) + 5)]
|
||||
for arm, row in table["arms"].items():
|
||||
t = row["tokens"]
|
||||
per_q = f"{t['sum'] / t['questions'] / 1e6:.2f} M" if t["questions"] else "not harvested"
|
||||
cells = [f"{c} of {n}" for c, n in (row["groups"][g] for g in GROUPS)]
|
||||
total = row["total"]
|
||||
lines.append(f"| {arm} | " + " | ".join(cells) + f" | {total[0]} of {total[1]} | {row['answer_ok']} | "
|
||||
f"{row['source_ok']} | {per_q} |")
|
||||
lines.append("")
|
||||
for name, pair in table["pairs"].items():
|
||||
lines.append(f"{name}: " + "; ".join(f"{g}: " + ", ".join(f"{k} {v}" for k, v in c.items())
|
||||
for g, c in pair.items()))
|
||||
m = table["mcnemar"][name]
|
||||
lines.append(f"{name}, all groups: " + ", ".join(f"{k} {v}" for k, v in m.items() if k != "p")
|
||||
+ f"; exact McNemar p = {m['p']:.3g}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- main
|
||||
|
||||
|
||||
def cut_names(manifest: Path) -> list[str]:
|
||||
names = load_json(manifest).get("cut", {}).get("names")
|
||||
if not isinstance(names, list) or not names:
|
||||
raise ProbeError(f"{manifest} records no cut names")
|
||||
return [str(n) for n in names]
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
f = sub.add_parser("fetch-astro")
|
||||
f.add_argument("--sources", required=True)
|
||||
f.add_argument("--cut-manifest", required=True)
|
||||
f.add_argument("--project", required=True)
|
||||
f.add_argument("--topic", required=True)
|
||||
f.add_argument("--collected", required=True)
|
||||
s = sub.add_parser("setup")
|
||||
s.add_argument("--run", required=True)
|
||||
s.add_argument("--arm", required=True)
|
||||
s.add_argument("--sources", required=True)
|
||||
s.add_argument("--cut-manifest", required=True)
|
||||
s.add_argument("--md-set", required=True)
|
||||
s.add_argument("--md-ids", required=True, help="comma-separated")
|
||||
s.add_argument("--wd-set", required=True)
|
||||
s.add_argument("--wiki")
|
||||
s.add_argument("--astro")
|
||||
c = sub.add_parser("collect")
|
||||
c.add_argument("--run", required=True)
|
||||
c.add_argument("--arm", required=True)
|
||||
h = sub.add_parser("harvest")
|
||||
h.add_argument("--run", required=True)
|
||||
h.add_argument("--transcripts", required=True)
|
||||
r = sub.add_parser("score")
|
||||
r.add_argument("--run", required=True)
|
||||
r.add_argument("--arms", required=True, help="comma-separated")
|
||||
r.add_argument("--md-ids", required=True, help="comma-separated")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "fetch-astro":
|
||||
written = fetch_astro(Path(args.sources), cut_names(Path(args.cut_manifest)), Path(args.project),
|
||||
args.topic, args.collected)
|
||||
print(f"raw files: {len(written)}")
|
||||
elif args.command == "setup":
|
||||
tasks = setup_arm(Path(args.run), args.arm, Path(args.sources), cut_names(Path(args.cut_manifest)),
|
||||
Path(args.md_set), args.md_ids.split(","), Path(args.wd_set),
|
||||
Path(args.wiki) if args.wiki else None, Path(args.astro) if args.astro else None)
|
||||
print("\n".join(str(t) for t in tasks))
|
||||
elif args.command == "collect":
|
||||
md_path, wd_path, problems = collect(Path(args.run), args.arm)
|
||||
print(f"{md_path}\n{wd_path}")
|
||||
for problem in problems:
|
||||
print(f"problem: {problem}")
|
||||
elif args.command == "harvest":
|
||||
rows = harvest(Path(args.transcripts), Path(args.run))
|
||||
write_json(Path(args.run) / "tokens.json", rows)
|
||||
print(f"tasks harvested: {len(rows)}; breaches: {sum(len(r['breaches']) for r in rows.values())}")
|
||||
else:
|
||||
table = score(Path(args.run), args.arms.split(","), args.md_ids.split(","))
|
||||
write_json(Path(args.run) / "table.json", table)
|
||||
print(render_table(table))
|
||||
except ProbeError as exc:
|
||||
print(f"FAIL: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
125
evals/p1/README.md
Normal file
125
evals/p1/README.md
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
# P1 method trial (TRIAL, branch `p1-metodeprove`, not v1 code)
|
||||
|
||||
The P1 method trial builds the 19 sources of the cut (the sources behind d01 d02 d03 d04 d37 d46 d53) three ways, in one
|
||||
fixed order, and measures cost and quality side by side, so the operator can choose the build method for v1 by seeing
|
||||
small real examples (PLAN.md § Rammer, order `20261002T094056Z-495685981`, plan v3).
|
||||
|
||||
- **K**: the fragment compiler. Code splits the sources into blocks, a narrow trial agent (`p1-compiler`) writes short
|
||||
fragments per pack of five sources, and code resolves concept identity and assembles the pages.
|
||||
- **K+**: a copy of K plus one consolidation pass (`p1-consolidator`) that rewrites pages with three or more fragments
|
||||
into flowing text, keeping every footnote label.
|
||||
- **D**: no model. A generated map of each source and a code-built concept register.
|
||||
|
||||
No module here calls a model. The model work is done in the operator's session by the two trial agent types under
|
||||
`.claude/agents/`. Every output lives outside the repository, under `<wiki>/prove-p1/` (CLAUDE.md rule 3); this README
|
||||
records numbers only.
|
||||
|
||||
## Modules
|
||||
|
||||
Run each as `PYTHONPATH=evals:. python3 -m p1.<module> ...`; each module's docstring is its usage.
|
||||
|
||||
| Module | What it does |
|
||||
|---|---|
|
||||
| `split.py` | the fixed source order, blocks with line spans, the block total, the three side-by-side concepts |
|
||||
| `dmap.py` | D: map pages and register pages |
|
||||
| `kpack.py` | K: the skeleton from every docs page, pack input parts, fragment validation, concept identity |
|
||||
| `assemble.py` | K's pages from accepted fragments; K+'s copy, tasks and apply |
|
||||
| `measure.py` | the audit record, coverage of source blocks, structure counts |
|
||||
| `cost.py` | tokens from the run sessions' transcripts, tokens per source, projection, the stop rule |
|
||||
| `sidebyside.py` | the side-by-side page and its link check |
|
||||
|
||||
Output files under `prove-p1/`: `rekkefolge.txt`, `blocks.json`, `begreper.txt`, `okter.txt`, `D/`, `K/`, `K+/`,
|
||||
`resultat.json`, `struktur-<V>.json`, `dekning-<V>.json`, `kost.json`, `side-om-side.html`, `rapport.md`.
|
||||
|
||||
Tests: `python3 -m unittest discover -s evals/tests -p 'test_p1_*.py'`. Checks: `bash tests/p1_repo.sh` (repository)
|
||||
and `bash tests/p1_trial.sh` (the trial's outputs; red until the runs are done).
|
||||
|
||||
## Results
|
||||
|
||||
Numbers only, filled in by the run steps (plan steps 8-11).
|
||||
|
||||
### Run 0 (step 8, 02.10.2026)
|
||||
|
||||
- Order: 19 sources (3 docs pages, then 16 release notes by version), equal to the files cited by the L4 cut.
|
||||
- Blocks: 1221 in the 19 sources (docs pages 599, release notes 622; pack 1 = sources 1-5 = 602 blocks); 33727 blocks
|
||||
in all 599 files of `kilde/` (the D12 projection base).
|
||||
- Blocks per source in order: 59, 308, 232, 1, 2, 22, 35, 76, 21, 25, 57, 35, 34, 52, 33, 22, 46, 53, 108.
|
||||
- Load test (the empty baseline run, not counted toward any cap): both trial types spawned and wrote under `prove-p1/`,
|
||||
so the agents' work directory is `prove-p1/` (no `runtime/p1` fallback). `p1-compiler`: startup 18604, 2 turns,
|
||||
39142 tokens in all; `p1-consolidator`: startup 18167, 2 turns, 38270 tokens in all.
|
||||
- D: 254 pages (19 map, 235 register), 0 model tokens. Auditor exit 0, 0 findings. Criterion 1: 0 of 0 (bodies are
|
||||
headings and link lists only, D8, by construction). Coverage 0 of 1221 blocks: by construction, because D cites no
|
||||
source in a body footnote (D21 counts body-cited entries only; D's locators live in `sources[]`). Duplicate pairs 9
|
||||
(5 by name, 4 by title Jaccard), merge candidates 0, islands 0, concept test 2 held by 254 pages, test 4 by 19.
|
||||
|
||||
### Run K (step 9, 02.10.2026)
|
||||
|
||||
- Skeleton: 3493 index entries from the titles and level 2-3 headings of all 198 docs pages (0 tokens).
|
||||
- The cost harvester was fixed twice during the run, each with a failing test first (`77b16ab`, `0a50286`): a long
|
||||
streamed message keeps only its first usage record in the transcript (output 2, no stop reason), so output was
|
||||
undercounted (373 recorded for pack 1's 336 fragments). Output is now derived from the next message's input growth,
|
||||
less the injected text counted high (2.5 characters per token), so every K output below is a lower bound. Checked on
|
||||
the load test's known short call, which keeps its recorded value. The recorded value stays beside it in `kost.json`.
|
||||
- K: per pack (blocks / index lines sent of 3493 / input parts / fragments accepted of proposed / rejected / accepted
|
||||
with a concept not among the lines sent / turns / tokens):
|
||||
- pack 1: 602 / 230 / 4 / 336 of 336 / 0 / 0 / 3 / 372996
|
||||
- pack 2: 179 / 117 / 1 / 87 of 87 / 0 / 0 / 3 / 125716
|
||||
- pack 3: 211 / 125 / 1 / 108 of 108 / 0 / 0 / 3 / 141320
|
||||
- pack 4: 229 / 127 / 1 / 119 of 119 / 0 / 0 / 3 / 153494
|
||||
- No fragment was rejected for any reason (incl. `release-note-new` 0); no new concept was proposed, so all 650
|
||||
accepted fragments enrich skeleton entries. Merge candidates 0.
|
||||
- Stop line after pack 1: projected per block 756525 (naive per source 1417385), cap 1000000: continue.
|
||||
- K total: 793526 tokens in 4 runs and 12 turns (input 24, cache read 228534, cache write 429550, output 135418);
|
||||
startup 74596 (4 x about 18.6 k). Tokens per source, mean of sources 1-5: 74599; of sources 15-19: 35119.
|
||||
- Projected to all 599 sources (33727 blocks): fit 52842866 (4 points, growth term b = 0.0554, not validated);
|
||||
flat 21919125.
|
||||
- Auditor exit 0, 0 findings. Criterion 1: 664 of 664. Quotes: 650 entries, 160 supplied by code, 0 on a structural
|
||||
line. Precision of body-cited entries: line 591, block 59, section 0, document 0.
|
||||
- Coverage: 623 of 1221 blocks. Per position in the pack (covered of blocks, n packs): 1: 85 of 160 (4); 2: 241 of 424
|
||||
(4); 3: 181 of 395 (4); 4: 83 of 182 (4); 5: 33 of 60 (3). A direction signal, n = 3-4.
|
||||
- Structure: 212 concept pages, 3281 skeleton entries unused. Concepts per source in order: 11, 61, 31, 1, 0, 5, 13,
|
||||
15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11. Duplicate pairs 3 (all title Jaccard), islands 63, documents per page mean
|
||||
1.65 (max 8), version numbers in running text 5 of 664 claims, concept test 2 held by 212, test 4 by 73.
|
||||
- Seen in the compiler's own reports: in one pack it wrote numbers as words to avoid the quote rule (D5 requires a
|
||||
quote when the text holds a digit); the compilers' own fragment counts differed from the files (the files count).
|
||||
|
||||
### Run K+ (step 10, 02.10.2026)
|
||||
|
||||
- K+: a copy of K; 76 pages carry 3 or more fragments, which gives 10 consolidation runs of up to 8 pages.
|
||||
- Run 1: 8 pages, applied 8, refused 0. 109651 tokens (input 8, cache read 72768, cache write 29909, output 6966, a lower
|
||||
bound), 4 turns, startup 18176. The consolidator took one more turn than its instructions allow (it read its
|
||||
output back).
|
||||
- Stop line after run 1: 109651 x 76 / 8 = 1041685 > cap 1000000: STOP. Runs 2-10 were not run; K+ stands as K with
|
||||
8 of 76 pages consolidated.
|
||||
- K+ total = K + consolidation = 793526 + 109651 = 903177 tokens.
|
||||
- Auditor exit 0, 0 findings. Criterion 1: 662 of 662 (2 fewer claims than K: sentences joined). Coverage 623 of 1221,
|
||||
precision and quotes as K (line 591, block 59; 160 code quotes, 0 structural). 212 pages, 3 duplicate pairs,
|
||||
63 islands, version numbers in running text 5 of 662.
|
||||
|
||||
### Step 11: the side-by-side table (02.10.2026)
|
||||
|
||||
Corrections after run 0, each its own commit with a failing test first: the backticked-term length is checked after
|
||||
trimming (`26f521f`) and the concept pick counts a document only where the term stands as a term (`09a50b2`). The
|
||||
first pick and D built before `26f521f` are kept in `prove-p1/forkastet/`; D was rebuilt (252 pages: 19 map, 233
|
||||
register; 9 duplicate pairs; auditor exit 0) and the concepts were picked again by the same rule.
|
||||
|
||||
| variant | built to source | concepts (total / per source 1-5, 15-19 / test 2 / test 4) | skeleton unused | merge candidates | coverage | tokens per source (input / cache read / cache write / output / startup) | 1-5 | 15-19 | turns / subagents | projected 599 (fit / flat) | criterion 1 | fragments accepted / code quotes / structural quotes | precision (line / block / section / document) | pages / islands / duplicates |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| K | 19 of 19 | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | 1 / 12028 / 22608 / 7127 / 3926 | 74599 | 35119 | 12 / 4 | 52842866 / 21919125 | 664 of 664 | 650 of 650 / 160 / 0 | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
|
||||
| K+ | 19 of 19, consolidation stopped after run 1 (8 of 76 pages) | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | consolidation 109651 for 8 pages; K+ total = K + consolidation = 903177 | - | - | 4 / 1 | not computed: stopped after 1 of 10 runs, and the pages to consolidate at 599 sources are not measured | 662 of 662 | as K | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
|
||||
| D | 19 of 19 | 0 concept pages (252 map and register) / 0, 0 by construction / 252 / 19 | - | 0 | 0 of 1221 (by construction) | 0 | 0 | 0 | 0 / 0 | 0 | 0 of 0 (by construction) | - | - | 252 / 0 / 9 |
|
||||
| L4 (structure counted today) | 19 of 19 | 151 / 33, 33 / 151 / 68 | - | 0 | 685 of 1221 | see the cost row | - | - | - | - | 1048 of 1048 | - / 0 / 41 of 832 | 832 / 0 / 0 / 0 | 151 / 54 / 2 |
|
||||
| L2b / L4 cost | measured 27-28.09, not measured again | | | | | L4 105.8 M for 19 sources (84 % cache read); L2b writing 30.93 M for 19 sources | | | | L2b about 1.6-1.8 bn writing + 8.6 bn reading | | | | |
|
||||
|
||||
- Every K and K+ output number is a lower bound (the harvester fix in run K). Tokens per source split each pack's cost
|
||||
by block share. The projection is not validated (4 points).
|
||||
- Side-by-side page: 3 concepts x 4 columns; 454 of 454 source links resolve. Main session (orchestration, apart):
|
||||
10.9 M tokens up to step 11, almost all cache read.
|
||||
- Concepts per source in order 1-19 (a page counts at the earliest source its body cites, D10): K and K+ 11, 61, 31,
|
||||
1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11; L4 2, 9, 19, 1, 2, 10, 14, 17, 5, 6, 14, 10, 7, 2, 9, 6, 3, 6, 9;
|
||||
D 0 for every source (its register pages cite nothing in a body).
|
||||
- K+ consolidation cost 13706 per page (lower bound) and is not split per source, because a page joins fragments from
|
||||
several sources.
|
||||
- Seen in the review (BLOCK, not fixed in this run): 45 of 1221 blocks are only MDX tag lines and cannot be covered
|
||||
(K 623 of 1176 without them); the index-line filter sends a line when the whole name occurs as a phrase, not per
|
||||
token as D7 is written; `cost.summarize` and the stop command have no test.
|
||||
5
evals/p1/__init__.py
Normal file
5
evals/p1/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""TRIAL, branch p1-metodeprove, not v1 code. Nothing outside evals/ imports it.
|
||||
|
||||
The P1 method trial (PLAN.md § Rammer, order 20261002T094056Z-495685981): three build methods for a concept wiki (K, K+,
|
||||
D) on the 19 sources of the cut, measured side by side for cost and quality. No module here calls a model.
|
||||
"""
|
||||
287
evals/p1/assemble.py
Normal file
287
evals/p1/assemble.py
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
"""K's assembler and K+'s apply step (TRIAL, P1): pages from accepted fragments, by code. It calls no model.
|
||||
|
||||
build --work DIR --kilde-root DIR --out DIR [--at ISO]
|
||||
One page per concept that at least one accepted fragment enriches (unused skeleton entries get none). Docs
|
||||
fragments form the body; release-note fragments go under "## History" in version order. Every sentence ends
|
||||
in a content footnote and, for a release note, a version footnote whose quote is the bare version on the
|
||||
note's heading line. A fragment without a model quote gets a code quote: the first line of its span that is
|
||||
not structural, marked x-quote-by: code. A fragment that names another concept's title links to it once.
|
||||
copy --src DIR --out DIR
|
||||
K+ starts as a copy of K.
|
||||
tasks --bundle DIR --work DIR [--min 3] [--per-run 8]
|
||||
One task file per consolidation run (pages with at least --min fragments, --per-run pages each) in
|
||||
<work>/Kplus-oppgaver/, output path <work>/Kplus-bodies/run-N.json.
|
||||
apply --bundle DIR --bodies FILE
|
||||
Replaces a page body only when the new body keeps every old footnote label and adds none; prints
|
||||
"applied X, refused Y".
|
||||
|
||||
--kilde-root is the folder that contains kilde/. Exit 0 = done, 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import FOOTNOTE_DEF, FOOTNOTE_REF, HEADING, read_document, sentence_spans
|
||||
from p1.dmap import frontmatter
|
||||
from p1.split import FENCE, headings_of, version_key
|
||||
|
||||
GENERATOR = "okf-wiki/p1-assemble"
|
||||
QUOTE_MAX = 200
|
||||
HTML_TAG_ONLY = re.compile(r"^\s*</?[A-Za-z][^<>]*/?>\s*$")
|
||||
TABLE_RULE = re.compile(r"^\s*\|?\s*:?-+:?\s*(\|\s*:?-+:?\s*)*\|?\s*$")
|
||||
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Run:
|
||||
task: Path
|
||||
output: Path
|
||||
pages: list[str]
|
||||
|
||||
|
||||
def structural(line: str) -> bool:
|
||||
return (not line.strip() or HEADING.match(line) is not None or FENCE.match(line) is not None
|
||||
or HTML_TAG_ONLY.match(line) is not None or TABLE_RULE.match(line) is not None)
|
||||
|
||||
|
||||
def code_quote(span: list[str], start: int) -> tuple[str, int]:
|
||||
"""The first non-structural line of a span (stripped, capped), and its file line."""
|
||||
for i, line in enumerate(span):
|
||||
if not structural(line):
|
||||
return line.strip()[:QUOTE_MAX], start + i
|
||||
return span[0].strip()[:QUOTE_MAX] or span[0], start
|
||||
|
||||
|
||||
def footnoted(text: str, labels: list[str]) -> str:
|
||||
text = text.strip()
|
||||
if text and text[-1] not in ".!?":
|
||||
text += "."
|
||||
marks = "".join(f"[^{x}]" for x in labels)
|
||||
out, last = [], 0
|
||||
for _a, b in sentence_spans(text):
|
||||
out.append(text[last:b] + marks)
|
||||
last = b
|
||||
out.append(text[last:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _link_titles(text: str, titles: dict[str, str], self_id: str, linked: set[str]) -> str:
|
||||
for cid, title in sorted(titles.items(), key=lambda kv: -len(kv[1])):
|
||||
if cid == self_id or cid in linked or len(title) < 4:
|
||||
continue
|
||||
pattern = re.compile(r"(?<![\w`\[])" + re.escape(title) + r"(?![\w`\]])", re.IGNORECASE)
|
||||
m = pattern.search(text)
|
||||
if m is None:
|
||||
continue
|
||||
text = text[:m.start()] + f"[{m.group(0)}]({cid}.md)" + text[m.end():]
|
||||
linked.add(cid)
|
||||
return text
|
||||
|
||||
|
||||
def _version_line(lines: list[str], version: str) -> int:
|
||||
for n, _level, text in headings_of(lines):
|
||||
m = VERSION_HEADING.match(text)
|
||||
if m and m.group(1) == version:
|
||||
return n
|
||||
raise RunError(f"no heading for version {version}")
|
||||
|
||||
|
||||
def assemble(work: Path, kilde_root: Path, out: Path, at: str | None = None) -> int:
|
||||
if out.exists():
|
||||
raise RunError(f"{out} exists; refusing to overwrite")
|
||||
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
index = json.loads((work / "K-work" / "index.json").read_text(encoding="utf-8"))
|
||||
frags = json.loads((work / "K-work" / "accepted.json").read_text(encoding="utf-8"))["fragments"]
|
||||
entries = {e["id"]: e for e in index["entries"]}
|
||||
kilde = kilde_root / "kilde"
|
||||
cache: dict[str, list[str]] = {}
|
||||
|
||||
def lines_of(name: str) -> list[str]:
|
||||
if name not in cache:
|
||||
cache[name] = (kilde / name).read_text(encoding="utf-8").split("\n")
|
||||
return cache[name]
|
||||
|
||||
by_concept: dict[str, list[dict[str, Any]]] = {}
|
||||
for f in frags:
|
||||
by_concept.setdefault(f["concept_id"], []).append(f)
|
||||
titles = {cid: str(entries[cid]["title"]) for cid in by_concept}
|
||||
pages: dict[str, str] = {}
|
||||
for cid, items in by_concept.items():
|
||||
e = entries[cid]
|
||||
sources: list[dict[str, Any]] = []
|
||||
versions: dict[str, str] = {}
|
||||
body_parts: list[str] = []
|
||||
history: list[tuple[tuple[int, ...], int, str]] = []
|
||||
linked: set[str] = set()
|
||||
for k, f in enumerate(items, 1):
|
||||
lines = lines_of(f["file"])
|
||||
src: dict[str, Any] = {"id": f"f{k}", "resource": f"/kilde/{f['file']}", "title": f["section"]}
|
||||
if f.get("quote"):
|
||||
src["x-locator"] = {"line": f["quote_line"]}
|
||||
src["x-quote"] = f["quote"]
|
||||
else:
|
||||
quote, line = code_quote(lines[f["start"] - 1:f["end"]], f["start"])
|
||||
src["x-locator"] = {"line": line}
|
||||
src["x-quote"] = quote
|
||||
src["x-quote-by"] = "code"
|
||||
src["x-span"] = [f["start"], f["end"]]
|
||||
sources.append(src)
|
||||
labels = [f"f{k}"]
|
||||
if f.get("version"):
|
||||
v = str(f["version"])
|
||||
if v not in versions:
|
||||
versions[v] = "v" + re.sub(r"\D", "", v)
|
||||
sources.append({"id": versions[v], "resource": f"/kilde/{f['file']}", "title": v,
|
||||
"x-locator": {"line": _version_line(lines, v)}, "x-quote": v})
|
||||
labels.append(versions[v])
|
||||
text = _link_titles(footnoted(f["text"], labels), titles, cid, linked)
|
||||
history.append((version_key(f"release-v{v}.md"), k, text))
|
||||
else:
|
||||
body_parts.append(_link_titles(footnoted(f["text"], labels), titles, cid, linked))
|
||||
body = [f"# {e['title']}", ""]
|
||||
for p in body_parts:
|
||||
body += [p, ""]
|
||||
if history:
|
||||
body += ["## History", ""]
|
||||
body += [f"- {t}" for _, _, t in sorted(history)]
|
||||
body.append("")
|
||||
body += [f"[^{s['id']}]: {Path(s['resource']).name}, line {s['x-locator']['line']}" for s in sources]
|
||||
desc = FOOTNOTE_REF.sub("", items[0]["text"]).strip()[:200]
|
||||
fm: dict[str, Any] = {"type": "Concept", "title": str(e["title"]), "description": desc, "sources": sources,
|
||||
"x-aliases": [str(a) for a in e.get("aliases", [])], "x-fragments": len(items)}
|
||||
if e.get("kind") == "merge-candidate" and e.get("merge_of"):
|
||||
fm["x-merge-candidate"] = str(e["merge_of"])
|
||||
fm["generated"] = {"by": GENERATOR, "at": at}
|
||||
pages[f"{cid}.md"] = frontmatter(fm) + "\n" + "\n".join(body) + "\n"
|
||||
out.mkdir(parents=True)
|
||||
for name, text in pages.items():
|
||||
(out / name).write_text(text, encoding="utf-8")
|
||||
index_md = ["---", 'okf_version: "0.2"', "---", "", "# Concepts", ""]
|
||||
index_md += [f"- [{titles[cid]}]({cid}.md)" for cid in sorted(by_concept)]
|
||||
(out / "index.md").write_text("\n".join(index_md) + "\n", encoding="utf-8")
|
||||
(out / "log.md").write_text(f"# Log\n\n## {at[:10]}\n* **Build**: K assembled by {GENERATOR}: {len(pages)} "
|
||||
f"pages from {len(frags)} accepted fragments.\n", encoding="utf-8")
|
||||
return len(pages)
|
||||
|
||||
|
||||
def copy(src: Path, dst: Path) -> None:
|
||||
if dst.exists():
|
||||
raise RunError(f"{dst} exists; refusing to overwrite")
|
||||
shutil.copytree(src, dst)
|
||||
|
||||
|
||||
def _split_page(text: str) -> tuple[str, str, list[str], list[str]]:
|
||||
"""(frontmatter block, title line, body lines without title and footnote definitions, definition lines)."""
|
||||
end = text.index("\n---\n", 4) + 5
|
||||
head, rest = text[:end], text[end:].split("\n")
|
||||
title = next(line for line in rest if line.startswith("# "))
|
||||
body = [line for line in rest if line != title and not FOOTNOTE_DEF.match(line.strip())]
|
||||
defs = [line for line in rest if FOOTNOTE_DEF.match(line.strip())]
|
||||
return head, title, body, defs
|
||||
|
||||
|
||||
def consolidation_tasks(bundle: Path, work: Path, min_fragments: int = 3, per_run: int = 8) -> list[Run]:
|
||||
chosen: list[str] = []
|
||||
for p in sorted(bundle.glob("*.md")):
|
||||
if p.name in ("index.md", "log.md"):
|
||||
continue
|
||||
fm = read_document(p, p.name).frontmatter or {}
|
||||
if int(fm.get("x-fragments", 0)) >= min_fragments:
|
||||
chosen.append(p.stem)
|
||||
runs: list[Run] = []
|
||||
for i in range(0, len(chosen), per_run):
|
||||
n = len(runs) + 1
|
||||
task = work / "Kplus-oppgaver" / f"run-{n}.md"
|
||||
output = work / "Kplus-bodies" / f"run-{n}.json"
|
||||
if task.exists():
|
||||
raise RunError(f"{task} exists; refusing to overwrite")
|
||||
parts = [f"# K+ consolidation run {n}\n\nRewrite each page body below into flowing text. Output: {output}\n"]
|
||||
for cid in chosen[i:i + per_run]:
|
||||
text = (bundle / f"{cid}.md").read_text(encoding="utf-8")
|
||||
fm = read_document(bundle / f"{cid}.md", f"{cid}.md").frontmatter or {}
|
||||
_head, title, body, _defs = _split_page(text)
|
||||
quotes = [f"- [^{s['id']}]: {s.get('x-quote', '')}" for s in fm.get("sources", [])]
|
||||
parts.append(f"## Page {cid} ({title[2:]})\n\nBody:\n\n" + "\n".join(body).strip()
|
||||
+ "\n\nWhat each label quotes:\n\n" + "\n".join(quotes) + "\n")
|
||||
task.parent.mkdir(parents=True, exist_ok=True)
|
||||
task.write_text("\n".join(parts), encoding="utf-8")
|
||||
runs.append(Run(task, output, chosen[i:i + per_run]))
|
||||
return runs
|
||||
|
||||
|
||||
def apply(bundle: Path, bodies_path: Path) -> tuple[int, int]:
|
||||
bodies = json.loads(bodies_path.read_text(encoding="utf-8"))
|
||||
applied = refused = 0
|
||||
for cid, new in bodies.items():
|
||||
page = bundle / f"{cid}.md"
|
||||
if not page.is_file() or not isinstance(new, str):
|
||||
refused += 1
|
||||
continue
|
||||
head, title, body, defs = _split_page(page.read_text(encoding="utf-8"))
|
||||
old_labels = set(FOOTNOTE_REF.findall("\n".join(body)))
|
||||
if set(FOOTNOTE_REF.findall(new)) != old_labels or FOOTNOTE_DEF.search(new):
|
||||
refused += 1
|
||||
continue
|
||||
page.write_text(head + "\n" + title + "\n\n" + new.strip() + "\n\n" + "\n".join(defs) + "\n", encoding="utf-8")
|
||||
applied += 1
|
||||
return applied, refused
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
b = sub.add_parser("build")
|
||||
b.add_argument("--work", required=True)
|
||||
b.add_argument("--kilde-root", required=True)
|
||||
b.add_argument("--out", required=True)
|
||||
b.add_argument("--at")
|
||||
c = sub.add_parser("copy")
|
||||
c.add_argument("--src", required=True)
|
||||
c.add_argument("--out", required=True)
|
||||
t = sub.add_parser("tasks")
|
||||
t.add_argument("--bundle", required=True)
|
||||
t.add_argument("--work", required=True)
|
||||
t.add_argument("--min", type=int, default=3)
|
||||
t.add_argument("--per-run", type=int, default=8)
|
||||
a = sub.add_parser("apply")
|
||||
a.add_argument("--bundle", required=True)
|
||||
a.add_argument("--bodies", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "build":
|
||||
n = assemble(Path(args.work), Path(args.kilde_root), Path(args.out), args.at)
|
||||
print(f"K: {n} pages -> {args.out}")
|
||||
elif args.command == "copy":
|
||||
copy(Path(args.src), Path(args.out))
|
||||
print(f"copied {args.src} -> {args.out}")
|
||||
elif args.command == "tasks":
|
||||
for r in consolidation_tasks(Path(args.bundle), Path(args.work), args.min, args.per_run):
|
||||
print(f"task: {r.task} output: {r.output} pages: {len(r.pages)}")
|
||||
else:
|
||||
applied, refused = apply(Path(args.bundle), Path(args.bodies))
|
||||
print(f"applied {applied}, refused {refused}")
|
||||
except (RunError, OSError, KeyError, ValueError) as exc:
|
||||
print(f"assemble cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
329
evals/p1/cost.py
Normal file
329
evals/p1/cost.py
Normal file
|
|
@ -0,0 +1,329 @@
|
|||
"""The trial's cost from the run sessions' transcripts (TRIAL, P1): harvest, allocation, projection, stop. No model.
|
||||
|
||||
harvest --project-dir D --okter FILE --blocks F --total N [--packs FILE] --out FILE
|
||||
Reads only the sessions listed in okter.txt ("<session-id> <step>" per line): each session's main
|
||||
transcript (orchestration, reported apart) and every subagent transcript under <session>/subagents/. Per
|
||||
subagent: input, cache read, cache write and output tokens kept apart (each message id counted once, at
|
||||
its largest value, because a streamed message repeats its id with growing output), turns (distinct
|
||||
message ids), startup (the first message's input side), agentType and description from its .meta.json.
|
||||
A message whose final usage was never written (no stop reason on any record) gets its output from the
|
||||
growth of the next message's input side less the injected text counted high (a lower bound); the
|
||||
recorded output stays beside it (output_recorded).
|
||||
A description that starts "P1-<variant>-<unit>" attributes the run; any other is listed as unattributed,
|
||||
never dropped. Per variant: totals, tokens per source (a pack's cost split by block share), the means of
|
||||
sources 1-5 and 15-19, and the projection to all sources (N blocks). The file is rewritten from the
|
||||
transcripts on every harvest; stop rows already in it are kept.
|
||||
stop VARIANT --kost FILE --blocks F [--pages-total N --pages-first M] [--record]
|
||||
The stop rule (cap 1 M tokens, all four categories): K after pack 1, projected per block processed and,
|
||||
beside it, naively per source; K+ after run 1, projected over the pages to consolidate. Exit 1 = stop.
|
||||
With --record the stop row is written into the kost file.
|
||||
|
||||
Projection: cost per block c = a + b*R by least squares over the measured units, R = blocks read before the unit;
|
||||
the projected total for T blocks is a*T + b*T*T/2 (the sum as R grows from 0 to T). One unit: b = 0. The flat
|
||||
projection (mean cost per block times T) stands beside it. Exit 0 = done, 1 = stop (stop only), 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from direction_probe import USAGE_FIELDS, read_transcript
|
||||
|
||||
SCHEMA = "okf-wiki/p1-cost/1"
|
||||
CAP = 1_000_000
|
||||
CATEGORIES = {"input_tokens": "input", "cache_read_input_tokens": "cache_read",
|
||||
"cache_creation_input_tokens": "cache_write", "output_tokens": "output"}
|
||||
PREFIX = re.compile(r"^P1-([A-Za-z0-9+]+)-(\S+)")
|
||||
PACK_UNIT = re.compile(r"^pack(\d+)")
|
||||
CLASS = {"LOAD": "load", "K": "write", "K+": "write"}
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
def sessions(okter: Path) -> list[str]:
|
||||
ids: list[str] = []
|
||||
for line in okter.read_text(encoding="utf-8").splitlines():
|
||||
parts = line.split()
|
||||
if parts and parts[0] not in ids:
|
||||
ids.append(parts[0])
|
||||
return ids
|
||||
|
||||
|
||||
INPUT_SIDE = ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")
|
||||
CHARS_PER_TOKEN_LOW = 2.5 # injected text counted high, so a derived output is a lower bound
|
||||
|
||||
|
||||
def _result_chars(message: dict[str, Any]) -> int:
|
||||
content = message.get("content")
|
||||
if not isinstance(content, list):
|
||||
return 0
|
||||
total = 0
|
||||
for item in content:
|
||||
if not (isinstance(item, dict) and item.get("type") == "tool_result"):
|
||||
continue
|
||||
inner = item.get("content")
|
||||
if isinstance(inner, str):
|
||||
total += len(inner)
|
||||
elif isinstance(inner, list):
|
||||
total += sum(len(str(x.get("text", ""))) for x in inner if isinstance(x, dict))
|
||||
return total
|
||||
|
||||
|
||||
def usage(path: Path) -> dict[str, Any]:
|
||||
"""Tokens of one transcript. A message whose records all lack a stop reason never got its final usage written
|
||||
(seen for a long streamed message: output 2); when another message follows, its output is the growth of the
|
||||
next message's input side less the text injected between them (tool results and rendered attachments, counted
|
||||
high at CHARS_PER_TOKEN_LOW), a lower bound, and never below what was recorded."""
|
||||
per: dict[str, dict[str, int]] = {}
|
||||
order: list[str] = []
|
||||
complete: set[str] = set()
|
||||
between: dict[str, int] = {}
|
||||
last_stop: str | None = None
|
||||
for line in read_transcript(path):
|
||||
message = line.get("message")
|
||||
if line.get("type") == "attachment" and order:
|
||||
between[order[-1]] = between.get(order[-1], 0) + len(str(line.get("rendered") or ""))
|
||||
continue
|
||||
if not isinstance(message, dict):
|
||||
continue
|
||||
if line.get("type") == "user" and order:
|
||||
between[order[-1]] = between.get(order[-1], 0) + _result_chars(message)
|
||||
continue
|
||||
if line.get("type") != "assistant":
|
||||
continue
|
||||
mid = str(message.get("id", ""))
|
||||
if mid not in per:
|
||||
order.append(mid)
|
||||
seen = per.setdefault(mid, {})
|
||||
fields = message.get("usage") or {}
|
||||
for f in USAGE_FIELDS:
|
||||
value = fields.get(f)
|
||||
if isinstance(value, int):
|
||||
seen[f] = max(seen.get(f, 0), value)
|
||||
stop = message.get("stop_reason")
|
||||
last_stop = stop if isinstance(stop, str) else None
|
||||
if isinstance(stop, str):
|
||||
complete.add(mid)
|
||||
out: dict[str, Any] = {CATEGORIES[f]: sum(v.get(f, 0) for v in per.values()) for f in USAGE_FIELDS}
|
||||
out["output_recorded"] = out["output"]
|
||||
derived = unrecorded = 0
|
||||
for i, mid in enumerate(order):
|
||||
if mid in complete:
|
||||
continue
|
||||
if i + 1 == len(order):
|
||||
unrecorded += 1
|
||||
continue
|
||||
growth = (sum(per[order[i + 1]].get(f, 0) for f in INPUT_SIDE) - sum(per[mid].get(f, 0) for f in INPUT_SIDE)
|
||||
- int(between.get(mid, 0) / CHARS_PER_TOKEN_LOW))
|
||||
recorded = per[mid].get("output_tokens", 0)
|
||||
if growth > recorded:
|
||||
out["output"] += growth - recorded
|
||||
derived += 1
|
||||
out["output_derived_messages"] = derived
|
||||
out["output_unrecorded"] = unrecorded
|
||||
out["total"] = sum(out[c] for c in CATEGORIES.values())
|
||||
out["turns"] = len(order)
|
||||
first = per[order[0]] if order else {}
|
||||
out["startup"] = sum(first.get(f, 0) for f in INPUT_SIDE)
|
||||
out["stop_reason"] = last_stop
|
||||
return out
|
||||
|
||||
|
||||
def harvest(project_dir: Path, ids: list[str]) -> dict[str, Any]:
|
||||
runs: list[dict[str, Any]] = []
|
||||
unattributed: list[dict[str, Any]] = []
|
||||
main: dict[str, Any] = {}
|
||||
for sid in ids:
|
||||
top = project_dir / f"{sid}.jsonl"
|
||||
if top.is_file():
|
||||
main[sid] = usage(top)
|
||||
for path in sorted((project_dir / sid / "subagents").glob("agent-*.jsonl")):
|
||||
meta_path = path.with_name(path.stem + ".meta.json")
|
||||
meta = json.loads(meta_path.read_text(encoding="utf-8")) if meta_path.is_file() else {}
|
||||
row = {"session": sid, "agent": path.stem, "agentType": meta.get("agentType"),
|
||||
"description": str(meta.get("description", "")), **usage(path)}
|
||||
m = PREFIX.match(row["description"])
|
||||
if m is None:
|
||||
unattributed.append(row)
|
||||
continue
|
||||
row["variant"], row["unit"] = m.group(1), m.group(2)
|
||||
row["class"] = CLASS.get(m.group(1), "other")
|
||||
runs.append(row)
|
||||
return {"sessions_read": ids, "runs": runs, "unattributed": unattributed, "main": main}
|
||||
|
||||
|
||||
def allocate(pack_costs: Mapping[int, float], packs: list[list[str]], blocks_doc: dict[str, Any]) -> dict[str, float]:
|
||||
"""Tokens per source: each pack's cost split over its sources by their share of the pack's blocks."""
|
||||
count: dict[str, int] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
count[b["file"]] = count.get(b["file"], 0) + 1
|
||||
per: dict[str, float] = {}
|
||||
for n, names in enumerate(packs, 1):
|
||||
if n not in pack_costs:
|
||||
continue
|
||||
total = sum(count.get(x, 0) for x in names) or 1
|
||||
for x in names:
|
||||
per[x] = pack_costs[n] * count.get(x, 0) / total
|
||||
return per
|
||||
|
||||
|
||||
def means(per: dict[str, float], order: list[str]) -> dict[str, float | None]:
|
||||
def mean(names: list[str]) -> float | None:
|
||||
vals = [per[x] for x in names if x in per]
|
||||
return sum(vals) / len(vals) if vals else None
|
||||
|
||||
return {"1-5": mean(order[0:5]), "15-19": mean(order[14:19])}
|
||||
|
||||
|
||||
def project(points: list[tuple[float, int, int]], total_blocks: int) -> dict[str, Any]:
|
||||
"""points: (cost, blocks in the unit, blocks read before it)."""
|
||||
if not points:
|
||||
return {"fit": None, "flat": None, "T": total_blocks}
|
||||
xs = [float(r) for _, _, r in points]
|
||||
ys = [c / b for c, b, _ in points]
|
||||
fit: dict[str, Any]
|
||||
if len(points) == 1 or len(set(xs)) == 1:
|
||||
a, b = sum(ys) / len(ys), 0.0
|
||||
fit = {"a": a, "b": 0, "note": "one point, no growth term"}
|
||||
else:
|
||||
mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
|
||||
b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
|
||||
a = my - b * mx
|
||||
fit = {"a": a, "b": b, "note": f"{len(points)} points, not validated"}
|
||||
fit["value"] = a * total_blocks + b * total_blocks * total_blocks / 2
|
||||
flat = sum(c for c, _, _ in points) / sum(bl for _, bl, _ in points) * total_blocks
|
||||
return {"fit": fit, "flat": flat, "T": total_blocks}
|
||||
|
||||
|
||||
def stop_k(pack1_cost: int, pack1_blocks: int, cut_blocks: int, pack1_sources: int, cut_sources: int
|
||||
) -> dict[str, Any]:
|
||||
per_block = pack1_cost * cut_blocks / pack1_blocks
|
||||
per_source = pack1_cost * cut_sources / pack1_sources
|
||||
return {"per_block": per_block, "per_source": per_source, "cap": CAP, "stop": per_block > CAP}
|
||||
|
||||
|
||||
def stop_kplus(run1_cost: int, run1_pages: int, pages_total: int) -> dict[str, Any]:
|
||||
projected = run1_cost * pages_total / run1_pages
|
||||
return {"projected": projected, "cap": CAP, "stop": projected > CAP}
|
||||
|
||||
|
||||
def _pack_number(unit: str) -> int | None:
|
||||
m = PACK_UNIT.match(unit)
|
||||
return int(m.group(1)) if m else None
|
||||
|
||||
|
||||
def summarize(h: dict[str, Any], blocks_doc: dict[str, Any], packs_log: dict[str, Any] | None,
|
||||
total_blocks: int) -> dict[str, Any]:
|
||||
order = [str(x) for x in blocks_doc["order"]]
|
||||
count: dict[str, int] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
count[b["file"]] = count.get(b["file"], 0) + 1
|
||||
variants: dict[str, Any] = {}
|
||||
for v in sorted({r["variant"] for r in h["runs"]}):
|
||||
mine = [r for r in h["runs"] if r["variant"] == v]
|
||||
row: dict[str, Any] = {c: sum(r[c] for r in mine) for c in (*CATEGORIES.values(), "total", "turns", "startup")}
|
||||
row["subagents"] = len(mine)
|
||||
row["class"] = mine[0]["class"]
|
||||
row["counts_toward_cap"] = v in ("K", "K+")
|
||||
if v == "K" and packs_log:
|
||||
packs = [list(p["sources"]) for p in sorted(packs_log["packs"], key=lambda p: int(p["pack"]))]
|
||||
pack_costs: dict[int, float] = {}
|
||||
per_cat: dict[str, dict[int, float]] = {c: {} for c in CATEGORIES.values()}
|
||||
for r in mine:
|
||||
n = _pack_number(r["unit"])
|
||||
if n is not None:
|
||||
pack_costs[n] = pack_costs.get(n, 0) + r["total"]
|
||||
for c in CATEGORIES.values():
|
||||
per_cat[c][n] = per_cat[c].get(n, 0) + r[c]
|
||||
row["per_pack"] = {str(n): c for n, c in sorted(pack_costs.items())}
|
||||
row["per_source"] = allocate(pack_costs, packs, blocks_doc)
|
||||
row["per_source_by_category"] = {c: allocate(per_cat[c], packs, blocks_doc) for c in CATEGORIES.values()}
|
||||
row["means"] = means(row["per_source"], order)
|
||||
points, read = [], 0
|
||||
for n, names in enumerate(packs, 1):
|
||||
blocks_n = sum(count.get(x, 0) for x in names)
|
||||
if n in pack_costs:
|
||||
points.append((pack_costs[n], blocks_n, read))
|
||||
read += blocks_n
|
||||
row["projection"] = project(points, total_blocks)
|
||||
variants[v] = row
|
||||
return variants
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
h = sub.add_parser("harvest")
|
||||
h.add_argument("--project-dir", required=True)
|
||||
h.add_argument("--okter", required=True)
|
||||
h.add_argument("--blocks", required=True)
|
||||
h.add_argument("--total", type=int, required=True)
|
||||
h.add_argument("--packs")
|
||||
h.add_argument("--out", required=True)
|
||||
s = sub.add_parser("stop")
|
||||
s.add_argument("variant", choices=("K", "K+"))
|
||||
s.add_argument("--kost", required=True)
|
||||
s.add_argument("--blocks", required=True)
|
||||
s.add_argument("--packs")
|
||||
s.add_argument("--pages-total", type=int)
|
||||
s.add_argument("--pages-first", type=int)
|
||||
s.add_argument("--record", action="store_true")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
if args.command == "harvest":
|
||||
out = Path(args.out)
|
||||
old = json.loads(out.read_text(encoding="utf-8")) if out.is_file() else {}
|
||||
h = harvest(Path(args.project_dir), sessions(Path(args.okter)))
|
||||
packs_log = json.loads(Path(args.packs).read_text(encoding="utf-8")) if args.packs else None
|
||||
doc = {"schema": SCHEMA, **h, "variants": summarize(h, blocks_doc, packs_log, args.total),
|
||||
"stops": old.get("stops", {})}
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
print(f"harvest: {len(h['runs'])} attributed runs, {len(h['unattributed'])} unattributed, sessions read: "
|
||||
f"{', '.join(h['sessions_read'])} -> {out}")
|
||||
return 0
|
||||
kost_path = Path(args.kost)
|
||||
kost = json.loads(kost_path.read_text(encoding="utf-8"))
|
||||
runs = [r for r in kost["runs"] if r["variant"] == args.variant]
|
||||
if args.variant == "K":
|
||||
first = [r for r in runs if _pack_number(r["unit"]) == 1]
|
||||
if not first:
|
||||
raise RunError("no pack 1 run in the kost file; harvest after pack 1")
|
||||
if not args.packs:
|
||||
raise RunError("--packs is needed for K")
|
||||
log = json.loads(Path(args.packs).read_text(encoding="utf-8"))
|
||||
p1 = next(p for p in log["packs"] if int(p["pack"]) == 1)
|
||||
row = stop_k(sum(r["total"] for r in first), int(p1["blocks"]), len(blocks_doc["blocks"]),
|
||||
len(p1["sources"]), len(blocks_doc["order"]))
|
||||
else:
|
||||
first = [r for r in runs if r["unit"].startswith("run1")]
|
||||
if not first or not args.pages_total or not args.pages_first:
|
||||
raise RunError("K+ needs a run 1 in the kost file, --pages-total and --pages-first")
|
||||
row = stop_kplus(sum(r["total"] for r in first), args.pages_first, args.pages_total)
|
||||
print(f"{args.variant}: projected {row.get('per_block', row.get('projected'))} tokens"
|
||||
+ (f" (naive per source {row['per_source']})" if "per_source" in row else "")
|
||||
+ f", cap {CAP}: {'STOP' if row['stop'] else 'continue'}")
|
||||
if args.record:
|
||||
kost.setdefault("stops", {})[args.variant] = row
|
||||
kost_path.write_text(json.dumps(kost, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
return 1 if row["stop"] else 0
|
||||
except (RunError, OSError, json.JSONDecodeError, KeyError, StopIteration) as exc:
|
||||
print(f"cost cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
178
evals/p1/dmap.py
Normal file
178
evals/p1/dmap.py
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
"""D, the trial's no-model baseline (TRIAL, P1): a generated map and a code-built concept register. It calls no model.
|
||||
|
||||
build --blocks FILE --kilde-root DIR --out DIR [--at ISO]
|
||||
One map page per source (k-<slug>.md) whose body mirrors the source's headings, with one sources[] entry per
|
||||
section; one register page per kept term (r-<slug>.md) whose body is a pure link list to the map sections
|
||||
where the term occurs, with one sources[] entry per occurrence. A term is kept when it is a docs-page heading
|
||||
that passes concept test 2, or when it occurs in at least 2 of the documents. Bodies hold only headings and
|
||||
pure link lists, so the claim count is 0 by construction; the exact locators live in sources[].
|
||||
|
||||
--kilde-root is the folder that contains kilde/ (the auditor's SOURCES argument). Exit 0 = written, 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from p1.split import headings_of, is_meta, occurs, terms
|
||||
|
||||
MIN_DOCUMENTS = 2
|
||||
GENERATOR = "okf-wiki/p1-dmap"
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class DStats:
|
||||
maps: int
|
||||
registers: int
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
|
||||
|
||||
|
||||
def frontmatter(data: dict[str, Any]) -> str:
|
||||
"""A frontmatter block; PyYAML quotes any string that would load as another type (no, 1.10)."""
|
||||
return "---\n" + yaml.safe_dump(data, sort_keys=False, allow_unicode=True, width=10_000) + "---\n"
|
||||
|
||||
|
||||
def _anchors(headings: list[tuple[int, int, str]]) -> dict[int, str]:
|
||||
seen: dict[str, int] = {}
|
||||
out: dict[int, str] = {}
|
||||
for line, _level, text in headings:
|
||||
base = slug(text)
|
||||
n = seen.get(base, 0)
|
||||
seen[base] = n + 1
|
||||
out[line] = base if n == 0 else f"{base}-{n}"
|
||||
return out
|
||||
|
||||
|
||||
def _title(name: str, headings: list[tuple[int, int, str]]) -> str:
|
||||
first = next((t for _, level, t in headings if level == 1), None)
|
||||
if first:
|
||||
return first
|
||||
return f"Release notes {headings[0][2]}" if name.startswith("release-") and headings else name
|
||||
|
||||
|
||||
def build(blocks_doc: dict[str, Any], kilde_root: Path, out: Path, at: str | None = None) -> DStats:
|
||||
if out.exists():
|
||||
raise RunError(f"{out} exists; refusing to overwrite")
|
||||
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
names = [str(n) for n in blocks_doc["order"]]
|
||||
kilde = kilde_root / "kilde"
|
||||
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
|
||||
lines = {n: t.split("\n") for n, t in texts.items()}
|
||||
heads = {n: headings_of(lines[n]) for n in names}
|
||||
anchors = {n: _anchors(heads[n]) for n in names}
|
||||
titles = {n: _title(n, heads[n]) for n in names}
|
||||
pages: dict[str, str] = {}
|
||||
gen = {"by": GENERATOR, "at": at}
|
||||
|
||||
for n in names:
|
||||
sources = []
|
||||
body = [f"# {titles[n]}", ""]
|
||||
for i, (line, level, text) in enumerate(heads[n], 1):
|
||||
sources.append({"id": f"s{i}", "resource": f"/kilde/{n}", "title": text, "x-locator": {"line": line},
|
||||
"x-quote": text})
|
||||
if level > 1 or text != titles[n]:
|
||||
body += [f"{'#' * max(level, 2)} {text}", ""]
|
||||
fm = {"type": "Document map", "title": titles[n], "description": f"The sections of {n}.", "sources": sources,
|
||||
"generated": gen}
|
||||
pages[f"k-{slug(n.removesuffix('.md'))}.md"] = frontmatter(fm) + "\n" + "\n".join(body).rstrip() + "\n"
|
||||
|
||||
candidates = [t for n in names if n.startswith("docs-") for _, level, t in heads[n] if level in (2, 3)]
|
||||
from_headings = {t for t in candidates if not is_meta(t)}
|
||||
pool = list(dict.fromkeys(list(from_headings) + [t for n in names for t in terms(texts[n])]))
|
||||
registers: dict[str, str] = {}
|
||||
for term in pool:
|
||||
if is_meta(term):
|
||||
continue
|
||||
holders = [n for n in names if occurs(term, texts[n])]
|
||||
if term not in from_headings and len(holders) < MIN_DOCUMENTS:
|
||||
continue
|
||||
pattern = re.compile(r"(?<!\w)" + re.escape(term) + r"(?!\w)", re.IGNORECASE)
|
||||
sources, links, seen = [], [], set()
|
||||
for n in holders:
|
||||
for i, raw in enumerate(lines[n], 1):
|
||||
m = pattern.search(raw)
|
||||
if m is None:
|
||||
continue
|
||||
section = max((h for h in heads[n] if h[0] <= i), default=None)
|
||||
key = (n, section[0] if section else 0)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
sources.append({"id": f"o{len(sources) + 1}", "resource": f"/kilde/{n}", "title": titles[n],
|
||||
"x-locator": {"line": i}, "x-quote": m.group(0)})
|
||||
target = f"k-{slug(n.removesuffix('.md'))}.md"
|
||||
if section is not None:
|
||||
target += f"#{anchors[n][section[0]]}"
|
||||
where = titles[n] + (f" › {section[2]}" if section and section[2] != titles[n] else "")
|
||||
links.append(f"- [{where.replace('[', '(').replace(']', ')')}]({target})")
|
||||
if not sources:
|
||||
continue
|
||||
name = f"r-{slug(term)}.md"
|
||||
k = 2
|
||||
while name in registers:
|
||||
name, k = f"r-{slug(term)}-{k}.md", k + 1
|
||||
fm = {"type": "Register entry", "title": term, "description": f"Where {term} occurs in the sources.",
|
||||
"sources": sources, "generated": gen}
|
||||
registers[name] = frontmatter(fm) + "\n" + f"# {term}\n\n" + "\n".join(links) + "\n"
|
||||
|
||||
out.mkdir(parents=True)
|
||||
for name, text in {**pages, **registers}.items():
|
||||
(out / name).write_text(text, encoding="utf-8")
|
||||
index = ["---", 'okf_version: "0.2"', "---", "", "# Document maps", ""]
|
||||
index += [f"- [{titles[n]}](k-{slug(n.removesuffix('.md'))}.md)" for n in names]
|
||||
index += ["", "# Register", ""]
|
||||
index += [f"- [{name[2:-3]}]({name})" for name in sorted(registers)]
|
||||
(out / "index.md").write_text("\n".join(index) + "\n", encoding="utf-8")
|
||||
day = at[:10]
|
||||
(out / "log.md").write_text(
|
||||
f"# Log\n\n## {day}\n"
|
||||
f"* **Build**: D by {GENERATOR}, {len(pages)} map pages and {len(registers)} register pages from {len(names)} "
|
||||
f"sources. Thresholds, recorded before any count: a term is kept when it is a docs-page heading of level 2 or "
|
||||
f"3 that passes concept test 2, or when it occurs in at least {MIN_DOCUMENTS} of the {len(names)} documents "
|
||||
f"(case-insensitive, whole phrase). Terms: backticked spans of 3-40 characters and capitalised phrases of 2-4 "
|
||||
f"words, a sentence-initial word dropped.\n",
|
||||
encoding="utf-8")
|
||||
return DStats(len(pages), len(registers))
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
b = sub.add_parser("build")
|
||||
b.add_argument("--blocks", required=True)
|
||||
b.add_argument("--kilde-root", required=True)
|
||||
b.add_argument("--out", required=True)
|
||||
b.add_argument("--at")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
s = build(doc, Path(args.kilde_root), Path(args.out), args.at)
|
||||
except (RunError, OSError, json.JSONDecodeError) as exc:
|
||||
print(f"dmap cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
print(f"D: {s.maps} map pages, {s.registers} register pages -> {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
374
evals/p1/kpack.py
Normal file
374
evals/p1/kpack.py
Normal file
|
|
@ -0,0 +1,374 @@
|
|||
"""K, the fragment compiler's code side (TRIAL, P1): skeleton, packs, input parts, validation, identity. No model.
|
||||
|
||||
skeleton --kilde DIR --out FILE
|
||||
The concept index from the titles and level 2-3 headings of every docs page under DIR that pass concept
|
||||
test 2. One entry per normalised name (the first page in name order wins); an index, not pages.
|
||||
pack N --blocks FILE --index FILE --kilde-root DIR --work DIR [--size 5]
|
||||
Pack N's input for the compiler: every block of its sources with its anchor (file › section › span) and
|
||||
numbered lines, then the index lines whose name occurs in those blocks, then the output contract, in parts
|
||||
of at most PART_CHARS characters (the Read tool's cap). Prints the part paths and the output path.
|
||||
accept N --work DIR --blocks FILE --kilde-root DIR
|
||||
Validates pack N's fragments, resolves each to a concept by code (existing, merge candidate, new or
|
||||
rejected), updates the index and appends the accepted fragments. Prints "accepted X of N" and the
|
||||
rejections by reason, and how many accepted fragments resolved to a concept whose index line was not
|
||||
sent with the pack (the pack log keeps the ids sent; the risk of the D7 phrase filter).
|
||||
|
||||
The work directory holds K-oppgaver/ (task parts), K-fragments/ (the compiler's JSON) and K-work/ (index.json,
|
||||
accepted.json, packs.json). Exit 0 = done, 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import sentence_spans
|
||||
from p1.split import docs_headings, is_meta
|
||||
|
||||
PART_CHARS = 60_000
|
||||
MERGE_JACCARD = 0.6
|
||||
NEEDS_QUOTE = re.compile(r"\d|`")
|
||||
REASONS = ("malformed", "anchor-outside-pack", "anchor-crosses-blocks", "sentence-count", "quote-missing",
|
||||
"quote-not-in-span", "tests-missing", "release-note-new", "meta")
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
def norm(name: str) -> str:
|
||||
text = name.replace("`", "").lower()
|
||||
text = re.sub(r"[^\w\s/.\-]", " ", text)
|
||||
words = text.split()
|
||||
if words and words[0] == "the":
|
||||
words = words[1:]
|
||||
words = [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
|
||||
return " ".join(w.strip(".") or w for w in words)
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
|
||||
|
||||
|
||||
def _unique_id(index: dict[str, Any], title: str) -> str:
|
||||
taken = {e["id"] for e in index["entries"]}
|
||||
base = slug(title)
|
||||
eid, k = base, 2
|
||||
while eid in taken:
|
||||
eid, k = f"{base}-{k}", k + 1
|
||||
return eid
|
||||
|
||||
|
||||
def skeleton(kilde: Path) -> dict[str, Any]:
|
||||
index: dict[str, Any] = {"schema": "okf-wiki/p1-index/1", "entries": []}
|
||||
seen: dict[str, dict[str, Any]] = {}
|
||||
for h in docs_headings(kilde):
|
||||
if is_meta(h.text):
|
||||
continue
|
||||
n = norm(h.text)
|
||||
if not n:
|
||||
continue
|
||||
if n in seen:
|
||||
if h.text not in seen[n]["aliases"]:
|
||||
seen[n]["aliases"].append(h.text)
|
||||
continue
|
||||
e = {"id": _unique_id(index, h.text), "title": h.text, "aliases": [h.text], "kind": "skeleton",
|
||||
"merge_of": None, "origin": {"file": h.file, "line": h.line}}
|
||||
index["entries"].append(e)
|
||||
seen[n] = e
|
||||
return index
|
||||
|
||||
|
||||
def _names(e: dict[str, Any]) -> set[str]:
|
||||
return {norm(a) for a in [e["title"], *e.get("aliases", [])]} - {""}
|
||||
|
||||
|
||||
def _jaccard(a: str, b: str) -> float:
|
||||
x, y = set(a.split()), set(b.split())
|
||||
return len(x & y) / len(x | y) if x | y else 0.0
|
||||
|
||||
|
||||
def resolve(fragment: dict[str, Any], index: dict[str, Any]) -> tuple[str, str | None]:
|
||||
"""existing <id> | merge-candidate <id> | new | rejected <reason> (D6, D3b)."""
|
||||
by_id = {e["id"]: e for e in index["entries"]}
|
||||
if fragment.get("status") == "existing" and fragment.get("id") in by_id:
|
||||
return "existing", str(fragment["id"])
|
||||
n = norm(str(fragment.get("concept", "")))
|
||||
for e in index["entries"]:
|
||||
if n in _names(e):
|
||||
return "existing", str(e["id"])
|
||||
if str(fragment.get("anchor", {}).get("file", "")).startswith("release-"):
|
||||
return "rejected", "release-note-new"
|
||||
if is_meta(str(fragment.get("concept", ""))):
|
||||
return "rejected", "meta"
|
||||
best = max(((max(_jaccard(n, m) for m in _names(e)), str(e["id"])) for e in index["entries"] if _names(e)),
|
||||
default=(0.0, ""))
|
||||
if best[0] >= MERGE_JACCARD:
|
||||
return "merge-candidate", best[1]
|
||||
return "new", None
|
||||
|
||||
|
||||
def relevant_lines(index: dict[str, Any], text: str) -> list[dict[str, Any]]:
|
||||
"""Index entries whose normalised name or alias occurs as a phrase in the normalised text (D7)."""
|
||||
hay = f" {norm(text)} "
|
||||
return [e for e in index["entries"] if any(f" {m} " in hay for m in _names(e))]
|
||||
|
||||
|
||||
def packs(names: list[str], size: int = 5) -> list[list[str]]:
|
||||
return [names[i:i + size] for i in range(0, len(names), size)]
|
||||
|
||||
|
||||
CONTRACT = """## Output contract
|
||||
|
||||
Write one JSON object `{"fragments": [...]}` to the output path named in your prompt. Each fragment:
|
||||
`concept`, `status` ("existing" with the index `id`, or "new" with `tests`: {"topic_shape", "citation"}),
|
||||
`text` (1-3 sentences), `anchor` {"file", "start", "end"} inside one block above, and `quote` (one piece copied from
|
||||
one line inside the anchor; required when the text holds a digit or a backticked name). A new concept only from a docs
|
||||
page, never from a release note.
|
||||
"""
|
||||
|
||||
|
||||
def render_input(pack_no: int, names: list[str], blocks_doc: dict[str, Any], index_lines: list[dict[str, Any]],
|
||||
kilde: Path) -> list[str]:
|
||||
lines = {n: (kilde / n).read_text(encoding="utf-8").split("\n") for n in names}
|
||||
pieces: list[str] = [f"# K pack {pack_no}: {len(names)} sources\n\nSources in order: {', '.join(names)}\n\n"
|
||||
+ CONTRACT + "\n## Blocks\n\n"]
|
||||
for b in blocks_doc["blocks"]:
|
||||
if b["file"] not in names:
|
||||
continue
|
||||
head = f"### {b['file']} › {b['section'] or '-'} › {b['start']}–{b['end']}"
|
||||
if b.get("version"):
|
||||
head += f" (version {b['version']})"
|
||||
body = [f"{n}: {lines[b['file']][n - 1]}" for n in range(b["start"], b["end"] + 1)]
|
||||
chunk = head + "\n"
|
||||
for row in body:
|
||||
if len(chunk) + len(row) + 1 > PART_CHARS // 2:
|
||||
pieces.append(chunk + "\n")
|
||||
chunk = head + " (continued)\n"
|
||||
chunk += row + "\n"
|
||||
pieces.append(chunk + "\n")
|
||||
pieces.append("## Index (concepts named in these blocks)\n\n")
|
||||
for e in index_lines:
|
||||
aliases = "; ".join(a for a in e["aliases"] if a != e["title"])
|
||||
pieces.append(f"- id: {e['id']} | {e['title']}" + (f" | aliases: {aliases}" if aliases else "") + "\n")
|
||||
room = PART_CHARS - len("<!-- part 99 of 99 -->\n")
|
||||
parts: list[str] = []
|
||||
current = ""
|
||||
for piece in pieces:
|
||||
if current and len(current) + len(piece) > room:
|
||||
parts.append(current)
|
||||
current = ""
|
||||
current += piece
|
||||
if current:
|
||||
parts.append(current)
|
||||
total = len(parts)
|
||||
return [f"<!-- part {i} of {total} -->\n" + p for i, p in enumerate(parts, 1)]
|
||||
|
||||
|
||||
def validate(fragments: list[Any], names: list[str], blocks_doc: dict[str, Any], kilde: Path
|
||||
) -> tuple[list[dict[str, Any]], Counter[str]]:
|
||||
by_file: dict[str, list[dict[str, Any]]] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
by_file.setdefault(b["file"], []).append(b)
|
||||
texts: dict[str, list[str]] = {}
|
||||
accepted: list[dict[str, Any]] = []
|
||||
rejected: Counter[str] = Counter()
|
||||
for f in fragments:
|
||||
a = f.get("anchor") if isinstance(f, dict) else None
|
||||
if not (isinstance(f, dict) and isinstance(f.get("concept"), str) and f["concept"].strip()
|
||||
and f.get("status") in ("new", "existing") and isinstance(f.get("text"), str)
|
||||
and isinstance(a, dict) and isinstance(a.get("file"), str)
|
||||
and type(a.get("start")) is int and type(a.get("end")) is int and a["start"] <= a["end"]):
|
||||
rejected["malformed"] += 1
|
||||
continue
|
||||
if a["file"] not in names:
|
||||
rejected["anchor-outside-pack"] += 1
|
||||
continue
|
||||
block = next((b for b in by_file.get(a["file"], []) if b["start"] <= a["start"] and a["end"] <= b["end"]), None)
|
||||
if block is None:
|
||||
rejected["anchor-crosses-blocks"] += 1
|
||||
continue
|
||||
if not 1 <= len(sentence_spans(f["text"].strip())) <= 3:
|
||||
rejected["sentence-count"] += 1
|
||||
continue
|
||||
if a["file"] not in texts:
|
||||
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
|
||||
span = texts[a["file"]][a["start"] - 1:a["end"]]
|
||||
quote = f.get("quote")
|
||||
if not (isinstance(quote, str) and quote.strip()):
|
||||
if NEEDS_QUOTE.search(f["text"]):
|
||||
rejected["quote-missing"] += 1
|
||||
continue
|
||||
quote, quote_line = None, None
|
||||
else:
|
||||
hit = next((a["start"] + i for i, line in enumerate(span) if quote in line), None)
|
||||
if hit is None:
|
||||
rejected["quote-not-in-span"] += 1
|
||||
continue
|
||||
quote_line = hit
|
||||
tests = f.get("tests")
|
||||
if f["status"] == "new" and not (isinstance(tests, dict) and str(tests.get("topic_shape", "")).strip()
|
||||
and str(tests.get("citation", "")).strip()):
|
||||
rejected["tests-missing"] += 1
|
||||
continue
|
||||
accepted.append({**f, "quote": quote, "quote_line": quote_line, "version": block.get("version"),
|
||||
"section": block["section"], "position": names.index(a["file"]) + 1})
|
||||
return accepted, rejected
|
||||
|
||||
|
||||
def _read_json(path: Path) -> Any:
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
raise RunError(f"cannot read {path}: {exc}") from exc
|
||||
|
||||
|
||||
def _write_json(path: Path, data: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _pack_log(work: Path) -> dict[str, Any]:
|
||||
path = work / "K-work" / "packs.json"
|
||||
return _read_json(path) if path.is_file() else {"schema": "okf-wiki/p1-packs/1", "packs": []}
|
||||
|
||||
|
||||
def _record(log: dict[str, Any], n: int) -> dict[str, Any]:
|
||||
for r in log["packs"]:
|
||||
if r["pack"] == n:
|
||||
return r # type: ignore[no-any-return]
|
||||
r = {"pack": n}
|
||||
log["packs"].append(r)
|
||||
return r
|
||||
|
||||
|
||||
def cmd_pack(args: argparse.Namespace) -> int:
|
||||
work, n = Path(args.work), args.n
|
||||
doc = _read_json(Path(args.blocks))
|
||||
all_packs = packs([str(x) for x in doc["order"]], args.size)
|
||||
if not 1 <= n <= len(all_packs):
|
||||
raise RunError(f"pack {n} does not exist; there are {len(all_packs)}")
|
||||
names = all_packs[n - 1]
|
||||
kilde = Path(args.kilde_root) / "kilde"
|
||||
index = _read_json(Path(args.index))
|
||||
text = "\n".join((kilde / name).read_text(encoding="utf-8") for name in names)
|
||||
lines = relevant_lines(index, text)
|
||||
parts = render_input(n, names, doc, lines, kilde)
|
||||
out_dir = work / "K-oppgaver"
|
||||
paths = [out_dir / f"pack-{n}-part-{i}.md" for i in range(1, len(parts) + 1)]
|
||||
if any(p.exists() for p in paths):
|
||||
raise RunError(f"pack {n}'s task parts exist in {out_dir}; refusing to overwrite")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
for p, text_part in zip(paths, parts):
|
||||
p.write_text(text_part, encoding="utf-8")
|
||||
log = _pack_log(work)
|
||||
rec = _record(log, n)
|
||||
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]), index_lines=len(lines),
|
||||
index_total=len(index["entries"]), index_ids=[str(e["id"]) for e in lines], parts=len(parts),
|
||||
chars=sum(len(p) for p in parts))
|
||||
_write_json(work / "K-work" / "packs.json", log)
|
||||
for p in paths:
|
||||
print(f"part: {p}")
|
||||
print(f"output: {work / 'K-fragments' / f'pack-{n}.json'}")
|
||||
print(f"index lines sent: {len(lines)} of {len(index['entries'])}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_accept(args: argparse.Namespace) -> int:
|
||||
work, n = Path(args.work), args.n
|
||||
doc = _read_json(Path(args.blocks))
|
||||
names = packs([str(x) for x in doc["order"]], args.size)[n - 1]
|
||||
kilde = Path(args.kilde_root) / "kilde"
|
||||
raw = _read_json(work / "K-fragments" / f"pack-{n}.json")
|
||||
fragments = raw.get("fragments") if isinstance(raw, dict) else None
|
||||
if not isinstance(fragments, list):
|
||||
raise RunError(f"pack {n}'s output has no fragments list")
|
||||
index_path = work / "K-work" / "index.json"
|
||||
index = _read_json(index_path)
|
||||
valid, rejected = validate(fragments, names, doc, kilde)
|
||||
acc_path = work / "K-work" / "accepted.json"
|
||||
acc: dict[str, Any] = (_read_json(acc_path) if acc_path.is_file()
|
||||
else {"schema": "okf-wiki/p1-accepted/1", "fragments": []})
|
||||
if any(a["pack"] == n for a in acc["fragments"]):
|
||||
raise RunError(f"pack {n} is already accepted")
|
||||
sent = set(_record(_pack_log(work), n).get("index_ids", []))
|
||||
not_sent: Counter[str] = Counter()
|
||||
accepted = 0
|
||||
for f in valid:
|
||||
outcome, ref = resolve(f, index)
|
||||
if outcome == "rejected":
|
||||
rejected[str(ref)] += 1
|
||||
continue
|
||||
if outcome == "existing":
|
||||
cid = str(ref)
|
||||
else:
|
||||
cid = _unique_id(index, f["concept"])
|
||||
index["entries"].append({"id": cid, "title": f["concept"], "aliases": [f["concept"]], "kind": outcome,
|
||||
"merge_of": ref, "origin": {"file": f["anchor"]["file"], "line": f["anchor"]["start"]},
|
||||
"tests": f.get("tests")})
|
||||
acc["fragments"].append({"pack": n, "position": f["position"], "file": f["anchor"]["file"],
|
||||
"start": f["anchor"]["start"], "end": f["anchor"]["end"], "section": f["section"],
|
||||
"text": f["text"].strip(), "quote": f["quote"], "quote_line": f["quote_line"],
|
||||
"version": f["version"], "concept_id": cid, "outcome": outcome})
|
||||
if cid not in sent:
|
||||
not_sent[outcome] += 1
|
||||
accepted += 1
|
||||
_write_json(index_path, index)
|
||||
_write_json(acc_path, acc)
|
||||
log = _pack_log(work)
|
||||
rec = _record(log, n)
|
||||
rec.update(proposed=len(fragments), accepted=accepted, rejected=dict(sorted(rejected.items())),
|
||||
not_sent={"total": sum(not_sent.values()), "by_outcome": dict(sorted(not_sent.items()))})
|
||||
_write_json(work / "K-work" / "packs.json", log)
|
||||
print(f"pack {n}: accepted {accepted} of {len(fragments)}")
|
||||
print(f" accepted with a concept not among the index lines sent: {sum(not_sent.values())}")
|
||||
for reason, count in sorted(rejected.items()):
|
||||
print(f" rejected {reason}: {count}")
|
||||
return 0
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
s = sub.add_parser("skeleton")
|
||||
s.add_argument("--kilde", required=True)
|
||||
s.add_argument("--out", required=True)
|
||||
for name in ("pack", "accept"):
|
||||
c = sub.add_parser(name)
|
||||
c.add_argument("n", type=int)
|
||||
c.add_argument("--blocks", required=True)
|
||||
c.add_argument("--kilde-root", required=True)
|
||||
c.add_argument("--work", required=True)
|
||||
c.add_argument("--size", type=int, default=5)
|
||||
if name == "pack":
|
||||
c.add_argument("--index", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "skeleton":
|
||||
out = Path(args.out)
|
||||
if out.exists():
|
||||
raise RunError(f"{out} exists; refusing to overwrite")
|
||||
index = skeleton(Path(args.kilde))
|
||||
_write_json(out, index)
|
||||
print(f"skeleton: {len(index['entries'])} entries -> {out}")
|
||||
return 0
|
||||
if args.command == "pack":
|
||||
return cmd_pack(args)
|
||||
return cmd_accept(args)
|
||||
except (RunError, OSError) as exc:
|
||||
print(f"kpack cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
361
evals/p1/measure.py
Normal file
361
evals/p1/measure.py
Normal file
|
|
@ -0,0 +1,361 @@
|
|||
"""The trial's measures without a reader (TRIAL, P1): audit record, coverage, structure. It calls no model.
|
||||
|
||||
audit --bundle B --root R --variant V --out FILE
|
||||
Runs the auditor in process and records, per variant, the exit status, findings per check, uncited
|
||||
sentences by cause (fence line, table row, link lead-in, prose), "yaml form" findings (a quote that YAML
|
||||
loaded as another type, a timestamp in the wrong form) and criterion 1 (N of N, claims.py's count).
|
||||
coverage --bundle B --root R --blocks F --built N [--pack-size K] --out FILE
|
||||
Blocks of the first N sources in the order with at least one body-cited source entry whose locator line
|
||||
lies inside the block (spans are not used), X of N blocks; per pack position with --pack-size.
|
||||
structure --bundle B --root R --blocks F [--skeleton I] --out FILE
|
||||
Pages, concepts per source (the earliest source in the order among a page's body-cited sources),
|
||||
skeleton entries without a page, duplicates (same name, alias or title Jaccard >= 0.6), merge
|
||||
candidates, islands, sources per page, version numbers in running text, concept tests 2 and 4,
|
||||
the share of quotes on a structural line and of code-supplied quotes (version sources left out), and the
|
||||
precision of each body-cited entry (line, block, section, document).
|
||||
|
||||
--root is the folder that contains kilde/. Before reporting, structure and coverage check that at least one quote
|
||||
stands at its locator under --root, because a wrong root makes every quote unreadable. Exit 0 = written, 2 = could
|
||||
not run (an existing output, or a variant already in the audit record, is never overwritten).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki import claims as claims_mod
|
||||
from okfwiki.audit import (
|
||||
FOOTNOTE_REF,
|
||||
LINK,
|
||||
LINK_ONLY,
|
||||
Document,
|
||||
audit,
|
||||
body_links,
|
||||
is_bundle_link,
|
||||
load_bundle,
|
||||
mask_code,
|
||||
prose_blocks,
|
||||
resolve,
|
||||
sentences,
|
||||
)
|
||||
from okfwiki.view import is_version_heading, related, versions
|
||||
from p1.assemble import structural
|
||||
from p1.kpack import MERGE_JACCARD, norm
|
||||
from p1.split import FENCE, is_meta
|
||||
|
||||
STRUCTURE_SCHEMA = "okf-wiki/p1-structure/1"
|
||||
RESULT_SCHEMA = "okf-wiki/p1-result/1"
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The measure cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Coverage:
|
||||
covered: int
|
||||
total: int
|
||||
per_position: dict[int, tuple[int, int, int]] = field(default_factory=dict)
|
||||
|
||||
|
||||
def _entries(doc: Document) -> dict[str, dict[str, Any]]:
|
||||
srcs = (doc.frontmatter or {}).get("sources")
|
||||
return {str(s["id"]): s for s in srcs if isinstance(s, dict) and "id" in s} if isinstance(srcs, list) else {}
|
||||
|
||||
|
||||
def cited(doc: Document) -> list[dict[str, Any]]:
|
||||
"""The source entries a body footnote uses, in label order of first use."""
|
||||
entries = _entries(doc)
|
||||
labels: list[str] = []
|
||||
for block in prose_blocks(doc):
|
||||
for label in FOOTNOTE_REF.findall(mask_code(block.text)):
|
||||
if label not in labels:
|
||||
labels.append(label)
|
||||
return [entries[x] for x in labels if x in entries]
|
||||
|
||||
|
||||
def _file(s: dict[str, Any]) -> str:
|
||||
return Path(str(s.get("resource", ""))).name
|
||||
|
||||
|
||||
def _line(s: dict[str, Any]) -> int | None:
|
||||
loc = s.get("x-locator")
|
||||
return loc["line"] if isinstance(loc, dict) and type(loc.get("line")) is int else None
|
||||
|
||||
|
||||
class _Lines:
|
||||
def __init__(self, root: Path) -> None:
|
||||
self.root = root
|
||||
self.cache: dict[str, list[str] | None] = {}
|
||||
|
||||
def get(self, name: str) -> list[str] | None:
|
||||
if name not in self.cache:
|
||||
p = self.root / "kilde" / name
|
||||
self.cache[name] = p.read_text(encoding="utf-8").split("\n") if p.is_file() else None
|
||||
return self.cache[name]
|
||||
|
||||
|
||||
def _check_root(concepts: list[Document], root: Path) -> None:
|
||||
lines = _Lines(root)
|
||||
seen = 0
|
||||
for d in concepts:
|
||||
for s in _entries(d).values():
|
||||
n, quote, text = _line(s), s.get("x-quote"), lines.get(_file(s))
|
||||
seen += 1
|
||||
if n and isinstance(quote, str) and text and n <= len(text) and quote in text[n - 1]:
|
||||
return
|
||||
if seen:
|
||||
raise RunError(f"no quote stands at its locator under {root}; is it the folder that contains kilde/?")
|
||||
|
||||
|
||||
def _block_of(blocks: list[dict[str, Any]], file: str, line: int) -> dict[str, Any] | None:
|
||||
return next((b for b in blocks if b["file"] == file and b["start"] <= line <= b["end"]), None)
|
||||
|
||||
|
||||
def coverage(bundle: Path, root: Path, blocks_doc: dict[str, Any], built: list[str],
|
||||
pack_size: int | None = None) -> Coverage:
|
||||
concepts = load_bundle(bundle, root).concepts
|
||||
_check_root(concepts, root)
|
||||
blocks = [b for b in blocks_doc["blocks"] if b["file"] in built]
|
||||
hit: set[tuple[str, int]] = set()
|
||||
for d in concepts:
|
||||
for s in cited(d):
|
||||
n = _line(s)
|
||||
b = _block_of(blocks, _file(s), n) if n else None
|
||||
if b is not None:
|
||||
hit.add((b["file"], b["start"]))
|
||||
result = Coverage(len(hit), len(blocks))
|
||||
if pack_size:
|
||||
names = [str(x) for x in blocks_doc["order"] if x in built]
|
||||
for pos in range(1, pack_size + 1):
|
||||
files = {n for i, n in enumerate(names) if i % pack_size == pos - 1}
|
||||
if files:
|
||||
mine = [b for b in blocks if b["file"] in files]
|
||||
result.per_position[pos] = (sum((b["file"], b["start"]) in hit for b in mine), len(mine), len(files))
|
||||
return result
|
||||
|
||||
|
||||
def _jaccard(a: str, b: str) -> float:
|
||||
x, y = set(a.split()), set(b.split())
|
||||
return len(x & y) / len(x | y) if x | y else 0.0
|
||||
|
||||
|
||||
def duplicates(concepts: list[Document]) -> list[tuple[str, str, str]]:
|
||||
info = []
|
||||
for d in concepts:
|
||||
fm = d.frontmatter or {}
|
||||
aliases = fm.get("x-aliases")
|
||||
info.append((d.rel, norm(str(fm.get("title", ""))),
|
||||
{norm(str(a)) for a in aliases} if isinstance(aliases, list) else set()))
|
||||
pairs: list[tuple[str, str, str]] = []
|
||||
for i, (a, na, aa) in enumerate(info):
|
||||
for b, nb, ab in info[i + 1:]:
|
||||
if na and na == nb:
|
||||
pairs.append((a, b, "norm"))
|
||||
elif na in ab or nb in aa:
|
||||
pairs.append((a, b, "alias"))
|
||||
elif _jaccard(na, nb) >= MERGE_JACCARD:
|
||||
pairs.append((a, b, "jaccard"))
|
||||
return pairs
|
||||
|
||||
|
||||
def _kind(d: Document) -> str:
|
||||
fm = d.frontmatter or {}
|
||||
if fm.get("x-merge-candidate"):
|
||||
return "merge-candidate"
|
||||
return {"Document map": "map", "Register entry": "register"}.get(str(fm.get("type", "")), "concept")
|
||||
|
||||
|
||||
def _precision(s: dict[str, Any], blocks: list[dict[str, Any]]) -> str:
|
||||
span = s.get("x-span")
|
||||
n = _line(s) or 0
|
||||
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (n, n)
|
||||
if a == z:
|
||||
return "line"
|
||||
inside = [b for b in blocks if b["file"] == _file(s) and b["end"] >= a and b["start"] <= z]
|
||||
if len(inside) == 1 and inside[0]["start"] <= a and z <= inside[0]["end"]:
|
||||
return "block"
|
||||
if inside and len({b["section"] for b in inside}) == 1:
|
||||
return "section"
|
||||
return "document"
|
||||
|
||||
|
||||
def structure(bundle: Path, root: Path, blocks_doc: dict[str, Any], order: list[str],
|
||||
skeleton: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
concepts = load_bundle(bundle, root).concepts
|
||||
_check_root(concepts, root)
|
||||
lines = _Lines(root)
|
||||
blocks = list(blocks_doc["blocks"])
|
||||
rank = {n: i for i, n in enumerate(order)}
|
||||
per_source: Counter[str] = Counter()
|
||||
incoming: Counter[str] = Counter()
|
||||
rels = {d.rel for d in concepts}
|
||||
for d in concepts:
|
||||
for _line_no, target in body_links(d):
|
||||
other = resolve(d.rel, target) if is_bundle_link(target) else None
|
||||
if other in rels and other != d.rel:
|
||||
incoming[str(other)] += 1
|
||||
quotes: Counter[str] = Counter()
|
||||
precision: Counter[str] = Counter()
|
||||
docs_per_page: list[int] = []
|
||||
test2 = test4 = 0
|
||||
for d in concepts:
|
||||
cs = cited(d)
|
||||
files = [_file(s) for s in cs if _file(s) in rank]
|
||||
if files:
|
||||
per_source[min(files, key=lambda f: rank[f])] += 1
|
||||
docs_per_page.append(len(set(_file(s) for s in cs)))
|
||||
title = str((d.frontmatter or {}).get("title", ""))
|
||||
test2 += not is_meta(title)
|
||||
test4 += incoming[d.rel] >= 2 or len(set(files)) >= 2
|
||||
for s in cs:
|
||||
if is_version_heading(s):
|
||||
continue
|
||||
quotes["entries"] += 1
|
||||
quotes["code"] += s.get("x-quote-by") == "code"
|
||||
text, n = lines.get(_file(s)), _line(s)
|
||||
quotes["structural"] += bool(text and n and n <= len(text) and structural(text[n - 1]))
|
||||
precision[_precision(s, blocks)] += 1
|
||||
rel_map = related(concepts)
|
||||
kinds = {d.rel: _kind(d) for d in concepts}
|
||||
islands: dict[str, Any] = {"total": sum(not v for v in rel_map.values())}
|
||||
for k in sorted(set(kinds.values())):
|
||||
islands[k] = sum(not rel_map[r] for r, kk in kinds.items() if kk == k)
|
||||
pages = {d.rel for d in concepts}
|
||||
unused = None
|
||||
if skeleton is not None:
|
||||
unused = sum(e.get("kind") == "skeleton" and f"{e['id']}.md" not in pages for e in skeleton["entries"])
|
||||
tests13 = None
|
||||
if skeleton is not None:
|
||||
new_ids = {e["id"] for e in skeleton["entries"] if e.get("kind") == "new" and e.get("tests")}
|
||||
tests13 = sum(f"{i}.md" in pages for i in new_ids)
|
||||
v = versions(bundle, root)
|
||||
dups = duplicates(concepts)
|
||||
return {
|
||||
"schema": STRUCTURE_SCHEMA,
|
||||
"pages": len(concepts),
|
||||
"page_kinds": dict(Counter(kinds.values())),
|
||||
"concepts_per_source": {n: per_source[n] for n in order if per_source[n]},
|
||||
"skeleton_unused": unused,
|
||||
"duplicates": {"pairs": len(dups), "by_rule": dict(Counter(r for _, _, r in dups)),
|
||||
"list": [list(p) for p in dups]},
|
||||
"merge_candidates": sum(k == "merge-candidate" for k in kinds.values()),
|
||||
"islands": islands,
|
||||
"documents_per_page": {"mean": round(sum(docs_per_page) / len(docs_per_page), 2) if docs_per_page else 0,
|
||||
"max": max(docs_per_page, default=0)},
|
||||
"version_numbers_in_text": {"flagged": len(v.flagged), "claims": v.total},
|
||||
"concept_tests": {"test2": test2, "test4": test4, "tests1and3_self_declared": tests13},
|
||||
"quotes": {"entries": quotes["entries"], "structural": quotes["structural"], "code": quotes["code"]},
|
||||
"precision": {k: precision[k] for k in ("line", "block", "section", "document")},
|
||||
}
|
||||
|
||||
|
||||
def _uncited_cause(sentence: str, block_text: str) -> str:
|
||||
s = sentence.strip()
|
||||
if FENCE.match(block_text): # the auditor's sentence text masks code spans, so a fence is read on the block
|
||||
return "fence line"
|
||||
if s.startswith("|"):
|
||||
return "table row"
|
||||
if LINK.search(block_text) and not LINK_ONLY.match(mask_code(block_text)):
|
||||
return "link lead-in"
|
||||
return "prose"
|
||||
|
||||
|
||||
def audit_record(bundle: Path, root: Path, variant: str, out: Path) -> dict[str, Any]:
|
||||
record_all: dict[str, Any] = (json.loads(out.read_text(encoding="utf-8")) if out.is_file()
|
||||
else {"schema": RESULT_SCHEMA, "variants": {}})
|
||||
if variant in record_all["variants"]:
|
||||
raise RunError(f"{out} already records variant {variant}")
|
||||
b = load_bundle(bundle, root)
|
||||
found = audit(bundle, root)
|
||||
uncited: Counter[str] = Counter()
|
||||
for d in b.concepts:
|
||||
for block in prose_blocks(d):
|
||||
if LINK_ONLY.match(mask_code(block.text)):
|
||||
continue
|
||||
for s in sentences(block.text):
|
||||
if not FOOTNOTE_REF.search(s):
|
||||
uncited[_uncited_cause(s, block.text)] += 1
|
||||
yaml_form = 0
|
||||
for d in b.concepts:
|
||||
for entry in _entries(d).values():
|
||||
q = entry.get("x-quote")
|
||||
yaml_form += q is not None and not isinstance(q, str)
|
||||
yaml_form += sum(f.check == "timestamp-form" for f in found)
|
||||
listed = claims_mod.extract(bundle, root)
|
||||
ok, total = claims_mod.criterion_1(listed)["total"]
|
||||
rec = {"exit": 1 if found else 0, "findings": dict(sorted(Counter(f.check for f in found).items())),
|
||||
"uncited": dict(sorted(uncited.items())), "yaml_form": yaml_form, "criterion_1": [ok, total]}
|
||||
record_all["variants"][variant] = rec
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(record_all, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
return rec
|
||||
|
||||
|
||||
def _write_new(path: Path, data: Any) -> None:
|
||||
if path.exists():
|
||||
raise RunError(f"{path} exists; refusing to overwrite")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
a = sub.add_parser("audit")
|
||||
a.add_argument("--bundle", required=True)
|
||||
a.add_argument("--root", required=True)
|
||||
a.add_argument("--variant", required=True)
|
||||
a.add_argument("--out", required=True)
|
||||
c = sub.add_parser("coverage")
|
||||
s = sub.add_parser("structure")
|
||||
for x in (c, s):
|
||||
x.add_argument("--bundle", required=True)
|
||||
x.add_argument("--root", required=True)
|
||||
x.add_argument("--blocks", required=True)
|
||||
x.add_argument("--out", required=True)
|
||||
c.add_argument("--built", type=int, required=True)
|
||||
c.add_argument("--pack-size", type=int)
|
||||
s.add_argument("--skeleton")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
bundle, root = Path(args.bundle), Path(args.root)
|
||||
if not bundle.is_dir():
|
||||
raise RunError(f"the bundle {bundle} is not a directory")
|
||||
if args.command == "audit":
|
||||
rec = audit_record(bundle, root, args.variant, Path(args.out))
|
||||
print(f"{args.variant}: audit exit {rec['exit']}, criterion 1 {rec['criterion_1'][0]} of "
|
||||
f"{rec['criterion_1'][1]}, findings {rec['findings']}")
|
||||
return 0
|
||||
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
order = [str(x) for x in blocks_doc["order"]]
|
||||
if args.command == "coverage":
|
||||
cov = coverage(bundle, root, blocks_doc, order[:args.built], args.pack_size)
|
||||
_write_new(Path(args.out), {"schema": "okf-wiki/p1-coverage/1", "covered": cov.covered, "total": cov.total,
|
||||
"built": args.built,
|
||||
"per_position": {str(k): list(v) for k, v in cov.per_position.items()}})
|
||||
print(f"coverage: {cov.covered} of {cov.total} blocks")
|
||||
return 0
|
||||
skeleton = json.loads(Path(args.skeleton).read_text(encoding="utf-8")) if args.skeleton else None
|
||||
data = structure(bundle, root, blocks_doc, order, skeleton)
|
||||
_write_new(Path(args.out), data)
|
||||
print(f"structure: {data['pages']} pages, {data['duplicates']['pairs']} duplicate pairs, "
|
||||
f"{data['islands']['total']} islands")
|
||||
return 0
|
||||
except (RunError, OSError, json.JSONDecodeError, KeyError) as exc:
|
||||
print(f"measure cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
185
evals/p1/sidebyside.py
Normal file
185
evals/p1/sidebyside.py
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
"""The side-by-side page (TRIAL, P1): the same concepts in every build, every source link checked. It calls no model.
|
||||
|
||||
render --begreper FILE --col LABEL=DIR ... [--col LABEL=-] --kilde-root DIR --out FILE
|
||||
One self-contained HTML page: one row per concept in begreper.txt (name, a tab, aliases joined by ;), one
|
||||
column per build. A cell shows the pages whose normalised title or alias equals the concept's (for the L4
|
||||
column: whose normalised title or description contains it), each with its body and its sources
|
||||
(file › line or span). An empty cell says why: "ikke bygget" (a build given as -, stopped), "ikke i
|
||||
registeret" (D has no register page for it), "ingen side" (built, no page).
|
||||
check --html FILE --kilde-root DIR
|
||||
A source link resolves when its file exists under kilde/ and its line or span lies inside the file.
|
||||
Prints "resolved X of N"; exit 1 when X < N.
|
||||
|
||||
Exit 0 = done, 1 = an unresolved link (check), 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import load_bundle
|
||||
from okfwiki.view import esc
|
||||
from p1.kpack import norm
|
||||
from p1.measure import cited
|
||||
|
||||
Concept = tuple[str, tuple[str, ...]]
|
||||
SRC = re.compile(r'<a class="src" href="[^"]*" data-file="([^"]+)" data-start="(\d+)" data-end="(\d+)">')
|
||||
STYLE = """
|
||||
:root { --bg: #fdfdfb; --fg: #1d1d1b; --muted: #6b6b66; --line: #ddd9cf; --cell: #ffffff; }
|
||||
@media (prefers-color-scheme: dark) { :root { --bg: #1b1b1a; --fg: #ecebe6; --muted: #a3a29b; --line: #3a3934;
|
||||
--cell: #242422; } }
|
||||
body { background: var(--bg); color: var(--fg); font: 15px/1.5 system-ui, sans-serif; margin: 16px; }
|
||||
table { border-collapse: collapse; width: 100%; table-layout: fixed; }
|
||||
th, td { border: 1px solid var(--line); vertical-align: top; padding: 8px; background: var(--cell); }
|
||||
th.concept-name { width: 10em; text-align: left; }
|
||||
.empty { color: var(--muted); font-style: italic; }
|
||||
.page h3 { margin: 0 0 4px; font-size: 15px; }
|
||||
.sources { color: var(--muted); font-size: 13px; }
|
||||
.wrap { overflow-x: auto; }
|
||||
"""
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
def read_begreper(path: Path) -> list[Concept]:
|
||||
out: list[Concept] = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
if line.strip():
|
||||
name, _, aliases = line.partition("\t")
|
||||
out.append((name.strip(), tuple(a.strip() for a in aliases.split(";") if a.strip()) or (name.strip(),)))
|
||||
return out
|
||||
|
||||
|
||||
def _l4_norm(text: str) -> str:
|
||||
return f" {norm(text.replace('-', ' '))} "
|
||||
|
||||
|
||||
def find(bundle: Path, concept: Concept, l4: bool = False) -> list[str]:
|
||||
wanted = {norm(n) for n in (concept[0], *concept[1])} - {""}
|
||||
found: list[str] = []
|
||||
for d in load_bundle(bundle, bundle).concepts:
|
||||
fm = d.frontmatter or {}
|
||||
if l4:
|
||||
hay = _l4_norm(str(fm.get("title", ""))) + _l4_norm(str(fm.get("description", "")))
|
||||
if any(f" {norm(w.replace('-', ' '))} " in hay for w in wanted):
|
||||
found.append(d.rel)
|
||||
continue
|
||||
aliases = fm.get("x-aliases")
|
||||
names = {norm(str(fm.get("title", "")))} | ({norm(str(a)) for a in aliases} if isinstance(aliases, list)
|
||||
else set())
|
||||
if names & wanted:
|
||||
found.append(d.rel)
|
||||
return found
|
||||
|
||||
|
||||
def _sources_html(entries: list[dict[str, Any]], kilde: Path, out_dir: Path) -> str:
|
||||
items = []
|
||||
for s in entries:
|
||||
file = Path(str(s.get("resource", ""))).name
|
||||
loc = s.get("x-locator")
|
||||
line = loc.get("line") if isinstance(loc, dict) else None
|
||||
span = s.get("x-span")
|
||||
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (line, line)
|
||||
if not isinstance(a, int) or not isinstance(z, int):
|
||||
continue
|
||||
href = os.path.relpath(kilde / file, out_dir).replace(os.sep, "/") + f"#L{a}"
|
||||
where = f"{a}" if a == z else f"{a}–{z}"
|
||||
items.append(f'<li><a class="src" href="{esc(href)}" data-file="{esc(file)}" data-start="{a}" '
|
||||
f'data-end="{z}">{esc(file)} › {where}</a></li>')
|
||||
return '<ul class="sources">' + "".join(items) + "</ul>"
|
||||
|
||||
|
||||
def _cell(bundle: Path | None, label: str, concept: Concept, kilde_root: Path, out_dir: Path) -> str:
|
||||
if bundle is None:
|
||||
return '<td class="cell empty">ikke bygget</td>'
|
||||
rels = find(bundle, concept, l4=label == "L4")
|
||||
if not rels:
|
||||
return f'<td class="cell empty">{"ikke i registeret" if label == "D" else "ingen side"}</td>'
|
||||
docs = {d.rel: d for d in load_bundle(bundle, kilde_root).concepts}
|
||||
parts = []
|
||||
for rel in rels:
|
||||
d = docs[rel]
|
||||
fm = d.frontmatter or {}
|
||||
srcs = fm.get("sources")
|
||||
entries = cited(d) or ([s for s in srcs if isinstance(s, dict)] if isinstance(srcs, list) else [])
|
||||
paras = [p.strip() for p in re.split(r"\n\s*\n", d.body) if p.strip() and not p.strip().startswith("# ")]
|
||||
body = "".join(f"<p>{esc(p)}</p>" for p in paras if not p.startswith("[^"))
|
||||
parts.append(f'<div class="page"><h3>{esc(str(fm.get("title", rel)))} <small>({esc(rel)})</small></h3>'
|
||||
f"{body}{_sources_html(entries, kilde_root / 'kilde', out_dir)}</div>")
|
||||
return '<td class="cell">' + "".join(parts) + "</td>"
|
||||
|
||||
|
||||
def render(begreper: list[Concept], columns: list[tuple[str, Path | None]], kilde_root: Path, out: Path) -> None:
|
||||
out_dir = out.parent
|
||||
head = "".join(f"<th>{esc(label)}</th>" for label, _ in columns)
|
||||
rows = []
|
||||
for concept in begreper:
|
||||
cells = "".join(_cell(b, label, concept, kilde_root, out_dir) for label, b in columns)
|
||||
rows.append(f'<tr class="concept"><th class="concept-name">{esc(concept[0])}</th>{cells}</tr>')
|
||||
html = (f'<!doctype html><html lang="no"><head><meta charset="utf-8">'
|
||||
f'<meta name="viewport" content="width=device-width, initial-scale=1"><title>P1 side om side</title>'
|
||||
f"<style>{STYLE}</style></head><body><h1>P1: tre begreper, side om side</h1>"
|
||||
f'<div class="wrap"><table><tr><th class="concept-name">Begrep</th>{head}</tr>{"".join(rows)}'
|
||||
f"</table></div></body></html>\n")
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(html, encoding="utf-8")
|
||||
|
||||
|
||||
def check(html_path: Path, kilde_root: Path) -> tuple[int, int]:
|
||||
html = html_path.read_text(encoding="utf-8")
|
||||
resolved = total = 0
|
||||
sizes: dict[str, int | None] = {}
|
||||
for file, a, z in SRC.findall(html):
|
||||
total += 1
|
||||
if file not in sizes:
|
||||
p = kilde_root / "kilde" / file
|
||||
sizes[file] = len(p.read_text(encoding="utf-8").splitlines()) if p.is_file() else None
|
||||
n = sizes[file]
|
||||
resolved += n is not None and 1 <= int(a) <= int(z) <= n
|
||||
return resolved, total
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
r = sub.add_parser("render")
|
||||
r.add_argument("--begreper", required=True)
|
||||
r.add_argument("--col", action="append", required=True, help="LABEL=DIR, or LABEL=- for a stopped build")
|
||||
r.add_argument("--kilde-root", required=True)
|
||||
r.add_argument("--out", required=True)
|
||||
c = sub.add_parser("check")
|
||||
c.add_argument("--html", required=True)
|
||||
c.add_argument("--kilde-root", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "render":
|
||||
columns: list[tuple[str, Path | None]] = []
|
||||
for col in args.col:
|
||||
label, _, path = col.partition("=")
|
||||
if not label or not path:
|
||||
raise RunError(f"--col {col!r} is not LABEL=DIR")
|
||||
columns.append((label, None if path == "-" else Path(path)))
|
||||
render(read_begreper(Path(args.begreper)), columns, Path(args.kilde_root), Path(args.out))
|
||||
print(f"side by side: {len(columns)} columns -> {args.out}")
|
||||
return 0
|
||||
resolved, total = check(Path(args.html), Path(args.kilde_root))
|
||||
print(f"resolved {resolved} of {total}")
|
||||
return 0 if total and resolved == total else 1
|
||||
except (RunError, OSError) as exc:
|
||||
print(f"sidebyside cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
328
evals/p1/split.py
Normal file
328
evals/p1/split.py
Normal file
|
|
@ -0,0 +1,328 @@
|
|||
"""Split the trial's sources into a fixed order and into blocks with line spans (TRIAL, P1). It calls no model.
|
||||
|
||||
order --manifest M --l4 L4DIR --out FILE
|
||||
The 19 names of the cut (gates/wholedoc.manifest.json cut.names) in the trial's fixed order: docs pages by
|
||||
name, then release notes by version. Refuses when the names differ from the files the L4 cut cites.
|
||||
blocks --kilde DIR --order FILE --out FILE
|
||||
Every block of every source in the order: a paragraph, one list item with its continuation lines, one
|
||||
table or one fenced code block, with its 1-based inclusive line span, its section (the heading path) and,
|
||||
for a release note, its version. Heading lines are section markers, not blocks.
|
||||
total --kilde DIR
|
||||
The number of blocks in every file under DIR (the denominator of the 599-source projection).
|
||||
pick --blocks FILE --kilde DIR [--focus NAME] --out FILE
|
||||
The three concepts of the side-by-side page, picked from the source text before any build: the focus page's
|
||||
heading with the most blocks, the term in the most release notes, and the term in at least two docs pages and
|
||||
two release notes with the highest document count; a document counts for a term only where the term stands
|
||||
as a term (backticked or a capitalised phrase), never as a word in prose. One line per concept: name, a tab,
|
||||
aliases joined by ;.
|
||||
|
||||
Exit 0 = written, 2 = could not run (an existing output is never overwritten).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import HEADING, LIST_ITEM, read_document
|
||||
|
||||
SCHEMA = "okf-wiki/p1-blocks/1"
|
||||
RELEASE = re.compile(r"^release-v(\d+(?:\.\d+)*)\.md$")
|
||||
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
|
||||
FENCE = re.compile(r"^\s*(```|~~~)")
|
||||
# Concept test 2 of the brief (Google's reference agent), plus the generic section names of docs pages.
|
||||
META_STOP = frozenset({
|
||||
"overview", "introduction", "getting started", "quickstart", "tutorial", "walkthrough", "release notes",
|
||||
"changelog", "roadmap", "faq", "prerequisites", "next steps", "before you begin", "troubleshooting", "see also",
|
||||
"related resources", "learn more",
|
||||
})
|
||||
CODE_TERM = re.compile(r"`([^`\n]{3,40})`")
|
||||
CAPITAL_PHRASE = re.compile(r"\b[A-Z][\w-]*(?:[ \t]+[A-Z][\w-]*){1,3}\b")
|
||||
SENTENCE_START = re.compile(r"(?:^|[.!?:]\s+|^\s*(?:[-*+]|\d+[.)])\s+)$")
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SBlock:
|
||||
file: str
|
||||
start: int
|
||||
end: int
|
||||
kind: str # paragraph | item | table | code
|
||||
section: str
|
||||
version: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Heading:
|
||||
file: str
|
||||
line: int
|
||||
level: int
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Picked:
|
||||
name: str
|
||||
aliases: tuple[str, ...]
|
||||
|
||||
|
||||
def is_meta(title: str) -> bool:
|
||||
return title.strip().lower() in META_STOP
|
||||
|
||||
|
||||
def version_key(name: str) -> tuple[int, ...]:
|
||||
m = RELEASE.match(name)
|
||||
if m is None:
|
||||
raise RunError(f"{name} is not a release note name")
|
||||
return tuple(int(p) for p in m.group(1).split("."))
|
||||
|
||||
|
||||
def order(names: list[str]) -> list[str]:
|
||||
docs = sorted(n for n in names if n.startswith("docs-"))
|
||||
releases = [n for n in names if n.startswith("release-")]
|
||||
other = [n for n in names if n not in docs and n not in releases]
|
||||
if other:
|
||||
raise RunError(f"names that are neither docs pages nor release notes: {other}")
|
||||
return docs + sorted(releases, key=version_key)
|
||||
|
||||
|
||||
def headings_of(lines: list[str]) -> list[tuple[int, int, str]]:
|
||||
"""(line, level, text) of every heading outside fenced code."""
|
||||
out: list[tuple[int, int, str]] = []
|
||||
fence: str | None = None
|
||||
for n, raw in enumerate(lines, 1):
|
||||
f = FENCE.match(raw)
|
||||
if f:
|
||||
fence = None if fence == f.group(1) else (fence or f.group(1))
|
||||
continue
|
||||
if fence is None:
|
||||
m = HEADING.match(raw)
|
||||
if m:
|
||||
out.append((n, len(m.group(1)), (m.group(2) or "").strip()))
|
||||
return out
|
||||
|
||||
|
||||
def blocks(path: Path) -> list[SBlock]:
|
||||
lines = path.read_text(encoding="utf-8").split("\n")
|
||||
name = path.name
|
||||
version: str | None = None
|
||||
if name.startswith("release-"):
|
||||
for _n, _level, text in headings_of(lines):
|
||||
m = VERSION_HEADING.match(text)
|
||||
if m:
|
||||
version = m.group(1)
|
||||
break
|
||||
out: list[SBlock] = []
|
||||
stack: list[tuple[int, str]] = []
|
||||
kind: str | None = None
|
||||
start = 0
|
||||
fence: str | None = None
|
||||
|
||||
def close(end: int) -> None:
|
||||
nonlocal kind
|
||||
if kind is not None:
|
||||
out.append(SBlock(name, start, end, kind, " › ".join(t for _, t in stack), version))
|
||||
kind = None
|
||||
|
||||
for n, raw in enumerate(lines, 1):
|
||||
f = FENCE.match(raw)
|
||||
if fence is not None:
|
||||
if f and f.group(1) == fence:
|
||||
fence = None
|
||||
close(n)
|
||||
continue
|
||||
if f:
|
||||
close(n - 1)
|
||||
fence, kind, start = f.group(1), "code", n
|
||||
continue
|
||||
if not raw.strip():
|
||||
close(n - 1)
|
||||
continue
|
||||
m = HEADING.match(raw)
|
||||
if m:
|
||||
close(n - 1)
|
||||
level = len(m.group(1))
|
||||
stack = [(lv, t) for lv, t in stack if lv < level] + [(level, (m.group(2) or "").strip())]
|
||||
continue
|
||||
if raw.lstrip().startswith("|"):
|
||||
if kind != "table":
|
||||
close(n - 1)
|
||||
kind, start = "table", n
|
||||
continue
|
||||
if LIST_ITEM.match(raw):
|
||||
close(n - 1)
|
||||
kind, start = "item", n
|
||||
continue
|
||||
if kind in (None, "table"):
|
||||
close(n - 1)
|
||||
kind, start = "paragraph", n
|
||||
if fence is not None:
|
||||
close(len(lines))
|
||||
else:
|
||||
last = len(lines)
|
||||
while last > 0 and not lines[last - 1].strip():
|
||||
last -= 1
|
||||
close(last)
|
||||
return out
|
||||
|
||||
|
||||
def docs_headings(kilde: Path) -> list[Heading]:
|
||||
out: list[Heading] = []
|
||||
for path in sorted(kilde.glob("docs-*.md")):
|
||||
for n, level, text in headings_of(path.read_text(encoding="utf-8").split("\n")):
|
||||
if level <= 3 and text:
|
||||
out.append(Heading(path.name, n, level, text))
|
||||
return out
|
||||
|
||||
|
||||
def blocks_document(kilde: Path, names: list[str]) -> dict[str, Any]:
|
||||
found: list[dict[str, Any]] = []
|
||||
for name in names:
|
||||
path = kilde / name
|
||||
if not path.is_file():
|
||||
raise RunError(f"{path} is not a file")
|
||||
found.extend(asdict(b) for b in blocks(path))
|
||||
return {"schema": SCHEMA, "order": names, "blocks": found}
|
||||
|
||||
|
||||
def blocks_total(kilde: Path) -> int:
|
||||
return sum(len(blocks(p)) for p in sorted(kilde.glob("*.md")))
|
||||
|
||||
|
||||
def terms(text: str) -> list[str]:
|
||||
"""Backticked spans of 3-40 characters and capitalised phrases of 2-4 words, a sentence-initial word dropped."""
|
||||
found: list[str] = [t for t in (m.group(1).strip() for m in CODE_TERM.finditer(text)) if 3 <= len(t) <= 40]
|
||||
masked = CODE_TERM.sub(lambda m: " " * len(m.group(0)), text)
|
||||
for raw in masked.split("\n"):
|
||||
for m in CAPITAL_PHRASE.finditer(raw):
|
||||
words = m.group(0).split()
|
||||
if SENTENCE_START.search(raw[:m.start()]):
|
||||
words = words[1:]
|
||||
if len(words) >= 2:
|
||||
found.append(" ".join(words))
|
||||
return list(dict.fromkeys(t for t in found if t))
|
||||
|
||||
|
||||
def occurs(term: str, text: str) -> bool:
|
||||
return re.search(r"(?<!\w)" + re.escape(term) + r"(?!\w)", text, re.IGNORECASE) is not None
|
||||
|
||||
|
||||
def pick(blocks_doc: dict[str, Any], kilde: Path, focus: str = "docs-en-permission-modes.md") -> list[Picked]:
|
||||
names = [str(n) for n in blocks_doc["order"]]
|
||||
if focus not in names:
|
||||
raise RunError(f"the focus page {focus} is not in the order")
|
||||
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
|
||||
per_section: dict[str, int] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
if b["file"] == focus and b["section"].count(" › ") >= 1:
|
||||
per_section[b["section"]] = per_section.get(b["section"], 0) + 1
|
||||
candidates = [(count, section.split(" › ")[-1]) for section, count in per_section.items()
|
||||
if not is_meta(section.split(" › ")[-1])]
|
||||
if not candidates:
|
||||
raise RunError(f"{focus} has no heading below its title that passes test 2")
|
||||
first = sorted(candidates, key=lambda c: (-c[0], c[1]))[0][1]
|
||||
pool = list(dict.fromkeys(t for n in names for t in terms(texts[n])))
|
||||
docs = [n for n in names if n.startswith("docs-")]
|
||||
releases = [n for n in names if n.startswith("release-")]
|
||||
|
||||
as_terms = {n: set(terms(texts[n])) for n in names}
|
||||
|
||||
def count(term: str, files: list[str]) -> int:
|
||||
return sum(term in as_terms[f] for f in files) # where it stands as a term, not as a word in prose
|
||||
|
||||
taken = {first.lower()}
|
||||
second_pool = [t for t in pool if t.lower() not in taken]
|
||||
if not second_pool:
|
||||
raise RunError("no term for the second concept")
|
||||
second = sorted(second_pool, key=lambda t: (-count(t, releases), t))[0]
|
||||
taken.add(second.lower())
|
||||
third_pool = [t for t in pool if t.lower() not in taken and count(t, docs) >= 2 and count(t, releases) >= 2]
|
||||
if not third_pool:
|
||||
raise RunError("no term in two docs pages and two release notes for the third concept")
|
||||
third = sorted(third_pool, key=lambda t: (-count(t, names), t))[0]
|
||||
return [Picked(first, (first,)), Picked(second, (second,)), Picked(third, (third,))]
|
||||
|
||||
|
||||
def _write_new(path: Path, text: str) -> None:
|
||||
if path.exists():
|
||||
raise RunError(f"{path} exists; refusing to overwrite")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
def l4_cited(l4: Path) -> set[str]:
|
||||
cited: set[str] = set()
|
||||
for page in sorted(l4.glob("*.md")):
|
||||
fm = read_document(page, page.name).frontmatter or {}
|
||||
srcs = fm.get("sources")
|
||||
for s in srcs if isinstance(srcs, list) else []:
|
||||
res = s.get("resource") if isinstance(s, dict) else None
|
||||
if isinstance(res, str) and res:
|
||||
cited.add(Path(res).name)
|
||||
return cited
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
o = sub.add_parser("order")
|
||||
o.add_argument("--manifest", required=True)
|
||||
o.add_argument("--l4", required=True)
|
||||
o.add_argument("--out", required=True)
|
||||
b = sub.add_parser("blocks")
|
||||
b.add_argument("--kilde", required=True)
|
||||
b.add_argument("--order", required=True)
|
||||
b.add_argument("--out", required=True)
|
||||
t = sub.add_parser("total")
|
||||
t.add_argument("--kilde", required=True)
|
||||
k = sub.add_parser("pick")
|
||||
k.add_argument("--blocks", required=True)
|
||||
k.add_argument("--kilde", required=True)
|
||||
k.add_argument("--focus", default="docs-en-permission-modes.md")
|
||||
k.add_argument("--out", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "order":
|
||||
manifest = json.loads(Path(args.manifest).read_text(encoding="utf-8"))
|
||||
names = manifest.get("cut", {}).get("names")
|
||||
if not isinstance(names, list) or not names:
|
||||
raise RunError(f"{args.manifest} records no cut names")
|
||||
cited = l4_cited(Path(args.l4))
|
||||
if set(map(str, names)) != cited:
|
||||
raise RunError(f"the manifest's {len(names)} names differ from the {len(cited)} files the L4 cut cites: "
|
||||
f"only in manifest {sorted(set(names) - cited)}, only in L4 {sorted(cited - set(names))}")
|
||||
fixed = order([str(n) for n in names])
|
||||
_write_new(Path(args.out), "\n".join(fixed) + "\n")
|
||||
print(f"order: {len(fixed)} sources -> {args.out}")
|
||||
elif args.command == "blocks":
|
||||
names = Path(args.order).read_text(encoding="utf-8").split()
|
||||
doc = blocks_document(Path(args.kilde), names)
|
||||
_write_new(Path(args.out), json.dumps(doc, ensure_ascii=False, indent=1) + "\n")
|
||||
print(f"blocks: {len(doc['blocks'])} in {len(names)} sources -> {args.out}")
|
||||
elif args.command == "total":
|
||||
print(f"blocks in every file of {args.kilde}: {blocks_total(Path(args.kilde))}")
|
||||
else:
|
||||
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
picked = pick(doc, Path(args.kilde), args.focus)
|
||||
_write_new(Path(args.out), "".join(f"{p.name}\t{';'.join(p.aliases)}\n" for p in picked))
|
||||
print(f"pick: {len(picked)} concepts -> {args.out}")
|
||||
except (RunError, OSError, json.JSONDecodeError) as exc:
|
||||
print(f"split cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
355
evals/tests/test_direction_probe.py
Normal file
355
evals/tests/test_direction_probe.py
Normal file
|
|
@ -0,0 +1,355 @@
|
|||
"""Tests for evals/direction_probe.py: the direction probe R (PLAN.md § Leveranser, row R).
|
||||
|
||||
Three arms answer the same questions from the same sources: file search alone, the okf-wiki plus file search, and a
|
||||
wiki built with Astro-Han's karpathy-llm-wiki skill. The module sets up each arm's basis and task files, collects the
|
||||
answers into the gates' format, harvests tokens and isolation breaches from the subagent transcripts, and prints the
|
||||
table. It never calls a model. The fixtures are synthetic text about gardening (CLAUDE.md rule 3).
|
||||
Run: python3 -m unittest discover -s evals/tests
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "evals"))
|
||||
|
||||
import direction_probe as dp # noqa: E402
|
||||
|
||||
TOMATO = "# Tomatoes\nTomatoes need full sun.\nWater them every morning.\n"
|
||||
BASIL = "# Basil\nBasil likes warm soil.\n"
|
||||
MINT = "# Mint\nMint spreads through runners.\n"
|
||||
NOT_IN_CUT = "# Cactus\nCacti need little water.\n"
|
||||
|
||||
MD_SET: dict[str, Any] = {
|
||||
"schema": "okf-wiki/multidoc-set/1",
|
||||
"questions": [
|
||||
{"id": "d01", "question": "What do tomatoes and basil need?",
|
||||
"answer_keys": [{"file": "tomato.md", "text": "SECRETKEYSUN"}],
|
||||
"evidence": [{"file": "tomato.md", "span": "SECRETSPANTOMATO"}]},
|
||||
{"id": "d02", "question": "A question outside the probe.",
|
||||
"answer_keys": [{"file": "mint.md", "text": "OTHERKEY"}],
|
||||
"evidence": [{"file": "mint.md", "span": "OTHERSPAN"}]},
|
||||
],
|
||||
}
|
||||
|
||||
WD_SET: dict[str, Any] = {
|
||||
"schema": "okf-wiki/wholedoc-set/1",
|
||||
"questions": [
|
||||
{"id": "d01", "type": "whole", "file": "tomato.md", "question": "What does the tomato page say overall?",
|
||||
"answer_keys": [{"text": "SECRETWHOLEKEY"}], "evidence": [{"span": "SECRETWHOLESPAN"}]},
|
||||
{"id": "d02", "type": "count", "file": "basil.md", "question": "How many basil lines are there?",
|
||||
"count": 424242, "item_pattern": "SECRETPATTERN", "evidence": [{"span": "SECRETCOUNTSPAN"}]},
|
||||
],
|
||||
}
|
||||
|
||||
SECRETS = ("SECRETKEYSUN", "SECRETSPANTOMATO", "SECRETWHOLEKEY", "SECRETWHOLESPAN", "424242", "SECRETPATTERN",
|
||||
"SECRETCOUNTSPAN", "OTHERKEY", "A question outside the probe")
|
||||
|
||||
|
||||
class Fixture:
|
||||
def __init__(self, root: Path) -> None:
|
||||
self.root = root
|
||||
self.sources = root / "kilde"
|
||||
self.sources.mkdir()
|
||||
for name, text in (("tomato.md", TOMATO), ("basil.md", BASIL), ("mint.md", MINT), ("cactus.md", NOT_IN_CUT)):
|
||||
(self.sources / name).write_text(text, encoding="utf-8")
|
||||
self.names = ["basil.md", "mint.md", "tomato.md"]
|
||||
self.md_set = root / "md.json"
|
||||
self.md_set.write_text(json.dumps(MD_SET), encoding="utf-8")
|
||||
self.wd_set = root / "wd.json"
|
||||
self.wd_set.write_text(json.dumps(WD_SET), encoding="utf-8")
|
||||
self.wiki = root / "wiki-src"
|
||||
self.wiki.mkdir()
|
||||
(self.wiki / "index.md").write_text("# Index\n* [Tomatoes](tomatoes.md)\n", encoding="utf-8")
|
||||
(self.wiki / "tomatoes.md").write_text("Tomatoes need full sun.\n", encoding="utf-8")
|
||||
self.run = root / "run"
|
||||
|
||||
def setup(self, arm: str, **extra: Path) -> list[Path]:
|
||||
return dp.setup_arm(run=self.run, arm=arm, sources=self.sources, names=self.names, md_set=self.md_set,
|
||||
md_ids=["d01"], wd_set=self.wd_set, **extra)
|
||||
|
||||
|
||||
class FetchAstroTest(unittest.TestCase):
|
||||
def test_raw_file_keeps_the_name_and_the_body_byte_exact_under_the_template_header(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
project = Path(tmp) / "astro"
|
||||
written = dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden",
|
||||
collected="2026-09-27")
|
||||
self.assertEqual([p.name for p in written], fx.names)
|
||||
raw = (project / "raw" / "garden" / "tomato.md").read_text(encoding="utf-8")
|
||||
self.assertTrue(raw.startswith("# tomato\n\n> Source: kilde/tomato.md\n> Collected: 2026-09-27\n"
|
||||
"> Published: Unknown\n\n"))
|
||||
self.assertTrue(raw.endswith(TOMATO))
|
||||
self.assertFalse((project / "raw" / "garden" / "cactus.md").exists())
|
||||
|
||||
def test_an_existing_raw_file_is_never_overwritten(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
project = Path(tmp) / "astro"
|
||||
dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden", collected="2026-09-27")
|
||||
with self.assertRaises(dp.ProbeError):
|
||||
dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden",
|
||||
collected="2026-09-27")
|
||||
|
||||
|
||||
class SetupTest(unittest.TestCase):
|
||||
def test_file_search_basis_holds_exactly_the_cut(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
fx.setup("filsok")
|
||||
basis = fx.run / "arms" / "filsok"
|
||||
self.assertEqual(sorted(p.name for p in basis.iterdir()), fx.names)
|
||||
self.assertEqual((basis / "tomato.md").read_text(encoding="utf-8"), TOMATO)
|
||||
|
||||
def test_one_task_per_probe_question_and_none_for_the_rest(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
tasks = fx.setup("filsok")
|
||||
self.assertEqual(sorted(p.name for p in tasks), ["md-d01.md", "wd-d01.md", "wd-d02.md"])
|
||||
self.assertTrue(all(p.parent == fx.run / "tasks" / "filsok" for p in tasks))
|
||||
|
||||
def test_a_task_carries_the_question_the_basis_and_the_answer_path_but_nothing_from_the_key(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
tasks = {p.name: p.read_text(encoding="utf-8") for p in fx.setup("filsok")}
|
||||
md = tasks["md-d01.md"]
|
||||
self.assertIn("What do tomatoes and basil need?", md)
|
||||
self.assertIn(str(fx.run / "arms" / "filsok"), md)
|
||||
self.assertIn(str(fx.run / "answers" / "filsok" / "md-d01.json"), md)
|
||||
self.assertIn('"count"', tasks["wd-d02.md"])
|
||||
self.assertNotIn('"count"', md)
|
||||
for text in tasks.values():
|
||||
for secret in SECRETS:
|
||||
self.assertNotIn(secret, text)
|
||||
|
||||
def test_the_wiki_arm_gets_the_wiki_and_the_cut_and_the_router_order(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
tasks = fx.setup("wiki", wiki=fx.wiki)
|
||||
basis = fx.run / "arms" / "wiki"
|
||||
self.assertEqual(sorted(p.name for p in (basis / "kilde").iterdir()), fx.names)
|
||||
self.assertTrue((basis / "wiki" / "index.md").is_file())
|
||||
text = tasks[0].read_text(encoding="utf-8")
|
||||
self.assertLess(text.index("index.md"), text.index("x-locator"))
|
||||
|
||||
def test_the_astro_arm_gets_raw_and_wiki_from_the_built_project(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
project = Path(tmp) / "astro"
|
||||
dp.fetch_astro(sources=fx.sources, names=fx.names, project=project, topic="garden", collected="2026-09-27")
|
||||
(project / "wiki").mkdir()
|
||||
(project / "wiki" / "index.md").write_text("# Knowledge Base Index\n", encoding="utf-8")
|
||||
tasks = fx.setup("astro", astro=project)
|
||||
basis = fx.run / "arms" / "astro"
|
||||
self.assertTrue((basis / "raw" / "garden" / "basil.md").is_file())
|
||||
self.assertTrue((basis / "wiki" / "index.md").is_file())
|
||||
self.assertIn("Raw", tasks[0].read_text(encoding="utf-8"))
|
||||
|
||||
def test_an_arm_without_its_basis_or_an_existing_arm_is_refused(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
with self.assertRaises(dp.ProbeError):
|
||||
fx.setup("wiki")
|
||||
fx.setup("filsok")
|
||||
with self.assertRaises(dp.ProbeError):
|
||||
fx.setup("filsok")
|
||||
with self.assertRaises(dp.ProbeError):
|
||||
fx.setup("grep-only")
|
||||
|
||||
def test_a_probe_id_missing_from_the_set_is_refused(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
with self.assertRaises(dp.ProbeError):
|
||||
dp.setup_arm(run=fx.run, arm="filsok", sources=fx.sources, names=fx.names, md_set=fx.md_set,
|
||||
md_ids=["d09"], wd_set=fx.wd_set)
|
||||
|
||||
|
||||
class CollectTest(unittest.TestCase):
|
||||
def test_answers_are_gathered_per_gate_with_ids_from_the_file_names(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
run = Path(tmp)
|
||||
out = run / "answers" / "wiki"
|
||||
out.mkdir(parents=True)
|
||||
(out / "md-d01.json").write_text(json.dumps({"id": "wrong", "answer": "a", "citations": []}))
|
||||
(out / "wd-d02.json").write_text(json.dumps({"answer": "7", "count": 7, "citations": []}))
|
||||
(out / "wd-d03.json").write_text("not json")
|
||||
md_path, wd_path, problems = dp.collect(run, "wiki")
|
||||
md = json.loads(md_path.read_text(encoding="utf-8"))
|
||||
wd = json.loads(wd_path.read_text(encoding="utf-8"))
|
||||
self.assertEqual(md, {"arm": "wiki", "answers": [{"id": "d01", "answer": "a", "citations": []}]})
|
||||
self.assertEqual(wd["answers"], [{"id": "d02", "answer": "7", "count": 7, "citations": []}])
|
||||
self.assertEqual(problems, ["wd-d03.json: not a JSON object"])
|
||||
|
||||
|
||||
def transcript(lines: list[dict[str, Any]]) -> str:
|
||||
return "".join(json.dumps(line) + "\n" for line in lines)
|
||||
|
||||
|
||||
def assistant(msg_id: str, tokens: tuple[int, int, int, int], *tools: tuple[str, dict[str, Any]]) -> dict[str, Any]:
|
||||
i, cc, cr, o = tokens
|
||||
content = [{"type": "tool_use", "name": name, "input": inp} for name, inp in tools]
|
||||
return {"type": "assistant", "message": {"id": msg_id, "content": content, "usage": {
|
||||
"input_tokens": i, "cache_creation_input_tokens": cc, "cache_read_input_tokens": cr, "output_tokens": o}}}
|
||||
|
||||
|
||||
class HarvestTest(unittest.TestCase):
|
||||
def write(self, folder: Path, name: str, lines: list[dict[str, Any]]) -> None:
|
||||
(folder / name).write_text(transcript(lines), encoding="utf-8")
|
||||
|
||||
def test_tokens_count_each_message_once_and_breaches_are_named(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
run, folder = Path(tmp) / "run", Path(tmp) / "subagents"
|
||||
folder.mkdir()
|
||||
task = run / "tasks" / "wiki" / "md-d01.md"
|
||||
basis = run / "arms" / "wiki"
|
||||
answer = run / "answers" / "wiki" / "md-d01.json"
|
||||
first = {"type": "user", "message": {"content": f"Read {task} and follow it."}}
|
||||
self.write(folder, "agent-a.jsonl", [
|
||||
first,
|
||||
assistant("m1", (2, 100, 1000, 5), ("Read", {"file_path": str(task)})),
|
||||
assistant("m1", (2, 100, 1000, 5)),
|
||||
assistant("m2", (1, 10, 2000, 50), ("Grep", {"pattern": "sun", "path": str(basis / "kilde")}),
|
||||
("Glob", {"pattern": "*.md", "path": str(basis / "wiki")})),
|
||||
assistant("m3", (1, 0, 3000, 80), ("Write", {"file_path": str(answer), "content": "{}"}),
|
||||
("SubagentHandback", {"message": "done"})),
|
||||
])
|
||||
self.write(folder, "agent-b.jsonl", [
|
||||
{"type": "user", "message": {"content": [{"type": "text", "text": f"Read {run}/tasks/filsok/wd-d02.md"}]}},
|
||||
assistant("n1", (1, 1, 1, 1), ("Read", {"file_path": str(run / "arms" / "wiki" / "wiki" / "x.md")}),
|
||||
("Grep", {"pattern": "sun"}), ("Bash", {"command": "ls"}),
|
||||
("Read", {"file_path": str(run / "arms" / "filsok" / ".." / "wiki" / "y.md")})),
|
||||
])
|
||||
self.write(folder, "agent-c.jsonl", [{"type": "user", "message": {"content": "Something else."}}])
|
||||
rows = dp.harvest(folder, run)
|
||||
self.assertEqual(sorted(rows), ["filsok/wd-d02", "wiki/md-d01"])
|
||||
wiki = rows["wiki/md-d01"]
|
||||
self.assertEqual(wiki["tokens"], 1107 + 2061 + 3081)
|
||||
self.assertEqual(wiki["breaches"], [])
|
||||
self.assertEqual(wiki["transcript"], "agent-a.jsonl")
|
||||
breaches = rows["filsok/wd-d02"]["breaches"]
|
||||
self.assertEqual(len(breaches), 4)
|
||||
self.assertTrue(any(b.startswith("Bash") for b in breaches))
|
||||
self.assertTrue(any(b.startswith("Grep without a path") for b in breaches))
|
||||
|
||||
def test_bash_is_read_only_search_that_starts_in_the_basis(self) -> None:
|
||||
basis = Path("/runs/r/arms/filsok")
|
||||
task, answer = Path("/runs/r/tasks/filsok/md-d01.md"), Path("/runs/r/answers/filsok/md-d01.json")
|
||||
allowed = [
|
||||
f"cd {basis} && grep -rn 'full sun' .",
|
||||
f"cd {basis} && grep -n sun tomato.md | head -5",
|
||||
f"cd {basis} && find . -name '*.md' | sort && wc -l tomato.md 2>/dev/null",
|
||||
f"cd '{basis}' && sed -n '1,40p' tomato.md",
|
||||
f"cd {basis}/kilde && ls",
|
||||
]
|
||||
refused = [
|
||||
"grep -rn sun /runs/r/arms/filsok",
|
||||
f"cd {basis} && cat ../wiki/x.md",
|
||||
f"cd {basis} && grep sun /runs/r/tasks/filsok/other.md",
|
||||
f"cd {basis} && rm tomato.md",
|
||||
f"cd {basis} && grep sun tomato.md > copy.txt",
|
||||
f"cd {basis} && find . -name '*.md' -exec cat {{}} ;",
|
||||
f"cd {basis} && sed -i 's/a/b/' tomato.md",
|
||||
f"cd {basis} && cat $(echo /etc/hosts)",
|
||||
f"cd {basis} && cd / && ls",
|
||||
f"cd {basis} && ls ~",
|
||||
"cd /runs/r/arms/wiki && ls",
|
||||
]
|
||||
for command in allowed:
|
||||
self.assertEqual(dp.breaches_of([("Bash", {"command": command})], basis, task, answer), [], command)
|
||||
for command in refused:
|
||||
self.assertEqual(len(dp.breaches_of([("Bash", {"command": command})], basis, task, answer)), 1, command)
|
||||
|
||||
def test_the_task_names_bash_as_the_search_tool_and_its_rule(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
fx = Fixture(Path(tmp))
|
||||
text = fx.setup("filsok")[0].read_text(encoding="utf-8")
|
||||
self.assertIn(f"cd {fx.run / 'arms' / 'filsok'} && ", text)
|
||||
self.assertIn("grep", text)
|
||||
|
||||
def test_two_transcripts_for_one_task_keep_the_latest_and_report_the_rerun(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
run, folder = Path(tmp) / "run", Path(tmp) / "subagents"
|
||||
folder.mkdir()
|
||||
task = run / "tasks" / "filsok" / "md-d01.md"
|
||||
first = {"type": "user", "timestamp": "2026-09-27T10:00:00Z", "message": {"content": f"Read {task}"}}
|
||||
later = {"type": "user", "timestamp": "2026-09-27T11:00:00Z", "message": {"content": f"Read {task}"}}
|
||||
self.write(folder, "agent-old.jsonl", [first, assistant("o", (1, 1, 1, 1))])
|
||||
self.write(folder, "agent-new.jsonl", [later, assistant("n", (2, 2, 2, 2))])
|
||||
rows = dp.harvest(folder, run)
|
||||
self.assertEqual(rows["filsok/md-d01"]["transcript"], "agent-new.jsonl")
|
||||
self.assertEqual(rows["filsok/md-d01"]["tokens"], 8)
|
||||
self.assertEqual(rows["filsok/md-d01"]["runs"], 2)
|
||||
|
||||
|
||||
def results(arm: str, correct: dict[str, bool], kind: str) -> dict[str, Any]:
|
||||
rows = []
|
||||
for qid, ok in correct.items():
|
||||
row: dict[str, Any] = {"id": qid, "answered": True, "answer_ok": ok, "source_ok": True, "correct": ok}
|
||||
if kind == "wd":
|
||||
row["type"] = "whole" if qid in ("d01", "d02") else "count"
|
||||
rows.append(row)
|
||||
return {"arm": arm, "set_sha256": "x", "results": rows}
|
||||
|
||||
|
||||
class ScoreTest(unittest.TestCase):
|
||||
def test_groups_are_counted_from_the_gate_results_and_pairs_compared(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
run = Path(tmp)
|
||||
res = run / "results"
|
||||
res.mkdir()
|
||||
md = {"filsok": {"d01": True, "d02": False, "d99": True}, "wiki": {"d01": True, "d02": True, "d99": False}}
|
||||
wd = {"filsok": {"d01": True, "d02": False, "d03": True}, "wiki": {"d01": False, "d02": True, "d03": True}}
|
||||
for arm in ("filsok", "wiki"):
|
||||
(res / f"{arm}.md.json").write_text(json.dumps(results(arm, md[arm], "md")))
|
||||
(res / f"{arm}.wd.json").write_text(json.dumps(results(arm, wd[arm], "wd")))
|
||||
(run / "tokens.json").write_text(json.dumps({
|
||||
"filsok/md-d01": {"tokens": 100, "breaches": [], "transcript": "a", "runs": 1},
|
||||
"wiki/md-d01": {"tokens": 300, "breaches": [], "transcript": "b", "runs": 1}}))
|
||||
table = dp.score(run, ["filsok", "wiki"], md_ids=["d01", "d02"])
|
||||
self.assertEqual(table["arms"]["filsok"]["groups"]["flerdokument"], [1, 2])
|
||||
self.assertEqual(table["arms"]["wiki"]["groups"]["flerdokument"], [2, 2])
|
||||
self.assertEqual(table["arms"]["wiki"]["groups"]["helt dokument"], [1, 2])
|
||||
self.assertEqual(table["arms"]["filsok"]["groups"]["telling"], [1, 1])
|
||||
self.assertEqual(table["arms"]["wiki"]["total"], [4, 5])
|
||||
self.assertEqual(table["arms"]["wiki"]["tokens"], {"sum": 300, "questions": 1})
|
||||
pair = table["pairs"]["wiki vs filsok"]["flerdokument"]
|
||||
self.assertEqual(pair, {"both": 1, "only wiki": 1, "only filsok": 0, "neither": 0})
|
||||
self.assertEqual(table["pairs"]["wiki vs filsok"]["helt dokument"]["only filsok"], 1)
|
||||
self.assertIn("| wiki |", dp.render_table(table))
|
||||
|
||||
def test_exact_mcnemar_is_the_two_sided_binomial_over_the_discordant_pairs(self) -> None:
|
||||
self.assertEqual(dp.mcnemar_exact(0, 0), 1.0)
|
||||
self.assertEqual(dp.mcnemar_exact(2, 0), 0.5)
|
||||
self.assertAlmostEqual(dp.mcnemar_exact(6, 0), 0.03125)
|
||||
self.assertAlmostEqual(dp.mcnemar_exact(1, 5), 0.21875)
|
||||
self.assertEqual(dp.mcnemar_exact(3, 3), 1.0)
|
||||
|
||||
def test_every_pair_carries_its_mcnemar_p_over_all_groups(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
run = Path(tmp)
|
||||
res = run / "results"
|
||||
res.mkdir()
|
||||
md = {"filsok": {"d01": False, "d02": False}, "wiki": {"d01": True, "d02": True}}
|
||||
for arm in ("filsok", "wiki"):
|
||||
(res / f"{arm}.md.json").write_text(json.dumps(results(arm, md[arm], "md")))
|
||||
(res / f"{arm}.wd.json").write_text(json.dumps(results(arm, {}, "wd")))
|
||||
table = dp.score(run, ["filsok", "wiki"], md_ids=["d01", "d02"])
|
||||
self.assertEqual(table["mcnemar"]["wiki vs filsok"], {"only wiki": 2, "only filsok": 0, "p": 0.5})
|
||||
|
||||
def test_a_missing_result_file_is_a_failure_not_a_zero(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
run = Path(tmp)
|
||||
(run / "results").mkdir()
|
||||
with self.assertRaises(dp.ProbeError):
|
||||
dp.score(run, ["filsok"], md_ids=["d01"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
609
evals/tests/test_p1_build.py
Normal file
609
evals/tests/test_p1_build.py
Normal file
|
|
@ -0,0 +1,609 @@
|
|||
"""Tests for the P1 method trial's build side (TRIAL, branch p1-metodeprove, not v1 code).
|
||||
|
||||
Covers the two trial agent types, the splitter (evals/p1/split.py), the D map and register (dmap.py), K packing
|
||||
(kpack.py) and assembly with K+ (assemble.py). None of these call a model. The fixtures are synthetic text about
|
||||
gardening and bread (CLAUDE.md rule 3: no text from the Claude Code documentation is committed).
|
||||
Run: python3 -m unittest discover -s evals/tests -p 'test_p1_build.py'
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / "evals"))
|
||||
|
||||
from okfwiki import audit, claims, view # noqa: E402
|
||||
from p1 import assemble, dmap, kpack, split # noqa: E402
|
||||
|
||||
READER_SHA256 = "13a93bc5a9e87a7282752316ac670664cbdca321b0dac9cdaf92d9e74c383445"
|
||||
BANNED = ("anthropic", "openai", "urllib", "http.client", "requests", "subprocess", "socket")
|
||||
|
||||
DOC_TOMATO = """# Growing tomatoes
|
||||
Source: https://example.org/garden/tomatoes
|
||||
|
||||
How to grow tomatoes in a small garden.
|
||||
|
||||
## Soil
|
||||
|
||||
Tomatoes need deep, loose soil with `compost` mixed in.
|
||||
Water the soil every morning.
|
||||
|
||||
* **Sun**: full sun for eight hours.
|
||||
* **Space**: forty centimetres between plants,
|
||||
more in a wet summer.
|
||||
|
||||
```bash
|
||||
# not a heading
|
||||
mix compost
|
||||
```
|
||||
|
||||
| Month | Task |
|
||||
|---|---|
|
||||
| May | plant |
|
||||
|
||||
## Overview
|
||||
|
||||
The Kitchen Garden keeps tomatoes and basil together.
|
||||
|
||||
### Pruning
|
||||
|
||||
Remove side shoots once a week.
|
||||
"""
|
||||
|
||||
DOC_BASIL = """# Growing basil
|
||||
Source: https://example.org/garden/basil
|
||||
|
||||
## Soil
|
||||
|
||||
Basil likes warm soil and the Kitchen Garden.
|
||||
Plant Sweet Genovese early.
|
||||
|
||||
## Watering
|
||||
|
||||
Water basil with `compost` tea when the leaves droop.
|
||||
"""
|
||||
|
||||
REL_1 = """## 1.2.3
|
||||
|
||||
- Added the `compost` command to the Kitchen Garden planner.
|
||||
- Fixed watering reminders.
|
||||
"""
|
||||
|
||||
REL_2 = """## 1.10.0
|
||||
|
||||
- The Kitchen Garden planner now shows `compost` levels.
|
||||
"""
|
||||
|
||||
REL_3 = """## 1.9.2
|
||||
|
||||
- Basil seedlings get their own tray.
|
||||
"""
|
||||
|
||||
|
||||
def runs8(text: str) -> set[tuple[str, ...]]:
|
||||
words = re.findall(r"[a-z0-9]+", text.lower())
|
||||
return {tuple(words[i:i + 8]) for i in range(len(words) - 7)}
|
||||
|
||||
|
||||
def before_link_rule(found: list[audit.Finding]) -> list[audit.Finding]:
|
||||
"""The P1 trial predates criterion 6 as structure (operator 04.10.2026), so its link-rule findings are left out."""
|
||||
return [f for f in found if f.check != "link-rule"]
|
||||
|
||||
|
||||
class Fixture(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.tmp = Path(self._tmp.name)
|
||||
self.kilde = self.tmp / "wiki" / "kilde"
|
||||
self.kilde.mkdir(parents=True)
|
||||
files = {
|
||||
"docs-en-tomatoes.md": DOC_TOMATO,
|
||||
"docs-en-basil.md": DOC_BASIL,
|
||||
"release-v1.2.3.md": REL_1,
|
||||
"release-v1.10.0.md": REL_2,
|
||||
"release-v1.9.2.md": REL_3,
|
||||
}
|
||||
for name, text in files.items():
|
||||
(self.kilde / name).write_text(text, encoding="utf-8")
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self._tmp.cleanup()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 1: agents
|
||||
|
||||
|
||||
class AgentFilesTest(unittest.TestCase):
|
||||
def frontmatter(self, text: str) -> dict[str, str]:
|
||||
self.assertTrue(text.startswith("---\n"))
|
||||
head = text[4:text.index("\n---\n", 4)]
|
||||
return dict(line.split(": ", 1) for line in head.splitlines())
|
||||
|
||||
def test_two_trial_agents_have_narrow_tools_and_a_trial_marker(self) -> None:
|
||||
for name in ("p1-compiler", "p1-consolidator"):
|
||||
text = (ROOT / ".claude" / "agents" / f"{name}.md").read_text(encoding="utf-8")
|
||||
fm = self.frontmatter(text)
|
||||
self.assertEqual(fm["name"], name)
|
||||
self.assertEqual(fm["tools"], "Read, Write")
|
||||
self.assertEqual(fm["model"], "opus")
|
||||
body = text[text.index("\n---\n", 4) + 5:].strip()
|
||||
self.assertTrue(body.startswith("TRIAL"), name)
|
||||
fixture = DOC_TOMATO + DOC_BASIL + REL_1 + REL_2 + REL_3
|
||||
self.assertEqual(runs8(body) & runs8(fixture), set(), name)
|
||||
|
||||
def test_the_reader_is_unchanged(self) -> None:
|
||||
shipped = (ROOT / "agents" / "okf-wiki-reader.md").read_bytes()
|
||||
self.assertEqual(hashlib.sha256(shipped).hexdigest(), READER_SHA256)
|
||||
self.assertEqual(shipped, (ROOT / ".claude" / "agents" / "okf-wiki-reader.md").read_bytes())
|
||||
|
||||
def test_compiler_names_the_rules_it_must_keep(self) -> None:
|
||||
body = (ROOT / ".claude" / "agents" / "p1-compiler.md").read_text(encoding="utf-8").lower()
|
||||
for idea in ("first turn", "never read", "release note", "four tests", "json"):
|
||||
self.assertIn(idea, body, idea)
|
||||
|
||||
def test_consolidator_keeps_every_label(self) -> None:
|
||||
body = (ROOT / ".claude" / "agents" / "p1-consolidator.md").read_text(encoding="utf-8").lower()
|
||||
for idea in ("every footnote label", "no new label", "json"):
|
||||
self.assertIn(idea, body, idea)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 1: split
|
||||
|
||||
|
||||
class OrderTest(unittest.TestCase):
|
||||
def test_docs_first_then_release_notes_by_version_tuple(self) -> None:
|
||||
names = ["release-v0.2.34.md", "docs-en-b.md", "release-v0.2.9.md", "docs-en-a.md", "release-v1.0.0.md"]
|
||||
self.assertEqual(split.order(names), ["docs-en-a.md", "docs-en-b.md", "release-v0.2.9.md",
|
||||
"release-v0.2.34.md", "release-v1.0.0.md"])
|
||||
|
||||
def test_an_unknown_name_is_refused(self) -> None:
|
||||
with self.assertRaises(split.RunError):
|
||||
split.order(["notes.md"])
|
||||
|
||||
|
||||
class BlocksTest(Fixture):
|
||||
def test_fence_aware_blocks_with_spans(self) -> None:
|
||||
got = split.blocks(self.kilde / "docs-en-tomatoes.md")
|
||||
kinds = [(b.kind, b.start, b.end) for b in got]
|
||||
# line 2 "Source:" and line 4 are paragraphs; headings are not blocks
|
||||
self.assertIn(("paragraph", 2, 2), kinds)
|
||||
self.assertIn(("paragraph", 8, 9), kinds)
|
||||
self.assertIn(("item", 11, 11), kinds)
|
||||
self.assertIn(("item", 12, 13), kinds) # a continuation line belongs to its item
|
||||
self.assertIn(("code", 15, 18), kinds) # the "#" inside the fence is not a heading
|
||||
self.assertIn(("table", 20, 22), kinds)
|
||||
heads = {b.section for b in got}
|
||||
self.assertIn("Growing tomatoes › Soil", heads)
|
||||
self.assertIn("Growing tomatoes › Overview › Pruning", heads)
|
||||
self.assertNotIn("not a heading", " ".join(heads))
|
||||
|
||||
def test_every_non_blank_non_heading_line_is_in_exactly_one_block(self) -> None:
|
||||
path = self.kilde / "docs-en-tomatoes.md"
|
||||
lines = path.read_text(encoding="utf-8").split("\n")
|
||||
got = split.blocks(path)
|
||||
covered: dict[int, int] = {}
|
||||
for b in got:
|
||||
for n in range(b.start, b.end + 1):
|
||||
covered[n] = covered.get(n, 0) + 1
|
||||
for n, raw in enumerate(lines, 1):
|
||||
heading = raw.startswith("#") and not (15 <= n <= 18)
|
||||
if raw.strip() and not heading:
|
||||
self.assertEqual(covered.get(n), 1, (n, raw))
|
||||
if heading:
|
||||
self.assertNotIn(n, covered, (n, raw))
|
||||
|
||||
def test_release_note_blocks_carry_the_version(self) -> None:
|
||||
got = split.blocks(self.kilde / "release-v1.10.0.md")
|
||||
self.assertEqual({b.version for b in got}, {"1.10.0"})
|
||||
self.assertEqual(split.blocks(self.kilde / "docs-en-basil.md")[0].version, None)
|
||||
|
||||
|
||||
class DocsHeadingsTest(Fixture):
|
||||
def test_titles_and_second_and_third_level_headings_of_docs_pages_only(self) -> None:
|
||||
got = split.docs_headings(self.kilde)
|
||||
texts = [(h.file, h.level, h.text) for h in got]
|
||||
self.assertIn(("docs-en-tomatoes.md", 1, "Growing tomatoes"), texts)
|
||||
self.assertIn(("docs-en-tomatoes.md", 3, "Pruning"), texts)
|
||||
self.assertFalse(any(f.startswith("release-") for f, _, _ in texts))
|
||||
self.assertFalse(any(t == "not a heading" for _, _, t in texts))
|
||||
|
||||
|
||||
class PickTest(Fixture):
|
||||
def blocks_doc(self, order: list[str]) -> dict[str, object]:
|
||||
return split.blocks_document(self.kilde, order)
|
||||
|
||||
def test_three_concepts_by_the_stated_rules(self) -> None:
|
||||
order = split.order([p.name for p in self.kilde.iterdir()])
|
||||
picked = split.pick(self.blocks_doc(order), self.kilde, focus="docs-en-tomatoes.md")
|
||||
names = [p.name for p in picked]
|
||||
self.assertEqual(len(names), 3)
|
||||
self.assertEqual(len(set(names)), 3)
|
||||
self.assertEqual(names[0], "Soil") # the focus page's heading with the most blocks
|
||||
# "Kitchen Garden" and "compost" are both in 2 release notes; the tie is broken by name
|
||||
self.assertEqual(names[1], "Kitchen Garden")
|
||||
self.assertEqual(names[2], "compost") # in 2 docs pages and 2 release notes, distinct from 1 and 2
|
||||
|
||||
def test_a_document_counts_for_a_term_only_where_it_stands_as_a_term(self) -> None:
|
||||
# Found in session 2: the word "and", backticked once, was counted in every document where it is prose.
|
||||
(self.kilde / "docs-en-zz.md").write_text("# Joining\n\n## Words\n\nUse `and` to join two beds.\n",
|
||||
encoding="utf-8")
|
||||
for v in ("2.0.0", "2.0.1", "2.0.2"):
|
||||
(self.kilde / f"release-v{v}.md").write_text(f"## {v}\n\n- Tomatoes and basil and mint.\n",
|
||||
encoding="utf-8")
|
||||
order = split.order([p.name for p in self.kilde.iterdir()])
|
||||
names = [p.name for p in split.pick(self.blocks_doc(order), self.kilde, focus="docs-en-tomatoes.md")]
|
||||
self.assertNotIn("and", names)
|
||||
self.assertEqual(names[1:], ["Kitchen Garden", "compost"])
|
||||
|
||||
def test_a_backticked_term_counts_its_length_after_trimming(self) -> None:
|
||||
# Found in session 2: "` or `" is four characters with its spaces, and became the two-letter term "or".
|
||||
got = split.terms("Use `compost` ` or ` ` ab ` ` mulch ` here.")
|
||||
self.assertEqual(got, ["compost", "mulch"])
|
||||
|
||||
def test_the_focus_page_must_be_in_the_order(self) -> None:
|
||||
with self.assertRaises(split.RunError):
|
||||
split.pick(self.blocks_doc(["docs-en-basil.md"]), self.kilde, focus="docs-en-tomatoes.md")
|
||||
|
||||
|
||||
class CliTest(Fixture):
|
||||
def test_order_refuses_names_that_differ_from_the_l4_cut(self) -> None:
|
||||
manifest = self.tmp / "m.json"
|
||||
manifest.write_text(json.dumps({"cut": {"names": ["docs-en-basil.md", "release-v1.2.3.md"]}}), encoding="utf-8")
|
||||
l4 = self.tmp / "l4"
|
||||
l4.mkdir()
|
||||
(l4 / "a.md").write_text("---\ntype: Concept\nsources:\n - id: a\n resource: /kilde/docs-en-basil.md\n"
|
||||
"---\n\nx\n", encoding="utf-8")
|
||||
out = self.tmp / "rekkefolge.txt"
|
||||
self.assertEqual(split.main(["order", "--manifest", str(manifest), "--l4", str(l4), "--out", str(out)]), 2)
|
||||
self.assertFalse(out.exists())
|
||||
(l4 / "b.md").write_text("---\ntype: Concept\nsources:\n - id: b\n resource: /kilde/release-v1.2.3.md\n"
|
||||
"---\n\nx\n", encoding="utf-8")
|
||||
self.assertEqual(split.main(["order", "--manifest", str(manifest), "--l4", str(l4), "--out", str(out)]), 0)
|
||||
self.assertEqual(out.read_text(encoding="utf-8").split(), ["docs-en-basil.md", "release-v1.2.3.md"])
|
||||
|
||||
def test_blocks_and_pick_refuse_to_overwrite(self) -> None:
|
||||
order = self.tmp / "order.txt"
|
||||
order.write_text("\n".join(split.order([p.name for p in self.kilde.iterdir()])) + "\n", encoding="utf-8")
|
||||
out = self.tmp / "blocks.json"
|
||||
self.assertEqual(split.main(["blocks", "--kilde", str(self.kilde), "--order", str(order), "--out", str(out)]), 0)
|
||||
data = json.loads(out.read_text(encoding="utf-8"))
|
||||
self.assertEqual(data["schema"], "okf-wiki/p1-blocks/1")
|
||||
self.assertEqual(split.main(["blocks", "--kilde", str(self.kilde), "--order", str(order), "--out", str(out)]), 2)
|
||||
picked = self.tmp / "begreper.txt"
|
||||
args = ["pick", "--blocks", str(out), "--kilde", str(self.kilde), "--focus", "docs-en-tomatoes.md",
|
||||
"--out", str(picked)]
|
||||
self.assertEqual(split.main(args), 0)
|
||||
self.assertEqual(len(picked.read_text(encoding="utf-8").strip().splitlines()), 3)
|
||||
self.assertEqual(split.main(args), 2)
|
||||
|
||||
def test_total_counts_blocks_in_every_file(self) -> None:
|
||||
expected = sum(len(split.blocks(p)) for p in self.kilde.iterdir())
|
||||
self.assertEqual(split.blocks_total(self.kilde), expected)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 2: D
|
||||
|
||||
|
||||
class DMapTest(Fixture):
|
||||
def build(self) -> Path:
|
||||
order = split.order([p.name for p in self.kilde.iterdir()])
|
||||
out = self.tmp / "D"
|
||||
dmap.build(split.blocks_document(self.kilde, order), self.kilde.parent, out, at="2026-10-02T12:00:00Z")
|
||||
return out
|
||||
|
||||
def test_d_passes_the_auditor_with_no_claims(self) -> None:
|
||||
out = self.build()
|
||||
self.assertEqual(before_link_rule(audit.audit(out, self.kilde.parent)), [])
|
||||
found = claims.extract(out, self.kilde.parent)
|
||||
self.assertEqual(found.claims, [])
|
||||
self.assertTrue(found.exceptions)
|
||||
|
||||
def test_register_links_point_to_map_sections(self) -> None:
|
||||
out = self.build()
|
||||
page = (out / "r-kitchen-garden.md").read_text(encoding="utf-8")
|
||||
self.assertIn("(k-docs-en-basil.md#soil)", page)
|
||||
self.assertIn("(k-release-v1-2-3.md#1-2-3)", page)
|
||||
|
||||
def test_register_keeps_docs_headings_and_terms_in_two_documents_only(self) -> None:
|
||||
out = self.build()
|
||||
names = {p.name for p in out.glob("r-*.md")}
|
||||
self.assertIn("r-pruning.md", names) # a docs heading (i), in one document
|
||||
self.assertIn("r-compost.md", names) # a backticked term in several documents
|
||||
self.assertNotIn("r-sweet-genovese.md", names) # a capitalised phrase in one document only
|
||||
self.assertNotIn("r-overview.md", names) # a stop-listed heading fails test 2
|
||||
|
||||
def test_quotes_stand_on_their_lines_and_stay_strings(self) -> None:
|
||||
out = self.build()
|
||||
doc = audit.read_document(out / "r-compost.md", "r-compost.md")
|
||||
assert doc.frontmatter is not None
|
||||
for s in doc.frontmatter["sources"]:
|
||||
self.assertIsInstance(s["x-quote"], str)
|
||||
line = (self.kilde / Path(s["resource"]).name).read_text(encoding="utf-8").split("\n")[s["x-locator"]["line"] - 1]
|
||||
self.assertIn(s["x-quote"], line)
|
||||
self.assertIsInstance(doc.frontmatter["generated"]["at"], str)
|
||||
|
||||
def test_a_quote_like_no_survives_a_yaml_round_trip(self) -> None:
|
||||
text = dmap.frontmatter({"type": "Concept", "sources": [{"id": "a", "x-quote": "no"}, {"id": "b", "x-quote": "1.10"}]})
|
||||
path = self.tmp / "x.md"
|
||||
path.write_text(text + "\n# x\n", encoding="utf-8")
|
||||
fm = audit.read_document(path, "x.md").frontmatter
|
||||
assert fm is not None
|
||||
self.assertEqual([s["x-quote"] for s in fm["sources"]], ["no", "1.10"])
|
||||
|
||||
def test_the_log_records_the_thresholds_and_build_refuses_an_existing_target(self) -> None:
|
||||
out = self.build()
|
||||
log = (out / "log.md").read_text(encoding="utf-8")
|
||||
self.assertIn("2 of the", log)
|
||||
with self.assertRaises(dmap.RunError):
|
||||
dmap.build({"order": [], "blocks": []}, self.kilde.parent, out, at="2026-10-02T12:00:00Z")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 3: K packing
|
||||
|
||||
|
||||
def entry(eid: str, title: str, aliases: tuple[str, ...] = ()) -> dict[str, object]:
|
||||
return {"id": eid, "title": title, "aliases": [title, *aliases], "kind": "skeleton", "merge_of": None,
|
||||
"origin": None}
|
||||
|
||||
|
||||
class KSkeletonTest(Fixture):
|
||||
def test_the_skeleton_is_an_index_of_docs_headings_that_pass_test_2(self) -> None:
|
||||
index = kpack.skeleton(self.kilde)
|
||||
titles = {e["title"] for e in index["entries"]}
|
||||
self.assertIn("Growing tomatoes", titles)
|
||||
self.assertIn("Pruning", titles)
|
||||
self.assertIn("Soil", titles)
|
||||
self.assertNotIn("Overview", titles)
|
||||
soil = [e for e in index["entries"] if e["title"] == "Soil"]
|
||||
self.assertEqual(len(soil), 1) # the same heading in two pages is one entry
|
||||
self.assertEqual(soil[0]["origin"]["file"], "docs-en-basil.md") # pages in name order, first one wins
|
||||
self.assertFalse(any(p.suffix == ".md" for p in self.tmp.rglob("*") if p.parent.name != "kilde"))
|
||||
|
||||
def test_norm(self) -> None:
|
||||
self.assertEqual(kpack.norm("The `Compost` bins"), "compost bin")
|
||||
self.assertEqual(kpack.norm("settings.json"), "settings.json")
|
||||
self.assertEqual(kpack.norm(" Kitchen Garden! "), "kitchen garden")
|
||||
self.assertEqual(kpack.norm("CLI flags"), "cli flag")
|
||||
|
||||
|
||||
class KIdentityTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.index = {"entries": [entry("kitchen-garden-planner", "Kitchen Garden planner"),
|
||||
entry("compost", "compost", ("compost heap",))]}
|
||||
|
||||
def frag(self, concept: str, status: str = "new", file: str = "docs-en-tomatoes.md", eid: str | None = None
|
||||
) -> dict[str, object]:
|
||||
f: dict[str, object] = {"concept": concept, "status": status, "anchor": {"file": file, "start": 8, "end": 9}}
|
||||
if eid:
|
||||
f["id"] = eid
|
||||
return f
|
||||
|
||||
def test_outcomes(self) -> None:
|
||||
r = kpack.resolve
|
||||
self.assertEqual(r(self.frag("x", "existing", eid="compost"), self.index), ("existing", "compost"))
|
||||
self.assertEqual(r(self.frag("Kitchen Garden planners"), self.index), ("existing", "kitchen-garden-planner"))
|
||||
self.assertEqual(r(self.frag("Compost heap"), self.index), ("existing", "compost"))
|
||||
self.assertEqual(r(self.frag("Kitchen Garden planner tool"), self.index),
|
||||
("merge-candidate", "kitchen-garden-planner"))
|
||||
self.assertEqual(r(self.frag("Garden hose"), self.index), ("new", None))
|
||||
self.assertEqual(r(self.frag("Garden hose", file="release-v1.2.3.md"), self.index), ("rejected", "release-note-new"))
|
||||
self.assertEqual(r(self.frag("Kitchen Garden planner tool", file="release-v1.2.3.md"), self.index),
|
||||
("rejected", "release-note-new"))
|
||||
self.assertEqual(r(self.frag("Troubleshooting"), self.index), ("rejected", "meta"))
|
||||
|
||||
def test_relevant_lines_by_phrase(self) -> None:
|
||||
got = kpack.relevant_lines(self.index, "We keep the compost heaps behind the shed.")
|
||||
self.assertEqual([e["id"] for e in got], ["compost"])
|
||||
self.assertEqual(kpack.relevant_lines(self.index, "Nothing about gardens here."), [])
|
||||
|
||||
|
||||
class KPackTest(Fixture):
|
||||
def blocks_doc(self) -> dict[str, object]:
|
||||
return split.blocks_document(self.kilde, split.order([p.name for p in self.kilde.iterdir()]))
|
||||
|
||||
def test_packs_of_five(self) -> None:
|
||||
self.assertEqual([len(p) for p in kpack.packs([f"n{i}" for i in range(19)], 5)], [5, 5, 5, 4])
|
||||
|
||||
def test_a_large_pack_is_split_into_parts_under_the_read_cap(self) -> None:
|
||||
big = "".join(f"Line {i} says the tomatoes are fine today and tomorrow, and the basil too.\n\n"
|
||||
for i in range(1110))
|
||||
(self.kilde / "docs-en-big.md").write_text("# Big\n\n" + big, encoding="utf-8")
|
||||
doc = split.blocks_document(self.kilde, ["docs-en-big.md"])
|
||||
parts = kpack.render_input(1, ["docs-en-big.md"], doc, [], self.kilde)
|
||||
self.assertGreater(sum(len(p) for p in parts), 130_000)
|
||||
self.assertEqual(len(parts), 3)
|
||||
self.assertTrue(all(len(p) <= kpack.PART_CHARS for p in parts))
|
||||
self.assertIn("docs-en-big.md › Big › 3–3", parts[0])
|
||||
|
||||
def test_validate(self) -> None:
|
||||
doc = self.blocks_doc()
|
||||
pack = ["docs-en-tomatoes.md", "release-v1.2.3.md"]
|
||||
good = {"concept": "Soil", "status": "existing", "id": "soil", "text": "Tomatoes need deep, loose soil.",
|
||||
"anchor": {"file": "docs-en-tomatoes.md", "start": 8, "end": 9}}
|
||||
cases = {
|
||||
"anchor-crosses-blocks": {**good, "anchor": {"file": "docs-en-tomatoes.md", "start": 8, "end": 11}},
|
||||
"quote-missing": {**good, "text": "Plants stand 40 cm apart."},
|
||||
"quote-not-in-span": {**good, "text": "Keep 40 cm.", "quote": "forty centimetres"},
|
||||
"sentence-count": {**good, "text": "One. Two. Three. Four."},
|
||||
"tests-missing": {**good, "status": "new", "concept": "Loose soil"},
|
||||
"anchor-outside-pack": {**good, "anchor": {"file": "docs-en-basil.md", "start": 7, "end": 7}},
|
||||
"malformed": {"concept": "Soil"},
|
||||
}
|
||||
no_text = {**good, "text": ""}
|
||||
accepted, rejected = kpack.validate([good, no_text, *cases.values()], pack, doc, self.kilde)
|
||||
self.assertEqual(len(accepted), 1)
|
||||
self.assertEqual(rejected["sentence-count"], 2) # the empty text and the four sentences
|
||||
for reason in cases:
|
||||
self.assertGreaterEqual(rejected[reason], 1, reason)
|
||||
ok = {**good, "text": "Plants stand forty centimetres apart.", "anchor": {"file": "docs-en-tomatoes.md",
|
||||
"start": 12, "end": 13}, "quote": "forty centimetres"}
|
||||
accepted, rejected = kpack.validate([ok], pack, doc, self.kilde)
|
||||
self.assertEqual((len(accepted), sum(rejected.values())), (1, 0))
|
||||
|
||||
|
||||
class KCliTest(Fixture):
|
||||
def test_skeleton_pack_and_accept(self) -> None:
|
||||
work = self.tmp / "work"
|
||||
blocks = self.tmp / "blocks.json"
|
||||
order = split.order([p.name for p in self.kilde.iterdir()])
|
||||
blocks.write_text(json.dumps(split.blocks_document(self.kilde, order)), encoding="utf-8")
|
||||
index = work / "K-work" / "index.json"
|
||||
self.assertEqual(kpack.main(["skeleton", "--kilde", str(self.kilde), "--out", str(index)]), 0)
|
||||
self.assertEqual(kpack.main(["skeleton", "--kilde", str(self.kilde), "--out", str(index)]), 2)
|
||||
root = str(self.kilde.parent)
|
||||
self.assertEqual(kpack.main(["pack", "1", "--blocks", str(blocks), "--index", str(index), "--kilde-root", root,
|
||||
"--work", str(work)]), 0)
|
||||
self.assertTrue((work / "K-oppgaver" / "pack-1-part-1.md").is_file())
|
||||
fragments = {"fragments": [
|
||||
{"concept": "Soil", "status": "existing", "id": "soil", "text": "Tomatoes need deep, loose soil.",
|
||||
"anchor": {"file": "docs-en-tomatoes.md", "start": 8, "end": 9}},
|
||||
{"concept": "Kitchen Garden", "status": "new", "text": "The Kitchen Garden keeps tomatoes and basil.",
|
||||
"anchor": {"file": "docs-en-tomatoes.md", "start": 26, "end": 26},
|
||||
"tests": {"topic_shape": "a named planner", "citation": "See the Kitchen Garden reference"}},
|
||||
{"concept": "Kitchen Garden", "status": "existing", "text": "The planner gained a command.",
|
||||
"anchor": {"file": "release-v1.2.3.md", "start": 3, "end": 3}},
|
||||
{"concept": "Watering reminders", "status": "new", "text": "Reminders were fixed.",
|
||||
"anchor": {"file": "release-v1.2.3.md", "start": 4, "end": 4},
|
||||
"tests": {"topic_shape": "x", "citation": "y"}},
|
||||
]}
|
||||
out = work / "K-fragments" / "pack-1.json"
|
||||
out.parent.mkdir(parents=True)
|
||||
out.write_text(json.dumps(fragments), encoding="utf-8")
|
||||
self.assertEqual(kpack.main(["accept", "1", "--work", str(work), "--blocks", str(blocks), "--kilde-root", root]), 0)
|
||||
acc = json.loads((work / "K-work" / "accepted.json").read_text(encoding="utf-8"))["fragments"]
|
||||
self.assertEqual([a["outcome"] for a in acc], ["existing", "new", "existing"])
|
||||
self.assertEqual(acc[2]["concept_id"], acc[1]["concept_id"]) # the release note enriches the new concept
|
||||
self.assertEqual(acc[2]["version"], "1.2.3")
|
||||
log = json.loads((work / "K-work" / "packs.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual(log["packs"][0]["rejected"], {"release-note-new": 1})
|
||||
self.assertEqual(log["packs"][0]["proposed"], 4)
|
||||
# PM 02.10: the ids of the index lines sent, and the accepted fragments whose concept was not among them
|
||||
sent = log["packs"][0]["index_ids"]
|
||||
self.assertIn("soil", sent)
|
||||
self.assertNotIn(acc[1]["concept_id"], sent)
|
||||
self.assertEqual(log["packs"][0]["not_sent"], {"total": 2, "by_outcome": {"existing": 1, "new": 1}})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 4: assembly
|
||||
|
||||
|
||||
def accepted(cid: str, file: str, start: int, end: int, text: str, pack: int, version: str | None = None,
|
||||
quote: str | None = None, quote_line: int | None = None) -> dict[str, object]:
|
||||
return {"pack": pack, "position": 1, "file": file, "start": start, "end": end, "section": "-", "text": text,
|
||||
"quote": quote, "quote_line": quote_line, "version": version, "concept_id": cid,
|
||||
"outcome": "existing"}
|
||||
|
||||
|
||||
class AssembleTest(Fixture):
|
||||
def setUp(self) -> None:
|
||||
super().setUp()
|
||||
self.work = self.tmp / "work"
|
||||
(self.work / "K-work").mkdir(parents=True)
|
||||
index = {"entries": [
|
||||
{**entry("soil", "Soil"), "origin": {"file": "docs-en-basil.md", "line": 4}},
|
||||
{**entry("pruning", "Pruning"), "origin": {"file": "docs-en-tomatoes.md", "line": 28}},
|
||||
{**entry("kitchen-garden", "Kitchen Garden"), "kind": "new"},
|
||||
{**entry("kitchen-garden-planner", "Kitchen Garden planner"), "kind": "merge-candidate",
|
||||
"merge_of": "kitchen-garden"},
|
||||
]}
|
||||
frags = [
|
||||
accepted("soil", "docs-en-tomatoes.md", 8, 9, "Tomatoes need deep, loose soil.", 1),
|
||||
accepted("kitchen-garden", "docs-en-tomatoes.md", 26, 26,
|
||||
"The Kitchen Garden keeps tomatoes and basil together.", 1, quote="Kitchen Garden", quote_line=26),
|
||||
accepted("kitchen-garden", "release-v1.2.3.md", 3, 3, "The planner gained a compost command.", 1, "1.2.3"),
|
||||
accepted("soil", "docs-en-basil.md", 6, 7, "Basil likes warm soil near the Kitchen Garden.", 1),
|
||||
accepted("kitchen-garden", "release-v1.10.0.md", 3, 3, "The planner shows compost levels.", 2, "1.10.0"),
|
||||
accepted("kitchen-garden-planner", "release-v1.9.2.md", 3, 3, "Seedlings get their own tray.", 2, "1.9.2"),
|
||||
]
|
||||
(self.work / "K-work" / "index.json").write_text(json.dumps(index), encoding="utf-8")
|
||||
(self.work / "K-work" / "accepted.json").write_text(json.dumps({"fragments": frags}), encoding="utf-8")
|
||||
self.out = self.tmp / "K"
|
||||
assemble.assemble(self.work, self.kilde.parent, self.out, at="2026-10-02T12:00:00Z")
|
||||
|
||||
def page(self, name: str) -> audit.Document:
|
||||
return audit.read_document(self.out / name, name)
|
||||
|
||||
def test_the_bundle_passes_the_auditor(self) -> None:
|
||||
self.assertEqual(before_link_rule(audit.audit(self.out, self.kilde.parent)), [])
|
||||
found = claims.extract(self.out, self.kilde.parent)
|
||||
self.assertEqual(claims.criterion_1(found)["total"], (6, 6))
|
||||
|
||||
def test_only_enriched_concepts_get_pages(self) -> None:
|
||||
names = {p.name for p in self.out.glob("*.md")} - {"index.md", "log.md"}
|
||||
self.assertEqual(names, {"soil.md", "kitchen-garden.md", "kitchen-garden-planner.md"})
|
||||
fm = self.page("kitchen-garden-planner.md").frontmatter
|
||||
assert fm is not None
|
||||
self.assertEqual(fm["x-merge-candidate"], "kitchen-garden")
|
||||
|
||||
def test_history_in_version_order_without_version_numbers_in_the_text(self) -> None:
|
||||
doc = self.page("kitchen-garden.md")
|
||||
body = doc.body
|
||||
self.assertLess(body.index("gained a compost command"), body.index("shows compost levels"))
|
||||
prose = " ".join(b.text for b in audit.prose_blocks(doc))
|
||||
self.assertEqual(view.version_numbers(audit.FOOTNOTE_REF.sub("", prose)), [])
|
||||
fm = doc.frontmatter
|
||||
assert fm is not None
|
||||
versions = [s for s in fm["sources"] if view.is_version_heading(s)]
|
||||
self.assertEqual(sorted(s["x-quote"] for s in versions), ["1.10.0", "1.2.3"])
|
||||
self.assertEqual(fm["x-fragments"], 3)
|
||||
|
||||
def test_code_quotes_skip_structural_lines_and_are_marked(self) -> None:
|
||||
fm = self.page("soil.md").frontmatter
|
||||
assert fm is not None
|
||||
first = fm["sources"][0]
|
||||
self.assertEqual(first["x-locator"], {"line": 8})
|
||||
self.assertEqual(first["x-quote-by"], "code")
|
||||
self.assertEqual(first["x-span"], [8, 9])
|
||||
kg = self.page("kitchen-garden.md").frontmatter
|
||||
assert kg is not None
|
||||
self.assertNotIn("x-quote-by", kg["sources"][0])
|
||||
self.assertEqual(assemble.code_quote(["```bash", "<Note>", "|---|---|", "", " real line "], 10),
|
||||
("real line", 14))
|
||||
|
||||
def test_a_concept_title_in_another_page_becomes_one_link(self) -> None:
|
||||
self.assertIn("[Kitchen Garden](kitchen-garden.md)", self.page("soil.md").body)
|
||||
|
||||
|
||||
class KPlusTest(AssembleTest):
|
||||
def test_copy_tasks_and_apply(self) -> None:
|
||||
kplus = self.tmp / "K+"
|
||||
assemble.copy(self.out, kplus)
|
||||
with self.assertRaises(assemble.RunError):
|
||||
assemble.copy(self.out, kplus)
|
||||
runs = assemble.consolidation_tasks(kplus, self.work, min_fragments=3, per_run=8)
|
||||
self.assertEqual([r.pages for r in runs], [["kitchen-garden"]])
|
||||
self.assertTrue(runs[0].task.is_file())
|
||||
old = self.page("kitchen-garden.md").body
|
||||
labels = sorted(set(audit.FOOTNOTE_REF.findall(old)))
|
||||
good = "The Kitchen Garden keeps tomatoes and basil together, and its planner tracks compost " + "".join(
|
||||
f"[^{x}]" for x in labels) + "."
|
||||
dropped = "The Kitchen Garden keeps tomatoes and basil together[^" + labels[0] + "]."
|
||||
added = good.replace(".", "[^zz].", 1)
|
||||
bodies = self.tmp / "bodies.json"
|
||||
bodies.write_text(json.dumps({"kitchen-garden": good, "soil": dropped}), encoding="utf-8")
|
||||
self.assertEqual(assemble.apply(kplus, bodies), (1, 1))
|
||||
self.assertIn("planner tracks compost", (kplus / "kitchen-garden.md").read_text(encoding="utf-8"))
|
||||
self.assertEqual(before_link_rule(audit.audit(kplus, self.kilde.parent)), [])
|
||||
bodies.write_text(json.dumps({"kitchen-garden": added}), encoding="utf-8")
|
||||
self.assertEqual(assemble.apply(kplus, bodies), (0, 1))
|
||||
|
||||
|
||||
class NoModelTest(unittest.TestCase):
|
||||
def test_trial_modules_import_no_network_or_model_client(self) -> None:
|
||||
for path in sorted((ROOT / "evals" / "p1").glob("*.py")):
|
||||
source = path.read_text(encoding="utf-8")
|
||||
for banned in BANNED:
|
||||
self.assertNotIn(f"import {banned}", source, path.name)
|
||||
self.assertNotIn(f"from {banned}", source, path.name)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
366
evals/tests/test_p1_report.py
Normal file
366
evals/tests/test_p1_report.py
Normal file
|
|
@ -0,0 +1,366 @@
|
|||
"""Tests for the P1 method trial's report side (TRIAL, branch p1-metodeprove, not v1 code).
|
||||
|
||||
Covers the measures (evals/p1/measure.py), the cost harvester (cost.py), the side-by-side page (sidebyside.py) and the
|
||||
two verification scripts. None of these call a model. The fixtures are synthetic text about gardening (CLAUDE.md
|
||||
rule 3: no text from the Claude Code documentation is committed).
|
||||
Run: python3 -m unittest discover -s evals/tests -p 'test_p1_report.py'
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / "evals"))
|
||||
|
||||
from okfwiki import audit, view # noqa: E402
|
||||
from p1 import cost, measure, sidebyside, split # noqa: E402
|
||||
|
||||
BANNED = ("anthropic", "openai", "urllib", "http.client", "requests", "subprocess", "socket")
|
||||
|
||||
DOC_A = """# Alpha
|
||||
|
||||
## Beds
|
||||
|
||||
Raised beds drain well.
|
||||
They warm early in spring.
|
||||
|
||||
- Line one item.
|
||||
- Line two item.
|
||||
|
||||
## Paths
|
||||
|
||||
Gravel paths stay dry.
|
||||
"""
|
||||
|
||||
REL = """## 2.0.0
|
||||
|
||||
- Beds got labels.
|
||||
"""
|
||||
|
||||
|
||||
def src(sid: str, file: str, line: int, quote: str, span: tuple[int, int] | None = None) -> dict[str, Any]:
|
||||
s: dict[str, Any] = {"id": sid, "resource": f"/kilde/{file}", "title": file, "x-locator": {"line": line},
|
||||
"x-quote": quote}
|
||||
if span:
|
||||
s["x-span"] = list(span)
|
||||
return s
|
||||
|
||||
|
||||
def write_page(folder: Path, name: str, title: str, body: str, sources: list[dict[str, Any]],
|
||||
**extra: Any) -> None:
|
||||
fm = {"type": "Concept", "title": title, "description": title, "sources": sources, **extra,
|
||||
"generated": {"by": "test", "at": "2026-10-02T12:00:00Z"}}
|
||||
text = "---\n" + yaml.safe_dump(fm, sort_keys=False) + "---\n\n# " + title + "\n\n" + body + "\n"
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
(folder / name).write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
class Fixture(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.tmp = Path(self._tmp.name)
|
||||
self.root = self.tmp / "wiki"
|
||||
self.kilde = self.root / "kilde"
|
||||
self.kilde.mkdir(parents=True)
|
||||
(self.kilde / "docs-en-a.md").write_text(DOC_A, encoding="utf-8")
|
||||
(self.kilde / "release-v2.0.0.md").write_text(REL, encoding="utf-8")
|
||||
self.order = ["docs-en-a.md", "release-v2.0.0.md"]
|
||||
self.blocks = split.blocks_document(self.kilde, self.order)
|
||||
self.bundle = self.tmp / "B"
|
||||
write_page(self.bundle, "beds.md", "Beds",
|
||||
"Raised beds drain well.[^b1] The beds have a heading.[^b2] Items begin.[^b3] Two items.[^b4]\n\n"
|
||||
"## History\n\n- Beds got labels.[^r1][^v1]\n\n| a | b |",
|
||||
[src("b1", "docs-en-a.md", 5, "Raised beds drain well.", (5, 6)),
|
||||
src("b2", "docs-en-a.md", 3, "Beds", (3, 3)),
|
||||
src("b3", "docs-en-a.md", 8, "Line one item.", (8, 13)),
|
||||
src("b4", "docs-en-a.md", 9, "Line two item.", (8, 9)),
|
||||
src("v1", "release-v2.0.0.md", 1, "2.0.0"),
|
||||
src("r1", "release-v2.0.0.md", 3, "Beds got labels.", (3, 3)),
|
||||
src("u1", "docs-en-a.md", 13, "Gravel paths stay dry.", (13, 13))],
|
||||
**{"x-aliases": ["Beds", "Raised beds"]})
|
||||
write_page(self.bundle, "labels.md", "Labels", "Beds got labels.[^r1][^v1] See also: [Beds](beds.md)",
|
||||
[src("v1", "release-v2.0.0.md", 1, "2.0.0"), src("r1", "release-v2.0.0.md", 3, "Beds got labels.")])
|
||||
(self.bundle / "index.md").write_text('---\nokf_version: "0.2"\n---\n\n# Concepts\n\n- [Beds](beds.md)\n'
|
||||
'- [Labels](labels.md)\n', encoding="utf-8")
|
||||
(self.bundle / "log.md").write_text("# Log\n\n## 2026-10-02\n* built\n", encoding="utf-8")
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self._tmp.cleanup()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 5: measures
|
||||
|
||||
|
||||
class CoverageTest(Fixture):
|
||||
def test_cited_entries_by_locator_line_only(self) -> None:
|
||||
got = measure.coverage(self.bundle, self.root, self.blocks, self.order)
|
||||
# covered: 5-6 (b1), 8 (b3's locator; its wide span is ignored), 9 (b4), release 3 (r1); not 13 (u1 uncited)
|
||||
self.assertEqual((got.covered, got.total), (4, 5))
|
||||
|
||||
def test_a_stopped_variant_counts_only_the_sources_it_built(self) -> None:
|
||||
got = measure.coverage(self.bundle, self.root, self.blocks, self.order[:1])
|
||||
self.assertEqual((got.covered, got.total), (3, 4))
|
||||
|
||||
def test_per_pack_position(self) -> None:
|
||||
got = measure.coverage(self.bundle, self.root, self.blocks, self.order, pack_size=5)
|
||||
self.assertEqual(got.per_position, {1: (3, 4, 1), 2: (1, 1, 1)})
|
||||
|
||||
|
||||
class StructureTest(Fixture):
|
||||
def test_shares_precision_and_concepts_per_source(self) -> None:
|
||||
s = measure.structure(self.bundle, self.root, self.blocks, self.order)
|
||||
self.assertEqual(s["pages"], 2)
|
||||
self.assertEqual(s["concepts_per_source"], {"docs-en-a.md": 1, "release-v2.0.0.md": 1})
|
||||
# version sources are left out of the quote and precision counts
|
||||
self.assertEqual(s["quotes"]["entries"], 6)
|
||||
self.assertEqual(s["quotes"]["structural"], 1) # b2 quotes a heading line
|
||||
self.assertEqual(s["precision"], {"line": 3, "block": 1, "section": 1, "document": 1})
|
||||
self.assertEqual(s["islands"]["total"], sum(not v for v in view.related(
|
||||
audit.load_bundle(self.bundle, self.root).concepts).values()))
|
||||
|
||||
def test_duplicates_by_norm_alias_and_jaccard_in_any_bundle(self) -> None:
|
||||
b = self.tmp / "dups"
|
||||
for name, title in (("a.md", "Beds"), ("b.md", "Bed"), ("c.md", "Raised beds"), ("d.md", "Gravel path"),
|
||||
("e.md", "Gravel path rules"), ("f.md", "Tools")):
|
||||
write_page(b, name, title, "x[^s].", [src("s", "docs-en-a.md", 5, "Raised")],
|
||||
**({"x-aliases": ["Beds", "Raised beds"]} if name == "a.md" else {}))
|
||||
pairs = measure.duplicates(audit.load_bundle(b, self.root).concepts)
|
||||
self.assertIn(("a.md", "b.md", "norm"), pairs)
|
||||
self.assertIn(("a.md", "c.md", "alias"), pairs)
|
||||
self.assertIn(("d.md", "e.md", "jaccard"), pairs)
|
||||
self.assertFalse(any("f.md" in p for p in pairs))
|
||||
|
||||
def test_a_wrong_root_raises(self) -> None:
|
||||
with self.assertRaises(measure.RunError):
|
||||
measure.structure(self.bundle, self.tmp / "nowhere", self.blocks, self.order)
|
||||
|
||||
|
||||
class AuditRecordTest(Fixture):
|
||||
def test_uncited_sentences_are_classed_by_cause_and_yaml_form_apart(self) -> None:
|
||||
write_page(self.bundle, "odd.md", "Odd", "```bash\n\nPlain words here.\n\nSee also: [Beds](beds.md)",
|
||||
[{"id": "y", "resource": "/kilde/docs-en-a.md", "x-locator": {"line": 5}, "x-quote": False}])
|
||||
out = self.tmp / "resultat.json"
|
||||
rec = measure.audit_record(self.bundle, self.root, "K", out)
|
||||
self.assertEqual(rec["exit"], 1)
|
||||
# labels.md in the shared fixture holds one more link lead-in ("See also: [Beds](beds.md)")
|
||||
self.assertEqual(rec["uncited"], {"fence line": 1, "link lead-in": 2, "prose": 1, "table row": 1})
|
||||
self.assertEqual(rec["yaml_form"], 1)
|
||||
self.assertEqual(json.loads(out.read_text(encoding="utf-8"))["variants"]["K"]["exit"], 1)
|
||||
with self.assertRaises(measure.RunError):
|
||||
measure.audit_record(self.bundle, self.root, "K", out) # one record per variant
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 6: cost
|
||||
|
||||
|
||||
def msg(mid: str, inp: int, cr: int, cw: int, out: int, stop: str | None = "end_turn") -> dict[str, Any]:
|
||||
return {"type": "assistant", "message": {"id": mid, "stop_reason": stop, "usage": {
|
||||
"input_tokens": inp, "cache_read_input_tokens": cr, "cache_creation_input_tokens": cw, "output_tokens": out}}}
|
||||
|
||||
|
||||
def jsonl(path: Path, lines: list[dict[str, Any]]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("\n".join(json.dumps(x) for x in lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
class HarvestTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.proj = Path(self._tmp.name)
|
||||
user = {"type": "user", "message": {"content": "task"}}
|
||||
jsonl(self.proj / "s1.jsonl", [user, msg("m0", 5, 100, 10, 7)])
|
||||
sub = self.proj / "s1" / "subagents"
|
||||
# a streamed message id appears twice with growing output: counted once, at its max
|
||||
jsonl(sub / "agent-a1.jsonl", [user, msg("x1", 3, 20_000, 2_000, 10, None), msg("x1", 3, 20_000, 2_000, 50),
|
||||
msg("x2", 1, 22_000, 500, 900)])
|
||||
(sub / "agent-a1.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "P1-K-pack1"}))
|
||||
jsonl(sub / "agent-a2.jsonl", [user, msg("y1", 2, 19_000, 1_000, 5)])
|
||||
(sub / "agent-a2.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "load test"}))
|
||||
jsonl(sub / "agent-a3.jsonl", [user, msg("z1", 1, 1, 1, 1, None)])
|
||||
(sub / "agent-a3.meta.json").write_text(json.dumps({"agentType": "p1-consolidator",
|
||||
"description": "P1-K+-run1"}))
|
||||
# a session not listed in okter.txt: its transcript is never opened
|
||||
jsonl(self.proj / "s9" / "subagents" / "agent-b1.jsonl", [user, msg("q", 1, 1, 1, 1)])
|
||||
(self.proj / "s9" / "subagents" / "agent-b1.meta.json").write_text(
|
||||
json.dumps({"agentType": "p1-compiler", "description": "P1-K-pack2"}))
|
||||
self.okter = self.proj / "okter.txt"
|
||||
self.okter.write_text("s1 step8\ns1 step9\n", encoding="utf-8")
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self._tmp.cleanup()
|
||||
|
||||
def test_four_categories_turns_startup_and_attribution(self) -> None:
|
||||
got = cost.harvest(self.proj, cost.sessions(self.okter))
|
||||
self.assertEqual(got["sessions_read"], ["s1"])
|
||||
runs = {r["unit"]: r for r in got["runs"]}
|
||||
k = runs["pack1"]
|
||||
self.assertEqual((k["variant"], k["input"], k["cache_read"], k["cache_write"], k["output"]),
|
||||
("K", 4, 42_000, 2_500, 950))
|
||||
self.assertEqual(k["turns"], 2)
|
||||
self.assertEqual(k["startup"], 3 + 20_000 + 2_000)
|
||||
self.assertEqual(runs["run1"]["variant"], "K+")
|
||||
self.assertEqual(runs["run1"]["stop_reason"], None) # a stopped agent is listed, not dropped
|
||||
self.assertEqual([u["description"] for u in got["unattributed"]], ["load test"])
|
||||
self.assertEqual(got["main"]["s1"]["cache_read"], 100)
|
||||
self.assertNotIn("pack2", runs)
|
||||
|
||||
def test_output_never_recorded_is_derived_from_the_next_messages_input_growth(self) -> None:
|
||||
# Found in session 2: a long streamed message keeps only its first usage record (output 2, no stop
|
||||
# reason). The next message's input side grew by that output plus the text injected between them (tool
|
||||
# results and rendered attachments), counted high at 2.5 characters per token so the output is a lower bound.
|
||||
def result(text: str) -> dict[str, Any]:
|
||||
return {"type": "user", "message": {"content": [{"type": "tool_result", "content": text}]}}
|
||||
|
||||
sub = self.proj / "s1" / "subagents"
|
||||
jsonl(sub / "agent-a4.jsonl", [{"type": "user", "message": {"content": "task"}},
|
||||
msg("w1", 2, 0, 10_000, 300, "tool_use"), result("x" * 400),
|
||||
msg("w2", 2, 10_000, 50_000, 2, None), msg("w2", 2, 10_000, 50_000, 2, None),
|
||||
result("y" * 40), msg("w3", 2, 60_000, 5_010, 20)])
|
||||
(sub / "agent-a4.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "P1-K-pack3"}))
|
||||
runs = {r["unit"]: r for r in cost.harvest(self.proj, cost.sessions(self.okter))["runs"]}
|
||||
w = runs["pack3"]
|
||||
self.assertEqual(w["output_recorded"], 300 + 2 + 20)
|
||||
self.assertEqual(w["output"], 300 + (5_010 - 40 * 2 // 5) + 20)
|
||||
self.assertEqual((w["output_derived_messages"], w["output_unrecorded"]), (1, 0))
|
||||
self.assertEqual(w["total"], 2 + 2 + 2 + 70_000 + 65_010 + w["output"])
|
||||
# the last message of a run has no next message: its output stays as recorded and is flagged
|
||||
self.assertEqual((runs["run1"]["output"], runs["run1"]["output_unrecorded"]), (1, 1))
|
||||
self.assertEqual(runs["pack1"]["output_derived_messages"], 0)
|
||||
|
||||
def test_growth_from_injected_attachments_is_not_counted_as_output(self) -> None:
|
||||
# The load test's pattern: a short call, then 5133 characters of rendered attachments before the next turn.
|
||||
sub = self.proj / "s1" / "subagents"
|
||||
jsonl(sub / "agent-a5.jsonl", [{"type": "user", "message": {"content": "task"}},
|
||||
msg("v1", 2, 0, 18_602, 16, None),
|
||||
{"type": "attachment", "rendered": "z" * 5_133},
|
||||
{"type": "user", "message": {"content": [{"type": "tool_result",
|
||||
"content": "w" * 151}]}},
|
||||
msg("v2", 2, 18_602, 1_894, 24, None)])
|
||||
(sub / "agent-a5.meta.json").write_text(json.dumps({"agentType": "p1-compiler", "description": "P1-LOAD-x"}))
|
||||
runs = {r["unit"]: r for r in cost.harvest(self.proj, cost.sessions(self.okter))["runs"]}
|
||||
self.assertEqual((runs["x"]["output"], runs["x"]["output_derived_messages"]), (16 + 24, 0))
|
||||
|
||||
|
||||
class ProjectionTest(unittest.TestCase):
|
||||
def test_least_squares_on_cost_per_block_against_blocks_read(self) -> None:
|
||||
p = cost.project([(600_000, 600, 0), (220_000, 200, 600)], total_blocks=3000)
|
||||
self.assertAlmostEqual(p["fit"]["a"], 1000)
|
||||
self.assertAlmostEqual(p["fit"]["b"], 1 / 6)
|
||||
self.assertAlmostEqual(p["fit"]["value"], 3_750_000)
|
||||
self.assertAlmostEqual(p["flat"], 3_075_000)
|
||||
|
||||
def test_one_point_has_no_growth_term(self) -> None:
|
||||
p = cost.project([(600_000, 600, 0)], total_blocks=3000)
|
||||
self.assertEqual(p["fit"]["b"], 0)
|
||||
self.assertEqual(p["fit"]["note"], "one point, no growth term")
|
||||
self.assertAlmostEqual(p["fit"]["value"], 3_000_000)
|
||||
|
||||
def test_stop_per_block_against_naive_per_source(self) -> None:
|
||||
s = cost.stop_k(pack1_cost=600_000, pack1_blocks=600, cut_blocks=1200, pack1_sources=5, cut_sources=19)
|
||||
self.assertEqual(s["per_block"], 1_200_000)
|
||||
self.assertEqual(s["per_source"], 2_280_000)
|
||||
self.assertTrue(s["stop"])
|
||||
self.assertFalse(cost.stop_k(500_000, 600, 1200, 5, 19)["stop"]) # exactly 1 M does not stop
|
||||
self.assertTrue(cost.stop_k(500_001, 600, 1200, 5, 19)["stop"])
|
||||
|
||||
def test_kplus_stop(self) -> None:
|
||||
self.assertFalse(cost.stop_kplus(run1_cost=200_000, run1_pages=8, pages_total=40)["stop"])
|
||||
self.assertTrue(cost.stop_kplus(run1_cost=200_001, run1_pages=8, pages_total=40)["stop"])
|
||||
|
||||
|
||||
class AllocationTest(unittest.TestCase):
|
||||
def test_tokens_per_source_by_block_share_and_the_two_means(self) -> None:
|
||||
names = [f"s{i:02d}.md" for i in range(1, 20)]
|
||||
blocks_doc = {"order": names, "blocks": [{"file": n, "start": k, "end": k} for n in names for k in (1, 2)]}
|
||||
packs = [names[i:i + 5] for i in range(0, 19, 5)]
|
||||
costs = {1: 1000, 2: 500, 3: 500, 4: 800}
|
||||
per = cost.allocate(costs, packs, blocks_doc)
|
||||
self.assertEqual(per["s01.md"], 200)
|
||||
self.assertEqual(per["s16.md"], 200)
|
||||
means = cost.means(per, names)
|
||||
self.assertEqual(means["1-5"], 200)
|
||||
self.assertEqual(means["15-19"], 180) # source 15 in pack 3 (100), 16-19 in pack 4 (200 each)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------- step 7: side by side
|
||||
|
||||
|
||||
class SideBySideTest(Fixture):
|
||||
def setUp(self) -> None:
|
||||
super().setUp()
|
||||
self.begreper = self.tmp / "begreper.txt"
|
||||
self.begreper.write_text("Beds\tBeds;Raised beds\nLabels\tLabels\nCompost\tCompost\n", encoding="utf-8")
|
||||
self.d = self.tmp / "D"
|
||||
write_page(self.d, "r-beds.md", "Raised beds", "- [Alpha](k.md)", [src("o1", "docs-en-a.md", 5, "Raised beds")])
|
||||
self.l4 = self.tmp / "L4"
|
||||
write_page(self.l4, "x.md", "THE raised-beds, explained!", "Beds drain.[^b1]",
|
||||
[src("b1", "docs-en-a.md", 5, "Raised beds drain well.")])
|
||||
write_page(self.l4, "y.md", "Compost", "Compost helps.[^b1]",
|
||||
[src("b1", "docs-en-a.md", 99, "Compost")]) # a link past the end of the file
|
||||
|
||||
def columns(self) -> list[tuple[str, Path | None]]:
|
||||
return [("K", self.bundle), ("K+", None), ("D", self.d), ("L4", self.l4)]
|
||||
|
||||
def test_three_concepts_in_four_columns_with_the_empty_states(self) -> None:
|
||||
out = self.tmp / "side.html"
|
||||
sidebyside.render(sidebyside.read_begreper(self.begreper), self.columns(), self.root, out)
|
||||
html = out.read_text(encoding="utf-8")
|
||||
self.assertEqual(html.count('<tr class="concept">'), 3)
|
||||
self.assertEqual(html.count('<td class="cell'), 12)
|
||||
self.assertIn("ikke bygget", html) # K+ stopped
|
||||
self.assertIn("ikke i registeret", html) # D has no Labels register page
|
||||
self.assertIn("ingen side", html) # K is built and has no Compost page
|
||||
self.assertNotIn("http", html)
|
||||
|
||||
def test_l4_finds_a_title_with_other_case_and_punctuation(self) -> None:
|
||||
found = sidebyside.find(self.l4, ("Raised beds", ("Raised beds",)), l4=True)
|
||||
self.assertEqual(found, ["x.md"])
|
||||
|
||||
def test_the_link_check_flags_a_line_past_the_end(self) -> None:
|
||||
out = self.tmp / "side.html"
|
||||
sidebyside.render(sidebyside.read_begreper(self.begreper), self.columns(), self.root, out)
|
||||
resolved, total = sidebyside.check(out, self.root)
|
||||
self.assertEqual(total - resolved, 1)
|
||||
self.assertEqual(sidebyside.main(["check", "--html", str(out), "--kilde-root", str(self.root)]), 1)
|
||||
|
||||
|
||||
class ScriptsTest(unittest.TestCase):
|
||||
SCRIPTS = (ROOT / "tests" / "p1_repo.sh", ROOT / "tests" / "p1_trial.sh")
|
||||
|
||||
def test_bash_syntax_and_ascii(self) -> None:
|
||||
for script in self.SCRIPTS:
|
||||
self.assertEqual(subprocess.run(["bash", "-n", str(script)]).returncode, 0, script.name)
|
||||
script.read_bytes().decode("ascii")
|
||||
|
||||
def test_the_trial_check_fails_every_check_when_nothing_was_built(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
env = {**os.environ, "P1_ROOT": tmp}
|
||||
res = subprocess.run(["bash", str(self.SCRIPTS[1])], env=env, capture_output=True, text=True, cwd=ROOT)
|
||||
self.assertEqual(res.returncode, 1)
|
||||
lines = [x for x in res.stdout.splitlines() if x.strip()]
|
||||
self.assertGreaterEqual(len(lines), 8)
|
||||
self.assertTrue(all(x.startswith("FAIL") for x in lines), res.stdout)
|
||||
|
||||
|
||||
class NoModelTest(unittest.TestCase):
|
||||
def test_trial_modules_import_no_network_or_model_client(self) -> None:
|
||||
for path in sorted((ROOT / "evals" / "p1").glob("*.py")):
|
||||
source = path.read_text(encoding="utf-8")
|
||||
for banned in BANNED:
|
||||
self.assertNotIn(f"import {banned}", source, path.name)
|
||||
self.assertNotIn(f"from {banned}", source, path.name)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Add table
Add a link
Reference in a new issue