okf-wiki 1.1.1

First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-11-14 21:03:20 +01:00
commit 0fd86a23a3
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
227 changed files with 96077 additions and 0 deletions

125
evals/p1/README.md Normal file
View file

@ -0,0 +1,125 @@
# P1 method trial (TRIAL, branch `p1-metodeprove`, not v1 code)
The P1 method trial builds the 19 sources of the cut (the sources behind d01 d02 d03 d04 d37 d46 d53) three ways, in one
fixed order, and measures cost and quality side by side, so the operator can choose the build method for v1 by seeing
small real examples (PLAN.md § Rammer, order `20261002T094056Z-495685981`, plan v3).
- **K**: the fragment compiler. Code splits the sources into blocks, a narrow trial agent (`p1-compiler`) writes short
fragments per pack of five sources, and code resolves concept identity and assembles the pages.
- **K+**: a copy of K plus one consolidation pass (`p1-consolidator`) that rewrites pages with three or more fragments
into flowing text, keeping every footnote label.
- **D**: no model. A generated map of each source and a code-built concept register.
No module here calls a model. The model work is done in the operator's session by the two trial agent types under
`.claude/agents/`. Every output lives outside the repository, under `<wiki>/prove-p1/` (CLAUDE.md rule 3); this README
records numbers only.
## Modules
Run each as `PYTHONPATH=evals:. python3 -m p1.<module> ...`; each module's docstring is its usage.
| Module | What it does |
|---|---|
| `split.py` | the fixed source order, blocks with line spans, the block total, the three side-by-side concepts |
| `dmap.py` | D: map pages and register pages |
| `kpack.py` | K: the skeleton from every docs page, pack input parts, fragment validation, concept identity |
| `assemble.py` | K's pages from accepted fragments; K+'s copy, tasks and apply |
| `measure.py` | the audit record, coverage of source blocks, structure counts |
| `cost.py` | tokens from the run sessions' transcripts, tokens per source, projection, the stop rule |
| `sidebyside.py` | the side-by-side page and its link check |
Output files under `prove-p1/`: `rekkefolge.txt`, `blocks.json`, `begreper.txt`, `okter.txt`, `D/`, `K/`, `K+/`,
`resultat.json`, `struktur-<V>.json`, `dekning-<V>.json`, `kost.json`, `side-om-side.html`, `rapport.md`.
Tests: `python3 -m unittest discover -s evals/tests -p 'test_p1_*.py'`. Checks: `bash tests/p1_repo.sh` (repository)
and `bash tests/p1_trial.sh` (the trial's outputs; red until the runs are done).
## Results
Numbers only, filled in by the run steps (plan steps 8-11).
### Run 0 (step 8, 02.10.2026)
- Order: 19 sources (3 docs pages, then 16 release notes by version), equal to the files cited by the L4 cut.
- Blocks: 1221 in the 19 sources (docs pages 599, release notes 622; pack 1 = sources 1-5 = 602 blocks); 33727 blocks
in all 599 files of `kilde/` (the D12 projection base).
- Blocks per source in order: 59, 308, 232, 1, 2, 22, 35, 76, 21, 25, 57, 35, 34, 52, 33, 22, 46, 53, 108.
- Load test (the empty baseline run, not counted toward any cap): both trial types spawned and wrote under `prove-p1/`,
so the agents' work directory is `prove-p1/` (no `runtime/p1` fallback). `p1-compiler`: startup 18604, 2 turns,
39142 tokens in all; `p1-consolidator`: startup 18167, 2 turns, 38270 tokens in all.
- D: 254 pages (19 map, 235 register), 0 model tokens. Auditor exit 0, 0 findings. Criterion 1: 0 of 0 (bodies are
headings and link lists only, D8, by construction). Coverage 0 of 1221 blocks: by construction, because D cites no
source in a body footnote (D21 counts body-cited entries only; D's locators live in `sources[]`). Duplicate pairs 9
(5 by name, 4 by title Jaccard), merge candidates 0, islands 0, concept test 2 held by 254 pages, test 4 by 19.
### Run K (step 9, 02.10.2026)
- Skeleton: 3493 index entries from the titles and level 2-3 headings of all 198 docs pages (0 tokens).
- The cost harvester was fixed twice during the run, each with a failing test first (`77b16ab`, `0a50286`): a long
streamed message keeps only its first usage record in the transcript (output 2, no stop reason), so output was
undercounted (373 recorded for pack 1's 336 fragments). Output is now derived from the next message's input growth,
less the injected text counted high (2.5 characters per token), so every K output below is a lower bound. Checked on
the load test's known short call, which keeps its recorded value. The recorded value stays beside it in `kost.json`.
- K: per pack (blocks / index lines sent of 3493 / input parts / fragments accepted of proposed / rejected / accepted
with a concept not among the lines sent / turns / tokens):
- pack 1: 602 / 230 / 4 / 336 of 336 / 0 / 0 / 3 / 372996
- pack 2: 179 / 117 / 1 / 87 of 87 / 0 / 0 / 3 / 125716
- pack 3: 211 / 125 / 1 / 108 of 108 / 0 / 0 / 3 / 141320
- pack 4: 229 / 127 / 1 / 119 of 119 / 0 / 0 / 3 / 153494
- No fragment was rejected for any reason (incl. `release-note-new` 0); no new concept was proposed, so all 650
accepted fragments enrich skeleton entries. Merge candidates 0.
- Stop line after pack 1: projected per block 756525 (naive per source 1417385), cap 1000000: continue.
- K total: 793526 tokens in 4 runs and 12 turns (input 24, cache read 228534, cache write 429550, output 135418);
startup 74596 (4 x about 18.6 k). Tokens per source, mean of sources 1-5: 74599; of sources 15-19: 35119.
- Projected to all 599 sources (33727 blocks): fit 52842866 (4 points, growth term b = 0.0554, not validated);
flat 21919125.
- Auditor exit 0, 0 findings. Criterion 1: 664 of 664. Quotes: 650 entries, 160 supplied by code, 0 on a structural
line. Precision of body-cited entries: line 591, block 59, section 0, document 0.
- Coverage: 623 of 1221 blocks. Per position in the pack (covered of blocks, n packs): 1: 85 of 160 (4); 2: 241 of 424
(4); 3: 181 of 395 (4); 4: 83 of 182 (4); 5: 33 of 60 (3). A direction signal, n = 3-4.
- Structure: 212 concept pages, 3281 skeleton entries unused. Concepts per source in order: 11, 61, 31, 1, 0, 5, 13,
15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11. Duplicate pairs 3 (all title Jaccard), islands 63, documents per page mean
1.65 (max 8), version numbers in running text 5 of 664 claims, concept test 2 held by 212, test 4 by 73.
- Seen in the compiler's own reports: in one pack it wrote numbers as words to avoid the quote rule (D5 requires a
quote when the text holds a digit); the compilers' own fragment counts differed from the files (the files count).
### Run K+ (step 10, 02.10.2026)
- K+: a copy of K; 76 pages carry 3 or more fragments, which gives 10 consolidation runs of up to 8 pages.
- Run 1: 8 pages, applied 8, refused 0. 109651 tokens (input 8, cache read 72768, cache write 29909, output 6966, a lower
bound), 4 turns, startup 18176. The consolidator took one more turn than its instructions allow (it read its
output back).
- Stop line after run 1: 109651 x 76 / 8 = 1041685 > cap 1000000: STOP. Runs 2-10 were not run; K+ stands as K with
8 of 76 pages consolidated.
- K+ total = K + consolidation = 793526 + 109651 = 903177 tokens.
- Auditor exit 0, 0 findings. Criterion 1: 662 of 662 (2 fewer claims than K: sentences joined). Coverage 623 of 1221,
precision and quotes as K (line 591, block 59; 160 code quotes, 0 structural). 212 pages, 3 duplicate pairs,
63 islands, version numbers in running text 5 of 662.
### Step 11: the side-by-side table (02.10.2026)
Corrections after run 0, each its own commit with a failing test first: the backticked-term length is checked after
trimming (`26f521f`) and the concept pick counts a document only where the term stands as a term (`09a50b2`). The
first pick and D built before `26f521f` are kept in `prove-p1/forkastet/`; D was rebuilt (252 pages: 19 map, 233
register; 9 duplicate pairs; auditor exit 0) and the concepts were picked again by the same rule.
| variant | built to source | concepts (total / per source 1-5, 15-19 / test 2 / test 4) | skeleton unused | merge candidates | coverage | tokens per source (input / cache read / cache write / output / startup) | 1-5 | 15-19 | turns / subagents | projected 599 (fit / flat) | criterion 1 | fragments accepted / code quotes / structural quotes | precision (line / block / section / document) | pages / islands / duplicates |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| K | 19 of 19 | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | 1 / 12028 / 22608 / 7127 / 3926 | 74599 | 35119 | 12 / 4 | 52842866 / 21919125 | 664 of 664 | 650 of 650 / 160 / 0 | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
| K+ | 19 of 19, consolidation stopped after run 1 (8 of 76 pages) | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | consolidation 109651 for 8 pages; K+ total = K + consolidation = 903177 | - | - | 4 / 1 | not computed: stopped after 1 of 10 runs, and the pages to consolidate at 599 sources are not measured | 662 of 662 | as K | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
| D | 19 of 19 | 0 concept pages (252 map and register) / 0, 0 by construction / 252 / 19 | - | 0 | 0 of 1221 (by construction) | 0 | 0 | 0 | 0 / 0 | 0 | 0 of 0 (by construction) | - | - | 252 / 0 / 9 |
| L4 (structure counted today) | 19 of 19 | 151 / 33, 33 / 151 / 68 | - | 0 | 685 of 1221 | see the cost row | - | - | - | - | 1048 of 1048 | - / 0 / 41 of 832 | 832 / 0 / 0 / 0 | 151 / 54 / 2 |
| L2b / L4 cost | measured 27-28.09, not measured again | | | | | L4 105.8 M for 19 sources (84 % cache read); L2b writing 30.93 M for 19 sources | | | | L2b about 1.6-1.8 bn writing + 8.6 bn reading | | | | |
- Every K and K+ output number is a lower bound (the harvester fix in run K). Tokens per source split each pack's cost
by block share. The projection is not validated (4 points).
- Side-by-side page: 3 concepts x 4 columns; 454 of 454 source links resolve. Main session (orchestration, apart):
10.9 M tokens up to step 11, almost all cache read.
- Concepts per source in order 1-19 (a page counts at the earliest source its body cites, D10): K and K+ 11, 61, 31,
1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11; L4 2, 9, 19, 1, 2, 10, 14, 17, 5, 6, 14, 10, 7, 2, 9, 6, 3, 6, 9;
D 0 for every source (its register pages cite nothing in a body).
- K+ consolidation cost 13706 per page (lower bound) and is not split per source, because a page joins fragments from
several sources.
- Seen in the review (BLOCK, not fixed in this run): 45 of 1221 blocks are only MDX tag lines and cannot be covered
(K 623 of 1176 without them); the index-line filter sends a line when the whole name occurs as a phrase, not per
token as D7 is written; `cost.summarize` and the stop command have no test.

5
evals/p1/__init__.py Normal file
View file

@ -0,0 +1,5 @@
"""TRIAL, branch p1-metodeprove, not v1 code. Nothing outside evals/ imports it.
The P1 method trial (PLAN.md § Rammer, order 20261002T094056Z-495685981): three build methods for a concept wiki (K, K+,
D) on the 19 sources of the cut, measured side by side for cost and quality. No module here calls a model.
"""

287
evals/p1/assemble.py Normal file
View file

@ -0,0 +1,287 @@
"""K's assembler and K+'s apply step (TRIAL, P1): pages from accepted fragments, by code. It calls no model.
build --work DIR --kilde-root DIR --out DIR [--at ISO]
One page per concept that at least one accepted fragment enriches (unused skeleton entries get none). Docs
fragments form the body; release-note fragments go under "## History" in version order. Every sentence ends
in a content footnote and, for a release note, a version footnote whose quote is the bare version on the
note's heading line. A fragment without a model quote gets a code quote: the first line of its span that is
not structural, marked x-quote-by: code. A fragment that names another concept's title links to it once.
copy --src DIR --out DIR
K+ starts as a copy of K.
tasks --bundle DIR --work DIR [--min 3] [--per-run 8]
One task file per consolidation run (pages with at least --min fragments, --per-run pages each) in
<work>/Kplus-oppgaver/, output path <work>/Kplus-bodies/run-N.json.
apply --bundle DIR --bodies FILE
Replaces a page body only when the new body keeps every old footnote label and adds none; prints
"applied X, refused Y".
--kilde-root is the folder that contains kilde/. Exit 0 = done, 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import shutil
import sys
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from okfwiki.audit import FOOTNOTE_DEF, FOOTNOTE_REF, HEADING, read_document, sentence_spans
from p1.dmap import frontmatter
from p1.split import FENCE, headings_of, version_key
GENERATOR = "okf-wiki/p1-assemble"
QUOTE_MAX = 200
HTML_TAG_ONLY = re.compile(r"^\s*</?[A-Za-z][^<>]*/?>\s*$")
TABLE_RULE = re.compile(r"^\s*\|?\s*:?-+:?\s*(\|\s*:?-+:?\s*)*\|?\s*$")
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
@dataclass
class Run:
task: Path
output: Path
pages: list[str]
def structural(line: str) -> bool:
return (not line.strip() or HEADING.match(line) is not None or FENCE.match(line) is not None
or HTML_TAG_ONLY.match(line) is not None or TABLE_RULE.match(line) is not None)
def code_quote(span: list[str], start: int) -> tuple[str, int]:
"""The first non-structural line of a span (stripped, capped), and its file line."""
for i, line in enumerate(span):
if not structural(line):
return line.strip()[:QUOTE_MAX], start + i
return span[0].strip()[:QUOTE_MAX] or span[0], start
def footnoted(text: str, labels: list[str]) -> str:
text = text.strip()
if text and text[-1] not in ".!?":
text += "."
marks = "".join(f"[^{x}]" for x in labels)
out, last = [], 0
for _a, b in sentence_spans(text):
out.append(text[last:b] + marks)
last = b
out.append(text[last:])
return "".join(out)
def _link_titles(text: str, titles: dict[str, str], self_id: str, linked: set[str]) -> str:
for cid, title in sorted(titles.items(), key=lambda kv: -len(kv[1])):
if cid == self_id or cid in linked or len(title) < 4:
continue
pattern = re.compile(r"(?<![\w`\[])" + re.escape(title) + r"(?![\w`\]])", re.IGNORECASE)
m = pattern.search(text)
if m is None:
continue
text = text[:m.start()] + f"[{m.group(0)}]({cid}.md)" + text[m.end():]
linked.add(cid)
return text
def _version_line(lines: list[str], version: str) -> int:
for n, _level, text in headings_of(lines):
m = VERSION_HEADING.match(text)
if m and m.group(1) == version:
return n
raise RunError(f"no heading for version {version}")
def assemble(work: Path, kilde_root: Path, out: Path, at: str | None = None) -> int:
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
index = json.loads((work / "K-work" / "index.json").read_text(encoding="utf-8"))
frags = json.loads((work / "K-work" / "accepted.json").read_text(encoding="utf-8"))["fragments"]
entries = {e["id"]: e for e in index["entries"]}
kilde = kilde_root / "kilde"
cache: dict[str, list[str]] = {}
def lines_of(name: str) -> list[str]:
if name not in cache:
cache[name] = (kilde / name).read_text(encoding="utf-8").split("\n")
return cache[name]
by_concept: dict[str, list[dict[str, Any]]] = {}
for f in frags:
by_concept.setdefault(f["concept_id"], []).append(f)
titles = {cid: str(entries[cid]["title"]) for cid in by_concept}
pages: dict[str, str] = {}
for cid, items in by_concept.items():
e = entries[cid]
sources: list[dict[str, Any]] = []
versions: dict[str, str] = {}
body_parts: list[str] = []
history: list[tuple[tuple[int, ...], int, str]] = []
linked: set[str] = set()
for k, f in enumerate(items, 1):
lines = lines_of(f["file"])
src: dict[str, Any] = {"id": f"f{k}", "resource": f"/kilde/{f['file']}", "title": f["section"]}
if f.get("quote"):
src["x-locator"] = {"line": f["quote_line"]}
src["x-quote"] = f["quote"]
else:
quote, line = code_quote(lines[f["start"] - 1:f["end"]], f["start"])
src["x-locator"] = {"line": line}
src["x-quote"] = quote
src["x-quote-by"] = "code"
src["x-span"] = [f["start"], f["end"]]
sources.append(src)
labels = [f"f{k}"]
if f.get("version"):
v = str(f["version"])
if v not in versions:
versions[v] = "v" + re.sub(r"\D", "", v)
sources.append({"id": versions[v], "resource": f"/kilde/{f['file']}", "title": v,
"x-locator": {"line": _version_line(lines, v)}, "x-quote": v})
labels.append(versions[v])
text = _link_titles(footnoted(f["text"], labels), titles, cid, linked)
history.append((version_key(f"release-v{v}.md"), k, text))
else:
body_parts.append(_link_titles(footnoted(f["text"], labels), titles, cid, linked))
body = [f"# {e['title']}", ""]
for p in body_parts:
body += [p, ""]
if history:
body += ["## History", ""]
body += [f"- {t}" for _, _, t in sorted(history)]
body.append("")
body += [f"[^{s['id']}]: {Path(s['resource']).name}, line {s['x-locator']['line']}" for s in sources]
desc = FOOTNOTE_REF.sub("", items[0]["text"]).strip()[:200]
fm: dict[str, Any] = {"type": "Concept", "title": str(e["title"]), "description": desc, "sources": sources,
"x-aliases": [str(a) for a in e.get("aliases", [])], "x-fragments": len(items)}
if e.get("kind") == "merge-candidate" and e.get("merge_of"):
fm["x-merge-candidate"] = str(e["merge_of"])
fm["generated"] = {"by": GENERATOR, "at": at}
pages[f"{cid}.md"] = frontmatter(fm) + "\n" + "\n".join(body) + "\n"
out.mkdir(parents=True)
for name, text in pages.items():
(out / name).write_text(text, encoding="utf-8")
index_md = ["---", 'okf_version: "0.2"', "---", "", "# Concepts", ""]
index_md += [f"- [{titles[cid]}]({cid}.md)" for cid in sorted(by_concept)]
(out / "index.md").write_text("\n".join(index_md) + "\n", encoding="utf-8")
(out / "log.md").write_text(f"# Log\n\n## {at[:10]}\n* **Build**: K assembled by {GENERATOR}: {len(pages)} "
f"pages from {len(frags)} accepted fragments.\n", encoding="utf-8")
return len(pages)
def copy(src: Path, dst: Path) -> None:
if dst.exists():
raise RunError(f"{dst} exists; refusing to overwrite")
shutil.copytree(src, dst)
def _split_page(text: str) -> tuple[str, str, list[str], list[str]]:
"""(frontmatter block, title line, body lines without title and footnote definitions, definition lines)."""
end = text.index("\n---\n", 4) + 5
head, rest = text[:end], text[end:].split("\n")
title = next(line for line in rest if line.startswith("# "))
body = [line for line in rest if line != title and not FOOTNOTE_DEF.match(line.strip())]
defs = [line for line in rest if FOOTNOTE_DEF.match(line.strip())]
return head, title, body, defs
def consolidation_tasks(bundle: Path, work: Path, min_fragments: int = 3, per_run: int = 8) -> list[Run]:
chosen: list[str] = []
for p in sorted(bundle.glob("*.md")):
if p.name in ("index.md", "log.md"):
continue
fm = read_document(p, p.name).frontmatter or {}
if int(fm.get("x-fragments", 0)) >= min_fragments:
chosen.append(p.stem)
runs: list[Run] = []
for i in range(0, len(chosen), per_run):
n = len(runs) + 1
task = work / "Kplus-oppgaver" / f"run-{n}.md"
output = work / "Kplus-bodies" / f"run-{n}.json"
if task.exists():
raise RunError(f"{task} exists; refusing to overwrite")
parts = [f"# K+ consolidation run {n}\n\nRewrite each page body below into flowing text. Output: {output}\n"]
for cid in chosen[i:i + per_run]:
text = (bundle / f"{cid}.md").read_text(encoding="utf-8")
fm = read_document(bundle / f"{cid}.md", f"{cid}.md").frontmatter or {}
_head, title, body, _defs = _split_page(text)
quotes = [f"- [^{s['id']}]: {s.get('x-quote', '')}" for s in fm.get("sources", [])]
parts.append(f"## Page {cid} ({title[2:]})\n\nBody:\n\n" + "\n".join(body).strip()
+ "\n\nWhat each label quotes:\n\n" + "\n".join(quotes) + "\n")
task.parent.mkdir(parents=True, exist_ok=True)
task.write_text("\n".join(parts), encoding="utf-8")
runs.append(Run(task, output, chosen[i:i + per_run]))
return runs
def apply(bundle: Path, bodies_path: Path) -> tuple[int, int]:
bodies = json.loads(bodies_path.read_text(encoding="utf-8"))
applied = refused = 0
for cid, new in bodies.items():
page = bundle / f"{cid}.md"
if not page.is_file() or not isinstance(new, str):
refused += 1
continue
head, title, body, defs = _split_page(page.read_text(encoding="utf-8"))
old_labels = set(FOOTNOTE_REF.findall("\n".join(body)))
if set(FOOTNOTE_REF.findall(new)) != old_labels or FOOTNOTE_DEF.search(new):
refused += 1
continue
page.write_text(head + "\n" + title + "\n\n" + new.strip() + "\n\n" + "\n".join(defs) + "\n", encoding="utf-8")
applied += 1
return applied, refused
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
b = sub.add_parser("build")
b.add_argument("--work", required=True)
b.add_argument("--kilde-root", required=True)
b.add_argument("--out", required=True)
b.add_argument("--at")
c = sub.add_parser("copy")
c.add_argument("--src", required=True)
c.add_argument("--out", required=True)
t = sub.add_parser("tasks")
t.add_argument("--bundle", required=True)
t.add_argument("--work", required=True)
t.add_argument("--min", type=int, default=3)
t.add_argument("--per-run", type=int, default=8)
a = sub.add_parser("apply")
a.add_argument("--bundle", required=True)
a.add_argument("--bodies", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "build":
n = assemble(Path(args.work), Path(args.kilde_root), Path(args.out), args.at)
print(f"K: {n} pages -> {args.out}")
elif args.command == "copy":
copy(Path(args.src), Path(args.out))
print(f"copied {args.src} -> {args.out}")
elif args.command == "tasks":
for r in consolidation_tasks(Path(args.bundle), Path(args.work), args.min, args.per_run):
print(f"task: {r.task} output: {r.output} pages: {len(r.pages)}")
else:
applied, refused = apply(Path(args.bundle), Path(args.bodies))
print(f"applied {applied}, refused {refused}")
except (RunError, OSError, KeyError, ValueError) as exc:
print(f"assemble cannot run: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())

329
evals/p1/cost.py Normal file
View file

@ -0,0 +1,329 @@
"""The trial's cost from the run sessions' transcripts (TRIAL, P1): harvest, allocation, projection, stop. No model.
harvest --project-dir D --okter FILE --blocks F --total N [--packs FILE] --out FILE
Reads only the sessions listed in okter.txt ("<session-id> <step>" per line): each session's main
transcript (orchestration, reported apart) and every subagent transcript under <session>/subagents/. Per
subagent: input, cache read, cache write and output tokens kept apart (each message id counted once, at
its largest value, because a streamed message repeats its id with growing output), turns (distinct
message ids), startup (the first message's input side), agentType and description from its .meta.json.
A message whose final usage was never written (no stop reason on any record) gets its output from the
growth of the next message's input side less the injected text counted high (a lower bound); the
recorded output stays beside it (output_recorded).
A description that starts "P1-<variant>-<unit>" attributes the run; any other is listed as unattributed,
never dropped. Per variant: totals, tokens per source (a pack's cost split by block share), the means of
sources 1-5 and 15-19, and the projection to all sources (N blocks). The file is rewritten from the
transcripts on every harvest; stop rows already in it are kept.
stop VARIANT --kost FILE --blocks F [--pages-total N --pages-first M] [--record]
The stop rule (cap 1 M tokens, all four categories): K after pack 1, projected per block processed and,
beside it, naively per source; K+ after run 1, projected over the pages to consolidate. Exit 1 = stop.
With --record the stop row is written into the kost file.
Projection: cost per block c = a + b*R by least squares over the measured units, R = blocks read before the unit;
the projected total for T blocks is a*T + b*T*T/2 (the sum as R grows from 0 to T). One unit: b = 0. The flat
projection (mean cost per block times T) stands beside it. Exit 0 = done, 1 = stop (stop only), 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections.abc import Mapping
from pathlib import Path
from typing import Any
from direction_probe import USAGE_FIELDS, read_transcript
SCHEMA = "okf-wiki/p1-cost/1"
CAP = 1_000_000
CATEGORIES = {"input_tokens": "input", "cache_read_input_tokens": "cache_read",
"cache_creation_input_tokens": "cache_write", "output_tokens": "output"}
PREFIX = re.compile(r"^P1-([A-Za-z0-9+]+)-(\S+)")
PACK_UNIT = re.compile(r"^pack(\d+)")
CLASS = {"LOAD": "load", "K": "write", "K+": "write"}
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
def sessions(okter: Path) -> list[str]:
ids: list[str] = []
for line in okter.read_text(encoding="utf-8").splitlines():
parts = line.split()
if parts and parts[0] not in ids:
ids.append(parts[0])
return ids
INPUT_SIDE = ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")
CHARS_PER_TOKEN_LOW = 2.5 # injected text counted high, so a derived output is a lower bound
def _result_chars(message: dict[str, Any]) -> int:
content = message.get("content")
if not isinstance(content, list):
return 0
total = 0
for item in content:
if not (isinstance(item, dict) and item.get("type") == "tool_result"):
continue
inner = item.get("content")
if isinstance(inner, str):
total += len(inner)
elif isinstance(inner, list):
total += sum(len(str(x.get("text", ""))) for x in inner if isinstance(x, dict))
return total
def usage(path: Path) -> dict[str, Any]:
"""Tokens of one transcript. A message whose records all lack a stop reason never got its final usage written
(seen for a long streamed message: output 2); when another message follows, its output is the growth of the
next message's input side less the text injected between them (tool results and rendered attachments, counted
high at CHARS_PER_TOKEN_LOW), a lower bound, and never below what was recorded."""
per: dict[str, dict[str, int]] = {}
order: list[str] = []
complete: set[str] = set()
between: dict[str, int] = {}
last_stop: str | None = None
for line in read_transcript(path):
message = line.get("message")
if line.get("type") == "attachment" and order:
between[order[-1]] = between.get(order[-1], 0) + len(str(line.get("rendered") or ""))
continue
if not isinstance(message, dict):
continue
if line.get("type") == "user" and order:
between[order[-1]] = between.get(order[-1], 0) + _result_chars(message)
continue
if line.get("type") != "assistant":
continue
mid = str(message.get("id", ""))
if mid not in per:
order.append(mid)
seen = per.setdefault(mid, {})
fields = message.get("usage") or {}
for f in USAGE_FIELDS:
value = fields.get(f)
if isinstance(value, int):
seen[f] = max(seen.get(f, 0), value)
stop = message.get("stop_reason")
last_stop = stop if isinstance(stop, str) else None
if isinstance(stop, str):
complete.add(mid)
out: dict[str, Any] = {CATEGORIES[f]: sum(v.get(f, 0) for v in per.values()) for f in USAGE_FIELDS}
out["output_recorded"] = out["output"]
derived = unrecorded = 0
for i, mid in enumerate(order):
if mid in complete:
continue
if i + 1 == len(order):
unrecorded += 1
continue
growth = (sum(per[order[i + 1]].get(f, 0) for f in INPUT_SIDE) - sum(per[mid].get(f, 0) for f in INPUT_SIDE)
- int(between.get(mid, 0) / CHARS_PER_TOKEN_LOW))
recorded = per[mid].get("output_tokens", 0)
if growth > recorded:
out["output"] += growth - recorded
derived += 1
out["output_derived_messages"] = derived
out["output_unrecorded"] = unrecorded
out["total"] = sum(out[c] for c in CATEGORIES.values())
out["turns"] = len(order)
first = per[order[0]] if order else {}
out["startup"] = sum(first.get(f, 0) for f in INPUT_SIDE)
out["stop_reason"] = last_stop
return out
def harvest(project_dir: Path, ids: list[str]) -> dict[str, Any]:
runs: list[dict[str, Any]] = []
unattributed: list[dict[str, Any]] = []
main: dict[str, Any] = {}
for sid in ids:
top = project_dir / f"{sid}.jsonl"
if top.is_file():
main[sid] = usage(top)
for path in sorted((project_dir / sid / "subagents").glob("agent-*.jsonl")):
meta_path = path.with_name(path.stem + ".meta.json")
meta = json.loads(meta_path.read_text(encoding="utf-8")) if meta_path.is_file() else {}
row = {"session": sid, "agent": path.stem, "agentType": meta.get("agentType"),
"description": str(meta.get("description", "")), **usage(path)}
m = PREFIX.match(row["description"])
if m is None:
unattributed.append(row)
continue
row["variant"], row["unit"] = m.group(1), m.group(2)
row["class"] = CLASS.get(m.group(1), "other")
runs.append(row)
return {"sessions_read": ids, "runs": runs, "unattributed": unattributed, "main": main}
def allocate(pack_costs: Mapping[int, float], packs: list[list[str]], blocks_doc: dict[str, Any]) -> dict[str, float]:
"""Tokens per source: each pack's cost split over its sources by their share of the pack's blocks."""
count: dict[str, int] = {}
for b in blocks_doc["blocks"]:
count[b["file"]] = count.get(b["file"], 0) + 1
per: dict[str, float] = {}
for n, names in enumerate(packs, 1):
if n not in pack_costs:
continue
total = sum(count.get(x, 0) for x in names) or 1
for x in names:
per[x] = pack_costs[n] * count.get(x, 0) / total
return per
def means(per: dict[str, float], order: list[str]) -> dict[str, float | None]:
def mean(names: list[str]) -> float | None:
vals = [per[x] for x in names if x in per]
return sum(vals) / len(vals) if vals else None
return {"1-5": mean(order[0:5]), "15-19": mean(order[14:19])}
def project(points: list[tuple[float, int, int]], total_blocks: int) -> dict[str, Any]:
"""points: (cost, blocks in the unit, blocks read before it)."""
if not points:
return {"fit": None, "flat": None, "T": total_blocks}
xs = [float(r) for _, _, r in points]
ys = [c / b for c, b, _ in points]
fit: dict[str, Any]
if len(points) == 1 or len(set(xs)) == 1:
a, b = sum(ys) / len(ys), 0.0
fit = {"a": a, "b": 0, "note": "one point, no growth term"}
else:
mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
a = my - b * mx
fit = {"a": a, "b": b, "note": f"{len(points)} points, not validated"}
fit["value"] = a * total_blocks + b * total_blocks * total_blocks / 2
flat = sum(c for c, _, _ in points) / sum(bl for _, bl, _ in points) * total_blocks
return {"fit": fit, "flat": flat, "T": total_blocks}
def stop_k(pack1_cost: int, pack1_blocks: int, cut_blocks: int, pack1_sources: int, cut_sources: int
) -> dict[str, Any]:
per_block = pack1_cost * cut_blocks / pack1_blocks
per_source = pack1_cost * cut_sources / pack1_sources
return {"per_block": per_block, "per_source": per_source, "cap": CAP, "stop": per_block > CAP}
def stop_kplus(run1_cost: int, run1_pages: int, pages_total: int) -> dict[str, Any]:
projected = run1_cost * pages_total / run1_pages
return {"projected": projected, "cap": CAP, "stop": projected > CAP}
def _pack_number(unit: str) -> int | None:
m = PACK_UNIT.match(unit)
return int(m.group(1)) if m else None
def summarize(h: dict[str, Any], blocks_doc: dict[str, Any], packs_log: dict[str, Any] | None,
total_blocks: int) -> dict[str, Any]:
order = [str(x) for x in blocks_doc["order"]]
count: dict[str, int] = {}
for b in blocks_doc["blocks"]:
count[b["file"]] = count.get(b["file"], 0) + 1
variants: dict[str, Any] = {}
for v in sorted({r["variant"] for r in h["runs"]}):
mine = [r for r in h["runs"] if r["variant"] == v]
row: dict[str, Any] = {c: sum(r[c] for r in mine) for c in (*CATEGORIES.values(), "total", "turns", "startup")}
row["subagents"] = len(mine)
row["class"] = mine[0]["class"]
row["counts_toward_cap"] = v in ("K", "K+")
if v == "K" and packs_log:
packs = [list(p["sources"]) for p in sorted(packs_log["packs"], key=lambda p: int(p["pack"]))]
pack_costs: dict[int, float] = {}
per_cat: dict[str, dict[int, float]] = {c: {} for c in CATEGORIES.values()}
for r in mine:
n = _pack_number(r["unit"])
if n is not None:
pack_costs[n] = pack_costs.get(n, 0) + r["total"]
for c in CATEGORIES.values():
per_cat[c][n] = per_cat[c].get(n, 0) + r[c]
row["per_pack"] = {str(n): c for n, c in sorted(pack_costs.items())}
row["per_source"] = allocate(pack_costs, packs, blocks_doc)
row["per_source_by_category"] = {c: allocate(per_cat[c], packs, blocks_doc) for c in CATEGORIES.values()}
row["means"] = means(row["per_source"], order)
points, read = [], 0
for n, names in enumerate(packs, 1):
blocks_n = sum(count.get(x, 0) for x in names)
if n in pack_costs:
points.append((pack_costs[n], blocks_n, read))
read += blocks_n
row["projection"] = project(points, total_blocks)
variants[v] = row
return variants
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
h = sub.add_parser("harvest")
h.add_argument("--project-dir", required=True)
h.add_argument("--okter", required=True)
h.add_argument("--blocks", required=True)
h.add_argument("--total", type=int, required=True)
h.add_argument("--packs")
h.add_argument("--out", required=True)
s = sub.add_parser("stop")
s.add_argument("variant", choices=("K", "K+"))
s.add_argument("--kost", required=True)
s.add_argument("--blocks", required=True)
s.add_argument("--packs")
s.add_argument("--pages-total", type=int)
s.add_argument("--pages-first", type=int)
s.add_argument("--record", action="store_true")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
if args.command == "harvest":
out = Path(args.out)
old = json.loads(out.read_text(encoding="utf-8")) if out.is_file() else {}
h = harvest(Path(args.project_dir), sessions(Path(args.okter)))
packs_log = json.loads(Path(args.packs).read_text(encoding="utf-8")) if args.packs else None
doc = {"schema": SCHEMA, **h, "variants": summarize(h, blocks_doc, packs_log, args.total),
"stops": old.get("stops", {})}
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
print(f"harvest: {len(h['runs'])} attributed runs, {len(h['unattributed'])} unattributed, sessions read: "
f"{', '.join(h['sessions_read'])} -> {out}")
return 0
kost_path = Path(args.kost)
kost = json.loads(kost_path.read_text(encoding="utf-8"))
runs = [r for r in kost["runs"] if r["variant"] == args.variant]
if args.variant == "K":
first = [r for r in runs if _pack_number(r["unit"]) == 1]
if not first:
raise RunError("no pack 1 run in the kost file; harvest after pack 1")
if not args.packs:
raise RunError("--packs is needed for K")
log = json.loads(Path(args.packs).read_text(encoding="utf-8"))
p1 = next(p for p in log["packs"] if int(p["pack"]) == 1)
row = stop_k(sum(r["total"] for r in first), int(p1["blocks"]), len(blocks_doc["blocks"]),
len(p1["sources"]), len(blocks_doc["order"]))
else:
first = [r for r in runs if r["unit"].startswith("run1")]
if not first or not args.pages_total or not args.pages_first:
raise RunError("K+ needs a run 1 in the kost file, --pages-total and --pages-first")
row = stop_kplus(sum(r["total"] for r in first), args.pages_first, args.pages_total)
print(f"{args.variant}: projected {row.get('per_block', row.get('projected'))} tokens"
+ (f" (naive per source {row['per_source']})" if "per_source" in row else "")
+ f", cap {CAP}: {'STOP' if row['stop'] else 'continue'}")
if args.record:
kost.setdefault("stops", {})[args.variant] = row
kost_path.write_text(json.dumps(kost, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
return 1 if row["stop"] else 0
except (RunError, OSError, json.JSONDecodeError, KeyError, StopIteration) as exc:
print(f"cost cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

178
evals/p1/dmap.py Normal file
View file

@ -0,0 +1,178 @@
"""D, the trial's no-model baseline (TRIAL, P1): a generated map and a code-built concept register. It calls no model.
build --blocks FILE --kilde-root DIR --out DIR [--at ISO]
One map page per source (k-<slug>.md) whose body mirrors the source's headings, with one sources[] entry per
section; one register page per kept term (r-<slug>.md) whose body is a pure link list to the map sections
where the term occurs, with one sources[] entry per occurrence. A term is kept when it is a docs-page heading
that passes concept test 2, or when it occurs in at least 2 of the documents. Bodies hold only headings and
pure link lists, so the claim count is 0 by construction; the exact locators live in sources[].
--kilde-root is the folder that contains kilde/ (the auditor's SOURCES argument). Exit 0 = written, 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import yaml
from p1.split import headings_of, is_meta, occurs, terms
MIN_DOCUMENTS = 2
GENERATOR = "okf-wiki/p1-dmap"
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
@dataclass
class DStats:
maps: int
registers: int
def slug(text: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
def frontmatter(data: dict[str, Any]) -> str:
"""A frontmatter block; PyYAML quotes any string that would load as another type (no, 1.10)."""
return "---\n" + yaml.safe_dump(data, sort_keys=False, allow_unicode=True, width=10_000) + "---\n"
def _anchors(headings: list[tuple[int, int, str]]) -> dict[int, str]:
seen: dict[str, int] = {}
out: dict[int, str] = {}
for line, _level, text in headings:
base = slug(text)
n = seen.get(base, 0)
seen[base] = n + 1
out[line] = base if n == 0 else f"{base}-{n}"
return out
def _title(name: str, headings: list[tuple[int, int, str]]) -> str:
first = next((t for _, level, t in headings if level == 1), None)
if first:
return first
return f"Release notes {headings[0][2]}" if name.startswith("release-") and headings else name
def build(blocks_doc: dict[str, Any], kilde_root: Path, out: Path, at: str | None = None) -> DStats:
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
names = [str(n) for n in blocks_doc["order"]]
kilde = kilde_root / "kilde"
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
lines = {n: t.split("\n") for n, t in texts.items()}
heads = {n: headings_of(lines[n]) for n in names}
anchors = {n: _anchors(heads[n]) for n in names}
titles = {n: _title(n, heads[n]) for n in names}
pages: dict[str, str] = {}
gen = {"by": GENERATOR, "at": at}
for n in names:
sources = []
body = [f"# {titles[n]}", ""]
for i, (line, level, text) in enumerate(heads[n], 1):
sources.append({"id": f"s{i}", "resource": f"/kilde/{n}", "title": text, "x-locator": {"line": line},
"x-quote": text})
if level > 1 or text != titles[n]:
body += [f"{'#' * max(level, 2)} {text}", ""]
fm = {"type": "Document map", "title": titles[n], "description": f"The sections of {n}.", "sources": sources,
"generated": gen}
pages[f"k-{slug(n.removesuffix('.md'))}.md"] = frontmatter(fm) + "\n" + "\n".join(body).rstrip() + "\n"
candidates = [t for n in names if n.startswith("docs-") for _, level, t in heads[n] if level in (2, 3)]
from_headings = {t for t in candidates if not is_meta(t)}
pool = list(dict.fromkeys(list(from_headings) + [t for n in names for t in terms(texts[n])]))
registers: dict[str, str] = {}
for term in pool:
if is_meta(term):
continue
holders = [n for n in names if occurs(term, texts[n])]
if term not in from_headings and len(holders) < MIN_DOCUMENTS:
continue
pattern = re.compile(r"(?<!\w)" + re.escape(term) + r"(?!\w)", re.IGNORECASE)
sources, links, seen = [], [], set()
for n in holders:
for i, raw in enumerate(lines[n], 1):
m = pattern.search(raw)
if m is None:
continue
section = max((h for h in heads[n] if h[0] <= i), default=None)
key = (n, section[0] if section else 0)
if key in seen:
continue
seen.add(key)
sources.append({"id": f"o{len(sources) + 1}", "resource": f"/kilde/{n}", "title": titles[n],
"x-locator": {"line": i}, "x-quote": m.group(0)})
target = f"k-{slug(n.removesuffix('.md'))}.md"
if section is not None:
target += f"#{anchors[n][section[0]]}"
where = titles[n] + (f" › {section[2]}" if section and section[2] != titles[n] else "")
links.append(f"- [{where.replace('[', '(').replace(']', ')')}]({target})")
if not sources:
continue
name = f"r-{slug(term)}.md"
k = 2
while name in registers:
name, k = f"r-{slug(term)}-{k}.md", k + 1
fm = {"type": "Register entry", "title": term, "description": f"Where {term} occurs in the sources.",
"sources": sources, "generated": gen}
registers[name] = frontmatter(fm) + "\n" + f"# {term}\n\n" + "\n".join(links) + "\n"
out.mkdir(parents=True)
for name, text in {**pages, **registers}.items():
(out / name).write_text(text, encoding="utf-8")
index = ["---", 'okf_version: "0.2"', "---", "", "# Document maps", ""]
index += [f"- [{titles[n]}](k-{slug(n.removesuffix('.md'))}.md)" for n in names]
index += ["", "# Register", ""]
index += [f"- [{name[2:-3]}]({name})" for name in sorted(registers)]
(out / "index.md").write_text("\n".join(index) + "\n", encoding="utf-8")
day = at[:10]
(out / "log.md").write_text(
f"# Log\n\n## {day}\n"
f"* **Build**: D by {GENERATOR}, {len(pages)} map pages and {len(registers)} register pages from {len(names)} "
f"sources. Thresholds, recorded before any count: a term is kept when it is a docs-page heading of level 2 or "
f"3 that passes concept test 2, or when it occurs in at least {MIN_DOCUMENTS} of the {len(names)} documents "
f"(case-insensitive, whole phrase). Terms: backticked spans of 3-40 characters and capitalised phrases of 2-4 "
f"words, a sentence-initial word dropped.\n",
encoding="utf-8")
return DStats(len(pages), len(registers))
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
b = sub.add_parser("build")
b.add_argument("--blocks", required=True)
b.add_argument("--kilde-root", required=True)
b.add_argument("--out", required=True)
b.add_argument("--at")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
s = build(doc, Path(args.kilde_root), Path(args.out), args.at)
except (RunError, OSError, json.JSONDecodeError) as exc:
print(f"dmap cannot run: {exc}", file=sys.stderr)
return 2
print(f"D: {s.maps} map pages, {s.registers} register pages -> {args.out}")
return 0
if __name__ == "__main__":
sys.exit(main())

374
evals/p1/kpack.py Normal file
View file

@ -0,0 +1,374 @@
"""K, the fragment compiler's code side (TRIAL, P1): skeleton, packs, input parts, validation, identity. No model.
skeleton --kilde DIR --out FILE
The concept index from the titles and level 2-3 headings of every docs page under DIR that pass concept
test 2. One entry per normalised name (the first page in name order wins); an index, not pages.
pack N --blocks FILE --index FILE --kilde-root DIR --work DIR [--size 5]
Pack N's input for the compiler: every block of its sources with its anchor (file › section › span) and
numbered lines, then the index lines whose name occurs in those blocks, then the output contract, in parts
of at most PART_CHARS characters (the Read tool's cap). Prints the part paths and the output path.
accept N --work DIR --blocks FILE --kilde-root DIR
Validates pack N's fragments, resolves each to a concept by code (existing, merge candidate, new or
rejected), updates the index and appends the accepted fragments. Prints "accepted X of N" and the
rejections by reason, and how many accepted fragments resolved to a concept whose index line was not
sent with the pack (the pack log keeps the ids sent; the risk of the D7 phrase filter).
The work directory holds K-oppgaver/ (task parts), K-fragments/ (the compiler's JSON) and K-work/ (index.json,
accepted.json, packs.json). Exit 0 = done, 2 = could not run.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import Counter
from pathlib import Path
from typing import Any
from okfwiki.audit import sentence_spans
from p1.split import docs_headings, is_meta
PART_CHARS = 60_000
MERGE_JACCARD = 0.6
NEEDS_QUOTE = re.compile(r"\d|`")
REASONS = ("malformed", "anchor-outside-pack", "anchor-crosses-blocks", "sentence-count", "quote-missing",
"quote-not-in-span", "tests-missing", "release-note-new", "meta")
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
def norm(name: str) -> str:
text = name.replace("`", "").lower()
text = re.sub(r"[^\w\s/.\-]", " ", text)
words = text.split()
if words and words[0] == "the":
words = words[1:]
words = [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
return " ".join(w.strip(".") or w for w in words)
def slug(text: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
def _unique_id(index: dict[str, Any], title: str) -> str:
taken = {e["id"] for e in index["entries"]}
base = slug(title)
eid, k = base, 2
while eid in taken:
eid, k = f"{base}-{k}", k + 1
return eid
def skeleton(kilde: Path) -> dict[str, Any]:
index: dict[str, Any] = {"schema": "okf-wiki/p1-index/1", "entries": []}
seen: dict[str, dict[str, Any]] = {}
for h in docs_headings(kilde):
if is_meta(h.text):
continue
n = norm(h.text)
if not n:
continue
if n in seen:
if h.text not in seen[n]["aliases"]:
seen[n]["aliases"].append(h.text)
continue
e = {"id": _unique_id(index, h.text), "title": h.text, "aliases": [h.text], "kind": "skeleton",
"merge_of": None, "origin": {"file": h.file, "line": h.line}}
index["entries"].append(e)
seen[n] = e
return index
def _names(e: dict[str, Any]) -> set[str]:
return {norm(a) for a in [e["title"], *e.get("aliases", [])]} - {""}
def _jaccard(a: str, b: str) -> float:
x, y = set(a.split()), set(b.split())
return len(x & y) / len(x | y) if x | y else 0.0
def resolve(fragment: dict[str, Any], index: dict[str, Any]) -> tuple[str, str | None]:
"""existing <id> | merge-candidate <id> | new | rejected <reason> (D6, D3b)."""
by_id = {e["id"]: e for e in index["entries"]}
if fragment.get("status") == "existing" and fragment.get("id") in by_id:
return "existing", str(fragment["id"])
n = norm(str(fragment.get("concept", "")))
for e in index["entries"]:
if n in _names(e):
return "existing", str(e["id"])
if str(fragment.get("anchor", {}).get("file", "")).startswith("release-"):
return "rejected", "release-note-new"
if is_meta(str(fragment.get("concept", ""))):
return "rejected", "meta"
best = max(((max(_jaccard(n, m) for m in _names(e)), str(e["id"])) for e in index["entries"] if _names(e)),
default=(0.0, ""))
if best[0] >= MERGE_JACCARD:
return "merge-candidate", best[1]
return "new", None
def relevant_lines(index: dict[str, Any], text: str) -> list[dict[str, Any]]:
"""Index entries whose normalised name or alias occurs as a phrase in the normalised text (D7)."""
hay = f" {norm(text)} "
return [e for e in index["entries"] if any(f" {m} " in hay for m in _names(e))]
def packs(names: list[str], size: int = 5) -> list[list[str]]:
return [names[i:i + size] for i in range(0, len(names), size)]
CONTRACT = """## Output contract
Write one JSON object `{"fragments": [...]}` to the output path named in your prompt. Each fragment:
`concept`, `status` ("existing" with the index `id`, or "new" with `tests`: {"topic_shape", "citation"}),
`text` (1-3 sentences), `anchor` {"file", "start", "end"} inside one block above, and `quote` (one piece copied from
one line inside the anchor; required when the text holds a digit or a backticked name). A new concept only from a docs
page, never from a release note.
"""
def render_input(pack_no: int, names: list[str], blocks_doc: dict[str, Any], index_lines: list[dict[str, Any]],
kilde: Path) -> list[str]:
lines = {n: (kilde / n).read_text(encoding="utf-8").split("\n") for n in names}
pieces: list[str] = [f"# K pack {pack_no}: {len(names)} sources\n\nSources in order: {', '.join(names)}\n\n"
+ CONTRACT + "\n## Blocks\n\n"]
for b in blocks_doc["blocks"]:
if b["file"] not in names:
continue
head = f"### {b['file']} › {b['section'] or '-'} › {b['start']}–{b['end']}"
if b.get("version"):
head += f" (version {b['version']})"
body = [f"{n}: {lines[b['file']][n - 1]}" for n in range(b["start"], b["end"] + 1)]
chunk = head + "\n"
for row in body:
if len(chunk) + len(row) + 1 > PART_CHARS // 2:
pieces.append(chunk + "\n")
chunk = head + " (continued)\n"
chunk += row + "\n"
pieces.append(chunk + "\n")
pieces.append("## Index (concepts named in these blocks)\n\n")
for e in index_lines:
aliases = "; ".join(a for a in e["aliases"] if a != e["title"])
pieces.append(f"- id: {e['id']} | {e['title']}" + (f" | aliases: {aliases}" if aliases else "") + "\n")
room = PART_CHARS - len("<!-- part 99 of 99 -->\n")
parts: list[str] = []
current = ""
for piece in pieces:
if current and len(current) + len(piece) > room:
parts.append(current)
current = ""
current += piece
if current:
parts.append(current)
total = len(parts)
return [f"<!-- part {i} of {total} -->\n" + p for i, p in enumerate(parts, 1)]
def validate(fragments: list[Any], names: list[str], blocks_doc: dict[str, Any], kilde: Path
) -> tuple[list[dict[str, Any]], Counter[str]]:
by_file: dict[str, list[dict[str, Any]]] = {}
for b in blocks_doc["blocks"]:
by_file.setdefault(b["file"], []).append(b)
texts: dict[str, list[str]] = {}
accepted: list[dict[str, Any]] = []
rejected: Counter[str] = Counter()
for f in fragments:
a = f.get("anchor") if isinstance(f, dict) else None
if not (isinstance(f, dict) and isinstance(f.get("concept"), str) and f["concept"].strip()
and f.get("status") in ("new", "existing") and isinstance(f.get("text"), str)
and isinstance(a, dict) and isinstance(a.get("file"), str)
and type(a.get("start")) is int and type(a.get("end")) is int and a["start"] <= a["end"]):
rejected["malformed"] += 1
continue
if a["file"] not in names:
rejected["anchor-outside-pack"] += 1
continue
block = next((b for b in by_file.get(a["file"], []) if b["start"] <= a["start"] and a["end"] <= b["end"]), None)
if block is None:
rejected["anchor-crosses-blocks"] += 1
continue
if not 1 <= len(sentence_spans(f["text"].strip())) <= 3:
rejected["sentence-count"] += 1
continue
if a["file"] not in texts:
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
span = texts[a["file"]][a["start"] - 1:a["end"]]
quote = f.get("quote")
if not (isinstance(quote, str) and quote.strip()):
if NEEDS_QUOTE.search(f["text"]):
rejected["quote-missing"] += 1
continue
quote, quote_line = None, None
else:
hit = next((a["start"] + i for i, line in enumerate(span) if quote in line), None)
if hit is None:
rejected["quote-not-in-span"] += 1
continue
quote_line = hit
tests = f.get("tests")
if f["status"] == "new" and not (isinstance(tests, dict) and str(tests.get("topic_shape", "")).strip()
and str(tests.get("citation", "")).strip()):
rejected["tests-missing"] += 1
continue
accepted.append({**f, "quote": quote, "quote_line": quote_line, "version": block.get("version"),
"section": block["section"], "position": names.index(a["file"]) + 1})
return accepted, rejected
def _read_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise RunError(f"cannot read {path}: {exc}") from exc
def _write_json(path: Path, data: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
def _pack_log(work: Path) -> dict[str, Any]:
path = work / "K-work" / "packs.json"
return _read_json(path) if path.is_file() else {"schema": "okf-wiki/p1-packs/1", "packs": []}
def _record(log: dict[str, Any], n: int) -> dict[str, Any]:
for r in log["packs"]:
if r["pack"] == n:
return r # type: ignore[no-any-return]
r = {"pack": n}
log["packs"].append(r)
return r
def cmd_pack(args: argparse.Namespace) -> int:
work, n = Path(args.work), args.n
doc = _read_json(Path(args.blocks))
all_packs = packs([str(x) for x in doc["order"]], args.size)
if not 1 <= n <= len(all_packs):
raise RunError(f"pack {n} does not exist; there are {len(all_packs)}")
names = all_packs[n - 1]
kilde = Path(args.kilde_root) / "kilde"
index = _read_json(Path(args.index))
text = "\n".join((kilde / name).read_text(encoding="utf-8") for name in names)
lines = relevant_lines(index, text)
parts = render_input(n, names, doc, lines, kilde)
out_dir = work / "K-oppgaver"
paths = [out_dir / f"pack-{n}-part-{i}.md" for i in range(1, len(parts) + 1)]
if any(p.exists() for p in paths):
raise RunError(f"pack {n}'s task parts exist in {out_dir}; refusing to overwrite")
out_dir.mkdir(parents=True, exist_ok=True)
for p, text_part in zip(paths, parts):
p.write_text(text_part, encoding="utf-8")
log = _pack_log(work)
rec = _record(log, n)
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]), index_lines=len(lines),
index_total=len(index["entries"]), index_ids=[str(e["id"]) for e in lines], parts=len(parts),
chars=sum(len(p) for p in parts))
_write_json(work / "K-work" / "packs.json", log)
for p in paths:
print(f"part: {p}")
print(f"output: {work / 'K-fragments' / f'pack-{n}.json'}")
print(f"index lines sent: {len(lines)} of {len(index['entries'])}")
return 0
def cmd_accept(args: argparse.Namespace) -> int:
work, n = Path(args.work), args.n
doc = _read_json(Path(args.blocks))
names = packs([str(x) for x in doc["order"]], args.size)[n - 1]
kilde = Path(args.kilde_root) / "kilde"
raw = _read_json(work / "K-fragments" / f"pack-{n}.json")
fragments = raw.get("fragments") if isinstance(raw, dict) else None
if not isinstance(fragments, list):
raise RunError(f"pack {n}'s output has no fragments list")
index_path = work / "K-work" / "index.json"
index = _read_json(index_path)
valid, rejected = validate(fragments, names, doc, kilde)
acc_path = work / "K-work" / "accepted.json"
acc: dict[str, Any] = (_read_json(acc_path) if acc_path.is_file()
else {"schema": "okf-wiki/p1-accepted/1", "fragments": []})
if any(a["pack"] == n for a in acc["fragments"]):
raise RunError(f"pack {n} is already accepted")
sent = set(_record(_pack_log(work), n).get("index_ids", []))
not_sent: Counter[str] = Counter()
accepted = 0
for f in valid:
outcome, ref = resolve(f, index)
if outcome == "rejected":
rejected[str(ref)] += 1
continue
if outcome == "existing":
cid = str(ref)
else:
cid = _unique_id(index, f["concept"])
index["entries"].append({"id": cid, "title": f["concept"], "aliases": [f["concept"]], "kind": outcome,
"merge_of": ref, "origin": {"file": f["anchor"]["file"], "line": f["anchor"]["start"]},
"tests": f.get("tests")})
acc["fragments"].append({"pack": n, "position": f["position"], "file": f["anchor"]["file"],
"start": f["anchor"]["start"], "end": f["anchor"]["end"], "section": f["section"],
"text": f["text"].strip(), "quote": f["quote"], "quote_line": f["quote_line"],
"version": f["version"], "concept_id": cid, "outcome": outcome})
if cid not in sent:
not_sent[outcome] += 1
accepted += 1
_write_json(index_path, index)
_write_json(acc_path, acc)
log = _pack_log(work)
rec = _record(log, n)
rec.update(proposed=len(fragments), accepted=accepted, rejected=dict(sorted(rejected.items())),
not_sent={"total": sum(not_sent.values()), "by_outcome": dict(sorted(not_sent.items()))})
_write_json(work / "K-work" / "packs.json", log)
print(f"pack {n}: accepted {accepted} of {len(fragments)}")
print(f" accepted with a concept not among the index lines sent: {sum(not_sent.values())}")
for reason, count in sorted(rejected.items()):
print(f" rejected {reason}: {count}")
return 0
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
s = sub.add_parser("skeleton")
s.add_argument("--kilde", required=True)
s.add_argument("--out", required=True)
for name in ("pack", "accept"):
c = sub.add_parser(name)
c.add_argument("n", type=int)
c.add_argument("--blocks", required=True)
c.add_argument("--kilde-root", required=True)
c.add_argument("--work", required=True)
c.add_argument("--size", type=int, default=5)
if name == "pack":
c.add_argument("--index", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "skeleton":
out = Path(args.out)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
index = skeleton(Path(args.kilde))
_write_json(out, index)
print(f"skeleton: {len(index['entries'])} entries -> {out}")
return 0
if args.command == "pack":
return cmd_pack(args)
return cmd_accept(args)
except (RunError, OSError) as exc:
print(f"kpack cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

361
evals/p1/measure.py Normal file
View file

@ -0,0 +1,361 @@
"""The trial's measures without a reader (TRIAL, P1): audit record, coverage, structure. It calls no model.
audit --bundle B --root R --variant V --out FILE
Runs the auditor in process and records, per variant, the exit status, findings per check, uncited
sentences by cause (fence line, table row, link lead-in, prose), "yaml form" findings (a quote that YAML
loaded as another type, a timestamp in the wrong form) and criterion 1 (N of N, claims.py's count).
coverage --bundle B --root R --blocks F --built N [--pack-size K] --out FILE
Blocks of the first N sources in the order with at least one body-cited source entry whose locator line
lies inside the block (spans are not used), X of N blocks; per pack position with --pack-size.
structure --bundle B --root R --blocks F [--skeleton I] --out FILE
Pages, concepts per source (the earliest source in the order among a page's body-cited sources),
skeleton entries without a page, duplicates (same name, alias or title Jaccard >= 0.6), merge
candidates, islands, sources per page, version numbers in running text, concept tests 2 and 4,
the share of quotes on a structural line and of code-supplied quotes (version sources left out), and the
precision of each body-cited entry (line, block, section, document).
--root is the folder that contains kilde/. Before reporting, structure and coverage check that at least one quote
stands at its locator under --root, because a wrong root makes every quote unreadable. Exit 0 = written, 2 = could
not run (an existing output, or a variant already in the audit record, is never overwritten).
"""
from __future__ import annotations
import argparse
import json
import sys
from collections import Counter
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from okfwiki import claims as claims_mod
from okfwiki.audit import (
FOOTNOTE_REF,
LINK,
LINK_ONLY,
Document,
audit,
body_links,
is_bundle_link,
load_bundle,
mask_code,
prose_blocks,
resolve,
sentences,
)
from okfwiki.view import is_version_heading, related, versions
from p1.assemble import structural
from p1.kpack import MERGE_JACCARD, norm
from p1.split import FENCE, is_meta
STRUCTURE_SCHEMA = "okf-wiki/p1-structure/1"
RESULT_SCHEMA = "okf-wiki/p1-result/1"
class RunError(Exception):
"""The measure cannot run as asked; nothing was written."""
@dataclass
class Coverage:
covered: int
total: int
per_position: dict[int, tuple[int, int, int]] = field(default_factory=dict)
def _entries(doc: Document) -> dict[str, dict[str, Any]]:
srcs = (doc.frontmatter or {}).get("sources")
return {str(s["id"]): s for s in srcs if isinstance(s, dict) and "id" in s} if isinstance(srcs, list) else {}
def cited(doc: Document) -> list[dict[str, Any]]:
"""The source entries a body footnote uses, in label order of first use."""
entries = _entries(doc)
labels: list[str] = []
for block in prose_blocks(doc):
for label in FOOTNOTE_REF.findall(mask_code(block.text)):
if label not in labels:
labels.append(label)
return [entries[x] for x in labels if x in entries]
def _file(s: dict[str, Any]) -> str:
return Path(str(s.get("resource", ""))).name
def _line(s: dict[str, Any]) -> int | None:
loc = s.get("x-locator")
return loc["line"] if isinstance(loc, dict) and type(loc.get("line")) is int else None
class _Lines:
def __init__(self, root: Path) -> None:
self.root = root
self.cache: dict[str, list[str] | None] = {}
def get(self, name: str) -> list[str] | None:
if name not in self.cache:
p = self.root / "kilde" / name
self.cache[name] = p.read_text(encoding="utf-8").split("\n") if p.is_file() else None
return self.cache[name]
def _check_root(concepts: list[Document], root: Path) -> None:
lines = _Lines(root)
seen = 0
for d in concepts:
for s in _entries(d).values():
n, quote, text = _line(s), s.get("x-quote"), lines.get(_file(s))
seen += 1
if n and isinstance(quote, str) and text and n <= len(text) and quote in text[n - 1]:
return
if seen:
raise RunError(f"no quote stands at its locator under {root}; is it the folder that contains kilde/?")
def _block_of(blocks: list[dict[str, Any]], file: str, line: int) -> dict[str, Any] | None:
return next((b for b in blocks if b["file"] == file and b["start"] <= line <= b["end"]), None)
def coverage(bundle: Path, root: Path, blocks_doc: dict[str, Any], built: list[str],
pack_size: int | None = None) -> Coverage:
concepts = load_bundle(bundle, root).concepts
_check_root(concepts, root)
blocks = [b for b in blocks_doc["blocks"] if b["file"] in built]
hit: set[tuple[str, int]] = set()
for d in concepts:
for s in cited(d):
n = _line(s)
b = _block_of(blocks, _file(s), n) if n else None
if b is not None:
hit.add((b["file"], b["start"]))
result = Coverage(len(hit), len(blocks))
if pack_size:
names = [str(x) for x in blocks_doc["order"] if x in built]
for pos in range(1, pack_size + 1):
files = {n for i, n in enumerate(names) if i % pack_size == pos - 1}
if files:
mine = [b for b in blocks if b["file"] in files]
result.per_position[pos] = (sum((b["file"], b["start"]) in hit for b in mine), len(mine), len(files))
return result
def _jaccard(a: str, b: str) -> float:
x, y = set(a.split()), set(b.split())
return len(x & y) / len(x | y) if x | y else 0.0
def duplicates(concepts: list[Document]) -> list[tuple[str, str, str]]:
info = []
for d in concepts:
fm = d.frontmatter or {}
aliases = fm.get("x-aliases")
info.append((d.rel, norm(str(fm.get("title", ""))),
{norm(str(a)) for a in aliases} if isinstance(aliases, list) else set()))
pairs: list[tuple[str, str, str]] = []
for i, (a, na, aa) in enumerate(info):
for b, nb, ab in info[i + 1:]:
if na and na == nb:
pairs.append((a, b, "norm"))
elif na in ab or nb in aa:
pairs.append((a, b, "alias"))
elif _jaccard(na, nb) >= MERGE_JACCARD:
pairs.append((a, b, "jaccard"))
return pairs
def _kind(d: Document) -> str:
fm = d.frontmatter or {}
if fm.get("x-merge-candidate"):
return "merge-candidate"
return {"Document map": "map", "Register entry": "register"}.get(str(fm.get("type", "")), "concept")
def _precision(s: dict[str, Any], blocks: list[dict[str, Any]]) -> str:
span = s.get("x-span")
n = _line(s) or 0
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (n, n)
if a == z:
return "line"
inside = [b for b in blocks if b["file"] == _file(s) and b["end"] >= a and b["start"] <= z]
if len(inside) == 1 and inside[0]["start"] <= a and z <= inside[0]["end"]:
return "block"
if inside and len({b["section"] for b in inside}) == 1:
return "section"
return "document"
def structure(bundle: Path, root: Path, blocks_doc: dict[str, Any], order: list[str],
skeleton: dict[str, Any] | None = None) -> dict[str, Any]:
concepts = load_bundle(bundle, root).concepts
_check_root(concepts, root)
lines = _Lines(root)
blocks = list(blocks_doc["blocks"])
rank = {n: i for i, n in enumerate(order)}
per_source: Counter[str] = Counter()
incoming: Counter[str] = Counter()
rels = {d.rel for d in concepts}
for d in concepts:
for _line_no, target in body_links(d):
other = resolve(d.rel, target) if is_bundle_link(target) else None
if other in rels and other != d.rel:
incoming[str(other)] += 1
quotes: Counter[str] = Counter()
precision: Counter[str] = Counter()
docs_per_page: list[int] = []
test2 = test4 = 0
for d in concepts:
cs = cited(d)
files = [_file(s) for s in cs if _file(s) in rank]
if files:
per_source[min(files, key=lambda f: rank[f])] += 1
docs_per_page.append(len(set(_file(s) for s in cs)))
title = str((d.frontmatter or {}).get("title", ""))
test2 += not is_meta(title)
test4 += incoming[d.rel] >= 2 or len(set(files)) >= 2
for s in cs:
if is_version_heading(s):
continue
quotes["entries"] += 1
quotes["code"] += s.get("x-quote-by") == "code"
text, n = lines.get(_file(s)), _line(s)
quotes["structural"] += bool(text and n and n <= len(text) and structural(text[n - 1]))
precision[_precision(s, blocks)] += 1
rel_map = related(concepts)
kinds = {d.rel: _kind(d) for d in concepts}
islands: dict[str, Any] = {"total": sum(not v for v in rel_map.values())}
for k in sorted(set(kinds.values())):
islands[k] = sum(not rel_map[r] for r, kk in kinds.items() if kk == k)
pages = {d.rel for d in concepts}
unused = None
if skeleton is not None:
unused = sum(e.get("kind") == "skeleton" and f"{e['id']}.md" not in pages for e in skeleton["entries"])
tests13 = None
if skeleton is not None:
new_ids = {e["id"] for e in skeleton["entries"] if e.get("kind") == "new" and e.get("tests")}
tests13 = sum(f"{i}.md" in pages for i in new_ids)
v = versions(bundle, root)
dups = duplicates(concepts)
return {
"schema": STRUCTURE_SCHEMA,
"pages": len(concepts),
"page_kinds": dict(Counter(kinds.values())),
"concepts_per_source": {n: per_source[n] for n in order if per_source[n]},
"skeleton_unused": unused,
"duplicates": {"pairs": len(dups), "by_rule": dict(Counter(r for _, _, r in dups)),
"list": [list(p) for p in dups]},
"merge_candidates": sum(k == "merge-candidate" for k in kinds.values()),
"islands": islands,
"documents_per_page": {"mean": round(sum(docs_per_page) / len(docs_per_page), 2) if docs_per_page else 0,
"max": max(docs_per_page, default=0)},
"version_numbers_in_text": {"flagged": len(v.flagged), "claims": v.total},
"concept_tests": {"test2": test2, "test4": test4, "tests1and3_self_declared": tests13},
"quotes": {"entries": quotes["entries"], "structural": quotes["structural"], "code": quotes["code"]},
"precision": {k: precision[k] for k in ("line", "block", "section", "document")},
}
def _uncited_cause(sentence: str, block_text: str) -> str:
s = sentence.strip()
if FENCE.match(block_text): # the auditor's sentence text masks code spans, so a fence is read on the block
return "fence line"
if s.startswith("|"):
return "table row"
if LINK.search(block_text) and not LINK_ONLY.match(mask_code(block_text)):
return "link lead-in"
return "prose"
def audit_record(bundle: Path, root: Path, variant: str, out: Path) -> dict[str, Any]:
record_all: dict[str, Any] = (json.loads(out.read_text(encoding="utf-8")) if out.is_file()
else {"schema": RESULT_SCHEMA, "variants": {}})
if variant in record_all["variants"]:
raise RunError(f"{out} already records variant {variant}")
b = load_bundle(bundle, root)
found = audit(bundle, root)
uncited: Counter[str] = Counter()
for d in b.concepts:
for block in prose_blocks(d):
if LINK_ONLY.match(mask_code(block.text)):
continue
for s in sentences(block.text):
if not FOOTNOTE_REF.search(s):
uncited[_uncited_cause(s, block.text)] += 1
yaml_form = 0
for d in b.concepts:
for entry in _entries(d).values():
q = entry.get("x-quote")
yaml_form += q is not None and not isinstance(q, str)
yaml_form += sum(f.check == "timestamp-form" for f in found)
listed = claims_mod.extract(bundle, root)
ok, total = claims_mod.criterion_1(listed)["total"]
rec = {"exit": 1 if found else 0, "findings": dict(sorted(Counter(f.check for f in found).items())),
"uncited": dict(sorted(uncited.items())), "yaml_form": yaml_form, "criterion_1": [ok, total]}
record_all["variants"][variant] = rec
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(record_all, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
return rec
def _write_new(path: Path, data: Any) -> None:
if path.exists():
raise RunError(f"{path} exists; refusing to overwrite")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
a = sub.add_parser("audit")
a.add_argument("--bundle", required=True)
a.add_argument("--root", required=True)
a.add_argument("--variant", required=True)
a.add_argument("--out", required=True)
c = sub.add_parser("coverage")
s = sub.add_parser("structure")
for x in (c, s):
x.add_argument("--bundle", required=True)
x.add_argument("--root", required=True)
x.add_argument("--blocks", required=True)
x.add_argument("--out", required=True)
c.add_argument("--built", type=int, required=True)
c.add_argument("--pack-size", type=int)
s.add_argument("--skeleton")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
bundle, root = Path(args.bundle), Path(args.root)
if not bundle.is_dir():
raise RunError(f"the bundle {bundle} is not a directory")
if args.command == "audit":
rec = audit_record(bundle, root, args.variant, Path(args.out))
print(f"{args.variant}: audit exit {rec['exit']}, criterion 1 {rec['criterion_1'][0]} of "
f"{rec['criterion_1'][1]}, findings {rec['findings']}")
return 0
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
order = [str(x) for x in blocks_doc["order"]]
if args.command == "coverage":
cov = coverage(bundle, root, blocks_doc, order[:args.built], args.pack_size)
_write_new(Path(args.out), {"schema": "okf-wiki/p1-coverage/1", "covered": cov.covered, "total": cov.total,
"built": args.built,
"per_position": {str(k): list(v) for k, v in cov.per_position.items()}})
print(f"coverage: {cov.covered} of {cov.total} blocks")
return 0
skeleton = json.loads(Path(args.skeleton).read_text(encoding="utf-8")) if args.skeleton else None
data = structure(bundle, root, blocks_doc, order, skeleton)
_write_new(Path(args.out), data)
print(f"structure: {data['pages']} pages, {data['duplicates']['pairs']} duplicate pairs, "
f"{data['islands']['total']} islands")
return 0
except (RunError, OSError, json.JSONDecodeError, KeyError) as exc:
print(f"measure cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

185
evals/p1/sidebyside.py Normal file
View file

@ -0,0 +1,185 @@
"""The side-by-side page (TRIAL, P1): the same concepts in every build, every source link checked. It calls no model.
render --begreper FILE --col LABEL=DIR ... [--col LABEL=-] --kilde-root DIR --out FILE
One self-contained HTML page: one row per concept in begreper.txt (name, a tab, aliases joined by ;), one
column per build. A cell shows the pages whose normalised title or alias equals the concept's (for the L4
column: whose normalised title or description contains it), each with its body and its sources
(file › line or span). An empty cell says why: "ikke bygget" (a build given as -, stopped), "ikke i
registeret" (D has no register page for it), "ingen side" (built, no page).
check --html FILE --kilde-root DIR
A source link resolves when its file exists under kilde/ and its line or span lies inside the file.
Prints "resolved X of N"; exit 1 when X < N.
Exit 0 = done, 1 = an unresolved link (check), 2 = could not run.
"""
from __future__ import annotations
import argparse
import os
import re
import sys
from pathlib import Path
from typing import Any
from okfwiki.audit import load_bundle
from okfwiki.view import esc
from p1.kpack import norm
from p1.measure import cited
Concept = tuple[str, tuple[str, ...]]
SRC = re.compile(r'<a class="src" href="[^"]*" data-file="([^"]+)" data-start="(\d+)" data-end="(\d+)">')
STYLE = """
:root { --bg: #fdfdfb; --fg: #1d1d1b; --muted: #6b6b66; --line: #ddd9cf; --cell: #ffffff; }
@media (prefers-color-scheme: dark) { :root { --bg: #1b1b1a; --fg: #ecebe6; --muted: #a3a29b; --line: #3a3934;
--cell: #242422; } }
body { background: var(--bg); color: var(--fg); font: 15px/1.5 system-ui, sans-serif; margin: 16px; }
table { border-collapse: collapse; width: 100%; table-layout: fixed; }
th, td { border: 1px solid var(--line); vertical-align: top; padding: 8px; background: var(--cell); }
th.concept-name { width: 10em; text-align: left; }
.empty { color: var(--muted); font-style: italic; }
.page h3 { margin: 0 0 4px; font-size: 15px; }
.sources { color: var(--muted); font-size: 13px; }
.wrap { overflow-x: auto; }
"""
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
def read_begreper(path: Path) -> list[Concept]:
out: list[Concept] = []
for line in path.read_text(encoding="utf-8").splitlines():
if line.strip():
name, _, aliases = line.partition("\t")
out.append((name.strip(), tuple(a.strip() for a in aliases.split(";") if a.strip()) or (name.strip(),)))
return out
def _l4_norm(text: str) -> str:
return f" {norm(text.replace('-', ' '))} "
def find(bundle: Path, concept: Concept, l4: bool = False) -> list[str]:
wanted = {norm(n) for n in (concept[0], *concept[1])} - {""}
found: list[str] = []
for d in load_bundle(bundle, bundle).concepts:
fm = d.frontmatter or {}
if l4:
hay = _l4_norm(str(fm.get("title", ""))) + _l4_norm(str(fm.get("description", "")))
if any(f" {norm(w.replace('-', ' '))} " in hay for w in wanted):
found.append(d.rel)
continue
aliases = fm.get("x-aliases")
names = {norm(str(fm.get("title", "")))} | ({norm(str(a)) for a in aliases} if isinstance(aliases, list)
else set())
if names & wanted:
found.append(d.rel)
return found
def _sources_html(entries: list[dict[str, Any]], kilde: Path, out_dir: Path) -> str:
items = []
for s in entries:
file = Path(str(s.get("resource", ""))).name
loc = s.get("x-locator")
line = loc.get("line") if isinstance(loc, dict) else None
span = s.get("x-span")
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (line, line)
if not isinstance(a, int) or not isinstance(z, int):
continue
href = os.path.relpath(kilde / file, out_dir).replace(os.sep, "/") + f"#L{a}"
where = f"{a}" if a == z else f"{a}–{z}"
items.append(f'<li><a class="src" href="{esc(href)}" data-file="{esc(file)}" data-start="{a}" '
f'data-end="{z}">{esc(file)} › {where}</a></li>')
return '<ul class="sources">' + "".join(items) + "</ul>"
def _cell(bundle: Path | None, label: str, concept: Concept, kilde_root: Path, out_dir: Path) -> str:
if bundle is None:
return '<td class="cell empty">ikke bygget</td>'
rels = find(bundle, concept, l4=label == "L4")
if not rels:
return f'<td class="cell empty">{"ikke i registeret" if label == "D" else "ingen side"}</td>'
docs = {d.rel: d for d in load_bundle(bundle, kilde_root).concepts}
parts = []
for rel in rels:
d = docs[rel]
fm = d.frontmatter or {}
srcs = fm.get("sources")
entries = cited(d) or ([s for s in srcs if isinstance(s, dict)] if isinstance(srcs, list) else [])
paras = [p.strip() for p in re.split(r"\n\s*\n", d.body) if p.strip() and not p.strip().startswith("# ")]
body = "".join(f"<p>{esc(p)}</p>" for p in paras if not p.startswith("[^"))
parts.append(f'<div class="page"><h3>{esc(str(fm.get("title", rel)))} <small>({esc(rel)})</small></h3>'
f"{body}{_sources_html(entries, kilde_root / 'kilde', out_dir)}</div>")
return '<td class="cell">' + "".join(parts) + "</td>"
def render(begreper: list[Concept], columns: list[tuple[str, Path | None]], kilde_root: Path, out: Path) -> None:
out_dir = out.parent
head = "".join(f"<th>{esc(label)}</th>" for label, _ in columns)
rows = []
for concept in begreper:
cells = "".join(_cell(b, label, concept, kilde_root, out_dir) for label, b in columns)
rows.append(f'<tr class="concept"><th class="concept-name">{esc(concept[0])}</th>{cells}</tr>')
html = (f'<!doctype html><html lang="no"><head><meta charset="utf-8">'
f'<meta name="viewport" content="width=device-width, initial-scale=1"><title>P1 side om side</title>'
f"<style>{STYLE}</style></head><body><h1>P1: tre begreper, side om side</h1>"
f'<div class="wrap"><table><tr><th class="concept-name">Begrep</th>{head}</tr>{"".join(rows)}'
f"</table></div></body></html>\n")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(html, encoding="utf-8")
def check(html_path: Path, kilde_root: Path) -> tuple[int, int]:
html = html_path.read_text(encoding="utf-8")
resolved = total = 0
sizes: dict[str, int | None] = {}
for file, a, z in SRC.findall(html):
total += 1
if file not in sizes:
p = kilde_root / "kilde" / file
sizes[file] = len(p.read_text(encoding="utf-8").splitlines()) if p.is_file() else None
n = sizes[file]
resolved += n is not None and 1 <= int(a) <= int(z) <= n
return resolved, total
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
r = sub.add_parser("render")
r.add_argument("--begreper", required=True)
r.add_argument("--col", action="append", required=True, help="LABEL=DIR, or LABEL=- for a stopped build")
r.add_argument("--kilde-root", required=True)
r.add_argument("--out", required=True)
c = sub.add_parser("check")
c.add_argument("--html", required=True)
c.add_argument("--kilde-root", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "render":
columns: list[tuple[str, Path | None]] = []
for col in args.col:
label, _, path = col.partition("=")
if not label or not path:
raise RunError(f"--col {col!r} is not LABEL=DIR")
columns.append((label, None if path == "-" else Path(path)))
render(read_begreper(Path(args.begreper)), columns, Path(args.kilde_root), Path(args.out))
print(f"side by side: {len(columns)} columns -> {args.out}")
return 0
resolved, total = check(Path(args.html), Path(args.kilde_root))
print(f"resolved {resolved} of {total}")
return 0 if total and resolved == total else 1
except (RunError, OSError) as exc:
print(f"sidebyside cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())

328
evals/p1/split.py Normal file
View file

@ -0,0 +1,328 @@
"""Split the trial's sources into a fixed order and into blocks with line spans (TRIAL, P1). It calls no model.
order --manifest M --l4 L4DIR --out FILE
The 19 names of the cut (gates/wholedoc.manifest.json cut.names) in the trial's fixed order: docs pages by
name, then release notes by version. Refuses when the names differ from the files the L4 cut cites.
blocks --kilde DIR --order FILE --out FILE
Every block of every source in the order: a paragraph, one list item with its continuation lines, one
table or one fenced code block, with its 1-based inclusive line span, its section (the heading path) and,
for a release note, its version. Heading lines are section markers, not blocks.
total --kilde DIR
The number of blocks in every file under DIR (the denominator of the 599-source projection).
pick --blocks FILE --kilde DIR [--focus NAME] --out FILE
The three concepts of the side-by-side page, picked from the source text before any build: the focus page's
heading with the most blocks, the term in the most release notes, and the term in at least two docs pages and
two release notes with the highest document count; a document counts for a term only where the term stands
as a term (backticked or a capitalised phrase), never as a word in prose. One line per concept: name, a tab,
aliases joined by ;.
Exit 0 = written, 2 = could not run (an existing output is never overwritten).
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any
from okfwiki.audit import HEADING, LIST_ITEM, read_document
SCHEMA = "okf-wiki/p1-blocks/1"
RELEASE = re.compile(r"^release-v(\d+(?:\.\d+)*)\.md$")
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
FENCE = re.compile(r"^\s*(```|~~~)")
# Concept test 2 of the brief (Google's reference agent), plus the generic section names of docs pages.
META_STOP = frozenset({
"overview", "introduction", "getting started", "quickstart", "tutorial", "walkthrough", "release notes",
"changelog", "roadmap", "faq", "prerequisites", "next steps", "before you begin", "troubleshooting", "see also",
"related resources", "learn more",
})
CODE_TERM = re.compile(r"`([^`\n]{3,40})`")
CAPITAL_PHRASE = re.compile(r"\b[A-Z][\w-]*(?:[ \t]+[A-Z][\w-]*){1,3}\b")
SENTENCE_START = re.compile(r"(?:^|[.!?:]\s+|^\s*(?:[-*+]|\d+[.)])\s+)$")
class RunError(Exception):
"""The command cannot run as asked; nothing was written."""
@dataclass(frozen=True)
class SBlock:
file: str
start: int
end: int
kind: str # paragraph | item | table | code
section: str
version: str | None
@dataclass(frozen=True)
class Heading:
file: str
line: int
level: int
text: str
@dataclass(frozen=True)
class Picked:
name: str
aliases: tuple[str, ...]
def is_meta(title: str) -> bool:
return title.strip().lower() in META_STOP
def version_key(name: str) -> tuple[int, ...]:
m = RELEASE.match(name)
if m is None:
raise RunError(f"{name} is not a release note name")
return tuple(int(p) for p in m.group(1).split("."))
def order(names: list[str]) -> list[str]:
docs = sorted(n for n in names if n.startswith("docs-"))
releases = [n for n in names if n.startswith("release-")]
other = [n for n in names if n not in docs and n not in releases]
if other:
raise RunError(f"names that are neither docs pages nor release notes: {other}")
return docs + sorted(releases, key=version_key)
def headings_of(lines: list[str]) -> list[tuple[int, int, str]]:
"""(line, level, text) of every heading outside fenced code."""
out: list[tuple[int, int, str]] = []
fence: str | None = None
for n, raw in enumerate(lines, 1):
f = FENCE.match(raw)
if f:
fence = None if fence == f.group(1) else (fence or f.group(1))
continue
if fence is None:
m = HEADING.match(raw)
if m:
out.append((n, len(m.group(1)), (m.group(2) or "").strip()))
return out
def blocks(path: Path) -> list[SBlock]:
lines = path.read_text(encoding="utf-8").split("\n")
name = path.name
version: str | None = None
if name.startswith("release-"):
for _n, _level, text in headings_of(lines):
m = VERSION_HEADING.match(text)
if m:
version = m.group(1)
break
out: list[SBlock] = []
stack: list[tuple[int, str]] = []
kind: str | None = None
start = 0
fence: str | None = None
def close(end: int) -> None:
nonlocal kind
if kind is not None:
out.append(SBlock(name, start, end, kind, " › ".join(t for _, t in stack), version))
kind = None
for n, raw in enumerate(lines, 1):
f = FENCE.match(raw)
if fence is not None:
if f and f.group(1) == fence:
fence = None
close(n)
continue
if f:
close(n - 1)
fence, kind, start = f.group(1), "code", n
continue
if not raw.strip():
close(n - 1)
continue
m = HEADING.match(raw)
if m:
close(n - 1)
level = len(m.group(1))
stack = [(lv, t) for lv, t in stack if lv < level] + [(level, (m.group(2) or "").strip())]
continue
if raw.lstrip().startswith("|"):
if kind != "table":
close(n - 1)
kind, start = "table", n
continue
if LIST_ITEM.match(raw):
close(n - 1)
kind, start = "item", n
continue
if kind in (None, "table"):
close(n - 1)
kind, start = "paragraph", n
if fence is not None:
close(len(lines))
else:
last = len(lines)
while last > 0 and not lines[last - 1].strip():
last -= 1
close(last)
return out
def docs_headings(kilde: Path) -> list[Heading]:
out: list[Heading] = []
for path in sorted(kilde.glob("docs-*.md")):
for n, level, text in headings_of(path.read_text(encoding="utf-8").split("\n")):
if level <= 3 and text:
out.append(Heading(path.name, n, level, text))
return out
def blocks_document(kilde: Path, names: list[str]) -> dict[str, Any]:
found: list[dict[str, Any]] = []
for name in names:
path = kilde / name
if not path.is_file():
raise RunError(f"{path} is not a file")
found.extend(asdict(b) for b in blocks(path))
return {"schema": SCHEMA, "order": names, "blocks": found}
def blocks_total(kilde: Path) -> int:
return sum(len(blocks(p)) for p in sorted(kilde.glob("*.md")))
def terms(text: str) -> list[str]:
"""Backticked spans of 3-40 characters and capitalised phrases of 2-4 words, a sentence-initial word dropped."""
found: list[str] = [t for t in (m.group(1).strip() for m in CODE_TERM.finditer(text)) if 3 <= len(t) <= 40]
masked = CODE_TERM.sub(lambda m: " " * len(m.group(0)), text)
for raw in masked.split("\n"):
for m in CAPITAL_PHRASE.finditer(raw):
words = m.group(0).split()
if SENTENCE_START.search(raw[:m.start()]):
words = words[1:]
if len(words) >= 2:
found.append(" ".join(words))
return list(dict.fromkeys(t for t in found if t))
def occurs(term: str, text: str) -> bool:
return re.search(r"(?<!\w)" + re.escape(term) + r"(?!\w)", text, re.IGNORECASE) is not None
def pick(blocks_doc: dict[str, Any], kilde: Path, focus: str = "docs-en-permission-modes.md") -> list[Picked]:
names = [str(n) for n in blocks_doc["order"]]
if focus not in names:
raise RunError(f"the focus page {focus} is not in the order")
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
per_section: dict[str, int] = {}
for b in blocks_doc["blocks"]:
if b["file"] == focus and b["section"].count(" › ") >= 1:
per_section[b["section"]] = per_section.get(b["section"], 0) + 1
candidates = [(count, section.split(" › ")[-1]) for section, count in per_section.items()
if not is_meta(section.split(" › ")[-1])]
if not candidates:
raise RunError(f"{focus} has no heading below its title that passes test 2")
first = sorted(candidates, key=lambda c: (-c[0], c[1]))[0][1]
pool = list(dict.fromkeys(t for n in names for t in terms(texts[n])))
docs = [n for n in names if n.startswith("docs-")]
releases = [n for n in names if n.startswith("release-")]
as_terms = {n: set(terms(texts[n])) for n in names}
def count(term: str, files: list[str]) -> int:
return sum(term in as_terms[f] for f in files) # where it stands as a term, not as a word in prose
taken = {first.lower()}
second_pool = [t for t in pool if t.lower() not in taken]
if not second_pool:
raise RunError("no term for the second concept")
second = sorted(second_pool, key=lambda t: (-count(t, releases), t))[0]
taken.add(second.lower())
third_pool = [t for t in pool if t.lower() not in taken and count(t, docs) >= 2 and count(t, releases) >= 2]
if not third_pool:
raise RunError("no term in two docs pages and two release notes for the third concept")
third = sorted(third_pool, key=lambda t: (-count(t, names), t))[0]
return [Picked(first, (first,)), Picked(second, (second,)), Picked(third, (third,))]
def _write_new(path: Path, text: str) -> None:
if path.exists():
raise RunError(f"{path} exists; refusing to overwrite")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
def l4_cited(l4: Path) -> set[str]:
cited: set[str] = set()
for page in sorted(l4.glob("*.md")):
fm = read_document(page, page.name).frontmatter or {}
srcs = fm.get("sources")
for s in srcs if isinstance(srcs, list) else []:
res = s.get("resource") if isinstance(s, dict) else None
if isinstance(res, str) and res:
cited.add(Path(res).name)
return cited
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
o = sub.add_parser("order")
o.add_argument("--manifest", required=True)
o.add_argument("--l4", required=True)
o.add_argument("--out", required=True)
b = sub.add_parser("blocks")
b.add_argument("--kilde", required=True)
b.add_argument("--order", required=True)
b.add_argument("--out", required=True)
t = sub.add_parser("total")
t.add_argument("--kilde", required=True)
k = sub.add_parser("pick")
k.add_argument("--blocks", required=True)
k.add_argument("--kilde", required=True)
k.add_argument("--focus", default="docs-en-permission-modes.md")
k.add_argument("--out", required=True)
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "order":
manifest = json.loads(Path(args.manifest).read_text(encoding="utf-8"))
names = manifest.get("cut", {}).get("names")
if not isinstance(names, list) or not names:
raise RunError(f"{args.manifest} records no cut names")
cited = l4_cited(Path(args.l4))
if set(map(str, names)) != cited:
raise RunError(f"the manifest's {len(names)} names differ from the {len(cited)} files the L4 cut cites: "
f"only in manifest {sorted(set(names) - cited)}, only in L4 {sorted(cited - set(names))}")
fixed = order([str(n) for n in names])
_write_new(Path(args.out), "\n".join(fixed) + "\n")
print(f"order: {len(fixed)} sources -> {args.out}")
elif args.command == "blocks":
names = Path(args.order).read_text(encoding="utf-8").split()
doc = blocks_document(Path(args.kilde), names)
_write_new(Path(args.out), json.dumps(doc, ensure_ascii=False, indent=1) + "\n")
print(f"blocks: {len(doc['blocks'])} in {len(names)} sources -> {args.out}")
elif args.command == "total":
print(f"blocks in every file of {args.kilde}: {blocks_total(Path(args.kilde))}")
else:
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
picked = pick(doc, Path(args.kilde), args.focus)
_write_new(Path(args.out), "".join(f"{p.name}\t{';'.join(p.aliases)}\n" for p in picked))
print(f"pick: {len(picked)} concepts -> {args.out}")
except (RunError, OSError, json.JSONDecodeError) as exc:
print(f"split cannot run: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())