okf-wiki 1.1.1
First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
commit
0fd86a23a3
227 changed files with 96077 additions and 0 deletions
125
evals/p1/README.md
Normal file
125
evals/p1/README.md
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
# P1 method trial (TRIAL, branch `p1-metodeprove`, not v1 code)
|
||||
|
||||
The P1 method trial builds the 19 sources of the cut (the sources behind d01 d02 d03 d04 d37 d46 d53) three ways, in one
|
||||
fixed order, and measures cost and quality side by side, so the operator can choose the build method for v1 by seeing
|
||||
small real examples (PLAN.md § Rammer, order `20261002T094056Z-495685981`, plan v3).
|
||||
|
||||
- **K**: the fragment compiler. Code splits the sources into blocks, a narrow trial agent (`p1-compiler`) writes short
|
||||
fragments per pack of five sources, and code resolves concept identity and assembles the pages.
|
||||
- **K+**: a copy of K plus one consolidation pass (`p1-consolidator`) that rewrites pages with three or more fragments
|
||||
into flowing text, keeping every footnote label.
|
||||
- **D**: no model. A generated map of each source and a code-built concept register.
|
||||
|
||||
No module here calls a model. The model work is done in the operator's session by the two trial agent types under
|
||||
`.claude/agents/`. Every output lives outside the repository, under `<wiki>/prove-p1/` (CLAUDE.md rule 3); this README
|
||||
records numbers only.
|
||||
|
||||
## Modules
|
||||
|
||||
Run each as `PYTHONPATH=evals:. python3 -m p1.<module> ...`; each module's docstring is its usage.
|
||||
|
||||
| Module | What it does |
|
||||
|---|---|
|
||||
| `split.py` | the fixed source order, blocks with line spans, the block total, the three side-by-side concepts |
|
||||
| `dmap.py` | D: map pages and register pages |
|
||||
| `kpack.py` | K: the skeleton from every docs page, pack input parts, fragment validation, concept identity |
|
||||
| `assemble.py` | K's pages from accepted fragments; K+'s copy, tasks and apply |
|
||||
| `measure.py` | the audit record, coverage of source blocks, structure counts |
|
||||
| `cost.py` | tokens from the run sessions' transcripts, tokens per source, projection, the stop rule |
|
||||
| `sidebyside.py` | the side-by-side page and its link check |
|
||||
|
||||
Output files under `prove-p1/`: `rekkefolge.txt`, `blocks.json`, `begreper.txt`, `okter.txt`, `D/`, `K/`, `K+/`,
|
||||
`resultat.json`, `struktur-<V>.json`, `dekning-<V>.json`, `kost.json`, `side-om-side.html`, `rapport.md`.
|
||||
|
||||
Tests: `python3 -m unittest discover -s evals/tests -p 'test_p1_*.py'`. Checks: `bash tests/p1_repo.sh` (repository)
|
||||
and `bash tests/p1_trial.sh` (the trial's outputs; red until the runs are done).
|
||||
|
||||
## Results
|
||||
|
||||
Numbers only, filled in by the run steps (plan steps 8-11).
|
||||
|
||||
### Run 0 (step 8, 02.10.2026)
|
||||
|
||||
- Order: 19 sources (3 docs pages, then 16 release notes by version), equal to the files cited by the L4 cut.
|
||||
- Blocks: 1221 in the 19 sources (docs pages 599, release notes 622; pack 1 = sources 1-5 = 602 blocks); 33727 blocks
|
||||
in all 599 files of `kilde/` (the D12 projection base).
|
||||
- Blocks per source in order: 59, 308, 232, 1, 2, 22, 35, 76, 21, 25, 57, 35, 34, 52, 33, 22, 46, 53, 108.
|
||||
- Load test (the empty baseline run, not counted toward any cap): both trial types spawned and wrote under `prove-p1/`,
|
||||
so the agents' work directory is `prove-p1/` (no `runtime/p1` fallback). `p1-compiler`: startup 18604, 2 turns,
|
||||
39142 tokens in all; `p1-consolidator`: startup 18167, 2 turns, 38270 tokens in all.
|
||||
- D: 254 pages (19 map, 235 register), 0 model tokens. Auditor exit 0, 0 findings. Criterion 1: 0 of 0 (bodies are
|
||||
headings and link lists only, D8, by construction). Coverage 0 of 1221 blocks: by construction, because D cites no
|
||||
source in a body footnote (D21 counts body-cited entries only; D's locators live in `sources[]`). Duplicate pairs 9
|
||||
(5 by name, 4 by title Jaccard), merge candidates 0, islands 0, concept test 2 held by 254 pages, test 4 by 19.
|
||||
|
||||
### Run K (step 9, 02.10.2026)
|
||||
|
||||
- Skeleton: 3493 index entries from the titles and level 2-3 headings of all 198 docs pages (0 tokens).
|
||||
- The cost harvester was fixed twice during the run, each with a failing test first (`77b16ab`, `0a50286`): a long
|
||||
streamed message keeps only its first usage record in the transcript (output 2, no stop reason), so output was
|
||||
undercounted (373 recorded for pack 1's 336 fragments). Output is now derived from the next message's input growth,
|
||||
less the injected text counted high (2.5 characters per token), so every K output below is a lower bound. Checked on
|
||||
the load test's known short call, which keeps its recorded value. The recorded value stays beside it in `kost.json`.
|
||||
- K: per pack (blocks / index lines sent of 3493 / input parts / fragments accepted of proposed / rejected / accepted
|
||||
with a concept not among the lines sent / turns / tokens):
|
||||
- pack 1: 602 / 230 / 4 / 336 of 336 / 0 / 0 / 3 / 372996
|
||||
- pack 2: 179 / 117 / 1 / 87 of 87 / 0 / 0 / 3 / 125716
|
||||
- pack 3: 211 / 125 / 1 / 108 of 108 / 0 / 0 / 3 / 141320
|
||||
- pack 4: 229 / 127 / 1 / 119 of 119 / 0 / 0 / 3 / 153494
|
||||
- No fragment was rejected for any reason (incl. `release-note-new` 0); no new concept was proposed, so all 650
|
||||
accepted fragments enrich skeleton entries. Merge candidates 0.
|
||||
- Stop line after pack 1: projected per block 756525 (naive per source 1417385), cap 1000000: continue.
|
||||
- K total: 793526 tokens in 4 runs and 12 turns (input 24, cache read 228534, cache write 429550, output 135418);
|
||||
startup 74596 (4 x about 18.6 k). Tokens per source, mean of sources 1-5: 74599; of sources 15-19: 35119.
|
||||
- Projected to all 599 sources (33727 blocks): fit 52842866 (4 points, growth term b = 0.0554, not validated);
|
||||
flat 21919125.
|
||||
- Auditor exit 0, 0 findings. Criterion 1: 664 of 664. Quotes: 650 entries, 160 supplied by code, 0 on a structural
|
||||
line. Precision of body-cited entries: line 591, block 59, section 0, document 0.
|
||||
- Coverage: 623 of 1221 blocks. Per position in the pack (covered of blocks, n packs): 1: 85 of 160 (4); 2: 241 of 424
|
||||
(4); 3: 181 of 395 (4); 4: 83 of 182 (4); 5: 33 of 60 (3). A direction signal, n = 3-4.
|
||||
- Structure: 212 concept pages, 3281 skeleton entries unused. Concepts per source in order: 11, 61, 31, 1, 0, 5, 13,
|
||||
15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11. Duplicate pairs 3 (all title Jaccard), islands 63, documents per page mean
|
||||
1.65 (max 8), version numbers in running text 5 of 664 claims, concept test 2 held by 212, test 4 by 73.
|
||||
- Seen in the compiler's own reports: in one pack it wrote numbers as words to avoid the quote rule (D5 requires a
|
||||
quote when the text holds a digit); the compilers' own fragment counts differed from the files (the files count).
|
||||
|
||||
### Run K+ (step 10, 02.10.2026)
|
||||
|
||||
- K+: a copy of K; 76 pages carry 3 or more fragments, which gives 10 consolidation runs of up to 8 pages.
|
||||
- Run 1: 8 pages, applied 8, refused 0. 109651 tokens (input 8, cache read 72768, cache write 29909, output 6966, a lower
|
||||
bound), 4 turns, startup 18176. The consolidator took one more turn than its instructions allow (it read its
|
||||
output back).
|
||||
- Stop line after run 1: 109651 x 76 / 8 = 1041685 > cap 1000000: STOP. Runs 2-10 were not run; K+ stands as K with
|
||||
8 of 76 pages consolidated.
|
||||
- K+ total = K + consolidation = 793526 + 109651 = 903177 tokens.
|
||||
- Auditor exit 0, 0 findings. Criterion 1: 662 of 662 (2 fewer claims than K: sentences joined). Coverage 623 of 1221,
|
||||
precision and quotes as K (line 591, block 59; 160 code quotes, 0 structural). 212 pages, 3 duplicate pairs,
|
||||
63 islands, version numbers in running text 5 of 662.
|
||||
|
||||
### Step 11: the side-by-side table (02.10.2026)
|
||||
|
||||
Corrections after run 0, each its own commit with a failing test first: the backticked-term length is checked after
|
||||
trimming (`26f521f`) and the concept pick counts a document only where the term stands as a term (`09a50b2`). The
|
||||
first pick and D built before `26f521f` are kept in `prove-p1/forkastet/`; D was rebuilt (252 pages: 19 map, 233
|
||||
register; 9 duplicate pairs; auditor exit 0) and the concepts were picked again by the same rule.
|
||||
|
||||
| variant | built to source | concepts (total / per source 1-5, 15-19 / test 2 / test 4) | skeleton unused | merge candidates | coverage | tokens per source (input / cache read / cache write / output / startup) | 1-5 | 15-19 | turns / subagents | projected 599 (fit / flat) | criterion 1 | fragments accepted / code quotes / structural quotes | precision (line / block / section / document) | pages / islands / duplicates |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| K | 19 of 19 | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | 1 / 12028 / 22608 / 7127 / 3926 | 74599 | 35119 | 12 / 4 | 52842866 / 21919125 | 664 of 664 | 650 of 650 / 160 / 0 | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
|
||||
| K+ | 19 of 19, consolidation stopped after run 1 (8 of 76 pages) | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | consolidation 109651 for 8 pages; K+ total = K + consolidation = 903177 | - | - | 4 / 1 | not computed: stopped after 1 of 10 runs, and the pages to consolidate at 599 sources are not measured | 662 of 662 | as K | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
|
||||
| D | 19 of 19 | 0 concept pages (252 map and register) / 0, 0 by construction / 252 / 19 | - | 0 | 0 of 1221 (by construction) | 0 | 0 | 0 | 0 / 0 | 0 | 0 of 0 (by construction) | - | - | 252 / 0 / 9 |
|
||||
| L4 (structure counted today) | 19 of 19 | 151 / 33, 33 / 151 / 68 | - | 0 | 685 of 1221 | see the cost row | - | - | - | - | 1048 of 1048 | - / 0 / 41 of 832 | 832 / 0 / 0 / 0 | 151 / 54 / 2 |
|
||||
| L2b / L4 cost | measured 27-28.09, not measured again | | | | | L4 105.8 M for 19 sources (84 % cache read); L2b writing 30.93 M for 19 sources | | | | L2b about 1.6-1.8 bn writing + 8.6 bn reading | | | | |
|
||||
|
||||
- Every K and K+ output number is a lower bound (the harvester fix in run K). Tokens per source split each pack's cost
|
||||
by block share. The projection is not validated (4 points).
|
||||
- Side-by-side page: 3 concepts x 4 columns; 454 of 454 source links resolve. Main session (orchestration, apart):
|
||||
10.9 M tokens up to step 11, almost all cache read.
|
||||
- Concepts per source in order 1-19 (a page counts at the earliest source its body cites, D10): K and K+ 11, 61, 31,
|
||||
1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11; L4 2, 9, 19, 1, 2, 10, 14, 17, 5, 6, 14, 10, 7, 2, 9, 6, 3, 6, 9;
|
||||
D 0 for every source (its register pages cite nothing in a body).
|
||||
- K+ consolidation cost 13706 per page (lower bound) and is not split per source, because a page joins fragments from
|
||||
several sources.
|
||||
- Seen in the review (BLOCK, not fixed in this run): 45 of 1221 blocks are only MDX tag lines and cannot be covered
|
||||
(K 623 of 1176 without them); the index-line filter sends a line when the whole name occurs as a phrase, not per
|
||||
token as D7 is written; `cost.summarize` and the stop command have no test.
|
||||
5
evals/p1/__init__.py
Normal file
5
evals/p1/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""TRIAL, branch p1-metodeprove, not v1 code. Nothing outside evals/ imports it.
|
||||
|
||||
The P1 method trial (PLAN.md § Rammer, order 20261002T094056Z-495685981): three build methods for a concept wiki (K, K+,
|
||||
D) on the 19 sources of the cut, measured side by side for cost and quality. No module here calls a model.
|
||||
"""
|
||||
287
evals/p1/assemble.py
Normal file
287
evals/p1/assemble.py
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
"""K's assembler and K+'s apply step (TRIAL, P1): pages from accepted fragments, by code. It calls no model.
|
||||
|
||||
build --work DIR --kilde-root DIR --out DIR [--at ISO]
|
||||
One page per concept that at least one accepted fragment enriches (unused skeleton entries get none). Docs
|
||||
fragments form the body; release-note fragments go under "## History" in version order. Every sentence ends
|
||||
in a content footnote and, for a release note, a version footnote whose quote is the bare version on the
|
||||
note's heading line. A fragment without a model quote gets a code quote: the first line of its span that is
|
||||
not structural, marked x-quote-by: code. A fragment that names another concept's title links to it once.
|
||||
copy --src DIR --out DIR
|
||||
K+ starts as a copy of K.
|
||||
tasks --bundle DIR --work DIR [--min 3] [--per-run 8]
|
||||
One task file per consolidation run (pages with at least --min fragments, --per-run pages each) in
|
||||
<work>/Kplus-oppgaver/, output path <work>/Kplus-bodies/run-N.json.
|
||||
apply --bundle DIR --bodies FILE
|
||||
Replaces a page body only when the new body keeps every old footnote label and adds none; prints
|
||||
"applied X, refused Y".
|
||||
|
||||
--kilde-root is the folder that contains kilde/. Exit 0 = done, 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import FOOTNOTE_DEF, FOOTNOTE_REF, HEADING, read_document, sentence_spans
|
||||
from p1.dmap import frontmatter
|
||||
from p1.split import FENCE, headings_of, version_key
|
||||
|
||||
GENERATOR = "okf-wiki/p1-assemble"
|
||||
QUOTE_MAX = 200
|
||||
HTML_TAG_ONLY = re.compile(r"^\s*</?[A-Za-z][^<>]*/?>\s*$")
|
||||
TABLE_RULE = re.compile(r"^\s*\|?\s*:?-+:?\s*(\|\s*:?-+:?\s*)*\|?\s*$")
|
||||
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Run:
|
||||
task: Path
|
||||
output: Path
|
||||
pages: list[str]
|
||||
|
||||
|
||||
def structural(line: str) -> bool:
|
||||
return (not line.strip() or HEADING.match(line) is not None or FENCE.match(line) is not None
|
||||
or HTML_TAG_ONLY.match(line) is not None or TABLE_RULE.match(line) is not None)
|
||||
|
||||
|
||||
def code_quote(span: list[str], start: int) -> tuple[str, int]:
|
||||
"""The first non-structural line of a span (stripped, capped), and its file line."""
|
||||
for i, line in enumerate(span):
|
||||
if not structural(line):
|
||||
return line.strip()[:QUOTE_MAX], start + i
|
||||
return span[0].strip()[:QUOTE_MAX] or span[0], start
|
||||
|
||||
|
||||
def footnoted(text: str, labels: list[str]) -> str:
|
||||
text = text.strip()
|
||||
if text and text[-1] not in ".!?":
|
||||
text += "."
|
||||
marks = "".join(f"[^{x}]" for x in labels)
|
||||
out, last = [], 0
|
||||
for _a, b in sentence_spans(text):
|
||||
out.append(text[last:b] + marks)
|
||||
last = b
|
||||
out.append(text[last:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _link_titles(text: str, titles: dict[str, str], self_id: str, linked: set[str]) -> str:
|
||||
for cid, title in sorted(titles.items(), key=lambda kv: -len(kv[1])):
|
||||
if cid == self_id or cid in linked or len(title) < 4:
|
||||
continue
|
||||
pattern = re.compile(r"(?<![\w`\[])" + re.escape(title) + r"(?![\w`\]])", re.IGNORECASE)
|
||||
m = pattern.search(text)
|
||||
if m is None:
|
||||
continue
|
||||
text = text[:m.start()] + f"[{m.group(0)}]({cid}.md)" + text[m.end():]
|
||||
linked.add(cid)
|
||||
return text
|
||||
|
||||
|
||||
def _version_line(lines: list[str], version: str) -> int:
|
||||
for n, _level, text in headings_of(lines):
|
||||
m = VERSION_HEADING.match(text)
|
||||
if m and m.group(1) == version:
|
||||
return n
|
||||
raise RunError(f"no heading for version {version}")
|
||||
|
||||
|
||||
def assemble(work: Path, kilde_root: Path, out: Path, at: str | None = None) -> int:
|
||||
if out.exists():
|
||||
raise RunError(f"{out} exists; refusing to overwrite")
|
||||
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
index = json.loads((work / "K-work" / "index.json").read_text(encoding="utf-8"))
|
||||
frags = json.loads((work / "K-work" / "accepted.json").read_text(encoding="utf-8"))["fragments"]
|
||||
entries = {e["id"]: e for e in index["entries"]}
|
||||
kilde = kilde_root / "kilde"
|
||||
cache: dict[str, list[str]] = {}
|
||||
|
||||
def lines_of(name: str) -> list[str]:
|
||||
if name not in cache:
|
||||
cache[name] = (kilde / name).read_text(encoding="utf-8").split("\n")
|
||||
return cache[name]
|
||||
|
||||
by_concept: dict[str, list[dict[str, Any]]] = {}
|
||||
for f in frags:
|
||||
by_concept.setdefault(f["concept_id"], []).append(f)
|
||||
titles = {cid: str(entries[cid]["title"]) for cid in by_concept}
|
||||
pages: dict[str, str] = {}
|
||||
for cid, items in by_concept.items():
|
||||
e = entries[cid]
|
||||
sources: list[dict[str, Any]] = []
|
||||
versions: dict[str, str] = {}
|
||||
body_parts: list[str] = []
|
||||
history: list[tuple[tuple[int, ...], int, str]] = []
|
||||
linked: set[str] = set()
|
||||
for k, f in enumerate(items, 1):
|
||||
lines = lines_of(f["file"])
|
||||
src: dict[str, Any] = {"id": f"f{k}", "resource": f"/kilde/{f['file']}", "title": f["section"]}
|
||||
if f.get("quote"):
|
||||
src["x-locator"] = {"line": f["quote_line"]}
|
||||
src["x-quote"] = f["quote"]
|
||||
else:
|
||||
quote, line = code_quote(lines[f["start"] - 1:f["end"]], f["start"])
|
||||
src["x-locator"] = {"line": line}
|
||||
src["x-quote"] = quote
|
||||
src["x-quote-by"] = "code"
|
||||
src["x-span"] = [f["start"], f["end"]]
|
||||
sources.append(src)
|
||||
labels = [f"f{k}"]
|
||||
if f.get("version"):
|
||||
v = str(f["version"])
|
||||
if v not in versions:
|
||||
versions[v] = "v" + re.sub(r"\D", "", v)
|
||||
sources.append({"id": versions[v], "resource": f"/kilde/{f['file']}", "title": v,
|
||||
"x-locator": {"line": _version_line(lines, v)}, "x-quote": v})
|
||||
labels.append(versions[v])
|
||||
text = _link_titles(footnoted(f["text"], labels), titles, cid, linked)
|
||||
history.append((version_key(f"release-v{v}.md"), k, text))
|
||||
else:
|
||||
body_parts.append(_link_titles(footnoted(f["text"], labels), titles, cid, linked))
|
||||
body = [f"# {e['title']}", ""]
|
||||
for p in body_parts:
|
||||
body += [p, ""]
|
||||
if history:
|
||||
body += ["## History", ""]
|
||||
body += [f"- {t}" for _, _, t in sorted(history)]
|
||||
body.append("")
|
||||
body += [f"[^{s['id']}]: {Path(s['resource']).name}, line {s['x-locator']['line']}" for s in sources]
|
||||
desc = FOOTNOTE_REF.sub("", items[0]["text"]).strip()[:200]
|
||||
fm: dict[str, Any] = {"type": "Concept", "title": str(e["title"]), "description": desc, "sources": sources,
|
||||
"x-aliases": [str(a) for a in e.get("aliases", [])], "x-fragments": len(items)}
|
||||
if e.get("kind") == "merge-candidate" and e.get("merge_of"):
|
||||
fm["x-merge-candidate"] = str(e["merge_of"])
|
||||
fm["generated"] = {"by": GENERATOR, "at": at}
|
||||
pages[f"{cid}.md"] = frontmatter(fm) + "\n" + "\n".join(body) + "\n"
|
||||
out.mkdir(parents=True)
|
||||
for name, text in pages.items():
|
||||
(out / name).write_text(text, encoding="utf-8")
|
||||
index_md = ["---", 'okf_version: "0.2"', "---", "", "# Concepts", ""]
|
||||
index_md += [f"- [{titles[cid]}]({cid}.md)" for cid in sorted(by_concept)]
|
||||
(out / "index.md").write_text("\n".join(index_md) + "\n", encoding="utf-8")
|
||||
(out / "log.md").write_text(f"# Log\n\n## {at[:10]}\n* **Build**: K assembled by {GENERATOR}: {len(pages)} "
|
||||
f"pages from {len(frags)} accepted fragments.\n", encoding="utf-8")
|
||||
return len(pages)
|
||||
|
||||
|
||||
def copy(src: Path, dst: Path) -> None:
|
||||
if dst.exists():
|
||||
raise RunError(f"{dst} exists; refusing to overwrite")
|
||||
shutil.copytree(src, dst)
|
||||
|
||||
|
||||
def _split_page(text: str) -> tuple[str, str, list[str], list[str]]:
|
||||
"""(frontmatter block, title line, body lines without title and footnote definitions, definition lines)."""
|
||||
end = text.index("\n---\n", 4) + 5
|
||||
head, rest = text[:end], text[end:].split("\n")
|
||||
title = next(line for line in rest if line.startswith("# "))
|
||||
body = [line for line in rest if line != title and not FOOTNOTE_DEF.match(line.strip())]
|
||||
defs = [line for line in rest if FOOTNOTE_DEF.match(line.strip())]
|
||||
return head, title, body, defs
|
||||
|
||||
|
||||
def consolidation_tasks(bundle: Path, work: Path, min_fragments: int = 3, per_run: int = 8) -> list[Run]:
|
||||
chosen: list[str] = []
|
||||
for p in sorted(bundle.glob("*.md")):
|
||||
if p.name in ("index.md", "log.md"):
|
||||
continue
|
||||
fm = read_document(p, p.name).frontmatter or {}
|
||||
if int(fm.get("x-fragments", 0)) >= min_fragments:
|
||||
chosen.append(p.stem)
|
||||
runs: list[Run] = []
|
||||
for i in range(0, len(chosen), per_run):
|
||||
n = len(runs) + 1
|
||||
task = work / "Kplus-oppgaver" / f"run-{n}.md"
|
||||
output = work / "Kplus-bodies" / f"run-{n}.json"
|
||||
if task.exists():
|
||||
raise RunError(f"{task} exists; refusing to overwrite")
|
||||
parts = [f"# K+ consolidation run {n}\n\nRewrite each page body below into flowing text. Output: {output}\n"]
|
||||
for cid in chosen[i:i + per_run]:
|
||||
text = (bundle / f"{cid}.md").read_text(encoding="utf-8")
|
||||
fm = read_document(bundle / f"{cid}.md", f"{cid}.md").frontmatter or {}
|
||||
_head, title, body, _defs = _split_page(text)
|
||||
quotes = [f"- [^{s['id']}]: {s.get('x-quote', '')}" for s in fm.get("sources", [])]
|
||||
parts.append(f"## Page {cid} ({title[2:]})\n\nBody:\n\n" + "\n".join(body).strip()
|
||||
+ "\n\nWhat each label quotes:\n\n" + "\n".join(quotes) + "\n")
|
||||
task.parent.mkdir(parents=True, exist_ok=True)
|
||||
task.write_text("\n".join(parts), encoding="utf-8")
|
||||
runs.append(Run(task, output, chosen[i:i + per_run]))
|
||||
return runs
|
||||
|
||||
|
||||
def apply(bundle: Path, bodies_path: Path) -> tuple[int, int]:
|
||||
bodies = json.loads(bodies_path.read_text(encoding="utf-8"))
|
||||
applied = refused = 0
|
||||
for cid, new in bodies.items():
|
||||
page = bundle / f"{cid}.md"
|
||||
if not page.is_file() or not isinstance(new, str):
|
||||
refused += 1
|
||||
continue
|
||||
head, title, body, defs = _split_page(page.read_text(encoding="utf-8"))
|
||||
old_labels = set(FOOTNOTE_REF.findall("\n".join(body)))
|
||||
if set(FOOTNOTE_REF.findall(new)) != old_labels or FOOTNOTE_DEF.search(new):
|
||||
refused += 1
|
||||
continue
|
||||
page.write_text(head + "\n" + title + "\n\n" + new.strip() + "\n\n" + "\n".join(defs) + "\n", encoding="utf-8")
|
||||
applied += 1
|
||||
return applied, refused
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
b = sub.add_parser("build")
|
||||
b.add_argument("--work", required=True)
|
||||
b.add_argument("--kilde-root", required=True)
|
||||
b.add_argument("--out", required=True)
|
||||
b.add_argument("--at")
|
||||
c = sub.add_parser("copy")
|
||||
c.add_argument("--src", required=True)
|
||||
c.add_argument("--out", required=True)
|
||||
t = sub.add_parser("tasks")
|
||||
t.add_argument("--bundle", required=True)
|
||||
t.add_argument("--work", required=True)
|
||||
t.add_argument("--min", type=int, default=3)
|
||||
t.add_argument("--per-run", type=int, default=8)
|
||||
a = sub.add_parser("apply")
|
||||
a.add_argument("--bundle", required=True)
|
||||
a.add_argument("--bodies", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "build":
|
||||
n = assemble(Path(args.work), Path(args.kilde_root), Path(args.out), args.at)
|
||||
print(f"K: {n} pages -> {args.out}")
|
||||
elif args.command == "copy":
|
||||
copy(Path(args.src), Path(args.out))
|
||||
print(f"copied {args.src} -> {args.out}")
|
||||
elif args.command == "tasks":
|
||||
for r in consolidation_tasks(Path(args.bundle), Path(args.work), args.min, args.per_run):
|
||||
print(f"task: {r.task} output: {r.output} pages: {len(r.pages)}")
|
||||
else:
|
||||
applied, refused = apply(Path(args.bundle), Path(args.bodies))
|
||||
print(f"applied {applied}, refused {refused}")
|
||||
except (RunError, OSError, KeyError, ValueError) as exc:
|
||||
print(f"assemble cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
329
evals/p1/cost.py
Normal file
329
evals/p1/cost.py
Normal file
|
|
@ -0,0 +1,329 @@
|
|||
"""The trial's cost from the run sessions' transcripts (TRIAL, P1): harvest, allocation, projection, stop. No model.
|
||||
|
||||
harvest --project-dir D --okter FILE --blocks F --total N [--packs FILE] --out FILE
|
||||
Reads only the sessions listed in okter.txt ("<session-id> <step>" per line): each session's main
|
||||
transcript (orchestration, reported apart) and every subagent transcript under <session>/subagents/. Per
|
||||
subagent: input, cache read, cache write and output tokens kept apart (each message id counted once, at
|
||||
its largest value, because a streamed message repeats its id with growing output), turns (distinct
|
||||
message ids), startup (the first message's input side), agentType and description from its .meta.json.
|
||||
A message whose final usage was never written (no stop reason on any record) gets its output from the
|
||||
growth of the next message's input side less the injected text counted high (a lower bound); the
|
||||
recorded output stays beside it (output_recorded).
|
||||
A description that starts "P1-<variant>-<unit>" attributes the run; any other is listed as unattributed,
|
||||
never dropped. Per variant: totals, tokens per source (a pack's cost split by block share), the means of
|
||||
sources 1-5 and 15-19, and the projection to all sources (N blocks). The file is rewritten from the
|
||||
transcripts on every harvest; stop rows already in it are kept.
|
||||
stop VARIANT --kost FILE --blocks F [--pages-total N --pages-first M] [--record]
|
||||
The stop rule (cap 1 M tokens, all four categories): K after pack 1, projected per block processed and,
|
||||
beside it, naively per source; K+ after run 1, projected over the pages to consolidate. Exit 1 = stop.
|
||||
With --record the stop row is written into the kost file.
|
||||
|
||||
Projection: cost per block c = a + b*R by least squares over the measured units, R = blocks read before the unit;
|
||||
the projected total for T blocks is a*T + b*T*T/2 (the sum as R grows from 0 to T). One unit: b = 0. The flat
|
||||
projection (mean cost per block times T) stands beside it. Exit 0 = done, 1 = stop (stop only), 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from direction_probe import USAGE_FIELDS, read_transcript
|
||||
|
||||
SCHEMA = "okf-wiki/p1-cost/1"
|
||||
CAP = 1_000_000
|
||||
CATEGORIES = {"input_tokens": "input", "cache_read_input_tokens": "cache_read",
|
||||
"cache_creation_input_tokens": "cache_write", "output_tokens": "output"}
|
||||
PREFIX = re.compile(r"^P1-([A-Za-z0-9+]+)-(\S+)")
|
||||
PACK_UNIT = re.compile(r"^pack(\d+)")
|
||||
CLASS = {"LOAD": "load", "K": "write", "K+": "write"}
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
def sessions(okter: Path) -> list[str]:
|
||||
ids: list[str] = []
|
||||
for line in okter.read_text(encoding="utf-8").splitlines():
|
||||
parts = line.split()
|
||||
if parts and parts[0] not in ids:
|
||||
ids.append(parts[0])
|
||||
return ids
|
||||
|
||||
|
||||
INPUT_SIDE = ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")
|
||||
CHARS_PER_TOKEN_LOW = 2.5 # injected text counted high, so a derived output is a lower bound
|
||||
|
||||
|
||||
def _result_chars(message: dict[str, Any]) -> int:
|
||||
content = message.get("content")
|
||||
if not isinstance(content, list):
|
||||
return 0
|
||||
total = 0
|
||||
for item in content:
|
||||
if not (isinstance(item, dict) and item.get("type") == "tool_result"):
|
||||
continue
|
||||
inner = item.get("content")
|
||||
if isinstance(inner, str):
|
||||
total += len(inner)
|
||||
elif isinstance(inner, list):
|
||||
total += sum(len(str(x.get("text", ""))) for x in inner if isinstance(x, dict))
|
||||
return total
|
||||
|
||||
|
||||
def usage(path: Path) -> dict[str, Any]:
|
||||
"""Tokens of one transcript. A message whose records all lack a stop reason never got its final usage written
|
||||
(seen for a long streamed message: output 2); when another message follows, its output is the growth of the
|
||||
next message's input side less the text injected between them (tool results and rendered attachments, counted
|
||||
high at CHARS_PER_TOKEN_LOW), a lower bound, and never below what was recorded."""
|
||||
per: dict[str, dict[str, int]] = {}
|
||||
order: list[str] = []
|
||||
complete: set[str] = set()
|
||||
between: dict[str, int] = {}
|
||||
last_stop: str | None = None
|
||||
for line in read_transcript(path):
|
||||
message = line.get("message")
|
||||
if line.get("type") == "attachment" and order:
|
||||
between[order[-1]] = between.get(order[-1], 0) + len(str(line.get("rendered") or ""))
|
||||
continue
|
||||
if not isinstance(message, dict):
|
||||
continue
|
||||
if line.get("type") == "user" and order:
|
||||
between[order[-1]] = between.get(order[-1], 0) + _result_chars(message)
|
||||
continue
|
||||
if line.get("type") != "assistant":
|
||||
continue
|
||||
mid = str(message.get("id", ""))
|
||||
if mid not in per:
|
||||
order.append(mid)
|
||||
seen = per.setdefault(mid, {})
|
||||
fields = message.get("usage") or {}
|
||||
for f in USAGE_FIELDS:
|
||||
value = fields.get(f)
|
||||
if isinstance(value, int):
|
||||
seen[f] = max(seen.get(f, 0), value)
|
||||
stop = message.get("stop_reason")
|
||||
last_stop = stop if isinstance(stop, str) else None
|
||||
if isinstance(stop, str):
|
||||
complete.add(mid)
|
||||
out: dict[str, Any] = {CATEGORIES[f]: sum(v.get(f, 0) for v in per.values()) for f in USAGE_FIELDS}
|
||||
out["output_recorded"] = out["output"]
|
||||
derived = unrecorded = 0
|
||||
for i, mid in enumerate(order):
|
||||
if mid in complete:
|
||||
continue
|
||||
if i + 1 == len(order):
|
||||
unrecorded += 1
|
||||
continue
|
||||
growth = (sum(per[order[i + 1]].get(f, 0) for f in INPUT_SIDE) - sum(per[mid].get(f, 0) for f in INPUT_SIDE)
|
||||
- int(between.get(mid, 0) / CHARS_PER_TOKEN_LOW))
|
||||
recorded = per[mid].get("output_tokens", 0)
|
||||
if growth > recorded:
|
||||
out["output"] += growth - recorded
|
||||
derived += 1
|
||||
out["output_derived_messages"] = derived
|
||||
out["output_unrecorded"] = unrecorded
|
||||
out["total"] = sum(out[c] for c in CATEGORIES.values())
|
||||
out["turns"] = len(order)
|
||||
first = per[order[0]] if order else {}
|
||||
out["startup"] = sum(first.get(f, 0) for f in INPUT_SIDE)
|
||||
out["stop_reason"] = last_stop
|
||||
return out
|
||||
|
||||
|
||||
def harvest(project_dir: Path, ids: list[str]) -> dict[str, Any]:
|
||||
runs: list[dict[str, Any]] = []
|
||||
unattributed: list[dict[str, Any]] = []
|
||||
main: dict[str, Any] = {}
|
||||
for sid in ids:
|
||||
top = project_dir / f"{sid}.jsonl"
|
||||
if top.is_file():
|
||||
main[sid] = usage(top)
|
||||
for path in sorted((project_dir / sid / "subagents").glob("agent-*.jsonl")):
|
||||
meta_path = path.with_name(path.stem + ".meta.json")
|
||||
meta = json.loads(meta_path.read_text(encoding="utf-8")) if meta_path.is_file() else {}
|
||||
row = {"session": sid, "agent": path.stem, "agentType": meta.get("agentType"),
|
||||
"description": str(meta.get("description", "")), **usage(path)}
|
||||
m = PREFIX.match(row["description"])
|
||||
if m is None:
|
||||
unattributed.append(row)
|
||||
continue
|
||||
row["variant"], row["unit"] = m.group(1), m.group(2)
|
||||
row["class"] = CLASS.get(m.group(1), "other")
|
||||
runs.append(row)
|
||||
return {"sessions_read": ids, "runs": runs, "unattributed": unattributed, "main": main}
|
||||
|
||||
|
||||
def allocate(pack_costs: Mapping[int, float], packs: list[list[str]], blocks_doc: dict[str, Any]) -> dict[str, float]:
|
||||
"""Tokens per source: each pack's cost split over its sources by their share of the pack's blocks."""
|
||||
count: dict[str, int] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
count[b["file"]] = count.get(b["file"], 0) + 1
|
||||
per: dict[str, float] = {}
|
||||
for n, names in enumerate(packs, 1):
|
||||
if n not in pack_costs:
|
||||
continue
|
||||
total = sum(count.get(x, 0) for x in names) or 1
|
||||
for x in names:
|
||||
per[x] = pack_costs[n] * count.get(x, 0) / total
|
||||
return per
|
||||
|
||||
|
||||
def means(per: dict[str, float], order: list[str]) -> dict[str, float | None]:
|
||||
def mean(names: list[str]) -> float | None:
|
||||
vals = [per[x] for x in names if x in per]
|
||||
return sum(vals) / len(vals) if vals else None
|
||||
|
||||
return {"1-5": mean(order[0:5]), "15-19": mean(order[14:19])}
|
||||
|
||||
|
||||
def project(points: list[tuple[float, int, int]], total_blocks: int) -> dict[str, Any]:
|
||||
"""points: (cost, blocks in the unit, blocks read before it)."""
|
||||
if not points:
|
||||
return {"fit": None, "flat": None, "T": total_blocks}
|
||||
xs = [float(r) for _, _, r in points]
|
||||
ys = [c / b for c, b, _ in points]
|
||||
fit: dict[str, Any]
|
||||
if len(points) == 1 or len(set(xs)) == 1:
|
||||
a, b = sum(ys) / len(ys), 0.0
|
||||
fit = {"a": a, "b": 0, "note": "one point, no growth term"}
|
||||
else:
|
||||
mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
|
||||
b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
|
||||
a = my - b * mx
|
||||
fit = {"a": a, "b": b, "note": f"{len(points)} points, not validated"}
|
||||
fit["value"] = a * total_blocks + b * total_blocks * total_blocks / 2
|
||||
flat = sum(c for c, _, _ in points) / sum(bl for _, bl, _ in points) * total_blocks
|
||||
return {"fit": fit, "flat": flat, "T": total_blocks}
|
||||
|
||||
|
||||
def stop_k(pack1_cost: int, pack1_blocks: int, cut_blocks: int, pack1_sources: int, cut_sources: int
|
||||
) -> dict[str, Any]:
|
||||
per_block = pack1_cost * cut_blocks / pack1_blocks
|
||||
per_source = pack1_cost * cut_sources / pack1_sources
|
||||
return {"per_block": per_block, "per_source": per_source, "cap": CAP, "stop": per_block > CAP}
|
||||
|
||||
|
||||
def stop_kplus(run1_cost: int, run1_pages: int, pages_total: int) -> dict[str, Any]:
|
||||
projected = run1_cost * pages_total / run1_pages
|
||||
return {"projected": projected, "cap": CAP, "stop": projected > CAP}
|
||||
|
||||
|
||||
def _pack_number(unit: str) -> int | None:
|
||||
m = PACK_UNIT.match(unit)
|
||||
return int(m.group(1)) if m else None
|
||||
|
||||
|
||||
def summarize(h: dict[str, Any], blocks_doc: dict[str, Any], packs_log: dict[str, Any] | None,
|
||||
total_blocks: int) -> dict[str, Any]:
|
||||
order = [str(x) for x in blocks_doc["order"]]
|
||||
count: dict[str, int] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
count[b["file"]] = count.get(b["file"], 0) + 1
|
||||
variants: dict[str, Any] = {}
|
||||
for v in sorted({r["variant"] for r in h["runs"]}):
|
||||
mine = [r for r in h["runs"] if r["variant"] == v]
|
||||
row: dict[str, Any] = {c: sum(r[c] for r in mine) for c in (*CATEGORIES.values(), "total", "turns", "startup")}
|
||||
row["subagents"] = len(mine)
|
||||
row["class"] = mine[0]["class"]
|
||||
row["counts_toward_cap"] = v in ("K", "K+")
|
||||
if v == "K" and packs_log:
|
||||
packs = [list(p["sources"]) for p in sorted(packs_log["packs"], key=lambda p: int(p["pack"]))]
|
||||
pack_costs: dict[int, float] = {}
|
||||
per_cat: dict[str, dict[int, float]] = {c: {} for c in CATEGORIES.values()}
|
||||
for r in mine:
|
||||
n = _pack_number(r["unit"])
|
||||
if n is not None:
|
||||
pack_costs[n] = pack_costs.get(n, 0) + r["total"]
|
||||
for c in CATEGORIES.values():
|
||||
per_cat[c][n] = per_cat[c].get(n, 0) + r[c]
|
||||
row["per_pack"] = {str(n): c for n, c in sorted(pack_costs.items())}
|
||||
row["per_source"] = allocate(pack_costs, packs, blocks_doc)
|
||||
row["per_source_by_category"] = {c: allocate(per_cat[c], packs, blocks_doc) for c in CATEGORIES.values()}
|
||||
row["means"] = means(row["per_source"], order)
|
||||
points, read = [], 0
|
||||
for n, names in enumerate(packs, 1):
|
||||
blocks_n = sum(count.get(x, 0) for x in names)
|
||||
if n in pack_costs:
|
||||
points.append((pack_costs[n], blocks_n, read))
|
||||
read += blocks_n
|
||||
row["projection"] = project(points, total_blocks)
|
||||
variants[v] = row
|
||||
return variants
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
h = sub.add_parser("harvest")
|
||||
h.add_argument("--project-dir", required=True)
|
||||
h.add_argument("--okter", required=True)
|
||||
h.add_argument("--blocks", required=True)
|
||||
h.add_argument("--total", type=int, required=True)
|
||||
h.add_argument("--packs")
|
||||
h.add_argument("--out", required=True)
|
||||
s = sub.add_parser("stop")
|
||||
s.add_argument("variant", choices=("K", "K+"))
|
||||
s.add_argument("--kost", required=True)
|
||||
s.add_argument("--blocks", required=True)
|
||||
s.add_argument("--packs")
|
||||
s.add_argument("--pages-total", type=int)
|
||||
s.add_argument("--pages-first", type=int)
|
||||
s.add_argument("--record", action="store_true")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
if args.command == "harvest":
|
||||
out = Path(args.out)
|
||||
old = json.loads(out.read_text(encoding="utf-8")) if out.is_file() else {}
|
||||
h = harvest(Path(args.project_dir), sessions(Path(args.okter)))
|
||||
packs_log = json.loads(Path(args.packs).read_text(encoding="utf-8")) if args.packs else None
|
||||
doc = {"schema": SCHEMA, **h, "variants": summarize(h, blocks_doc, packs_log, args.total),
|
||||
"stops": old.get("stops", {})}
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
print(f"harvest: {len(h['runs'])} attributed runs, {len(h['unattributed'])} unattributed, sessions read: "
|
||||
f"{', '.join(h['sessions_read'])} -> {out}")
|
||||
return 0
|
||||
kost_path = Path(args.kost)
|
||||
kost = json.loads(kost_path.read_text(encoding="utf-8"))
|
||||
runs = [r for r in kost["runs"] if r["variant"] == args.variant]
|
||||
if args.variant == "K":
|
||||
first = [r for r in runs if _pack_number(r["unit"]) == 1]
|
||||
if not first:
|
||||
raise RunError("no pack 1 run in the kost file; harvest after pack 1")
|
||||
if not args.packs:
|
||||
raise RunError("--packs is needed for K")
|
||||
log = json.loads(Path(args.packs).read_text(encoding="utf-8"))
|
||||
p1 = next(p for p in log["packs"] if int(p["pack"]) == 1)
|
||||
row = stop_k(sum(r["total"] for r in first), int(p1["blocks"]), len(blocks_doc["blocks"]),
|
||||
len(p1["sources"]), len(blocks_doc["order"]))
|
||||
else:
|
||||
first = [r for r in runs if r["unit"].startswith("run1")]
|
||||
if not first or not args.pages_total or not args.pages_first:
|
||||
raise RunError("K+ needs a run 1 in the kost file, --pages-total and --pages-first")
|
||||
row = stop_kplus(sum(r["total"] for r in first), args.pages_first, args.pages_total)
|
||||
print(f"{args.variant}: projected {row.get('per_block', row.get('projected'))} tokens"
|
||||
+ (f" (naive per source {row['per_source']})" if "per_source" in row else "")
|
||||
+ f", cap {CAP}: {'STOP' if row['stop'] else 'continue'}")
|
||||
if args.record:
|
||||
kost.setdefault("stops", {})[args.variant] = row
|
||||
kost_path.write_text(json.dumps(kost, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
return 1 if row["stop"] else 0
|
||||
except (RunError, OSError, json.JSONDecodeError, KeyError, StopIteration) as exc:
|
||||
print(f"cost cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
178
evals/p1/dmap.py
Normal file
178
evals/p1/dmap.py
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
"""D, the trial's no-model baseline (TRIAL, P1): a generated map and a code-built concept register. It calls no model.
|
||||
|
||||
build --blocks FILE --kilde-root DIR --out DIR [--at ISO]
|
||||
One map page per source (k-<slug>.md) whose body mirrors the source's headings, with one sources[] entry per
|
||||
section; one register page per kept term (r-<slug>.md) whose body is a pure link list to the map sections
|
||||
where the term occurs, with one sources[] entry per occurrence. A term is kept when it is a docs-page heading
|
||||
that passes concept test 2, or when it occurs in at least 2 of the documents. Bodies hold only headings and
|
||||
pure link lists, so the claim count is 0 by construction; the exact locators live in sources[].
|
||||
|
||||
--kilde-root is the folder that contains kilde/ (the auditor's SOURCES argument). Exit 0 = written, 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from p1.split import headings_of, is_meta, occurs, terms
|
||||
|
||||
MIN_DOCUMENTS = 2
|
||||
GENERATOR = "okf-wiki/p1-dmap"
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class DStats:
|
||||
maps: int
|
||||
registers: int
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
|
||||
|
||||
|
||||
def frontmatter(data: dict[str, Any]) -> str:
|
||||
"""A frontmatter block; PyYAML quotes any string that would load as another type (no, 1.10)."""
|
||||
return "---\n" + yaml.safe_dump(data, sort_keys=False, allow_unicode=True, width=10_000) + "---\n"
|
||||
|
||||
|
||||
def _anchors(headings: list[tuple[int, int, str]]) -> dict[int, str]:
|
||||
seen: dict[str, int] = {}
|
||||
out: dict[int, str] = {}
|
||||
for line, _level, text in headings:
|
||||
base = slug(text)
|
||||
n = seen.get(base, 0)
|
||||
seen[base] = n + 1
|
||||
out[line] = base if n == 0 else f"{base}-{n}"
|
||||
return out
|
||||
|
||||
|
||||
def _title(name: str, headings: list[tuple[int, int, str]]) -> str:
|
||||
first = next((t for _, level, t in headings if level == 1), None)
|
||||
if first:
|
||||
return first
|
||||
return f"Release notes {headings[0][2]}" if name.startswith("release-") and headings else name
|
||||
|
||||
|
||||
def build(blocks_doc: dict[str, Any], kilde_root: Path, out: Path, at: str | None = None) -> DStats:
|
||||
if out.exists():
|
||||
raise RunError(f"{out} exists; refusing to overwrite")
|
||||
at = at or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
names = [str(n) for n in blocks_doc["order"]]
|
||||
kilde = kilde_root / "kilde"
|
||||
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
|
||||
lines = {n: t.split("\n") for n, t in texts.items()}
|
||||
heads = {n: headings_of(lines[n]) for n in names}
|
||||
anchors = {n: _anchors(heads[n]) for n in names}
|
||||
titles = {n: _title(n, heads[n]) for n in names}
|
||||
pages: dict[str, str] = {}
|
||||
gen = {"by": GENERATOR, "at": at}
|
||||
|
||||
for n in names:
|
||||
sources = []
|
||||
body = [f"# {titles[n]}", ""]
|
||||
for i, (line, level, text) in enumerate(heads[n], 1):
|
||||
sources.append({"id": f"s{i}", "resource": f"/kilde/{n}", "title": text, "x-locator": {"line": line},
|
||||
"x-quote": text})
|
||||
if level > 1 or text != titles[n]:
|
||||
body += [f"{'#' * max(level, 2)} {text}", ""]
|
||||
fm = {"type": "Document map", "title": titles[n], "description": f"The sections of {n}.", "sources": sources,
|
||||
"generated": gen}
|
||||
pages[f"k-{slug(n.removesuffix('.md'))}.md"] = frontmatter(fm) + "\n" + "\n".join(body).rstrip() + "\n"
|
||||
|
||||
candidates = [t for n in names if n.startswith("docs-") for _, level, t in heads[n] if level in (2, 3)]
|
||||
from_headings = {t for t in candidates if not is_meta(t)}
|
||||
pool = list(dict.fromkeys(list(from_headings) + [t for n in names for t in terms(texts[n])]))
|
||||
registers: dict[str, str] = {}
|
||||
for term in pool:
|
||||
if is_meta(term):
|
||||
continue
|
||||
holders = [n for n in names if occurs(term, texts[n])]
|
||||
if term not in from_headings and len(holders) < MIN_DOCUMENTS:
|
||||
continue
|
||||
pattern = re.compile(r"(?<!\w)" + re.escape(term) + r"(?!\w)", re.IGNORECASE)
|
||||
sources, links, seen = [], [], set()
|
||||
for n in holders:
|
||||
for i, raw in enumerate(lines[n], 1):
|
||||
m = pattern.search(raw)
|
||||
if m is None:
|
||||
continue
|
||||
section = max((h for h in heads[n] if h[0] <= i), default=None)
|
||||
key = (n, section[0] if section else 0)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
sources.append({"id": f"o{len(sources) + 1}", "resource": f"/kilde/{n}", "title": titles[n],
|
||||
"x-locator": {"line": i}, "x-quote": m.group(0)})
|
||||
target = f"k-{slug(n.removesuffix('.md'))}.md"
|
||||
if section is not None:
|
||||
target += f"#{anchors[n][section[0]]}"
|
||||
where = titles[n] + (f" › {section[2]}" if section and section[2] != titles[n] else "")
|
||||
links.append(f"- [{where.replace('[', '(').replace(']', ')')}]({target})")
|
||||
if not sources:
|
||||
continue
|
||||
name = f"r-{slug(term)}.md"
|
||||
k = 2
|
||||
while name in registers:
|
||||
name, k = f"r-{slug(term)}-{k}.md", k + 1
|
||||
fm = {"type": "Register entry", "title": term, "description": f"Where {term} occurs in the sources.",
|
||||
"sources": sources, "generated": gen}
|
||||
registers[name] = frontmatter(fm) + "\n" + f"# {term}\n\n" + "\n".join(links) + "\n"
|
||||
|
||||
out.mkdir(parents=True)
|
||||
for name, text in {**pages, **registers}.items():
|
||||
(out / name).write_text(text, encoding="utf-8")
|
||||
index = ["---", 'okf_version: "0.2"', "---", "", "# Document maps", ""]
|
||||
index += [f"- [{titles[n]}](k-{slug(n.removesuffix('.md'))}.md)" for n in names]
|
||||
index += ["", "# Register", ""]
|
||||
index += [f"- [{name[2:-3]}]({name})" for name in sorted(registers)]
|
||||
(out / "index.md").write_text("\n".join(index) + "\n", encoding="utf-8")
|
||||
day = at[:10]
|
||||
(out / "log.md").write_text(
|
||||
f"# Log\n\n## {day}\n"
|
||||
f"* **Build**: D by {GENERATOR}, {len(pages)} map pages and {len(registers)} register pages from {len(names)} "
|
||||
f"sources. Thresholds, recorded before any count: a term is kept when it is a docs-page heading of level 2 or "
|
||||
f"3 that passes concept test 2, or when it occurs in at least {MIN_DOCUMENTS} of the {len(names)} documents "
|
||||
f"(case-insensitive, whole phrase). Terms: backticked spans of 3-40 characters and capitalised phrases of 2-4 "
|
||||
f"words, a sentence-initial word dropped.\n",
|
||||
encoding="utf-8")
|
||||
return DStats(len(pages), len(registers))
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
b = sub.add_parser("build")
|
||||
b.add_argument("--blocks", required=True)
|
||||
b.add_argument("--kilde-root", required=True)
|
||||
b.add_argument("--out", required=True)
|
||||
b.add_argument("--at")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
s = build(doc, Path(args.kilde_root), Path(args.out), args.at)
|
||||
except (RunError, OSError, json.JSONDecodeError) as exc:
|
||||
print(f"dmap cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
print(f"D: {s.maps} map pages, {s.registers} register pages -> {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
374
evals/p1/kpack.py
Normal file
374
evals/p1/kpack.py
Normal file
|
|
@ -0,0 +1,374 @@
|
|||
"""K, the fragment compiler's code side (TRIAL, P1): skeleton, packs, input parts, validation, identity. No model.
|
||||
|
||||
skeleton --kilde DIR --out FILE
|
||||
The concept index from the titles and level 2-3 headings of every docs page under DIR that pass concept
|
||||
test 2. One entry per normalised name (the first page in name order wins); an index, not pages.
|
||||
pack N --blocks FILE --index FILE --kilde-root DIR --work DIR [--size 5]
|
||||
Pack N's input for the compiler: every block of its sources with its anchor (file › section › span) and
|
||||
numbered lines, then the index lines whose name occurs in those blocks, then the output contract, in parts
|
||||
of at most PART_CHARS characters (the Read tool's cap). Prints the part paths and the output path.
|
||||
accept N --work DIR --blocks FILE --kilde-root DIR
|
||||
Validates pack N's fragments, resolves each to a concept by code (existing, merge candidate, new or
|
||||
rejected), updates the index and appends the accepted fragments. Prints "accepted X of N" and the
|
||||
rejections by reason, and how many accepted fragments resolved to a concept whose index line was not
|
||||
sent with the pack (the pack log keeps the ids sent; the risk of the D7 phrase filter).
|
||||
|
||||
The work directory holds K-oppgaver/ (task parts), K-fragments/ (the compiler's JSON) and K-work/ (index.json,
|
||||
accepted.json, packs.json). Exit 0 = done, 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import sentence_spans
|
||||
from p1.split import docs_headings, is_meta
|
||||
|
||||
PART_CHARS = 60_000
|
||||
MERGE_JACCARD = 0.6
|
||||
NEEDS_QUOTE = re.compile(r"\d|`")
|
||||
REASONS = ("malformed", "anchor-outside-pack", "anchor-crosses-blocks", "sentence-count", "quote-missing",
|
||||
"quote-not-in-span", "tests-missing", "release-note-new", "meta")
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
def norm(name: str) -> str:
|
||||
text = name.replace("`", "").lower()
|
||||
text = re.sub(r"[^\w\s/.\-]", " ", text)
|
||||
words = text.split()
|
||||
if words and words[0] == "the":
|
||||
words = words[1:]
|
||||
words = [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
|
||||
return " ".join(w.strip(".") or w for w in words)
|
||||
|
||||
|
||||
def slug(text: str) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")[:60] or "x"
|
||||
|
||||
|
||||
def _unique_id(index: dict[str, Any], title: str) -> str:
|
||||
taken = {e["id"] for e in index["entries"]}
|
||||
base = slug(title)
|
||||
eid, k = base, 2
|
||||
while eid in taken:
|
||||
eid, k = f"{base}-{k}", k + 1
|
||||
return eid
|
||||
|
||||
|
||||
def skeleton(kilde: Path) -> dict[str, Any]:
|
||||
index: dict[str, Any] = {"schema": "okf-wiki/p1-index/1", "entries": []}
|
||||
seen: dict[str, dict[str, Any]] = {}
|
||||
for h in docs_headings(kilde):
|
||||
if is_meta(h.text):
|
||||
continue
|
||||
n = norm(h.text)
|
||||
if not n:
|
||||
continue
|
||||
if n in seen:
|
||||
if h.text not in seen[n]["aliases"]:
|
||||
seen[n]["aliases"].append(h.text)
|
||||
continue
|
||||
e = {"id": _unique_id(index, h.text), "title": h.text, "aliases": [h.text], "kind": "skeleton",
|
||||
"merge_of": None, "origin": {"file": h.file, "line": h.line}}
|
||||
index["entries"].append(e)
|
||||
seen[n] = e
|
||||
return index
|
||||
|
||||
|
||||
def _names(e: dict[str, Any]) -> set[str]:
|
||||
return {norm(a) for a in [e["title"], *e.get("aliases", [])]} - {""}
|
||||
|
||||
|
||||
def _jaccard(a: str, b: str) -> float:
|
||||
x, y = set(a.split()), set(b.split())
|
||||
return len(x & y) / len(x | y) if x | y else 0.0
|
||||
|
||||
|
||||
def resolve(fragment: dict[str, Any], index: dict[str, Any]) -> tuple[str, str | None]:
|
||||
"""existing <id> | merge-candidate <id> | new | rejected <reason> (D6, D3b)."""
|
||||
by_id = {e["id"]: e for e in index["entries"]}
|
||||
if fragment.get("status") == "existing" and fragment.get("id") in by_id:
|
||||
return "existing", str(fragment["id"])
|
||||
n = norm(str(fragment.get("concept", "")))
|
||||
for e in index["entries"]:
|
||||
if n in _names(e):
|
||||
return "existing", str(e["id"])
|
||||
if str(fragment.get("anchor", {}).get("file", "")).startswith("release-"):
|
||||
return "rejected", "release-note-new"
|
||||
if is_meta(str(fragment.get("concept", ""))):
|
||||
return "rejected", "meta"
|
||||
best = max(((max(_jaccard(n, m) for m in _names(e)), str(e["id"])) for e in index["entries"] if _names(e)),
|
||||
default=(0.0, ""))
|
||||
if best[0] >= MERGE_JACCARD:
|
||||
return "merge-candidate", best[1]
|
||||
return "new", None
|
||||
|
||||
|
||||
def relevant_lines(index: dict[str, Any], text: str) -> list[dict[str, Any]]:
|
||||
"""Index entries whose normalised name or alias occurs as a phrase in the normalised text (D7)."""
|
||||
hay = f" {norm(text)} "
|
||||
return [e for e in index["entries"] if any(f" {m} " in hay for m in _names(e))]
|
||||
|
||||
|
||||
def packs(names: list[str], size: int = 5) -> list[list[str]]:
|
||||
return [names[i:i + size] for i in range(0, len(names), size)]
|
||||
|
||||
|
||||
CONTRACT = """## Output contract
|
||||
|
||||
Write one JSON object `{"fragments": [...]}` to the output path named in your prompt. Each fragment:
|
||||
`concept`, `status` ("existing" with the index `id`, or "new" with `tests`: {"topic_shape", "citation"}),
|
||||
`text` (1-3 sentences), `anchor` {"file", "start", "end"} inside one block above, and `quote` (one piece copied from
|
||||
one line inside the anchor; required when the text holds a digit or a backticked name). A new concept only from a docs
|
||||
page, never from a release note.
|
||||
"""
|
||||
|
||||
|
||||
def render_input(pack_no: int, names: list[str], blocks_doc: dict[str, Any], index_lines: list[dict[str, Any]],
|
||||
kilde: Path) -> list[str]:
|
||||
lines = {n: (kilde / n).read_text(encoding="utf-8").split("\n") for n in names}
|
||||
pieces: list[str] = [f"# K pack {pack_no}: {len(names)} sources\n\nSources in order: {', '.join(names)}\n\n"
|
||||
+ CONTRACT + "\n## Blocks\n\n"]
|
||||
for b in blocks_doc["blocks"]:
|
||||
if b["file"] not in names:
|
||||
continue
|
||||
head = f"### {b['file']} › {b['section'] or '-'} › {b['start']}–{b['end']}"
|
||||
if b.get("version"):
|
||||
head += f" (version {b['version']})"
|
||||
body = [f"{n}: {lines[b['file']][n - 1]}" for n in range(b["start"], b["end"] + 1)]
|
||||
chunk = head + "\n"
|
||||
for row in body:
|
||||
if len(chunk) + len(row) + 1 > PART_CHARS // 2:
|
||||
pieces.append(chunk + "\n")
|
||||
chunk = head + " (continued)\n"
|
||||
chunk += row + "\n"
|
||||
pieces.append(chunk + "\n")
|
||||
pieces.append("## Index (concepts named in these blocks)\n\n")
|
||||
for e in index_lines:
|
||||
aliases = "; ".join(a for a in e["aliases"] if a != e["title"])
|
||||
pieces.append(f"- id: {e['id']} | {e['title']}" + (f" | aliases: {aliases}" if aliases else "") + "\n")
|
||||
room = PART_CHARS - len("<!-- part 99 of 99 -->\n")
|
||||
parts: list[str] = []
|
||||
current = ""
|
||||
for piece in pieces:
|
||||
if current and len(current) + len(piece) > room:
|
||||
parts.append(current)
|
||||
current = ""
|
||||
current += piece
|
||||
if current:
|
||||
parts.append(current)
|
||||
total = len(parts)
|
||||
return [f"<!-- part {i} of {total} -->\n" + p for i, p in enumerate(parts, 1)]
|
||||
|
||||
|
||||
def validate(fragments: list[Any], names: list[str], blocks_doc: dict[str, Any], kilde: Path
|
||||
) -> tuple[list[dict[str, Any]], Counter[str]]:
|
||||
by_file: dict[str, list[dict[str, Any]]] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
by_file.setdefault(b["file"], []).append(b)
|
||||
texts: dict[str, list[str]] = {}
|
||||
accepted: list[dict[str, Any]] = []
|
||||
rejected: Counter[str] = Counter()
|
||||
for f in fragments:
|
||||
a = f.get("anchor") if isinstance(f, dict) else None
|
||||
if not (isinstance(f, dict) and isinstance(f.get("concept"), str) and f["concept"].strip()
|
||||
and f.get("status") in ("new", "existing") and isinstance(f.get("text"), str)
|
||||
and isinstance(a, dict) and isinstance(a.get("file"), str)
|
||||
and type(a.get("start")) is int and type(a.get("end")) is int and a["start"] <= a["end"]):
|
||||
rejected["malformed"] += 1
|
||||
continue
|
||||
if a["file"] not in names:
|
||||
rejected["anchor-outside-pack"] += 1
|
||||
continue
|
||||
block = next((b for b in by_file.get(a["file"], []) if b["start"] <= a["start"] and a["end"] <= b["end"]), None)
|
||||
if block is None:
|
||||
rejected["anchor-crosses-blocks"] += 1
|
||||
continue
|
||||
if not 1 <= len(sentence_spans(f["text"].strip())) <= 3:
|
||||
rejected["sentence-count"] += 1
|
||||
continue
|
||||
if a["file"] not in texts:
|
||||
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
|
||||
span = texts[a["file"]][a["start"] - 1:a["end"]]
|
||||
quote = f.get("quote")
|
||||
if not (isinstance(quote, str) and quote.strip()):
|
||||
if NEEDS_QUOTE.search(f["text"]):
|
||||
rejected["quote-missing"] += 1
|
||||
continue
|
||||
quote, quote_line = None, None
|
||||
else:
|
||||
hit = next((a["start"] + i for i, line in enumerate(span) if quote in line), None)
|
||||
if hit is None:
|
||||
rejected["quote-not-in-span"] += 1
|
||||
continue
|
||||
quote_line = hit
|
||||
tests = f.get("tests")
|
||||
if f["status"] == "new" and not (isinstance(tests, dict) and str(tests.get("topic_shape", "")).strip()
|
||||
and str(tests.get("citation", "")).strip()):
|
||||
rejected["tests-missing"] += 1
|
||||
continue
|
||||
accepted.append({**f, "quote": quote, "quote_line": quote_line, "version": block.get("version"),
|
||||
"section": block["section"], "position": names.index(a["file"]) + 1})
|
||||
return accepted, rejected
|
||||
|
||||
|
||||
def _read_json(path: Path) -> Any:
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
raise RunError(f"cannot read {path}: {exc}") from exc
|
||||
|
||||
|
||||
def _write_json(path: Path, data: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _pack_log(work: Path) -> dict[str, Any]:
|
||||
path = work / "K-work" / "packs.json"
|
||||
return _read_json(path) if path.is_file() else {"schema": "okf-wiki/p1-packs/1", "packs": []}
|
||||
|
||||
|
||||
def _record(log: dict[str, Any], n: int) -> dict[str, Any]:
|
||||
for r in log["packs"]:
|
||||
if r["pack"] == n:
|
||||
return r # type: ignore[no-any-return]
|
||||
r = {"pack": n}
|
||||
log["packs"].append(r)
|
||||
return r
|
||||
|
||||
|
||||
def cmd_pack(args: argparse.Namespace) -> int:
|
||||
work, n = Path(args.work), args.n
|
||||
doc = _read_json(Path(args.blocks))
|
||||
all_packs = packs([str(x) for x in doc["order"]], args.size)
|
||||
if not 1 <= n <= len(all_packs):
|
||||
raise RunError(f"pack {n} does not exist; there are {len(all_packs)}")
|
||||
names = all_packs[n - 1]
|
||||
kilde = Path(args.kilde_root) / "kilde"
|
||||
index = _read_json(Path(args.index))
|
||||
text = "\n".join((kilde / name).read_text(encoding="utf-8") for name in names)
|
||||
lines = relevant_lines(index, text)
|
||||
parts = render_input(n, names, doc, lines, kilde)
|
||||
out_dir = work / "K-oppgaver"
|
||||
paths = [out_dir / f"pack-{n}-part-{i}.md" for i in range(1, len(parts) + 1)]
|
||||
if any(p.exists() for p in paths):
|
||||
raise RunError(f"pack {n}'s task parts exist in {out_dir}; refusing to overwrite")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
for p, text_part in zip(paths, parts):
|
||||
p.write_text(text_part, encoding="utf-8")
|
||||
log = _pack_log(work)
|
||||
rec = _record(log, n)
|
||||
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]), index_lines=len(lines),
|
||||
index_total=len(index["entries"]), index_ids=[str(e["id"]) for e in lines], parts=len(parts),
|
||||
chars=sum(len(p) for p in parts))
|
||||
_write_json(work / "K-work" / "packs.json", log)
|
||||
for p in paths:
|
||||
print(f"part: {p}")
|
||||
print(f"output: {work / 'K-fragments' / f'pack-{n}.json'}")
|
||||
print(f"index lines sent: {len(lines)} of {len(index['entries'])}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_accept(args: argparse.Namespace) -> int:
|
||||
work, n = Path(args.work), args.n
|
||||
doc = _read_json(Path(args.blocks))
|
||||
names = packs([str(x) for x in doc["order"]], args.size)[n - 1]
|
||||
kilde = Path(args.kilde_root) / "kilde"
|
||||
raw = _read_json(work / "K-fragments" / f"pack-{n}.json")
|
||||
fragments = raw.get("fragments") if isinstance(raw, dict) else None
|
||||
if not isinstance(fragments, list):
|
||||
raise RunError(f"pack {n}'s output has no fragments list")
|
||||
index_path = work / "K-work" / "index.json"
|
||||
index = _read_json(index_path)
|
||||
valid, rejected = validate(fragments, names, doc, kilde)
|
||||
acc_path = work / "K-work" / "accepted.json"
|
||||
acc: dict[str, Any] = (_read_json(acc_path) if acc_path.is_file()
|
||||
else {"schema": "okf-wiki/p1-accepted/1", "fragments": []})
|
||||
if any(a["pack"] == n for a in acc["fragments"]):
|
||||
raise RunError(f"pack {n} is already accepted")
|
||||
sent = set(_record(_pack_log(work), n).get("index_ids", []))
|
||||
not_sent: Counter[str] = Counter()
|
||||
accepted = 0
|
||||
for f in valid:
|
||||
outcome, ref = resolve(f, index)
|
||||
if outcome == "rejected":
|
||||
rejected[str(ref)] += 1
|
||||
continue
|
||||
if outcome == "existing":
|
||||
cid = str(ref)
|
||||
else:
|
||||
cid = _unique_id(index, f["concept"])
|
||||
index["entries"].append({"id": cid, "title": f["concept"], "aliases": [f["concept"]], "kind": outcome,
|
||||
"merge_of": ref, "origin": {"file": f["anchor"]["file"], "line": f["anchor"]["start"]},
|
||||
"tests": f.get("tests")})
|
||||
acc["fragments"].append({"pack": n, "position": f["position"], "file": f["anchor"]["file"],
|
||||
"start": f["anchor"]["start"], "end": f["anchor"]["end"], "section": f["section"],
|
||||
"text": f["text"].strip(), "quote": f["quote"], "quote_line": f["quote_line"],
|
||||
"version": f["version"], "concept_id": cid, "outcome": outcome})
|
||||
if cid not in sent:
|
||||
not_sent[outcome] += 1
|
||||
accepted += 1
|
||||
_write_json(index_path, index)
|
||||
_write_json(acc_path, acc)
|
||||
log = _pack_log(work)
|
||||
rec = _record(log, n)
|
||||
rec.update(proposed=len(fragments), accepted=accepted, rejected=dict(sorted(rejected.items())),
|
||||
not_sent={"total": sum(not_sent.values()), "by_outcome": dict(sorted(not_sent.items()))})
|
||||
_write_json(work / "K-work" / "packs.json", log)
|
||||
print(f"pack {n}: accepted {accepted} of {len(fragments)}")
|
||||
print(f" accepted with a concept not among the index lines sent: {sum(not_sent.values())}")
|
||||
for reason, count in sorted(rejected.items()):
|
||||
print(f" rejected {reason}: {count}")
|
||||
return 0
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
s = sub.add_parser("skeleton")
|
||||
s.add_argument("--kilde", required=True)
|
||||
s.add_argument("--out", required=True)
|
||||
for name in ("pack", "accept"):
|
||||
c = sub.add_parser(name)
|
||||
c.add_argument("n", type=int)
|
||||
c.add_argument("--blocks", required=True)
|
||||
c.add_argument("--kilde-root", required=True)
|
||||
c.add_argument("--work", required=True)
|
||||
c.add_argument("--size", type=int, default=5)
|
||||
if name == "pack":
|
||||
c.add_argument("--index", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "skeleton":
|
||||
out = Path(args.out)
|
||||
if out.exists():
|
||||
raise RunError(f"{out} exists; refusing to overwrite")
|
||||
index = skeleton(Path(args.kilde))
|
||||
_write_json(out, index)
|
||||
print(f"skeleton: {len(index['entries'])} entries -> {out}")
|
||||
return 0
|
||||
if args.command == "pack":
|
||||
return cmd_pack(args)
|
||||
return cmd_accept(args)
|
||||
except (RunError, OSError) as exc:
|
||||
print(f"kpack cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
361
evals/p1/measure.py
Normal file
361
evals/p1/measure.py
Normal file
|
|
@ -0,0 +1,361 @@
|
|||
"""The trial's measures without a reader (TRIAL, P1): audit record, coverage, structure. It calls no model.
|
||||
|
||||
audit --bundle B --root R --variant V --out FILE
|
||||
Runs the auditor in process and records, per variant, the exit status, findings per check, uncited
|
||||
sentences by cause (fence line, table row, link lead-in, prose), "yaml form" findings (a quote that YAML
|
||||
loaded as another type, a timestamp in the wrong form) and criterion 1 (N of N, claims.py's count).
|
||||
coverage --bundle B --root R --blocks F --built N [--pack-size K] --out FILE
|
||||
Blocks of the first N sources in the order with at least one body-cited source entry whose locator line
|
||||
lies inside the block (spans are not used), X of N blocks; per pack position with --pack-size.
|
||||
structure --bundle B --root R --blocks F [--skeleton I] --out FILE
|
||||
Pages, concepts per source (the earliest source in the order among a page's body-cited sources),
|
||||
skeleton entries without a page, duplicates (same name, alias or title Jaccard >= 0.6), merge
|
||||
candidates, islands, sources per page, version numbers in running text, concept tests 2 and 4,
|
||||
the share of quotes on a structural line and of code-supplied quotes (version sources left out), and the
|
||||
precision of each body-cited entry (line, block, section, document).
|
||||
|
||||
--root is the folder that contains kilde/. Before reporting, structure and coverage check that at least one quote
|
||||
stands at its locator under --root, because a wrong root makes every quote unreadable. Exit 0 = written, 2 = could
|
||||
not run (an existing output, or a variant already in the audit record, is never overwritten).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki import claims as claims_mod
|
||||
from okfwiki.audit import (
|
||||
FOOTNOTE_REF,
|
||||
LINK,
|
||||
LINK_ONLY,
|
||||
Document,
|
||||
audit,
|
||||
body_links,
|
||||
is_bundle_link,
|
||||
load_bundle,
|
||||
mask_code,
|
||||
prose_blocks,
|
||||
resolve,
|
||||
sentences,
|
||||
)
|
||||
from okfwiki.view import is_version_heading, related, versions
|
||||
from p1.assemble import structural
|
||||
from p1.kpack import MERGE_JACCARD, norm
|
||||
from p1.split import FENCE, is_meta
|
||||
|
||||
STRUCTURE_SCHEMA = "okf-wiki/p1-structure/1"
|
||||
RESULT_SCHEMA = "okf-wiki/p1-result/1"
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The measure cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Coverage:
|
||||
covered: int
|
||||
total: int
|
||||
per_position: dict[int, tuple[int, int, int]] = field(default_factory=dict)
|
||||
|
||||
|
||||
def _entries(doc: Document) -> dict[str, dict[str, Any]]:
|
||||
srcs = (doc.frontmatter or {}).get("sources")
|
||||
return {str(s["id"]): s for s in srcs if isinstance(s, dict) and "id" in s} if isinstance(srcs, list) else {}
|
||||
|
||||
|
||||
def cited(doc: Document) -> list[dict[str, Any]]:
|
||||
"""The source entries a body footnote uses, in label order of first use."""
|
||||
entries = _entries(doc)
|
||||
labels: list[str] = []
|
||||
for block in prose_blocks(doc):
|
||||
for label in FOOTNOTE_REF.findall(mask_code(block.text)):
|
||||
if label not in labels:
|
||||
labels.append(label)
|
||||
return [entries[x] for x in labels if x in entries]
|
||||
|
||||
|
||||
def _file(s: dict[str, Any]) -> str:
|
||||
return Path(str(s.get("resource", ""))).name
|
||||
|
||||
|
||||
def _line(s: dict[str, Any]) -> int | None:
|
||||
loc = s.get("x-locator")
|
||||
return loc["line"] if isinstance(loc, dict) and type(loc.get("line")) is int else None
|
||||
|
||||
|
||||
class _Lines:
|
||||
def __init__(self, root: Path) -> None:
|
||||
self.root = root
|
||||
self.cache: dict[str, list[str] | None] = {}
|
||||
|
||||
def get(self, name: str) -> list[str] | None:
|
||||
if name not in self.cache:
|
||||
p = self.root / "kilde" / name
|
||||
self.cache[name] = p.read_text(encoding="utf-8").split("\n") if p.is_file() else None
|
||||
return self.cache[name]
|
||||
|
||||
|
||||
def _check_root(concepts: list[Document], root: Path) -> None:
|
||||
lines = _Lines(root)
|
||||
seen = 0
|
||||
for d in concepts:
|
||||
for s in _entries(d).values():
|
||||
n, quote, text = _line(s), s.get("x-quote"), lines.get(_file(s))
|
||||
seen += 1
|
||||
if n and isinstance(quote, str) and text and n <= len(text) and quote in text[n - 1]:
|
||||
return
|
||||
if seen:
|
||||
raise RunError(f"no quote stands at its locator under {root}; is it the folder that contains kilde/?")
|
||||
|
||||
|
||||
def _block_of(blocks: list[dict[str, Any]], file: str, line: int) -> dict[str, Any] | None:
|
||||
return next((b for b in blocks if b["file"] == file and b["start"] <= line <= b["end"]), None)
|
||||
|
||||
|
||||
def coverage(bundle: Path, root: Path, blocks_doc: dict[str, Any], built: list[str],
|
||||
pack_size: int | None = None) -> Coverage:
|
||||
concepts = load_bundle(bundle, root).concepts
|
||||
_check_root(concepts, root)
|
||||
blocks = [b for b in blocks_doc["blocks"] if b["file"] in built]
|
||||
hit: set[tuple[str, int]] = set()
|
||||
for d in concepts:
|
||||
for s in cited(d):
|
||||
n = _line(s)
|
||||
b = _block_of(blocks, _file(s), n) if n else None
|
||||
if b is not None:
|
||||
hit.add((b["file"], b["start"]))
|
||||
result = Coverage(len(hit), len(blocks))
|
||||
if pack_size:
|
||||
names = [str(x) for x in blocks_doc["order"] if x in built]
|
||||
for pos in range(1, pack_size + 1):
|
||||
files = {n for i, n in enumerate(names) if i % pack_size == pos - 1}
|
||||
if files:
|
||||
mine = [b for b in blocks if b["file"] in files]
|
||||
result.per_position[pos] = (sum((b["file"], b["start"]) in hit for b in mine), len(mine), len(files))
|
||||
return result
|
||||
|
||||
|
||||
def _jaccard(a: str, b: str) -> float:
|
||||
x, y = set(a.split()), set(b.split())
|
||||
return len(x & y) / len(x | y) if x | y else 0.0
|
||||
|
||||
|
||||
def duplicates(concepts: list[Document]) -> list[tuple[str, str, str]]:
|
||||
info = []
|
||||
for d in concepts:
|
||||
fm = d.frontmatter or {}
|
||||
aliases = fm.get("x-aliases")
|
||||
info.append((d.rel, norm(str(fm.get("title", ""))),
|
||||
{norm(str(a)) for a in aliases} if isinstance(aliases, list) else set()))
|
||||
pairs: list[tuple[str, str, str]] = []
|
||||
for i, (a, na, aa) in enumerate(info):
|
||||
for b, nb, ab in info[i + 1:]:
|
||||
if na and na == nb:
|
||||
pairs.append((a, b, "norm"))
|
||||
elif na in ab or nb in aa:
|
||||
pairs.append((a, b, "alias"))
|
||||
elif _jaccard(na, nb) >= MERGE_JACCARD:
|
||||
pairs.append((a, b, "jaccard"))
|
||||
return pairs
|
||||
|
||||
|
||||
def _kind(d: Document) -> str:
|
||||
fm = d.frontmatter or {}
|
||||
if fm.get("x-merge-candidate"):
|
||||
return "merge-candidate"
|
||||
return {"Document map": "map", "Register entry": "register"}.get(str(fm.get("type", "")), "concept")
|
||||
|
||||
|
||||
def _precision(s: dict[str, Any], blocks: list[dict[str, Any]]) -> str:
|
||||
span = s.get("x-span")
|
||||
n = _line(s) or 0
|
||||
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (n, n)
|
||||
if a == z:
|
||||
return "line"
|
||||
inside = [b for b in blocks if b["file"] == _file(s) and b["end"] >= a and b["start"] <= z]
|
||||
if len(inside) == 1 and inside[0]["start"] <= a and z <= inside[0]["end"]:
|
||||
return "block"
|
||||
if inside and len({b["section"] for b in inside}) == 1:
|
||||
return "section"
|
||||
return "document"
|
||||
|
||||
|
||||
def structure(bundle: Path, root: Path, blocks_doc: dict[str, Any], order: list[str],
|
||||
skeleton: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
concepts = load_bundle(bundle, root).concepts
|
||||
_check_root(concepts, root)
|
||||
lines = _Lines(root)
|
||||
blocks = list(blocks_doc["blocks"])
|
||||
rank = {n: i for i, n in enumerate(order)}
|
||||
per_source: Counter[str] = Counter()
|
||||
incoming: Counter[str] = Counter()
|
||||
rels = {d.rel for d in concepts}
|
||||
for d in concepts:
|
||||
for _line_no, target in body_links(d):
|
||||
other = resolve(d.rel, target) if is_bundle_link(target) else None
|
||||
if other in rels and other != d.rel:
|
||||
incoming[str(other)] += 1
|
||||
quotes: Counter[str] = Counter()
|
||||
precision: Counter[str] = Counter()
|
||||
docs_per_page: list[int] = []
|
||||
test2 = test4 = 0
|
||||
for d in concepts:
|
||||
cs = cited(d)
|
||||
files = [_file(s) for s in cs if _file(s) in rank]
|
||||
if files:
|
||||
per_source[min(files, key=lambda f: rank[f])] += 1
|
||||
docs_per_page.append(len(set(_file(s) for s in cs)))
|
||||
title = str((d.frontmatter or {}).get("title", ""))
|
||||
test2 += not is_meta(title)
|
||||
test4 += incoming[d.rel] >= 2 or len(set(files)) >= 2
|
||||
for s in cs:
|
||||
if is_version_heading(s):
|
||||
continue
|
||||
quotes["entries"] += 1
|
||||
quotes["code"] += s.get("x-quote-by") == "code"
|
||||
text, n = lines.get(_file(s)), _line(s)
|
||||
quotes["structural"] += bool(text and n and n <= len(text) and structural(text[n - 1]))
|
||||
precision[_precision(s, blocks)] += 1
|
||||
rel_map = related(concepts)
|
||||
kinds = {d.rel: _kind(d) for d in concepts}
|
||||
islands: dict[str, Any] = {"total": sum(not v for v in rel_map.values())}
|
||||
for k in sorted(set(kinds.values())):
|
||||
islands[k] = sum(not rel_map[r] for r, kk in kinds.items() if kk == k)
|
||||
pages = {d.rel for d in concepts}
|
||||
unused = None
|
||||
if skeleton is not None:
|
||||
unused = sum(e.get("kind") == "skeleton" and f"{e['id']}.md" not in pages for e in skeleton["entries"])
|
||||
tests13 = None
|
||||
if skeleton is not None:
|
||||
new_ids = {e["id"] for e in skeleton["entries"] if e.get("kind") == "new" and e.get("tests")}
|
||||
tests13 = sum(f"{i}.md" in pages for i in new_ids)
|
||||
v = versions(bundle, root)
|
||||
dups = duplicates(concepts)
|
||||
return {
|
||||
"schema": STRUCTURE_SCHEMA,
|
||||
"pages": len(concepts),
|
||||
"page_kinds": dict(Counter(kinds.values())),
|
||||
"concepts_per_source": {n: per_source[n] for n in order if per_source[n]},
|
||||
"skeleton_unused": unused,
|
||||
"duplicates": {"pairs": len(dups), "by_rule": dict(Counter(r for _, _, r in dups)),
|
||||
"list": [list(p) for p in dups]},
|
||||
"merge_candidates": sum(k == "merge-candidate" for k in kinds.values()),
|
||||
"islands": islands,
|
||||
"documents_per_page": {"mean": round(sum(docs_per_page) / len(docs_per_page), 2) if docs_per_page else 0,
|
||||
"max": max(docs_per_page, default=0)},
|
||||
"version_numbers_in_text": {"flagged": len(v.flagged), "claims": v.total},
|
||||
"concept_tests": {"test2": test2, "test4": test4, "tests1and3_self_declared": tests13},
|
||||
"quotes": {"entries": quotes["entries"], "structural": quotes["structural"], "code": quotes["code"]},
|
||||
"precision": {k: precision[k] for k in ("line", "block", "section", "document")},
|
||||
}
|
||||
|
||||
|
||||
def _uncited_cause(sentence: str, block_text: str) -> str:
|
||||
s = sentence.strip()
|
||||
if FENCE.match(block_text): # the auditor's sentence text masks code spans, so a fence is read on the block
|
||||
return "fence line"
|
||||
if s.startswith("|"):
|
||||
return "table row"
|
||||
if LINK.search(block_text) and not LINK_ONLY.match(mask_code(block_text)):
|
||||
return "link lead-in"
|
||||
return "prose"
|
||||
|
||||
|
||||
def audit_record(bundle: Path, root: Path, variant: str, out: Path) -> dict[str, Any]:
|
||||
record_all: dict[str, Any] = (json.loads(out.read_text(encoding="utf-8")) if out.is_file()
|
||||
else {"schema": RESULT_SCHEMA, "variants": {}})
|
||||
if variant in record_all["variants"]:
|
||||
raise RunError(f"{out} already records variant {variant}")
|
||||
b = load_bundle(bundle, root)
|
||||
found = audit(bundle, root)
|
||||
uncited: Counter[str] = Counter()
|
||||
for d in b.concepts:
|
||||
for block in prose_blocks(d):
|
||||
if LINK_ONLY.match(mask_code(block.text)):
|
||||
continue
|
||||
for s in sentences(block.text):
|
||||
if not FOOTNOTE_REF.search(s):
|
||||
uncited[_uncited_cause(s, block.text)] += 1
|
||||
yaml_form = 0
|
||||
for d in b.concepts:
|
||||
for entry in _entries(d).values():
|
||||
q = entry.get("x-quote")
|
||||
yaml_form += q is not None and not isinstance(q, str)
|
||||
yaml_form += sum(f.check == "timestamp-form" for f in found)
|
||||
listed = claims_mod.extract(bundle, root)
|
||||
ok, total = claims_mod.criterion_1(listed)["total"]
|
||||
rec = {"exit": 1 if found else 0, "findings": dict(sorted(Counter(f.check for f in found).items())),
|
||||
"uncited": dict(sorted(uncited.items())), "yaml_form": yaml_form, "criterion_1": [ok, total]}
|
||||
record_all["variants"][variant] = rec
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(record_all, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
return rec
|
||||
|
||||
|
||||
def _write_new(path: Path, data: Any) -> None:
|
||||
if path.exists():
|
||||
raise RunError(f"{path} exists; refusing to overwrite")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
a = sub.add_parser("audit")
|
||||
a.add_argument("--bundle", required=True)
|
||||
a.add_argument("--root", required=True)
|
||||
a.add_argument("--variant", required=True)
|
||||
a.add_argument("--out", required=True)
|
||||
c = sub.add_parser("coverage")
|
||||
s = sub.add_parser("structure")
|
||||
for x in (c, s):
|
||||
x.add_argument("--bundle", required=True)
|
||||
x.add_argument("--root", required=True)
|
||||
x.add_argument("--blocks", required=True)
|
||||
x.add_argument("--out", required=True)
|
||||
c.add_argument("--built", type=int, required=True)
|
||||
c.add_argument("--pack-size", type=int)
|
||||
s.add_argument("--skeleton")
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
bundle, root = Path(args.bundle), Path(args.root)
|
||||
if not bundle.is_dir():
|
||||
raise RunError(f"the bundle {bundle} is not a directory")
|
||||
if args.command == "audit":
|
||||
rec = audit_record(bundle, root, args.variant, Path(args.out))
|
||||
print(f"{args.variant}: audit exit {rec['exit']}, criterion 1 {rec['criterion_1'][0]} of "
|
||||
f"{rec['criterion_1'][1]}, findings {rec['findings']}")
|
||||
return 0
|
||||
blocks_doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
order = [str(x) for x in blocks_doc["order"]]
|
||||
if args.command == "coverage":
|
||||
cov = coverage(bundle, root, blocks_doc, order[:args.built], args.pack_size)
|
||||
_write_new(Path(args.out), {"schema": "okf-wiki/p1-coverage/1", "covered": cov.covered, "total": cov.total,
|
||||
"built": args.built,
|
||||
"per_position": {str(k): list(v) for k, v in cov.per_position.items()}})
|
||||
print(f"coverage: {cov.covered} of {cov.total} blocks")
|
||||
return 0
|
||||
skeleton = json.loads(Path(args.skeleton).read_text(encoding="utf-8")) if args.skeleton else None
|
||||
data = structure(bundle, root, blocks_doc, order, skeleton)
|
||||
_write_new(Path(args.out), data)
|
||||
print(f"structure: {data['pages']} pages, {data['duplicates']['pairs']} duplicate pairs, "
|
||||
f"{data['islands']['total']} islands")
|
||||
return 0
|
||||
except (RunError, OSError, json.JSONDecodeError, KeyError) as exc:
|
||||
print(f"measure cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
185
evals/p1/sidebyside.py
Normal file
185
evals/p1/sidebyside.py
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
"""The side-by-side page (TRIAL, P1): the same concepts in every build, every source link checked. It calls no model.
|
||||
|
||||
render --begreper FILE --col LABEL=DIR ... [--col LABEL=-] --kilde-root DIR --out FILE
|
||||
One self-contained HTML page: one row per concept in begreper.txt (name, a tab, aliases joined by ;), one
|
||||
column per build. A cell shows the pages whose normalised title or alias equals the concept's (for the L4
|
||||
column: whose normalised title or description contains it), each with its body and its sources
|
||||
(file › line or span). An empty cell says why: "ikke bygget" (a build given as -, stopped), "ikke i
|
||||
registeret" (D has no register page for it), "ingen side" (built, no page).
|
||||
check --html FILE --kilde-root DIR
|
||||
A source link resolves when its file exists under kilde/ and its line or span lies inside the file.
|
||||
Prints "resolved X of N"; exit 1 when X < N.
|
||||
|
||||
Exit 0 = done, 1 = an unresolved link (check), 2 = could not run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import load_bundle
|
||||
from okfwiki.view import esc
|
||||
from p1.kpack import norm
|
||||
from p1.measure import cited
|
||||
|
||||
Concept = tuple[str, tuple[str, ...]]
|
||||
SRC = re.compile(r'<a class="src" href="[^"]*" data-file="([^"]+)" data-start="(\d+)" data-end="(\d+)">')
|
||||
STYLE = """
|
||||
:root { --bg: #fdfdfb; --fg: #1d1d1b; --muted: #6b6b66; --line: #ddd9cf; --cell: #ffffff; }
|
||||
@media (prefers-color-scheme: dark) { :root { --bg: #1b1b1a; --fg: #ecebe6; --muted: #a3a29b; --line: #3a3934;
|
||||
--cell: #242422; } }
|
||||
body { background: var(--bg); color: var(--fg); font: 15px/1.5 system-ui, sans-serif; margin: 16px; }
|
||||
table { border-collapse: collapse; width: 100%; table-layout: fixed; }
|
||||
th, td { border: 1px solid var(--line); vertical-align: top; padding: 8px; background: var(--cell); }
|
||||
th.concept-name { width: 10em; text-align: left; }
|
||||
.empty { color: var(--muted); font-style: italic; }
|
||||
.page h3 { margin: 0 0 4px; font-size: 15px; }
|
||||
.sources { color: var(--muted); font-size: 13px; }
|
||||
.wrap { overflow-x: auto; }
|
||||
"""
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
def read_begreper(path: Path) -> list[Concept]:
|
||||
out: list[Concept] = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
if line.strip():
|
||||
name, _, aliases = line.partition("\t")
|
||||
out.append((name.strip(), tuple(a.strip() for a in aliases.split(";") if a.strip()) or (name.strip(),)))
|
||||
return out
|
||||
|
||||
|
||||
def _l4_norm(text: str) -> str:
|
||||
return f" {norm(text.replace('-', ' '))} "
|
||||
|
||||
|
||||
def find(bundle: Path, concept: Concept, l4: bool = False) -> list[str]:
|
||||
wanted = {norm(n) for n in (concept[0], *concept[1])} - {""}
|
||||
found: list[str] = []
|
||||
for d in load_bundle(bundle, bundle).concepts:
|
||||
fm = d.frontmatter or {}
|
||||
if l4:
|
||||
hay = _l4_norm(str(fm.get("title", ""))) + _l4_norm(str(fm.get("description", "")))
|
||||
if any(f" {norm(w.replace('-', ' '))} " in hay for w in wanted):
|
||||
found.append(d.rel)
|
||||
continue
|
||||
aliases = fm.get("x-aliases")
|
||||
names = {norm(str(fm.get("title", "")))} | ({norm(str(a)) for a in aliases} if isinstance(aliases, list)
|
||||
else set())
|
||||
if names & wanted:
|
||||
found.append(d.rel)
|
||||
return found
|
||||
|
||||
|
||||
def _sources_html(entries: list[dict[str, Any]], kilde: Path, out_dir: Path) -> str:
|
||||
items = []
|
||||
for s in entries:
|
||||
file = Path(str(s.get("resource", ""))).name
|
||||
loc = s.get("x-locator")
|
||||
line = loc.get("line") if isinstance(loc, dict) else None
|
||||
span = s.get("x-span")
|
||||
a, z = (span[0], span[1]) if isinstance(span, list) and len(span) == 2 else (line, line)
|
||||
if not isinstance(a, int) or not isinstance(z, int):
|
||||
continue
|
||||
href = os.path.relpath(kilde / file, out_dir).replace(os.sep, "/") + f"#L{a}"
|
||||
where = f"{a}" if a == z else f"{a}–{z}"
|
||||
items.append(f'<li><a class="src" href="{esc(href)}" data-file="{esc(file)}" data-start="{a}" '
|
||||
f'data-end="{z}">{esc(file)} › {where}</a></li>')
|
||||
return '<ul class="sources">' + "".join(items) + "</ul>"
|
||||
|
||||
|
||||
def _cell(bundle: Path | None, label: str, concept: Concept, kilde_root: Path, out_dir: Path) -> str:
|
||||
if bundle is None:
|
||||
return '<td class="cell empty">ikke bygget</td>'
|
||||
rels = find(bundle, concept, l4=label == "L4")
|
||||
if not rels:
|
||||
return f'<td class="cell empty">{"ikke i registeret" if label == "D" else "ingen side"}</td>'
|
||||
docs = {d.rel: d for d in load_bundle(bundle, kilde_root).concepts}
|
||||
parts = []
|
||||
for rel in rels:
|
||||
d = docs[rel]
|
||||
fm = d.frontmatter or {}
|
||||
srcs = fm.get("sources")
|
||||
entries = cited(d) or ([s for s in srcs if isinstance(s, dict)] if isinstance(srcs, list) else [])
|
||||
paras = [p.strip() for p in re.split(r"\n\s*\n", d.body) if p.strip() and not p.strip().startswith("# ")]
|
||||
body = "".join(f"<p>{esc(p)}</p>" for p in paras if not p.startswith("[^"))
|
||||
parts.append(f'<div class="page"><h3>{esc(str(fm.get("title", rel)))} <small>({esc(rel)})</small></h3>'
|
||||
f"{body}{_sources_html(entries, kilde_root / 'kilde', out_dir)}</div>")
|
||||
return '<td class="cell">' + "".join(parts) + "</td>"
|
||||
|
||||
|
||||
def render(begreper: list[Concept], columns: list[tuple[str, Path | None]], kilde_root: Path, out: Path) -> None:
|
||||
out_dir = out.parent
|
||||
head = "".join(f"<th>{esc(label)}</th>" for label, _ in columns)
|
||||
rows = []
|
||||
for concept in begreper:
|
||||
cells = "".join(_cell(b, label, concept, kilde_root, out_dir) for label, b in columns)
|
||||
rows.append(f'<tr class="concept"><th class="concept-name">{esc(concept[0])}</th>{cells}</tr>')
|
||||
html = (f'<!doctype html><html lang="no"><head><meta charset="utf-8">'
|
||||
f'<meta name="viewport" content="width=device-width, initial-scale=1"><title>P1 side om side</title>'
|
||||
f"<style>{STYLE}</style></head><body><h1>P1: tre begreper, side om side</h1>"
|
||||
f'<div class="wrap"><table><tr><th class="concept-name">Begrep</th>{head}</tr>{"".join(rows)}'
|
||||
f"</table></div></body></html>\n")
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(html, encoding="utf-8")
|
||||
|
||||
|
||||
def check(html_path: Path, kilde_root: Path) -> tuple[int, int]:
|
||||
html = html_path.read_text(encoding="utf-8")
|
||||
resolved = total = 0
|
||||
sizes: dict[str, int | None] = {}
|
||||
for file, a, z in SRC.findall(html):
|
||||
total += 1
|
||||
if file not in sizes:
|
||||
p = kilde_root / "kilde" / file
|
||||
sizes[file] = len(p.read_text(encoding="utf-8").splitlines()) if p.is_file() else None
|
||||
n = sizes[file]
|
||||
resolved += n is not None and 1 <= int(a) <= int(z) <= n
|
||||
return resolved, total
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
r = sub.add_parser("render")
|
||||
r.add_argument("--begreper", required=True)
|
||||
r.add_argument("--col", action="append", required=True, help="LABEL=DIR, or LABEL=- for a stopped build")
|
||||
r.add_argument("--kilde-root", required=True)
|
||||
r.add_argument("--out", required=True)
|
||||
c = sub.add_parser("check")
|
||||
c.add_argument("--html", required=True)
|
||||
c.add_argument("--kilde-root", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "render":
|
||||
columns: list[tuple[str, Path | None]] = []
|
||||
for col in args.col:
|
||||
label, _, path = col.partition("=")
|
||||
if not label or not path:
|
||||
raise RunError(f"--col {col!r} is not LABEL=DIR")
|
||||
columns.append((label, None if path == "-" else Path(path)))
|
||||
render(read_begreper(Path(args.begreper)), columns, Path(args.kilde_root), Path(args.out))
|
||||
print(f"side by side: {len(columns)} columns -> {args.out}")
|
||||
return 0
|
||||
resolved, total = check(Path(args.html), Path(args.kilde_root))
|
||||
print(f"resolved {resolved} of {total}")
|
||||
return 0 if total and resolved == total else 1
|
||||
except (RunError, OSError) as exc:
|
||||
print(f"sidebyside cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
328
evals/p1/split.py
Normal file
328
evals/p1/split.py
Normal file
|
|
@ -0,0 +1,328 @@
|
|||
"""Split the trial's sources into a fixed order and into blocks with line spans (TRIAL, P1). It calls no model.
|
||||
|
||||
order --manifest M --l4 L4DIR --out FILE
|
||||
The 19 names of the cut (gates/wholedoc.manifest.json cut.names) in the trial's fixed order: docs pages by
|
||||
name, then release notes by version. Refuses when the names differ from the files the L4 cut cites.
|
||||
blocks --kilde DIR --order FILE --out FILE
|
||||
Every block of every source in the order: a paragraph, one list item with its continuation lines, one
|
||||
table or one fenced code block, with its 1-based inclusive line span, its section (the heading path) and,
|
||||
for a release note, its version. Heading lines are section markers, not blocks.
|
||||
total --kilde DIR
|
||||
The number of blocks in every file under DIR (the denominator of the 599-source projection).
|
||||
pick --blocks FILE --kilde DIR [--focus NAME] --out FILE
|
||||
The three concepts of the side-by-side page, picked from the source text before any build: the focus page's
|
||||
heading with the most blocks, the term in the most release notes, and the term in at least two docs pages and
|
||||
two release notes with the highest document count; a document counts for a term only where the term stands
|
||||
as a term (backticked or a capitalised phrase), never as a word in prose. One line per concept: name, a tab,
|
||||
aliases joined by ;.
|
||||
|
||||
Exit 0 = written, 2 = could not run (an existing output is never overwritten).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from okfwiki.audit import HEADING, LIST_ITEM, read_document
|
||||
|
||||
SCHEMA = "okf-wiki/p1-blocks/1"
|
||||
RELEASE = re.compile(r"^release-v(\d+(?:\.\d+)*)\.md$")
|
||||
VERSION_HEADING = re.compile(r"^v?(\d+(?:\.\d+){2,})$")
|
||||
FENCE = re.compile(r"^\s*(```|~~~)")
|
||||
# Concept test 2 of the brief (Google's reference agent), plus the generic section names of docs pages.
|
||||
META_STOP = frozenset({
|
||||
"overview", "introduction", "getting started", "quickstart", "tutorial", "walkthrough", "release notes",
|
||||
"changelog", "roadmap", "faq", "prerequisites", "next steps", "before you begin", "troubleshooting", "see also",
|
||||
"related resources", "learn more",
|
||||
})
|
||||
CODE_TERM = re.compile(r"`([^`\n]{3,40})`")
|
||||
CAPITAL_PHRASE = re.compile(r"\b[A-Z][\w-]*(?:[ \t]+[A-Z][\w-]*){1,3}\b")
|
||||
SENTENCE_START = re.compile(r"(?:^|[.!?:]\s+|^\s*(?:[-*+]|\d+[.)])\s+)$")
|
||||
|
||||
|
||||
class RunError(Exception):
|
||||
"""The command cannot run as asked; nothing was written."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SBlock:
|
||||
file: str
|
||||
start: int
|
||||
end: int
|
||||
kind: str # paragraph | item | table | code
|
||||
section: str
|
||||
version: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Heading:
|
||||
file: str
|
||||
line: int
|
||||
level: int
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Picked:
|
||||
name: str
|
||||
aliases: tuple[str, ...]
|
||||
|
||||
|
||||
def is_meta(title: str) -> bool:
|
||||
return title.strip().lower() in META_STOP
|
||||
|
||||
|
||||
def version_key(name: str) -> tuple[int, ...]:
|
||||
m = RELEASE.match(name)
|
||||
if m is None:
|
||||
raise RunError(f"{name} is not a release note name")
|
||||
return tuple(int(p) for p in m.group(1).split("."))
|
||||
|
||||
|
||||
def order(names: list[str]) -> list[str]:
|
||||
docs = sorted(n for n in names if n.startswith("docs-"))
|
||||
releases = [n for n in names if n.startswith("release-")]
|
||||
other = [n for n in names if n not in docs and n not in releases]
|
||||
if other:
|
||||
raise RunError(f"names that are neither docs pages nor release notes: {other}")
|
||||
return docs + sorted(releases, key=version_key)
|
||||
|
||||
|
||||
def headings_of(lines: list[str]) -> list[tuple[int, int, str]]:
|
||||
"""(line, level, text) of every heading outside fenced code."""
|
||||
out: list[tuple[int, int, str]] = []
|
||||
fence: str | None = None
|
||||
for n, raw in enumerate(lines, 1):
|
||||
f = FENCE.match(raw)
|
||||
if f:
|
||||
fence = None if fence == f.group(1) else (fence or f.group(1))
|
||||
continue
|
||||
if fence is None:
|
||||
m = HEADING.match(raw)
|
||||
if m:
|
||||
out.append((n, len(m.group(1)), (m.group(2) or "").strip()))
|
||||
return out
|
||||
|
||||
|
||||
def blocks(path: Path) -> list[SBlock]:
|
||||
lines = path.read_text(encoding="utf-8").split("\n")
|
||||
name = path.name
|
||||
version: str | None = None
|
||||
if name.startswith("release-"):
|
||||
for _n, _level, text in headings_of(lines):
|
||||
m = VERSION_HEADING.match(text)
|
||||
if m:
|
||||
version = m.group(1)
|
||||
break
|
||||
out: list[SBlock] = []
|
||||
stack: list[tuple[int, str]] = []
|
||||
kind: str | None = None
|
||||
start = 0
|
||||
fence: str | None = None
|
||||
|
||||
def close(end: int) -> None:
|
||||
nonlocal kind
|
||||
if kind is not None:
|
||||
out.append(SBlock(name, start, end, kind, " › ".join(t for _, t in stack), version))
|
||||
kind = None
|
||||
|
||||
for n, raw in enumerate(lines, 1):
|
||||
f = FENCE.match(raw)
|
||||
if fence is not None:
|
||||
if f and f.group(1) == fence:
|
||||
fence = None
|
||||
close(n)
|
||||
continue
|
||||
if f:
|
||||
close(n - 1)
|
||||
fence, kind, start = f.group(1), "code", n
|
||||
continue
|
||||
if not raw.strip():
|
||||
close(n - 1)
|
||||
continue
|
||||
m = HEADING.match(raw)
|
||||
if m:
|
||||
close(n - 1)
|
||||
level = len(m.group(1))
|
||||
stack = [(lv, t) for lv, t in stack if lv < level] + [(level, (m.group(2) or "").strip())]
|
||||
continue
|
||||
if raw.lstrip().startswith("|"):
|
||||
if kind != "table":
|
||||
close(n - 1)
|
||||
kind, start = "table", n
|
||||
continue
|
||||
if LIST_ITEM.match(raw):
|
||||
close(n - 1)
|
||||
kind, start = "item", n
|
||||
continue
|
||||
if kind in (None, "table"):
|
||||
close(n - 1)
|
||||
kind, start = "paragraph", n
|
||||
if fence is not None:
|
||||
close(len(lines))
|
||||
else:
|
||||
last = len(lines)
|
||||
while last > 0 and not lines[last - 1].strip():
|
||||
last -= 1
|
||||
close(last)
|
||||
return out
|
||||
|
||||
|
||||
def docs_headings(kilde: Path) -> list[Heading]:
|
||||
out: list[Heading] = []
|
||||
for path in sorted(kilde.glob("docs-*.md")):
|
||||
for n, level, text in headings_of(path.read_text(encoding="utf-8").split("\n")):
|
||||
if level <= 3 and text:
|
||||
out.append(Heading(path.name, n, level, text))
|
||||
return out
|
||||
|
||||
|
||||
def blocks_document(kilde: Path, names: list[str]) -> dict[str, Any]:
|
||||
found: list[dict[str, Any]] = []
|
||||
for name in names:
|
||||
path = kilde / name
|
||||
if not path.is_file():
|
||||
raise RunError(f"{path} is not a file")
|
||||
found.extend(asdict(b) for b in blocks(path))
|
||||
return {"schema": SCHEMA, "order": names, "blocks": found}
|
||||
|
||||
|
||||
def blocks_total(kilde: Path) -> int:
|
||||
return sum(len(blocks(p)) for p in sorted(kilde.glob("*.md")))
|
||||
|
||||
|
||||
def terms(text: str) -> list[str]:
|
||||
"""Backticked spans of 3-40 characters and capitalised phrases of 2-4 words, a sentence-initial word dropped."""
|
||||
found: list[str] = [t for t in (m.group(1).strip() for m in CODE_TERM.finditer(text)) if 3 <= len(t) <= 40]
|
||||
masked = CODE_TERM.sub(lambda m: " " * len(m.group(0)), text)
|
||||
for raw in masked.split("\n"):
|
||||
for m in CAPITAL_PHRASE.finditer(raw):
|
||||
words = m.group(0).split()
|
||||
if SENTENCE_START.search(raw[:m.start()]):
|
||||
words = words[1:]
|
||||
if len(words) >= 2:
|
||||
found.append(" ".join(words))
|
||||
return list(dict.fromkeys(t for t in found if t))
|
||||
|
||||
|
||||
def occurs(term: str, text: str) -> bool:
|
||||
return re.search(r"(?<!\w)" + re.escape(term) + r"(?!\w)", text, re.IGNORECASE) is not None
|
||||
|
||||
|
||||
def pick(blocks_doc: dict[str, Any], kilde: Path, focus: str = "docs-en-permission-modes.md") -> list[Picked]:
|
||||
names = [str(n) for n in blocks_doc["order"]]
|
||||
if focus not in names:
|
||||
raise RunError(f"the focus page {focus} is not in the order")
|
||||
texts = {n: (kilde / n).read_text(encoding="utf-8") for n in names}
|
||||
per_section: dict[str, int] = {}
|
||||
for b in blocks_doc["blocks"]:
|
||||
if b["file"] == focus and b["section"].count(" › ") >= 1:
|
||||
per_section[b["section"]] = per_section.get(b["section"], 0) + 1
|
||||
candidates = [(count, section.split(" › ")[-1]) for section, count in per_section.items()
|
||||
if not is_meta(section.split(" › ")[-1])]
|
||||
if not candidates:
|
||||
raise RunError(f"{focus} has no heading below its title that passes test 2")
|
||||
first = sorted(candidates, key=lambda c: (-c[0], c[1]))[0][1]
|
||||
pool = list(dict.fromkeys(t for n in names for t in terms(texts[n])))
|
||||
docs = [n for n in names if n.startswith("docs-")]
|
||||
releases = [n for n in names if n.startswith("release-")]
|
||||
|
||||
as_terms = {n: set(terms(texts[n])) for n in names}
|
||||
|
||||
def count(term: str, files: list[str]) -> int:
|
||||
return sum(term in as_terms[f] for f in files) # where it stands as a term, not as a word in prose
|
||||
|
||||
taken = {first.lower()}
|
||||
second_pool = [t for t in pool if t.lower() not in taken]
|
||||
if not second_pool:
|
||||
raise RunError("no term for the second concept")
|
||||
second = sorted(second_pool, key=lambda t: (-count(t, releases), t))[0]
|
||||
taken.add(second.lower())
|
||||
third_pool = [t for t in pool if t.lower() not in taken and count(t, docs) >= 2 and count(t, releases) >= 2]
|
||||
if not third_pool:
|
||||
raise RunError("no term in two docs pages and two release notes for the third concept")
|
||||
third = sorted(third_pool, key=lambda t: (-count(t, names), t))[0]
|
||||
return [Picked(first, (first,)), Picked(second, (second,)), Picked(third, (third,))]
|
||||
|
||||
|
||||
def _write_new(path: Path, text: str) -> None:
|
||||
if path.exists():
|
||||
raise RunError(f"{path} exists; refusing to overwrite")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
def l4_cited(l4: Path) -> set[str]:
|
||||
cited: set[str] = set()
|
||||
for page in sorted(l4.glob("*.md")):
|
||||
fm = read_document(page, page.name).frontmatter or {}
|
||||
srcs = fm.get("sources")
|
||||
for s in srcs if isinstance(srcs, list) else []:
|
||||
res = s.get("resource") if isinstance(s, dict) else None
|
||||
if isinstance(res, str) and res:
|
||||
cited.add(Path(res).name)
|
||||
return cited
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
o = sub.add_parser("order")
|
||||
o.add_argument("--manifest", required=True)
|
||||
o.add_argument("--l4", required=True)
|
||||
o.add_argument("--out", required=True)
|
||||
b = sub.add_parser("blocks")
|
||||
b.add_argument("--kilde", required=True)
|
||||
b.add_argument("--order", required=True)
|
||||
b.add_argument("--out", required=True)
|
||||
t = sub.add_parser("total")
|
||||
t.add_argument("--kilde", required=True)
|
||||
k = sub.add_parser("pick")
|
||||
k.add_argument("--blocks", required=True)
|
||||
k.add_argument("--kilde", required=True)
|
||||
k.add_argument("--focus", default="docs-en-permission-modes.md")
|
||||
k.add_argument("--out", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "order":
|
||||
manifest = json.loads(Path(args.manifest).read_text(encoding="utf-8"))
|
||||
names = manifest.get("cut", {}).get("names")
|
||||
if not isinstance(names, list) or not names:
|
||||
raise RunError(f"{args.manifest} records no cut names")
|
||||
cited = l4_cited(Path(args.l4))
|
||||
if set(map(str, names)) != cited:
|
||||
raise RunError(f"the manifest's {len(names)} names differ from the {len(cited)} files the L4 cut cites: "
|
||||
f"only in manifest {sorted(set(names) - cited)}, only in L4 {sorted(cited - set(names))}")
|
||||
fixed = order([str(n) for n in names])
|
||||
_write_new(Path(args.out), "\n".join(fixed) + "\n")
|
||||
print(f"order: {len(fixed)} sources -> {args.out}")
|
||||
elif args.command == "blocks":
|
||||
names = Path(args.order).read_text(encoding="utf-8").split()
|
||||
doc = blocks_document(Path(args.kilde), names)
|
||||
_write_new(Path(args.out), json.dumps(doc, ensure_ascii=False, indent=1) + "\n")
|
||||
print(f"blocks: {len(doc['blocks'])} in {len(names)} sources -> {args.out}")
|
||||
elif args.command == "total":
|
||||
print(f"blocks in every file of {args.kilde}: {blocks_total(Path(args.kilde))}")
|
||||
else:
|
||||
doc = json.loads(Path(args.blocks).read_text(encoding="utf-8"))
|
||||
picked = pick(doc, Path(args.kilde), args.focus)
|
||||
_write_new(Path(args.out), "".join(f"{p.name}\t{';'.join(p.aliases)}\n" for p in picked))
|
||||
print(f"pick: {len(picked)} concepts -> {args.out}")
|
||||
except (RunError, OSError, json.JSONDecodeError) as exc:
|
||||
print(f"split cannot run: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Add a link
Reference in a new issue