"""The repair command: one fragment of a built collection repaired or removed, one link record taken out or moved, the helper's task, and the measurement of what a repair touched. No model call, no network, no program started (CLAUDE.md rule 1). Order 34, contracts K18 and K19; the contract below is the order's, point for point, in English. Usage: python3 -m okfwiki.krepair list --claims CLAIMS --verdicts VERDICTS --out LIST [--ids IDS] python3 -m okfwiki.krepair task --list LIST --claims CLAIMS --pairs PAIRS --work DIR --blocks FILE --sources DIR --agent FILE --out-dir DIR python3 -m okfwiki.krepair fix --answers FILE --bundle WIKI --work DIR --blocks FILE --sources DIR --links FILE python3 -m okfwiki.krepair unlink --links FILE --page P --sentence KEY --anchor A --at N [--work DIR --bundle WIKI] python3 -m okfwiki.krepair relink --links FILE --page P --sentence KEY --anchor A --at N --to TARGET --bundle WIKI [--work DIR] python3 -m okfwiki.krepair linkset --list LIST --bundle WIKI --out-dir DIR [--claims CLAIMS] python3 -m okfwiki.krepair diff --old WIKI --new WIKI --sources DIR --repairs FILE --out DIR Exit 0 done; 1 refused or not equal, with one line per reason of the form `refused: `; 2 cannot run (also for an input the code did not foresee, with its traceback). The contract: - The list in is K17's schema (okf-wiki/repair-list/1); task reads only `claims`, linkset only `links`. Both refuse a list with an unknown top-level key, without one of K17's keys, of another schema, or with a `claims_sha256` that is not the sha256 of the claim list given (task --claims; linkset with --claims, which K18 leaves out and this command takes optionally), and task a claim the claim list lacks or shows with another `excerpt_sha256`: refusal code `stale-list`. - list --claims CLAIMS --verdicts VERDICTS --out LIST [--ids IDS]: the list from a reading (v1.2, part 1; before it only the repair gate rendered one). Every claim the reading covered (all of CLAIMS, or those of --ids, the file claims calls --ids reads) that does not have exactly one verdict, `supported`, is listed as claims score counts it: `not-supported`, or `no-verdict` when the reader gave it none. Each row has the claim's id, `excerpt_sha256`, page, line, provenance, its excerpts as `sources` and origin `reader`; K17's `kind` is left out (task does not read it). `pages`, `pairs` and `links` are empty. A verdict for a claim CLAIMS lacks, or an --out that exists, cannot run (exit 2), and nothing is written. - task: the helper's tasks. The unit is the fragment, keyed by (`concept_id`, the `s...` footnote): the footnote alone is not unique (kassemble.label takes neither text nor concept; measured 20.10.2026 in b5-18a/work: 47 keys shared by 95 records in different concepts, 0 shared pairs of concept and key). The claims of the list that come from one fragment stand together, and a task holds at most TASK_FRAGMENTS fragments from one page. Per fragment: the page's title, every sentence of the fragment (those on the list marked with the list's reason: the reader did not accept it, or the operator's category), the excerpts the sentences show, the fragment's quote and context quotes, and the block the fragment stands in with numbered lines. The parts are at most kpack.PART_CHARS characters. A fragment whose section with the whole block is over the room of a part is shown with the block's lines from EXCERPT_LINES before its quote's line to EXCERPT_LINES after it, within the block, under a line that says so, and with its context quote lines that fall outside them on their own (order 35 A2; the operator 21.10.2026, PLAN.md § Rammer point 8); fix still checks against the whole block. A fragment whose part is larger even so exits 2 with the fragment named. The instructions and the answer contract stand in the task (INSTRUCTIONS, as the reader's in okfwiki/reader.py), so the agent file is thin and a change reaches the helper without a restart (CLAUDE.md). task refuses when the agent file or the instructions share a run of eight words with a claim, a quote or the block text in the task (`shared-words`). Each task gets a part list (`.parts.json`: the parts it may read and the one answer path it writes), which the breach count reads (okfwiki.cost isolation --line repair). task writes only in --out-dir and never over a task already written. - The answer (okf-wiki/repair-answers/1) is keyed by (`concept_id`, the `s...` footnote): replace with one to three new fragments (`text`, `quote`, `anchor` {file, start, end}, `provenance`, and optionally `context` as a list of quote strings, as kpack.with_context takes them), or remove. Code sets `concept` from the old record and `status` to "existing" (the concept exists and is never decided again), so kpack.validate asks for no `tests`, also when the old record made the concept; the record keeps the old record's `outcome`. - fix: every fragment of the answer is found against the same built collection and the present accepted.json before anything is written: each key must be (`concept_id`, kassemble.label) of exactly one record of accepted.json (`unknown-key`, `ambiguous-key`) and stand as a footnote on that page of the collection (`stale-bundle`: a collection built before the last repair has old keys and is refused so); two answers for one key are `duplicate-answer`. New fragments are checked with kpack.validate against their source's blocks alone (the quote verbatim on a line, the mark from PROVENANCE; its reasons are the refusal codes, `anchor-outside-pack` for an anchor in another source) and get their context from kpack.with_context; when the answer gives no `context`, the fragment keeps the old record's context quotes (records keep them as {line, quote, section}; they are given to with_context as their `quote` values) where they still stand on a heading of the path or a line of the section, and those that do not are listed. A new fragment takes the old record's place (same `concept_id`, `pack`, `position` and order). A new footnote that another fragment of the concept has, or that two new fragments share, is `label-taken`: the key stays unique. All or nothing: one refused fragment, and nothing is written. - The sentences that move: a sentence key ends in `-`, the occurrence of the same text on the page. When a repair changes another sentence's key (two equal sentences become one, or a new one equals one that stands), the link records on that sentence move to the new key, and its claim is logged as changed (it is read again). - The links on a repaired sentence: a link record (klink's {page, sentence, anchor, at, target}) on a sentence that is repaired moves to the new sentence's key when the anchor text stands in the new sentence with the same occurrence number (`at` counted again from that occurrence, outside code, as a whole word); the i-th sentence of the old fragment is paired with the i-th sentence of its new fragments. Otherwise, and when the sentence is removed, the record is taken out and listed; so is a record on or into a page that goes. Code never places a new link (links are made by the model, CLAUDE.md). - unlink --links FILE --page P --sentence KEY --anchor A --at N, and relink ... --to TARGET --bundle WIKI: take out one link record, or give it a new target; the record must stand in the file exactly so (`link-not-found`), and the target must be a concept page that exists (`target-missing`), not the record's own page (`self-link`) and not a target the sentence already links to (`duplicate-link`). No other record changes. With --work (and --bundle, which unlink then needs), the write is logged like a fix's, so diff can measure it. - linkset --list LIST --bundle WIKI --out-dir DIR: for the list's link candidates (K17 part 4), a sentence file in the form klink task --sentences reads and, per page, a catalogue (--catalogue) with the present targets and the pages that share a source with the sentence. klink task takes one catalogue per run, so linkset groups the pages into tasks in which the union of the catalogues stays within klink.CATALOGUE_CHARS, and writes one sentence file and one catalogue per group (`group-.sentences.json`, `group-.catalogue`); a page whose catalogue alone is over is cut to it, the present targets kept first, and the cut is counted. The number of groups is printed (the repair order prices the link pass from it). The frozen linker then gets an ordinary link pass on exactly those sentences (klink task, klink accept), and the repair order needs no new code. A candidate whose sentence or target is not in the collection is `stale-list`. - The home and the main pages: when the last fragment on a page is removed, the page goes as in kpack remove (drop_dangling, left_without, pass_homes), and all it brings with it is listed. When a main page loses its last fragment from the source it is the main page of while the page stands, it is listed as a main page without its source; the guard in the repair order asks the model again, as when a source lacks a main page. The pages without a home after the repair that had one before are listed. - Every write is logged in WORK/K-work/repairs.json (okf-wiki/repairs/1, appended, never rewritten): what was done, the fragments before and after (label), the claims that change (also the other sentences of a replaced fragment and the sentences that moved), the link records moved or taken out, and the pages that change. What a fix expects to change is measured, not guessed: the new state is built in a temporary copy of the collection with kassemble build and compared with it by the same measurement as diff, and the log names the collection by its sha256 (tree_sha256). accepted.json and the log are written with kpack.write_state (with its backup); the links file is written after them through a temporary file beside it, its old copy kept in K-work/backup/. A temporary links file left by a write that did not finish stops the next fix, unlink and relink (exit 2, with what to do). - diff --old WIKI --new WIKI --sources DIR --repairs FILE --out DIR: measures after kassemble build which concept pages have other bytes (not index.md and log.md; a page gone counts) and which claims are new, gone or have another `excerpt_sha256` (the frozen claims extract). It compares them with what the repairs logged against --old (by its sha256) expect, and exits 1 when the measured and the logged are not equal both ways. Out: `ids.json` with the new and changed claims in the form claims calls --ids reads, and `sider.json` with the changed pages and those gone (K15, K19), written also when they differ. The refusal codes (REFUSALS below; those the repair gate tries stand also in gates/repair.manifest.json): stale-list task, linkset: not K17's list for this claim list, or a claim, sentence or target it names is not in the collection as listed unknown-key fix, task: (concept_id, footnote) is no record of accepted.json ambiguous-key fix, task: (concept_id, footnote) is more than one record of accepted.json stale-bundle fix: the key is a record but no footnote on its page in --bundle duplicate-answer fix: two answers for one key label-taken fix: a new footnote another fragment of the concept has, or two new fragments share malformed fix: not in the answer form (and kpack.validate's malformed) anchor-outside-pack, anchor-crosses-blocks, sentence-count, source-missing, quote-missing, quote-not-in-span, provenance-missing, provenance-unknown, tests-missing fix: kpack.validate's reasons, the one hard check at writing, against the source alone shared-words task: the agent file or the instructions share eight words with the task's text target-missing relink: no concept page --to in --bundle self-link relink: --to is the record's own page duplicate-link relink: the sentence links to --to already link-not-found unlink, relink: no record with this page, sentence, anchor and at """ from __future__ import annotations import argparse import copy import hashlib import json import os import re import shutil import stat import sys import tempfile import traceback from collections import Counter from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import Any from okfwiki import claims, kassemble, klink, kpack, reader from okfwiki.common import RunError from okfwiki.kblocks import kilde_of LIST_SCHEMA = "okf-wiki/repair-list/1" LIST_KEYS = ("schema", "claims_sha256", "verdicts_sha256", "claims", "pages", "pairs", "links") ANSWERS_SCHEMA = "okf-wiki/repair-answers/1" REPAIRS_SCHEMA = "okf-wiki/repairs/1" PARTS_SCHEMA = "okf-wiki/repair-parts/1" IDS_SCHEMA = "okf-wiki/repair-ids/1" PAGES_SCHEMA = "okf-wiki/repair-pages/1" REPAIRS = "repairs.json" # in WORK/K-work TASK_FRAGMENTS = 10 # fragments of one page per task NEW_FRAGMENTS = 3 # a fragment is replaced by one to three EXCERPT_LINES = 40 # lines on each side of the quote when the block is too large (measured in order 35 A2: 40 fits both) FRAGMENT_LABEL = re.compile(rf"s[0-9a-f]{{{kassemble.LABEL_HEX}}}") # kassemble.label; context labels start with c EXPECTED = ("pages", "pages_gone", "claims", "claims_gone") REASON_TEXT = {"not-supported": "the reader did not accept it", "no-verdict": "the reader gave it no verdict"} REFUSALS = { "stale-list": "task and linkset: the list is not K17's for this claim list (claims_sha256, schema, keys), or names " "a claim, sentence or target the collection does not have as listed", "unknown-key": "fix and task: (concept_id, footnote) is no record of accepted.json", "ambiguous-key": "fix and task: (concept_id, footnote) is more than one record of accepted.json", "stale-bundle": "fix: the key is a record of accepted.json but not a footnote on its page in --bundle", "duplicate-answer": "fix: two answers in one file have the same key", "label-taken": "fix: a new fragment's footnote is another fragment's of the concept, or two new ones share it", "malformed": "fix: an answer or a new fragment is not in the answer form (kpack.validate's malformed too)", "anchor-outside-pack": "kpack.validate against the fragment's own source alone: the anchor is in another source", "anchor-crosses-blocks": "kpack.validate: the anchor is not inside one block", "sentence-count": "kpack.validate: the text is not one to three sentences", "source-missing": "kpack.validate: the source cannot be read", "quote-missing": "kpack.validate: no quote", "quote-not-in-span": "kpack.validate: the quote is not verbatim on one line of the anchor span", "provenance-missing": "kpack.validate: no provenance mark", "provenance-unknown": "kpack.validate: a provenance mark not in kpack.PROVENANCE", "tests-missing": "kpack.validate: a new concept without tests (never, as code sets status existing)", "shared-words": "task: the agent file or the instructions share a run of eight words with a claim, a quote or the " "block text in the task", "target-missing": "relink: the new target is no concept page of --bundle", "self-link": "relink: the new target is the record's own page", "duplicate-link": "relink: the sentence already links to the new target", "link-not-found": "unlink and relink: no record of the links file has this page, sentence, anchor and at", } INSTRUCTIONS = """\ # How to repair the fragments You repair fragments of a knowledge collection. A fragment is one to three sentences on a concept page, written from one block of a source document and resting on a quote from it. A reader, or the operator, did not accept at least one sentence of each fragment below; those sentences are marked with the reason. Read every part of this task first: its fragments may stand in more than one part. For each fragment, give exactly one answer. - `replace` it with one to three new fragments when the block bears what the fragment is there to say. Each new fragment says what its quote and its context say, and no more. A condition the source sets stays with the statement it limits. An instruction stays an instruction, and a possibility stays a possibility. The new fragments take the place of the whole fragment, so carry over what its other sentences say where the block bears it. - `remove` it when the block does not bear it. Each new fragment has: - `text`: one to three sentences. - `quote`: one piece copied character for character from one numbered line of the block, inside `anchor`. - `anchor`: {"file", "start", "end"}: the file the fragment names, and a range of lines inside its block. - `provenance`: "documented" when the text says what its quote says; "inferred" when it draws a conclusion from lines of its anchor that no single line states; "merged" when it combines what several lines of its anchor say. - `context` (optional): at most two pieces, each copied character for character from a heading of the block's path or from another line of its section: the subject a heading names, or a condition, step or setting another line states. Leave `context` out to keep the context quotes the fragment has now; give an empty list for none. An example from another subject. A block reads: 12 | Knead the dough for ten minutes. 13 | Let it rest in a warm place until it doubles in size. A fragment says "Knead the dough for an hour." Its quote holds and its text does not, so replace it with {"text": "Knead the dough for ten minutes.", "quote": "Knead the dough for ten minutes", "anchor": {"file": "bread.md", "start": 12, "end": 13}, "provenance": "documented"}. A fragment that says "The dough must rest overnight." has no line that bears it: remove it. Write one JSON object to the answer path your prompt names: {"schema": "okf-wiki/repair-answers/1", "answers": [ {"concept_id": "...", "footnote": "...", "action": "replace", "fragments": [...]}, {"concept_id": "...", "footnote": "...", "action": "remove"}]} Copy `concept_id` and `footnote` as the fragment shows them, and answer every fragment of the task once. Then hand back only the word done. """ @dataclass(frozen=True) class Refusal: code: str what: str def refuse(refusals: list[Refusal]) -> int: for r in refusals: print(f"refused: {r.code} {r.what}") return 1 # --- common ---------------------------------------------------------------------------------------------------- def file_sha256(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() def tree_sha256(root: Path) -> str: """The sha256 of a collection: every file's relative path and sha256, in path order (folder metadata left out).""" h = hashlib.sha256() for p in sorted(root.rglob("*")): if p.is_file() and p.name not in kassemble.TOLERATED: h.update(f"{p.relative_to(root).as_posix()}\0{file_sha256(p)}\n".encode("utf-8")) return h.hexdigest() def stem(page: str) -> str: return klink.concept_id(klink.page_rel(page)) def fragment_of(labels: Any) -> str | None: """The fragment's footnote among a sentence's labels (its context, version and week labels left out).""" return next((str(x) for x in labels if FRAGMENT_LABEL.fullmatch(str(x))), None) def keyed(acc: dict[str, Any]) -> dict[tuple[str, str], list[int]]: out: dict[tuple[str, str], list[int]] = {} for i, r in enumerate(acc["fragments"]): out.setdefault((str(r["concept_id"]), kassemble.label(r)), []).append(i) return out def footnotes_on(bundle: Path, cid: str) -> set[str]: page = bundle / klink.page_rel(cid) if not page.is_file(): return set() return {x for s in klink.page_sentences(page.read_text(encoding="utf-8")) for x in s.labels} def block_of(blocks_doc: dict[str, Any], r: dict[str, Any]) -> dict[str, Any]: block = next((b for b in blocks_doc["blocks"] if b["file"] == r["file"] and b["start"] <= r["start"] and r["end"] <= b["end"]), None) if block is None: raise RunError(f"--blocks has no block for {r['concept_id']} {kassemble.label(r)} ({r['file']} " f"{r['start']}-{r['end']})") return dict(block) def source_lines(kilde: Path, file: str, cache: dict[str, list[str]]) -> list[str]: if file not in cache: cache[file] = (kilde / file).read_text(encoding="utf-8").split("\n") return cache[file] def links_tmp(links: Path) -> Path: return links.with_name(f".{links.name}.krepair-tmp") def check_links_tmp(links: Path) -> None: tmp = links_tmp(links) if tmp.exists(): raise RunError(f"{tmp} exists: an earlier repair did not finish its links write. When the last entry of " f"K-work/{REPAIRS} is that repair, move {tmp} to {links}; else delete it") def read_repairs(work: Path) -> dict[str, Any]: path = kpack.k_work(work) / REPAIRS if not path.is_file(): return {"schema": REPAIRS_SCHEMA, "entries": []} doc: dict[str, Any] = kpack.read_json(path) if doc.get("schema") != REPAIRS_SCHEMA or not isinstance(doc.get("entries"), list): raise RunError(f"{path} is not {REPAIRS_SCHEMA}") return doc def now() -> str: return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") def commit(work: Path, label: str, files: dict[str, Any], links: Path, links_doc: dict[str, Any] | None) -> None: """accepted.json and the log under kpack's marker and backup, then the links file through a temporary file.""" tmp = links_tmp(links) if links_doc is not None: tmp.write_text(klink.dump(links_doc), encoding="utf-8") kpack.write_state(work, label, files) if links_doc is not None: shutil.copy2(links, kpack.k_work(work) / "backup" / links.name) os.replace(tmp, links) def write_links(links: Path, links_doc: dict[str, Any]) -> None: tmp = links_tmp(links) tmp.write_text(klink.dump(links_doc), encoding="utf-8") os.replace(tmp, links) def with_records(doc: dict[str, Any], records: list[dict[str, Any]]) -> dict[str, Any]: return {**doc, "links": records, **({"accepted": len(records)} if "accepted" in doc else {})} # --- the measurement (diff, and what a fix expects) ------------------------------------------------------------ @dataclass class Measured: pages: list[str] # concept pages with other bytes, a page gone among them pages_gone: list[str] claims: list[str] # ids new or with another excerpt_sha256 claims_gone: list[str] # ids gone or with another excerpt_sha256 def as_dict(self) -> dict[str, list[str]]: return {"pages": self.pages, "pages_gone": self.pages_gone, "claims": self.claims, "claims_gone": self.claims_gone} def concept_pages(wiki: Path) -> dict[str, bytes]: return {p.relative_to(wiki).as_posix(): p.read_bytes() for p in sorted(wiki.rglob("*.md")) if klink.is_concept(p.relative_to(wiki).as_posix())} def measure(old: Path, new: Path, sources: Path) -> Measured: a, z = concept_pages(old), concept_pages(new) before = Counter((c.id, c.excerpt_sha256) for c in claims.extract(old, sources).claims) after = Counter((c.id, c.excerpt_sha256) for c in claims.extract(new, sources).claims) return Measured(sorted(n for n in set(a) | set(z) if a.get(n) != z.get(n)), sorted(set(a) - set(z)), sorted({i for i, _ in after - before}), sorted({i for i, _ in before - after})) # --- lists ----------------------------------------------------------------------------------------------------- def read_list(path: Path, claims_path: Path | None) -> tuple[dict[str, Any], list[Refusal]]: doc = kpack.read_json(path) if not isinstance(doc, dict): raise RunError(f"{path} is not a repair list") name = path.name out = [] if doc.get("schema") != LIST_SCHEMA: out.append(Refusal("stale-list", f"{name} (schema {doc.get('schema')!r}, not {LIST_SCHEMA})")) out += [Refusal("stale-list", f"{name} (unknown top-level key {k!r})") for k in sorted(set(doc) - set(LIST_KEYS))] out += [Refusal("stale-list", f"{name} (no {k})") for k in LIST_KEYS if k not in doc] if claims_path is not None and doc.get("claims_sha256") != file_sha256(claims_path): out.append(Refusal("stale-list", f"{name} (claims_sha256 is not the sha256 of {claims_path.name})")) if out: return doc, out rows, links = doc["claims"], doc["links"] if not (isinstance(rows, list) and all( isinstance(c, dict) and isinstance(c.get("id"), str) and isinstance(c.get("excerpt_sha256"), str) and isinstance(c.get("reasons", []), list) for c in rows)): raise RunError(f"{path}: claims is not a list of records with id, excerpt_sha256 and reasons") if not (isinstance(links, list) and all( isinstance(x, dict) and all(type(x.get(k)) is t for k, t in kassemble.LINK_KEYS.items()) for x in links)): raise RunError(f"{path}: links is not a list of records with {', '.join(kassemble.LINK_KEYS)}") return doc, out # --- list ------------------------------------------------------------------------------------------------------ def cmd_list(args: argparse.Namespace) -> int: claims_path, verdicts_path, out = Path(args.claims), Path(args.verdicts), Path(args.out) if out.exists(): raise RunError(f"{out} exists: a list already written is never overwritten") found = claims.load_claims(claims_path) verdicts = kpack.read_json(verdicts_path) rows_in = verdicts.get("verdicts") if isinstance(verdicts, dict) else None if not (isinstance(rows_in, list) and all(isinstance(v, dict) and isinstance(v.get("id"), str) for v in rows_in)): raise RunError(f"{verdicts_path} is not the reader's verdicts (reader collect --out)") given: dict[str, list[str]] = {} for v in rows_in: given.setdefault(v["id"], []).append(str(v.get("verdict"))) known = {c.id for c in found.claims} strangers = sorted(set(given) - known) if strangers: raise RunError(f"{verdicts_path.name} has verdicts for {len(strangers)} claims that {claims_path.name} lacks: " + " ".join(strangers[:5])) covered = known if args.ids: ids = kpack.read_json(Path(args.ids)) wanted = ids.get("ids") if isinstance(ids, dict) else None if not (isinstance(wanted, list) and all(isinstance(i, str) for i in wanted)): raise RunError(f"{args.ids} has no list of ids") covered = set(wanted) if covered - known: raise RunError(f"{args.ids} names {len(covered - known)} claims that {claims_path.name} lacks: " + " ".join(sorted(covered - known)[:5])) rows = [{"id": c.id, "excerpt_sha256": c.excerpt_sha256, "page": c.page, "line": c.line, "provenance": c.provenance, "sources": [{"file": f"kilde/{e.file}", "line": e.line} for e in c.excerpts], "origins": ["reader"], "reasons": ["not-supported" if c.id in given else "no-verdict"]} for c in found.claims if c.id in covered and given.get(c.id) != [reader.SUPPORTED]] doc = {"schema": LIST_SCHEMA, "claims_sha256": file_sha256(claims_path), "verdicts_sha256": file_sha256(verdicts_path), "claims": rows, "pages": [], "pairs": [], "links": []} out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8") silent = sum(1 for r in rows if r["reasons"] == ["no-verdict"]) print(f"list: {len(rows)} of {len(covered)} claims read are on the list ({len(rows) - silent} not supported, " f"{silent} without a verdict) -> {out}") return 0 # --- task ------------------------------------------------------------------------------------------------------ @dataclass class Item: """One fragment of a task: its record, its page's title and the sentences it gives the page.""" concept_id: str label: str record: dict[str, Any] title: str sentences: list[dict[str, Any]] # the claims of the fragment on its page, in page order (from --claims) reasons: dict[str, list[str]] = field(default_factory=dict) # listed claim id -> its reasons def reason_text(reasons: list[str]) -> str: return "; ".join(dict.fromkeys(REASON_TEXT.get(r, f"the operator: {r}") for r in reasons)) def excerpts_of(c: dict[str, Any], pairs: dict[str, reader.Pair]) -> list[reader.Excerpt]: p = pairs.get(str(c["id"])) if p is not None: return list(p.excerpts) return [reader.Excerpt(str(e["file"]), int(e["line"]), str(e["text"])) for e in c.get("excerpts", [])] def block_text(item: Item, blocks_doc: dict[str, Any], kilde: Path, cache: dict[str, list[str]]) -> tuple[ dict[str, Any], list[str]]: block = block_of(blocks_doc, item.record) lines = source_lines(kilde, str(item.record["file"]), cache) return block, [lines[n - 1] if n <= len(lines) else "" for n in range(block["start"], block["end"] + 1)] def render_item(n: int, item: Item, pairs: dict[str, reader.Pair], blocks_doc: dict[str, Any], kilde: Path, cache: dict[str, list[str]], excerpt: bool = False) -> str: r = item.record out = [f"## Fragment {n}", "", f"- concept_id: `{item.concept_id}`", f"- footnote: `{item.label}`", f"- page: {item.title}", "", "Its sentences, as the page shows them:", ""] for k, c in enumerate(item.sentences, 1): mark = f" (on the list: {reason_text(item.reasons[c['id']])})" if c["id"] in item.reasons else "" out.append(f"{k}. {klink.key_text(str(c['claim']))}{mark}") out += [f' {e.text}' for e in excerpts_of(c, pairs)] context = r.get("context", []) out += ["", f"Quote: `{r['quote']}` ({r['file']}, line {r['quote_line']}), provenance {r['provenance']}.", "Context quotes: " + ("; ".join(f"`{c['quote']}` (line {c['line']})" for c in context) if context else "none") + "."] block, lines = block_text(item, blocks_doc, kilde, cache) out += [f"Anchor: {r['file']}, lines {r['start']}-{r['end']}.", f"Block: {r['file']} › {block['section']}, lines {block['start']}-{block['end']}:"] first, last = block["start"], block["end"] if excerpt: # the quote stands on one line (kpack.validate), so its first and last line are quote_line q = int(r["quote_line"]) first, last = max(first, q - EXCERPT_LINES), min(last, q + EXCERPT_LINES) out.append(f"The block is shown only in part: lines {first}-{last} of lines {block['start']}-{block['end']} " f"({len(lines)} lines). Judge the fragment from the lines shown.") out.append("") out += [f"{k:>5} | {lines[k - block['start']]}" for k in range(first, last + 1)] outside = sorted({int(c["line"]) for c in context} - set(range(first, last + 1))) if excerpt and outside: src = source_lines(kilde, str(r["file"]), cache) out += ["", "Context quote lines outside the lines shown:", ""] out += [f"{k:>5} | {src[k - 1] if k <= len(src) else ''}" for k in outside] return "\n".join(out) + "\n\n" def part_head(task: str, i: int, n: int) -> str: return f"\n# Repair task {task}, part {i} of {n}\n\n" def part_room(task: str) -> int: return kpack.PART_CHARS - len(part_head(task, 999, 999)) def pack_parts(task: str, sections: list[tuple[str, str]]) -> list[str]: """The task's parts: the instructions first, then whole fragment sections, each part within kpack.PART_CHARS.""" limit = kpack.PART_CHARS room = part_room(task) for what, text in sections[1:] + sections[:1]: # a fragment that cannot fit is named before the instructions if len(text) > room: raise RunError(f"{what} needs {len(text)} characters, over the room of {room} in a part of {limit}: it " f"cannot stand in one part") bodies: list[str] = [] for _, text in sections: if bodies and len(bodies[-1]) + len(text) <= room: bodies[-1] += text else: bodies.append(text) return [part_head(task, i, len(bodies)) + body for i, body in enumerate(bodies, 1)] def task_items(lst: dict[str, Any], claims_doc: dict[str, Any], acc: dict[str, Any], index: dict[str, Any], claims_name: str) -> tuple[list[Item], list[Refusal]]: by_id = {str(c["id"]): c for c in claims_doc["claims"]} keys = keyed(acc) titles = {str(e["id"]): kassemble.one_line(e["title"]) for e in index["entries"]} items: dict[tuple[str, str], Item] = {} refusals: list[Refusal] = [] for row in lst["claims"]: cid_claim = str(row["id"]) c = by_id.get(cid_claim) if c is None or c.get("excerpt_sha256") != row["excerpt_sha256"]: why = "not a claim of" if c is None else "shown with another excerpt_sha256 in" refusals.append(Refusal("stale-list", f"{cid_claim} ({why} {claims_name})")) continue label = fragment_of(c.get("footnotes", [])) concept = stem(str(c["page"])) hits = keys.get((concept, label or ""), []) if label is None or len(hits) != 1: code = "ambiguous-key" if len(hits) > 1 else "unknown-key" refusals.append(Refusal(code, f"{cid_claim} ({concept} {label or 'no footnote'})")) continue key = (concept, label) if key not in items: sentences = [x for x in claims_doc["claims"] if str(x["page"]) == str(c["page"]) and fragment_of(x.get("footnotes", [])) == label] items[key] = Item(concept, label, acc["fragments"][hits[0]], titles.get(concept, concept), sentences) items[key].reasons[cid_claim] = [str(x) for x in row.get("reasons", [])] or ["not-supported"] return list(items.values()), refusals def cmd_task(args: argparse.Namespace) -> int: claims_path, work, out = Path(args.claims), Path(args.work), Path(args.out_dir).resolve() kpack.check_pending(work) lst, refusals = read_list(Path(args.list), claims_path) if refusals: return refuse(refusals) claims_doc: dict[str, Any] = kpack.read_json(claims_path) pairs = {p.id: p for p in reader.load_pairs(Path(args.pairs))} acc, index = kpack.read_accepted(work), kpack.read_index(work) blocks_doc: dict[str, Any] = kpack.read_json(Path(args.blocks)) kilde = kilde_of(Path(args.sources)) items, refusals = task_items(lst, claims_doc, acc, index, claims_path.name) if refusals: return refuse(refusals) if not items: raise RunError(f"{args.list} lists no claim") cache: dict[str, list[str]] = {} material: list[reader.Pair] = [] for it in items: what = f"{it.concept_id} {it.label}" material += [reader.Pair(f"{c['id']}", klink.key_text(str(c["claim"])), tuple(excerpts_of(c, pairs))) for c in it.sentences] quotes = [str(it.record["quote"]), *(str(c["quote"]) for c in it.record.get("context", []))] material.append(reader.Pair(f"{what} (quotes)", " ".join(quotes), ())) material.append(reader.Pair(f"{what} (block)", "\n".join(block_text(it, blocks_doc, kilde, cache)[1]), ())) agent = Path(args.agent) for name, text in ((agent.name, agent.read_text(encoding="utf-8")), ("the instructions", INSTRUCTIONS)): refusals += [Refusal("shared-words", f"{name} and {pid}") for pid in reader.shared_runs(text, material)] if refusals: return refuse(refusals) by_page: dict[str, list[Item]] = {} for it in items: by_page.setdefault(it.concept_id, []).append(it) files: dict[Path, str] = {} tasks = 0 for concept in by_page: chunk = by_page[concept] for i in range(0, len(chunk), TASK_FRAGMENTS): tasks += 1 name = f"r{tasks:03d}" group = chunk[i:i + TASK_FRAGMENTS] sections = [("the instructions", INSTRUCTIONS + "\n")] for n, it in enumerate(group, 1): text = render_item(n, it, pairs, blocks_doc, kilde, cache) if len(text) > part_room(name): text = render_item(n, it, pairs, blocks_doc, kilde, cache, excerpt=True) sections.append((f"fragment {it.concept_id} {it.label}", text)) parts = pack_parts(name, sections) paths = [out / f"{name}-{k}.md" for k in range(1, len(parts) + 1)] files.update(zip(paths, parts)) files[out / f"{name}.parts.json"] = kpack.dump({ "schema": PARTS_SCHEMA, "task": name, "parts": [str(p) for p in paths], "answer": str(out / f"{name}.answers.json"), "chars": [len(p) for p in parts], "fragments": [{"concept_id": it.concept_id, "footnote": it.label, "claims": sorted(it.reasons)} for it in group]}) taken = sorted(str(p) for p in files if p.exists()) + sorted(str(p) for p in out.glob("*.parts.json")) if taken: raise RunError(f"tasks exist in {out} ({taken[0]}); a task already written is never changed") out.mkdir(parents=True, exist_ok=True) for path, text in files.items(): path.write_text(text, encoding="utf-8") listed = sum(len(it.reasons) for it in items) print(f"task: {listed} claims, {len(items)} fragments, {len(by_page)} pages, {tasks} tasks, " f"{sum(1 for p in files if p.suffix == '.md')} parts -> {out}") return 0 # --- fix ------------------------------------------------------------------------------------------------------- @dataclass class Answer: concept_id: str footnote: str action: str fragments: list[dict[str, Any]] @property def what(self) -> str: return f"{self.concept_id} {self.footnote}" def read_answers(path: Path) -> tuple[list[Answer], list[Refusal]]: doc = kpack.read_json(path) if not (isinstance(doc, dict) and doc.get("schema") == ANSWERS_SCHEMA and isinstance(doc.get("answers"), list)): raise RunError(f"{path} is not {ANSWERS_SCHEMA} with a list of answers") if not doc["answers"]: raise RunError(f"{path} holds no answer") out: list[Answer] = [] refusals: list[Refusal] = [] for n, a in enumerate(doc["answers"], 1): cid, fn, act = (a.get(k) if isinstance(a, dict) else None for k in ("concept_id", "footnote", "action")) what = f"{cid} {fn}" if isinstance(cid, str) and isinstance(fn, str) else f"answer {n}" if not (isinstance(cid, str) and isinstance(fn, str) and act in ("replace", "remove")): refusals.append(Refusal("malformed", f"{what} (needs concept_id, footnote and action replace or remove)")) continue frags = a.get("fragments") if act == "replace" and not (isinstance(frags, list) and 1 <= len(frags) <= NEW_FRAGMENTS and all(isinstance(f, dict) for f in frags)): refusals.append(Refusal("malformed", f"{what} (replace takes one to {NEW_FRAGMENTS} fragments)")) continue if act == "remove" and frags is not None: refusals.append(Refusal("malformed", f"{what} (remove takes no fragments)")) continue out.append(Answer(cid, fn, str(act), list(frags or []))) return out, refusals @dataclass class Checked: """The answers that passed, by the index of the record they repair, with what fix lists.""" new: dict[int, list[dict[str, Any]]] = field(default_factory=dict) # old record index -> new records ([] removes) keys: dict[int, tuple[str, str]] = field(default_factory=dict) context_not_kept: list[dict[str, Any]] = field(default_factory=list) def check_answers(answers: list[Answer], acc: dict[str, Any], index: dict[str, Any], bundle: Path, blocks_doc: dict[str, Any], kilde: Path) -> tuple[Checked, list[Refusal]]: keys = keyed(acc) counts = Counter((a.concept_id, a.footnote) for a in answers) refusals = [Refusal("duplicate-answer", f"{c} {f} ({n} answers)") for (c, f), n in counts.items() if n > 1] done = Checked() shown: dict[str, set[str]] = {} fresh: dict[str, set[str]] = {} for a in answers: hits = keys.get((a.concept_id, a.footnote), []) if len(hits) != 1: refusals.append(Refusal("ambiguous-key" if hits else "unknown-key", a.what)) continue if a.concept_id not in shown: shown[a.concept_id] = footnotes_on(bundle, a.concept_id) if a.footnote not in shown[a.concept_id]: refusals.append(Refusal("stale-bundle", f"{a.what} (no footnote on {klink.page_rel(a.concept_id)} in " f"{bundle})")) continue i = hits[0] old = acc["fragments"][i] records: list[dict[str, Any]] = [] ok = True for k, given in enumerate(a.fragments, 1): f: dict[str, Any] = {"concept": old["concept_id"], "status": "existing", **{x: given[x] for x in ("text", "quote", "anchor", "provenance") if x in given}} inherited = "context" not in given f["context"] = [c["quote"] for c in old.get("context", [])] if inherited else given["context"] valid, rejected = kpack.validate([f], [str(old["file"])], blocks_doc, kilde) if rejected: refusals += [Refusal(reason, f"{a.what} (new fragment {k})") for reason in sorted(rejected)] ok = False continue placed, _ = kpack.with_context(valid, blocks_doc, kilde) rec = {**kpack.new_record(placed[0], "existing", str(old["concept_id"]), index, int(old["pack"]), int(old["position"])), "outcome": old.get("outcome", "existing")} label = kassemble.label(rec) other = [j for j in keys.get((a.concept_id, label), []) if j != i] if other or label in fresh.setdefault(a.concept_id, set()): whose = f"of {kassemble.label(acc['fragments'][other[0]])}" if other else "of another new fragment" refusals.append(Refusal("label-taken", f"{a.what} (new fragment {k} has the footnote {label}, the " f"footnote {whose} in {a.concept_id})")) ok = False continue fresh[a.concept_id].add(label) if inherited: kept = {str(c["quote"]) for c in rec.get("context", [])} done.context_not_kept += [{"concept_id": a.concept_id, "footnote": a.footnote, "quote": str(q)} for q in f["context"] if str(q) not in kept] records.append(rec) if ok: done.new[i], done.keys[i] = records, (a.concept_id, a.footnote) return done, refusals Slot = tuple[str, int] def slotted(page: kassemble.Page) -> list[tuple[klink.Sentence, Slot | None]]: """The sentences of an assembled page as klink keys them, each with its slot: (its fragment's footnote, its place among that fragment's sentences on the page).""" seen: Counter[str] = Counter() out: list[tuple[klink.Sentence, Slot | None]] = [] for s in klink.body_sentences(klink.reduce_links(page.body)[0]): label = fragment_of(s.labels) if label is None: out.append((s, None)) continue out.append((s, (label, seen[label]))) seen[label] += 1 return out def sentences_of(by_slot: dict[Slot, klink.Sentence], label: str) -> list[klink.Sentence]: out: list[klink.Sentence] = [] while (label, len(out)) in by_slot: out.append(by_slot[(label, len(out))]) return out def occurrences(text: str, anchor: str) -> list[int]: out, at = [], text.find(anchor) while at >= 0: out.append(at) at = text.find(anchor, at + 1) return out def same_occurrence(old: klink.Sentence, anchor: str, at: int, new: klink.Sentence) -> int | None: """The offset of the anchor in the new sentence at the occurrence number it has in the old one, when it stands there outside code and as a whole word (as klink.find_anchor accepts it); else None.""" before = occurrences(old.text, anchor) if at not in before: return None after = occurrences(new.text, anchor) k = before.index(at) if k >= len(after): return None p, e = after[k], after[k] + len(anchor) whole = (p == 0 or not new.text[p - 1].isalnum()) and (e == len(new.text) or not new.text[e].isalnum()) return p if whole and new.masked[p:e] == anchor else None @dataclass class Moves: records: list[dict[str, Any]] moved: list[dict[str, Any]] = field(default_factory=list) removed: list[dict[str, Any]] = field(default_factory=list) sentences_moved: list[dict[str, str]] = field(default_factory=list) touched_before: list[str] = field(default_factory=list) touched_after: list[str] = field(default_factory=list) def move_links(records: list[dict[str, Any]], old_pages: dict[str, kassemble.Page], new_pages: dict[str, kassemble.Page], changed: dict[tuple[str, str], list[str]], gone: set[str]) -> Moves: """The link records after the repair: `changed` maps each repaired (concept, footnote) to its new footnotes ([] for a removal); `gone` holds the pages that go.""" m = Moves([]) old_s = {rel: slotted(p) for rel, p in old_pages.items()} old_keyed = {rel: {s.key: (s, slot) for s, slot in sentences if slot is not None} for rel, sentences in old_s.items()} new_s = {rel: {slot: s for s, slot in slotted(p) if slot is not None} for rel, p in new_pages.items()} for rel, sentences in old_s.items(): cid = stem(rel) for s, slot in sentences: if slot is None: continue if (cid, slot[0]) in changed: m.touched_before.append(f"{rel}#{s.key}") continue now_s = new_s.get(rel, {}).get(slot) if now_s is not None and now_s.key != s.key: m.sentences_moved.append({"page": rel, "before": s.key, "after": now_s.key}) m.touched_before.append(f"{rel}#{s.key}") m.touched_after.append(f"{rel}#{now_s.key}") for rel, by_slot in new_s.items(): cid = stem(rel) new_labels = {x for (c, _), labels in changed.items() if c == cid for x in labels} m.touched_after += [f"{rel}#{s.key}" for slot, s in by_slot.items() if slot[0] in new_labels] for r in records: page, target = klink.page_rel(str(r["page"])), klink.page_rel(str(r["target"])) if page in gone or target in gone: m.removed.append({"record": r, "why": "its page goes" if page in gone else "its target goes"}) continue hit = old_keyed.get(page, {}).get(str(r["sentence"])) if hit is None: m.records.append(r) # on a page the repair does not touch, or stale already: left as it was continue s, (label, i) = hit cid = stem(page) new = new_s.get(page, {}) if (cid, label) in changed: pool = [s2 for x in changed[(cid, label)] for s2 in sentences_of(new, x)] if not pool: m.removed.append({"record": r, "why": "its sentence is removed"}) continue if i >= len(pool): m.removed.append({"record": r, "why": "its fragment has no sentence in its place now"}) continue to = pool[i] at = same_occurrence(s, str(r["anchor"]), int(r["at"]), to) elif (label, i) not in new: m.removed.append({"record": r, "why": "its sentence is not on the page now"}) continue else: to = new[(label, i)] at = int(r["at"]) if to.text[int(r["at"]):int(r["at"]) + len(str(r["anchor"]))] == r["anchor"] else None if at is None: m.removed.append({"record": r, "why": "its anchor is not in the new sentence at its occurrence"}) continue moved = {**r, "sentence": to.key, "at": at} if moved != r: m.moved.append({"before": r, "after": moved}) m.records.append(moved) return m def roles(acc: dict[str, Any], facts: dict[str, kpack.Fact]) -> tuple[set[str], set[str], dict[str, set[str]]]: """(the main pages, the pages with a home, the sources each page cites) as kassemble renders them: a main page cites the main-topic source it is named for, and a home is a link from a main page.""" cited: dict[str, set[str]] = {} for r in acc["fragments"]: cited.setdefault(str(r["concept_id"]), set()).add(str(r["file"])) mains = {str(m["concept_id"]) for m in acc["main_pages"] if str(m["source"]) in cited.get(str(m["concept_id"]), set()) and str(m["source"]) in facts and facts[str(m["source"])].main_topic} homed = {str(h["concept_id"]) for h in acc["homes"] if str(h["home"]) in mains and h["home"] != h["concept_id"] and str(h["concept_id"]) in cited} return mains, homed, cited def without_home(acc: dict[str, Any], facts: dict[str, kpack.Fact]) -> set[str]: mains, homed, cited = roles(acc, facts) return set(cited) - mains - homed def trial(work: Path, bundle: Path, sources: Path, acc: dict[str, Any], links_doc: dict[str, Any]) -> Measured: """What the repair will change: the new state built into a copy of the collection, measured against it.""" with tempfile.TemporaryDirectory(prefix="krepair-trial-") as tmp_name: tmp = Path(tmp_name) kw = tmp / "work" / "K-work" kw.mkdir(parents=True) shutil.copyfile(kpack.k_work(work) / "index.json", kw / "index.json") (kw / "accepted.json").write_text(kpack.dump(acc), encoding="utf-8") (tmp / "links.json").write_text(klink.dump(links_doc), encoding="utf-8") wiki = tmp / "wiki" shutil.copytree(bundle, wiki, copy_function=shutil.copyfile) # the copy is ours, whatever the bundle's modes for p in [wiki, *wiki.rglob("*")]: p.chmod(p.stat().st_mode | stat.S_IWUSR) report = kassemble.build(tmp / "work", sources, wiki, tmp / "links.json", replace=True) if report.stale: raise RunError(f"after the repair {report.stale} link records would not stand: a fault in krepair") return measure(bundle, wiki, sources) def cmd_fix(args: argparse.Namespace) -> int: work, bundle, sources, links = Path(args.work), Path(args.bundle), Path(args.sources), Path(args.links) kpack.check_pending(work) check_links_tmp(links) answers_path = Path(args.answers) answers, refusals = read_answers(answers_path) acc, index = kpack.read_accepted(work), kpack.read_index(work) blocks_doc: dict[str, Any] = kpack.read_json(Path(args.blocks)) kilde = kilde_of(sources) links_doc = kassemble.check_links(links) checked, more = check_answers(answers, acc, index, bundle, blocks_doc, kilde) refusals += more if refusals: return refuse(refusals) new_acc = copy.deepcopy(acc) new_acc["fragments"] = [copy.deepcopy(x) for i, r in enumerate(acc["fragments"]) for x in (checked.new[i] if i in checked.new else [r])] dropped = kpack.drop_dangling(new_acc) passed = kpack.pass_homes(new_acc) files = sorted({str(r["file"]) for r in acc["fragments"] + new_acc["fragments"]} | {str(m["source"]) for m in acc["main_pages"]}) facts = kpack.source_facts(sources, files) changed = {checked.keys[i]: [kassemble.label(r) for r in recs] for i, recs in checked.new.items()} concepts = {c for c, _ in changed} old_pages = kassemble.page_models(index, [r for r in acc["fragments"] if r["concept_id"] in concepts], kilde, facts, acc["main_pages"]) new_pages = kassemble.page_models(index, [r for r in new_acc["fragments"] if r["concept_id"] in concepts], kilde, facts, new_acc["main_pages"]) gone = {klink.page_rel(c) for c in kpack.pages_of(acc) - kpack.pages_of(new_acc)} moves = move_links([dict(r) for r in links_doc["links"]], old_pages, new_pages, changed, gone) new_links = with_records(links_doc, moves.records) expected = trial(work, bundle, sources, new_acc, new_links) _, _, cited_before = roles(acc, facts) _, _, cited_after = roles(new_acc, facts) no_source = [{"concept_id": str(m["concept_id"]), "source": str(m["source"])} for m in new_acc["main_pages"] if str(m["concept_id"]) in cited_after and str(m["source"]) not in cited_after[str(m["concept_id"])] and str(m["source"]) in cited_before.get(str(m["concept_id"]), set())] left = kpack.left_without(new_acc, set(kpack.main_ids(acc)), dropped[2]) homeless = sorted(set(left) | (without_home(new_acc, facts) - without_home(acc, facts))) log = read_repairs(work) entry = {"n": len(log["entries"]) + 1, "command": "fix", "at": now(), "answers": str(answers_path), "answers_sha256": file_sha256(answers_path), "bundle": str(bundle), "bundle_sha256": tree_sha256(bundle), "fragments": [{"concept_id": checked.keys[i][0], "before": checked.keys[i][1], "action": "replace" if recs else "remove", "after": [kassemble.label(r) for r in recs]} for i, recs in sorted(checked.new.items())], "claims_touched": {"before": sorted(set(moves.touched_before)), "after": sorted(set(moves.touched_after))}, "sentences_moved": moves.sentences_moved, "links": {"moved": moves.moved, "removed": moves.removed}, "listed": {"pages_gone": sorted(gone), "without_home": homeless, "main_page_without_source": no_source, "context_not_kept": checked.context_not_kept, "dropped": {"main_pages": dropped[0], "homes": dropped[1], "homes_passed": passed}}, "expected": expected.as_dict()} log["entries"].append(entry) commit(work, "repair", {"accepted.json": new_acc, REPAIRS: log}, links, new_links if moves.moved or moves.removed else None) removed = sum(not recs for recs in checked.new.values()) print(f"fix: {len(checked.new)} answers: replaced {len(checked.new) - removed} fragments with " f"{sum(len(recs) for recs in checked.new.values())}, removed {removed} -> " f"{kpack.k_work(work) / 'accepted.json'}") print(f" expected to change: {len(expected.pages)} pages ({len(expected.pages_gone)} gone), " f"{len(expected.claims)} claims new or changed, {len(expected.claims_gone)} gone or changed") print(f" links: moved {len(moves.moved)}, taken out {len(moves.removed)}; sentences moved " f"{len(moves.sentences_moved)}") for x in moves.removed: r = x["record"] print(f" taken out: {r['page']} {r['sentence']} {r['anchor']!r} -> {r['target']} ({x['why']})") for x in checked.context_not_kept: print(f" context not kept: {x['concept_id']} {x['footnote']}: {x['quote']!r}") for name in sorted(gone): print(f" page gone: {name}") for name in homeless: print(f" without a home now: {name}") for x in no_source: print(f" main page without its source: {x['concept_id']} ({x['source']})") print(f" logged as repair {entry['n']} in {kpack.k_work(work) / REPAIRS}") return 0 # --- unlink and relink ----------------------------------------------------------------------------------------- def find_record(records: list[dict[str, Any]], args: argparse.Namespace) -> int | None: page = klink.page_rel(args.page) return next((i for i, r in enumerate(records) if klink.page_rel(str(r["page"])) == page and r["sentence"] == args.sentence and r["anchor"] == args.anchor and int(r["at"]) == args.at), None) def link_entry(work: Path, command: str, bundle: Path, page: str, before: dict[str, Any], after: dict[str, Any] | None) -> dict[str, Any]: links = {"moved": [{"before": before, "after": after}], "removed": []} if after else { "moved": [], "removed": [{"record": before, "why": "unlink"}]} return {"n": len(read_repairs(work)["entries"]) + 1, "command": command, "at": now(), "bundle": str(bundle), "bundle_sha256": tree_sha256(bundle), "links": links, "expected": {"pages": [page], "pages_gone": [], "claims": [], "claims_gone": []}} def write_link_change(args: argparse.Namespace, command: str, links: Path, doc: dict[str, Any], records: list[dict[str, Any]], before: dict[str, Any], after: dict[str, Any] | None) -> None: new_doc = with_records(doc, records) if args.work is None: write_links(links, new_doc) return work = Path(args.work) kpack.check_pending(work) log = read_repairs(work) log["entries"].append(link_entry(work, command, Path(args.bundle), klink.page_rel(args.page), before, after)) commit(work, command, {REPAIRS: log}, links, new_doc) def link_args(args: argparse.Namespace) -> tuple[Path, dict[str, Any], list[dict[str, Any]]]: if args.work is not None and args.bundle is None: raise RunError("--work needs --bundle: the log names the collection the record was taken out of") links = Path(args.links) check_links_tmp(links) doc = kassemble.check_links(links) return links, doc, [dict(r) for r in doc["links"]] def cmd_unlink(args: argparse.Namespace) -> int: links, doc, records = link_args(args) i = find_record(records, args) if i is None: return refuse([Refusal("link-not-found", f"{klink.page_rel(args.page)} {args.sentence} {args.anchor!r} at " f"{args.at}")]) before = records.pop(i) write_link_change(args, "unlink", links, doc, records, before, None) print(f"unlink: took out {before['page']} {before['sentence']} {before['anchor']!r} -> {before['target']}") return 0 def cmd_relink(args: argparse.Namespace) -> int: links, doc, records = link_args(args) i = find_record(records, args) page, target = klink.page_rel(args.page), klink.page_rel(args.to) what = f"{page} {args.sentence} {args.anchor!r} at {args.at} -> {target}" if i is None: return refuse([Refusal("link-not-found", what)]) if target not in klink.read_pages(Path(args.bundle)): return refuse([Refusal("target-missing", what)]) if target == page: return refuse([Refusal("self-link", what)]) if any(klink.page_rel(str(r["page"])) == page and r["sentence"] == args.sentence and klink.page_rel(str(r["target"])) == target for r in records): return refuse([Refusal("duplicate-link", what)]) before = records[i] after = {**before, "target": target} records[i] = after write_link_change(args, "relink", links, doc, records, before, after) print(f"relink: {before['page']} {before['sentence']} {before['anchor']!r} -> {target} (was {before['target']})") return 0 # --- linkset --------------------------------------------------------------------------------------------------- def cmd_linkset(args: argparse.Namespace) -> int: lst, refusals = read_list(Path(args.list), Path(args.claims) if args.claims else None) if refusals: return refuse(refusals) bundle, out = Path(args.bundle), Path(args.out_dir) pages = klink.read_pages(bundle) sentences = {rel: {s.key: s for s in klink.body_sentences(p.body)} for rel, p in pages.items()} wanted: dict[str, list[str]] = {} targets: dict[str, list[str]] = {} sharing: dict[str, set[str]] = {} for x in lst["links"]: page, target = klink.page_rel(str(x["page"])), klink.page_rel(str(x["target"])) s = sentences.get(page, {}).get(str(x["sentence"])) if s is None or target not in pages: why = "no such sentence on the page" if s is None else f"no page {target}" refusals.append(Refusal("stale-list", f"{page} {x['sentence']} ({why})")) continue if s.key not in wanted.setdefault(page, []): wanted[page].append(s.key) if stem(target) not in targets.setdefault(page, []): targets[page].append(stem(target)) cites = {pages[page].resources[x] for x in s.labels if x in pages[page].resources} sharing.setdefault(page, set()).update(rel for rel, p in pages.items() if rel != page and cites & set(p.resources.values())) if refusals: return refuse(refusals) if not wanted: raise RunError(f"{args.list} lists no link candidate") budget = klink.CATALOGUE_CHARS groups: list[tuple[list[str], list[str]]] = [] # (pages, catalogue ids) cut = 0 for page in sorted(wanted): ids = targets[page] + sorted({stem(rel) for rel in sharing[page]} - set(targets[page])) if groups: union = groups[-1][1] + [i for i in ids if i not in groups[-1][1]] if len(klink.catalogue_text(union, pages)) <= budget: groups[-1] = (groups[-1][0] + [page], union) continue keep = list(targets[page]) for i in ids[len(targets[page]):]: if len(klink.catalogue_text(keep + [i], pages)) <= budget: keep.append(i) cut += len(ids) - len(keep) groups.append(([page], keep)) files: dict[Path, str] = {} for n, (group_pages, ids) in enumerate(groups, 1): files[out / f"group-{n}.sentences.json"] = klink.dump({"schema": klink.SENTENCES_SCHEMA, "pages": {p: wanted[p] for p in group_pages}}) files[out / f"group-{n}.catalogue"] = "".join(f"{i}\n" for i in ids) taken = sorted(str(p) for p in files if p.exists()) if taken: raise RunError(f"{taken[0]} exists; refusing to overwrite") out.mkdir(parents=True, exist_ok=True) for path, text in files.items(): path.write_text(text, encoding="utf-8") g = len(groups) print(f"linkset: {sum(len(v) for v in wanted.values())} candidates on {len(wanted)} pages, {g} " f"group{'s' if g != 1 else ''} -> {out}") for n, (group_pages, ids) in enumerate(groups, 1): print(f" group-{n}: {len(group_pages)} pages, {sum(len(wanted[p]) for p in group_pages)} sentences, " f"catalogue {len(ids)} ids ({len(klink.catalogue_text(ids, pages))} characters)") if cut: print(f" cut {cut} catalogue ids that did not fit {budget} characters (the present targets kept)") return 0 # --- diff ------------------------------------------------------------------------------------------------------ def cmd_diff(args: argparse.Namespace) -> int: old, new, sources, out = Path(args.old), Path(args.new), Path(args.sources), Path(args.out) log = kpack.read_json(Path(args.repairs)) if not (isinstance(log, dict) and log.get("schema") == REPAIRS_SCHEMA and isinstance(log.get("entries"), list)): raise RunError(f"{args.repairs} is not {REPAIRS_SCHEMA}") fp = tree_sha256(old) entries = [e for e in log["entries"] if e.get("bundle_sha256") == fp] m = measure(old, new, sources) logged = {k: sorted({x for e in entries for x in e["expected"][k]}) for k in EXPECTED} got = m.as_dict() out.mkdir(parents=True, exist_ok=True) (out / "ids.json").write_text(kpack.dump({"schema": IDS_SCHEMA, "old": str(old), "new": str(new), "ids": m.claims}), encoding="utf-8") (out / "sider.json").write_text(kpack.dump({"schema": PAGES_SCHEMA, "old": str(old), "new": str(new), "pages": m.pages, "gone": m.pages_gone}), encoding="utf-8") print(f"diff: {len(m.pages)} pages changed ({len(m.pages_gone)} gone), {len(m.claims)} claims new or changed, " f"{len(m.claims_gone)} gone or changed; {len(entries)} of {len(log['entries'])} repairs logged against " f"{old} -> {out}") differs = False if not entries and any(got.values()): print(f"differs: no repair is logged against {old} (sha256 {fp[:12]})") for k in EXPECTED: extra, missing = sorted(set(got[k]) - set(logged[k])), sorted(set(logged[k]) - set(got[k])) if extra: print(f"differs: {k} measured, not logged: {len(extra)} ({', '.join(extra[:5])})") if missing: print(f"differs: {k} logged, not measured: {len(missing)} ({', '.join(missing[:5])})") differs = differs or bool(extra or missing) return 1 if differs else 0 # --- command line ---------------------------------------------------------------------------------------------- def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(prog="python3 -m okfwiki.krepair", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) sub = p.add_subparsers(dest="command", required=True) ls = sub.add_parser("list") for name in ("--claims", "--verdicts", "--out"): ls.add_argument(name, required=True) ls.add_argument("--ids") t = sub.add_parser("task") for name in ("--list", "--claims", "--pairs", "--work", "--blocks", "--sources", "--agent", "--out-dir"): t.add_argument(name, required=True) f = sub.add_parser("fix") for name in ("--answers", "--bundle", "--work", "--blocks", "--sources", "--links"): f.add_argument(name, required=True) for command in ("unlink", "relink"): u = sub.add_parser(command) for name in ("--links", "--page", "--sentence", "--anchor"): u.add_argument(name, required=True) u.add_argument("--at", type=int, required=True) u.add_argument("--work") u.add_argument("--bundle", required=command == "relink") if command == "relink": u.add_argument("--to", required=True) s = sub.add_parser("linkset") for name in ("--list", "--bundle", "--out-dir"): s.add_argument(name, required=True) s.add_argument("--claims") d = sub.add_parser("diff") for name in ("--old", "--new", "--sources", "--repairs", "--out"): d.add_argument(name, required=True) return p COMMANDS = {"list": cmd_list, "task": cmd_task, "fix": cmd_fix, "unlink": cmd_unlink, "relink": cmd_relink, "linkset": cmd_linkset, "diff": cmd_diff} def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) try: return COMMANDS[args.command](args) except (RunError, reader.RunError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: print(f"krepair cannot run: {exc}", file=sys.stderr) return 2 except Exception: # an input this code did not foresee: still "could not run", with the traceback traceback.print_exc() return 2 if __name__ == "__main__": sys.exit(main())