okf-wiki/okfwiki/krepair.py
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

1212 lines
66 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""The repair command: one fragment of a built collection repaired or removed, one link record taken out or moved, the
helper's task, and the measurement of what a repair touched. No model call, no network, no program started (CLAUDE.md
rule 1). Order 34, contracts K18 and K19; the contract below is the order's, point for point, in English.
Usage:
python3 -m okfwiki.krepair list --claims CLAIMS --verdicts VERDICTS --out LIST [--ids IDS]
python3 -m okfwiki.krepair task --list LIST --claims CLAIMS --pairs PAIRS --work DIR --blocks FILE --sources DIR
--agent FILE --out-dir DIR
python3 -m okfwiki.krepair fix --answers FILE --bundle WIKI --work DIR --blocks FILE --sources DIR --links FILE
python3 -m okfwiki.krepair unlink --links FILE --page P --sentence KEY --anchor A --at N [--work DIR --bundle WIKI]
python3 -m okfwiki.krepair relink --links FILE --page P --sentence KEY --anchor A --at N --to TARGET --bundle WIKI
[--work DIR]
python3 -m okfwiki.krepair linkset --list LIST --bundle WIKI --out-dir DIR [--claims CLAIMS]
python3 -m okfwiki.krepair diff --old WIKI --new WIKI --sources DIR --repairs FILE --out DIR
Exit 0 done; 1 refused or not equal, with one line per reason of the form `refused: <code> <claim or record>`; 2 cannot
run (also for an input the code did not foresee, with its traceback).
The contract:
- The list in is K17's schema (okf-wiki/repair-list/1); task reads only `claims`, linkset only `links`. Both refuse a
list with an unknown top-level key, without one of K17's keys, of another schema, or with a `claims_sha256` that is
not the sha256 of the claim list given (task --claims; linkset with --claims, which K18 leaves out and this command
takes optionally), and task a claim the claim list lacks or shows with another `excerpt_sha256`: refusal code
`stale-list`.
- list --claims CLAIMS --verdicts VERDICTS --out LIST [--ids IDS]: the list from a reading (v1.2, part 1; before it
only the repair gate rendered one). Every claim the reading covered (all of CLAIMS, or those of --ids, the file
claims calls --ids reads) that does not have exactly one verdict, `supported`, is listed as claims score counts it:
`not-supported`, or `no-verdict` when the reader gave it none. Each row has the claim's id, `excerpt_sha256`, page,
line, provenance, its excerpts as `sources` and origin `reader`; K17's `kind` is left out (task does not read it).
`pages`, `pairs` and `links` are empty. A verdict for a claim CLAIMS lacks, or an --out that exists, cannot run
(exit 2), and nothing is written.
- task: the helper's tasks. The unit is the fragment, keyed by (`concept_id`, the `s...` footnote): the footnote alone
is not unique (kassemble.label takes neither text nor concept; measured 20.10.2026 in b5-18a/work: 47 keys shared by
95 records in different concepts, 0 shared pairs of concept and key). The claims of the list that come from one
fragment stand together, and a task holds at most TASK_FRAGMENTS fragments from one page. Per fragment: the page's
title, every sentence of the fragment (those on the list marked with the list's reason: the reader did not accept it,
or the operator's category), the excerpts the sentences show, the fragment's quote and context quotes, and the block
the fragment stands in with numbered lines. The parts are at most kpack.PART_CHARS characters. A fragment whose
section with the whole block is over the room of a part is shown with the block's lines from EXCERPT_LINES before its
quote's line to EXCERPT_LINES after it, within the block, under a line that says so, and with its context quote lines
that fall outside them on their own (order 35 A2; the operator 21.10.2026, PLAN.md § Rammer point 8); fix still checks
against the whole block. A fragment whose part is larger even so exits 2 with the fragment named. The instructions and the answer contract stand in the task
(INSTRUCTIONS, as the reader's in okfwiki/reader.py), so the agent file is thin and a change reaches the helper
without a restart (CLAUDE.md). task refuses when the agent file or the instructions share a run of eight words with a
claim, a quote or the block text in the task (`shared-words`). Each task gets a part list (`<task>.parts.json`: the
parts it may read and the one answer path it writes), which the breach count reads (okfwiki.cost isolation --line
repair). task writes only in --out-dir and never over a task already written.
- The answer (okf-wiki/repair-answers/1) is keyed by (`concept_id`, the `s...` footnote): replace with one to three
new fragments (`text`, `quote`, `anchor` {file, start, end}, `provenance`, and optionally `context` as a list of
quote strings, as kpack.with_context takes them), or remove. Code sets `concept` from the old record and `status` to
"existing" (the concept exists and is never decided again), so kpack.validate asks for no `tests`, also when the old
record made the concept; the record keeps the old record's `outcome`.
- fix: every fragment of the answer is found against the same built collection and the present accepted.json before
anything is written: each key must be (`concept_id`, kassemble.label) of exactly one record of accepted.json
(`unknown-key`, `ambiguous-key`) and stand as a footnote on that page of the collection (`stale-bundle`: a collection
built before the last repair has old keys and is refused so); two answers for one key are `duplicate-answer`. New
fragments are checked with kpack.validate against their source's blocks alone (the quote verbatim on a line, the mark
from PROVENANCE; its reasons are the refusal codes, `anchor-outside-pack` for an anchor in another source) and get
their context from kpack.with_context; when the answer gives no `context`, the fragment keeps the old record's context
quotes (records keep them as {line, quote, section}; they are given to with_context as their `quote` values) where
they still stand on a heading of the path or a line of the section, and those that do not are listed. A new fragment
takes the old record's place (same `concept_id`, `pack`, `position` and order). A new footnote that another fragment
of the concept has, or that two new fragments share, is `label-taken`: the key stays unique. All or nothing: one
refused fragment, and nothing is written.
- The sentences that move: a sentence key ends in `-<n>`, the occurrence of the same text on the page. When a repair
changes another sentence's key (two equal sentences become one, or a new one equals one that stands), the link
records on that sentence move to the new key, and its claim is logged as changed (it is read again).
- The links on a repaired sentence: a link record (klink's {page, sentence, anchor, at, target}) on a sentence that is
repaired moves to the new sentence's key when the anchor text stands in the new sentence with the same occurrence
number (`at` counted again from that occurrence, outside code, as a whole word); the i-th sentence of the old fragment
is paired with the i-th sentence of its new fragments. Otherwise, and when the sentence is removed, the record is
taken out and listed; so is a record on or into a page that goes. Code never places a new link (links are made by
the model, CLAUDE.md).
- unlink --links FILE --page P --sentence KEY --anchor A --at N, and relink ... --to TARGET --bundle WIKI: take out one
link record, or give it a new target; the record must stand in the file exactly so (`link-not-found`), and the target
must be a concept page that exists (`target-missing`), not the record's own page (`self-link`) and not a target the
sentence already links to (`duplicate-link`). No other record changes. With --work (and --bundle, which unlink then
needs), the write is logged like a fix's, so diff can measure it.
- linkset --list LIST --bundle WIKI --out-dir DIR: for the list's link candidates (K17 part 4), a sentence file in the
form klink task --sentences reads and, per page, a catalogue (--catalogue) with the present targets and the pages that
share a source with the sentence. klink task takes one catalogue per run, so linkset groups the pages into tasks in
which the union of the catalogues stays within klink.CATALOGUE_CHARS, and writes one sentence file and one catalogue
per group (`group-<n>.sentences.json`, `group-<n>.catalogue`); a page whose catalogue alone is over is cut to it, the
present targets kept first, and the cut is counted. The number of groups is printed (the repair order prices the link
pass from it). The frozen linker then gets an ordinary link pass on exactly those sentences (klink task, klink
accept), and the repair order needs no new code. A candidate whose sentence or target is not in the collection is
`stale-list`.
- The home and the main pages: when the last fragment on a page is removed, the page goes as in kpack remove
(drop_dangling, left_without, pass_homes), and all it brings with it is listed. When a main page loses its last
fragment from the source it is the main page of while the page stands, it is listed as a main page without its
source; the guard in the repair order asks the model again, as when a source lacks a main page. The pages without a
home after the repair that had one before are listed.
- Every write is logged in WORK/K-work/repairs.json (okf-wiki/repairs/1, appended, never rewritten): what was done, the
fragments before and after (label), the claims that change (also the other sentences of a replaced fragment and the
sentences that moved), the link records moved or taken out, and the pages that change. What a fix expects to change
is measured, not guessed: the new state is built in a temporary copy of the collection with kassemble build and
compared with it by the same measurement as diff, and the log names the collection by its sha256 (tree_sha256).
accepted.json and the log are written with kpack.write_state (with its backup); the links file is written after them
through a temporary file beside it, its old copy kept in K-work/backup/. A temporary links file left by a write that
did not finish stops the next fix, unlink and relink (exit 2, with what to do).
- diff --old WIKI --new WIKI --sources DIR --repairs FILE --out DIR: measures after kassemble build which concept pages
have other bytes (not index.md and log.md; a page gone counts) and which claims are new, gone or have another
`excerpt_sha256` (the frozen claims extract). It compares them with what the repairs logged against --old (by its
sha256) expect, and exits 1 when the measured and the logged are not equal both ways. Out: `ids.json` with the new and
changed claims in the form claims calls --ids reads, and `sider.json` with the changed pages and those gone (K15,
K19), written also when they differ.
The refusal codes (REFUSALS below; those the repair gate tries stand also in gates/repair.manifest.json):
stale-list task, linkset: not K17's list for this claim list, or a claim, sentence or target it names is
not in the collection as listed
unknown-key fix, task: (concept_id, footnote) is no record of accepted.json
ambiguous-key fix, task: (concept_id, footnote) is more than one record of accepted.json
stale-bundle fix: the key is a record but no footnote on its page in --bundle
duplicate-answer fix: two answers for one key
label-taken fix: a new footnote another fragment of the concept has, or two new fragments share
malformed fix: not in the answer form (and kpack.validate's malformed)
anchor-outside-pack, anchor-crosses-blocks, sentence-count, source-missing, quote-missing, quote-not-in-span,
provenance-missing, provenance-unknown, tests-missing
fix: kpack.validate's reasons, the one hard check at writing, against the source alone
shared-words task: the agent file or the instructions share eight words with the task's text
target-missing relink: no concept page --to in --bundle
self-link relink: --to is the record's own page
duplicate-link relink: the sentence links to --to already
link-not-found unlink, relink: no record with this page, sentence, anchor and at
"""
from __future__ import annotations
import argparse
import copy
import hashlib
import json
import os
import re
import shutil
import stat
import sys
import tempfile
import traceback
from collections import Counter
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from okfwiki import claims, kassemble, klink, kpack, reader
from okfwiki.common import RunError
from okfwiki.kblocks import kilde_of
LIST_SCHEMA = "okf-wiki/repair-list/1"
LIST_KEYS = ("schema", "claims_sha256", "verdicts_sha256", "claims", "pages", "pairs", "links")
ANSWERS_SCHEMA = "okf-wiki/repair-answers/1"
REPAIRS_SCHEMA = "okf-wiki/repairs/1"
PARTS_SCHEMA = "okf-wiki/repair-parts/1"
IDS_SCHEMA = "okf-wiki/repair-ids/1"
PAGES_SCHEMA = "okf-wiki/repair-pages/1"
REPAIRS = "repairs.json" # in WORK/K-work
TASK_FRAGMENTS = 10 # fragments of one page per task
NEW_FRAGMENTS = 3 # a fragment is replaced by one to three
EXCERPT_LINES = 40 # lines on each side of the quote when the block is too large (measured in order 35 A2: 40 fits both)
FRAGMENT_LABEL = re.compile(rf"s[0-9a-f]{{{kassemble.LABEL_HEX}}}") # kassemble.label; context labels start with c
EXPECTED = ("pages", "pages_gone", "claims", "claims_gone")
REASON_TEXT = {"not-supported": "the reader did not accept it", "no-verdict": "the reader gave it no verdict"}
REFUSALS = {
"stale-list": "task and linkset: the list is not K17's for this claim list (claims_sha256, schema, keys), or names "
"a claim, sentence or target the collection does not have as listed",
"unknown-key": "fix and task: (concept_id, footnote) is no record of accepted.json",
"ambiguous-key": "fix and task: (concept_id, footnote) is more than one record of accepted.json",
"stale-bundle": "fix: the key is a record of accepted.json but not a footnote on its page in --bundle",
"duplicate-answer": "fix: two answers in one file have the same key",
"label-taken": "fix: a new fragment's footnote is another fragment's of the concept, or two new ones share it",
"malformed": "fix: an answer or a new fragment is not in the answer form (kpack.validate's malformed too)",
"anchor-outside-pack": "kpack.validate against the fragment's own source alone: the anchor is in another source",
"anchor-crosses-blocks": "kpack.validate: the anchor is not inside one block",
"sentence-count": "kpack.validate: the text is not one to three sentences",
"source-missing": "kpack.validate: the source cannot be read",
"quote-missing": "kpack.validate: no quote",
"quote-not-in-span": "kpack.validate: the quote is not verbatim on one line of the anchor span",
"provenance-missing": "kpack.validate: no provenance mark",
"provenance-unknown": "kpack.validate: a provenance mark not in kpack.PROVENANCE",
"tests-missing": "kpack.validate: a new concept without tests (never, as code sets status existing)",
"shared-words": "task: the agent file or the instructions share a run of eight words with a claim, a quote or the "
"block text in the task",
"target-missing": "relink: the new target is no concept page of --bundle",
"self-link": "relink: the new target is the record's own page",
"duplicate-link": "relink: the sentence already links to the new target",
"link-not-found": "unlink and relink: no record of the links file has this page, sentence, anchor and at",
}
INSTRUCTIONS = """\
# How to repair the fragments
You repair fragments of a knowledge collection. A fragment is one to three sentences on a concept page, written from
one block of a source document and resting on a quote from it. A reader, or the operator, did not accept at least one
sentence of each fragment below; those sentences are marked with the reason. Read every part of this task first: its
fragments may stand in more than one part.
For each fragment, give exactly one answer.
- `replace` it with one to three new fragments when the block bears what the fragment is there to say. Each new
fragment says what its quote and its context say, and no more. A condition the source sets stays with the statement
it limits. An instruction stays an instruction, and a possibility stays a possibility. The new fragments take the
place of the whole fragment, so carry over what its other sentences say where the block bears it.
- `remove` it when the block does not bear it.
Each new fragment has:
- `text`: one to three sentences.
- `quote`: one piece copied character for character from one numbered line of the block, inside `anchor`.
- `anchor`: {"file", "start", "end"}: the file the fragment names, and a range of lines inside its block.
- `provenance`: "documented" when the text says what its quote says; "inferred" when it draws a conclusion from lines
of its anchor that no single line states; "merged" when it combines what several lines of its anchor say.
- `context` (optional): at most two pieces, each copied character for character from a heading of the block's path or
from another line of its section: the subject a heading names, or a condition, step or setting another line states.
Leave `context` out to keep the context quotes the fragment has now; give an empty list for none.
An example from another subject. A block reads:
12 | Knead the dough for ten minutes.
13 | Let it rest in a warm place until it doubles in size.
A fragment says "Knead the dough for an hour." Its quote holds and its text does not, so replace it with
{"text": "Knead the dough for ten minutes.", "quote": "Knead the dough for ten minutes", "anchor": {"file":
"bread.md", "start": 12, "end": 13}, "provenance": "documented"}. A fragment that says "The dough must rest
overnight." has no line that bears it: remove it.
Write one JSON object to the answer path your prompt names:
{"schema": "okf-wiki/repair-answers/1", "answers": [
{"concept_id": "...", "footnote": "...", "action": "replace", "fragments": [...]},
{"concept_id": "...", "footnote": "...", "action": "remove"}]}
Copy `concept_id` and `footnote` as the fragment shows them, and answer every fragment of the task once. Then hand back
only the word done.
"""
@dataclass(frozen=True)
class Refusal:
code: str
what: str
def refuse(refusals: list[Refusal]) -> int:
for r in refusals:
print(f"refused: {r.code} {r.what}")
return 1
# --- common ----------------------------------------------------------------------------------------------------
def file_sha256(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def tree_sha256(root: Path) -> str:
"""The sha256 of a collection: every file's relative path and sha256, in path order (folder metadata left out)."""
h = hashlib.sha256()
for p in sorted(root.rglob("*")):
if p.is_file() and p.name not in kassemble.TOLERATED:
h.update(f"{p.relative_to(root).as_posix()}\0{file_sha256(p)}\n".encode("utf-8"))
return h.hexdigest()
def stem(page: str) -> str:
return klink.concept_id(klink.page_rel(page))
def fragment_of(labels: Any) -> str | None:
"""The fragment's footnote among a sentence's labels (its context, version and week labels left out)."""
return next((str(x) for x in labels if FRAGMENT_LABEL.fullmatch(str(x))), None)
def keyed(acc: dict[str, Any]) -> dict[tuple[str, str], list[int]]:
out: dict[tuple[str, str], list[int]] = {}
for i, r in enumerate(acc["fragments"]):
out.setdefault((str(r["concept_id"]), kassemble.label(r)), []).append(i)
return out
def footnotes_on(bundle: Path, cid: str) -> set[str]:
page = bundle / klink.page_rel(cid)
if not page.is_file():
return set()
return {x for s in klink.page_sentences(page.read_text(encoding="utf-8")) for x in s.labels}
def block_of(blocks_doc: dict[str, Any], r: dict[str, Any]) -> dict[str, Any]:
block = next((b for b in blocks_doc["blocks"] if b["file"] == r["file"] and b["start"] <= r["start"]
and r["end"] <= b["end"]), None)
if block is None:
raise RunError(f"--blocks has no block for {r['concept_id']} {kassemble.label(r)} ({r['file']} "
f"{r['start']}-{r['end']})")
return dict(block)
def source_lines(kilde: Path, file: str, cache: dict[str, list[str]]) -> list[str]:
if file not in cache:
cache[file] = (kilde / file).read_text(encoding="utf-8").split("\n")
return cache[file]
def links_tmp(links: Path) -> Path:
return links.with_name(f".{links.name}.krepair-tmp")
def check_links_tmp(links: Path) -> None:
tmp = links_tmp(links)
if tmp.exists():
raise RunError(f"{tmp} exists: an earlier repair did not finish its links write. When the last entry of "
f"K-work/{REPAIRS} is that repair, move {tmp} to {links}; else delete it")
def read_repairs(work: Path) -> dict[str, Any]:
path = kpack.k_work(work) / REPAIRS
if not path.is_file():
return {"schema": REPAIRS_SCHEMA, "entries": []}
doc: dict[str, Any] = kpack.read_json(path)
if doc.get("schema") != REPAIRS_SCHEMA or not isinstance(doc.get("entries"), list):
raise RunError(f"{path} is not {REPAIRS_SCHEMA}")
return doc
def now() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def commit(work: Path, label: str, files: dict[str, Any], links: Path, links_doc: dict[str, Any] | None) -> None:
"""accepted.json and the log under kpack's marker and backup, then the links file through a temporary file."""
tmp = links_tmp(links)
if links_doc is not None:
tmp.write_text(klink.dump(links_doc), encoding="utf-8")
kpack.write_state(work, label, files)
if links_doc is not None:
shutil.copy2(links, kpack.k_work(work) / "backup" / links.name)
os.replace(tmp, links)
def write_links(links: Path, links_doc: dict[str, Any]) -> None:
tmp = links_tmp(links)
tmp.write_text(klink.dump(links_doc), encoding="utf-8")
os.replace(tmp, links)
def with_records(doc: dict[str, Any], records: list[dict[str, Any]]) -> dict[str, Any]:
return {**doc, "links": records, **({"accepted": len(records)} if "accepted" in doc else {})}
# --- the measurement (diff, and what a fix expects) ------------------------------------------------------------
@dataclass
class Measured:
pages: list[str] # concept pages with other bytes, a page gone among them
pages_gone: list[str]
claims: list[str] # ids new or with another excerpt_sha256
claims_gone: list[str] # ids gone or with another excerpt_sha256
def as_dict(self) -> dict[str, list[str]]:
return {"pages": self.pages, "pages_gone": self.pages_gone, "claims": self.claims,
"claims_gone": self.claims_gone}
def concept_pages(wiki: Path) -> dict[str, bytes]:
return {p.relative_to(wiki).as_posix(): p.read_bytes() for p in sorted(wiki.rglob("*.md"))
if klink.is_concept(p.relative_to(wiki).as_posix())}
def measure(old: Path, new: Path, sources: Path) -> Measured:
a, z = concept_pages(old), concept_pages(new)
before = Counter((c.id, c.excerpt_sha256) for c in claims.extract(old, sources).claims)
after = Counter((c.id, c.excerpt_sha256) for c in claims.extract(new, sources).claims)
return Measured(sorted(n for n in set(a) | set(z) if a.get(n) != z.get(n)), sorted(set(a) - set(z)),
sorted({i for i, _ in after - before}), sorted({i for i, _ in before - after}))
# --- lists -----------------------------------------------------------------------------------------------------
def read_list(path: Path, claims_path: Path | None) -> tuple[dict[str, Any], list[Refusal]]:
doc = kpack.read_json(path)
if not isinstance(doc, dict):
raise RunError(f"{path} is not a repair list")
name = path.name
out = []
if doc.get("schema") != LIST_SCHEMA:
out.append(Refusal("stale-list", f"{name} (schema {doc.get('schema')!r}, not {LIST_SCHEMA})"))
out += [Refusal("stale-list", f"{name} (unknown top-level key {k!r})") for k in sorted(set(doc) - set(LIST_KEYS))]
out += [Refusal("stale-list", f"{name} (no {k})") for k in LIST_KEYS if k not in doc]
if claims_path is not None and doc.get("claims_sha256") != file_sha256(claims_path):
out.append(Refusal("stale-list", f"{name} (claims_sha256 is not the sha256 of {claims_path.name})"))
if out:
return doc, out
rows, links = doc["claims"], doc["links"]
if not (isinstance(rows, list) and all(
isinstance(c, dict) and isinstance(c.get("id"), str) and isinstance(c.get("excerpt_sha256"), str)
and isinstance(c.get("reasons", []), list) for c in rows)):
raise RunError(f"{path}: claims is not a list of records with id, excerpt_sha256 and reasons")
if not (isinstance(links, list) and all(
isinstance(x, dict) and all(type(x.get(k)) is t for k, t in kassemble.LINK_KEYS.items()) for x in links)):
raise RunError(f"{path}: links is not a list of records with {', '.join(kassemble.LINK_KEYS)}")
return doc, out
# --- list ------------------------------------------------------------------------------------------------------
def cmd_list(args: argparse.Namespace) -> int:
claims_path, verdicts_path, out = Path(args.claims), Path(args.verdicts), Path(args.out)
if out.exists():
raise RunError(f"{out} exists: a list already written is never overwritten")
found = claims.load_claims(claims_path)
verdicts = kpack.read_json(verdicts_path)
rows_in = verdicts.get("verdicts") if isinstance(verdicts, dict) else None
if not (isinstance(rows_in, list) and all(isinstance(v, dict) and isinstance(v.get("id"), str) for v in rows_in)):
raise RunError(f"{verdicts_path} is not the reader's verdicts (reader collect --out)")
given: dict[str, list[str]] = {}
for v in rows_in:
given.setdefault(v["id"], []).append(str(v.get("verdict")))
known = {c.id for c in found.claims}
strangers = sorted(set(given) - known)
if strangers:
raise RunError(f"{verdicts_path.name} has verdicts for {len(strangers)} claims that {claims_path.name} lacks: "
+ " ".join(strangers[:5]))
covered = known
if args.ids:
ids = kpack.read_json(Path(args.ids))
wanted = ids.get("ids") if isinstance(ids, dict) else None
if not (isinstance(wanted, list) and all(isinstance(i, str) for i in wanted)):
raise RunError(f"{args.ids} has no list of ids")
covered = set(wanted)
if covered - known:
raise RunError(f"{args.ids} names {len(covered - known)} claims that {claims_path.name} lacks: "
+ " ".join(sorted(covered - known)[:5]))
rows = [{"id": c.id, "excerpt_sha256": c.excerpt_sha256, "page": c.page, "line": c.line,
"provenance": c.provenance, "sources": [{"file": f"kilde/{e.file}", "line": e.line} for e in c.excerpts],
"origins": ["reader"], "reasons": ["not-supported" if c.id in given else "no-verdict"]}
for c in found.claims if c.id in covered and given.get(c.id) != [reader.SUPPORTED]]
doc = {"schema": LIST_SCHEMA, "claims_sha256": file_sha256(claims_path),
"verdicts_sha256": file_sha256(verdicts_path), "claims": rows, "pages": [], "pairs": [], "links": []}
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
silent = sum(1 for r in rows if r["reasons"] == ["no-verdict"])
print(f"list: {len(rows)} of {len(covered)} claims read are on the list ({len(rows) - silent} not supported, "
f"{silent} without a verdict) -> {out}")
return 0
# --- task ------------------------------------------------------------------------------------------------------
@dataclass
class Item:
"""One fragment of a task: its record, its page's title and the sentences it gives the page."""
concept_id: str
label: str
record: dict[str, Any]
title: str
sentences: list[dict[str, Any]] # the claims of the fragment on its page, in page order (from --claims)
reasons: dict[str, list[str]] = field(default_factory=dict) # listed claim id -> its reasons
def reason_text(reasons: list[str]) -> str:
return "; ".join(dict.fromkeys(REASON_TEXT.get(r, f"the operator: {r}") for r in reasons))
def excerpts_of(c: dict[str, Any], pairs: dict[str, reader.Pair]) -> list[reader.Excerpt]:
p = pairs.get(str(c["id"]))
if p is not None:
return list(p.excerpts)
return [reader.Excerpt(str(e["file"]), int(e["line"]), str(e["text"])) for e in c.get("excerpts", [])]
def block_text(item: Item, blocks_doc: dict[str, Any], kilde: Path, cache: dict[str, list[str]]) -> tuple[
dict[str, Any], list[str]]:
block = block_of(blocks_doc, item.record)
lines = source_lines(kilde, str(item.record["file"]), cache)
return block, [lines[n - 1] if n <= len(lines) else "" for n in range(block["start"], block["end"] + 1)]
def render_item(n: int, item: Item, pairs: dict[str, reader.Pair], blocks_doc: dict[str, Any], kilde: Path,
cache: dict[str, list[str]], excerpt: bool = False) -> str:
r = item.record
out = [f"## Fragment {n}", "", f"- concept_id: `{item.concept_id}`", f"- footnote: `{item.label}`",
f"- page: {item.title}", "", "Its sentences, as the page shows them:", ""]
for k, c in enumerate(item.sentences, 1):
mark = f" (on the list: {reason_text(item.reasons[c['id']])})" if c["id"] in item.reasons else ""
out.append(f"{k}. {klink.key_text(str(c['claim']))}{mark}")
out += [f' <excerpt file="{e.file}" line="{e.line}">{e.text}</excerpt>' for e in excerpts_of(c, pairs)]
context = r.get("context", [])
out += ["", f"Quote: `{r['quote']}` ({r['file']}, line {r['quote_line']}), provenance {r['provenance']}.",
"Context quotes: " + ("; ".join(f"`{c['quote']}` (line {c['line']})" for c in context) if context
else "none") + "."]
block, lines = block_text(item, blocks_doc, kilde, cache)
out += [f"Anchor: {r['file']}, lines {r['start']}-{r['end']}.",
f"Block: {r['file']} › {block['section']}, lines {block['start']}-{block['end']}:"]
first, last = block["start"], block["end"]
if excerpt: # the quote stands on one line (kpack.validate), so its first and last line are quote_line
q = int(r["quote_line"])
first, last = max(first, q - EXCERPT_LINES), min(last, q + EXCERPT_LINES)
out.append(f"The block is shown only in part: lines {first}-{last} of lines {block['start']}-{block['end']} "
f"({len(lines)} lines). Judge the fragment from the lines shown.")
out.append("")
out += [f"{k:>5} | {lines[k - block['start']]}" for k in range(first, last + 1)]
outside = sorted({int(c["line"]) for c in context} - set(range(first, last + 1)))
if excerpt and outside:
src = source_lines(kilde, str(r["file"]), cache)
out += ["", "Context quote lines outside the lines shown:", ""]
out += [f"{k:>5} | {src[k - 1] if k <= len(src) else ''}" for k in outside]
return "\n".join(out) + "\n\n"
def part_head(task: str, i: int, n: int) -> str:
return f"<!-- part {i} of {n} -->\n# Repair task {task}, part {i} of {n}\n\n"
def part_room(task: str) -> int:
return kpack.PART_CHARS - len(part_head(task, 999, 999))
def pack_parts(task: str, sections: list[tuple[str, str]]) -> list[str]:
"""The task's parts: the instructions first, then whole fragment sections, each part within kpack.PART_CHARS."""
limit = kpack.PART_CHARS
room = part_room(task)
for what, text in sections[1:] + sections[:1]: # a fragment that cannot fit is named before the instructions
if len(text) > room:
raise RunError(f"{what} needs {len(text)} characters, over the room of {room} in a part of {limit}: it "
f"cannot stand in one part")
bodies: list[str] = []
for _, text in sections:
if bodies and len(bodies[-1]) + len(text) <= room:
bodies[-1] += text
else:
bodies.append(text)
return [part_head(task, i, len(bodies)) + body for i, body in enumerate(bodies, 1)]
def task_items(lst: dict[str, Any], claims_doc: dict[str, Any], acc: dict[str, Any], index: dict[str, Any],
claims_name: str) -> tuple[list[Item], list[Refusal]]:
by_id = {str(c["id"]): c for c in claims_doc["claims"]}
keys = keyed(acc)
titles = {str(e["id"]): kassemble.one_line(e["title"]) for e in index["entries"]}
items: dict[tuple[str, str], Item] = {}
refusals: list[Refusal] = []
for row in lst["claims"]:
cid_claim = str(row["id"])
c = by_id.get(cid_claim)
if c is None or c.get("excerpt_sha256") != row["excerpt_sha256"]:
why = "not a claim of" if c is None else "shown with another excerpt_sha256 in"
refusals.append(Refusal("stale-list", f"{cid_claim} ({why} {claims_name})"))
continue
label = fragment_of(c.get("footnotes", []))
concept = stem(str(c["page"]))
hits = keys.get((concept, label or ""), [])
if label is None or len(hits) != 1:
code = "ambiguous-key" if len(hits) > 1 else "unknown-key"
refusals.append(Refusal(code, f"{cid_claim} ({concept} {label or 'no footnote'})"))
continue
key = (concept, label)
if key not in items:
sentences = [x for x in claims_doc["claims"] if str(x["page"]) == str(c["page"])
and fragment_of(x.get("footnotes", [])) == label]
items[key] = Item(concept, label, acc["fragments"][hits[0]], titles.get(concept, concept), sentences)
items[key].reasons[cid_claim] = [str(x) for x in row.get("reasons", [])] or ["not-supported"]
return list(items.values()), refusals
def cmd_task(args: argparse.Namespace) -> int:
claims_path, work, out = Path(args.claims), Path(args.work), Path(args.out_dir).resolve()
kpack.check_pending(work)
lst, refusals = read_list(Path(args.list), claims_path)
if refusals:
return refuse(refusals)
claims_doc: dict[str, Any] = kpack.read_json(claims_path)
pairs = {p.id: p for p in reader.load_pairs(Path(args.pairs))}
acc, index = kpack.read_accepted(work), kpack.read_index(work)
blocks_doc: dict[str, Any] = kpack.read_json(Path(args.blocks))
kilde = kilde_of(Path(args.sources))
items, refusals = task_items(lst, claims_doc, acc, index, claims_path.name)
if refusals:
return refuse(refusals)
if not items:
raise RunError(f"{args.list} lists no claim")
cache: dict[str, list[str]] = {}
material: list[reader.Pair] = []
for it in items:
what = f"{it.concept_id} {it.label}"
material += [reader.Pair(f"{c['id']}", klink.key_text(str(c["claim"])), tuple(excerpts_of(c, pairs)))
for c in it.sentences]
quotes = [str(it.record["quote"]), *(str(c["quote"]) for c in it.record.get("context", []))]
material.append(reader.Pair(f"{what} (quotes)", " ".join(quotes), ()))
material.append(reader.Pair(f"{what} (block)", "\n".join(block_text(it, blocks_doc, kilde, cache)[1]), ()))
agent = Path(args.agent)
for name, text in ((agent.name, agent.read_text(encoding="utf-8")), ("the instructions", INSTRUCTIONS)):
refusals += [Refusal("shared-words", f"{name} and {pid}") for pid in reader.shared_runs(text, material)]
if refusals:
return refuse(refusals)
by_page: dict[str, list[Item]] = {}
for it in items:
by_page.setdefault(it.concept_id, []).append(it)
files: dict[Path, str] = {}
tasks = 0
for concept in by_page:
chunk = by_page[concept]
for i in range(0, len(chunk), TASK_FRAGMENTS):
tasks += 1
name = f"r{tasks:03d}"
group = chunk[i:i + TASK_FRAGMENTS]
sections = [("the instructions", INSTRUCTIONS + "\n")]
for n, it in enumerate(group, 1):
text = render_item(n, it, pairs, blocks_doc, kilde, cache)
if len(text) > part_room(name):
text = render_item(n, it, pairs, blocks_doc, kilde, cache, excerpt=True)
sections.append((f"fragment {it.concept_id} {it.label}", text))
parts = pack_parts(name, sections)
paths = [out / f"{name}-{k}.md" for k in range(1, len(parts) + 1)]
files.update(zip(paths, parts))
files[out / f"{name}.parts.json"] = kpack.dump({
"schema": PARTS_SCHEMA, "task": name, "parts": [str(p) for p in paths],
"answer": str(out / f"{name}.answers.json"), "chars": [len(p) for p in parts],
"fragments": [{"concept_id": it.concept_id, "footnote": it.label, "claims": sorted(it.reasons)}
for it in group]})
taken = sorted(str(p) for p in files if p.exists()) + sorted(str(p) for p in out.glob("*.parts.json"))
if taken:
raise RunError(f"tasks exist in {out} ({taken[0]}); a task already written is never changed")
out.mkdir(parents=True, exist_ok=True)
for path, text in files.items():
path.write_text(text, encoding="utf-8")
listed = sum(len(it.reasons) for it in items)
print(f"task: {listed} claims, {len(items)} fragments, {len(by_page)} pages, {tasks} tasks, "
f"{sum(1 for p in files if p.suffix == '.md')} parts -> {out}")
return 0
# --- fix -------------------------------------------------------------------------------------------------------
@dataclass
class Answer:
concept_id: str
footnote: str
action: str
fragments: list[dict[str, Any]]
@property
def what(self) -> str:
return f"{self.concept_id} {self.footnote}"
def read_answers(path: Path) -> tuple[list[Answer], list[Refusal]]:
doc = kpack.read_json(path)
if not (isinstance(doc, dict) and doc.get("schema") == ANSWERS_SCHEMA and isinstance(doc.get("answers"), list)):
raise RunError(f"{path} is not {ANSWERS_SCHEMA} with a list of answers")
if not doc["answers"]:
raise RunError(f"{path} holds no answer")
out: list[Answer] = []
refusals: list[Refusal] = []
for n, a in enumerate(doc["answers"], 1):
cid, fn, act = (a.get(k) if isinstance(a, dict) else None for k in ("concept_id", "footnote", "action"))
what = f"{cid} {fn}" if isinstance(cid, str) and isinstance(fn, str) else f"answer {n}"
if not (isinstance(cid, str) and isinstance(fn, str) and act in ("replace", "remove")):
refusals.append(Refusal("malformed", f"{what} (needs concept_id, footnote and action replace or remove)"))
continue
frags = a.get("fragments")
if act == "replace" and not (isinstance(frags, list) and 1 <= len(frags) <= NEW_FRAGMENTS
and all(isinstance(f, dict) for f in frags)):
refusals.append(Refusal("malformed", f"{what} (replace takes one to {NEW_FRAGMENTS} fragments)"))
continue
if act == "remove" and frags is not None:
refusals.append(Refusal("malformed", f"{what} (remove takes no fragments)"))
continue
out.append(Answer(cid, fn, str(act), list(frags or [])))
return out, refusals
@dataclass
class Checked:
"""The answers that passed, by the index of the record they repair, with what fix lists."""
new: dict[int, list[dict[str, Any]]] = field(default_factory=dict) # old record index -> new records ([] removes)
keys: dict[int, tuple[str, str]] = field(default_factory=dict)
context_not_kept: list[dict[str, Any]] = field(default_factory=list)
def check_answers(answers: list[Answer], acc: dict[str, Any], index: dict[str, Any], bundle: Path,
blocks_doc: dict[str, Any], kilde: Path) -> tuple[Checked, list[Refusal]]:
keys = keyed(acc)
counts = Counter((a.concept_id, a.footnote) for a in answers)
refusals = [Refusal("duplicate-answer", f"{c} {f} ({n} answers)") for (c, f), n in counts.items() if n > 1]
done = Checked()
shown: dict[str, set[str]] = {}
fresh: dict[str, set[str]] = {}
for a in answers:
hits = keys.get((a.concept_id, a.footnote), [])
if len(hits) != 1:
refusals.append(Refusal("ambiguous-key" if hits else "unknown-key", a.what))
continue
if a.concept_id not in shown:
shown[a.concept_id] = footnotes_on(bundle, a.concept_id)
if a.footnote not in shown[a.concept_id]:
refusals.append(Refusal("stale-bundle", f"{a.what} (no footnote on {klink.page_rel(a.concept_id)} in "
f"{bundle})"))
continue
i = hits[0]
old = acc["fragments"][i]
records: list[dict[str, Any]] = []
ok = True
for k, given in enumerate(a.fragments, 1):
f: dict[str, Any] = {"concept": old["concept_id"], "status": "existing",
**{x: given[x] for x in ("text", "quote", "anchor", "provenance") if x in given}}
inherited = "context" not in given
f["context"] = [c["quote"] for c in old.get("context", [])] if inherited else given["context"]
valid, rejected = kpack.validate([f], [str(old["file"])], blocks_doc, kilde)
if rejected:
refusals += [Refusal(reason, f"{a.what} (new fragment {k})") for reason in sorted(rejected)]
ok = False
continue
placed, _ = kpack.with_context(valid, blocks_doc, kilde)
rec = {**kpack.new_record(placed[0], "existing", str(old["concept_id"]), index, int(old["pack"]),
int(old["position"])), "outcome": old.get("outcome", "existing")}
label = kassemble.label(rec)
other = [j for j in keys.get((a.concept_id, label), []) if j != i]
if other or label in fresh.setdefault(a.concept_id, set()):
whose = f"of {kassemble.label(acc['fragments'][other[0]])}" if other else "of another new fragment"
refusals.append(Refusal("label-taken", f"{a.what} (new fragment {k} has the footnote {label}, the "
f"footnote {whose} in {a.concept_id})"))
ok = False
continue
fresh[a.concept_id].add(label)
if inherited:
kept = {str(c["quote"]) for c in rec.get("context", [])}
done.context_not_kept += [{"concept_id": a.concept_id, "footnote": a.footnote, "quote": str(q)}
for q in f["context"] if str(q) not in kept]
records.append(rec)
if ok:
done.new[i], done.keys[i] = records, (a.concept_id, a.footnote)
return done, refusals
Slot = tuple[str, int]
def slotted(page: kassemble.Page) -> list[tuple[klink.Sentence, Slot | None]]:
"""The sentences of an assembled page as klink keys them, each with its slot: (its fragment's footnote, its place
among that fragment's sentences on the page)."""
seen: Counter[str] = Counter()
out: list[tuple[klink.Sentence, Slot | None]] = []
for s in klink.body_sentences(klink.reduce_links(page.body)[0]):
label = fragment_of(s.labels)
if label is None:
out.append((s, None))
continue
out.append((s, (label, seen[label])))
seen[label] += 1
return out
def sentences_of(by_slot: dict[Slot, klink.Sentence], label: str) -> list[klink.Sentence]:
out: list[klink.Sentence] = []
while (label, len(out)) in by_slot:
out.append(by_slot[(label, len(out))])
return out
def occurrences(text: str, anchor: str) -> list[int]:
out, at = [], text.find(anchor)
while at >= 0:
out.append(at)
at = text.find(anchor, at + 1)
return out
def same_occurrence(old: klink.Sentence, anchor: str, at: int, new: klink.Sentence) -> int | None:
"""The offset of the anchor in the new sentence at the occurrence number it has in the old one, when it stands
there outside code and as a whole word (as klink.find_anchor accepts it); else None."""
before = occurrences(old.text, anchor)
if at not in before:
return None
after = occurrences(new.text, anchor)
k = before.index(at)
if k >= len(after):
return None
p, e = after[k], after[k] + len(anchor)
whole = (p == 0 or not new.text[p - 1].isalnum()) and (e == len(new.text) or not new.text[e].isalnum())
return p if whole and new.masked[p:e] == anchor else None
@dataclass
class Moves:
records: list[dict[str, Any]]
moved: list[dict[str, Any]] = field(default_factory=list)
removed: list[dict[str, Any]] = field(default_factory=list)
sentences_moved: list[dict[str, str]] = field(default_factory=list)
touched_before: list[str] = field(default_factory=list)
touched_after: list[str] = field(default_factory=list)
def move_links(records: list[dict[str, Any]], old_pages: dict[str, kassemble.Page],
new_pages: dict[str, kassemble.Page], changed: dict[tuple[str, str], list[str]],
gone: set[str]) -> Moves:
"""The link records after the repair: `changed` maps each repaired (concept, footnote) to its new footnotes ([]
for a removal); `gone` holds the pages that go."""
m = Moves([])
old_s = {rel: slotted(p) for rel, p in old_pages.items()}
old_keyed = {rel: {s.key: (s, slot) for s, slot in sentences if slot is not None}
for rel, sentences in old_s.items()}
new_s = {rel: {slot: s for s, slot in slotted(p) if slot is not None} for rel, p in new_pages.items()}
for rel, sentences in old_s.items():
cid = stem(rel)
for s, slot in sentences:
if slot is None:
continue
if (cid, slot[0]) in changed:
m.touched_before.append(f"{rel}#{s.key}")
continue
now_s = new_s.get(rel, {}).get(slot)
if now_s is not None and now_s.key != s.key:
m.sentences_moved.append({"page": rel, "before": s.key, "after": now_s.key})
m.touched_before.append(f"{rel}#{s.key}")
m.touched_after.append(f"{rel}#{now_s.key}")
for rel, by_slot in new_s.items():
cid = stem(rel)
new_labels = {x for (c, _), labels in changed.items() if c == cid for x in labels}
m.touched_after += [f"{rel}#{s.key}" for slot, s in by_slot.items() if slot[0] in new_labels]
for r in records:
page, target = klink.page_rel(str(r["page"])), klink.page_rel(str(r["target"]))
if page in gone or target in gone:
m.removed.append({"record": r, "why": "its page goes" if page in gone else "its target goes"})
continue
hit = old_keyed.get(page, {}).get(str(r["sentence"]))
if hit is None:
m.records.append(r) # on a page the repair does not touch, or stale already: left as it was
continue
s, (label, i) = hit
cid = stem(page)
new = new_s.get(page, {})
if (cid, label) in changed:
pool = [s2 for x in changed[(cid, label)] for s2 in sentences_of(new, x)]
if not pool:
m.removed.append({"record": r, "why": "its sentence is removed"})
continue
if i >= len(pool):
m.removed.append({"record": r, "why": "its fragment has no sentence in its place now"})
continue
to = pool[i]
at = same_occurrence(s, str(r["anchor"]), int(r["at"]), to)
elif (label, i) not in new:
m.removed.append({"record": r, "why": "its sentence is not on the page now"})
continue
else:
to = new[(label, i)]
at = int(r["at"]) if to.text[int(r["at"]):int(r["at"]) + len(str(r["anchor"]))] == r["anchor"] else None
if at is None:
m.removed.append({"record": r, "why": "its anchor is not in the new sentence at its occurrence"})
continue
moved = {**r, "sentence": to.key, "at": at}
if moved != r:
m.moved.append({"before": r, "after": moved})
m.records.append(moved)
return m
def roles(acc: dict[str, Any], facts: dict[str, kpack.Fact]) -> tuple[set[str], set[str], dict[str, set[str]]]:
"""(the main pages, the pages with a home, the sources each page cites) as kassemble renders them: a main page
cites the main-topic source it is named for, and a home is a link from a main page."""
cited: dict[str, set[str]] = {}
for r in acc["fragments"]:
cited.setdefault(str(r["concept_id"]), set()).add(str(r["file"]))
mains = {str(m["concept_id"]) for m in acc["main_pages"]
if str(m["source"]) in cited.get(str(m["concept_id"]), set())
and str(m["source"]) in facts and facts[str(m["source"])].main_topic}
homed = {str(h["concept_id"]) for h in acc["homes"] if str(h["home"]) in mains and h["home"] != h["concept_id"]
and str(h["concept_id"]) in cited}
return mains, homed, cited
def without_home(acc: dict[str, Any], facts: dict[str, kpack.Fact]) -> set[str]:
mains, homed, cited = roles(acc, facts)
return set(cited) - mains - homed
def trial(work: Path, bundle: Path, sources: Path, acc: dict[str, Any], links_doc: dict[str, Any]) -> Measured:
"""What the repair will change: the new state built into a copy of the collection, measured against it."""
with tempfile.TemporaryDirectory(prefix="krepair-trial-") as tmp_name:
tmp = Path(tmp_name)
kw = tmp / "work" / "K-work"
kw.mkdir(parents=True)
shutil.copyfile(kpack.k_work(work) / "index.json", kw / "index.json")
(kw / "accepted.json").write_text(kpack.dump(acc), encoding="utf-8")
(tmp / "links.json").write_text(klink.dump(links_doc), encoding="utf-8")
wiki = tmp / "wiki"
shutil.copytree(bundle, wiki, copy_function=shutil.copyfile) # the copy is ours, whatever the bundle's modes
for p in [wiki, *wiki.rglob("*")]:
p.chmod(p.stat().st_mode | stat.S_IWUSR)
report = kassemble.build(tmp / "work", sources, wiki, tmp / "links.json", replace=True)
if report.stale:
raise RunError(f"after the repair {report.stale} link records would not stand: a fault in krepair")
return measure(bundle, wiki, sources)
def cmd_fix(args: argparse.Namespace) -> int:
work, bundle, sources, links = Path(args.work), Path(args.bundle), Path(args.sources), Path(args.links)
kpack.check_pending(work)
check_links_tmp(links)
answers_path = Path(args.answers)
answers, refusals = read_answers(answers_path)
acc, index = kpack.read_accepted(work), kpack.read_index(work)
blocks_doc: dict[str, Any] = kpack.read_json(Path(args.blocks))
kilde = kilde_of(sources)
links_doc = kassemble.check_links(links)
checked, more = check_answers(answers, acc, index, bundle, blocks_doc, kilde)
refusals += more
if refusals:
return refuse(refusals)
new_acc = copy.deepcopy(acc)
new_acc["fragments"] = [copy.deepcopy(x) for i, r in enumerate(acc["fragments"])
for x in (checked.new[i] if i in checked.new else [r])]
dropped = kpack.drop_dangling(new_acc)
passed = kpack.pass_homes(new_acc)
files = sorted({str(r["file"]) for r in acc["fragments"] + new_acc["fragments"]}
| {str(m["source"]) for m in acc["main_pages"]})
facts = kpack.source_facts(sources, files)
changed = {checked.keys[i]: [kassemble.label(r) for r in recs] for i, recs in checked.new.items()}
concepts = {c for c, _ in changed}
old_pages = kassemble.page_models(index, [r for r in acc["fragments"] if r["concept_id"] in concepts], kilde, facts,
acc["main_pages"])
new_pages = kassemble.page_models(index, [r for r in new_acc["fragments"] if r["concept_id"] in concepts], kilde,
facts, new_acc["main_pages"])
gone = {klink.page_rel(c) for c in kpack.pages_of(acc) - kpack.pages_of(new_acc)}
moves = move_links([dict(r) for r in links_doc["links"]], old_pages, new_pages, changed, gone)
new_links = with_records(links_doc, moves.records)
expected = trial(work, bundle, sources, new_acc, new_links)
_, _, cited_before = roles(acc, facts)
_, _, cited_after = roles(new_acc, facts)
no_source = [{"concept_id": str(m["concept_id"]), "source": str(m["source"])} for m in new_acc["main_pages"]
if str(m["concept_id"]) in cited_after and str(m["source"]) not in cited_after[str(m["concept_id"])]
and str(m["source"]) in cited_before.get(str(m["concept_id"]), set())]
left = kpack.left_without(new_acc, set(kpack.main_ids(acc)), dropped[2])
homeless = sorted(set(left) | (without_home(new_acc, facts) - without_home(acc, facts)))
log = read_repairs(work)
entry = {"n": len(log["entries"]) + 1, "command": "fix", "at": now(), "answers": str(answers_path),
"answers_sha256": file_sha256(answers_path), "bundle": str(bundle), "bundle_sha256": tree_sha256(bundle),
"fragments": [{"concept_id": checked.keys[i][0], "before": checked.keys[i][1],
"action": "replace" if recs else "remove", "after": [kassemble.label(r) for r in recs]}
for i, recs in sorted(checked.new.items())],
"claims_touched": {"before": sorted(set(moves.touched_before)), "after": sorted(set(moves.touched_after))},
"sentences_moved": moves.sentences_moved,
"links": {"moved": moves.moved, "removed": moves.removed},
"listed": {"pages_gone": sorted(gone), "without_home": homeless, "main_page_without_source": no_source,
"context_not_kept": checked.context_not_kept,
"dropped": {"main_pages": dropped[0], "homes": dropped[1], "homes_passed": passed}},
"expected": expected.as_dict()}
log["entries"].append(entry)
commit(work, "repair", {"accepted.json": new_acc, REPAIRS: log}, links,
new_links if moves.moved or moves.removed else None)
removed = sum(not recs for recs in checked.new.values())
print(f"fix: {len(checked.new)} answers: replaced {len(checked.new) - removed} fragments with "
f"{sum(len(recs) for recs in checked.new.values())}, removed {removed} -> "
f"{kpack.k_work(work) / 'accepted.json'}")
print(f" expected to change: {len(expected.pages)} pages ({len(expected.pages_gone)} gone), "
f"{len(expected.claims)} claims new or changed, {len(expected.claims_gone)} gone or changed")
print(f" links: moved {len(moves.moved)}, taken out {len(moves.removed)}; sentences moved "
f"{len(moves.sentences_moved)}")
for x in moves.removed:
r = x["record"]
print(f" taken out: {r['page']} {r['sentence']} {r['anchor']!r} -> {r['target']} ({x['why']})")
for x in checked.context_not_kept:
print(f" context not kept: {x['concept_id']} {x['footnote']}: {x['quote']!r}")
for name in sorted(gone):
print(f" page gone: {name}")
for name in homeless:
print(f" without a home now: {name}")
for x in no_source:
print(f" main page without its source: {x['concept_id']} ({x['source']})")
print(f" logged as repair {entry['n']} in {kpack.k_work(work) / REPAIRS}")
return 0
# --- unlink and relink -----------------------------------------------------------------------------------------
def find_record(records: list[dict[str, Any]], args: argparse.Namespace) -> int | None:
page = klink.page_rel(args.page)
return next((i for i, r in enumerate(records) if klink.page_rel(str(r["page"])) == page
and r["sentence"] == args.sentence and r["anchor"] == args.anchor and int(r["at"]) == args.at), None)
def link_entry(work: Path, command: str, bundle: Path, page: str, before: dict[str, Any],
after: dict[str, Any] | None) -> dict[str, Any]:
links = {"moved": [{"before": before, "after": after}], "removed": []} if after else {
"moved": [], "removed": [{"record": before, "why": "unlink"}]}
return {"n": len(read_repairs(work)["entries"]) + 1, "command": command, "at": now(), "bundle": str(bundle),
"bundle_sha256": tree_sha256(bundle), "links": links,
"expected": {"pages": [page], "pages_gone": [], "claims": [], "claims_gone": []}}
def write_link_change(args: argparse.Namespace, command: str, links: Path, doc: dict[str, Any],
records: list[dict[str, Any]], before: dict[str, Any], after: dict[str, Any] | None) -> None:
new_doc = with_records(doc, records)
if args.work is None:
write_links(links, new_doc)
return
work = Path(args.work)
kpack.check_pending(work)
log = read_repairs(work)
log["entries"].append(link_entry(work, command, Path(args.bundle), klink.page_rel(args.page), before, after))
commit(work, command, {REPAIRS: log}, links, new_doc)
def link_args(args: argparse.Namespace) -> tuple[Path, dict[str, Any], list[dict[str, Any]]]:
if args.work is not None and args.bundle is None:
raise RunError("--work needs --bundle: the log names the collection the record was taken out of")
links = Path(args.links)
check_links_tmp(links)
doc = kassemble.check_links(links)
return links, doc, [dict(r) for r in doc["links"]]
def cmd_unlink(args: argparse.Namespace) -> int:
links, doc, records = link_args(args)
i = find_record(records, args)
if i is None:
return refuse([Refusal("link-not-found", f"{klink.page_rel(args.page)} {args.sentence} {args.anchor!r} at "
f"{args.at}")])
before = records.pop(i)
write_link_change(args, "unlink", links, doc, records, before, None)
print(f"unlink: took out {before['page']} {before['sentence']} {before['anchor']!r} -> {before['target']}")
return 0
def cmd_relink(args: argparse.Namespace) -> int:
links, doc, records = link_args(args)
i = find_record(records, args)
page, target = klink.page_rel(args.page), klink.page_rel(args.to)
what = f"{page} {args.sentence} {args.anchor!r} at {args.at} -> {target}"
if i is None:
return refuse([Refusal("link-not-found", what)])
if target not in klink.read_pages(Path(args.bundle)):
return refuse([Refusal("target-missing", what)])
if target == page:
return refuse([Refusal("self-link", what)])
if any(klink.page_rel(str(r["page"])) == page and r["sentence"] == args.sentence
and klink.page_rel(str(r["target"])) == target for r in records):
return refuse([Refusal("duplicate-link", what)])
before = records[i]
after = {**before, "target": target}
records[i] = after
write_link_change(args, "relink", links, doc, records, before, after)
print(f"relink: {before['page']} {before['sentence']} {before['anchor']!r} -> {target} (was {before['target']})")
return 0
# --- linkset ---------------------------------------------------------------------------------------------------
def cmd_linkset(args: argparse.Namespace) -> int:
lst, refusals = read_list(Path(args.list), Path(args.claims) if args.claims else None)
if refusals:
return refuse(refusals)
bundle, out = Path(args.bundle), Path(args.out_dir)
pages = klink.read_pages(bundle)
sentences = {rel: {s.key: s for s in klink.body_sentences(p.body)} for rel, p in pages.items()}
wanted: dict[str, list[str]] = {}
targets: dict[str, list[str]] = {}
sharing: dict[str, set[str]] = {}
for x in lst["links"]:
page, target = klink.page_rel(str(x["page"])), klink.page_rel(str(x["target"]))
s = sentences.get(page, {}).get(str(x["sentence"]))
if s is None or target not in pages:
why = "no such sentence on the page" if s is None else f"no page {target}"
refusals.append(Refusal("stale-list", f"{page} {x['sentence']} ({why})"))
continue
if s.key not in wanted.setdefault(page, []):
wanted[page].append(s.key)
if stem(target) not in targets.setdefault(page, []):
targets[page].append(stem(target))
cites = {pages[page].resources[x] for x in s.labels if x in pages[page].resources}
sharing.setdefault(page, set()).update(rel for rel, p in pages.items() if rel != page
and cites & set(p.resources.values()))
if refusals:
return refuse(refusals)
if not wanted:
raise RunError(f"{args.list} lists no link candidate")
budget = klink.CATALOGUE_CHARS
groups: list[tuple[list[str], list[str]]] = [] # (pages, catalogue ids)
cut = 0
for page in sorted(wanted):
ids = targets[page] + sorted({stem(rel) for rel in sharing[page]} - set(targets[page]))
if groups:
union = groups[-1][1] + [i for i in ids if i not in groups[-1][1]]
if len(klink.catalogue_text(union, pages)) <= budget:
groups[-1] = (groups[-1][0] + [page], union)
continue
keep = list(targets[page])
for i in ids[len(targets[page]):]:
if len(klink.catalogue_text(keep + [i], pages)) <= budget:
keep.append(i)
cut += len(ids) - len(keep)
groups.append(([page], keep))
files: dict[Path, str] = {}
for n, (group_pages, ids) in enumerate(groups, 1):
files[out / f"group-{n}.sentences.json"] = klink.dump({"schema": klink.SENTENCES_SCHEMA,
"pages": {p: wanted[p] for p in group_pages}})
files[out / f"group-{n}.catalogue"] = "".join(f"{i}\n" for i in ids)
taken = sorted(str(p) for p in files if p.exists())
if taken:
raise RunError(f"{taken[0]} exists; refusing to overwrite")
out.mkdir(parents=True, exist_ok=True)
for path, text in files.items():
path.write_text(text, encoding="utf-8")
g = len(groups)
print(f"linkset: {sum(len(v) for v in wanted.values())} candidates on {len(wanted)} pages, {g} "
f"group{'s' if g != 1 else ''} -> {out}")
for n, (group_pages, ids) in enumerate(groups, 1):
print(f" group-{n}: {len(group_pages)} pages, {sum(len(wanted[p]) for p in group_pages)} sentences, "
f"catalogue {len(ids)} ids ({len(klink.catalogue_text(ids, pages))} characters)")
if cut:
print(f" cut {cut} catalogue ids that did not fit {budget} characters (the present targets kept)")
return 0
# --- diff ------------------------------------------------------------------------------------------------------
def cmd_diff(args: argparse.Namespace) -> int:
old, new, sources, out = Path(args.old), Path(args.new), Path(args.sources), Path(args.out)
log = kpack.read_json(Path(args.repairs))
if not (isinstance(log, dict) and log.get("schema") == REPAIRS_SCHEMA and isinstance(log.get("entries"), list)):
raise RunError(f"{args.repairs} is not {REPAIRS_SCHEMA}")
fp = tree_sha256(old)
entries = [e for e in log["entries"] if e.get("bundle_sha256") == fp]
m = measure(old, new, sources)
logged = {k: sorted({x for e in entries for x in e["expected"][k]}) for k in EXPECTED}
got = m.as_dict()
out.mkdir(parents=True, exist_ok=True)
(out / "ids.json").write_text(kpack.dump({"schema": IDS_SCHEMA, "old": str(old), "new": str(new),
"ids": m.claims}), encoding="utf-8")
(out / "sider.json").write_text(kpack.dump({"schema": PAGES_SCHEMA, "old": str(old), "new": str(new),
"pages": m.pages, "gone": m.pages_gone}), encoding="utf-8")
print(f"diff: {len(m.pages)} pages changed ({len(m.pages_gone)} gone), {len(m.claims)} claims new or changed, "
f"{len(m.claims_gone)} gone or changed; {len(entries)} of {len(log['entries'])} repairs logged against "
f"{old} -> {out}")
differs = False
if not entries and any(got.values()):
print(f"differs: no repair is logged against {old} (sha256 {fp[:12]})")
for k in EXPECTED:
extra, missing = sorted(set(got[k]) - set(logged[k])), sorted(set(logged[k]) - set(got[k]))
if extra:
print(f"differs: {k} measured, not logged: {len(extra)} ({', '.join(extra[:5])})")
if missing:
print(f"differs: {k} logged, not measured: {len(missing)} ({', '.join(missing[:5])})")
differs = differs or bool(extra or missing)
return 1 if differs else 0
# --- command line ----------------------------------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="python3 -m okfwiki.krepair", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
ls = sub.add_parser("list")
for name in ("--claims", "--verdicts", "--out"):
ls.add_argument(name, required=True)
ls.add_argument("--ids")
t = sub.add_parser("task")
for name in ("--list", "--claims", "--pairs", "--work", "--blocks", "--sources", "--agent", "--out-dir"):
t.add_argument(name, required=True)
f = sub.add_parser("fix")
for name in ("--answers", "--bundle", "--work", "--blocks", "--sources", "--links"):
f.add_argument(name, required=True)
for command in ("unlink", "relink"):
u = sub.add_parser(command)
for name in ("--links", "--page", "--sentence", "--anchor"):
u.add_argument(name, required=True)
u.add_argument("--at", type=int, required=True)
u.add_argument("--work")
u.add_argument("--bundle", required=command == "relink")
if command == "relink":
u.add_argument("--to", required=True)
s = sub.add_parser("linkset")
for name in ("--list", "--bundle", "--out-dir"):
s.add_argument(name, required=True)
s.add_argument("--claims")
d = sub.add_parser("diff")
for name in ("--old", "--new", "--sources", "--repairs", "--out"):
d.add_argument(name, required=True)
return p
COMMANDS = {"list": cmd_list, "task": cmd_task, "fix": cmd_fix, "unlink": cmd_unlink, "relink": cmd_relink, "linkset": cmd_linkset,
"diff": cmd_diff}
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
return COMMANDS[args.command](args)
except (RunError, reader.RunError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
print(f"krepair cannot run: {exc}", file=sys.stderr)
return 2
except Exception: # an input this code did not foresee: still "could not run", with the traceback
traceback.print_exc()
return 2
if __name__ == "__main__":
sys.exit(main())