First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
1212 lines
66 KiB
Python
1212 lines
66 KiB
Python
"""The repair command: one fragment of a built collection repaired or removed, one link record taken out or moved, the
|
||
helper's task, and the measurement of what a repair touched. No model call, no network, no program started (CLAUDE.md
|
||
rule 1). Order 34, contracts K18 and K19; the contract below is the order's, point for point, in English.
|
||
|
||
Usage:
|
||
python3 -m okfwiki.krepair list --claims CLAIMS --verdicts VERDICTS --out LIST [--ids IDS]
|
||
python3 -m okfwiki.krepair task --list LIST --claims CLAIMS --pairs PAIRS --work DIR --blocks FILE --sources DIR
|
||
--agent FILE --out-dir DIR
|
||
python3 -m okfwiki.krepair fix --answers FILE --bundle WIKI --work DIR --blocks FILE --sources DIR --links FILE
|
||
python3 -m okfwiki.krepair unlink --links FILE --page P --sentence KEY --anchor A --at N [--work DIR --bundle WIKI]
|
||
python3 -m okfwiki.krepair relink --links FILE --page P --sentence KEY --anchor A --at N --to TARGET --bundle WIKI
|
||
[--work DIR]
|
||
python3 -m okfwiki.krepair linkset --list LIST --bundle WIKI --out-dir DIR [--claims CLAIMS]
|
||
python3 -m okfwiki.krepair diff --old WIKI --new WIKI --sources DIR --repairs FILE --out DIR
|
||
|
||
Exit 0 done; 1 refused or not equal, with one line per reason of the form `refused: <code> <claim or record>`; 2 cannot
|
||
run (also for an input the code did not foresee, with its traceback).
|
||
|
||
The contract:
|
||
|
||
- The list in is K17's schema (okf-wiki/repair-list/1); task reads only `claims`, linkset only `links`. Both refuse a
|
||
list with an unknown top-level key, without one of K17's keys, of another schema, or with a `claims_sha256` that is
|
||
not the sha256 of the claim list given (task --claims; linkset with --claims, which K18 leaves out and this command
|
||
takes optionally), and task a claim the claim list lacks or shows with another `excerpt_sha256`: refusal code
|
||
`stale-list`.
|
||
- list --claims CLAIMS --verdicts VERDICTS --out LIST [--ids IDS]: the list from a reading (v1.2, part 1; before it
|
||
only the repair gate rendered one). Every claim the reading covered (all of CLAIMS, or those of --ids, the file
|
||
claims calls --ids reads) that does not have exactly one verdict, `supported`, is listed as claims score counts it:
|
||
`not-supported`, or `no-verdict` when the reader gave it none. Each row has the claim's id, `excerpt_sha256`, page,
|
||
line, provenance, its excerpts as `sources` and origin `reader`; K17's `kind` is left out (task does not read it).
|
||
`pages`, `pairs` and `links` are empty. A verdict for a claim CLAIMS lacks, or an --out that exists, cannot run
|
||
(exit 2), and nothing is written.
|
||
- task: the helper's tasks. The unit is the fragment, keyed by (`concept_id`, the `s...` footnote): the footnote alone
|
||
is not unique (kassemble.label takes neither text nor concept; measured 20.10.2026 in b5-18a/work: 47 keys shared by
|
||
95 records in different concepts, 0 shared pairs of concept and key). The claims of the list that come from one
|
||
fragment stand together, and a task holds at most TASK_FRAGMENTS fragments from one page. Per fragment: the page's
|
||
title, every sentence of the fragment (those on the list marked with the list's reason: the reader did not accept it,
|
||
or the operator's category), the excerpts the sentences show, the fragment's quote and context quotes, and the block
|
||
the fragment stands in with numbered lines. The parts are at most kpack.PART_CHARS characters. A fragment whose
|
||
section with the whole block is over the room of a part is shown with the block's lines from EXCERPT_LINES before its
|
||
quote's line to EXCERPT_LINES after it, within the block, under a line that says so, and with its context quote lines
|
||
that fall outside them on their own (order 35 A2; the operator 21.10.2026, PLAN.md § Rammer point 8); fix still checks
|
||
against the whole block. A fragment whose part is larger even so exits 2 with the fragment named. The instructions and the answer contract stand in the task
|
||
(INSTRUCTIONS, as the reader's in okfwiki/reader.py), so the agent file is thin and a change reaches the helper
|
||
without a restart (CLAUDE.md). task refuses when the agent file or the instructions share a run of eight words with a
|
||
claim, a quote or the block text in the task (`shared-words`). Each task gets a part list (`<task>.parts.json`: the
|
||
parts it may read and the one answer path it writes), which the breach count reads (okfwiki.cost isolation --line
|
||
repair). task writes only in --out-dir and never over a task already written.
|
||
- The answer (okf-wiki/repair-answers/1) is keyed by (`concept_id`, the `s...` footnote): replace with one to three
|
||
new fragments (`text`, `quote`, `anchor` {file, start, end}, `provenance`, and optionally `context` as a list of
|
||
quote strings, as kpack.with_context takes them), or remove. Code sets `concept` from the old record and `status` to
|
||
"existing" (the concept exists and is never decided again), so kpack.validate asks for no `tests`, also when the old
|
||
record made the concept; the record keeps the old record's `outcome`.
|
||
- fix: every fragment of the answer is found against the same built collection and the present accepted.json before
|
||
anything is written: each key must be (`concept_id`, kassemble.label) of exactly one record of accepted.json
|
||
(`unknown-key`, `ambiguous-key`) and stand as a footnote on that page of the collection (`stale-bundle`: a collection
|
||
built before the last repair has old keys and is refused so); two answers for one key are `duplicate-answer`. New
|
||
fragments are checked with kpack.validate against their source's blocks alone (the quote verbatim on a line, the mark
|
||
from PROVENANCE; its reasons are the refusal codes, `anchor-outside-pack` for an anchor in another source) and get
|
||
their context from kpack.with_context; when the answer gives no `context`, the fragment keeps the old record's context
|
||
quotes (records keep them as {line, quote, section}; they are given to with_context as their `quote` values) where
|
||
they still stand on a heading of the path or a line of the section, and those that do not are listed. A new fragment
|
||
takes the old record's place (same `concept_id`, `pack`, `position` and order). A new footnote that another fragment
|
||
of the concept has, or that two new fragments share, is `label-taken`: the key stays unique. All or nothing: one
|
||
refused fragment, and nothing is written.
|
||
- The sentences that move: a sentence key ends in `-<n>`, the occurrence of the same text on the page. When a repair
|
||
changes another sentence's key (two equal sentences become one, or a new one equals one that stands), the link
|
||
records on that sentence move to the new key, and its claim is logged as changed (it is read again).
|
||
- The links on a repaired sentence: a link record (klink's {page, sentence, anchor, at, target}) on a sentence that is
|
||
repaired moves to the new sentence's key when the anchor text stands in the new sentence with the same occurrence
|
||
number (`at` counted again from that occurrence, outside code, as a whole word); the i-th sentence of the old fragment
|
||
is paired with the i-th sentence of its new fragments. Otherwise, and when the sentence is removed, the record is
|
||
taken out and listed; so is a record on or into a page that goes. Code never places a new link (links are made by
|
||
the model, CLAUDE.md).
|
||
- unlink --links FILE --page P --sentence KEY --anchor A --at N, and relink ... --to TARGET --bundle WIKI: take out one
|
||
link record, or give it a new target; the record must stand in the file exactly so (`link-not-found`), and the target
|
||
must be a concept page that exists (`target-missing`), not the record's own page (`self-link`) and not a target the
|
||
sentence already links to (`duplicate-link`). No other record changes. With --work (and --bundle, which unlink then
|
||
needs), the write is logged like a fix's, so diff can measure it.
|
||
- linkset --list LIST --bundle WIKI --out-dir DIR: for the list's link candidates (K17 part 4), a sentence file in the
|
||
form klink task --sentences reads and, per page, a catalogue (--catalogue) with the present targets and the pages that
|
||
share a source with the sentence. klink task takes one catalogue per run, so linkset groups the pages into tasks in
|
||
which the union of the catalogues stays within klink.CATALOGUE_CHARS, and writes one sentence file and one catalogue
|
||
per group (`group-<n>.sentences.json`, `group-<n>.catalogue`); a page whose catalogue alone is over is cut to it, the
|
||
present targets kept first, and the cut is counted. The number of groups is printed (the repair order prices the link
|
||
pass from it). The frozen linker then gets an ordinary link pass on exactly those sentences (klink task, klink
|
||
accept), and the repair order needs no new code. A candidate whose sentence or target is not in the collection is
|
||
`stale-list`.
|
||
- The home and the main pages: when the last fragment on a page is removed, the page goes as in kpack remove
|
||
(drop_dangling, left_without, pass_homes), and all it brings with it is listed. When a main page loses its last
|
||
fragment from the source it is the main page of while the page stands, it is listed as a main page without its
|
||
source; the guard in the repair order asks the model again, as when a source lacks a main page. The pages without a
|
||
home after the repair that had one before are listed.
|
||
- Every write is logged in WORK/K-work/repairs.json (okf-wiki/repairs/1, appended, never rewritten): what was done, the
|
||
fragments before and after (label), the claims that change (also the other sentences of a replaced fragment and the
|
||
sentences that moved), the link records moved or taken out, and the pages that change. What a fix expects to change
|
||
is measured, not guessed: the new state is built in a temporary copy of the collection with kassemble build and
|
||
compared with it by the same measurement as diff, and the log names the collection by its sha256 (tree_sha256).
|
||
accepted.json and the log are written with kpack.write_state (with its backup); the links file is written after them
|
||
through a temporary file beside it, its old copy kept in K-work/backup/. A temporary links file left by a write that
|
||
did not finish stops the next fix, unlink and relink (exit 2, with what to do).
|
||
- diff --old WIKI --new WIKI --sources DIR --repairs FILE --out DIR: measures after kassemble build which concept pages
|
||
have other bytes (not index.md and log.md; a page gone counts) and which claims are new, gone or have another
|
||
`excerpt_sha256` (the frozen claims extract). It compares them with what the repairs logged against --old (by its
|
||
sha256) expect, and exits 1 when the measured and the logged are not equal both ways. Out: `ids.json` with the new and
|
||
changed claims in the form claims calls --ids reads, and `sider.json` with the changed pages and those gone (K15,
|
||
K19), written also when they differ.
|
||
|
||
The refusal codes (REFUSALS below; those the repair gate tries stand also in gates/repair.manifest.json):
|
||
stale-list task, linkset: not K17's list for this claim list, or a claim, sentence or target it names is
|
||
not in the collection as listed
|
||
unknown-key fix, task: (concept_id, footnote) is no record of accepted.json
|
||
ambiguous-key fix, task: (concept_id, footnote) is more than one record of accepted.json
|
||
stale-bundle fix: the key is a record but no footnote on its page in --bundle
|
||
duplicate-answer fix: two answers for one key
|
||
label-taken fix: a new footnote another fragment of the concept has, or two new fragments share
|
||
malformed fix: not in the answer form (and kpack.validate's malformed)
|
||
anchor-outside-pack, anchor-crosses-blocks, sentence-count, source-missing, quote-missing, quote-not-in-span,
|
||
provenance-missing, provenance-unknown, tests-missing
|
||
fix: kpack.validate's reasons, the one hard check at writing, against the source alone
|
||
shared-words task: the agent file or the instructions share eight words with the task's text
|
||
target-missing relink: no concept page --to in --bundle
|
||
self-link relink: --to is the record's own page
|
||
duplicate-link relink: the sentence links to --to already
|
||
link-not-found unlink, relink: no record with this page, sentence, anchor and at
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import argparse
|
||
import copy
|
||
import hashlib
|
||
import json
|
||
import os
|
||
import re
|
||
import shutil
|
||
import stat
|
||
import sys
|
||
import tempfile
|
||
import traceback
|
||
from collections import Counter
|
||
from dataclasses import dataclass, field
|
||
from datetime import datetime, timezone
|
||
from pathlib import Path
|
||
from typing import Any
|
||
|
||
from okfwiki import claims, kassemble, klink, kpack, reader
|
||
from okfwiki.common import RunError
|
||
from okfwiki.kblocks import kilde_of
|
||
|
||
LIST_SCHEMA = "okf-wiki/repair-list/1"
|
||
LIST_KEYS = ("schema", "claims_sha256", "verdicts_sha256", "claims", "pages", "pairs", "links")
|
||
ANSWERS_SCHEMA = "okf-wiki/repair-answers/1"
|
||
REPAIRS_SCHEMA = "okf-wiki/repairs/1"
|
||
PARTS_SCHEMA = "okf-wiki/repair-parts/1"
|
||
IDS_SCHEMA = "okf-wiki/repair-ids/1"
|
||
PAGES_SCHEMA = "okf-wiki/repair-pages/1"
|
||
REPAIRS = "repairs.json" # in WORK/K-work
|
||
TASK_FRAGMENTS = 10 # fragments of one page per task
|
||
NEW_FRAGMENTS = 3 # a fragment is replaced by one to three
|
||
EXCERPT_LINES = 40 # lines on each side of the quote when the block is too large (measured in order 35 A2: 40 fits both)
|
||
FRAGMENT_LABEL = re.compile(rf"s[0-9a-f]{{{kassemble.LABEL_HEX}}}") # kassemble.label; context labels start with c
|
||
EXPECTED = ("pages", "pages_gone", "claims", "claims_gone")
|
||
REASON_TEXT = {"not-supported": "the reader did not accept it", "no-verdict": "the reader gave it no verdict"}
|
||
|
||
REFUSALS = {
|
||
"stale-list": "task and linkset: the list is not K17's for this claim list (claims_sha256, schema, keys), or names "
|
||
"a claim, sentence or target the collection does not have as listed",
|
||
"unknown-key": "fix and task: (concept_id, footnote) is no record of accepted.json",
|
||
"ambiguous-key": "fix and task: (concept_id, footnote) is more than one record of accepted.json",
|
||
"stale-bundle": "fix: the key is a record of accepted.json but not a footnote on its page in --bundle",
|
||
"duplicate-answer": "fix: two answers in one file have the same key",
|
||
"label-taken": "fix: a new fragment's footnote is another fragment's of the concept, or two new ones share it",
|
||
"malformed": "fix: an answer or a new fragment is not in the answer form (kpack.validate's malformed too)",
|
||
"anchor-outside-pack": "kpack.validate against the fragment's own source alone: the anchor is in another source",
|
||
"anchor-crosses-blocks": "kpack.validate: the anchor is not inside one block",
|
||
"sentence-count": "kpack.validate: the text is not one to three sentences",
|
||
"source-missing": "kpack.validate: the source cannot be read",
|
||
"quote-missing": "kpack.validate: no quote",
|
||
"quote-not-in-span": "kpack.validate: the quote is not verbatim on one line of the anchor span",
|
||
"provenance-missing": "kpack.validate: no provenance mark",
|
||
"provenance-unknown": "kpack.validate: a provenance mark not in kpack.PROVENANCE",
|
||
"tests-missing": "kpack.validate: a new concept without tests (never, as code sets status existing)",
|
||
"shared-words": "task: the agent file or the instructions share a run of eight words with a claim, a quote or the "
|
||
"block text in the task",
|
||
"target-missing": "relink: the new target is no concept page of --bundle",
|
||
"self-link": "relink: the new target is the record's own page",
|
||
"duplicate-link": "relink: the sentence already links to the new target",
|
||
"link-not-found": "unlink and relink: no record of the links file has this page, sentence, anchor and at",
|
||
}
|
||
|
||
INSTRUCTIONS = """\
|
||
# How to repair the fragments
|
||
|
||
You repair fragments of a knowledge collection. A fragment is one to three sentences on a concept page, written from
|
||
one block of a source document and resting on a quote from it. A reader, or the operator, did not accept at least one
|
||
sentence of each fragment below; those sentences are marked with the reason. Read every part of this task first: its
|
||
fragments may stand in more than one part.
|
||
|
||
For each fragment, give exactly one answer.
|
||
|
||
- `replace` it with one to three new fragments when the block bears what the fragment is there to say. Each new
|
||
fragment says what its quote and its context say, and no more. A condition the source sets stays with the statement
|
||
it limits. An instruction stays an instruction, and a possibility stays a possibility. The new fragments take the
|
||
place of the whole fragment, so carry over what its other sentences say where the block bears it.
|
||
- `remove` it when the block does not bear it.
|
||
|
||
Each new fragment has:
|
||
|
||
- `text`: one to three sentences.
|
||
- `quote`: one piece copied character for character from one numbered line of the block, inside `anchor`.
|
||
- `anchor`: {"file", "start", "end"}: the file the fragment names, and a range of lines inside its block.
|
||
- `provenance`: "documented" when the text says what its quote says; "inferred" when it draws a conclusion from lines
|
||
of its anchor that no single line states; "merged" when it combines what several lines of its anchor say.
|
||
- `context` (optional): at most two pieces, each copied character for character from a heading of the block's path or
|
||
from another line of its section: the subject a heading names, or a condition, step or setting another line states.
|
||
Leave `context` out to keep the context quotes the fragment has now; give an empty list for none.
|
||
|
||
An example from another subject. A block reads:
|
||
|
||
12 | Knead the dough for ten minutes.
|
||
13 | Let it rest in a warm place until it doubles in size.
|
||
|
||
A fragment says "Knead the dough for an hour." Its quote holds and its text does not, so replace it with
|
||
{"text": "Knead the dough for ten minutes.", "quote": "Knead the dough for ten minutes", "anchor": {"file":
|
||
"bread.md", "start": 12, "end": 13}, "provenance": "documented"}. A fragment that says "The dough must rest
|
||
overnight." has no line that bears it: remove it.
|
||
|
||
Write one JSON object to the answer path your prompt names:
|
||
|
||
{"schema": "okf-wiki/repair-answers/1", "answers": [
|
||
{"concept_id": "...", "footnote": "...", "action": "replace", "fragments": [...]},
|
||
{"concept_id": "...", "footnote": "...", "action": "remove"}]}
|
||
|
||
Copy `concept_id` and `footnote` as the fragment shows them, and answer every fragment of the task once. Then hand back
|
||
only the word done.
|
||
"""
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class Refusal:
|
||
code: str
|
||
what: str
|
||
|
||
|
||
def refuse(refusals: list[Refusal]) -> int:
|
||
for r in refusals:
|
||
print(f"refused: {r.code} {r.what}")
|
||
return 1
|
||
|
||
|
||
# --- common ----------------------------------------------------------------------------------------------------
|
||
|
||
|
||
def file_sha256(path: Path) -> str:
|
||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||
|
||
|
||
def tree_sha256(root: Path) -> str:
|
||
"""The sha256 of a collection: every file's relative path and sha256, in path order (folder metadata left out)."""
|
||
h = hashlib.sha256()
|
||
for p in sorted(root.rglob("*")):
|
||
if p.is_file() and p.name not in kassemble.TOLERATED:
|
||
h.update(f"{p.relative_to(root).as_posix()}\0{file_sha256(p)}\n".encode("utf-8"))
|
||
return h.hexdigest()
|
||
|
||
|
||
def stem(page: str) -> str:
|
||
return klink.concept_id(klink.page_rel(page))
|
||
|
||
|
||
def fragment_of(labels: Any) -> str | None:
|
||
"""The fragment's footnote among a sentence's labels (its context, version and week labels left out)."""
|
||
return next((str(x) for x in labels if FRAGMENT_LABEL.fullmatch(str(x))), None)
|
||
|
||
|
||
def keyed(acc: dict[str, Any]) -> dict[tuple[str, str], list[int]]:
|
||
out: dict[tuple[str, str], list[int]] = {}
|
||
for i, r in enumerate(acc["fragments"]):
|
||
out.setdefault((str(r["concept_id"]), kassemble.label(r)), []).append(i)
|
||
return out
|
||
|
||
|
||
def footnotes_on(bundle: Path, cid: str) -> set[str]:
|
||
page = bundle / klink.page_rel(cid)
|
||
if not page.is_file():
|
||
return set()
|
||
return {x for s in klink.page_sentences(page.read_text(encoding="utf-8")) for x in s.labels}
|
||
|
||
|
||
def block_of(blocks_doc: dict[str, Any], r: dict[str, Any]) -> dict[str, Any]:
|
||
block = next((b for b in blocks_doc["blocks"] if b["file"] == r["file"] and b["start"] <= r["start"]
|
||
and r["end"] <= b["end"]), None)
|
||
if block is None:
|
||
raise RunError(f"--blocks has no block for {r['concept_id']} {kassemble.label(r)} ({r['file']} "
|
||
f"{r['start']}-{r['end']})")
|
||
return dict(block)
|
||
|
||
|
||
def source_lines(kilde: Path, file: str, cache: dict[str, list[str]]) -> list[str]:
|
||
if file not in cache:
|
||
cache[file] = (kilde / file).read_text(encoding="utf-8").split("\n")
|
||
return cache[file]
|
||
|
||
|
||
def links_tmp(links: Path) -> Path:
|
||
return links.with_name(f".{links.name}.krepair-tmp")
|
||
|
||
|
||
def check_links_tmp(links: Path) -> None:
|
||
tmp = links_tmp(links)
|
||
if tmp.exists():
|
||
raise RunError(f"{tmp} exists: an earlier repair did not finish its links write. When the last entry of "
|
||
f"K-work/{REPAIRS} is that repair, move {tmp} to {links}; else delete it")
|
||
|
||
|
||
def read_repairs(work: Path) -> dict[str, Any]:
|
||
path = kpack.k_work(work) / REPAIRS
|
||
if not path.is_file():
|
||
return {"schema": REPAIRS_SCHEMA, "entries": []}
|
||
doc: dict[str, Any] = kpack.read_json(path)
|
||
if doc.get("schema") != REPAIRS_SCHEMA or not isinstance(doc.get("entries"), list):
|
||
raise RunError(f"{path} is not {REPAIRS_SCHEMA}")
|
||
return doc
|
||
|
||
|
||
def now() -> str:
|
||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||
|
||
|
||
def commit(work: Path, label: str, files: dict[str, Any], links: Path, links_doc: dict[str, Any] | None) -> None:
|
||
"""accepted.json and the log under kpack's marker and backup, then the links file through a temporary file."""
|
||
tmp = links_tmp(links)
|
||
if links_doc is not None:
|
||
tmp.write_text(klink.dump(links_doc), encoding="utf-8")
|
||
kpack.write_state(work, label, files)
|
||
if links_doc is not None:
|
||
shutil.copy2(links, kpack.k_work(work) / "backup" / links.name)
|
||
os.replace(tmp, links)
|
||
|
||
|
||
def write_links(links: Path, links_doc: dict[str, Any]) -> None:
|
||
tmp = links_tmp(links)
|
||
tmp.write_text(klink.dump(links_doc), encoding="utf-8")
|
||
os.replace(tmp, links)
|
||
|
||
|
||
def with_records(doc: dict[str, Any], records: list[dict[str, Any]]) -> dict[str, Any]:
|
||
return {**doc, "links": records, **({"accepted": len(records)} if "accepted" in doc else {})}
|
||
|
||
|
||
# --- the measurement (diff, and what a fix expects) ------------------------------------------------------------
|
||
|
||
|
||
@dataclass
|
||
class Measured:
|
||
pages: list[str] # concept pages with other bytes, a page gone among them
|
||
pages_gone: list[str]
|
||
claims: list[str] # ids new or with another excerpt_sha256
|
||
claims_gone: list[str] # ids gone or with another excerpt_sha256
|
||
|
||
def as_dict(self) -> dict[str, list[str]]:
|
||
return {"pages": self.pages, "pages_gone": self.pages_gone, "claims": self.claims,
|
||
"claims_gone": self.claims_gone}
|
||
|
||
|
||
def concept_pages(wiki: Path) -> dict[str, bytes]:
|
||
return {p.relative_to(wiki).as_posix(): p.read_bytes() for p in sorted(wiki.rglob("*.md"))
|
||
if klink.is_concept(p.relative_to(wiki).as_posix())}
|
||
|
||
|
||
def measure(old: Path, new: Path, sources: Path) -> Measured:
|
||
a, z = concept_pages(old), concept_pages(new)
|
||
before = Counter((c.id, c.excerpt_sha256) for c in claims.extract(old, sources).claims)
|
||
after = Counter((c.id, c.excerpt_sha256) for c in claims.extract(new, sources).claims)
|
||
return Measured(sorted(n for n in set(a) | set(z) if a.get(n) != z.get(n)), sorted(set(a) - set(z)),
|
||
sorted({i for i, _ in after - before}), sorted({i for i, _ in before - after}))
|
||
|
||
|
||
# --- lists -----------------------------------------------------------------------------------------------------
|
||
|
||
|
||
def read_list(path: Path, claims_path: Path | None) -> tuple[dict[str, Any], list[Refusal]]:
|
||
doc = kpack.read_json(path)
|
||
if not isinstance(doc, dict):
|
||
raise RunError(f"{path} is not a repair list")
|
||
name = path.name
|
||
out = []
|
||
if doc.get("schema") != LIST_SCHEMA:
|
||
out.append(Refusal("stale-list", f"{name} (schema {doc.get('schema')!r}, not {LIST_SCHEMA})"))
|
||
out += [Refusal("stale-list", f"{name} (unknown top-level key {k!r})") for k in sorted(set(doc) - set(LIST_KEYS))]
|
||
out += [Refusal("stale-list", f"{name} (no {k})") for k in LIST_KEYS if k not in doc]
|
||
if claims_path is not None and doc.get("claims_sha256") != file_sha256(claims_path):
|
||
out.append(Refusal("stale-list", f"{name} (claims_sha256 is not the sha256 of {claims_path.name})"))
|
||
if out:
|
||
return doc, out
|
||
rows, links = doc["claims"], doc["links"]
|
||
if not (isinstance(rows, list) and all(
|
||
isinstance(c, dict) and isinstance(c.get("id"), str) and isinstance(c.get("excerpt_sha256"), str)
|
||
and isinstance(c.get("reasons", []), list) for c in rows)):
|
||
raise RunError(f"{path}: claims is not a list of records with id, excerpt_sha256 and reasons")
|
||
if not (isinstance(links, list) and all(
|
||
isinstance(x, dict) and all(type(x.get(k)) is t for k, t in kassemble.LINK_KEYS.items()) for x in links)):
|
||
raise RunError(f"{path}: links is not a list of records with {', '.join(kassemble.LINK_KEYS)}")
|
||
return doc, out
|
||
|
||
|
||
# --- list ------------------------------------------------------------------------------------------------------
|
||
|
||
|
||
def cmd_list(args: argparse.Namespace) -> int:
|
||
claims_path, verdicts_path, out = Path(args.claims), Path(args.verdicts), Path(args.out)
|
||
if out.exists():
|
||
raise RunError(f"{out} exists: a list already written is never overwritten")
|
||
found = claims.load_claims(claims_path)
|
||
verdicts = kpack.read_json(verdicts_path)
|
||
rows_in = verdicts.get("verdicts") if isinstance(verdicts, dict) else None
|
||
if not (isinstance(rows_in, list) and all(isinstance(v, dict) and isinstance(v.get("id"), str) for v in rows_in)):
|
||
raise RunError(f"{verdicts_path} is not the reader's verdicts (reader collect --out)")
|
||
given: dict[str, list[str]] = {}
|
||
for v in rows_in:
|
||
given.setdefault(v["id"], []).append(str(v.get("verdict")))
|
||
known = {c.id for c in found.claims}
|
||
strangers = sorted(set(given) - known)
|
||
if strangers:
|
||
raise RunError(f"{verdicts_path.name} has verdicts for {len(strangers)} claims that {claims_path.name} lacks: "
|
||
+ " ".join(strangers[:5]))
|
||
covered = known
|
||
if args.ids:
|
||
ids = kpack.read_json(Path(args.ids))
|
||
wanted = ids.get("ids") if isinstance(ids, dict) else None
|
||
if not (isinstance(wanted, list) and all(isinstance(i, str) for i in wanted)):
|
||
raise RunError(f"{args.ids} has no list of ids")
|
||
covered = set(wanted)
|
||
if covered - known:
|
||
raise RunError(f"{args.ids} names {len(covered - known)} claims that {claims_path.name} lacks: "
|
||
+ " ".join(sorted(covered - known)[:5]))
|
||
rows = [{"id": c.id, "excerpt_sha256": c.excerpt_sha256, "page": c.page, "line": c.line,
|
||
"provenance": c.provenance, "sources": [{"file": f"kilde/{e.file}", "line": e.line} for e in c.excerpts],
|
||
"origins": ["reader"], "reasons": ["not-supported" if c.id in given else "no-verdict"]}
|
||
for c in found.claims if c.id in covered and given.get(c.id) != [reader.SUPPORTED]]
|
||
doc = {"schema": LIST_SCHEMA, "claims_sha256": file_sha256(claims_path),
|
||
"verdicts_sha256": file_sha256(verdicts_path), "claims": rows, "pages": [], "pairs": [], "links": []}
|
||
out.write_text(json.dumps(doc, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||
silent = sum(1 for r in rows if r["reasons"] == ["no-verdict"])
|
||
print(f"list: {len(rows)} of {len(covered)} claims read are on the list ({len(rows) - silent} not supported, "
|
||
f"{silent} without a verdict) -> {out}")
|
||
return 0
|
||
|
||
|
||
# --- task ------------------------------------------------------------------------------------------------------
|
||
|
||
|
||
@dataclass
|
||
class Item:
|
||
"""One fragment of a task: its record, its page's title and the sentences it gives the page."""
|
||
|
||
concept_id: str
|
||
label: str
|
||
record: dict[str, Any]
|
||
title: str
|
||
sentences: list[dict[str, Any]] # the claims of the fragment on its page, in page order (from --claims)
|
||
reasons: dict[str, list[str]] = field(default_factory=dict) # listed claim id -> its reasons
|
||
|
||
|
||
def reason_text(reasons: list[str]) -> str:
|
||
return "; ".join(dict.fromkeys(REASON_TEXT.get(r, f"the operator: {r}") for r in reasons))
|
||
|
||
|
||
def excerpts_of(c: dict[str, Any], pairs: dict[str, reader.Pair]) -> list[reader.Excerpt]:
|
||
p = pairs.get(str(c["id"]))
|
||
if p is not None:
|
||
return list(p.excerpts)
|
||
return [reader.Excerpt(str(e["file"]), int(e["line"]), str(e["text"])) for e in c.get("excerpts", [])]
|
||
|
||
|
||
def block_text(item: Item, blocks_doc: dict[str, Any], kilde: Path, cache: dict[str, list[str]]) -> tuple[
|
||
dict[str, Any], list[str]]:
|
||
block = block_of(blocks_doc, item.record)
|
||
lines = source_lines(kilde, str(item.record["file"]), cache)
|
||
return block, [lines[n - 1] if n <= len(lines) else "" for n in range(block["start"], block["end"] + 1)]
|
||
|
||
|
||
def render_item(n: int, item: Item, pairs: dict[str, reader.Pair], blocks_doc: dict[str, Any], kilde: Path,
|
||
cache: dict[str, list[str]], excerpt: bool = False) -> str:
|
||
r = item.record
|
||
out = [f"## Fragment {n}", "", f"- concept_id: `{item.concept_id}`", f"- footnote: `{item.label}`",
|
||
f"- page: {item.title}", "", "Its sentences, as the page shows them:", ""]
|
||
for k, c in enumerate(item.sentences, 1):
|
||
mark = f" (on the list: {reason_text(item.reasons[c['id']])})" if c["id"] in item.reasons else ""
|
||
out.append(f"{k}. {klink.key_text(str(c['claim']))}{mark}")
|
||
out += [f' <excerpt file="{e.file}" line="{e.line}">{e.text}</excerpt>' for e in excerpts_of(c, pairs)]
|
||
context = r.get("context", [])
|
||
out += ["", f"Quote: `{r['quote']}` ({r['file']}, line {r['quote_line']}), provenance {r['provenance']}.",
|
||
"Context quotes: " + ("; ".join(f"`{c['quote']}` (line {c['line']})" for c in context) if context
|
||
else "none") + "."]
|
||
block, lines = block_text(item, blocks_doc, kilde, cache)
|
||
out += [f"Anchor: {r['file']}, lines {r['start']}-{r['end']}.",
|
||
f"Block: {r['file']} › {block['section']}, lines {block['start']}-{block['end']}:"]
|
||
first, last = block["start"], block["end"]
|
||
if excerpt: # the quote stands on one line (kpack.validate), so its first and last line are quote_line
|
||
q = int(r["quote_line"])
|
||
first, last = max(first, q - EXCERPT_LINES), min(last, q + EXCERPT_LINES)
|
||
out.append(f"The block is shown only in part: lines {first}-{last} of lines {block['start']}-{block['end']} "
|
||
f"({len(lines)} lines). Judge the fragment from the lines shown.")
|
||
out.append("")
|
||
out += [f"{k:>5} | {lines[k - block['start']]}" for k in range(first, last + 1)]
|
||
outside = sorted({int(c["line"]) for c in context} - set(range(first, last + 1)))
|
||
if excerpt and outside:
|
||
src = source_lines(kilde, str(r["file"]), cache)
|
||
out += ["", "Context quote lines outside the lines shown:", ""]
|
||
out += [f"{k:>5} | {src[k - 1] if k <= len(src) else ''}" for k in outside]
|
||
return "\n".join(out) + "\n\n"
|
||
|
||
|
||
def part_head(task: str, i: int, n: int) -> str:
|
||
return f"<!-- part {i} of {n} -->\n# Repair task {task}, part {i} of {n}\n\n"
|
||
|
||
|
||
def part_room(task: str) -> int:
|
||
return kpack.PART_CHARS - len(part_head(task, 999, 999))
|
||
|
||
|
||
def pack_parts(task: str, sections: list[tuple[str, str]]) -> list[str]:
|
||
"""The task's parts: the instructions first, then whole fragment sections, each part within kpack.PART_CHARS."""
|
||
limit = kpack.PART_CHARS
|
||
room = part_room(task)
|
||
for what, text in sections[1:] + sections[:1]: # a fragment that cannot fit is named before the instructions
|
||
if len(text) > room:
|
||
raise RunError(f"{what} needs {len(text)} characters, over the room of {room} in a part of {limit}: it "
|
||
f"cannot stand in one part")
|
||
bodies: list[str] = []
|
||
for _, text in sections:
|
||
if bodies and len(bodies[-1]) + len(text) <= room:
|
||
bodies[-1] += text
|
||
else:
|
||
bodies.append(text)
|
||
return [part_head(task, i, len(bodies)) + body for i, body in enumerate(bodies, 1)]
|
||
|
||
|
||
def task_items(lst: dict[str, Any], claims_doc: dict[str, Any], acc: dict[str, Any], index: dict[str, Any],
|
||
claims_name: str) -> tuple[list[Item], list[Refusal]]:
|
||
by_id = {str(c["id"]): c for c in claims_doc["claims"]}
|
||
keys = keyed(acc)
|
||
titles = {str(e["id"]): kassemble.one_line(e["title"]) for e in index["entries"]}
|
||
items: dict[tuple[str, str], Item] = {}
|
||
refusals: list[Refusal] = []
|
||
for row in lst["claims"]:
|
||
cid_claim = str(row["id"])
|
||
c = by_id.get(cid_claim)
|
||
if c is None or c.get("excerpt_sha256") != row["excerpt_sha256"]:
|
||
why = "not a claim of" if c is None else "shown with another excerpt_sha256 in"
|
||
refusals.append(Refusal("stale-list", f"{cid_claim} ({why} {claims_name})"))
|
||
continue
|
||
label = fragment_of(c.get("footnotes", []))
|
||
concept = stem(str(c["page"]))
|
||
hits = keys.get((concept, label or ""), [])
|
||
if label is None or len(hits) != 1:
|
||
code = "ambiguous-key" if len(hits) > 1 else "unknown-key"
|
||
refusals.append(Refusal(code, f"{cid_claim} ({concept} {label or 'no footnote'})"))
|
||
continue
|
||
key = (concept, label)
|
||
if key not in items:
|
||
sentences = [x for x in claims_doc["claims"] if str(x["page"]) == str(c["page"])
|
||
and fragment_of(x.get("footnotes", [])) == label]
|
||
items[key] = Item(concept, label, acc["fragments"][hits[0]], titles.get(concept, concept), sentences)
|
||
items[key].reasons[cid_claim] = [str(x) for x in row.get("reasons", [])] or ["not-supported"]
|
||
return list(items.values()), refusals
|
||
|
||
|
||
def cmd_task(args: argparse.Namespace) -> int:
|
||
claims_path, work, out = Path(args.claims), Path(args.work), Path(args.out_dir).resolve()
|
||
kpack.check_pending(work)
|
||
lst, refusals = read_list(Path(args.list), claims_path)
|
||
if refusals:
|
||
return refuse(refusals)
|
||
claims_doc: dict[str, Any] = kpack.read_json(claims_path)
|
||
pairs = {p.id: p for p in reader.load_pairs(Path(args.pairs))}
|
||
acc, index = kpack.read_accepted(work), kpack.read_index(work)
|
||
blocks_doc: dict[str, Any] = kpack.read_json(Path(args.blocks))
|
||
kilde = kilde_of(Path(args.sources))
|
||
items, refusals = task_items(lst, claims_doc, acc, index, claims_path.name)
|
||
if refusals:
|
||
return refuse(refusals)
|
||
if not items:
|
||
raise RunError(f"{args.list} lists no claim")
|
||
cache: dict[str, list[str]] = {}
|
||
material: list[reader.Pair] = []
|
||
for it in items:
|
||
what = f"{it.concept_id} {it.label}"
|
||
material += [reader.Pair(f"{c['id']}", klink.key_text(str(c["claim"])), tuple(excerpts_of(c, pairs)))
|
||
for c in it.sentences]
|
||
quotes = [str(it.record["quote"]), *(str(c["quote"]) for c in it.record.get("context", []))]
|
||
material.append(reader.Pair(f"{what} (quotes)", " ".join(quotes), ()))
|
||
material.append(reader.Pair(f"{what} (block)", "\n".join(block_text(it, blocks_doc, kilde, cache)[1]), ()))
|
||
agent = Path(args.agent)
|
||
for name, text in ((agent.name, agent.read_text(encoding="utf-8")), ("the instructions", INSTRUCTIONS)):
|
||
refusals += [Refusal("shared-words", f"{name} and {pid}") for pid in reader.shared_runs(text, material)]
|
||
if refusals:
|
||
return refuse(refusals)
|
||
by_page: dict[str, list[Item]] = {}
|
||
for it in items:
|
||
by_page.setdefault(it.concept_id, []).append(it)
|
||
files: dict[Path, str] = {}
|
||
tasks = 0
|
||
for concept in by_page:
|
||
chunk = by_page[concept]
|
||
for i in range(0, len(chunk), TASK_FRAGMENTS):
|
||
tasks += 1
|
||
name = f"r{tasks:03d}"
|
||
group = chunk[i:i + TASK_FRAGMENTS]
|
||
sections = [("the instructions", INSTRUCTIONS + "\n")]
|
||
for n, it in enumerate(group, 1):
|
||
text = render_item(n, it, pairs, blocks_doc, kilde, cache)
|
||
if len(text) > part_room(name):
|
||
text = render_item(n, it, pairs, blocks_doc, kilde, cache, excerpt=True)
|
||
sections.append((f"fragment {it.concept_id} {it.label}", text))
|
||
parts = pack_parts(name, sections)
|
||
paths = [out / f"{name}-{k}.md" for k in range(1, len(parts) + 1)]
|
||
files.update(zip(paths, parts))
|
||
files[out / f"{name}.parts.json"] = kpack.dump({
|
||
"schema": PARTS_SCHEMA, "task": name, "parts": [str(p) for p in paths],
|
||
"answer": str(out / f"{name}.answers.json"), "chars": [len(p) for p in parts],
|
||
"fragments": [{"concept_id": it.concept_id, "footnote": it.label, "claims": sorted(it.reasons)}
|
||
for it in group]})
|
||
taken = sorted(str(p) for p in files if p.exists()) + sorted(str(p) for p in out.glob("*.parts.json"))
|
||
if taken:
|
||
raise RunError(f"tasks exist in {out} ({taken[0]}); a task already written is never changed")
|
||
out.mkdir(parents=True, exist_ok=True)
|
||
for path, text in files.items():
|
||
path.write_text(text, encoding="utf-8")
|
||
listed = sum(len(it.reasons) for it in items)
|
||
print(f"task: {listed} claims, {len(items)} fragments, {len(by_page)} pages, {tasks} tasks, "
|
||
f"{sum(1 for p in files if p.suffix == '.md')} parts -> {out}")
|
||
return 0
|
||
|
||
|
||
# --- fix -------------------------------------------------------------------------------------------------------
|
||
|
||
|
||
@dataclass
|
||
class Answer:
|
||
concept_id: str
|
||
footnote: str
|
||
action: str
|
||
fragments: list[dict[str, Any]]
|
||
|
||
@property
|
||
def what(self) -> str:
|
||
return f"{self.concept_id} {self.footnote}"
|
||
|
||
|
||
def read_answers(path: Path) -> tuple[list[Answer], list[Refusal]]:
|
||
doc = kpack.read_json(path)
|
||
if not (isinstance(doc, dict) and doc.get("schema") == ANSWERS_SCHEMA and isinstance(doc.get("answers"), list)):
|
||
raise RunError(f"{path} is not {ANSWERS_SCHEMA} with a list of answers")
|
||
if not doc["answers"]:
|
||
raise RunError(f"{path} holds no answer")
|
||
out: list[Answer] = []
|
||
refusals: list[Refusal] = []
|
||
for n, a in enumerate(doc["answers"], 1):
|
||
cid, fn, act = (a.get(k) if isinstance(a, dict) else None for k in ("concept_id", "footnote", "action"))
|
||
what = f"{cid} {fn}" if isinstance(cid, str) and isinstance(fn, str) else f"answer {n}"
|
||
if not (isinstance(cid, str) and isinstance(fn, str) and act in ("replace", "remove")):
|
||
refusals.append(Refusal("malformed", f"{what} (needs concept_id, footnote and action replace or remove)"))
|
||
continue
|
||
frags = a.get("fragments")
|
||
if act == "replace" and not (isinstance(frags, list) and 1 <= len(frags) <= NEW_FRAGMENTS
|
||
and all(isinstance(f, dict) for f in frags)):
|
||
refusals.append(Refusal("malformed", f"{what} (replace takes one to {NEW_FRAGMENTS} fragments)"))
|
||
continue
|
||
if act == "remove" and frags is not None:
|
||
refusals.append(Refusal("malformed", f"{what} (remove takes no fragments)"))
|
||
continue
|
||
out.append(Answer(cid, fn, str(act), list(frags or [])))
|
||
return out, refusals
|
||
|
||
|
||
@dataclass
|
||
class Checked:
|
||
"""The answers that passed, by the index of the record they repair, with what fix lists."""
|
||
|
||
new: dict[int, list[dict[str, Any]]] = field(default_factory=dict) # old record index -> new records ([] removes)
|
||
keys: dict[int, tuple[str, str]] = field(default_factory=dict)
|
||
context_not_kept: list[dict[str, Any]] = field(default_factory=list)
|
||
|
||
|
||
def check_answers(answers: list[Answer], acc: dict[str, Any], index: dict[str, Any], bundle: Path,
|
||
blocks_doc: dict[str, Any], kilde: Path) -> tuple[Checked, list[Refusal]]:
|
||
keys = keyed(acc)
|
||
counts = Counter((a.concept_id, a.footnote) for a in answers)
|
||
refusals = [Refusal("duplicate-answer", f"{c} {f} ({n} answers)") for (c, f), n in counts.items() if n > 1]
|
||
done = Checked()
|
||
shown: dict[str, set[str]] = {}
|
||
fresh: dict[str, set[str]] = {}
|
||
for a in answers:
|
||
hits = keys.get((a.concept_id, a.footnote), [])
|
||
if len(hits) != 1:
|
||
refusals.append(Refusal("ambiguous-key" if hits else "unknown-key", a.what))
|
||
continue
|
||
if a.concept_id not in shown:
|
||
shown[a.concept_id] = footnotes_on(bundle, a.concept_id)
|
||
if a.footnote not in shown[a.concept_id]:
|
||
refusals.append(Refusal("stale-bundle", f"{a.what} (no footnote on {klink.page_rel(a.concept_id)} in "
|
||
f"{bundle})"))
|
||
continue
|
||
i = hits[0]
|
||
old = acc["fragments"][i]
|
||
records: list[dict[str, Any]] = []
|
||
ok = True
|
||
for k, given in enumerate(a.fragments, 1):
|
||
f: dict[str, Any] = {"concept": old["concept_id"], "status": "existing",
|
||
**{x: given[x] for x in ("text", "quote", "anchor", "provenance") if x in given}}
|
||
inherited = "context" not in given
|
||
f["context"] = [c["quote"] for c in old.get("context", [])] if inherited else given["context"]
|
||
valid, rejected = kpack.validate([f], [str(old["file"])], blocks_doc, kilde)
|
||
if rejected:
|
||
refusals += [Refusal(reason, f"{a.what} (new fragment {k})") for reason in sorted(rejected)]
|
||
ok = False
|
||
continue
|
||
placed, _ = kpack.with_context(valid, blocks_doc, kilde)
|
||
rec = {**kpack.new_record(placed[0], "existing", str(old["concept_id"]), index, int(old["pack"]),
|
||
int(old["position"])), "outcome": old.get("outcome", "existing")}
|
||
label = kassemble.label(rec)
|
||
other = [j for j in keys.get((a.concept_id, label), []) if j != i]
|
||
if other or label in fresh.setdefault(a.concept_id, set()):
|
||
whose = f"of {kassemble.label(acc['fragments'][other[0]])}" if other else "of another new fragment"
|
||
refusals.append(Refusal("label-taken", f"{a.what} (new fragment {k} has the footnote {label}, the "
|
||
f"footnote {whose} in {a.concept_id})"))
|
||
ok = False
|
||
continue
|
||
fresh[a.concept_id].add(label)
|
||
if inherited:
|
||
kept = {str(c["quote"]) for c in rec.get("context", [])}
|
||
done.context_not_kept += [{"concept_id": a.concept_id, "footnote": a.footnote, "quote": str(q)}
|
||
for q in f["context"] if str(q) not in kept]
|
||
records.append(rec)
|
||
if ok:
|
||
done.new[i], done.keys[i] = records, (a.concept_id, a.footnote)
|
||
return done, refusals
|
||
|
||
|
||
Slot = tuple[str, int]
|
||
|
||
|
||
def slotted(page: kassemble.Page) -> list[tuple[klink.Sentence, Slot | None]]:
|
||
"""The sentences of an assembled page as klink keys them, each with its slot: (its fragment's footnote, its place
|
||
among that fragment's sentences on the page)."""
|
||
seen: Counter[str] = Counter()
|
||
out: list[tuple[klink.Sentence, Slot | None]] = []
|
||
for s in klink.body_sentences(klink.reduce_links(page.body)[0]):
|
||
label = fragment_of(s.labels)
|
||
if label is None:
|
||
out.append((s, None))
|
||
continue
|
||
out.append((s, (label, seen[label])))
|
||
seen[label] += 1
|
||
return out
|
||
|
||
|
||
def sentences_of(by_slot: dict[Slot, klink.Sentence], label: str) -> list[klink.Sentence]:
|
||
out: list[klink.Sentence] = []
|
||
while (label, len(out)) in by_slot:
|
||
out.append(by_slot[(label, len(out))])
|
||
return out
|
||
|
||
|
||
def occurrences(text: str, anchor: str) -> list[int]:
|
||
out, at = [], text.find(anchor)
|
||
while at >= 0:
|
||
out.append(at)
|
||
at = text.find(anchor, at + 1)
|
||
return out
|
||
|
||
|
||
def same_occurrence(old: klink.Sentence, anchor: str, at: int, new: klink.Sentence) -> int | None:
|
||
"""The offset of the anchor in the new sentence at the occurrence number it has in the old one, when it stands
|
||
there outside code and as a whole word (as klink.find_anchor accepts it); else None."""
|
||
before = occurrences(old.text, anchor)
|
||
if at not in before:
|
||
return None
|
||
after = occurrences(new.text, anchor)
|
||
k = before.index(at)
|
||
if k >= len(after):
|
||
return None
|
||
p, e = after[k], after[k] + len(anchor)
|
||
whole = (p == 0 or not new.text[p - 1].isalnum()) and (e == len(new.text) or not new.text[e].isalnum())
|
||
return p if whole and new.masked[p:e] == anchor else None
|
||
|
||
|
||
@dataclass
|
||
class Moves:
|
||
records: list[dict[str, Any]]
|
||
moved: list[dict[str, Any]] = field(default_factory=list)
|
||
removed: list[dict[str, Any]] = field(default_factory=list)
|
||
sentences_moved: list[dict[str, str]] = field(default_factory=list)
|
||
touched_before: list[str] = field(default_factory=list)
|
||
touched_after: list[str] = field(default_factory=list)
|
||
|
||
|
||
def move_links(records: list[dict[str, Any]], old_pages: dict[str, kassemble.Page],
|
||
new_pages: dict[str, kassemble.Page], changed: dict[tuple[str, str], list[str]],
|
||
gone: set[str]) -> Moves:
|
||
"""The link records after the repair: `changed` maps each repaired (concept, footnote) to its new footnotes ([]
|
||
for a removal); `gone` holds the pages that go."""
|
||
m = Moves([])
|
||
old_s = {rel: slotted(p) for rel, p in old_pages.items()}
|
||
old_keyed = {rel: {s.key: (s, slot) for s, slot in sentences if slot is not None}
|
||
for rel, sentences in old_s.items()}
|
||
new_s = {rel: {slot: s for s, slot in slotted(p) if slot is not None} for rel, p in new_pages.items()}
|
||
for rel, sentences in old_s.items():
|
||
cid = stem(rel)
|
||
for s, slot in sentences:
|
||
if slot is None:
|
||
continue
|
||
if (cid, slot[0]) in changed:
|
||
m.touched_before.append(f"{rel}#{s.key}")
|
||
continue
|
||
now_s = new_s.get(rel, {}).get(slot)
|
||
if now_s is not None and now_s.key != s.key:
|
||
m.sentences_moved.append({"page": rel, "before": s.key, "after": now_s.key})
|
||
m.touched_before.append(f"{rel}#{s.key}")
|
||
m.touched_after.append(f"{rel}#{now_s.key}")
|
||
for rel, by_slot in new_s.items():
|
||
cid = stem(rel)
|
||
new_labels = {x for (c, _), labels in changed.items() if c == cid for x in labels}
|
||
m.touched_after += [f"{rel}#{s.key}" for slot, s in by_slot.items() if slot[0] in new_labels]
|
||
for r in records:
|
||
page, target = klink.page_rel(str(r["page"])), klink.page_rel(str(r["target"]))
|
||
if page in gone or target in gone:
|
||
m.removed.append({"record": r, "why": "its page goes" if page in gone else "its target goes"})
|
||
continue
|
||
hit = old_keyed.get(page, {}).get(str(r["sentence"]))
|
||
if hit is None:
|
||
m.records.append(r) # on a page the repair does not touch, or stale already: left as it was
|
||
continue
|
||
s, (label, i) = hit
|
||
cid = stem(page)
|
||
new = new_s.get(page, {})
|
||
if (cid, label) in changed:
|
||
pool = [s2 for x in changed[(cid, label)] for s2 in sentences_of(new, x)]
|
||
if not pool:
|
||
m.removed.append({"record": r, "why": "its sentence is removed"})
|
||
continue
|
||
if i >= len(pool):
|
||
m.removed.append({"record": r, "why": "its fragment has no sentence in its place now"})
|
||
continue
|
||
to = pool[i]
|
||
at = same_occurrence(s, str(r["anchor"]), int(r["at"]), to)
|
||
elif (label, i) not in new:
|
||
m.removed.append({"record": r, "why": "its sentence is not on the page now"})
|
||
continue
|
||
else:
|
||
to = new[(label, i)]
|
||
at = int(r["at"]) if to.text[int(r["at"]):int(r["at"]) + len(str(r["anchor"]))] == r["anchor"] else None
|
||
if at is None:
|
||
m.removed.append({"record": r, "why": "its anchor is not in the new sentence at its occurrence"})
|
||
continue
|
||
moved = {**r, "sentence": to.key, "at": at}
|
||
if moved != r:
|
||
m.moved.append({"before": r, "after": moved})
|
||
m.records.append(moved)
|
||
return m
|
||
|
||
|
||
def roles(acc: dict[str, Any], facts: dict[str, kpack.Fact]) -> tuple[set[str], set[str], dict[str, set[str]]]:
|
||
"""(the main pages, the pages with a home, the sources each page cites) as kassemble renders them: a main page
|
||
cites the main-topic source it is named for, and a home is a link from a main page."""
|
||
cited: dict[str, set[str]] = {}
|
||
for r in acc["fragments"]:
|
||
cited.setdefault(str(r["concept_id"]), set()).add(str(r["file"]))
|
||
mains = {str(m["concept_id"]) for m in acc["main_pages"]
|
||
if str(m["source"]) in cited.get(str(m["concept_id"]), set())
|
||
and str(m["source"]) in facts and facts[str(m["source"])].main_topic}
|
||
homed = {str(h["concept_id"]) for h in acc["homes"] if str(h["home"]) in mains and h["home"] != h["concept_id"]
|
||
and str(h["concept_id"]) in cited}
|
||
return mains, homed, cited
|
||
|
||
|
||
def without_home(acc: dict[str, Any], facts: dict[str, kpack.Fact]) -> set[str]:
|
||
mains, homed, cited = roles(acc, facts)
|
||
return set(cited) - mains - homed
|
||
|
||
|
||
def trial(work: Path, bundle: Path, sources: Path, acc: dict[str, Any], links_doc: dict[str, Any]) -> Measured:
|
||
"""What the repair will change: the new state built into a copy of the collection, measured against it."""
|
||
with tempfile.TemporaryDirectory(prefix="krepair-trial-") as tmp_name:
|
||
tmp = Path(tmp_name)
|
||
kw = tmp / "work" / "K-work"
|
||
kw.mkdir(parents=True)
|
||
shutil.copyfile(kpack.k_work(work) / "index.json", kw / "index.json")
|
||
(kw / "accepted.json").write_text(kpack.dump(acc), encoding="utf-8")
|
||
(tmp / "links.json").write_text(klink.dump(links_doc), encoding="utf-8")
|
||
wiki = tmp / "wiki"
|
||
shutil.copytree(bundle, wiki, copy_function=shutil.copyfile) # the copy is ours, whatever the bundle's modes
|
||
for p in [wiki, *wiki.rglob("*")]:
|
||
p.chmod(p.stat().st_mode | stat.S_IWUSR)
|
||
report = kassemble.build(tmp / "work", sources, wiki, tmp / "links.json", replace=True)
|
||
if report.stale:
|
||
raise RunError(f"after the repair {report.stale} link records would not stand: a fault in krepair")
|
||
return measure(bundle, wiki, sources)
|
||
|
||
|
||
def cmd_fix(args: argparse.Namespace) -> int:
|
||
work, bundle, sources, links = Path(args.work), Path(args.bundle), Path(args.sources), Path(args.links)
|
||
kpack.check_pending(work)
|
||
check_links_tmp(links)
|
||
answers_path = Path(args.answers)
|
||
answers, refusals = read_answers(answers_path)
|
||
acc, index = kpack.read_accepted(work), kpack.read_index(work)
|
||
blocks_doc: dict[str, Any] = kpack.read_json(Path(args.blocks))
|
||
kilde = kilde_of(sources)
|
||
links_doc = kassemble.check_links(links)
|
||
checked, more = check_answers(answers, acc, index, bundle, blocks_doc, kilde)
|
||
refusals += more
|
||
if refusals:
|
||
return refuse(refusals)
|
||
new_acc = copy.deepcopy(acc)
|
||
new_acc["fragments"] = [copy.deepcopy(x) for i, r in enumerate(acc["fragments"])
|
||
for x in (checked.new[i] if i in checked.new else [r])]
|
||
dropped = kpack.drop_dangling(new_acc)
|
||
passed = kpack.pass_homes(new_acc)
|
||
files = sorted({str(r["file"]) for r in acc["fragments"] + new_acc["fragments"]}
|
||
| {str(m["source"]) for m in acc["main_pages"]})
|
||
facts = kpack.source_facts(sources, files)
|
||
changed = {checked.keys[i]: [kassemble.label(r) for r in recs] for i, recs in checked.new.items()}
|
||
concepts = {c for c, _ in changed}
|
||
old_pages = kassemble.page_models(index, [r for r in acc["fragments"] if r["concept_id"] in concepts], kilde, facts,
|
||
acc["main_pages"])
|
||
new_pages = kassemble.page_models(index, [r for r in new_acc["fragments"] if r["concept_id"] in concepts], kilde,
|
||
facts, new_acc["main_pages"])
|
||
gone = {klink.page_rel(c) for c in kpack.pages_of(acc) - kpack.pages_of(new_acc)}
|
||
moves = move_links([dict(r) for r in links_doc["links"]], old_pages, new_pages, changed, gone)
|
||
new_links = with_records(links_doc, moves.records)
|
||
expected = trial(work, bundle, sources, new_acc, new_links)
|
||
_, _, cited_before = roles(acc, facts)
|
||
_, _, cited_after = roles(new_acc, facts)
|
||
no_source = [{"concept_id": str(m["concept_id"]), "source": str(m["source"])} for m in new_acc["main_pages"]
|
||
if str(m["concept_id"]) in cited_after and str(m["source"]) not in cited_after[str(m["concept_id"])]
|
||
and str(m["source"]) in cited_before.get(str(m["concept_id"]), set())]
|
||
left = kpack.left_without(new_acc, set(kpack.main_ids(acc)), dropped[2])
|
||
homeless = sorted(set(left) | (without_home(new_acc, facts) - without_home(acc, facts)))
|
||
log = read_repairs(work)
|
||
entry = {"n": len(log["entries"]) + 1, "command": "fix", "at": now(), "answers": str(answers_path),
|
||
"answers_sha256": file_sha256(answers_path), "bundle": str(bundle), "bundle_sha256": tree_sha256(bundle),
|
||
"fragments": [{"concept_id": checked.keys[i][0], "before": checked.keys[i][1],
|
||
"action": "replace" if recs else "remove", "after": [kassemble.label(r) for r in recs]}
|
||
for i, recs in sorted(checked.new.items())],
|
||
"claims_touched": {"before": sorted(set(moves.touched_before)), "after": sorted(set(moves.touched_after))},
|
||
"sentences_moved": moves.sentences_moved,
|
||
"links": {"moved": moves.moved, "removed": moves.removed},
|
||
"listed": {"pages_gone": sorted(gone), "without_home": homeless, "main_page_without_source": no_source,
|
||
"context_not_kept": checked.context_not_kept,
|
||
"dropped": {"main_pages": dropped[0], "homes": dropped[1], "homes_passed": passed}},
|
||
"expected": expected.as_dict()}
|
||
log["entries"].append(entry)
|
||
commit(work, "repair", {"accepted.json": new_acc, REPAIRS: log}, links,
|
||
new_links if moves.moved or moves.removed else None)
|
||
removed = sum(not recs for recs in checked.new.values())
|
||
print(f"fix: {len(checked.new)} answers: replaced {len(checked.new) - removed} fragments with "
|
||
f"{sum(len(recs) for recs in checked.new.values())}, removed {removed} -> "
|
||
f"{kpack.k_work(work) / 'accepted.json'}")
|
||
print(f" expected to change: {len(expected.pages)} pages ({len(expected.pages_gone)} gone), "
|
||
f"{len(expected.claims)} claims new or changed, {len(expected.claims_gone)} gone or changed")
|
||
print(f" links: moved {len(moves.moved)}, taken out {len(moves.removed)}; sentences moved "
|
||
f"{len(moves.sentences_moved)}")
|
||
for x in moves.removed:
|
||
r = x["record"]
|
||
print(f" taken out: {r['page']} {r['sentence']} {r['anchor']!r} -> {r['target']} ({x['why']})")
|
||
for x in checked.context_not_kept:
|
||
print(f" context not kept: {x['concept_id']} {x['footnote']}: {x['quote']!r}")
|
||
for name in sorted(gone):
|
||
print(f" page gone: {name}")
|
||
for name in homeless:
|
||
print(f" without a home now: {name}")
|
||
for x in no_source:
|
||
print(f" main page without its source: {x['concept_id']} ({x['source']})")
|
||
print(f" logged as repair {entry['n']} in {kpack.k_work(work) / REPAIRS}")
|
||
return 0
|
||
|
||
|
||
# --- unlink and relink -----------------------------------------------------------------------------------------
|
||
|
||
|
||
def find_record(records: list[dict[str, Any]], args: argparse.Namespace) -> int | None:
|
||
page = klink.page_rel(args.page)
|
||
return next((i for i, r in enumerate(records) if klink.page_rel(str(r["page"])) == page
|
||
and r["sentence"] == args.sentence and r["anchor"] == args.anchor and int(r["at"]) == args.at), None)
|
||
|
||
|
||
def link_entry(work: Path, command: str, bundle: Path, page: str, before: dict[str, Any],
|
||
after: dict[str, Any] | None) -> dict[str, Any]:
|
||
links = {"moved": [{"before": before, "after": after}], "removed": []} if after else {
|
||
"moved": [], "removed": [{"record": before, "why": "unlink"}]}
|
||
return {"n": len(read_repairs(work)["entries"]) + 1, "command": command, "at": now(), "bundle": str(bundle),
|
||
"bundle_sha256": tree_sha256(bundle), "links": links,
|
||
"expected": {"pages": [page], "pages_gone": [], "claims": [], "claims_gone": []}}
|
||
|
||
|
||
def write_link_change(args: argparse.Namespace, command: str, links: Path, doc: dict[str, Any],
|
||
records: list[dict[str, Any]], before: dict[str, Any], after: dict[str, Any] | None) -> None:
|
||
new_doc = with_records(doc, records)
|
||
if args.work is None:
|
||
write_links(links, new_doc)
|
||
return
|
||
work = Path(args.work)
|
||
kpack.check_pending(work)
|
||
log = read_repairs(work)
|
||
log["entries"].append(link_entry(work, command, Path(args.bundle), klink.page_rel(args.page), before, after))
|
||
commit(work, command, {REPAIRS: log}, links, new_doc)
|
||
|
||
|
||
def link_args(args: argparse.Namespace) -> tuple[Path, dict[str, Any], list[dict[str, Any]]]:
|
||
if args.work is not None and args.bundle is None:
|
||
raise RunError("--work needs --bundle: the log names the collection the record was taken out of")
|
||
links = Path(args.links)
|
||
check_links_tmp(links)
|
||
doc = kassemble.check_links(links)
|
||
return links, doc, [dict(r) for r in doc["links"]]
|
||
|
||
|
||
def cmd_unlink(args: argparse.Namespace) -> int:
|
||
links, doc, records = link_args(args)
|
||
i = find_record(records, args)
|
||
if i is None:
|
||
return refuse([Refusal("link-not-found", f"{klink.page_rel(args.page)} {args.sentence} {args.anchor!r} at "
|
||
f"{args.at}")])
|
||
before = records.pop(i)
|
||
write_link_change(args, "unlink", links, doc, records, before, None)
|
||
print(f"unlink: took out {before['page']} {before['sentence']} {before['anchor']!r} -> {before['target']}")
|
||
return 0
|
||
|
||
|
||
def cmd_relink(args: argparse.Namespace) -> int:
|
||
links, doc, records = link_args(args)
|
||
i = find_record(records, args)
|
||
page, target = klink.page_rel(args.page), klink.page_rel(args.to)
|
||
what = f"{page} {args.sentence} {args.anchor!r} at {args.at} -> {target}"
|
||
if i is None:
|
||
return refuse([Refusal("link-not-found", what)])
|
||
if target not in klink.read_pages(Path(args.bundle)):
|
||
return refuse([Refusal("target-missing", what)])
|
||
if target == page:
|
||
return refuse([Refusal("self-link", what)])
|
||
if any(klink.page_rel(str(r["page"])) == page and r["sentence"] == args.sentence
|
||
and klink.page_rel(str(r["target"])) == target for r in records):
|
||
return refuse([Refusal("duplicate-link", what)])
|
||
before = records[i]
|
||
after = {**before, "target": target}
|
||
records[i] = after
|
||
write_link_change(args, "relink", links, doc, records, before, after)
|
||
print(f"relink: {before['page']} {before['sentence']} {before['anchor']!r} -> {target} (was {before['target']})")
|
||
return 0
|
||
|
||
|
||
# --- linkset ---------------------------------------------------------------------------------------------------
|
||
|
||
|
||
def cmd_linkset(args: argparse.Namespace) -> int:
|
||
lst, refusals = read_list(Path(args.list), Path(args.claims) if args.claims else None)
|
||
if refusals:
|
||
return refuse(refusals)
|
||
bundle, out = Path(args.bundle), Path(args.out_dir)
|
||
pages = klink.read_pages(bundle)
|
||
sentences = {rel: {s.key: s for s in klink.body_sentences(p.body)} for rel, p in pages.items()}
|
||
wanted: dict[str, list[str]] = {}
|
||
targets: dict[str, list[str]] = {}
|
||
sharing: dict[str, set[str]] = {}
|
||
for x in lst["links"]:
|
||
page, target = klink.page_rel(str(x["page"])), klink.page_rel(str(x["target"]))
|
||
s = sentences.get(page, {}).get(str(x["sentence"]))
|
||
if s is None or target not in pages:
|
||
why = "no such sentence on the page" if s is None else f"no page {target}"
|
||
refusals.append(Refusal("stale-list", f"{page} {x['sentence']} ({why})"))
|
||
continue
|
||
if s.key not in wanted.setdefault(page, []):
|
||
wanted[page].append(s.key)
|
||
if stem(target) not in targets.setdefault(page, []):
|
||
targets[page].append(stem(target))
|
||
cites = {pages[page].resources[x] for x in s.labels if x in pages[page].resources}
|
||
sharing.setdefault(page, set()).update(rel for rel, p in pages.items() if rel != page
|
||
and cites & set(p.resources.values()))
|
||
if refusals:
|
||
return refuse(refusals)
|
||
if not wanted:
|
||
raise RunError(f"{args.list} lists no link candidate")
|
||
budget = klink.CATALOGUE_CHARS
|
||
groups: list[tuple[list[str], list[str]]] = [] # (pages, catalogue ids)
|
||
cut = 0
|
||
for page in sorted(wanted):
|
||
ids = targets[page] + sorted({stem(rel) for rel in sharing[page]} - set(targets[page]))
|
||
if groups:
|
||
union = groups[-1][1] + [i for i in ids if i not in groups[-1][1]]
|
||
if len(klink.catalogue_text(union, pages)) <= budget:
|
||
groups[-1] = (groups[-1][0] + [page], union)
|
||
continue
|
||
keep = list(targets[page])
|
||
for i in ids[len(targets[page]):]:
|
||
if len(klink.catalogue_text(keep + [i], pages)) <= budget:
|
||
keep.append(i)
|
||
cut += len(ids) - len(keep)
|
||
groups.append(([page], keep))
|
||
files: dict[Path, str] = {}
|
||
for n, (group_pages, ids) in enumerate(groups, 1):
|
||
files[out / f"group-{n}.sentences.json"] = klink.dump({"schema": klink.SENTENCES_SCHEMA,
|
||
"pages": {p: wanted[p] for p in group_pages}})
|
||
files[out / f"group-{n}.catalogue"] = "".join(f"{i}\n" for i in ids)
|
||
taken = sorted(str(p) for p in files if p.exists())
|
||
if taken:
|
||
raise RunError(f"{taken[0]} exists; refusing to overwrite")
|
||
out.mkdir(parents=True, exist_ok=True)
|
||
for path, text in files.items():
|
||
path.write_text(text, encoding="utf-8")
|
||
g = len(groups)
|
||
print(f"linkset: {sum(len(v) for v in wanted.values())} candidates on {len(wanted)} pages, {g} "
|
||
f"group{'s' if g != 1 else ''} -> {out}")
|
||
for n, (group_pages, ids) in enumerate(groups, 1):
|
||
print(f" group-{n}: {len(group_pages)} pages, {sum(len(wanted[p]) for p in group_pages)} sentences, "
|
||
f"catalogue {len(ids)} ids ({len(klink.catalogue_text(ids, pages))} characters)")
|
||
if cut:
|
||
print(f" cut {cut} catalogue ids that did not fit {budget} characters (the present targets kept)")
|
||
return 0
|
||
|
||
|
||
# --- diff ------------------------------------------------------------------------------------------------------
|
||
|
||
|
||
def cmd_diff(args: argparse.Namespace) -> int:
|
||
old, new, sources, out = Path(args.old), Path(args.new), Path(args.sources), Path(args.out)
|
||
log = kpack.read_json(Path(args.repairs))
|
||
if not (isinstance(log, dict) and log.get("schema") == REPAIRS_SCHEMA and isinstance(log.get("entries"), list)):
|
||
raise RunError(f"{args.repairs} is not {REPAIRS_SCHEMA}")
|
||
fp = tree_sha256(old)
|
||
entries = [e for e in log["entries"] if e.get("bundle_sha256") == fp]
|
||
m = measure(old, new, sources)
|
||
logged = {k: sorted({x for e in entries for x in e["expected"][k]}) for k in EXPECTED}
|
||
got = m.as_dict()
|
||
out.mkdir(parents=True, exist_ok=True)
|
||
(out / "ids.json").write_text(kpack.dump({"schema": IDS_SCHEMA, "old": str(old), "new": str(new),
|
||
"ids": m.claims}), encoding="utf-8")
|
||
(out / "sider.json").write_text(kpack.dump({"schema": PAGES_SCHEMA, "old": str(old), "new": str(new),
|
||
"pages": m.pages, "gone": m.pages_gone}), encoding="utf-8")
|
||
print(f"diff: {len(m.pages)} pages changed ({len(m.pages_gone)} gone), {len(m.claims)} claims new or changed, "
|
||
f"{len(m.claims_gone)} gone or changed; {len(entries)} of {len(log['entries'])} repairs logged against "
|
||
f"{old} -> {out}")
|
||
differs = False
|
||
if not entries and any(got.values()):
|
||
print(f"differs: no repair is logged against {old} (sha256 {fp[:12]})")
|
||
for k in EXPECTED:
|
||
extra, missing = sorted(set(got[k]) - set(logged[k])), sorted(set(logged[k]) - set(got[k]))
|
||
if extra:
|
||
print(f"differs: {k} measured, not logged: {len(extra)} ({', '.join(extra[:5])})")
|
||
if missing:
|
||
print(f"differs: {k} logged, not measured: {len(missing)} ({', '.join(missing[:5])})")
|
||
differs = differs or bool(extra or missing)
|
||
return 1 if differs else 0
|
||
|
||
|
||
# --- command line ----------------------------------------------------------------------------------------------
|
||
|
||
|
||
def build_parser() -> argparse.ArgumentParser:
|
||
p = argparse.ArgumentParser(prog="python3 -m okfwiki.krepair", description=__doc__,
|
||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||
sub = p.add_subparsers(dest="command", required=True)
|
||
ls = sub.add_parser("list")
|
||
for name in ("--claims", "--verdicts", "--out"):
|
||
ls.add_argument(name, required=True)
|
||
ls.add_argument("--ids")
|
||
t = sub.add_parser("task")
|
||
for name in ("--list", "--claims", "--pairs", "--work", "--blocks", "--sources", "--agent", "--out-dir"):
|
||
t.add_argument(name, required=True)
|
||
f = sub.add_parser("fix")
|
||
for name in ("--answers", "--bundle", "--work", "--blocks", "--sources", "--links"):
|
||
f.add_argument(name, required=True)
|
||
for command in ("unlink", "relink"):
|
||
u = sub.add_parser(command)
|
||
for name in ("--links", "--page", "--sentence", "--anchor"):
|
||
u.add_argument(name, required=True)
|
||
u.add_argument("--at", type=int, required=True)
|
||
u.add_argument("--work")
|
||
u.add_argument("--bundle", required=command == "relink")
|
||
if command == "relink":
|
||
u.add_argument("--to", required=True)
|
||
s = sub.add_parser("linkset")
|
||
for name in ("--list", "--bundle", "--out-dir"):
|
||
s.add_argument(name, required=True)
|
||
s.add_argument("--claims")
|
||
d = sub.add_parser("diff")
|
||
for name in ("--old", "--new", "--sources", "--repairs", "--out"):
|
||
d.add_argument(name, required=True)
|
||
return p
|
||
|
||
|
||
COMMANDS = {"list": cmd_list, "task": cmd_task, "fix": cmd_fix, "unlink": cmd_unlink, "relink": cmd_relink, "linkset": cmd_linkset,
|
||
"diff": cmd_diff}
|
||
|
||
|
||
def main(argv: list[str] | None = None) -> int:
|
||
args = build_parser().parse_args(argv)
|
||
try:
|
||
return COMMANDS[args.command](args)
|
||
except (RunError, reader.RunError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||
print(f"krepair cannot run: {exc}", file=sys.stderr)
|
||
return 2
|
||
except Exception: # an input this code did not foresee: still "could not run", with the traceback
|
||
traceback.print_exc()
|
||
return 2
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|