First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
1012 lines
53 KiB
Python
1012 lines
53 KiB
Python
"""Step 18a's merge of like concepts (order 29 A3): the model judges each candidate pair in the user's session, this
|
|
code finds the candidate pairs (B14), draws the control pairs, writes the tasks, reads the answers and merges the pairs
|
|
judged `same` in the build's work folder, so kassemble builds the merged collection. No model call, no network
|
|
(CLAUDE.md rule 1).
|
|
|
|
Usage:
|
|
python3 -m okfwiki.kmerge propose --work DIR --wiki DIR --claims FILE --out FILE [--judged PROPOSAL VERDICTS]...
|
|
python3 -m okfwiki.kmerge controls --proposal FILE --work DIR --wiki DIR --claims FILE [--home ID] --seed N
|
|
[--count 6] --out FILE
|
|
python3 -m okfwiki.kmerge tasks --proposal FILE --controls FILE --work DIR --wiki DIR --claims FILE --out DIR
|
|
[--only IDS]
|
|
python3 -m okfwiki.kmerge verdicts --tasks DIR --transcripts DIR --run RUN --out FILE
|
|
python3 -m okfwiki.kmerge apply --work DIR --wiki DIR --proposal FILE --claims FILE --verdicts FILE --report FILE
|
|
[--operator FILE]
|
|
|
|
The proposal lists candidate pairs as `groups`, each with `keep` and `merge_in` page names; a pair's id is its number
|
|
from 1 in `groups`. A page may stand in several pairs; a pair whose merge_in does not hold exactly one page, that pairs
|
|
a page with itself or repeats an earlier pair, cannot run. `keep` is read from the file and checked against the keep
|
|
rule (it says which page of the pair the proposal lists first, not which page a merge keeps: apply goes by the name): a
|
|
main page first, then the page with more claims (the claim list of okfwiki.claims for WIKI), more inbound links (pages
|
|
of WIKI that link to it), the earlier first pack (its fragments in WORK/K-work/accepted.json); a deviation cannot run
|
|
(STOP).
|
|
|
|
propose writes the proposal from the names alone (the title of each concept page of WIKI), for any language: every
|
|
pair of pages with a name each that is the same once spacing, punctuation, case and code spelling are set
|
|
aside (kind `compact`: listSessions, list_sessions(), List sessions), else whose words all stand in the
|
|
other's name (kind `contained`: Channel, Push messages with channels), a word matching one that adds at most
|
|
ENDING letters to a stem of STEM or more (agent/agentic, kube/kubene). Words: code spellings split, lower
|
|
case, letters beyond a-z kept, a leading "the" and a plural s dropped as kpack.norm drops them. No pair is
|
|
left out and none is ranked away; keep by the rule; prints the pairs, the calls and the tokens to judge them
|
|
at step 18a's measured rate. Measured 08.11.2026 on collection 1 before step 18a (wiki-15, 3,162 pages; 14
|
|
pairs known to be one concept, 10 by the judge in 18a and 4 by the operator): compact 19 pairs holding 10,
|
|
contained 3,676 more holding 3; the signals tried and left out held fewer: shared quoted lines 3 pairs holding
|
|
0, shared source sections 3,771 pairs holding 1, word likeness of the claims (tf-idf, 0.3) 3,664 pairs holding
|
|
10. Catalogue aliases added 0 pairs (15 pages have one; kpack already resolves a name to an alias at accept),
|
|
so they are not read. Never written again. --judged (repeatable) names an earlier proposal and the
|
|
verdicts read for it (its proposal_sha256 must match, else nothing is written): a pair of pages judged
|
|
`different` there, whichever page was kept, is left out and counted (summary judged_different, and per
|
|
earlier judging under judged); a pair judged `same` or not judged is proposed again.
|
|
controls draws at most --count control pairs, ids k1..kN, that must be judged `different`: two concept pages, neither
|
|
a main page nor in a proposal pair, with titles that are no compact pair (propose), and no cited line in
|
|
common (a sources[] resource and x-locator line of the one page on the other). First the pairs whose two
|
|
pages have --home as home (accepted.json homes), then, when they give fewer than --count, the other such
|
|
pairs; without --home only the latter. Each set in name order, shuffled with --seed, taken greedily so no
|
|
page stands twice. A build that has fewer gets the ones it has, and the count is printed (drawn); none at
|
|
all is written and said, and the verdicts then have no control to check them.
|
|
tasks writes one task per call (k001.md, ...) and tasks.json: the proposal's pairs and the controls together, at
|
|
most eight a call and at most TASK_CHARS characters a task, so the merger reads it with one Read call (a pair
|
|
over that stands alone), in an order fixed by the controls' seed. Per pair the two pages in name order, each
|
|
with its id, its title, its home (the title of each main page whose home list holds it) and its claims (a link
|
|
reduced to its text) with the excerpts they cite, at most twenty a page (the first in page order) and the
|
|
count of the rest; nothing else of the pages. With --only, the listed pairs go into new calls after the last,
|
|
numbered on, and no task already written changes.
|
|
verdicts reads each helper's reply from the subagent transcripts (agent type okf-wiki-merger, description
|
|
"V1-<run>-merge-<call>", started from a session or a workflow; the reply as okfwiki.reader.final_reply reads
|
|
it). Every non-empty line must be "<pair id> same" or "<pair id> different" for a pair of the call: one line
|
|
that cannot be read or names an id outside the call gives no verdict for any pair of the call, as the reader's
|
|
rule (K3 point 3), because the line cannot be tied to one pair; a pair with two lines gets none; a call
|
|
answered by two transcripts gets none. A pair asked once without exactly one valid line is missing (exit 1,
|
|
listed: the session runs one new call with tasks --only, then verdicts again); a pair asked in two calls that
|
|
still lacks exactly one is written without a verdict and is not merged. A control judged `same` is exit 3,
|
|
STOP before apply.
|
|
apply merges the pairs judged `same` in WORK (run on a copy first): pairs that share a page join one group, which
|
|
goes into the page with the shortest name, counted in letters once spacing, punctuation, case and code
|
|
spelling are set aside, so the merged page carries the concept's short name (Skills, not Extend Claude with
|
|
skills); names of one length go by the keep rule. Measured 13.11.2026 on collection 1's first merge, which
|
|
went by the keep rule alone: 48 of 118 merged pages carried a longer name than a page that went into them.
|
|
A group with a pair inside judged `different` is not merged and is listed under conflicts. Each page merged
|
|
in hands its fragments to the kept page's concept id, each marked `merged_from` with the id it had, so
|
|
kassemble takes the page's description from its own fragments; main page
|
|
records naming it name the kept page (x-main-for then lists both sources); homes on it point to the kept page
|
|
and its own homes are taken out; link records on it or to it name the kept page, and a record that then links
|
|
a page to itself, or names the page, sentence and target of an earlier record (klink's duplicate), is taken
|
|
out and counted; its catalogue entry is taken out and its title and aliases become aliases of the kept page
|
|
in WORK/K-work/index.json, which kassemble writes on the page as `x-aliases` and on its index line (the
|
|
operator 13.11.2026, in place of his "never on a page" of 18.10.2026). A home or main page record that then
|
|
points a page to itself or repeats an earlier one is taken out and counted. A sentence stands once on the
|
|
merged page: a sentence of a fragment that moves, when the same sentence (links and footnote marks set
|
|
aside) already stands on the kept page or on a page merged in before it, is cut from the fragment and listed
|
|
under cut with its source line; the rest of the fragment moves, a fragment with nothing left does not, and a
|
|
link record on the cut sentence goes to the one that stands. Code cannot tell a sentence that reads the same
|
|
but means another thing in its place (It requires version 2 or later), so every cut is printed to be read; a
|
|
sentence one page holds twice itself stays twice. --operator names a file of verdicts a person set in place
|
|
of the judge's (schema okf-wiki/merge-operator/1, `by` saying who, `verdicts` a map of pair ids of the
|
|
proposal to same or different; a control cannot be set); the report lists each one that differs. Refused
|
|
(exit 2, nothing written) when the verdicts are for another proposal or still miss a pair, the keep rule
|
|
fails, or the operator file does not fit; a control judged `same` is exit 3. Without a page to merge, WORK
|
|
is not written at all. The writes go under a marker as kpack's do. The report lists, per page merged in,
|
|
the pairs judged same it stands in and what moved, was cut, was taken out and was pointed again, and the
|
|
groups in conflict.
|
|
|
|
Exit 0 = done, 1 = verdicts missing (verdicts), 2 = cannot run or refused, 3 = a control judged same (STOP).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import random
|
|
import re
|
|
import shutil
|
|
import sys
|
|
import traceback
|
|
from collections import Counter
|
|
from dataclasses import dataclass, field
|
|
from pathlib import Path, PurePosixPath
|
|
from typing import Any
|
|
|
|
from okfwiki import audit, claims, kassemble, klink, kpack
|
|
from okfwiki.common import RunError
|
|
from okfwiki.reader import final_reply
|
|
from okfwiki.transcript import is_agent_type
|
|
|
|
TASKS_SCHEMA = "okf-wiki/merge-tasks/1"
|
|
CONTROLS_SCHEMA = "okf-wiki/merge-controls/1"
|
|
VERDICTS_SCHEMA = "okf-wiki/merge-verdicts/1"
|
|
APPLIED_SCHEMA = "okf-wiki/merge-applied/3"
|
|
OPERATOR_SCHEMA = "okf-wiki/merge-operator/1"
|
|
PROPOSAL_SCHEMA = "okf-wiki/merge-proposal/1"
|
|
KINDS = ("compact", "contained")
|
|
STEM, ENDING = 4, 2 # word_match: a stem of 4 letters or more, an ending of 2 at most (B14, measured 08.11.2026)
|
|
PER_PAIR_TOKENS = 154_545 // 28 # step 18a: four merger calls over 28 pairs on collection 1 (kost-M.json), a guide
|
|
CAMEL = re.compile(r"(?<=[a-zà-ÿ0-9])(?=[A-ZÀ-Þ])|(?<=[A-ZÀ-Þ])(?=[A-ZÀ-Þ][a-zà-ÿ]{2})")
|
|
AGENT_TYPE = "okf-wiki-merger"
|
|
PER_CALL = 8
|
|
TASK_CHARS = kpack.PART_CHARS # a task the merger reads whole with one Read call (kpack: the Read tool's cap)
|
|
CLAIMS_SHOWN = 20
|
|
SAME, DIFFERENT = "same", "different"
|
|
CONTROL_ID = re.compile(r"k[1-9][0-9]*")
|
|
CALL_NAME = re.compile(r"k([0-9]{3,})")
|
|
LINE = re.compile(r"(\S+)\s+(same|different)")
|
|
MARKER = "kmerge-apply.pending"
|
|
|
|
CONTRACT = """\
|
|
# Are the two pages one concept?
|
|
|
|
Each pair below shows two pages of a knowledge collection. Code found them as candidates for one concept from their
|
|
names alone: the names are the same once spacing and spelling are set aside, or the words of one name stand in the
|
|
other. Many such pairs are two different things. A page can stand in more than one pair. Decide for each pair, from
|
|
the meaning of the claims and the quotes they rest on, whether the two pages are about one and the same concept, so
|
|
that one page can hold the claims of both without mixing two things up, or about two different concepts.
|
|
|
|
- `same`: both pages are about one and the same thing, even when they come from different sources or describe it from
|
|
different sides.
|
|
- `different`: the pages are about two things, even when the two are related, have nearly the same name, or are listed
|
|
together.
|
|
|
|
Judge each pair on its own, from what this file shows. Read no other file.
|
|
|
|
Reply with one line per pair and nothing else: the pair id, one space, and `same` or `different`, for example
|
|
`12 different`. No heading, no explanation, no other line.
|
|
"""
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Pair:
|
|
id: str
|
|
keep: str
|
|
merge_in: str
|
|
kind: str = ""
|
|
|
|
|
|
@dataclass
|
|
class Facts:
|
|
titles: dict[str, str] # page -> title
|
|
main: set[str] # pages named as a main page in accepted.json
|
|
claims: dict[str, int] # page -> claims
|
|
inbound: dict[str, int] # page -> pages that link to it
|
|
first_pack: dict[str, int] # page -> earliest pack of its fragments
|
|
homes: dict[str, list[str]] # page -> the pages whose home list holds it
|
|
cited: dict[str, set[tuple[str, int]]] # page -> (resource, line) of its sources[]
|
|
|
|
|
|
@dataclass
|
|
class Applied:
|
|
pairs: list[dict[str, Any]] = field(default_factory=list)
|
|
counts: Counter[str] = field(default_factory=Counter)
|
|
cut: list[dict[str, Any]] = field(default_factory=list)
|
|
|
|
|
|
# --- inputs ----------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def read_json(path: Path) -> Any:
|
|
try:
|
|
return json.loads(path.read_text(encoding="utf-8"))
|
|
except ValueError as exc:
|
|
raise RunError(f"{path} is not JSON: {exc}") from exc
|
|
|
|
|
|
def sha(path: Path) -> str:
|
|
return hashlib.sha256(path.read_bytes()).hexdigest()
|
|
|
|
|
|
def cid(page: str) -> str:
|
|
return page.removesuffix(".md")
|
|
|
|
|
|
def load_proposal(path: Path) -> list[Pair]:
|
|
doc = read_json(path)
|
|
groups = doc.get("groups") if isinstance(doc, dict) else None
|
|
if not isinstance(groups, list):
|
|
raise RunError(f"{path} has no groups list")
|
|
pairs: list[Pair] = []
|
|
seen: set[frozenset[str]] = set()
|
|
for i, g in enumerate(groups, 1):
|
|
merge_in = g.get("merge_in") if isinstance(g, dict) else None
|
|
if not isinstance(merge_in, list) or len(merge_in) != 1 or not isinstance(g.get("keep"), str):
|
|
got = len(merge_in) if isinstance(merge_in, list) else "no list"
|
|
raise RunError(f"pair {i}: merge_in has {got} pages, not exactly one, or keep is missing")
|
|
pair = Pair(str(i), klink.page_rel(g["keep"]), klink.page_rel(str(merge_in[0])), str(g.get("kind", "")))
|
|
both = frozenset([pair.keep, pair.merge_in])
|
|
if len(both) == 1 or both in seen:
|
|
raise RunError(f"pair {i}: {pair.keep} and {pair.merge_in} are one page or an earlier pair")
|
|
seen.add(both)
|
|
pairs.append(pair)
|
|
return pairs
|
|
|
|
|
|
def load_controls(path: Path) -> tuple[list[Pair], dict[str, Any]]:
|
|
doc = read_json(path)
|
|
if not isinstance(doc, dict) or doc.get("schema") != CONTROLS_SCHEMA:
|
|
raise RunError(f"{path} is not a controls file ({CONTROLS_SCHEMA})")
|
|
pairs = [Pair(str(p["id"]), str(p["pages"][0]), str(p["pages"][1]), "control") for p in doc["pairs"]]
|
|
return pairs, doc
|
|
|
|
|
|
def page_facts(work: Path, wiki: Path, claims_path: Path) -> Facts:
|
|
accepted = kpack.read_accepted(work)
|
|
titles: dict[str, str] = {}
|
|
cited: dict[str, set[tuple[str, int]]] = {}
|
|
inbound: Counter[str] = Counter()
|
|
for path in sorted(wiki.glob("*.md")):
|
|
if not klink.is_concept(path.name):
|
|
continue
|
|
doc = audit.read_document(path, path.name)
|
|
fm = doc.frontmatter or {}
|
|
titles[path.name] = " ".join(str(fm.get("title") or cid(path.name)).split())
|
|
srcs = fm.get("sources")
|
|
cited[path.name] = {(str(s.get("resource")), int(s["x-locator"]["line"]))
|
|
for s in (srcs if isinstance(srcs, list) else []) if isinstance(s, dict)
|
|
and isinstance(s.get("x-locator"), dict) and type(s["x-locator"].get("line")) is int}
|
|
targets = {PurePosixPath(t.split("#", 1)[0]).name for _, t in audit.body_links(doc) if audit.is_bundle_link(t)}
|
|
inbound.update(targets - {path.name})
|
|
first: dict[str, int] = {}
|
|
for f in accepted["fragments"]:
|
|
page = f"{f['concept_id']}.md"
|
|
first[page] = min(first.get(page, int(f["pack"])), int(f["pack"]))
|
|
homes: dict[str, list[str]] = {}
|
|
for h in accepted["homes"]:
|
|
homes.setdefault(f"{h['concept_id']}.md", []).append(f"{h['home']}.md")
|
|
return Facts(titles, {f"{m['concept_id']}.md" for m in accepted["main_pages"]},
|
|
dict(Counter(c.page for c in claims.load_claims(claims_path).claims)), dict(inbound), first, homes,
|
|
cited)
|
|
|
|
|
|
def keep_key(page: str, facts: Facts) -> tuple[bool, int, int, int, str]:
|
|
return (page not in facts.main, -facts.claims.get(page, 0), -facts.inbound.get(page, 0),
|
|
facts.first_pack.get(page, 10**9), page)
|
|
|
|
|
|
def keep_of(pages: list[str], facts: Facts) -> str:
|
|
"""The keep rule of a proposal's pair: a main page, more claims, more inbound links, the earlier first pack."""
|
|
return min(pages, key=lambda x: keep_key(x, facts))
|
|
|
|
|
|
def named_of(pages: list[str], facts: Facts) -> str:
|
|
"""The page a group merges into: the one with the shortest name, counted in letters once spacing, punctuation,
|
|
case and code spelling are set aside (compact), so the merged page carries the concept's short name (Skills, not
|
|
Extend Claude with skills); names of one length go by the keep rule."""
|
|
return min(pages, key=lambda x: (len(compact(facts.titles[x])), keep_key(x, facts)))
|
|
|
|
|
|
def keep_deviations(pairs: list[Pair], facts: Facts) -> list[str]:
|
|
return [p.id for p in pairs if keep_of([p.keep, p.merge_in], facts) != p.keep]
|
|
|
|
|
|
def checked_pairs(proposal: Path, work: Path, wiki: Path, claims_path: Path) -> tuple[list[Pair], Facts]:
|
|
pairs = load_proposal(proposal)
|
|
facts = page_facts(work, wiki, claims_path)
|
|
absent = sorted({x for p in pairs for x in (p.keep, p.merge_in)} - set(facts.titles))
|
|
if absent:
|
|
raise RunError(f"pages of the proposal that {wiki} lacks: {', '.join(absent)}")
|
|
wrong = keep_deviations(pairs, facts)
|
|
if wrong:
|
|
raise RunError(f"STOP: keep differs from the rule (main page, claims, inbound links, first pack) in pair "
|
|
f"{', '.join(wrong)}")
|
|
return pairs, facts
|
|
|
|
|
|
# --- propose ---------------------------------------------------------------------------------------------------
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Candidate:
|
|
a: str
|
|
b: str
|
|
kind: str
|
|
names: tuple[str, str]
|
|
|
|
|
|
def name_words(name: str) -> list[str]:
|
|
"""The words of a name in any language: code spellings split (listSessions, list_sessions, SDKMessage), lower
|
|
case, letters beyond a-z kept, a leading "the" and a plural s dropped as kpack.norm drops them."""
|
|
text = CAMEL.sub(" ", name.replace("`", ""))
|
|
words = re.findall(r"[^\W_]+", text.lower())
|
|
if words and words[0] == "the":
|
|
words = words[1:]
|
|
return [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
|
|
|
|
|
|
def compact(name: str) -> str:
|
|
return "".join(name_words(name))
|
|
|
|
|
|
def word_match(x: str, y: str) -> bool:
|
|
"""One word, or one with an ending: the shorter is a stem of at least STEM letters and the longer adds at most
|
|
ENDING (agent/agentic, kube/kubene, channel/channels)."""
|
|
if x == y:
|
|
return True
|
|
short, long = (x, y) if len(x) <= len(y) else (y, x)
|
|
return len(short) >= STEM and long.startswith(short) and len(long) - len(short) <= ENDING
|
|
|
|
|
|
def candidates(names: dict[str, list[str]]) -> list[Candidate]:
|
|
"""Every pair of pages with a name each that is the same once spacing, punctuation, case and code spelling are
|
|
set aside (`compact`), or whose words all stand in the other's name (`contained`, word_match). Each pair once,
|
|
under the first kind; compact first, then by page name. No pair is left out, however many one name gives."""
|
|
keyed = [(page, name, name_words(name)) for page in sorted(names) for name in names[page]]
|
|
keyed = [k for k in keyed if k[2]]
|
|
found: dict[tuple[str, str], Candidate] = {}
|
|
|
|
def add(x: tuple[str, str], y: tuple[str, str], kind: str) -> None:
|
|
if x[0] == y[0]:
|
|
return
|
|
(pa, na), (pb, nb) = sorted([x, y])
|
|
found.setdefault((pa, pb), Candidate(pa, pb, kind, (na, nb)))
|
|
|
|
by_compact: dict[str, list[tuple[str, str]]] = {}
|
|
for page, name, words in keyed:
|
|
by_compact.setdefault("".join(words), []).append((page, name))
|
|
for group in by_compact.values():
|
|
for i, x in enumerate(group):
|
|
for y in group[i + 1:]:
|
|
add(x, y, "compact")
|
|
|
|
holders: dict[str, set[int]] = {} # word -> the names (by position in keyed) that hold it
|
|
for n, (_, _, words) in enumerate(keyed):
|
|
for w in words:
|
|
holders.setdefault(w, set()).add(n)
|
|
by_stem: dict[str, list[str]] = {}
|
|
for w in holders:
|
|
by_stem.setdefault(w[:STEM], []).append(w)
|
|
like = {w: [u for u in by_stem.get(w[:STEM], [w]) if word_match(w, u)] if len(w) >= STEM else [w]
|
|
for w in holders}
|
|
for n, (page, name, words) in enumerate(keyed):
|
|
inside: set[int] | None = None
|
|
for w in dict.fromkeys(words):
|
|
here = set().union(*(holders[u] for u in like[w]))
|
|
inside = here if inside is None else inside & here
|
|
for m in sorted(inside or set()):
|
|
add((page, name), keyed[m][:2], "contained")
|
|
return sorted(found.values(), key=lambda c: (KINDS.index(c.kind), c.a, c.b))
|
|
|
|
|
|
def judged_different(proposal: Path, verdicts: Path) -> set[frozenset[str]]:
|
|
"""The page pairs of an earlier proposal judged `different` in the verdicts read for it."""
|
|
doc = read_json(verdicts)
|
|
if not isinstance(doc, dict) or doc.get("schema") != VERDICTS_SCHEMA or not isinstance(doc.get("verdicts"), dict):
|
|
raise RunError(f"{verdicts} is not a verdicts file ({VERDICTS_SCHEMA})")
|
|
if doc.get("proposal_sha256") != sha(proposal):
|
|
raise RunError(f"{verdicts} holds verdicts for another proposal than {proposal}")
|
|
return {frozenset([p.keep, p.merge_in]) for p in load_proposal(proposal) if doc["verdicts"].get(p.id) == DIFFERENT}
|
|
|
|
|
|
def cmd_propose(args: argparse.Namespace) -> int:
|
|
work, out = Path(args.work), Path(args.out)
|
|
if out.exists():
|
|
raise RunError(f"{out} exists; a proposal is never written again")
|
|
facts = page_facts(work, Path(args.wiki), Path(args.claims))
|
|
found = candidates({page: [title] for page, title in facts.titles.items()})
|
|
judged: list[dict[str, Any]] = []
|
|
gone: set[frozenset[str]] = set()
|
|
for earlier, verdicts in args.judged or []:
|
|
different = judged_different(Path(earlier), Path(verdicts))
|
|
judged.append({"proposal_sha256": sha(Path(earlier)), "verdicts_sha256": sha(Path(verdicts)),
|
|
"different": len(different),
|
|
"left_out": sum(frozenset([c.a, c.b]) in different for c in found)})
|
|
gone |= different
|
|
left_out = sum(frozenset([c.a, c.b]) in gone for c in found)
|
|
found = [c for c in found if frozenset([c.a, c.b]) not in gone]
|
|
groups = []
|
|
for c in found:
|
|
keep = keep_of([c.a, c.b], facts)
|
|
names = c.names if keep == c.a else c.names[::-1]
|
|
groups.append({"kind": c.kind, "keep": keep, "merge_in": [c.b if keep == c.a else c.a],
|
|
"names": list(names)})
|
|
kinds = Counter(c.kind for c in found)
|
|
calls = -(-len(found) // PER_CALL)
|
|
summary = {"pairs": len(found), **{k: kinds[k] for k in KINDS},
|
|
"pages": len({x for c in found for x in (c.a, c.b)}), "calls": calls, "judged_different": left_out}
|
|
out.write_text(kpack.dump({"schema": PROPOSAL_SCHEMA, "summary": summary, "judged": judged, "groups": groups}),
|
|
encoding="utf-8")
|
|
print(f"kmerge propose: {len(found)} pairs ({', '.join(f'{k} {kinds[k]}' for k in KINDS)}) over "
|
|
f"{summary['pages']} pages, {calls} calls of at most {PER_CALL} besides the controls -> {out}")
|
|
if args.judged:
|
|
print(f" {left_out} pair{'' if left_out == 1 else 's'} judged different before left out "
|
|
f"({len(args.judged)} earlier judging{'' if len(args.judged) == 1 else 's'})")
|
|
print(f" at step 18a's measured {PER_PAIR_TOKENS:,} tokens a pair (collection 1, 18.10.2026): about "
|
|
f"{len(found) * PER_PAIR_TOKENS:,} tokens to judge them all")
|
|
return 0
|
|
|
|
|
|
# --- controls --------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def draw_controls(pairs: list[Pair], facts: Facts, home: str | None, seed: int, count: int) -> list[Pair]:
|
|
"""At most count control pairs: under home first, then from the other concept pages; fewer when the build has
|
|
fewer."""
|
|
taken = {x for p in pairs for x in (p.keep, p.merge_in)}
|
|
free = sorted(p for p in facts.titles if p not in facts.main and p not in taken)
|
|
|
|
def possible(pages: list[str]) -> list[tuple[str, str]]:
|
|
return [(a, b) for i, a in enumerate(pages) for b in pages[i + 1:]
|
|
if compact(facts.titles[a]) != compact(facts.titles[b])
|
|
and not facts.cited.get(a, set()) & facts.cited.get(b, set())]
|
|
|
|
rng = random.Random(seed)
|
|
under = possible([p for p in free if home is not None and klink.page_rel(home) in facts.homes.get(p, [])])
|
|
rng.shuffle(under)
|
|
first = set(under)
|
|
rest = [c for c in possible(free) if c not in first]
|
|
rng.shuffle(rest)
|
|
out: list[Pair] = []
|
|
used: set[str] = set()
|
|
for a, b in [*under, *rest]:
|
|
if len(out) == count:
|
|
break
|
|
if a in used or b in used:
|
|
continue
|
|
used.update([a, b])
|
|
out.append(Pair(f"k{len(out) + 1}", a, b, "control"))
|
|
return out
|
|
|
|
|
|
def cmd_controls(args: argparse.Namespace) -> int:
|
|
proposal, out = Path(args.proposal), Path(args.out)
|
|
if out.exists():
|
|
raise RunError(f"{out} exists; a draw is never written again")
|
|
pairs, facts = checked_pairs(proposal, Path(args.work), Path(args.wiki), Path(args.claims))
|
|
drawn = draw_controls(pairs, facts, args.home, args.seed, args.count)
|
|
doc = {"schema": CONTROLS_SCHEMA, "seed": args.seed, "home": args.home, "count": args.count,
|
|
"drawn": len(drawn), "proposal_sha256": sha(proposal),
|
|
"pairs": [{"id": p.id, "pages": [p.keep, p.merge_in]} for p in drawn]}
|
|
out.write_text(kpack.dump(doc), encoding="utf-8")
|
|
where = f"under {args.home} first" if args.home else "under any home"
|
|
print(f"kmerge controls: {len(drawn)} of {args.count} pairs, {where}, seed {args.seed} -> {out}")
|
|
if not drawn:
|
|
print(" no control pair can be drawn in this build: the verdicts have no control to check them")
|
|
elif len(drawn) < args.count:
|
|
print(f" the build has no more: {len(drawn)} of {args.count}")
|
|
return 0
|
|
|
|
|
|
# --- tasks -----------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def id_key(x: str) -> tuple[int, int, str]:
|
|
m = re.fullmatch(r"(k?)([0-9]+)", x)
|
|
return (1, int(m.group(2)), x) if m and m.group(1) else (0, int(m.group(2)), x) if m else (2, 0, x)
|
|
|
|
|
|
def chunk(order: list[str], per_call: int = PER_CALL, sizes: dict[str, int] | None = None,
|
|
cap: int | None = None) -> list[list[str]]:
|
|
"""The pairs in order, a call closed at per_call pairs or before the pair that would take its sizes over cap; a
|
|
pair over cap on its own stands alone."""
|
|
calls: list[list[str]] = []
|
|
used = 0
|
|
for x in order:
|
|
size = (sizes or {}).get(x, 0)
|
|
if not calls or len(calls[-1]) == per_call or (cap is not None and used + size > cap):
|
|
calls.append([])
|
|
used = 0
|
|
calls[-1].append(x)
|
|
used += size
|
|
return calls
|
|
|
|
|
|
def layout(ids: list[str], seed: int, per_call: int = PER_CALL, sizes: dict[str, int] | None = None,
|
|
cap: int | None = None) -> list[list[str]]:
|
|
order = sorted(ids, key=id_key)
|
|
random.Random(seed).shuffle(order)
|
|
return chunk(order, per_call, sizes, cap)
|
|
|
|
|
|
def page_block(page: str, title: str, homes: list[str], items: list[claims.Claim]) -> str:
|
|
shown = items[:CLAIMS_SHOWN]
|
|
lines = [f"Page `{cid(page)}`: {title}", "",
|
|
f"Home: {'; '.join(homes) if homes else 'none (a main page, or no main page lists it)'}", "",
|
|
f"Claims: {len(shown)} of {len(items)} shown"
|
|
+ (f" ({len(items) - len(shown)} more claims not shown)" if len(items) > len(shown) else ""), ""]
|
|
for n, c in enumerate(shown, 1):
|
|
lines.append(f"{n}. {' '.join(audit.sub_links(lambda m: m.group(1), c.claim).split())}")
|
|
lines += [f" - quote ({e.file}, line {e.line}): {' '.join(e.text.split())}" for e in c.excerpts]
|
|
return "\n".join(lines) + "\n"
|
|
|
|
|
|
def pair_text(pid: str, pairs: dict[str, Pair], facts: Facts, by_page: dict[str, list[claims.Claim]]) -> str:
|
|
p = pairs[pid]
|
|
parts = [f"## Pair {pid}", ""]
|
|
for page in sorted([p.keep, p.merge_in]):
|
|
homes = [facts.titles.get(h, cid(h)) for h in facts.homes.get(page, []) if h in facts.main]
|
|
parts.append(page_block(page, facts.titles[page], homes, by_page.get(page, [])))
|
|
return "\n".join(parts)
|
|
|
|
|
|
def render_task(call: list[str], pairs: dict[str, Pair], facts: Facts, by_page: dict[str, list[claims.Claim]]) -> str:
|
|
parts = [CONTRACT, f"# The pairs ({len(call)})", ""]
|
|
parts += [pair_text(pid, pairs, facts, by_page) for pid in call]
|
|
return "\n".join(parts).rstrip("\n") + "\n"
|
|
|
|
|
|
def cmd_tasks(args: argparse.Namespace) -> int:
|
|
proposal, controls_path, out = Path(args.proposal), Path(args.controls), Path(args.out)
|
|
pairs, facts = checked_pairs(proposal, Path(args.work), Path(args.wiki), Path(args.claims))
|
|
controls, cdoc = load_controls(controls_path)
|
|
if cdoc.get("proposal_sha256") != sha(proposal):
|
|
raise RunError(f"{controls_path} was drawn for another proposal")
|
|
every = {p.id: p for p in [*pairs, *controls]}
|
|
by_page: dict[str, list[claims.Claim]] = {}
|
|
for c in claims.load_claims(Path(args.claims)).claims:
|
|
by_page.setdefault(c.page, []).append(c)
|
|
sizes = {x: len(pair_text(x, every, facts, by_page)) + 1 for x in every}
|
|
cap = TASK_CHARS - len(render_task([], every, facts, by_page)) - 8 # the head, and digits in its pair count
|
|
index = out / "tasks.json"
|
|
if args.only is None:
|
|
if index.exists():
|
|
raise RunError(f"{index} exists; a new call for some pairs takes --only")
|
|
doc: dict[str, Any] = {"schema": TASKS_SCHEMA, "proposal_sha256": sha(proposal),
|
|
"controls_sha256": sha(controls_path), "seed": int(cdoc["seed"]),
|
|
"controls": [p.id for p in controls], "calls": []}
|
|
calls = layout(list(every), int(cdoc["seed"]), sizes=sizes, cap=cap)
|
|
else:
|
|
doc = read_json(index)
|
|
if doc.get("proposal_sha256") != sha(proposal) or doc.get("controls_sha256") != sha(controls_path):
|
|
raise RunError(f"{index} was written for another proposal or other controls")
|
|
wanted = [x.strip() for x in args.only.split(",") if x.strip()]
|
|
unknown = [x for x in wanted if x not in every]
|
|
if unknown or not wanted:
|
|
raise RunError(f"--only names no pair or unknown pairs: {', '.join(unknown)}")
|
|
calls = chunk(sorted(wanted, key=id_key), sizes=sizes, cap=cap)
|
|
out.mkdir(parents=True, exist_ok=True)
|
|
first = 1 + max((int(m.group(1)) for c in doc["calls"] if (m := CALL_NAME.fullmatch(str(c["name"])))), default=0)
|
|
for n, call in enumerate(calls, first):
|
|
name = f"k{n:03d}"
|
|
path = out / f"{name}.md"
|
|
if path.exists():
|
|
raise RunError(f"{path} exists; a task already written never changes")
|
|
path.write_text(render_task(call, every, facts, by_page), encoding="utf-8")
|
|
doc["calls"].append({"name": name, "pairs": call, "sha256": sha(path)})
|
|
tmp = out / ".tasks.json.tmp"
|
|
tmp.write_text(kpack.dump(doc), encoding="utf-8")
|
|
os.replace(tmp, index)
|
|
print(f"kmerge tasks: {sum(len(c) for c in calls)} pairs in {len(calls)} calls "
|
|
f"({', '.join(c['name'] for c in doc['calls'][-len(calls):])}) -> {out}")
|
|
return 0
|
|
|
|
|
|
# --- verdicts --------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def merge_transcripts(transcripts: Path, run: str) -> dict[str, list[Path]]:
|
|
"""Transcripts per call name of the merge helpers of one run (agent type okf-wiki-merger, bare or
|
|
plugin-namespaced), from a session or from a workflow."""
|
|
prefix = re.compile(rf"^V1-{re.escape(run)}-merge-(\S+)$")
|
|
found: dict[str, list[Path]] = {}
|
|
metas = [*transcripts.glob("*/subagents/agent-*.meta.json"),
|
|
*transcripts.glob("*/subagents/workflows/*/agent-*.meta.json")]
|
|
for meta_path in sorted(metas):
|
|
try:
|
|
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
|
except (OSError, ValueError):
|
|
continue
|
|
if not isinstance(meta, dict) or not is_agent_type(meta.get("agentType"), AGENT_TYPE):
|
|
continue
|
|
m = prefix.match(str(meta.get("description", "")))
|
|
if m is not None:
|
|
found.setdefault(m.group(1), []).append(meta_path.with_name(
|
|
meta_path.name.removesuffix(".meta.json") + ".jsonl"))
|
|
return found
|
|
|
|
|
|
def parse_reply(reply: str, ids: list[str]) -> tuple[dict[str, str], str]:
|
|
"""(verdict per pair of the call, what was wrong with the reply or ""): one unreadable line or unknown id voids
|
|
the call; a pair with two lines gets none."""
|
|
given: dict[str, list[str]] = {}
|
|
for raw in reply.splitlines():
|
|
line = raw.strip()
|
|
if not line:
|
|
continue
|
|
m = LINE.fullmatch(line)
|
|
if m is None:
|
|
return {}, "a line that cannot be read"
|
|
if m.group(1) not in ids:
|
|
return {}, f"an unknown id {m.group(1)}"
|
|
given.setdefault(m.group(1), []).append(m.group(2))
|
|
twice = [x for x, v in given.items() if len(v) > 1]
|
|
return {x: v[0] for x, v in given.items() if len(v) == 1}, f"two lines for {', '.join(twice)}" if twice else ""
|
|
|
|
|
|
def cmd_verdicts(args: argparse.Namespace) -> int:
|
|
tasks = Path(args.tasks)
|
|
doc = read_json(tasks / "tasks.json")
|
|
if not isinstance(doc, dict) or doc.get("schema") != TASKS_SCHEMA:
|
|
raise RunError(f"{tasks / 'tasks.json'} is not a tasks file ({TASKS_SCHEMA})")
|
|
found = merge_transcripts(Path(args.transcripts), args.run)
|
|
asked: Counter[str] = Counter()
|
|
valid: dict[str, list[str]] = {}
|
|
calls: dict[str, dict[str, Any]] = {}
|
|
for call in doc["calls"]:
|
|
name, ids = str(call["name"]), [str(x) for x in call["pairs"]]
|
|
asked.update(ids)
|
|
paths = found.get(name, [])
|
|
if len(paths) != 1:
|
|
calls[name] = {"transcripts": len(paths), "problem": "no reply" if not paths else "answered twice"}
|
|
continue
|
|
given, problem = parse_reply(final_reply(paths[0]) if paths[0].is_file() else "", ids)
|
|
calls[name] = {"transcripts": 1, "verdicts": len(given), "problem": problem}
|
|
for x, v in given.items():
|
|
valid.setdefault(x, []).append(v)
|
|
every = sorted(asked, key=id_key)
|
|
verdicts = {x: valid[x][0] for x in every if len(valid.get(x, [])) == 1}
|
|
missing = [x for x in every if x not in verdicts and asked[x] < 2]
|
|
without = [x for x in every if x not in verdicts and asked[x] >= 2]
|
|
controls = [str(x) for x in doc.get("controls", [])]
|
|
same_controls = [x for x in controls if verdicts.get(x) == SAME]
|
|
out = {"schema": VERDICTS_SCHEMA, "proposal_sha256": doc["proposal_sha256"],
|
|
"controls_sha256": doc["controls_sha256"], "controls": controls, "verdicts": verdicts, "missing": missing,
|
|
"without": without, "controls_same": same_controls, "calls": calls,
|
|
"stray": sorted(x for x in found if x not in calls)}
|
|
Path(args.out).write_text(kpack.dump(out), encoding="utf-8")
|
|
pairs = [x for x in every if x not in controls]
|
|
print(f"kmerge verdicts: pairs {len(pairs)}: same {sum(verdicts.get(x) == SAME for x in pairs)}, different "
|
|
f"{sum(verdicts.get(x) == DIFFERENT for x in pairs)}, missing {sum(x in missing for x in pairs)}, without "
|
|
f"{sum(x in without for x in pairs)}; controls {len(controls)}: different "
|
|
f"{sum(verdicts.get(x) == DIFFERENT for x in controls)}, same {len(same_controls)}, missing "
|
|
f"{sum(x in missing for x in controls)}, without {sum(x in without for x in controls)}")
|
|
for name, c in calls.items():
|
|
if c.get("problem"):
|
|
print(f" {name}: {c['problem']}")
|
|
if same_controls:
|
|
print(f"STOP: control pairs judged same: {', '.join(same_controls)}; nothing is merged")
|
|
return 3
|
|
if missing:
|
|
print(f"pairs without a verdict, for one new call: {', '.join(missing)}")
|
|
return 1
|
|
if without:
|
|
print(f"pairs without a verdict after their new call (not merged): {', '.join(without)}")
|
|
return 0
|
|
|
|
|
|
# --- apply -----------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def operator_verdicts(path: Path, pairs: list[Pair]) -> tuple[str, dict[str, str]]:
|
|
"""(who set them, the verdict per pair) of an operator file: verdicts a person set in place of the judge's. It
|
|
names who, and only pairs of the proposal; a control cannot be set."""
|
|
doc = read_json(path)
|
|
if not isinstance(doc, dict) or doc.get("schema") != OPERATOR_SCHEMA:
|
|
raise RunError(f"{path} is not an operator file ({OPERATOR_SCHEMA})")
|
|
by, given = doc.get("by"), doc.get("verdicts")
|
|
if not isinstance(by, str) or not by.strip():
|
|
raise RunError(f"{path}: by does not say who set the verdicts")
|
|
ids = {p.id for p in pairs}
|
|
if not isinstance(given, dict) or not given or any(x not in ids or v not in (SAME, DIFFERENT)
|
|
for x, v in given.items()):
|
|
raise RunError(f"{path}: verdicts is not a map of pair ids of the proposal to same or different")
|
|
return by.strip(), {str(x): str(v) for x, v in given.items()}
|
|
|
|
|
|
def given_verdicts(pairs: list[Pair], verdicts_path: Path, proposal: Path) -> tuple[dict[str, str], int]:
|
|
"""(the verdict of each pair, exit code 3 when a control is judged same else 0); a file that does not fit cannot
|
|
run."""
|
|
doc = read_json(verdicts_path)
|
|
if not isinstance(doc, dict) or doc.get("schema") != VERDICTS_SCHEMA:
|
|
raise RunError(f"{verdicts_path} is not a verdicts file ({VERDICTS_SCHEMA})")
|
|
if doc.get("proposal_sha256") != sha(proposal):
|
|
raise RunError(f"{verdicts_path} holds verdicts for another proposal")
|
|
if doc.get("missing"):
|
|
raise RunError(f"pairs still missing a verdict: {', '.join(map(str, doc['missing']))}; run verdicts first")
|
|
given = doc.get("verdicts")
|
|
if not isinstance(given, dict) or any(v not in (SAME, DIFFERENT) for v in given.values()):
|
|
raise RunError(f"{verdicts_path}: verdicts is not a map of pair ids to same or different")
|
|
ids = {p.id for p in pairs}
|
|
unknown = sorted(x for x in given if x not in ids and not CONTROL_ID.fullmatch(x))
|
|
if unknown:
|
|
raise RunError(f"{verdicts_path}: verdicts for unknown pairs {', '.join(unknown)}")
|
|
controls = sorted({*(x for x, v in given.items() if CONTROL_ID.fullmatch(x) and v == SAME),
|
|
*map(str, doc.get("controls_same") or [])})
|
|
if controls:
|
|
print(f"STOP: control pairs judged same: {', '.join(controls)}; nothing is merged")
|
|
return {}, 3
|
|
return {p.id: str(given[p.id]) for p in pairs if p.id in given}, 0
|
|
|
|
|
|
@dataclass
|
|
class Grouped:
|
|
into: dict[str, str] # page merged in -> page kept, in the order the pages first stand in a pair
|
|
why: dict[str, list[str]] # page merged in -> the pairs judged same it stands in
|
|
conflicts: list[dict[str, Any]]
|
|
|
|
|
|
def grouped(pairs: list[Pair], given: dict[str, str], facts: Facts) -> Grouped:
|
|
"""The pairs judged same joined into groups (a page in two such pairs joins them); each group goes into the page
|
|
with the shortest name (named_of). A group that holds a pair judged different is not merged and is listed as a
|
|
conflict: the verdicts disagree, and which page goes with which is the model's call, not code's."""
|
|
parent: dict[str, str] = {}
|
|
|
|
def root(x: str) -> str:
|
|
while parent.setdefault(x, x) != x:
|
|
x = parent[x]
|
|
return x
|
|
|
|
same = [p for p in pairs if given.get(p.id) == SAME]
|
|
for p in same:
|
|
parent[root(p.keep)] = root(p.merge_in)
|
|
members: dict[str, list[str]] = {}
|
|
for p in pairs: # pages in the order they first stand in a pair
|
|
for x in (p.keep, p.merge_in):
|
|
if x in parent and x not in members.get(root(x), []):
|
|
members.setdefault(root(x), []).append(x)
|
|
out = Grouped({}, {}, [])
|
|
for pages in members.values():
|
|
inside = set(pages)
|
|
against = [p.id for p in pairs if given.get(p.id) == DIFFERENT and {p.keep, p.merge_in} <= inside]
|
|
if against:
|
|
out.conflicts.append({"pages": sorted(pages), "same": sorted(
|
|
(p.id for p in same if p.keep in inside), key=id_key), "different": sorted(against, key=id_key)})
|
|
continue
|
|
keep = named_of(pages, facts)
|
|
for x in pages:
|
|
if x != keep:
|
|
out.into[x] = keep
|
|
out.why[x] = sorted((p.id for p in same if x in (p.keep, p.merge_in)), key=id_key)
|
|
return out
|
|
|
|
|
|
def sentence_digest(text: str) -> str:
|
|
return hashlib.sha256(klink.key_text(text).encode("utf-8")).hexdigest()[:16]
|
|
|
|
|
|
def fragment_sentences(text: str) -> list[tuple[str, str]]:
|
|
"""(digest, sentence) per sentence of a fragment's text as kassemble writes it on the page; the digest is the one
|
|
klink's sentence key starts with (measured 13.11.2026: equal on 3,335 of 3,335 pages of collection 1)."""
|
|
plain = kassemble.footnoted(kassemble.clean(text), [])
|
|
return [(sentence_digest(plain[a:b]), plain[a:b].strip()) for a, b in audit.sentence_spans(plain)]
|
|
|
|
|
|
def merged(work: Path, wiki: Path, pages: dict[str, str], why: dict[str, list[str]]
|
|
) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any] | None, Applied]:
|
|
"""accepted.json, index.json and links.json (None when work has none) with each page of `pages` merged into the
|
|
page it maps to, and the counts."""
|
|
into = {cid(m): cid(k) for m, k in pages.items()}
|
|
acc = kpack.read_accepted(work)
|
|
index = kpack.read_index(work)
|
|
links_path = work / "links.json"
|
|
links = read_json(links_path) if links_path.is_file() else None
|
|
applied = Applied()
|
|
n = applied.counts
|
|
per = {m: Counter[str]() for m in pages}
|
|
|
|
# A sentence stands once on the merged page: one that already stands there, from the kept page or from another
|
|
# page merged in, is cut from the fragment that moves; a sentence a page holds twice itself stays twice.
|
|
landed: dict[str, dict[str, str]] = {} # kept page -> sentence digest -> the page it stood on
|
|
for keep in dict.fromkeys(pages.values()):
|
|
text = (wiki / keep).read_text(encoding="utf-8")
|
|
landed[keep] = {s.key.rsplit("-", 1)[0]: keep for s in klink.page_sentences(text) if s.footnoted}
|
|
cut_keys: dict[str, set[str]] = {m: set() for m in pages} # page merged in -> digests of its cut sentences
|
|
order = sorted(range(len(acc["fragments"])), key=lambda i: tuple(acc["fragments"][i][k] for k in (
|
|
"pack", "position", "start", "end"))) # the order kassemble puts them on the page
|
|
dropped: set[int] = set()
|
|
for i in order:
|
|
f = acc["fragments"][i]
|
|
if f["concept_id"] not in into:
|
|
continue
|
|
origin = f"{f['concept_id']}.md"
|
|
here = landed[pages[origin]]
|
|
sentences = fragment_sentences(f["text"])
|
|
stays = [(d, t) for d, t in sentences if here.setdefault(d, origin) == origin]
|
|
for d, t in sentences:
|
|
if here[d] != origin:
|
|
cut_keys[origin].add(d)
|
|
applied.cut.append({"keep": pages[origin], "merge_in": origin, "file": f["file"],
|
|
"quote_line": f["quote_line"], "sentence": t,
|
|
"fragment": "moved" if stays else "dropped"})
|
|
per[origin]["sentences_cut"] += 1
|
|
if not stays:
|
|
dropped.add(i)
|
|
per[origin]["fragments_dropped"] += 1
|
|
continue
|
|
if len(stays) < len(sentences):
|
|
f["text"] = " ".join(t for _, t in stays)
|
|
per[origin]["fragments_moved"] += 1
|
|
f.setdefault("merged_from", f["concept_id"])
|
|
f["concept_id"] = into[f["concept_id"]]
|
|
acc["fragments"] = [f for i, f in enumerate(acc["fragments"]) if i not in dropped]
|
|
|
|
mains, seen = [], set()
|
|
for m in acc["main_pages"]:
|
|
if m["concept_id"] in into:
|
|
per[f"{m['concept_id']}.md"]["main_pages_repointed"] += 1
|
|
m = {**m, "concept_id": into[m["concept_id"]]}
|
|
key = (m["source"], m["concept_id"])
|
|
if key in seen:
|
|
n["main_pages_removed_duplicate"] += 1
|
|
continue
|
|
seen.add(key)
|
|
mains.append(m)
|
|
acc["main_pages"] = mains
|
|
|
|
homes, seen_h = [], set()
|
|
for h in acc["homes"]:
|
|
if h["concept_id"] in into:
|
|
per[f"{h['concept_id']}.md"]["homes_removed_own"] += 1
|
|
continue
|
|
if h["home"] in into:
|
|
per[f"{h['home']}.md"]["homes_repointed"] += 1
|
|
h = {**h, "home": into[h["home"]]}
|
|
key_h = (h["source"], h["concept_id"], h["home"])
|
|
if h["concept_id"] == h["home"]:
|
|
n["homes_removed_self"] += 1
|
|
continue
|
|
if key_h in seen_h:
|
|
n["homes_removed_duplicate"] += 1
|
|
continue
|
|
seen_h.add(key_h)
|
|
homes.append(h)
|
|
acc["homes"] = homes
|
|
|
|
entries = {str(e["id"]): e for e in index["entries"]}
|
|
for e in index["entries"]:
|
|
if e.get("merge_of") in into:
|
|
raise RunError(f"index entry {e['id']} is a merge candidate of {e['merge_of']}, a page merged in")
|
|
for m_id, k_id in into.items():
|
|
if m_id not in entries or k_id not in entries:
|
|
raise RunError(f"the catalogue lacks {m_id if m_id not in entries else k_id}")
|
|
old = list(entries[k_id].get("aliases") or [])
|
|
new = [str(entries[m_id]["title"]), *map(str, entries[m_id].get("aliases") or [])]
|
|
entries[k_id]["aliases"] = list(dict.fromkeys([*old, *new]))
|
|
per[f"{m_id}.md"]["aliases_added"] += len(entries[k_id]["aliases"]) - len(old)
|
|
index["entries"] = [e for e in index["entries"] if e["id"] not in into]
|
|
|
|
if links is not None:
|
|
kept, seen_l = [], set()
|
|
for r in links["links"]:
|
|
r = dict(r)
|
|
target = klink.page_rel(str(r["target"]))
|
|
if r["page"] in pages:
|
|
per[r["page"]]["links_page_repointed"] += 1
|
|
digest = str(r["sentence"]).rsplit("-", 1)[0]
|
|
if digest in cut_keys[r["page"]]: # its sentence was cut: the record goes to the one that stands
|
|
r["sentence"] = f"{digest}-0"
|
|
r["page"] = pages[r["page"]]
|
|
if target in pages:
|
|
per[target]["links_target_repointed"] += 1
|
|
r["target"] = pages[target]
|
|
if r["page"] == klink.page_rel(str(r["target"])):
|
|
n["links_removed_self"] += 1
|
|
continue
|
|
key_l = (r["page"], r["sentence"], klink.page_rel(str(r["target"])))
|
|
if key_l in seen_l:
|
|
n["links_removed_duplicate"] += 1
|
|
continue
|
|
seen_l.add(key_l)
|
|
kept.append(r)
|
|
links = {**links, "links": kept, "accepted": len(kept)}
|
|
|
|
for m, k in pages.items():
|
|
applied.pairs.append({"pairs": why[m], "keep": k, "merge_in": m, **per[m]})
|
|
n.update(per[m])
|
|
n["pages_gone"] = len(pages)
|
|
return acc, index, links, applied
|
|
|
|
|
|
def write_merged(work: Path, acc: dict[str, Any], index: dict[str, Any], links: dict[str, Any] | None) -> None:
|
|
"""kpack's way: back up the files about to be replaced, set a marker naming them, write, clear the marker."""
|
|
kw = kpack.k_work(work)
|
|
backup = kw / "backup"
|
|
if backup.exists():
|
|
shutil.rmtree(backup)
|
|
backup.mkdir()
|
|
files: dict[Path, str] = {kw / "accepted.json": kpack.dump(acc), kw / "index.json": kpack.dump(index)}
|
|
if links is not None:
|
|
files[work / "links.json"] = klink.dump(links)
|
|
for path in files:
|
|
shutil.copy2(path, backup / path.name)
|
|
marker = kw / MARKER
|
|
restore = [p.name if p.parent == kw else f"{p.name} (to {p})" for p in files]
|
|
marker.write_text(kpack.dump({"command": "kmerge apply", "restore": restore, "delete": [], "move_back": []}),
|
|
encoding="utf-8")
|
|
for path, text in files.items():
|
|
tmp = path.with_name(f".{path.name}.tmp")
|
|
tmp.write_text(text, encoding="utf-8")
|
|
os.replace(tmp, path)
|
|
marker.unlink()
|
|
|
|
|
|
def cmd_apply(args: argparse.Namespace) -> int:
|
|
work, wiki, proposal, report = Path(args.work), Path(args.wiki), Path(args.proposal), Path(args.report)
|
|
kpack.check_pending(work)
|
|
pairs, facts = checked_pairs(proposal, work, wiki, Path(args.claims))
|
|
given, code = given_verdicts(pairs, Path(args.verdicts), proposal)
|
|
if code:
|
|
return code
|
|
operator: dict[str, Any] | None = None
|
|
if args.operator:
|
|
by, chosen = operator_verdicts(Path(args.operator), pairs)
|
|
operator = {"by": by, "sha256": sha(Path(args.operator)),
|
|
"set": [{"pair": x, "from": given.get(x), "to": v} for x, v in sorted(
|
|
chosen.items(), key=lambda kv: id_key(kv[0])) if given.get(x) != v]}
|
|
given = {**given, **chosen}
|
|
g = grouped(pairs, given, facts)
|
|
if g.into:
|
|
acc, index, links, applied = merged(work, wiki, g.into, g.why)
|
|
write_merged(work, acc, index, links)
|
|
else:
|
|
applied = Applied()
|
|
applied.counts["pages_gone"] = 0
|
|
keys = ("pages_gone", "fragments_moved", "sentences_cut", "fragments_dropped", "main_pages_repointed",
|
|
"main_pages_removed_duplicate", "homes_removed_own", "homes_repointed", "homes_removed_self",
|
|
"homes_removed_duplicate", "links_page_repointed", "links_target_repointed", "links_removed_self",
|
|
"links_removed_duplicate", "aliases_added")
|
|
doc = {"schema": APPLIED_SCHEMA, "proposal_sha256": sha(proposal), "verdicts_sha256": sha(Path(args.verdicts)),
|
|
"operator": operator, **{k: int(applied.counts[k]) for k in keys}, "pairs": applied.pairs,
|
|
"conflicts": g.conflicts, "cut": applied.cut}
|
|
report.write_text(kpack.dump(doc), encoding="utf-8")
|
|
same = sum(v == SAME for x, v in given.items() if not CONTROL_ID.fullmatch(x))
|
|
print(f"kmerge apply: {len(g.into)} pages merged into {len(set(g.into.values()))} kept, from {same} of "
|
|
f"{len(pairs)} pairs judged same; {len(g.conflicts)} groups in conflict, {len(applied.cut)} sentences cut "
|
|
f"-> {work}")
|
|
print(" " + ", ".join(f"{k.replace('_', ' ')} {doc[k]}" for k in keys))
|
|
if operator is not None:
|
|
print(f" set by the operator ({operator['by']}): "
|
|
+ (", ".join(f"pair {x['pair']} {x['from']} -> {x['to']}" for x in operator["set"]) or "nothing new"))
|
|
for c in applied.cut:
|
|
print(f" cut, it stands on {c['keep']} already: a sentence of {c['merge_in']} ({c['file']}, line "
|
|
f"{c['quote_line']}; its fragment {c['fragment']}): {c['sentence']}")
|
|
for c in g.conflicts:
|
|
print(f" conflict, not merged: {', '.join(c['pages'])} (same {', '.join(c['same'])}; different "
|
|
f"{', '.join(c['different'])})")
|
|
return 0
|
|
|
|
|
|
# --- command line ----------------------------------------------------------------------------------------------
|
|
|
|
|
|
def build_parser() -> argparse.ArgumentParser:
|
|
p = argparse.ArgumentParser(prog="python3 -m okfwiki.kmerge", description=__doc__,
|
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
sub = p.add_subparsers(dest="command", required=True)
|
|
pr = sub.add_parser("propose")
|
|
pr.add_argument("--work", required=True)
|
|
pr.add_argument("--wiki", required=True)
|
|
pr.add_argument("--claims", required=True)
|
|
pr.add_argument("--out", required=True)
|
|
pr.add_argument("--judged", nargs=2, action="append", metavar=("PROPOSAL", "VERDICTS"))
|
|
c = sub.add_parser("controls")
|
|
t = sub.add_parser("tasks")
|
|
a = sub.add_parser("apply")
|
|
for s in (c, t, a):
|
|
s.add_argument("--proposal", required=True)
|
|
s.add_argument("--work", required=True)
|
|
s.add_argument("--wiki", required=True)
|
|
s.add_argument("--claims", required=True)
|
|
c.add_argument("--home")
|
|
c.add_argument("--seed", type=int, required=True)
|
|
c.add_argument("--count", type=int, default=6)
|
|
c.add_argument("--out", required=True)
|
|
t.add_argument("--controls", required=True)
|
|
t.add_argument("--out", required=True)
|
|
t.add_argument("--only")
|
|
v = sub.add_parser("verdicts")
|
|
v.add_argument("--tasks", required=True)
|
|
v.add_argument("--transcripts", required=True)
|
|
v.add_argument("--run", required=True)
|
|
v.add_argument("--out", required=True)
|
|
a.add_argument("--verdicts", required=True)
|
|
a.add_argument("--report", required=True)
|
|
a.add_argument("--operator")
|
|
return p
|
|
|
|
|
|
COMMANDS = {"propose": cmd_propose, "controls": cmd_controls, "tasks": cmd_tasks, "verdicts": cmd_verdicts,
|
|
"apply": cmd_apply}
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
args = build_parser().parse_args(argv)
|
|
try:
|
|
return COMMANDS[args.command](args)
|
|
except (RunError, OSError, UnicodeDecodeError, KeyError, ValueError, TypeError) as exc:
|
|
print(f"kmerge cannot run: {exc}", file=sys.stderr)
|
|
return 2
|
|
except Exception: # an input this code did not foresee
|
|
traceback.print_exc()
|
|
return 2
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|