"""Step 18a's merge of like concepts (order 29 A3): the model judges each candidate pair in the user's session, this code finds the candidate pairs (B14), draws the control pairs, writes the tasks, reads the answers and merges the pairs judged `same` in the build's work folder, so kassemble builds the merged collection. No model call, no network (CLAUDE.md rule 1). Usage: python3 -m okfwiki.kmerge propose --work DIR --wiki DIR --claims FILE --out FILE [--judged PROPOSAL VERDICTS]... python3 -m okfwiki.kmerge controls --proposal FILE --work DIR --wiki DIR --claims FILE [--home ID] --seed N [--count 6] --out FILE python3 -m okfwiki.kmerge tasks --proposal FILE --controls FILE --work DIR --wiki DIR --claims FILE --out DIR [--only IDS] python3 -m okfwiki.kmerge verdicts --tasks DIR --transcripts DIR --run RUN --out FILE python3 -m okfwiki.kmerge apply --work DIR --wiki DIR --proposal FILE --claims FILE --verdicts FILE --report FILE [--operator FILE] The proposal lists candidate pairs as `groups`, each with `keep` and `merge_in` page names; a pair's id is its number from 1 in `groups`. A page may stand in several pairs; a pair whose merge_in does not hold exactly one page, that pairs a page with itself or repeats an earlier pair, cannot run. `keep` is read from the file and checked against the keep rule (it says which page of the pair the proposal lists first, not which page a merge keeps: apply goes by the name): a main page first, then the page with more claims (the claim list of okfwiki.claims for WIKI), more inbound links (pages of WIKI that link to it), the earlier first pack (its fragments in WORK/K-work/accepted.json); a deviation cannot run (STOP). propose writes the proposal from the names alone (the title of each concept page of WIKI), for any language: every pair of pages with a name each that is the same once spacing, punctuation, case and code spelling are set aside (kind `compact`: listSessions, list_sessions(), List sessions), else whose words all stand in the other's name (kind `contained`: Channel, Push messages with channels), a word matching one that adds at most ENDING letters to a stem of STEM or more (agent/agentic, kube/kubene). Words: code spellings split, lower case, letters beyond a-z kept, a leading "the" and a plural s dropped as kpack.norm drops them. No pair is left out and none is ranked away; keep by the rule; prints the pairs, the calls and the tokens to judge them at step 18a's measured rate. Measured 08.11.2026 on collection 1 before step 18a (wiki-15, 3,162 pages; 14 pairs known to be one concept, 10 by the judge in 18a and 4 by the operator): compact 19 pairs holding 10, contained 3,676 more holding 3; the signals tried and left out held fewer: shared quoted lines 3 pairs holding 0, shared source sections 3,771 pairs holding 1, word likeness of the claims (tf-idf, 0.3) 3,664 pairs holding 10. Catalogue aliases added 0 pairs (15 pages have one; kpack already resolves a name to an alias at accept), so they are not read. Never written again. --judged (repeatable) names an earlier proposal and the verdicts read for it (its proposal_sha256 must match, else nothing is written): a pair of pages judged `different` there, whichever page was kept, is left out and counted (summary judged_different, and per earlier judging under judged); a pair judged `same` or not judged is proposed again. controls draws at most --count control pairs, ids k1..kN, that must be judged `different`: two concept pages, neither a main page nor in a proposal pair, with titles that are no compact pair (propose), and no cited line in common (a sources[] resource and x-locator line of the one page on the other). First the pairs whose two pages have --home as home (accepted.json homes), then, when they give fewer than --count, the other such pairs; without --home only the latter. Each set in name order, shuffled with --seed, taken greedily so no page stands twice. A build that has fewer gets the ones it has, and the count is printed (drawn); none at all is written and said, and the verdicts then have no control to check them. tasks writes one task per call (k001.md, ...) and tasks.json: the proposal's pairs and the controls together, at most eight a call and at most TASK_CHARS characters a task, so the merger reads it with one Read call (a pair over that stands alone), in an order fixed by the controls' seed. Per pair the two pages in name order, each with its id, its title, its home (the title of each main page whose home list holds it) and its claims (a link reduced to its text) with the excerpts they cite, at most twenty a page (the first in page order) and the count of the rest; nothing else of the pages. With --only, the listed pairs go into new calls after the last, numbered on, and no task already written changes. verdicts reads each helper's reply from the subagent transcripts (agent type okf-wiki-merger, description "V1--merge-", started from a session or a workflow; the reply as okfwiki.reader.final_reply reads it). Every non-empty line must be " same" or " different" for a pair of the call: one line that cannot be read or names an id outside the call gives no verdict for any pair of the call, as the reader's rule (K3 point 3), because the line cannot be tied to one pair; a pair with two lines gets none; a call answered by two transcripts gets none. A pair asked once without exactly one valid line is missing (exit 1, listed: the session runs one new call with tasks --only, then verdicts again); a pair asked in two calls that still lacks exactly one is written without a verdict and is not merged. A control judged `same` is exit 3, STOP before apply. apply merges the pairs judged `same` in WORK (run on a copy first): pairs that share a page join one group, which goes into the page with the shortest name, counted in letters once spacing, punctuation, case and code spelling are set aside, so the merged page carries the concept's short name (Skills, not Extend Claude with skills); names of one length go by the keep rule. Measured 13.11.2026 on collection 1's first merge, which went by the keep rule alone: 48 of 118 merged pages carried a longer name than a page that went into them. A group with a pair inside judged `different` is not merged and is listed under conflicts. Each page merged in hands its fragments to the kept page's concept id, each marked `merged_from` with the id it had, so kassemble takes the page's description from its own fragments; main page records naming it name the kept page (x-main-for then lists both sources); homes on it point to the kept page and its own homes are taken out; link records on it or to it name the kept page, and a record that then links a page to itself, or names the page, sentence and target of an earlier record (klink's duplicate), is taken out and counted; its catalogue entry is taken out and its title and aliases become aliases of the kept page in WORK/K-work/index.json, which kassemble writes on the page as `x-aliases` and on its index line (the operator 13.11.2026, in place of his "never on a page" of 18.10.2026). A home or main page record that then points a page to itself or repeats an earlier one is taken out and counted. A sentence stands once on the merged page: a sentence of a fragment that moves, when the same sentence (links and footnote marks set aside) already stands on the kept page or on a page merged in before it, is cut from the fragment and listed under cut with its source line; the rest of the fragment moves, a fragment with nothing left does not, and a link record on the cut sentence goes to the one that stands. Code cannot tell a sentence that reads the same but means another thing in its place (It requires version 2 or later), so every cut is printed to be read; a sentence one page holds twice itself stays twice. --operator names a file of verdicts a person set in place of the judge's (schema okf-wiki/merge-operator/1, `by` saying who, `verdicts` a map of pair ids of the proposal to same or different; a control cannot be set); the report lists each one that differs. Refused (exit 2, nothing written) when the verdicts are for another proposal or still miss a pair, the keep rule fails, or the operator file does not fit; a control judged `same` is exit 3. Without a page to merge, WORK is not written at all. The writes go under a marker as kpack's do. The report lists, per page merged in, the pairs judged same it stands in and what moved, was cut, was taken out and was pointed again, and the groups in conflict. Exit 0 = done, 1 = verdicts missing (verdicts), 2 = cannot run or refused, 3 = a control judged same (STOP). """ from __future__ import annotations import argparse import hashlib import json import os import random import re import shutil import sys import traceback from collections import Counter from dataclasses import dataclass, field from pathlib import Path, PurePosixPath from typing import Any from okfwiki import audit, claims, kassemble, klink, kpack from okfwiki.common import RunError from okfwiki.reader import final_reply from okfwiki.transcript import is_agent_type TASKS_SCHEMA = "okf-wiki/merge-tasks/1" CONTROLS_SCHEMA = "okf-wiki/merge-controls/1" VERDICTS_SCHEMA = "okf-wiki/merge-verdicts/1" APPLIED_SCHEMA = "okf-wiki/merge-applied/3" OPERATOR_SCHEMA = "okf-wiki/merge-operator/1" PROPOSAL_SCHEMA = "okf-wiki/merge-proposal/1" KINDS = ("compact", "contained") STEM, ENDING = 4, 2 # word_match: a stem of 4 letters or more, an ending of 2 at most (B14, measured 08.11.2026) PER_PAIR_TOKENS = 154_545 // 28 # step 18a: four merger calls over 28 pairs on collection 1 (kost-M.json), a guide CAMEL = re.compile(r"(?<=[a-zà-ÿ0-9])(?=[A-ZÀ-Þ])|(?<=[A-ZÀ-Þ])(?=[A-ZÀ-Þ][a-zà-ÿ]{2})") AGENT_TYPE = "okf-wiki-merger" PER_CALL = 8 TASK_CHARS = kpack.PART_CHARS # a task the merger reads whole with one Read call (kpack: the Read tool's cap) CLAIMS_SHOWN = 20 SAME, DIFFERENT = "same", "different" CONTROL_ID = re.compile(r"k[1-9][0-9]*") CALL_NAME = re.compile(r"k([0-9]{3,})") LINE = re.compile(r"(\S+)\s+(same|different)") MARKER = "kmerge-apply.pending" CONTRACT = """\ # Are the two pages one concept? Each pair below shows two pages of a knowledge collection. Code found them as candidates for one concept from their names alone: the names are the same once spacing and spelling are set aside, or the words of one name stand in the other. Many such pairs are two different things. A page can stand in more than one pair. Decide for each pair, from the meaning of the claims and the quotes they rest on, whether the two pages are about one and the same concept, so that one page can hold the claims of both without mixing two things up, or about two different concepts. - `same`: both pages are about one and the same thing, even when they come from different sources or describe it from different sides. - `different`: the pages are about two things, even when the two are related, have nearly the same name, or are listed together. Judge each pair on its own, from what this file shows. Read no other file. Reply with one line per pair and nothing else: the pair id, one space, and `same` or `different`, for example `12 different`. No heading, no explanation, no other line. """ @dataclass(frozen=True) class Pair: id: str keep: str merge_in: str kind: str = "" @dataclass class Facts: titles: dict[str, str] # page -> title main: set[str] # pages named as a main page in accepted.json claims: dict[str, int] # page -> claims inbound: dict[str, int] # page -> pages that link to it first_pack: dict[str, int] # page -> earliest pack of its fragments homes: dict[str, list[str]] # page -> the pages whose home list holds it cited: dict[str, set[tuple[str, int]]] # page -> (resource, line) of its sources[] @dataclass class Applied: pairs: list[dict[str, Any]] = field(default_factory=list) counts: Counter[str] = field(default_factory=Counter) cut: list[dict[str, Any]] = field(default_factory=list) # --- inputs ---------------------------------------------------------------------------------------------------- def read_json(path: Path) -> Any: try: return json.loads(path.read_text(encoding="utf-8")) except ValueError as exc: raise RunError(f"{path} is not JSON: {exc}") from exc def sha(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() def cid(page: str) -> str: return page.removesuffix(".md") def load_proposal(path: Path) -> list[Pair]: doc = read_json(path) groups = doc.get("groups") if isinstance(doc, dict) else None if not isinstance(groups, list): raise RunError(f"{path} has no groups list") pairs: list[Pair] = [] seen: set[frozenset[str]] = set() for i, g in enumerate(groups, 1): merge_in = g.get("merge_in") if isinstance(g, dict) else None if not isinstance(merge_in, list) or len(merge_in) != 1 or not isinstance(g.get("keep"), str): got = len(merge_in) if isinstance(merge_in, list) else "no list" raise RunError(f"pair {i}: merge_in has {got} pages, not exactly one, or keep is missing") pair = Pair(str(i), klink.page_rel(g["keep"]), klink.page_rel(str(merge_in[0])), str(g.get("kind", ""))) both = frozenset([pair.keep, pair.merge_in]) if len(both) == 1 or both in seen: raise RunError(f"pair {i}: {pair.keep} and {pair.merge_in} are one page or an earlier pair") seen.add(both) pairs.append(pair) return pairs def load_controls(path: Path) -> tuple[list[Pair], dict[str, Any]]: doc = read_json(path) if not isinstance(doc, dict) or doc.get("schema") != CONTROLS_SCHEMA: raise RunError(f"{path} is not a controls file ({CONTROLS_SCHEMA})") pairs = [Pair(str(p["id"]), str(p["pages"][0]), str(p["pages"][1]), "control") for p in doc["pairs"]] return pairs, doc def page_facts(work: Path, wiki: Path, claims_path: Path) -> Facts: accepted = kpack.read_accepted(work) titles: dict[str, str] = {} cited: dict[str, set[tuple[str, int]]] = {} inbound: Counter[str] = Counter() for path in sorted(wiki.glob("*.md")): if not klink.is_concept(path.name): continue doc = audit.read_document(path, path.name) fm = doc.frontmatter or {} titles[path.name] = " ".join(str(fm.get("title") or cid(path.name)).split()) srcs = fm.get("sources") cited[path.name] = {(str(s.get("resource")), int(s["x-locator"]["line"])) for s in (srcs if isinstance(srcs, list) else []) if isinstance(s, dict) and isinstance(s.get("x-locator"), dict) and type(s["x-locator"].get("line")) is int} targets = {PurePosixPath(t.split("#", 1)[0]).name for _, t in audit.body_links(doc) if audit.is_bundle_link(t)} inbound.update(targets - {path.name}) first: dict[str, int] = {} for f in accepted["fragments"]: page = f"{f['concept_id']}.md" first[page] = min(first.get(page, int(f["pack"])), int(f["pack"])) homes: dict[str, list[str]] = {} for h in accepted["homes"]: homes.setdefault(f"{h['concept_id']}.md", []).append(f"{h['home']}.md") return Facts(titles, {f"{m['concept_id']}.md" for m in accepted["main_pages"]}, dict(Counter(c.page for c in claims.load_claims(claims_path).claims)), dict(inbound), first, homes, cited) def keep_key(page: str, facts: Facts) -> tuple[bool, int, int, int, str]: return (page not in facts.main, -facts.claims.get(page, 0), -facts.inbound.get(page, 0), facts.first_pack.get(page, 10**9), page) def keep_of(pages: list[str], facts: Facts) -> str: """The keep rule of a proposal's pair: a main page, more claims, more inbound links, the earlier first pack.""" return min(pages, key=lambda x: keep_key(x, facts)) def named_of(pages: list[str], facts: Facts) -> str: """The page a group merges into: the one with the shortest name, counted in letters once spacing, punctuation, case and code spelling are set aside (compact), so the merged page carries the concept's short name (Skills, not Extend Claude with skills); names of one length go by the keep rule.""" return min(pages, key=lambda x: (len(compact(facts.titles[x])), keep_key(x, facts))) def keep_deviations(pairs: list[Pair], facts: Facts) -> list[str]: return [p.id for p in pairs if keep_of([p.keep, p.merge_in], facts) != p.keep] def checked_pairs(proposal: Path, work: Path, wiki: Path, claims_path: Path) -> tuple[list[Pair], Facts]: pairs = load_proposal(proposal) facts = page_facts(work, wiki, claims_path) absent = sorted({x for p in pairs for x in (p.keep, p.merge_in)} - set(facts.titles)) if absent: raise RunError(f"pages of the proposal that {wiki} lacks: {', '.join(absent)}") wrong = keep_deviations(pairs, facts) if wrong: raise RunError(f"STOP: keep differs from the rule (main page, claims, inbound links, first pack) in pair " f"{', '.join(wrong)}") return pairs, facts # --- propose --------------------------------------------------------------------------------------------------- @dataclass(frozen=True) class Candidate: a: str b: str kind: str names: tuple[str, str] def name_words(name: str) -> list[str]: """The words of a name in any language: code spellings split (listSessions, list_sessions, SDKMessage), lower case, letters beyond a-z kept, a leading "the" and a plural s dropped as kpack.norm drops them.""" text = CAMEL.sub(" ", name.replace("`", "")) words = re.findall(r"[^\W_]+", text.lower()) if words and words[0] == "the": words = words[1:] return [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words] def compact(name: str) -> str: return "".join(name_words(name)) def word_match(x: str, y: str) -> bool: """One word, or one with an ending: the shorter is a stem of at least STEM letters and the longer adds at most ENDING (agent/agentic, kube/kubene, channel/channels).""" if x == y: return True short, long = (x, y) if len(x) <= len(y) else (y, x) return len(short) >= STEM and long.startswith(short) and len(long) - len(short) <= ENDING def candidates(names: dict[str, list[str]]) -> list[Candidate]: """Every pair of pages with a name each that is the same once spacing, punctuation, case and code spelling are set aside (`compact`), or whose words all stand in the other's name (`contained`, word_match). Each pair once, under the first kind; compact first, then by page name. No pair is left out, however many one name gives.""" keyed = [(page, name, name_words(name)) for page in sorted(names) for name in names[page]] keyed = [k for k in keyed if k[2]] found: dict[tuple[str, str], Candidate] = {} def add(x: tuple[str, str], y: tuple[str, str], kind: str) -> None: if x[0] == y[0]: return (pa, na), (pb, nb) = sorted([x, y]) found.setdefault((pa, pb), Candidate(pa, pb, kind, (na, nb))) by_compact: dict[str, list[tuple[str, str]]] = {} for page, name, words in keyed: by_compact.setdefault("".join(words), []).append((page, name)) for group in by_compact.values(): for i, x in enumerate(group): for y in group[i + 1:]: add(x, y, "compact") holders: dict[str, set[int]] = {} # word -> the names (by position in keyed) that hold it for n, (_, _, words) in enumerate(keyed): for w in words: holders.setdefault(w, set()).add(n) by_stem: dict[str, list[str]] = {} for w in holders: by_stem.setdefault(w[:STEM], []).append(w) like = {w: [u for u in by_stem.get(w[:STEM], [w]) if word_match(w, u)] if len(w) >= STEM else [w] for w in holders} for n, (page, name, words) in enumerate(keyed): inside: set[int] | None = None for w in dict.fromkeys(words): here = set().union(*(holders[u] for u in like[w])) inside = here if inside is None else inside & here for m in sorted(inside or set()): add((page, name), keyed[m][:2], "contained") return sorted(found.values(), key=lambda c: (KINDS.index(c.kind), c.a, c.b)) def judged_different(proposal: Path, verdicts: Path) -> set[frozenset[str]]: """The page pairs of an earlier proposal judged `different` in the verdicts read for it.""" doc = read_json(verdicts) if not isinstance(doc, dict) or doc.get("schema") != VERDICTS_SCHEMA or not isinstance(doc.get("verdicts"), dict): raise RunError(f"{verdicts} is not a verdicts file ({VERDICTS_SCHEMA})") if doc.get("proposal_sha256") != sha(proposal): raise RunError(f"{verdicts} holds verdicts for another proposal than {proposal}") return {frozenset([p.keep, p.merge_in]) for p in load_proposal(proposal) if doc["verdicts"].get(p.id) == DIFFERENT} def cmd_propose(args: argparse.Namespace) -> int: work, out = Path(args.work), Path(args.out) if out.exists(): raise RunError(f"{out} exists; a proposal is never written again") facts = page_facts(work, Path(args.wiki), Path(args.claims)) found = candidates({page: [title] for page, title in facts.titles.items()}) judged: list[dict[str, Any]] = [] gone: set[frozenset[str]] = set() for earlier, verdicts in args.judged or []: different = judged_different(Path(earlier), Path(verdicts)) judged.append({"proposal_sha256": sha(Path(earlier)), "verdicts_sha256": sha(Path(verdicts)), "different": len(different), "left_out": sum(frozenset([c.a, c.b]) in different for c in found)}) gone |= different left_out = sum(frozenset([c.a, c.b]) in gone for c in found) found = [c for c in found if frozenset([c.a, c.b]) not in gone] groups = [] for c in found: keep = keep_of([c.a, c.b], facts) names = c.names if keep == c.a else c.names[::-1] groups.append({"kind": c.kind, "keep": keep, "merge_in": [c.b if keep == c.a else c.a], "names": list(names)}) kinds = Counter(c.kind for c in found) calls = -(-len(found) // PER_CALL) summary = {"pairs": len(found), **{k: kinds[k] for k in KINDS}, "pages": len({x for c in found for x in (c.a, c.b)}), "calls": calls, "judged_different": left_out} out.write_text(kpack.dump({"schema": PROPOSAL_SCHEMA, "summary": summary, "judged": judged, "groups": groups}), encoding="utf-8") print(f"kmerge propose: {len(found)} pairs ({', '.join(f'{k} {kinds[k]}' for k in KINDS)}) over " f"{summary['pages']} pages, {calls} calls of at most {PER_CALL} besides the controls -> {out}") if args.judged: print(f" {left_out} pair{'' if left_out == 1 else 's'} judged different before left out " f"({len(args.judged)} earlier judging{'' if len(args.judged) == 1 else 's'})") print(f" at step 18a's measured {PER_PAIR_TOKENS:,} tokens a pair (collection 1, 18.10.2026): about " f"{len(found) * PER_PAIR_TOKENS:,} tokens to judge them all") return 0 # --- controls -------------------------------------------------------------------------------------------------- def draw_controls(pairs: list[Pair], facts: Facts, home: str | None, seed: int, count: int) -> list[Pair]: """At most count control pairs: under home first, then from the other concept pages; fewer when the build has fewer.""" taken = {x for p in pairs for x in (p.keep, p.merge_in)} free = sorted(p for p in facts.titles if p not in facts.main and p not in taken) def possible(pages: list[str]) -> list[tuple[str, str]]: return [(a, b) for i, a in enumerate(pages) for b in pages[i + 1:] if compact(facts.titles[a]) != compact(facts.titles[b]) and not facts.cited.get(a, set()) & facts.cited.get(b, set())] rng = random.Random(seed) under = possible([p for p in free if home is not None and klink.page_rel(home) in facts.homes.get(p, [])]) rng.shuffle(under) first = set(under) rest = [c for c in possible(free) if c not in first] rng.shuffle(rest) out: list[Pair] = [] used: set[str] = set() for a, b in [*under, *rest]: if len(out) == count: break if a in used or b in used: continue used.update([a, b]) out.append(Pair(f"k{len(out) + 1}", a, b, "control")) return out def cmd_controls(args: argparse.Namespace) -> int: proposal, out = Path(args.proposal), Path(args.out) if out.exists(): raise RunError(f"{out} exists; a draw is never written again") pairs, facts = checked_pairs(proposal, Path(args.work), Path(args.wiki), Path(args.claims)) drawn = draw_controls(pairs, facts, args.home, args.seed, args.count) doc = {"schema": CONTROLS_SCHEMA, "seed": args.seed, "home": args.home, "count": args.count, "drawn": len(drawn), "proposal_sha256": sha(proposal), "pairs": [{"id": p.id, "pages": [p.keep, p.merge_in]} for p in drawn]} out.write_text(kpack.dump(doc), encoding="utf-8") where = f"under {args.home} first" if args.home else "under any home" print(f"kmerge controls: {len(drawn)} of {args.count} pairs, {where}, seed {args.seed} -> {out}") if not drawn: print(" no control pair can be drawn in this build: the verdicts have no control to check them") elif len(drawn) < args.count: print(f" the build has no more: {len(drawn)} of {args.count}") return 0 # --- tasks ----------------------------------------------------------------------------------------------------- def id_key(x: str) -> tuple[int, int, str]: m = re.fullmatch(r"(k?)([0-9]+)", x) return (1, int(m.group(2)), x) if m and m.group(1) else (0, int(m.group(2)), x) if m else (2, 0, x) def chunk(order: list[str], per_call: int = PER_CALL, sizes: dict[str, int] | None = None, cap: int | None = None) -> list[list[str]]: """The pairs in order, a call closed at per_call pairs or before the pair that would take its sizes over cap; a pair over cap on its own stands alone.""" calls: list[list[str]] = [] used = 0 for x in order: size = (sizes or {}).get(x, 0) if not calls or len(calls[-1]) == per_call or (cap is not None and used + size > cap): calls.append([]) used = 0 calls[-1].append(x) used += size return calls def layout(ids: list[str], seed: int, per_call: int = PER_CALL, sizes: dict[str, int] | None = None, cap: int | None = None) -> list[list[str]]: order = sorted(ids, key=id_key) random.Random(seed).shuffle(order) return chunk(order, per_call, sizes, cap) def page_block(page: str, title: str, homes: list[str], items: list[claims.Claim]) -> str: shown = items[:CLAIMS_SHOWN] lines = [f"Page `{cid(page)}`: {title}", "", f"Home: {'; '.join(homes) if homes else 'none (a main page, or no main page lists it)'}", "", f"Claims: {len(shown)} of {len(items)} shown" + (f" ({len(items) - len(shown)} more claims not shown)" if len(items) > len(shown) else ""), ""] for n, c in enumerate(shown, 1): lines.append(f"{n}. {' '.join(audit.sub_links(lambda m: m.group(1), c.claim).split())}") lines += [f" - quote ({e.file}, line {e.line}): {' '.join(e.text.split())}" for e in c.excerpts] return "\n".join(lines) + "\n" def pair_text(pid: str, pairs: dict[str, Pair], facts: Facts, by_page: dict[str, list[claims.Claim]]) -> str: p = pairs[pid] parts = [f"## Pair {pid}", ""] for page in sorted([p.keep, p.merge_in]): homes = [facts.titles.get(h, cid(h)) for h in facts.homes.get(page, []) if h in facts.main] parts.append(page_block(page, facts.titles[page], homes, by_page.get(page, []))) return "\n".join(parts) def render_task(call: list[str], pairs: dict[str, Pair], facts: Facts, by_page: dict[str, list[claims.Claim]]) -> str: parts = [CONTRACT, f"# The pairs ({len(call)})", ""] parts += [pair_text(pid, pairs, facts, by_page) for pid in call] return "\n".join(parts).rstrip("\n") + "\n" def cmd_tasks(args: argparse.Namespace) -> int: proposal, controls_path, out = Path(args.proposal), Path(args.controls), Path(args.out) pairs, facts = checked_pairs(proposal, Path(args.work), Path(args.wiki), Path(args.claims)) controls, cdoc = load_controls(controls_path) if cdoc.get("proposal_sha256") != sha(proposal): raise RunError(f"{controls_path} was drawn for another proposal") every = {p.id: p for p in [*pairs, *controls]} by_page: dict[str, list[claims.Claim]] = {} for c in claims.load_claims(Path(args.claims)).claims: by_page.setdefault(c.page, []).append(c) sizes = {x: len(pair_text(x, every, facts, by_page)) + 1 for x in every} cap = TASK_CHARS - len(render_task([], every, facts, by_page)) - 8 # the head, and digits in its pair count index = out / "tasks.json" if args.only is None: if index.exists(): raise RunError(f"{index} exists; a new call for some pairs takes --only") doc: dict[str, Any] = {"schema": TASKS_SCHEMA, "proposal_sha256": sha(proposal), "controls_sha256": sha(controls_path), "seed": int(cdoc["seed"]), "controls": [p.id for p in controls], "calls": []} calls = layout(list(every), int(cdoc["seed"]), sizes=sizes, cap=cap) else: doc = read_json(index) if doc.get("proposal_sha256") != sha(proposal) or doc.get("controls_sha256") != sha(controls_path): raise RunError(f"{index} was written for another proposal or other controls") wanted = [x.strip() for x in args.only.split(",") if x.strip()] unknown = [x for x in wanted if x not in every] if unknown or not wanted: raise RunError(f"--only names no pair or unknown pairs: {', '.join(unknown)}") calls = chunk(sorted(wanted, key=id_key), sizes=sizes, cap=cap) out.mkdir(parents=True, exist_ok=True) first = 1 + max((int(m.group(1)) for c in doc["calls"] if (m := CALL_NAME.fullmatch(str(c["name"])))), default=0) for n, call in enumerate(calls, first): name = f"k{n:03d}" path = out / f"{name}.md" if path.exists(): raise RunError(f"{path} exists; a task already written never changes") path.write_text(render_task(call, every, facts, by_page), encoding="utf-8") doc["calls"].append({"name": name, "pairs": call, "sha256": sha(path)}) tmp = out / ".tasks.json.tmp" tmp.write_text(kpack.dump(doc), encoding="utf-8") os.replace(tmp, index) print(f"kmerge tasks: {sum(len(c) for c in calls)} pairs in {len(calls)} calls " f"({', '.join(c['name'] for c in doc['calls'][-len(calls):])}) -> {out}") return 0 # --- verdicts -------------------------------------------------------------------------------------------------- def merge_transcripts(transcripts: Path, run: str) -> dict[str, list[Path]]: """Transcripts per call name of the merge helpers of one run (agent type okf-wiki-merger, bare or plugin-namespaced), from a session or from a workflow.""" prefix = re.compile(rf"^V1-{re.escape(run)}-merge-(\S+)$") found: dict[str, list[Path]] = {} metas = [*transcripts.glob("*/subagents/agent-*.meta.json"), *transcripts.glob("*/subagents/workflows/*/agent-*.meta.json")] for meta_path in sorted(metas): try: meta = json.loads(meta_path.read_text(encoding="utf-8")) except (OSError, ValueError): continue if not isinstance(meta, dict) or not is_agent_type(meta.get("agentType"), AGENT_TYPE): continue m = prefix.match(str(meta.get("description", ""))) if m is not None: found.setdefault(m.group(1), []).append(meta_path.with_name( meta_path.name.removesuffix(".meta.json") + ".jsonl")) return found def parse_reply(reply: str, ids: list[str]) -> tuple[dict[str, str], str]: """(verdict per pair of the call, what was wrong with the reply or ""): one unreadable line or unknown id voids the call; a pair with two lines gets none.""" given: dict[str, list[str]] = {} for raw in reply.splitlines(): line = raw.strip() if not line: continue m = LINE.fullmatch(line) if m is None: return {}, "a line that cannot be read" if m.group(1) not in ids: return {}, f"an unknown id {m.group(1)}" given.setdefault(m.group(1), []).append(m.group(2)) twice = [x for x, v in given.items() if len(v) > 1] return {x: v[0] for x, v in given.items() if len(v) == 1}, f"two lines for {', '.join(twice)}" if twice else "" def cmd_verdicts(args: argparse.Namespace) -> int: tasks = Path(args.tasks) doc = read_json(tasks / "tasks.json") if not isinstance(doc, dict) or doc.get("schema") != TASKS_SCHEMA: raise RunError(f"{tasks / 'tasks.json'} is not a tasks file ({TASKS_SCHEMA})") found = merge_transcripts(Path(args.transcripts), args.run) asked: Counter[str] = Counter() valid: dict[str, list[str]] = {} calls: dict[str, dict[str, Any]] = {} for call in doc["calls"]: name, ids = str(call["name"]), [str(x) for x in call["pairs"]] asked.update(ids) paths = found.get(name, []) if len(paths) != 1: calls[name] = {"transcripts": len(paths), "problem": "no reply" if not paths else "answered twice"} continue given, problem = parse_reply(final_reply(paths[0]) if paths[0].is_file() else "", ids) calls[name] = {"transcripts": 1, "verdicts": len(given), "problem": problem} for x, v in given.items(): valid.setdefault(x, []).append(v) every = sorted(asked, key=id_key) verdicts = {x: valid[x][0] for x in every if len(valid.get(x, [])) == 1} missing = [x for x in every if x not in verdicts and asked[x] < 2] without = [x for x in every if x not in verdicts and asked[x] >= 2] controls = [str(x) for x in doc.get("controls", [])] same_controls = [x for x in controls if verdicts.get(x) == SAME] out = {"schema": VERDICTS_SCHEMA, "proposal_sha256": doc["proposal_sha256"], "controls_sha256": doc["controls_sha256"], "controls": controls, "verdicts": verdicts, "missing": missing, "without": without, "controls_same": same_controls, "calls": calls, "stray": sorted(x for x in found if x not in calls)} Path(args.out).write_text(kpack.dump(out), encoding="utf-8") pairs = [x for x in every if x not in controls] print(f"kmerge verdicts: pairs {len(pairs)}: same {sum(verdicts.get(x) == SAME for x in pairs)}, different " f"{sum(verdicts.get(x) == DIFFERENT for x in pairs)}, missing {sum(x in missing for x in pairs)}, without " f"{sum(x in without for x in pairs)}; controls {len(controls)}: different " f"{sum(verdicts.get(x) == DIFFERENT for x in controls)}, same {len(same_controls)}, missing " f"{sum(x in missing for x in controls)}, without {sum(x in without for x in controls)}") for name, c in calls.items(): if c.get("problem"): print(f" {name}: {c['problem']}") if same_controls: print(f"STOP: control pairs judged same: {', '.join(same_controls)}; nothing is merged") return 3 if missing: print(f"pairs without a verdict, for one new call: {', '.join(missing)}") return 1 if without: print(f"pairs without a verdict after their new call (not merged): {', '.join(without)}") return 0 # --- apply ----------------------------------------------------------------------------------------------------- def operator_verdicts(path: Path, pairs: list[Pair]) -> tuple[str, dict[str, str]]: """(who set them, the verdict per pair) of an operator file: verdicts a person set in place of the judge's. It names who, and only pairs of the proposal; a control cannot be set.""" doc = read_json(path) if not isinstance(doc, dict) or doc.get("schema") != OPERATOR_SCHEMA: raise RunError(f"{path} is not an operator file ({OPERATOR_SCHEMA})") by, given = doc.get("by"), doc.get("verdicts") if not isinstance(by, str) or not by.strip(): raise RunError(f"{path}: by does not say who set the verdicts") ids = {p.id for p in pairs} if not isinstance(given, dict) or not given or any(x not in ids or v not in (SAME, DIFFERENT) for x, v in given.items()): raise RunError(f"{path}: verdicts is not a map of pair ids of the proposal to same or different") return by.strip(), {str(x): str(v) for x, v in given.items()} def given_verdicts(pairs: list[Pair], verdicts_path: Path, proposal: Path) -> tuple[dict[str, str], int]: """(the verdict of each pair, exit code 3 when a control is judged same else 0); a file that does not fit cannot run.""" doc = read_json(verdicts_path) if not isinstance(doc, dict) or doc.get("schema") != VERDICTS_SCHEMA: raise RunError(f"{verdicts_path} is not a verdicts file ({VERDICTS_SCHEMA})") if doc.get("proposal_sha256") != sha(proposal): raise RunError(f"{verdicts_path} holds verdicts for another proposal") if doc.get("missing"): raise RunError(f"pairs still missing a verdict: {', '.join(map(str, doc['missing']))}; run verdicts first") given = doc.get("verdicts") if not isinstance(given, dict) or any(v not in (SAME, DIFFERENT) for v in given.values()): raise RunError(f"{verdicts_path}: verdicts is not a map of pair ids to same or different") ids = {p.id for p in pairs} unknown = sorted(x for x in given if x not in ids and not CONTROL_ID.fullmatch(x)) if unknown: raise RunError(f"{verdicts_path}: verdicts for unknown pairs {', '.join(unknown)}") controls = sorted({*(x for x, v in given.items() if CONTROL_ID.fullmatch(x) and v == SAME), *map(str, doc.get("controls_same") or [])}) if controls: print(f"STOP: control pairs judged same: {', '.join(controls)}; nothing is merged") return {}, 3 return {p.id: str(given[p.id]) for p in pairs if p.id in given}, 0 @dataclass class Grouped: into: dict[str, str] # page merged in -> page kept, in the order the pages first stand in a pair why: dict[str, list[str]] # page merged in -> the pairs judged same it stands in conflicts: list[dict[str, Any]] def grouped(pairs: list[Pair], given: dict[str, str], facts: Facts) -> Grouped: """The pairs judged same joined into groups (a page in two such pairs joins them); each group goes into the page with the shortest name (named_of). A group that holds a pair judged different is not merged and is listed as a conflict: the verdicts disagree, and which page goes with which is the model's call, not code's.""" parent: dict[str, str] = {} def root(x: str) -> str: while parent.setdefault(x, x) != x: x = parent[x] return x same = [p for p in pairs if given.get(p.id) == SAME] for p in same: parent[root(p.keep)] = root(p.merge_in) members: dict[str, list[str]] = {} for p in pairs: # pages in the order they first stand in a pair for x in (p.keep, p.merge_in): if x in parent and x not in members.get(root(x), []): members.setdefault(root(x), []).append(x) out = Grouped({}, {}, []) for pages in members.values(): inside = set(pages) against = [p.id for p in pairs if given.get(p.id) == DIFFERENT and {p.keep, p.merge_in} <= inside] if against: out.conflicts.append({"pages": sorted(pages), "same": sorted( (p.id for p in same if p.keep in inside), key=id_key), "different": sorted(against, key=id_key)}) continue keep = named_of(pages, facts) for x in pages: if x != keep: out.into[x] = keep out.why[x] = sorted((p.id for p in same if x in (p.keep, p.merge_in)), key=id_key) return out def sentence_digest(text: str) -> str: return hashlib.sha256(klink.key_text(text).encode("utf-8")).hexdigest()[:16] def fragment_sentences(text: str) -> list[tuple[str, str]]: """(digest, sentence) per sentence of a fragment's text as kassemble writes it on the page; the digest is the one klink's sentence key starts with (measured 13.11.2026: equal on 3,335 of 3,335 pages of collection 1).""" plain = kassemble.footnoted(kassemble.clean(text), []) return [(sentence_digest(plain[a:b]), plain[a:b].strip()) for a, b in audit.sentence_spans(plain)] def merged(work: Path, wiki: Path, pages: dict[str, str], why: dict[str, list[str]] ) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any] | None, Applied]: """accepted.json, index.json and links.json (None when work has none) with each page of `pages` merged into the page it maps to, and the counts.""" into = {cid(m): cid(k) for m, k in pages.items()} acc = kpack.read_accepted(work) index = kpack.read_index(work) links_path = work / "links.json" links = read_json(links_path) if links_path.is_file() else None applied = Applied() n = applied.counts per = {m: Counter[str]() for m in pages} # A sentence stands once on the merged page: one that already stands there, from the kept page or from another # page merged in, is cut from the fragment that moves; a sentence a page holds twice itself stays twice. landed: dict[str, dict[str, str]] = {} # kept page -> sentence digest -> the page it stood on for keep in dict.fromkeys(pages.values()): text = (wiki / keep).read_text(encoding="utf-8") landed[keep] = {s.key.rsplit("-", 1)[0]: keep for s in klink.page_sentences(text) if s.footnoted} cut_keys: dict[str, set[str]] = {m: set() for m in pages} # page merged in -> digests of its cut sentences order = sorted(range(len(acc["fragments"])), key=lambda i: tuple(acc["fragments"][i][k] for k in ( "pack", "position", "start", "end"))) # the order kassemble puts them on the page dropped: set[int] = set() for i in order: f = acc["fragments"][i] if f["concept_id"] not in into: continue origin = f"{f['concept_id']}.md" here = landed[pages[origin]] sentences = fragment_sentences(f["text"]) stays = [(d, t) for d, t in sentences if here.setdefault(d, origin) == origin] for d, t in sentences: if here[d] != origin: cut_keys[origin].add(d) applied.cut.append({"keep": pages[origin], "merge_in": origin, "file": f["file"], "quote_line": f["quote_line"], "sentence": t, "fragment": "moved" if stays else "dropped"}) per[origin]["sentences_cut"] += 1 if not stays: dropped.add(i) per[origin]["fragments_dropped"] += 1 continue if len(stays) < len(sentences): f["text"] = " ".join(t for _, t in stays) per[origin]["fragments_moved"] += 1 f.setdefault("merged_from", f["concept_id"]) f["concept_id"] = into[f["concept_id"]] acc["fragments"] = [f for i, f in enumerate(acc["fragments"]) if i not in dropped] mains, seen = [], set() for m in acc["main_pages"]: if m["concept_id"] in into: per[f"{m['concept_id']}.md"]["main_pages_repointed"] += 1 m = {**m, "concept_id": into[m["concept_id"]]} key = (m["source"], m["concept_id"]) if key in seen: n["main_pages_removed_duplicate"] += 1 continue seen.add(key) mains.append(m) acc["main_pages"] = mains homes, seen_h = [], set() for h in acc["homes"]: if h["concept_id"] in into: per[f"{h['concept_id']}.md"]["homes_removed_own"] += 1 continue if h["home"] in into: per[f"{h['home']}.md"]["homes_repointed"] += 1 h = {**h, "home": into[h["home"]]} key_h = (h["source"], h["concept_id"], h["home"]) if h["concept_id"] == h["home"]: n["homes_removed_self"] += 1 continue if key_h in seen_h: n["homes_removed_duplicate"] += 1 continue seen_h.add(key_h) homes.append(h) acc["homes"] = homes entries = {str(e["id"]): e for e in index["entries"]} for e in index["entries"]: if e.get("merge_of") in into: raise RunError(f"index entry {e['id']} is a merge candidate of {e['merge_of']}, a page merged in") for m_id, k_id in into.items(): if m_id not in entries or k_id not in entries: raise RunError(f"the catalogue lacks {m_id if m_id not in entries else k_id}") old = list(entries[k_id].get("aliases") or []) new = [str(entries[m_id]["title"]), *map(str, entries[m_id].get("aliases") or [])] entries[k_id]["aliases"] = list(dict.fromkeys([*old, *new])) per[f"{m_id}.md"]["aliases_added"] += len(entries[k_id]["aliases"]) - len(old) index["entries"] = [e for e in index["entries"] if e["id"] not in into] if links is not None: kept, seen_l = [], set() for r in links["links"]: r = dict(r) target = klink.page_rel(str(r["target"])) if r["page"] in pages: per[r["page"]]["links_page_repointed"] += 1 digest = str(r["sentence"]).rsplit("-", 1)[0] if digest in cut_keys[r["page"]]: # its sentence was cut: the record goes to the one that stands r["sentence"] = f"{digest}-0" r["page"] = pages[r["page"]] if target in pages: per[target]["links_target_repointed"] += 1 r["target"] = pages[target] if r["page"] == klink.page_rel(str(r["target"])): n["links_removed_self"] += 1 continue key_l = (r["page"], r["sentence"], klink.page_rel(str(r["target"]))) if key_l in seen_l: n["links_removed_duplicate"] += 1 continue seen_l.add(key_l) kept.append(r) links = {**links, "links": kept, "accepted": len(kept)} for m, k in pages.items(): applied.pairs.append({"pairs": why[m], "keep": k, "merge_in": m, **per[m]}) n.update(per[m]) n["pages_gone"] = len(pages) return acc, index, links, applied def write_merged(work: Path, acc: dict[str, Any], index: dict[str, Any], links: dict[str, Any] | None) -> None: """kpack's way: back up the files about to be replaced, set a marker naming them, write, clear the marker.""" kw = kpack.k_work(work) backup = kw / "backup" if backup.exists(): shutil.rmtree(backup) backup.mkdir() files: dict[Path, str] = {kw / "accepted.json": kpack.dump(acc), kw / "index.json": kpack.dump(index)} if links is not None: files[work / "links.json"] = klink.dump(links) for path in files: shutil.copy2(path, backup / path.name) marker = kw / MARKER restore = [p.name if p.parent == kw else f"{p.name} (to {p})" for p in files] marker.write_text(kpack.dump({"command": "kmerge apply", "restore": restore, "delete": [], "move_back": []}), encoding="utf-8") for path, text in files.items(): tmp = path.with_name(f".{path.name}.tmp") tmp.write_text(text, encoding="utf-8") os.replace(tmp, path) marker.unlink() def cmd_apply(args: argparse.Namespace) -> int: work, wiki, proposal, report = Path(args.work), Path(args.wiki), Path(args.proposal), Path(args.report) kpack.check_pending(work) pairs, facts = checked_pairs(proposal, work, wiki, Path(args.claims)) given, code = given_verdicts(pairs, Path(args.verdicts), proposal) if code: return code operator: dict[str, Any] | None = None if args.operator: by, chosen = operator_verdicts(Path(args.operator), pairs) operator = {"by": by, "sha256": sha(Path(args.operator)), "set": [{"pair": x, "from": given.get(x), "to": v} for x, v in sorted( chosen.items(), key=lambda kv: id_key(kv[0])) if given.get(x) != v]} given = {**given, **chosen} g = grouped(pairs, given, facts) if g.into: acc, index, links, applied = merged(work, wiki, g.into, g.why) write_merged(work, acc, index, links) else: applied = Applied() applied.counts["pages_gone"] = 0 keys = ("pages_gone", "fragments_moved", "sentences_cut", "fragments_dropped", "main_pages_repointed", "main_pages_removed_duplicate", "homes_removed_own", "homes_repointed", "homes_removed_self", "homes_removed_duplicate", "links_page_repointed", "links_target_repointed", "links_removed_self", "links_removed_duplicate", "aliases_added") doc = {"schema": APPLIED_SCHEMA, "proposal_sha256": sha(proposal), "verdicts_sha256": sha(Path(args.verdicts)), "operator": operator, **{k: int(applied.counts[k]) for k in keys}, "pairs": applied.pairs, "conflicts": g.conflicts, "cut": applied.cut} report.write_text(kpack.dump(doc), encoding="utf-8") same = sum(v == SAME for x, v in given.items() if not CONTROL_ID.fullmatch(x)) print(f"kmerge apply: {len(g.into)} pages merged into {len(set(g.into.values()))} kept, from {same} of " f"{len(pairs)} pairs judged same; {len(g.conflicts)} groups in conflict, {len(applied.cut)} sentences cut " f"-> {work}") print(" " + ", ".join(f"{k.replace('_', ' ')} {doc[k]}" for k in keys)) if operator is not None: print(f" set by the operator ({operator['by']}): " + (", ".join(f"pair {x['pair']} {x['from']} -> {x['to']}" for x in operator["set"]) or "nothing new")) for c in applied.cut: print(f" cut, it stands on {c['keep']} already: a sentence of {c['merge_in']} ({c['file']}, line " f"{c['quote_line']}; its fragment {c['fragment']}): {c['sentence']}") for c in g.conflicts: print(f" conflict, not merged: {', '.join(c['pages'])} (same {', '.join(c['same'])}; different " f"{', '.join(c['different'])})") return 0 # --- command line ---------------------------------------------------------------------------------------------- def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(prog="python3 -m okfwiki.kmerge", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) sub = p.add_subparsers(dest="command", required=True) pr = sub.add_parser("propose") pr.add_argument("--work", required=True) pr.add_argument("--wiki", required=True) pr.add_argument("--claims", required=True) pr.add_argument("--out", required=True) pr.add_argument("--judged", nargs=2, action="append", metavar=("PROPOSAL", "VERDICTS")) c = sub.add_parser("controls") t = sub.add_parser("tasks") a = sub.add_parser("apply") for s in (c, t, a): s.add_argument("--proposal", required=True) s.add_argument("--work", required=True) s.add_argument("--wiki", required=True) s.add_argument("--claims", required=True) c.add_argument("--home") c.add_argument("--seed", type=int, required=True) c.add_argument("--count", type=int, default=6) c.add_argument("--out", required=True) t.add_argument("--controls", required=True) t.add_argument("--out", required=True) t.add_argument("--only") v = sub.add_parser("verdicts") v.add_argument("--tasks", required=True) v.add_argument("--transcripts", required=True) v.add_argument("--run", required=True) v.add_argument("--out", required=True) a.add_argument("--verdicts", required=True) a.add_argument("--report", required=True) a.add_argument("--operator") return p COMMANDS = {"propose": cmd_propose, "controls": cmd_controls, "tasks": cmd_tasks, "verdicts": cmd_verdicts, "apply": cmd_apply} def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) try: return COMMANDS[args.command](args) except (RunError, OSError, UnicodeDecodeError, KeyError, ValueError, TypeError) as exc: print(f"kmerge cannot run: {exc}", file=sys.stderr) return 2 except Exception: # an input this code did not foresee traceback.print_exc() return 2 if __name__ == "__main__": sys.exit(main())