"""The source manifest of a frozen copy (contract K1): check, freeze and subset; the change warning (contract K4): diff and marks. No network, no model. Usage: python3 -m okfwiki.sources check ROOT python3 -m okfwiki.sources freeze STAGING --meta FILE --out ROOT [--frozen-at YYYY-MM-DDTHH:MM:SSZ] python3 -m okfwiki.sources folder DIR --collection NAME --out ROOT [--frozen-at YYYY-MM-DDTHH:MM:SSZ] python3 -m okfwiki.sources subset ROOT --paths FILE --out ROOT2 [--frozen-at YYYY-MM-DDTHH:MM:SSZ] python3 -m okfwiki.sources diff OLD NEW python3 -m okfwiki.sources marks COLLECTION --diff FILE python3 -m okfwiki.sources reread ROOT --out ROOT2 [--frozen-at YYYY-MM-DDTHH:MM:SSZ] python3 -m okfwiki.sources watch ROOT --into DIR [--bundle COLLECTION] [--frozen-at YYYY-MM-DDTHH:MM:SSZ] A frozen copy is a root folder with source-manifest.json and the source files under kilde/. A page cites a source as sources[].resource "/" + path. check Exit 0 when every listed file exists with the same sha256, bytes and lines, every file under kilde/ is listed, and the manifest has the form of K1; exit 1 with the problems listed; exit 2 when it cannot run (no manifest, not JSON, another schema, sources not a list). freeze Copies the files of STAGING (flat, named as the part of path after kilde/) to ROOT/kilde/, makes them read only and writes the manifest. The meta file says per file what freeze cannot know: {"collection": "...", "index": {...} (optional), "schema": "okf-wiki/source-meta/1", "sources": {"": {"kind": "...", "main_topic": true|false, "url": "...", "version": "..."}}} Refused with exit 1 and the files listed: a file the meta lacks or a meta entry without a file, an invalid entry, a file that is not UTF-8, and a file whose lines differ between splitlines() and split("\\n") (the auditor uses the one, the trial code the other, so line N must be the same for both). Exit 2 when ROOT exists. folder Freezes a folder of documents as it is, with no staging and no hand-written meta: every file under DIR, in subfolders too, whose suffix has a reader (.md, .markdown, .txt, and .pdf, .docx, .xlsx, .pptx through okfwiki.readers). A PDF or Office file is frozen as the text a reader of it sees (its safe name plus ".md"), with the file itself copied read-only to original/ (`original_copy`), `reader` (the type), `original_sha256`, `places` (the page, paragraph, cell or slide of each line, null for a blank line or a heading) and `left_out` (hidden parts left out, also reported); every document records `lang` (common.language) when its words tell it, for the view; one that cannot be read or has no text is skipped and named, and a PDF without pypdf at the pinned version is refused with exit 1 and the install command. Each gets a safe name under kilde/: its path relative to DIR with "/" as "__", letters spelled in ASCII (æ ø å as ae oe aa, accents dropped), lower case, any other character as "-"; a name that would read as a release note (release-vX.Y.Z.md) ends in "-doc.md" instead, since the release-note rules go by the name. The entry keeps the path as it was (`original`, NFC), the file's own URL (`url`, file:///) and what was changed to make it a source (`normalised`, [] when the copy is byte for byte): "bom" (a UTF-8 byte order mark dropped), "cp1252" (decoded as Windows-1252, the Western Latin-1 superset, when not UTF-8) and "line-breaks" (\r\n, \r, form feed and every other line break written as \n), so a quote stands verbatim in the frozen copy. Kind "document", one main topic. A file is skipped, and named with the reason, when it is hidden (a name starting with "."; a hidden folder is skipped whole), has no reader, is empty or is a link. Two files with one safe name are refused with exit 1; exit 2 when no file is left, or DIR is not a folder, or ROOT exists. The manifest names the folder read (`folder`, its resolved path), so the copy can be read again from itself. subset A part of ROOT with the listed paths (one kilde/... per line, sorted, no duplicates), its own frozen_at, the index unchanged and subset_of pointing at ROOT's frozen_at and the sha256 of ROOT's manifest. Exit 2 when ROOT fails check, the paths file is not as described, or ROOT2 exists. diff Which sources of NEW are new, changed or gone since OLD, by path. Changed = another sha256 or another kind (the kind decides what a source is to the build). Exit 0 = no change, 1 = at least one, 2 = cannot run (a root fails check, or the two roots name another collection). marks The pages of COLLECTION (every .md under it, named by its path with "/") that cite a changed or gone source of the diff as sources[].resource "/" + path. New sources mark nothing. A page without frontmatter cites nothing; one whose frontmatter cannot be read stops marks. Exit 0 = no page marked, 1 = at least one, 2 = cannot run. Both print their report with the one written form, so two runs are byte-identical. A source read from a folder is named in diff with its `original` path too, as the user knows the file. reread Reads the folder ROOT was read from again, as the same collection, into ROOT2 (the folder command's rules). The folder is the manifest's `folder`, or for a copy frozen before that field, the one folder its entries name (each url less its original path). Exit 2 when ROOT fails check, was not read from a folder, its entries name two folders, the folder is gone, or ROOT2 exists. watch The change warning for a folder, one command for any scheduler (motor B15): reads the folder of ROOT again into DIR/latest (replaced whole each run), writes diff.json there and, with --bundle, marks.json, and prints each difference by its original path with the pages marked on it. DIR/watch-state.json keeps the lines reported, so a run says only what is new since the last one (a new frozen copy starts over). Exit 0 = nothing new, 1 = something new, 2 = cannot run (also DIR inside the watched folder: the next reading would take the watch's files for documents). Only a warning: the pages are rebuilt from DIR/latest in a session. """ from __future__ import annotations import argparse import hashlib import json import os import re import shutil import sys import tempfile import unicodedata import urllib.parse from datetime import datetime, timezone from pathlib import Path from typing import Any import yaml from okfwiki import readers from okfwiki.common import RunError, ascii_fold, language from okfwiki.kblocks import RELEASE SCHEMA = "okf-wiki/source-manifest/1" META_SCHEMA = "okf-wiki/source-meta/1" DIFF_SCHEMA = "okf-wiki/source-diff/1" MARKS_SCHEMA = "okf-wiki/source-marks/1" MANIFEST = "source-manifest.json" KILDE = "kilde" PATH = re.compile(r"^kilde/[a-z0-9._-]+$") NAME = re.compile(r"^[a-z0-9._-]+$") HEX64 = re.compile(r"^[0-9a-f]{64}$") FROZEN_AT = re.compile(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$") READ_ONLY = 0o444 class Refused(Exception): """The input breaks a rule; nothing was written. A command reports the problems with exit 1.""" def __init__(self, problems: list[str]) -> None: super().__init__("; ".join(problems)) self.problems = problems def dump(data: Any) -> str: """The one written form of every file here, so two writings of the same content are byte-identical.""" return json.dumps(data, indent=1, sort_keys=True, ensure_ascii=False) + "\n" def sha256(data: bytes) -> str: return hashlib.sha256(data).hexdigest() def ambiguous_lines(text: str) -> bool: """True when splitlines() and split("\\n") (less a final empty part) give different lines.""" parts = text.split("\n") if parts[-1] == "": parts.pop() return parts != text.splitlines() def utc_now() -> str: return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") def load_json(path: Path) -> Any: try: return json.loads(path.read_text(encoding="utf-8")) except FileNotFoundError: raise RunError(f"{path} does not exist") from None except (json.JSONDecodeError, UnicodeDecodeError) as exc: raise RunError(f"{path} is not JSON: {exc}") from None def load_manifest(root: Path) -> dict[str, Any]: """The manifest of ROOT, or RunError when there is none that check can read.""" if not root.is_dir(): raise RunError(f"{root} is not a directory") doc = load_json(root / MANIFEST) if not isinstance(doc, dict): raise RunError(f"{root / MANIFEST} is not a JSON object") if doc.get("schema") != SCHEMA: raise RunError(f"{root / MANIFEST} has schema {doc.get('schema')!r}, not {SCHEMA}") if not isinstance(doc.get("sources"), list): raise RunError(f"{root / MANIFEST} has no list of sources") return doc def nonempty_str(value: Any) -> bool: return isinstance(value, str) and value != "" def describe_problems(meta: Any) -> list[str]: """The rules for what the meta says about one source (K1 rules 3, 4, 6).""" if not isinstance(meta, dict): return ["is not an object"] out = [] if not nonempty_str(meta.get("kind")): out.append("kind is not a non-empty string") if not isinstance(meta.get("main_topic"), bool): out.append(f"main_topic is {meta.get('main_topic')!r}, not true or false") if not nonempty_str(meta.get("url")): out.append("url is not a non-empty string") if meta.get("kind") == "release-notes" and not nonempty_str(meta.get("version")): out.append("release-notes has no version") if "version" in meta and not nonempty_str(meta["version"]): out.append("version is not a non-empty string") return out def index_problem(value: Any) -> str | None: if (isinstance(value, dict) and isinstance(value.get("pages"), int) and not isinstance(value["pages"], bool) and isinstance(value.get("sha256"), str) and HEX64.match(value["sha256"]) and nonempty_str(value.get("url"))): return None return f"index is {value!r}, not {{pages, sha256, url}}" def subset_of_problem(value: Any) -> str | None: if value is None: return None if (isinstance(value, dict) and isinstance(value.get("frozen_at"), str) and FROZEN_AT.match(value["frozen_at"]) and isinstance(value.get("manifest_sha256"), str) and HEX64.match(value["manifest_sha256"])): return None return f"subset_of is {value!r}, not null or {{frozen_at, manifest_sha256}}" def file_facts(data: bytes) -> tuple[dict[str, Any], str | None]: """sha256, bytes and lines of a source file, and the reason it cannot be a source, if any.""" facts: dict[str, Any] = {"bytes": len(data), "sha256": sha256(data)} try: text = data.decode("utf-8") except UnicodeDecodeError: return facts, "is not UTF-8" facts["lines"] = len(text.splitlines()) if ambiguous_lines(text): return facts, "has ambiguous lines (a line break other than \\n)" return facts, None def check(root: Path) -> list[str]: """K1 rule 11 and the form of the manifest; [] when the root is sound. RunError when check cannot run.""" doc = load_manifest(root) problems: list[str] = [] if not nonempty_str(doc.get("collection")): problems.append("collection is not a non-empty string") if not (isinstance(doc.get("frozen_at"), str) and FROZEN_AT.match(doc["frozen_at"])): problems.append(f"frozen_at is {doc.get('frozen_at')!r}, not YYYY-MM-DDTHH:MM:SSZ in UTC") if "index" in doc and (p := index_problem(doc["index"])): problems.append(p) if "folder" in doc and not (nonempty_str(doc["folder"]) and Path(doc["folder"]).is_absolute()): problems.append(f"folder is {doc['folder']!r}, not the absolute path of the folder the copy was read from") if "subset_of" not in doc: problems.append("subset_of is missing (null for a full copy)") elif p := subset_of_problem(doc["subset_of"]): problems.append(p) listed: set[str] = set() paths: list[str] = [] for n, e in enumerate(doc["sources"], 1): if not isinstance(e, dict) or not isinstance(e.get("path"), str): problems.append(f"sources entry {n} has no path") continue path = e["path"] paths.append(path) if not PATH.match(path): problems.append(f"{path}: path is not kilde/ followed by [a-z0-9._-]") continue if path in listed: problems.append(f"{path}: listed more than once") continue listed.add(path) problems += [f"{path}: {p}" for p in describe_problems(e)] f = root / path if not f.is_file() or f.is_symlink(): problems.append(f"{path}: missing") continue facts, why = file_facts(f.read_bytes()) for key in ("sha256", "bytes", "lines"): if key in facts and e.get(key) != facts[key]: problems.append(f"{path}: {key} is {facts[key]!r} on disk, {e.get(key)!r} in the manifest") if why: problems.append(f"{path}: {why}") if "places" in e and not (isinstance(e["places"], list) and len(e["places"]) == facts.get("lines")): problems.append(f"{path}: places do not name one place per line ({facts.get('lines')} lines)") if "original_copy" in e: copy = e["original_copy"] c = root / copy if isinstance(copy, str) and re.fullmatch(rf"{ORIGINAL}/[a-z0-9._-]+", copy) else None if c is None: problems.append(f"{path}: original_copy {copy!r} is not {ORIGINAL}/ followed by [a-z0-9._-]") elif not c.is_file() or c.is_symlink(): problems.append(f"{copy}: missing") elif sha256(c.read_bytes()) != e.get("original_sha256"): problems.append(f"{copy}: sha256 differs from original_sha256") if paths != sorted(paths): problems.append("sources are not sorted on path") kilde = root / KILDE if kilde.is_dir(): for f in sorted(kilde.rglob("*")): rel = f.relative_to(root).as_posix() if (f.is_file() or f.is_symlink()) and rel not in listed: problems.append(f"{rel}: under kilde/ but not listed") return problems def write_root(out: Path, files: dict[str, bytes], doc: dict[str, Any], others: dict[str, bytes] | None = None) -> None: """Write a root in a temporary folder beside OUT and move it into place, so a failure leaves no half root. files go under kilde/; others (relative path -> bytes) beside it, such as the original copies under original/.""" if out.exists(): raise RunError(f"{out} exists; refusing to overwrite") out.parent.mkdir(parents=True, exist_ok=True) tmp = Path(tempfile.mkdtemp(prefix=f".{out.name}-", dir=out.parent)) try: (tmp / KILDE).mkdir() for name, data in files.items(): p = tmp / KILDE / name p.write_bytes(data) os.chmod(p, READ_ONLY) for rel, data in (others or {}).items(): p = tmp / rel p.parent.mkdir(parents=True, exist_ok=True) p.write_bytes(data) os.chmod(p, READ_ONLY) (tmp / MANIFEST).write_text(dump(doc), encoding="utf-8") problems = check(tmp) if problems: raise Refused(problems) tmp.rename(out) except BaseException: shutil.rmtree(tmp, ignore_errors=True) raise def checked_frozen_at(value: str | None) -> str: frozen_at = value or utc_now() if not FROZEN_AT.match(frozen_at): raise RunError(f"--frozen-at {frozen_at!r} is not YYYY-MM-DDTHH:MM:SSZ") return frozen_at def freeze(staging: Path, meta_path: Path, out: Path, frozen_at: str | None = None) -> dict[str, Any]: when = checked_frozen_at(frozen_at) if out.exists(): raise RunError(f"{out} exists; refusing to overwrite") if not staging.is_dir(): raise RunError(f"{staging} is not a directory") meta = load_json(meta_path) if not isinstance(meta, dict) or meta.get("schema") != META_SCHEMA or not isinstance(meta.get("sources"), dict): raise RunError(f"{meta_path} is not a {META_SCHEMA} file with an object of sources") described: dict[str, Any] = meta["sources"] problems: list[str] = [] if not nonempty_str(meta.get("collection")): problems.append("meta: collection is not a non-empty string") if "index" in meta and (p := index_problem(meta["index"])): problems.append(f"meta: {p}") on_disk = {p.name: p for p in staging.iterdir()} for name in sorted(set(on_disk) - set(described)): problems.append(f"{name}: in the staging folder but not in the meta") for name in sorted(set(described) - set(on_disk)): problems.append(f"{name}: in the meta but not in the staging folder") files: dict[str, bytes] = {} entries = [] for name in sorted(set(on_disk) & set(described)): if not NAME.match(name) or not on_disk[name].is_file() or on_disk[name].is_symlink(): problems.append(f"{name}: not a plain file named [a-z0-9._-]") continue problems += [f"{name}: {p}" for p in describe_problems(described[name])] data = on_disk[name].read_bytes() facts, why = file_facts(data) if why: problems.append(f"{name}: {why}") continue files[name] = data entries.append({**described[name], **facts, "path": f"{KILDE}/{name}"}) if problems: raise Refused(problems) doc: dict[str, Any] = {"collection": meta["collection"], "frozen_at": when, "schema": SCHEMA, "sources": entries, "subset_of": None} if "index" in meta: doc["index"] = meta["index"] write_root(out, files, doc) return doc TEXT_READERS = (".md", ".markdown", ".txt") READERS = TEXT_READERS + readers.TYPES # Every line break str.splitlines() knows besides \n; a source keeps \n only, so line N is the same for every reader. LINE_BREAKS = re.compile("\r\n|[\r\x0b\x0c\x1c\x1d\x1e\x85\u2028\u2029]") FOLDER_KIND = "document" ORIGINAL = "original" # beside kilde/: the PDF and Office files as they were, so the view can open them at the place def safe_name(rel: str) -> str: """The kilde/ name of a file at rel (a path with "/") in a folder: see the folder command.""" parts = [re.sub(r"-+", "-", re.sub(r"[^a-z0-9._-]", "-", ascii_fold(part))).strip("-") or "x" for part in rel.split("/")] return "__".join(parts) def normalised_text(data: bytes) -> tuple[bytes, list[str]]: """The bytes of a text file as a source (UTF-8, lines broken at \n only) and what was changed to get them.""" changed: list[str] = [] if data.startswith(b"\xef\xbb\xbf"): data, changed = data[3:], ["bom"] try: text = data.decode("utf-8") except UnicodeDecodeError: try: text = data.decode("cp1252") except UnicodeDecodeError: text = data.decode("latin-1") changed.append("cp1252") flat = LINE_BREAKS.sub("\n", text) if flat != text: changed.append("line-breaks") return flat.encode("utf-8"), changed def walk(folder: Path) -> tuple[list[tuple[str, Path]], list[str]]: """(every file under folder that has a reader, as (relative path in NFC, path), sorted on the relative path), and one line per thing skipped with its reason.""" found: list[tuple[str, Path]] = [] skipped: list[str] = [] def visit(d: Path) -> None: for p in sorted(d.iterdir()): rel = unicodedata.normalize("NFC", p.relative_to(folder).as_posix()) if p.name.startswith("."): skipped.append(f"{rel}: skipped, hidden") elif p.is_symlink(): skipped.append(f"{rel}: skipped, a link") elif p.is_dir(): visit(p) elif not p.is_file(): skipped.append(f"{rel}: skipped, not a plain file") elif p.suffix.lower() not in READERS: skipped.append(f"{rel}: skipped, no reader for {p.suffix.lower() or 'a file without a suffix'}") else: found.append((rel, p)) visit(folder) return sorted(found), sorted(skipped) def freeze_folder(folder: Path, collection: str, out: Path, frozen_at: str | None = None) -> tuple[dict[str, Any], list[str]]: """The folder command: (the manifest written, the lines of what was skipped).""" when = checked_frozen_at(frozen_at) if out.exists(): raise RunError(f"{out} exists; refusing to overwrite") if not folder.is_dir(): raise RunError(f"{folder} is not a directory") if not nonempty_str(collection): raise RunError("the collection has no name") found, skipped = walk(folder) files: dict[str, bytes] = {} entries: list[dict[str, Any]] = [] origin: dict[str, list[str]] = {} missing: list[str] = [] originals: dict[str, bytes] = {} for rel, path in found: raw = path.read_bytes() read: dict[str, Any] = {} if path.suffix.lower() in readers.TYPES: try: rendering = readers.render(path) except readers.NoReader as e: missing.append(str(e)) continue except readers.Unreadable as e: skipped.append(f"{rel}: skipped, unreadable: {e}") continue if not rendering.text.strip(): skipped.append(f"{rel}: skipped, no text (a scanned page needs text recognition first)" if path.suffix.lower() == ".pdf" else f"{rel}: skipped, no text") continue if rendering.left_out: skipped.append(f"{rel}: {rendering.left_out} hidden parts left out") data, name = rendering.text.encode("utf-8"), safe_name(rel) + ".md" changed: list[str] = [] copy = f"{ORIGINAL}/{safe_name(rel)}" originals[copy] = raw read = {"left_out": rendering.left_out, "original_copy": copy, "original_sha256": sha256(raw), "places": rendering.locators, "reader": path.suffix.lower()[1:]} else: data, changed = normalised_text(raw) if not data.strip(): skipped.append(f"{rel}: skipped, empty") continue name = safe_name(rel) if RELEASE.match(name): # a folder's document has one main topic, never a release note's name (B13) name = name[:-len(".md")] + "-doc.md" origin.setdefault(name, []).append(rel) files[name] = data facts, _ = file_facts(data) lang = language(data.decode("utf-8")) entries.append({"kind": FOLDER_KIND, "main_topic": True, "normalised": changed, "original": rel, "url": path.resolve().as_uri(), **facts, **read, **({"lang": lang} if lang else {}), "path": f"{KILDE}/{name}"}) problems = sorted(set(missing)) + [f"{name}: the safe name of {' and '.join(rels)}" for name, rels in sorted(origin.items()) if len(rels) > 1] if problems: raise Refused(problems) if not entries: raise RunError(f"{folder} holds no document with a reader ({', '.join(READERS)})") entries.sort(key=lambda e: str(e["path"])) doc: dict[str, Any] = {"collection": collection, "folder": str(folder.resolve()), "frozen_at": when, "schema": SCHEMA, "sources": entries, "subset_of": None} write_root(out, files, doc, originals) return doc, sorted(skipped) def subset(root: Path, paths_file: Path, out: Path, frozen_at: str | None = None) -> dict[str, Any]: when = checked_frozen_at(frozen_at) if out.exists(): raise RunError(f"{out} exists; refusing to overwrite") problems = check(root) if problems: raise RunError(f"{root} fails check: {'; '.join(problems)}") text = paths_file.read_text(encoding="utf-8") wanted = text.split("\n") if wanted[-1] == "": wanted.pop() if not wanted: raise RunError(f"{paths_file} lists no path") if "" in wanted or wanted != sorted(set(wanted)): raise RunError(f"{paths_file} is not one path per line, sorted, without duplicates or empty lines") manifest_bytes = (root / MANIFEST).read_bytes() doc: dict[str, Any] = json.loads(manifest_bytes) by_path = {e["path"]: e for e in doc["sources"]} unknown = [p for p in wanted if p not in by_path] if unknown: raise RunError(f"not in {root / MANIFEST}: {', '.join(unknown)}") part = {**doc, "frozen_at": when, "sources": [by_path[p] for p in wanted], "subset_of": {"frozen_at": doc["frozen_at"], "manifest_sha256": sha256(manifest_bytes)}} write_root(out, {p[len(KILDE) + 1:]: (root / p).read_bytes() for p in wanted}, part, {e["original_copy"]: (root / e["original_copy"]).read_bytes() for e in part["sources"] if "original_copy" in e}) return part def sound_manifest(root: Path) -> tuple[dict[str, Any], dict[str, str]]: """The manifest of a root that passes check, and its stamp (frozen_at and the sha256 of the manifest's bytes).""" problems = check(root) if problems: raise RunError(f"{root} fails check: {'; '.join(problems)}") data = (root / MANIFEST).read_bytes() doc: dict[str, Any] = json.loads(data) return doc, {"frozen_at": doc["frozen_at"], "manifest_sha256": sha256(data)} def diff(old: Path, new: Path) -> dict[str, Any]: """K4 item 1: the sources of NEW that are new, changed or gone since OLD, each list sorted on path.""" old_doc, old_stamp = sound_manifest(old) new_doc, new_stamp = sound_manifest(new) if old_doc["collection"] != new_doc["collection"]: raise RunError(f"{old} is collection {old_doc['collection']!r} and {new} is {new_doc['collection']!r}; " "the paths of two collections can only match by chance") before = {e["path"]: e for e in old_doc["sources"]} after = {e["path"]: e for e in new_doc["sources"]} changed: list[dict[str, Any]] = [] gone: list[dict[str, Any]] = [] new_sources: list[dict[str, Any]] = [] unchanged = 0 for path in sorted(before.keys() | after.keys()): a, b = before.get(path), after.get(path) if b is None: assert a is not None gone.append({"kind": a["kind"], **as_known(a), "path": path}) elif a is None: new_sources.append({"kind": b["kind"], **as_known(b), "path": path}) elif a["sha256"] != b["sha256"] or a["kind"] != b["kind"]: changed.append({"kind": b["kind"], "new_sha256": b["sha256"], "old_sha256": a["sha256"], **as_known(b), "path": path}) else: unchanged += 1 return {"changed": changed, "gone": gone, "new": new_stamp, "new_sources": new_sources, "old": old_stamp, "schema": DIFF_SCHEMA, "unchanged": unchanged} def as_known(entry: dict[str, Any]) -> dict[str, Any]: """The file as the user knows it ({"original": path in the folder}) for a source read from a folder, else {}.""" return {"original": entry["original"]} if isinstance(entry.get("original"), str) else {} def has_change(report: dict[str, Any]) -> bool: return bool(report["changed"] or report["gone"] or report["new_sources"]) def read_diff(path: Path) -> dict[str, str]: """The changed and gone paths of a diff file, by path: "changed" or "gone". RunError when it is not a K4 diff.""" doc = load_json(path) if not isinstance(doc, dict) or doc.get("schema") != DIFF_SCHEMA: raise RunError(f"{path} is not a {DIFF_SCHEMA} file") out: dict[str, str] = {} for change in ("changed", "gone"): entries = doc.get(change) if not isinstance(entries, list) or not all(isinstance(e, dict) and isinstance(e.get("path"), str) for e in entries): raise RunError(f"{path}: {change} is not a list of entries with a path") out.update({e["path"]: change for e in entries}) return out def page_resources(path: Path, rel: str) -> list[str]: """Every sources[].resource string in the frontmatter of one page; [] for a page without frontmatter.""" lines = path.read_text(encoding="utf-8").split("\n") if lines[0].strip() != "---": return [] end = next((i for i in range(1, len(lines)) if lines[i].strip() == "---"), None) if end is None: raise RunError(f"{rel}: the frontmatter is not closed with ---") try: data = yaml.safe_load("\n".join(lines[1:end])) except yaml.YAMLError as exc: raise RunError(f"{rel}: the frontmatter does not parse: {str(exc).splitlines()[0]}") from None if data is None: return [] if not isinstance(data, dict): raise RunError(f"{rel}: the frontmatter is not a mapping") cited = data.get("sources") or [] if not isinstance(cited, list): raise RunError(f"{rel}: sources is not a list") return [s["resource"] for s in cited if isinstance(s, dict) and isinstance(s.get("resource"), str)] def marks(collection: Path, diff_path: Path) -> dict[str, Any]: """K4 item 2: the pages built on a changed or gone source, sorted on page, their sources sorted on path.""" changes = read_diff(diff_path) if not collection.is_dir(): raise RunError(f"{collection} is not a directory") pages: list[dict[str, Any]] = [] for p in sorted(collection.rglob("*.md")): if not p.is_file(): continue rel = p.relative_to(collection).as_posix() hit = {r[1:] for r in page_resources(p, rel) if r.startswith("/") and r[1:] in changes} if hit: pages.append({"page": rel, "sources": [{"change": changes[s], "path": s} for s in sorted(hit)]}) pages.sort(key=lambda e: e["page"]) return {"pages": pages, "schema": MARKS_SCHEMA} WATCH_SCHEMA = "okf-wiki/folder-watch/1" WATCH_STATE = "watch-state.json" LATEST = "latest" def source_folder(doc: dict[str, Any]) -> Path: """The folder a copy was read from: its `folder`, or for a copy frozen before that field, the one folder its entries name (each `url` less its `original` path). RunError when the copy was not read from a folder.""" if nonempty_str(doc.get("folder")): return Path(doc["folder"]) found: set[Path] = set() for e in doc["sources"]: url, original = e.get("url"), e.get("original") if not (isinstance(url, str) and url.startswith("file://") and nonempty_str(original)): continue path = Path(urllib.parse.unquote(urllib.parse.urlparse(url).path)) depth = len(Path(original).parts) if len(path.parts) <= depth: continue found.add(Path(*path.parts[:-depth])) if not found: raise RunError("the copy was not read from a folder (no folder, and no entry with a file:// url and an " "original path); only a folder copy can be read again") if len(found) > 1: raise RunError(f"the entries name more than one folder ({', '.join(sorted(map(str, found)))}); " "read the folder again with the folder command") return found.pop() def sound_root(root: Path) -> dict[str, Any]: problems = check(root) if problems: raise RunError(f"{root} fails check: {'; '.join(problems[:5])}") return load_manifest(root) def reread(root: Path, out: Path, frozen_at: str | None = None) -> tuple[dict[str, Any], list[str], Path]: """The reread command: the folder ROOT was read from, read again as the same collection into OUT.""" doc = sound_root(root) folder = source_folder(doc) if not folder.is_dir(): raise RunError(f"the folder {folder} that {root} was read from is not there") try: new, skipped = freeze_folder(folder, doc["collection"], out, frozen_at) except Refused as exc: raise RunError(f"{folder} cannot be read again: {'; '.join(exc.problems)}") from exc return new, skipped, folder def watch_lines(report: dict[str, Any]) -> list[str]: """One line per difference, "change path [new sha256]", so a later change of the same file is a new line.""" lines = [f"new {e['path']}" for e in report["new_sources"]] lines += [f"changed {e['path']} {e['new_sha256']}" for e in report["changed"]] lines += [f"gone {e['path']}" for e in report["gone"]] return sorted(lines) def watch(root: Path, into: Path, bundle: Path | None, frozen_at: str | None = None) -> tuple[str, bool]: """The watch command: (the report, whether something is new since the last run). See the module doc.""" doc = sound_root(root) folder = source_folder(doc).resolve() target = into.resolve() if target == folder or folder in target.parents: raise RunError(f"{into} is inside the folder {folder}; the next reading would take the watch's files for " "documents. Put it beside the folder") state_path = into / WATCH_STATE state = load_json(state_path) if state_path.is_file() else None if state is not None and not (isinstance(state, dict) and state.get("schema") == WATCH_SCHEMA and isinstance(state.get("entries"), list)): raise RunError(f"{state_path} is not a watch state ({WATCH_SCHEMA})") into.mkdir(parents=True, exist_ok=True) work = Path(tempfile.mkdtemp(prefix=".watch-", dir=into)) try: _, skipped, _ = reread(root, work / LATEST, frozen_at) report = diff(root, work / LATEST) (work / LATEST / "diff.json").write_text(dump(report), encoding="utf-8") marked: dict[str, int] = {} pages = 0 if bundle is not None: found = marks(bundle, work / LATEST / "diff.json") (work / LATEST / "marks.json").write_text(dump(found), encoding="utf-8") pages = len(found["pages"]) for pg in found["pages"]: for src in pg["sources"]: marked[src["path"]] = marked.get(src["path"], 0) + 1 latest = into / LATEST if latest.exists(): latest.rename(work / "replaced") (work / LATEST).rename(latest) finally: shutil.rmtree(work, ignore_errors=True) entries = watch_lines(report) same_copy = state is not None and state.get("frozen") == report["old"] seen = {str(e) for e in state["entries"]} if state is not None and same_copy else set() news = [e for e in entries if e not in seen] resolved = sorted(seen - set(entries)) state_path.write_text(dump({"entries": entries, "frozen": report["old"], "schema": WATCH_SCHEMA}), encoding="utf-8") by_path = {e["path"]: e for k in ("new_sources", "changed", "gone") for e in report[k]} counts = (f"{len(report['new_sources'])} new, {len(report['changed'])} changed, {len(report['gone'])} gone, " f"{report['unchanged']} unchanged") lines = [f"watch: {folder} read again into {latest}, against {root} (frozen {report['old']['frozen_at']}): " f"{counts}"] if not news: lines.append(" nothing differs" if not entries else " nothing new since the last run") else: lines.append("New since the last run:" if seen or same_copy else "Every difference (first run against this copy):") for e in news: change, path = e.split(" ")[:2] known = by_path[path].get("original", path) n = marked.get(path) lines.append(f" {change:<8} {known} ({path})" + (f", {n} page(s) marked" if n else "")) if resolved: lines.append("No longer differs from the frozen copy:") lines += [f" {e.split(' ')[1]}" for e in resolved] if skipped: lines.append("Skipped in the folder:") lines += [f" {line}" for line in skipped] if bundle is not None: lines.append(f"{pages} page(s) marked in {bundle}; the list: {latest / 'marks.json'}") lines.append(f"Every difference: {latest / 'diff.json'}. This only warns; rebuild from {latest} to update.") return "\n".join(lines) + "\n", bool(news) def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) sub = p.add_subparsers(dest="command", required=True) c = sub.add_parser("check") c.add_argument("root") f = sub.add_parser("freeze") f.add_argument("staging") f.add_argument("--meta", required=True) f.add_argument("--out", required=True) f.add_argument("--frozen-at") fo = sub.add_parser("folder") fo.add_argument("folder") fo.add_argument("--collection", required=True) fo.add_argument("--out", required=True) fo.add_argument("--frozen-at") s = sub.add_parser("subset") s.add_argument("root") s.add_argument("--paths", required=True) s.add_argument("--out", required=True) s.add_argument("--frozen-at") d = sub.add_parser("diff") d.add_argument("old") d.add_argument("new") m = sub.add_parser("marks") m.add_argument("collection") m.add_argument("--diff", required=True) r = sub.add_parser("reread") r.add_argument("root") r.add_argument("--out", required=True) r.add_argument("--frozen-at") w = sub.add_parser("watch") w.add_argument("root") w.add_argument("--into", required=True) w.add_argument("--bundle") w.add_argument("--frozen-at") return p def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) try: if args.command == "check": root = Path(args.root) problems = check(root) n = len(load_manifest(root)["sources"]) if problems: print(f"check: {len(problems)} problems in {root} ({n} sources listed)") for p in problems: print(f" {p}") return 1 print(f"check: {n} sources in {root}, every file present and listed") return 0 if args.command == "diff": report = diff(Path(args.old), Path(args.new)) sys.stdout.write(dump(report)) return 1 if has_change(report) else 0 if args.command == "marks": marked = marks(Path(args.collection), Path(args.diff)) sys.stdout.write(dump(marked)) return 1 if marked["pages"] else 0 if args.command == "reread": new, skipped, folder = reread(Path(args.root), Path(args.out), args.frozen_at) print(f"reread: {len(new['sources'])} sources from {folder} -> {args.out}; {len(skipped)} skipped") for line in skipped: print(f" {line}") return 0 if args.command == "watch": text, news = watch(Path(args.root), Path(args.into), Path(args.bundle) if args.bundle else None, args.frozen_at) sys.stdout.write(text) return 1 if news else 0 if args.command == "folder": doc, skipped = freeze_folder(Path(args.folder), args.collection, Path(args.out), args.frozen_at) print(f"folder: {len(doc['sources'])} sources -> {args.out}; {len(skipped)} skipped") for line in skipped: print(f" {line}") return 0 if args.command == "freeze": doc = freeze(Path(args.staging), Path(args.meta), Path(args.out), args.frozen_at) else: doc = subset(Path(args.root), Path(args.paths), Path(args.out), args.frozen_at) print(f"{args.command}: {len(doc['sources'])} sources -> {args.out}") return 0 except Refused as exc: print(f"{args.command} refused: {len(exc.problems)} problems, nothing written", file=sys.stderr) for p in exc.problems: print(f" {p}", file=sys.stderr) return 1 except (RunError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: print(f"{args.command} cannot run: {exc}", file=sys.stderr) return 2 if __name__ == "__main__": sys.exit(main())