okf-wiki/okfwiki/sources.py
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

830 lines
40 KiB
Python

"""The source manifest of a frozen copy (contract K1): check, freeze and subset; the change warning (contract K4): diff
and marks. No network, no model.
Usage:
python3 -m okfwiki.sources check ROOT
python3 -m okfwiki.sources freeze STAGING --meta FILE --out ROOT [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
python3 -m okfwiki.sources folder DIR --collection NAME --out ROOT [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
python3 -m okfwiki.sources subset ROOT --paths FILE --out ROOT2 [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
python3 -m okfwiki.sources diff OLD NEW
python3 -m okfwiki.sources marks COLLECTION --diff FILE
python3 -m okfwiki.sources reread ROOT --out ROOT2 [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
python3 -m okfwiki.sources watch ROOT --into DIR [--bundle COLLECTION] [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
A frozen copy is a root folder with source-manifest.json and the source files under kilde/. A page cites a source as
sources[].resource "/" + path.
check Exit 0 when every listed file exists with the same sha256, bytes and lines, every file under kilde/ is
listed, and the manifest has the form of K1; exit 1 with the problems listed; exit 2 when it cannot run (no
manifest, not JSON, another schema, sources not a list).
freeze Copies the files of STAGING (flat, named as the part of path after kilde/) to ROOT/kilde/, makes them read
only and writes the manifest. The meta file says per file what freeze cannot know:
{"collection": "...", "index": {...} (optional), "schema": "okf-wiki/source-meta/1",
"sources": {"<name>": {"kind": "...", "main_topic": true|false, "url": "...", "version": "..."}}}
Refused with exit 1 and the files listed: a file the meta lacks or a meta entry without a file, an invalid
entry, a file that is not UTF-8, and a file whose lines differ between splitlines() and split("\\n") (the
auditor uses the one, the trial code the other, so line N must be the same for both). Exit 2 when ROOT exists.
folder Freezes a folder of documents as it is, with no staging and no hand-written meta: every file under DIR, in
subfolders too, whose suffix has a reader (.md, .markdown, .txt, and .pdf, .docx, .xlsx, .pptx through
okfwiki.readers). A PDF or Office file is frozen as the text a reader of it sees (its safe name plus ".md"),
with the file itself copied read-only to original/ (`original_copy`), `reader` (the type), `original_sha256`,
`places` (the page, paragraph, cell or slide of each line, null
for a blank line or a heading) and `left_out` (hidden parts left out, also reported); every document records
`lang` (common.language) when its words tell it, for the view; one that cannot be read
or has no text is skipped and named, and a PDF without pypdf at the pinned version is refused with exit 1
and the install command. Each gets a safe name under kilde/: its path
relative to DIR with "/" as "__", letters spelled in ASCII (æ ø å as ae oe aa, accents dropped), lower case,
any other character as "-"; a name that would read as a release note (release-vX.Y.Z.md) ends in "-doc.md"
instead, since the release-note rules go by the name. The entry keeps the path as it was (`original`, NFC), the file's own URL (`url`,
file:///) and what was changed to make it a source (`normalised`, [] when the copy is byte for byte): "bom" (a
UTF-8 byte order mark dropped), "cp1252" (decoded as Windows-1252, the Western Latin-1 superset, when not
UTF-8) and "line-breaks" (\r\n, \r, form feed and every other line break written as \n), so a quote
stands verbatim in the frozen copy. Kind "document", one main topic. A file is skipped, and named with the
reason, when it is hidden (a name starting with "."; a hidden folder is skipped whole), has no reader, is
empty or is a link. Two files with one safe name are refused with exit 1; exit 2 when no file is left, or DIR
is not a folder, or ROOT exists. The manifest names the folder read (`folder`, its resolved path), so the copy
can be read again from itself.
subset A part of ROOT with the listed paths (one kilde/... per line, sorted, no duplicates), its own frozen_at, the
index unchanged and subset_of pointing at ROOT's frozen_at and the sha256 of ROOT's manifest. Exit 2 when ROOT
fails check, the paths file is not as described, or ROOT2 exists.
diff Which sources of NEW are new, changed or gone since OLD, by path. Changed = another sha256 or another kind (the
kind decides what a source is to the build). Exit 0 = no change, 1 = at least one, 2 = cannot run (a root
fails check, or the two roots name another collection).
marks The pages of COLLECTION (every .md under it, named by its path with "/") that cite a changed or gone source
of the diff as sources[].resource "/" + path. New sources mark nothing. A page without frontmatter cites
nothing; one whose frontmatter cannot be read stops marks. Exit 0 = no page marked, 1 = at least one, 2 =
cannot run.
Both print their report with the one written form, so two runs are byte-identical. A source read from a folder
is named in diff with its `original` path too, as the user knows the file.
reread Reads the folder ROOT was read from again, as the same collection, into ROOT2 (the folder command's rules). The
folder is the manifest's `folder`, or for a copy frozen before that field, the one folder its entries name
(each url less its original path). Exit 2 when ROOT fails check, was not read from a folder, its entries name
two folders, the folder is gone, or ROOT2 exists.
watch The change warning for a folder, one command for any scheduler (motor B15): reads the folder of ROOT again into
DIR/latest (replaced whole each run), writes diff.json there and, with --bundle, marks.json, and prints each
difference by its original path with the pages marked on it. DIR/watch-state.json keeps the lines reported,
so a run says only what is new since the last one (a new frozen copy starts over). Exit 0 = nothing new,
1 = something new, 2 = cannot run (also DIR inside the watched folder: the next reading would take the
watch's files for documents). Only a warning: the pages are rebuilt from DIR/latest in a session.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import shutil
import sys
import tempfile
import unicodedata
import urllib.parse
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import yaml
from okfwiki import readers
from okfwiki.common import RunError, ascii_fold, language
from okfwiki.kblocks import RELEASE
SCHEMA = "okf-wiki/source-manifest/1"
META_SCHEMA = "okf-wiki/source-meta/1"
DIFF_SCHEMA = "okf-wiki/source-diff/1"
MARKS_SCHEMA = "okf-wiki/source-marks/1"
MANIFEST = "source-manifest.json"
KILDE = "kilde"
PATH = re.compile(r"^kilde/[a-z0-9._-]+$")
NAME = re.compile(r"^[a-z0-9._-]+$")
HEX64 = re.compile(r"^[0-9a-f]{64}$")
FROZEN_AT = re.compile(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$")
READ_ONLY = 0o444
class Refused(Exception):
"""The input breaks a rule; nothing was written. A command reports the problems with exit 1."""
def __init__(self, problems: list[str]) -> None:
super().__init__("; ".join(problems))
self.problems = problems
def dump(data: Any) -> str:
"""The one written form of every file here, so two writings of the same content are byte-identical."""
return json.dumps(data, indent=1, sort_keys=True, ensure_ascii=False) + "\n"
def sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def ambiguous_lines(text: str) -> bool:
"""True when splitlines() and split("\\n") (less a final empty part) give different lines."""
parts = text.split("\n")
if parts[-1] == "":
parts.pop()
return parts != text.splitlines()
def utc_now() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def load_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except FileNotFoundError:
raise RunError(f"{path} does not exist") from None
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
raise RunError(f"{path} is not JSON: {exc}") from None
def load_manifest(root: Path) -> dict[str, Any]:
"""The manifest of ROOT, or RunError when there is none that check can read."""
if not root.is_dir():
raise RunError(f"{root} is not a directory")
doc = load_json(root / MANIFEST)
if not isinstance(doc, dict):
raise RunError(f"{root / MANIFEST} is not a JSON object")
if doc.get("schema") != SCHEMA:
raise RunError(f"{root / MANIFEST} has schema {doc.get('schema')!r}, not {SCHEMA}")
if not isinstance(doc.get("sources"), list):
raise RunError(f"{root / MANIFEST} has no list of sources")
return doc
def nonempty_str(value: Any) -> bool:
return isinstance(value, str) and value != ""
def describe_problems(meta: Any) -> list[str]:
"""The rules for what the meta says about one source (K1 rules 3, 4, 6)."""
if not isinstance(meta, dict):
return ["is not an object"]
out = []
if not nonempty_str(meta.get("kind")):
out.append("kind is not a non-empty string")
if not isinstance(meta.get("main_topic"), bool):
out.append(f"main_topic is {meta.get('main_topic')!r}, not true or false")
if not nonempty_str(meta.get("url")):
out.append("url is not a non-empty string")
if meta.get("kind") == "release-notes" and not nonempty_str(meta.get("version")):
out.append("release-notes has no version")
if "version" in meta and not nonempty_str(meta["version"]):
out.append("version is not a non-empty string")
return out
def index_problem(value: Any) -> str | None:
if (isinstance(value, dict) and isinstance(value.get("pages"), int) and not isinstance(value["pages"], bool)
and isinstance(value.get("sha256"), str) and HEX64.match(value["sha256"]) and nonempty_str(value.get("url"))):
return None
return f"index is {value!r}, not {{pages, sha256, url}}"
def subset_of_problem(value: Any) -> str | None:
if value is None:
return None
if (isinstance(value, dict) and isinstance(value.get("frozen_at"), str) and FROZEN_AT.match(value["frozen_at"])
and isinstance(value.get("manifest_sha256"), str) and HEX64.match(value["manifest_sha256"])):
return None
return f"subset_of is {value!r}, not null or {{frozen_at, manifest_sha256}}"
def file_facts(data: bytes) -> tuple[dict[str, Any], str | None]:
"""sha256, bytes and lines of a source file, and the reason it cannot be a source, if any."""
facts: dict[str, Any] = {"bytes": len(data), "sha256": sha256(data)}
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
return facts, "is not UTF-8"
facts["lines"] = len(text.splitlines())
if ambiguous_lines(text):
return facts, "has ambiguous lines (a line break other than \\n)"
return facts, None
def check(root: Path) -> list[str]:
"""K1 rule 11 and the form of the manifest; [] when the root is sound. RunError when check cannot run."""
doc = load_manifest(root)
problems: list[str] = []
if not nonempty_str(doc.get("collection")):
problems.append("collection is not a non-empty string")
if not (isinstance(doc.get("frozen_at"), str) and FROZEN_AT.match(doc["frozen_at"])):
problems.append(f"frozen_at is {doc.get('frozen_at')!r}, not YYYY-MM-DDTHH:MM:SSZ in UTC")
if "index" in doc and (p := index_problem(doc["index"])):
problems.append(p)
if "folder" in doc and not (nonempty_str(doc["folder"]) and Path(doc["folder"]).is_absolute()):
problems.append(f"folder is {doc['folder']!r}, not the absolute path of the folder the copy was read from")
if "subset_of" not in doc:
problems.append("subset_of is missing (null for a full copy)")
elif p := subset_of_problem(doc["subset_of"]):
problems.append(p)
listed: set[str] = set()
paths: list[str] = []
for n, e in enumerate(doc["sources"], 1):
if not isinstance(e, dict) or not isinstance(e.get("path"), str):
problems.append(f"sources entry {n} has no path")
continue
path = e["path"]
paths.append(path)
if not PATH.match(path):
problems.append(f"{path}: path is not kilde/ followed by [a-z0-9._-]")
continue
if path in listed:
problems.append(f"{path}: listed more than once")
continue
listed.add(path)
problems += [f"{path}: {p}" for p in describe_problems(e)]
f = root / path
if not f.is_file() or f.is_symlink():
problems.append(f"{path}: missing")
continue
facts, why = file_facts(f.read_bytes())
for key in ("sha256", "bytes", "lines"):
if key in facts and e.get(key) != facts[key]:
problems.append(f"{path}: {key} is {facts[key]!r} on disk, {e.get(key)!r} in the manifest")
if why:
problems.append(f"{path}: {why}")
if "places" in e and not (isinstance(e["places"], list) and len(e["places"]) == facts.get("lines")):
problems.append(f"{path}: places do not name one place per line ({facts.get('lines')} lines)")
if "original_copy" in e:
copy = e["original_copy"]
c = root / copy if isinstance(copy, str) and re.fullmatch(rf"{ORIGINAL}/[a-z0-9._-]+", copy) else None
if c is None:
problems.append(f"{path}: original_copy {copy!r} is not {ORIGINAL}/ followed by [a-z0-9._-]")
elif not c.is_file() or c.is_symlink():
problems.append(f"{copy}: missing")
elif sha256(c.read_bytes()) != e.get("original_sha256"):
problems.append(f"{copy}: sha256 differs from original_sha256")
if paths != sorted(paths):
problems.append("sources are not sorted on path")
kilde = root / KILDE
if kilde.is_dir():
for f in sorted(kilde.rglob("*")):
rel = f.relative_to(root).as_posix()
if (f.is_file() or f.is_symlink()) and rel not in listed:
problems.append(f"{rel}: under kilde/ but not listed")
return problems
def write_root(out: Path, files: dict[str, bytes], doc: dict[str, Any], others: dict[str, bytes] | None = None) -> None:
"""Write a root in a temporary folder beside OUT and move it into place, so a failure leaves no half root. files
go under kilde/; others (relative path -> bytes) beside it, such as the original copies under original/."""
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
out.parent.mkdir(parents=True, exist_ok=True)
tmp = Path(tempfile.mkdtemp(prefix=f".{out.name}-", dir=out.parent))
try:
(tmp / KILDE).mkdir()
for name, data in files.items():
p = tmp / KILDE / name
p.write_bytes(data)
os.chmod(p, READ_ONLY)
for rel, data in (others or {}).items():
p = tmp / rel
p.parent.mkdir(parents=True, exist_ok=True)
p.write_bytes(data)
os.chmod(p, READ_ONLY)
(tmp / MANIFEST).write_text(dump(doc), encoding="utf-8")
problems = check(tmp)
if problems:
raise Refused(problems)
tmp.rename(out)
except BaseException:
shutil.rmtree(tmp, ignore_errors=True)
raise
def checked_frozen_at(value: str | None) -> str:
frozen_at = value or utc_now()
if not FROZEN_AT.match(frozen_at):
raise RunError(f"--frozen-at {frozen_at!r} is not YYYY-MM-DDTHH:MM:SSZ")
return frozen_at
def freeze(staging: Path, meta_path: Path, out: Path, frozen_at: str | None = None) -> dict[str, Any]:
when = checked_frozen_at(frozen_at)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
if not staging.is_dir():
raise RunError(f"{staging} is not a directory")
meta = load_json(meta_path)
if not isinstance(meta, dict) or meta.get("schema") != META_SCHEMA or not isinstance(meta.get("sources"), dict):
raise RunError(f"{meta_path} is not a {META_SCHEMA} file with an object of sources")
described: dict[str, Any] = meta["sources"]
problems: list[str] = []
if not nonempty_str(meta.get("collection")):
problems.append("meta: collection is not a non-empty string")
if "index" in meta and (p := index_problem(meta["index"])):
problems.append(f"meta: {p}")
on_disk = {p.name: p for p in staging.iterdir()}
for name in sorted(set(on_disk) - set(described)):
problems.append(f"{name}: in the staging folder but not in the meta")
for name in sorted(set(described) - set(on_disk)):
problems.append(f"{name}: in the meta but not in the staging folder")
files: dict[str, bytes] = {}
entries = []
for name in sorted(set(on_disk) & set(described)):
if not NAME.match(name) or not on_disk[name].is_file() or on_disk[name].is_symlink():
problems.append(f"{name}: not a plain file named [a-z0-9._-]")
continue
problems += [f"{name}: {p}" for p in describe_problems(described[name])]
data = on_disk[name].read_bytes()
facts, why = file_facts(data)
if why:
problems.append(f"{name}: {why}")
continue
files[name] = data
entries.append({**described[name], **facts, "path": f"{KILDE}/{name}"})
if problems:
raise Refused(problems)
doc: dict[str, Any] = {"collection": meta["collection"], "frozen_at": when, "schema": SCHEMA,
"sources": entries, "subset_of": None}
if "index" in meta:
doc["index"] = meta["index"]
write_root(out, files, doc)
return doc
TEXT_READERS = (".md", ".markdown", ".txt")
READERS = TEXT_READERS + readers.TYPES
# Every line break str.splitlines() knows besides \n; a source keeps \n only, so line N is the same for every reader.
LINE_BREAKS = re.compile("\r\n|[\r\x0b\x0c\x1c\x1d\x1e\x85\u2028\u2029]")
FOLDER_KIND = "document"
ORIGINAL = "original" # beside kilde/: the PDF and Office files as they were, so the view can open them at the place
def safe_name(rel: str) -> str:
"""The kilde/ name of a file at rel (a path with "/") in a folder: see the folder command."""
parts = [re.sub(r"-+", "-", re.sub(r"[^a-z0-9._-]", "-", ascii_fold(part))).strip("-") or "x"
for part in rel.split("/")]
return "__".join(parts)
def normalised_text(data: bytes) -> tuple[bytes, list[str]]:
"""The bytes of a text file as a source (UTF-8, lines broken at \n only) and what was changed to get them."""
changed: list[str] = []
if data.startswith(b"\xef\xbb\xbf"):
data, changed = data[3:], ["bom"]
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
try:
text = data.decode("cp1252")
except UnicodeDecodeError:
text = data.decode("latin-1")
changed.append("cp1252")
flat = LINE_BREAKS.sub("\n", text)
if flat != text:
changed.append("line-breaks")
return flat.encode("utf-8"), changed
def walk(folder: Path) -> tuple[list[tuple[str, Path]], list[str]]:
"""(every file under folder that has a reader, as (relative path in NFC, path), sorted on the relative path), and
one line per thing skipped with its reason."""
found: list[tuple[str, Path]] = []
skipped: list[str] = []
def visit(d: Path) -> None:
for p in sorted(d.iterdir()):
rel = unicodedata.normalize("NFC", p.relative_to(folder).as_posix())
if p.name.startswith("."):
skipped.append(f"{rel}: skipped, hidden")
elif p.is_symlink():
skipped.append(f"{rel}: skipped, a link")
elif p.is_dir():
visit(p)
elif not p.is_file():
skipped.append(f"{rel}: skipped, not a plain file")
elif p.suffix.lower() not in READERS:
skipped.append(f"{rel}: skipped, no reader for {p.suffix.lower() or 'a file without a suffix'}")
else:
found.append((rel, p))
visit(folder)
return sorted(found), sorted(skipped)
def freeze_folder(folder: Path, collection: str, out: Path,
frozen_at: str | None = None) -> tuple[dict[str, Any], list[str]]:
"""The folder command: (the manifest written, the lines of what was skipped)."""
when = checked_frozen_at(frozen_at)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
if not folder.is_dir():
raise RunError(f"{folder} is not a directory")
if not nonempty_str(collection):
raise RunError("the collection has no name")
found, skipped = walk(folder)
files: dict[str, bytes] = {}
entries: list[dict[str, Any]] = []
origin: dict[str, list[str]] = {}
missing: list[str] = []
originals: dict[str, bytes] = {}
for rel, path in found:
raw = path.read_bytes()
read: dict[str, Any] = {}
if path.suffix.lower() in readers.TYPES:
try:
rendering = readers.render(path)
except readers.NoReader as e:
missing.append(str(e))
continue
except readers.Unreadable as e:
skipped.append(f"{rel}: skipped, unreadable: {e}")
continue
if not rendering.text.strip():
skipped.append(f"{rel}: skipped, no text (a scanned page needs text recognition first)"
if path.suffix.lower() == ".pdf" else f"{rel}: skipped, no text")
continue
if rendering.left_out:
skipped.append(f"{rel}: {rendering.left_out} hidden parts left out")
data, name = rendering.text.encode("utf-8"), safe_name(rel) + ".md"
changed: list[str] = []
copy = f"{ORIGINAL}/{safe_name(rel)}"
originals[copy] = raw
read = {"left_out": rendering.left_out, "original_copy": copy, "original_sha256": sha256(raw),
"places": rendering.locators, "reader": path.suffix.lower()[1:]}
else:
data, changed = normalised_text(raw)
if not data.strip():
skipped.append(f"{rel}: skipped, empty")
continue
name = safe_name(rel)
if RELEASE.match(name): # a folder's document has one main topic, never a release note's name (B13)
name = name[:-len(".md")] + "-doc.md"
origin.setdefault(name, []).append(rel)
files[name] = data
facts, _ = file_facts(data)
lang = language(data.decode("utf-8"))
entries.append({"kind": FOLDER_KIND, "main_topic": True, "normalised": changed, "original": rel,
"url": path.resolve().as_uri(), **facts, **read, **({"lang": lang} if lang else {}),
"path": f"{KILDE}/{name}"})
problems = sorted(set(missing)) + [f"{name}: the safe name of {' and '.join(rels)}"
for name, rels in sorted(origin.items()) if len(rels) > 1]
if problems:
raise Refused(problems)
if not entries:
raise RunError(f"{folder} holds no document with a reader ({', '.join(READERS)})")
entries.sort(key=lambda e: str(e["path"]))
doc: dict[str, Any] = {"collection": collection, "folder": str(folder.resolve()), "frozen_at": when,
"schema": SCHEMA, "sources": entries, "subset_of": None}
write_root(out, files, doc, originals)
return doc, sorted(skipped)
def subset(root: Path, paths_file: Path, out: Path, frozen_at: str | None = None) -> dict[str, Any]:
when = checked_frozen_at(frozen_at)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
problems = check(root)
if problems:
raise RunError(f"{root} fails check: {'; '.join(problems)}")
text = paths_file.read_text(encoding="utf-8")
wanted = text.split("\n")
if wanted[-1] == "":
wanted.pop()
if not wanted:
raise RunError(f"{paths_file} lists no path")
if "" in wanted or wanted != sorted(set(wanted)):
raise RunError(f"{paths_file} is not one path per line, sorted, without duplicates or empty lines")
manifest_bytes = (root / MANIFEST).read_bytes()
doc: dict[str, Any] = json.loads(manifest_bytes)
by_path = {e["path"]: e for e in doc["sources"]}
unknown = [p for p in wanted if p not in by_path]
if unknown:
raise RunError(f"not in {root / MANIFEST}: {', '.join(unknown)}")
part = {**doc, "frozen_at": when, "sources": [by_path[p] for p in wanted],
"subset_of": {"frozen_at": doc["frozen_at"], "manifest_sha256": sha256(manifest_bytes)}}
write_root(out, {p[len(KILDE) + 1:]: (root / p).read_bytes() for p in wanted}, part,
{e["original_copy"]: (root / e["original_copy"]).read_bytes() for e in part["sources"]
if "original_copy" in e})
return part
def sound_manifest(root: Path) -> tuple[dict[str, Any], dict[str, str]]:
"""The manifest of a root that passes check, and its stamp (frozen_at and the sha256 of the manifest's bytes)."""
problems = check(root)
if problems:
raise RunError(f"{root} fails check: {'; '.join(problems)}")
data = (root / MANIFEST).read_bytes()
doc: dict[str, Any] = json.loads(data)
return doc, {"frozen_at": doc["frozen_at"], "manifest_sha256": sha256(data)}
def diff(old: Path, new: Path) -> dict[str, Any]:
"""K4 item 1: the sources of NEW that are new, changed or gone since OLD, each list sorted on path."""
old_doc, old_stamp = sound_manifest(old)
new_doc, new_stamp = sound_manifest(new)
if old_doc["collection"] != new_doc["collection"]:
raise RunError(f"{old} is collection {old_doc['collection']!r} and {new} is {new_doc['collection']!r}; "
"the paths of two collections can only match by chance")
before = {e["path"]: e for e in old_doc["sources"]}
after = {e["path"]: e for e in new_doc["sources"]}
changed: list[dict[str, Any]] = []
gone: list[dict[str, Any]] = []
new_sources: list[dict[str, Any]] = []
unchanged = 0
for path in sorted(before.keys() | after.keys()):
a, b = before.get(path), after.get(path)
if b is None:
assert a is not None
gone.append({"kind": a["kind"], **as_known(a), "path": path})
elif a is None:
new_sources.append({"kind": b["kind"], **as_known(b), "path": path})
elif a["sha256"] != b["sha256"] or a["kind"] != b["kind"]:
changed.append({"kind": b["kind"], "new_sha256": b["sha256"], "old_sha256": a["sha256"], **as_known(b),
"path": path})
else:
unchanged += 1
return {"changed": changed, "gone": gone, "new": new_stamp, "new_sources": new_sources, "old": old_stamp,
"schema": DIFF_SCHEMA, "unchanged": unchanged}
def as_known(entry: dict[str, Any]) -> dict[str, Any]:
"""The file as the user knows it ({"original": path in the folder}) for a source read from a folder, else {}."""
return {"original": entry["original"]} if isinstance(entry.get("original"), str) else {}
def has_change(report: dict[str, Any]) -> bool:
return bool(report["changed"] or report["gone"] or report["new_sources"])
def read_diff(path: Path) -> dict[str, str]:
"""The changed and gone paths of a diff file, by path: "changed" or "gone". RunError when it is not a K4 diff."""
doc = load_json(path)
if not isinstance(doc, dict) or doc.get("schema") != DIFF_SCHEMA:
raise RunError(f"{path} is not a {DIFF_SCHEMA} file")
out: dict[str, str] = {}
for change in ("changed", "gone"):
entries = doc.get(change)
if not isinstance(entries, list) or not all(isinstance(e, dict) and isinstance(e.get("path"), str)
for e in entries):
raise RunError(f"{path}: {change} is not a list of entries with a path")
out.update({e["path"]: change for e in entries})
return out
def page_resources(path: Path, rel: str) -> list[str]:
"""Every sources[].resource string in the frontmatter of one page; [] for a page without frontmatter."""
lines = path.read_text(encoding="utf-8").split("\n")
if lines[0].strip() != "---":
return []
end = next((i for i in range(1, len(lines)) if lines[i].strip() == "---"), None)
if end is None:
raise RunError(f"{rel}: the frontmatter is not closed with ---")
try:
data = yaml.safe_load("\n".join(lines[1:end]))
except yaml.YAMLError as exc:
raise RunError(f"{rel}: the frontmatter does not parse: {str(exc).splitlines()[0]}") from None
if data is None:
return []
if not isinstance(data, dict):
raise RunError(f"{rel}: the frontmatter is not a mapping")
cited = data.get("sources") or []
if not isinstance(cited, list):
raise RunError(f"{rel}: sources is not a list")
return [s["resource"] for s in cited if isinstance(s, dict) and isinstance(s.get("resource"), str)]
def marks(collection: Path, diff_path: Path) -> dict[str, Any]:
"""K4 item 2: the pages built on a changed or gone source, sorted on page, their sources sorted on path."""
changes = read_diff(diff_path)
if not collection.is_dir():
raise RunError(f"{collection} is not a directory")
pages: list[dict[str, Any]] = []
for p in sorted(collection.rglob("*.md")):
if not p.is_file():
continue
rel = p.relative_to(collection).as_posix()
hit = {r[1:] for r in page_resources(p, rel) if r.startswith("/") and r[1:] in changes}
if hit:
pages.append({"page": rel, "sources": [{"change": changes[s], "path": s} for s in sorted(hit)]})
pages.sort(key=lambda e: e["page"])
return {"pages": pages, "schema": MARKS_SCHEMA}
WATCH_SCHEMA = "okf-wiki/folder-watch/1"
WATCH_STATE = "watch-state.json"
LATEST = "latest"
def source_folder(doc: dict[str, Any]) -> Path:
"""The folder a copy was read from: its `folder`, or for a copy frozen before that field, the one folder its
entries name (each `url` less its `original` path). RunError when the copy was not read from a folder."""
if nonempty_str(doc.get("folder")):
return Path(doc["folder"])
found: set[Path] = set()
for e in doc["sources"]:
url, original = e.get("url"), e.get("original")
if not (isinstance(url, str) and url.startswith("file://") and nonempty_str(original)):
continue
path = Path(urllib.parse.unquote(urllib.parse.urlparse(url).path))
depth = len(Path(original).parts)
if len(path.parts) <= depth:
continue
found.add(Path(*path.parts[:-depth]))
if not found:
raise RunError("the copy was not read from a folder (no folder, and no entry with a file:// url and an "
"original path); only a folder copy can be read again")
if len(found) > 1:
raise RunError(f"the entries name more than one folder ({', '.join(sorted(map(str, found)))}); "
"read the folder again with the folder command")
return found.pop()
def sound_root(root: Path) -> dict[str, Any]:
problems = check(root)
if problems:
raise RunError(f"{root} fails check: {'; '.join(problems[:5])}")
return load_manifest(root)
def reread(root: Path, out: Path, frozen_at: str | None = None) -> tuple[dict[str, Any], list[str], Path]:
"""The reread command: the folder ROOT was read from, read again as the same collection into OUT."""
doc = sound_root(root)
folder = source_folder(doc)
if not folder.is_dir():
raise RunError(f"the folder {folder} that {root} was read from is not there")
try:
new, skipped = freeze_folder(folder, doc["collection"], out, frozen_at)
except Refused as exc:
raise RunError(f"{folder} cannot be read again: {'; '.join(exc.problems)}") from exc
return new, skipped, folder
def watch_lines(report: dict[str, Any]) -> list[str]:
"""One line per difference, "change path [new sha256]", so a later change of the same file is a new line."""
lines = [f"new {e['path']}" for e in report["new_sources"]]
lines += [f"changed {e['path']} {e['new_sha256']}" for e in report["changed"]]
lines += [f"gone {e['path']}" for e in report["gone"]]
return sorted(lines)
def watch(root: Path, into: Path, bundle: Path | None, frozen_at: str | None = None) -> tuple[str, bool]:
"""The watch command: (the report, whether something is new since the last run). See the module doc."""
doc = sound_root(root)
folder = source_folder(doc).resolve()
target = into.resolve()
if target == folder or folder in target.parents:
raise RunError(f"{into} is inside the folder {folder}; the next reading would take the watch's files for "
"documents. Put it beside the folder")
state_path = into / WATCH_STATE
state = load_json(state_path) if state_path.is_file() else None
if state is not None and not (isinstance(state, dict) and state.get("schema") == WATCH_SCHEMA
and isinstance(state.get("entries"), list)):
raise RunError(f"{state_path} is not a watch state ({WATCH_SCHEMA})")
into.mkdir(parents=True, exist_ok=True)
work = Path(tempfile.mkdtemp(prefix=".watch-", dir=into))
try:
_, skipped, _ = reread(root, work / LATEST, frozen_at)
report = diff(root, work / LATEST)
(work / LATEST / "diff.json").write_text(dump(report), encoding="utf-8")
marked: dict[str, int] = {}
pages = 0
if bundle is not None:
found = marks(bundle, work / LATEST / "diff.json")
(work / LATEST / "marks.json").write_text(dump(found), encoding="utf-8")
pages = len(found["pages"])
for pg in found["pages"]:
for src in pg["sources"]:
marked[src["path"]] = marked.get(src["path"], 0) + 1
latest = into / LATEST
if latest.exists():
latest.rename(work / "replaced")
(work / LATEST).rename(latest)
finally:
shutil.rmtree(work, ignore_errors=True)
entries = watch_lines(report)
same_copy = state is not None and state.get("frozen") == report["old"]
seen = {str(e) for e in state["entries"]} if state is not None and same_copy else set()
news = [e for e in entries if e not in seen]
resolved = sorted(seen - set(entries))
state_path.write_text(dump({"entries": entries, "frozen": report["old"], "schema": WATCH_SCHEMA}),
encoding="utf-8")
by_path = {e["path"]: e for k in ("new_sources", "changed", "gone") for e in report[k]}
counts = (f"{len(report['new_sources'])} new, {len(report['changed'])} changed, {len(report['gone'])} gone, "
f"{report['unchanged']} unchanged")
lines = [f"watch: {folder} read again into {latest}, against {root} (frozen {report['old']['frozen_at']}): "
f"{counts}"]
if not news:
lines.append(" nothing differs" if not entries else " nothing new since the last run")
else:
lines.append("New since the last run:" if seen or same_copy else "Every difference (first run against this copy):")
for e in news:
change, path = e.split(" ")[:2]
known = by_path[path].get("original", path)
n = marked.get(path)
lines.append(f" {change:<8} {known} ({path})" + (f", {n} page(s) marked" if n else ""))
if resolved:
lines.append("No longer differs from the frozen copy:")
lines += [f" {e.split(' ')[1]}" for e in resolved]
if skipped:
lines.append("Skipped in the folder:")
lines += [f" {line}" for line in skipped]
if bundle is not None:
lines.append(f"{pages} page(s) marked in {bundle}; the list: {latest / 'marks.json'}")
lines.append(f"Every difference: {latest / 'diff.json'}. This only warns; rebuild from {latest} to update.")
return "\n".join(lines) + "\n", bool(news)
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
c = sub.add_parser("check")
c.add_argument("root")
f = sub.add_parser("freeze")
f.add_argument("staging")
f.add_argument("--meta", required=True)
f.add_argument("--out", required=True)
f.add_argument("--frozen-at")
fo = sub.add_parser("folder")
fo.add_argument("folder")
fo.add_argument("--collection", required=True)
fo.add_argument("--out", required=True)
fo.add_argument("--frozen-at")
s = sub.add_parser("subset")
s.add_argument("root")
s.add_argument("--paths", required=True)
s.add_argument("--out", required=True)
s.add_argument("--frozen-at")
d = sub.add_parser("diff")
d.add_argument("old")
d.add_argument("new")
m = sub.add_parser("marks")
m.add_argument("collection")
m.add_argument("--diff", required=True)
r = sub.add_parser("reread")
r.add_argument("root")
r.add_argument("--out", required=True)
r.add_argument("--frozen-at")
w = sub.add_parser("watch")
w.add_argument("root")
w.add_argument("--into", required=True)
w.add_argument("--bundle")
w.add_argument("--frozen-at")
return p
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "check":
root = Path(args.root)
problems = check(root)
n = len(load_manifest(root)["sources"])
if problems:
print(f"check: {len(problems)} problems in {root} ({n} sources listed)")
for p in problems:
print(f" {p}")
return 1
print(f"check: {n} sources in {root}, every file present and listed")
return 0
if args.command == "diff":
report = diff(Path(args.old), Path(args.new))
sys.stdout.write(dump(report))
return 1 if has_change(report) else 0
if args.command == "marks":
marked = marks(Path(args.collection), Path(args.diff))
sys.stdout.write(dump(marked))
return 1 if marked["pages"] else 0
if args.command == "reread":
new, skipped, folder = reread(Path(args.root), Path(args.out), args.frozen_at)
print(f"reread: {len(new['sources'])} sources from {folder} -> {args.out}; {len(skipped)} skipped")
for line in skipped:
print(f" {line}")
return 0
if args.command == "watch":
text, news = watch(Path(args.root), Path(args.into), Path(args.bundle) if args.bundle else None,
args.frozen_at)
sys.stdout.write(text)
return 1 if news else 0
if args.command == "folder":
doc, skipped = freeze_folder(Path(args.folder), args.collection, Path(args.out), args.frozen_at)
print(f"folder: {len(doc['sources'])} sources -> {args.out}; {len(skipped)} skipped")
for line in skipped:
print(f" {line}")
return 0
if args.command == "freeze":
doc = freeze(Path(args.staging), Path(args.meta), Path(args.out), args.frozen_at)
else:
doc = subset(Path(args.root), Path(args.paths), Path(args.out), args.frozen_at)
print(f"{args.command}: {len(doc['sources'])} sources -> {args.out}")
return 0
except Refused as exc:
print(f"{args.command} refused: {len(exc.problems)} problems, nothing written", file=sys.stderr)
for p in exc.problems:
print(f" {p}", file=sys.stderr)
return 1
except (RunError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
print(f"{args.command} cannot run: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())