First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
830 lines
40 KiB
Python
830 lines
40 KiB
Python
"""The source manifest of a frozen copy (contract K1): check, freeze and subset; the change warning (contract K4): diff
|
|
and marks. No network, no model.
|
|
|
|
Usage:
|
|
python3 -m okfwiki.sources check ROOT
|
|
python3 -m okfwiki.sources freeze STAGING --meta FILE --out ROOT [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
|
|
python3 -m okfwiki.sources folder DIR --collection NAME --out ROOT [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
|
|
python3 -m okfwiki.sources subset ROOT --paths FILE --out ROOT2 [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
|
|
python3 -m okfwiki.sources diff OLD NEW
|
|
python3 -m okfwiki.sources marks COLLECTION --diff FILE
|
|
python3 -m okfwiki.sources reread ROOT --out ROOT2 [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
|
|
python3 -m okfwiki.sources watch ROOT --into DIR [--bundle COLLECTION] [--frozen-at YYYY-MM-DDTHH:MM:SSZ]
|
|
|
|
A frozen copy is a root folder with source-manifest.json and the source files under kilde/. A page cites a source as
|
|
sources[].resource "/" + path.
|
|
|
|
check Exit 0 when every listed file exists with the same sha256, bytes and lines, every file under kilde/ is
|
|
listed, and the manifest has the form of K1; exit 1 with the problems listed; exit 2 when it cannot run (no
|
|
manifest, not JSON, another schema, sources not a list).
|
|
freeze Copies the files of STAGING (flat, named as the part of path after kilde/) to ROOT/kilde/, makes them read
|
|
only and writes the manifest. The meta file says per file what freeze cannot know:
|
|
{"collection": "...", "index": {...} (optional), "schema": "okf-wiki/source-meta/1",
|
|
"sources": {"<name>": {"kind": "...", "main_topic": true|false, "url": "...", "version": "..."}}}
|
|
Refused with exit 1 and the files listed: a file the meta lacks or a meta entry without a file, an invalid
|
|
entry, a file that is not UTF-8, and a file whose lines differ between splitlines() and split("\\n") (the
|
|
auditor uses the one, the trial code the other, so line N must be the same for both). Exit 2 when ROOT exists.
|
|
folder Freezes a folder of documents as it is, with no staging and no hand-written meta: every file under DIR, in
|
|
subfolders too, whose suffix has a reader (.md, .markdown, .txt, and .pdf, .docx, .xlsx, .pptx through
|
|
okfwiki.readers). A PDF or Office file is frozen as the text a reader of it sees (its safe name plus ".md"),
|
|
with the file itself copied read-only to original/ (`original_copy`), `reader` (the type), `original_sha256`,
|
|
`places` (the page, paragraph, cell or slide of each line, null
|
|
for a blank line or a heading) and `left_out` (hidden parts left out, also reported); every document records
|
|
`lang` (common.language) when its words tell it, for the view; one that cannot be read
|
|
or has no text is skipped and named, and a PDF without pypdf at the pinned version is refused with exit 1
|
|
and the install command. Each gets a safe name under kilde/: its path
|
|
relative to DIR with "/" as "__", letters spelled in ASCII (æ ø å as ae oe aa, accents dropped), lower case,
|
|
any other character as "-"; a name that would read as a release note (release-vX.Y.Z.md) ends in "-doc.md"
|
|
instead, since the release-note rules go by the name. The entry keeps the path as it was (`original`, NFC), the file's own URL (`url`,
|
|
file:///) and what was changed to make it a source (`normalised`, [] when the copy is byte for byte): "bom" (a
|
|
UTF-8 byte order mark dropped), "cp1252" (decoded as Windows-1252, the Western Latin-1 superset, when not
|
|
UTF-8) and "line-breaks" (\r\n, \r, form feed and every other line break written as \n), so a quote
|
|
stands verbatim in the frozen copy. Kind "document", one main topic. A file is skipped, and named with the
|
|
reason, when it is hidden (a name starting with "."; a hidden folder is skipped whole), has no reader, is
|
|
empty or is a link. Two files with one safe name are refused with exit 1; exit 2 when no file is left, or DIR
|
|
is not a folder, or ROOT exists. The manifest names the folder read (`folder`, its resolved path), so the copy
|
|
can be read again from itself.
|
|
subset A part of ROOT with the listed paths (one kilde/... per line, sorted, no duplicates), its own frozen_at, the
|
|
index unchanged and subset_of pointing at ROOT's frozen_at and the sha256 of ROOT's manifest. Exit 2 when ROOT
|
|
fails check, the paths file is not as described, or ROOT2 exists.
|
|
diff Which sources of NEW are new, changed or gone since OLD, by path. Changed = another sha256 or another kind (the
|
|
kind decides what a source is to the build). Exit 0 = no change, 1 = at least one, 2 = cannot run (a root
|
|
fails check, or the two roots name another collection).
|
|
marks The pages of COLLECTION (every .md under it, named by its path with "/") that cite a changed or gone source
|
|
of the diff as sources[].resource "/" + path. New sources mark nothing. A page without frontmatter cites
|
|
nothing; one whose frontmatter cannot be read stops marks. Exit 0 = no page marked, 1 = at least one, 2 =
|
|
cannot run.
|
|
Both print their report with the one written form, so two runs are byte-identical. A source read from a folder
|
|
is named in diff with its `original` path too, as the user knows the file.
|
|
reread Reads the folder ROOT was read from again, as the same collection, into ROOT2 (the folder command's rules). The
|
|
folder is the manifest's `folder`, or for a copy frozen before that field, the one folder its entries name
|
|
(each url less its original path). Exit 2 when ROOT fails check, was not read from a folder, its entries name
|
|
two folders, the folder is gone, or ROOT2 exists.
|
|
watch The change warning for a folder, one command for any scheduler (motor B15): reads the folder of ROOT again into
|
|
DIR/latest (replaced whole each run), writes diff.json there and, with --bundle, marks.json, and prints each
|
|
difference by its original path with the pages marked on it. DIR/watch-state.json keeps the lines reported,
|
|
so a run says only what is new since the last one (a new frozen copy starts over). Exit 0 = nothing new,
|
|
1 = something new, 2 = cannot run (also DIR inside the watched folder: the next reading would take the
|
|
watch's files for documents). Only a warning: the pages are rebuilt from DIR/latest in a session.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import sys
|
|
import tempfile
|
|
import unicodedata
|
|
import urllib.parse
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import yaml
|
|
|
|
from okfwiki import readers
|
|
from okfwiki.common import RunError, ascii_fold, language
|
|
from okfwiki.kblocks import RELEASE
|
|
|
|
SCHEMA = "okf-wiki/source-manifest/1"
|
|
META_SCHEMA = "okf-wiki/source-meta/1"
|
|
DIFF_SCHEMA = "okf-wiki/source-diff/1"
|
|
MARKS_SCHEMA = "okf-wiki/source-marks/1"
|
|
MANIFEST = "source-manifest.json"
|
|
KILDE = "kilde"
|
|
PATH = re.compile(r"^kilde/[a-z0-9._-]+$")
|
|
NAME = re.compile(r"^[a-z0-9._-]+$")
|
|
HEX64 = re.compile(r"^[0-9a-f]{64}$")
|
|
FROZEN_AT = re.compile(r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$")
|
|
READ_ONLY = 0o444
|
|
|
|
|
|
class Refused(Exception):
|
|
"""The input breaks a rule; nothing was written. A command reports the problems with exit 1."""
|
|
|
|
def __init__(self, problems: list[str]) -> None:
|
|
super().__init__("; ".join(problems))
|
|
self.problems = problems
|
|
|
|
|
|
def dump(data: Any) -> str:
|
|
"""The one written form of every file here, so two writings of the same content are byte-identical."""
|
|
return json.dumps(data, indent=1, sort_keys=True, ensure_ascii=False) + "\n"
|
|
|
|
|
|
def sha256(data: bytes) -> str:
|
|
return hashlib.sha256(data).hexdigest()
|
|
|
|
|
|
def ambiguous_lines(text: str) -> bool:
|
|
"""True when splitlines() and split("\\n") (less a final empty part) give different lines."""
|
|
parts = text.split("\n")
|
|
if parts[-1] == "":
|
|
parts.pop()
|
|
return parts != text.splitlines()
|
|
|
|
|
|
def utc_now() -> str:
|
|
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
|
|
|
|
def load_json(path: Path) -> Any:
|
|
try:
|
|
return json.loads(path.read_text(encoding="utf-8"))
|
|
except FileNotFoundError:
|
|
raise RunError(f"{path} does not exist") from None
|
|
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
|
|
raise RunError(f"{path} is not JSON: {exc}") from None
|
|
|
|
|
|
def load_manifest(root: Path) -> dict[str, Any]:
|
|
"""The manifest of ROOT, or RunError when there is none that check can read."""
|
|
if not root.is_dir():
|
|
raise RunError(f"{root} is not a directory")
|
|
doc = load_json(root / MANIFEST)
|
|
if not isinstance(doc, dict):
|
|
raise RunError(f"{root / MANIFEST} is not a JSON object")
|
|
if doc.get("schema") != SCHEMA:
|
|
raise RunError(f"{root / MANIFEST} has schema {doc.get('schema')!r}, not {SCHEMA}")
|
|
if not isinstance(doc.get("sources"), list):
|
|
raise RunError(f"{root / MANIFEST} has no list of sources")
|
|
return doc
|
|
|
|
|
|
def nonempty_str(value: Any) -> bool:
|
|
return isinstance(value, str) and value != ""
|
|
|
|
|
|
def describe_problems(meta: Any) -> list[str]:
|
|
"""The rules for what the meta says about one source (K1 rules 3, 4, 6)."""
|
|
if not isinstance(meta, dict):
|
|
return ["is not an object"]
|
|
out = []
|
|
if not nonempty_str(meta.get("kind")):
|
|
out.append("kind is not a non-empty string")
|
|
if not isinstance(meta.get("main_topic"), bool):
|
|
out.append(f"main_topic is {meta.get('main_topic')!r}, not true or false")
|
|
if not nonempty_str(meta.get("url")):
|
|
out.append("url is not a non-empty string")
|
|
if meta.get("kind") == "release-notes" and not nonempty_str(meta.get("version")):
|
|
out.append("release-notes has no version")
|
|
if "version" in meta and not nonempty_str(meta["version"]):
|
|
out.append("version is not a non-empty string")
|
|
return out
|
|
|
|
|
|
def index_problem(value: Any) -> str | None:
|
|
if (isinstance(value, dict) and isinstance(value.get("pages"), int) and not isinstance(value["pages"], bool)
|
|
and isinstance(value.get("sha256"), str) and HEX64.match(value["sha256"]) and nonempty_str(value.get("url"))):
|
|
return None
|
|
return f"index is {value!r}, not {{pages, sha256, url}}"
|
|
|
|
|
|
def subset_of_problem(value: Any) -> str | None:
|
|
if value is None:
|
|
return None
|
|
if (isinstance(value, dict) and isinstance(value.get("frozen_at"), str) and FROZEN_AT.match(value["frozen_at"])
|
|
and isinstance(value.get("manifest_sha256"), str) and HEX64.match(value["manifest_sha256"])):
|
|
return None
|
|
return f"subset_of is {value!r}, not null or {{frozen_at, manifest_sha256}}"
|
|
|
|
|
|
def file_facts(data: bytes) -> tuple[dict[str, Any], str | None]:
|
|
"""sha256, bytes and lines of a source file, and the reason it cannot be a source, if any."""
|
|
facts: dict[str, Any] = {"bytes": len(data), "sha256": sha256(data)}
|
|
try:
|
|
text = data.decode("utf-8")
|
|
except UnicodeDecodeError:
|
|
return facts, "is not UTF-8"
|
|
facts["lines"] = len(text.splitlines())
|
|
if ambiguous_lines(text):
|
|
return facts, "has ambiguous lines (a line break other than \\n)"
|
|
return facts, None
|
|
|
|
|
|
def check(root: Path) -> list[str]:
|
|
"""K1 rule 11 and the form of the manifest; [] when the root is sound. RunError when check cannot run."""
|
|
doc = load_manifest(root)
|
|
problems: list[str] = []
|
|
if not nonempty_str(doc.get("collection")):
|
|
problems.append("collection is not a non-empty string")
|
|
if not (isinstance(doc.get("frozen_at"), str) and FROZEN_AT.match(doc["frozen_at"])):
|
|
problems.append(f"frozen_at is {doc.get('frozen_at')!r}, not YYYY-MM-DDTHH:MM:SSZ in UTC")
|
|
if "index" in doc and (p := index_problem(doc["index"])):
|
|
problems.append(p)
|
|
if "folder" in doc and not (nonempty_str(doc["folder"]) and Path(doc["folder"]).is_absolute()):
|
|
problems.append(f"folder is {doc['folder']!r}, not the absolute path of the folder the copy was read from")
|
|
if "subset_of" not in doc:
|
|
problems.append("subset_of is missing (null for a full copy)")
|
|
elif p := subset_of_problem(doc["subset_of"]):
|
|
problems.append(p)
|
|
listed: set[str] = set()
|
|
paths: list[str] = []
|
|
for n, e in enumerate(doc["sources"], 1):
|
|
if not isinstance(e, dict) or not isinstance(e.get("path"), str):
|
|
problems.append(f"sources entry {n} has no path")
|
|
continue
|
|
path = e["path"]
|
|
paths.append(path)
|
|
if not PATH.match(path):
|
|
problems.append(f"{path}: path is not kilde/ followed by [a-z0-9._-]")
|
|
continue
|
|
if path in listed:
|
|
problems.append(f"{path}: listed more than once")
|
|
continue
|
|
listed.add(path)
|
|
problems += [f"{path}: {p}" for p in describe_problems(e)]
|
|
f = root / path
|
|
if not f.is_file() or f.is_symlink():
|
|
problems.append(f"{path}: missing")
|
|
continue
|
|
facts, why = file_facts(f.read_bytes())
|
|
for key in ("sha256", "bytes", "lines"):
|
|
if key in facts and e.get(key) != facts[key]:
|
|
problems.append(f"{path}: {key} is {facts[key]!r} on disk, {e.get(key)!r} in the manifest")
|
|
if why:
|
|
problems.append(f"{path}: {why}")
|
|
if "places" in e and not (isinstance(e["places"], list) and len(e["places"]) == facts.get("lines")):
|
|
problems.append(f"{path}: places do not name one place per line ({facts.get('lines')} lines)")
|
|
if "original_copy" in e:
|
|
copy = e["original_copy"]
|
|
c = root / copy if isinstance(copy, str) and re.fullmatch(rf"{ORIGINAL}/[a-z0-9._-]+", copy) else None
|
|
if c is None:
|
|
problems.append(f"{path}: original_copy {copy!r} is not {ORIGINAL}/ followed by [a-z0-9._-]")
|
|
elif not c.is_file() or c.is_symlink():
|
|
problems.append(f"{copy}: missing")
|
|
elif sha256(c.read_bytes()) != e.get("original_sha256"):
|
|
problems.append(f"{copy}: sha256 differs from original_sha256")
|
|
if paths != sorted(paths):
|
|
problems.append("sources are not sorted on path")
|
|
kilde = root / KILDE
|
|
if kilde.is_dir():
|
|
for f in sorted(kilde.rglob("*")):
|
|
rel = f.relative_to(root).as_posix()
|
|
if (f.is_file() or f.is_symlink()) and rel not in listed:
|
|
problems.append(f"{rel}: under kilde/ but not listed")
|
|
return problems
|
|
|
|
|
|
def write_root(out: Path, files: dict[str, bytes], doc: dict[str, Any], others: dict[str, bytes] | None = None) -> None:
|
|
"""Write a root in a temporary folder beside OUT and move it into place, so a failure leaves no half root. files
|
|
go under kilde/; others (relative path -> bytes) beside it, such as the original copies under original/."""
|
|
if out.exists():
|
|
raise RunError(f"{out} exists; refusing to overwrite")
|
|
out.parent.mkdir(parents=True, exist_ok=True)
|
|
tmp = Path(tempfile.mkdtemp(prefix=f".{out.name}-", dir=out.parent))
|
|
try:
|
|
(tmp / KILDE).mkdir()
|
|
for name, data in files.items():
|
|
p = tmp / KILDE / name
|
|
p.write_bytes(data)
|
|
os.chmod(p, READ_ONLY)
|
|
for rel, data in (others or {}).items():
|
|
p = tmp / rel
|
|
p.parent.mkdir(parents=True, exist_ok=True)
|
|
p.write_bytes(data)
|
|
os.chmod(p, READ_ONLY)
|
|
(tmp / MANIFEST).write_text(dump(doc), encoding="utf-8")
|
|
problems = check(tmp)
|
|
if problems:
|
|
raise Refused(problems)
|
|
tmp.rename(out)
|
|
except BaseException:
|
|
shutil.rmtree(tmp, ignore_errors=True)
|
|
raise
|
|
|
|
|
|
def checked_frozen_at(value: str | None) -> str:
|
|
frozen_at = value or utc_now()
|
|
if not FROZEN_AT.match(frozen_at):
|
|
raise RunError(f"--frozen-at {frozen_at!r} is not YYYY-MM-DDTHH:MM:SSZ")
|
|
return frozen_at
|
|
|
|
|
|
def freeze(staging: Path, meta_path: Path, out: Path, frozen_at: str | None = None) -> dict[str, Any]:
|
|
when = checked_frozen_at(frozen_at)
|
|
if out.exists():
|
|
raise RunError(f"{out} exists; refusing to overwrite")
|
|
if not staging.is_dir():
|
|
raise RunError(f"{staging} is not a directory")
|
|
meta = load_json(meta_path)
|
|
if not isinstance(meta, dict) or meta.get("schema") != META_SCHEMA or not isinstance(meta.get("sources"), dict):
|
|
raise RunError(f"{meta_path} is not a {META_SCHEMA} file with an object of sources")
|
|
described: dict[str, Any] = meta["sources"]
|
|
problems: list[str] = []
|
|
if not nonempty_str(meta.get("collection")):
|
|
problems.append("meta: collection is not a non-empty string")
|
|
if "index" in meta and (p := index_problem(meta["index"])):
|
|
problems.append(f"meta: {p}")
|
|
on_disk = {p.name: p for p in staging.iterdir()}
|
|
for name in sorted(set(on_disk) - set(described)):
|
|
problems.append(f"{name}: in the staging folder but not in the meta")
|
|
for name in sorted(set(described) - set(on_disk)):
|
|
problems.append(f"{name}: in the meta but not in the staging folder")
|
|
files: dict[str, bytes] = {}
|
|
entries = []
|
|
for name in sorted(set(on_disk) & set(described)):
|
|
if not NAME.match(name) or not on_disk[name].is_file() or on_disk[name].is_symlink():
|
|
problems.append(f"{name}: not a plain file named [a-z0-9._-]")
|
|
continue
|
|
problems += [f"{name}: {p}" for p in describe_problems(described[name])]
|
|
data = on_disk[name].read_bytes()
|
|
facts, why = file_facts(data)
|
|
if why:
|
|
problems.append(f"{name}: {why}")
|
|
continue
|
|
files[name] = data
|
|
entries.append({**described[name], **facts, "path": f"{KILDE}/{name}"})
|
|
if problems:
|
|
raise Refused(problems)
|
|
doc: dict[str, Any] = {"collection": meta["collection"], "frozen_at": when, "schema": SCHEMA,
|
|
"sources": entries, "subset_of": None}
|
|
if "index" in meta:
|
|
doc["index"] = meta["index"]
|
|
write_root(out, files, doc)
|
|
return doc
|
|
|
|
|
|
TEXT_READERS = (".md", ".markdown", ".txt")
|
|
READERS = TEXT_READERS + readers.TYPES
|
|
# Every line break str.splitlines() knows besides \n; a source keeps \n only, so line N is the same for every reader.
|
|
LINE_BREAKS = re.compile("\r\n|[\r\x0b\x0c\x1c\x1d\x1e\x85\u2028\u2029]")
|
|
FOLDER_KIND = "document"
|
|
ORIGINAL = "original" # beside kilde/: the PDF and Office files as they were, so the view can open them at the place
|
|
|
|
|
|
def safe_name(rel: str) -> str:
|
|
"""The kilde/ name of a file at rel (a path with "/") in a folder: see the folder command."""
|
|
parts = [re.sub(r"-+", "-", re.sub(r"[^a-z0-9._-]", "-", ascii_fold(part))).strip("-") or "x"
|
|
for part in rel.split("/")]
|
|
return "__".join(parts)
|
|
|
|
|
|
def normalised_text(data: bytes) -> tuple[bytes, list[str]]:
|
|
"""The bytes of a text file as a source (UTF-8, lines broken at \n only) and what was changed to get them."""
|
|
changed: list[str] = []
|
|
if data.startswith(b"\xef\xbb\xbf"):
|
|
data, changed = data[3:], ["bom"]
|
|
try:
|
|
text = data.decode("utf-8")
|
|
except UnicodeDecodeError:
|
|
try:
|
|
text = data.decode("cp1252")
|
|
except UnicodeDecodeError:
|
|
text = data.decode("latin-1")
|
|
changed.append("cp1252")
|
|
flat = LINE_BREAKS.sub("\n", text)
|
|
if flat != text:
|
|
changed.append("line-breaks")
|
|
return flat.encode("utf-8"), changed
|
|
|
|
|
|
def walk(folder: Path) -> tuple[list[tuple[str, Path]], list[str]]:
|
|
"""(every file under folder that has a reader, as (relative path in NFC, path), sorted on the relative path), and
|
|
one line per thing skipped with its reason."""
|
|
found: list[tuple[str, Path]] = []
|
|
skipped: list[str] = []
|
|
|
|
def visit(d: Path) -> None:
|
|
for p in sorted(d.iterdir()):
|
|
rel = unicodedata.normalize("NFC", p.relative_to(folder).as_posix())
|
|
if p.name.startswith("."):
|
|
skipped.append(f"{rel}: skipped, hidden")
|
|
elif p.is_symlink():
|
|
skipped.append(f"{rel}: skipped, a link")
|
|
elif p.is_dir():
|
|
visit(p)
|
|
elif not p.is_file():
|
|
skipped.append(f"{rel}: skipped, not a plain file")
|
|
elif p.suffix.lower() not in READERS:
|
|
skipped.append(f"{rel}: skipped, no reader for {p.suffix.lower() or 'a file without a suffix'}")
|
|
else:
|
|
found.append((rel, p))
|
|
|
|
visit(folder)
|
|
return sorted(found), sorted(skipped)
|
|
|
|
|
|
def freeze_folder(folder: Path, collection: str, out: Path,
|
|
frozen_at: str | None = None) -> tuple[dict[str, Any], list[str]]:
|
|
"""The folder command: (the manifest written, the lines of what was skipped)."""
|
|
when = checked_frozen_at(frozen_at)
|
|
if out.exists():
|
|
raise RunError(f"{out} exists; refusing to overwrite")
|
|
if not folder.is_dir():
|
|
raise RunError(f"{folder} is not a directory")
|
|
if not nonempty_str(collection):
|
|
raise RunError("the collection has no name")
|
|
found, skipped = walk(folder)
|
|
files: dict[str, bytes] = {}
|
|
entries: list[dict[str, Any]] = []
|
|
origin: dict[str, list[str]] = {}
|
|
missing: list[str] = []
|
|
originals: dict[str, bytes] = {}
|
|
for rel, path in found:
|
|
raw = path.read_bytes()
|
|
read: dict[str, Any] = {}
|
|
if path.suffix.lower() in readers.TYPES:
|
|
try:
|
|
rendering = readers.render(path)
|
|
except readers.NoReader as e:
|
|
missing.append(str(e))
|
|
continue
|
|
except readers.Unreadable as e:
|
|
skipped.append(f"{rel}: skipped, unreadable: {e}")
|
|
continue
|
|
if not rendering.text.strip():
|
|
skipped.append(f"{rel}: skipped, no text (a scanned page needs text recognition first)"
|
|
if path.suffix.lower() == ".pdf" else f"{rel}: skipped, no text")
|
|
continue
|
|
if rendering.left_out:
|
|
skipped.append(f"{rel}: {rendering.left_out} hidden parts left out")
|
|
data, name = rendering.text.encode("utf-8"), safe_name(rel) + ".md"
|
|
changed: list[str] = []
|
|
copy = f"{ORIGINAL}/{safe_name(rel)}"
|
|
originals[copy] = raw
|
|
read = {"left_out": rendering.left_out, "original_copy": copy, "original_sha256": sha256(raw),
|
|
"places": rendering.locators, "reader": path.suffix.lower()[1:]}
|
|
else:
|
|
data, changed = normalised_text(raw)
|
|
if not data.strip():
|
|
skipped.append(f"{rel}: skipped, empty")
|
|
continue
|
|
name = safe_name(rel)
|
|
if RELEASE.match(name): # a folder's document has one main topic, never a release note's name (B13)
|
|
name = name[:-len(".md")] + "-doc.md"
|
|
origin.setdefault(name, []).append(rel)
|
|
files[name] = data
|
|
facts, _ = file_facts(data)
|
|
lang = language(data.decode("utf-8"))
|
|
entries.append({"kind": FOLDER_KIND, "main_topic": True, "normalised": changed, "original": rel,
|
|
"url": path.resolve().as_uri(), **facts, **read, **({"lang": lang} if lang else {}),
|
|
"path": f"{KILDE}/{name}"})
|
|
problems = sorted(set(missing)) + [f"{name}: the safe name of {' and '.join(rels)}"
|
|
for name, rels in sorted(origin.items()) if len(rels) > 1]
|
|
if problems:
|
|
raise Refused(problems)
|
|
if not entries:
|
|
raise RunError(f"{folder} holds no document with a reader ({', '.join(READERS)})")
|
|
entries.sort(key=lambda e: str(e["path"]))
|
|
doc: dict[str, Any] = {"collection": collection, "folder": str(folder.resolve()), "frozen_at": when,
|
|
"schema": SCHEMA, "sources": entries, "subset_of": None}
|
|
write_root(out, files, doc, originals)
|
|
return doc, sorted(skipped)
|
|
|
|
|
|
def subset(root: Path, paths_file: Path, out: Path, frozen_at: str | None = None) -> dict[str, Any]:
|
|
when = checked_frozen_at(frozen_at)
|
|
if out.exists():
|
|
raise RunError(f"{out} exists; refusing to overwrite")
|
|
problems = check(root)
|
|
if problems:
|
|
raise RunError(f"{root} fails check: {'; '.join(problems)}")
|
|
text = paths_file.read_text(encoding="utf-8")
|
|
wanted = text.split("\n")
|
|
if wanted[-1] == "":
|
|
wanted.pop()
|
|
if not wanted:
|
|
raise RunError(f"{paths_file} lists no path")
|
|
if "" in wanted or wanted != sorted(set(wanted)):
|
|
raise RunError(f"{paths_file} is not one path per line, sorted, without duplicates or empty lines")
|
|
manifest_bytes = (root / MANIFEST).read_bytes()
|
|
doc: dict[str, Any] = json.loads(manifest_bytes)
|
|
by_path = {e["path"]: e for e in doc["sources"]}
|
|
unknown = [p for p in wanted if p not in by_path]
|
|
if unknown:
|
|
raise RunError(f"not in {root / MANIFEST}: {', '.join(unknown)}")
|
|
part = {**doc, "frozen_at": when, "sources": [by_path[p] for p in wanted],
|
|
"subset_of": {"frozen_at": doc["frozen_at"], "manifest_sha256": sha256(manifest_bytes)}}
|
|
write_root(out, {p[len(KILDE) + 1:]: (root / p).read_bytes() for p in wanted}, part,
|
|
{e["original_copy"]: (root / e["original_copy"]).read_bytes() for e in part["sources"]
|
|
if "original_copy" in e})
|
|
return part
|
|
|
|
|
|
def sound_manifest(root: Path) -> tuple[dict[str, Any], dict[str, str]]:
|
|
"""The manifest of a root that passes check, and its stamp (frozen_at and the sha256 of the manifest's bytes)."""
|
|
problems = check(root)
|
|
if problems:
|
|
raise RunError(f"{root} fails check: {'; '.join(problems)}")
|
|
data = (root / MANIFEST).read_bytes()
|
|
doc: dict[str, Any] = json.loads(data)
|
|
return doc, {"frozen_at": doc["frozen_at"], "manifest_sha256": sha256(data)}
|
|
|
|
|
|
def diff(old: Path, new: Path) -> dict[str, Any]:
|
|
"""K4 item 1: the sources of NEW that are new, changed or gone since OLD, each list sorted on path."""
|
|
old_doc, old_stamp = sound_manifest(old)
|
|
new_doc, new_stamp = sound_manifest(new)
|
|
if old_doc["collection"] != new_doc["collection"]:
|
|
raise RunError(f"{old} is collection {old_doc['collection']!r} and {new} is {new_doc['collection']!r}; "
|
|
"the paths of two collections can only match by chance")
|
|
before = {e["path"]: e for e in old_doc["sources"]}
|
|
after = {e["path"]: e for e in new_doc["sources"]}
|
|
changed: list[dict[str, Any]] = []
|
|
gone: list[dict[str, Any]] = []
|
|
new_sources: list[dict[str, Any]] = []
|
|
unchanged = 0
|
|
for path in sorted(before.keys() | after.keys()):
|
|
a, b = before.get(path), after.get(path)
|
|
if b is None:
|
|
assert a is not None
|
|
gone.append({"kind": a["kind"], **as_known(a), "path": path})
|
|
elif a is None:
|
|
new_sources.append({"kind": b["kind"], **as_known(b), "path": path})
|
|
elif a["sha256"] != b["sha256"] or a["kind"] != b["kind"]:
|
|
changed.append({"kind": b["kind"], "new_sha256": b["sha256"], "old_sha256": a["sha256"], **as_known(b),
|
|
"path": path})
|
|
else:
|
|
unchanged += 1
|
|
return {"changed": changed, "gone": gone, "new": new_stamp, "new_sources": new_sources, "old": old_stamp,
|
|
"schema": DIFF_SCHEMA, "unchanged": unchanged}
|
|
|
|
|
|
def as_known(entry: dict[str, Any]) -> dict[str, Any]:
|
|
"""The file as the user knows it ({"original": path in the folder}) for a source read from a folder, else {}."""
|
|
return {"original": entry["original"]} if isinstance(entry.get("original"), str) else {}
|
|
|
|
|
|
def has_change(report: dict[str, Any]) -> bool:
|
|
return bool(report["changed"] or report["gone"] or report["new_sources"])
|
|
|
|
|
|
def read_diff(path: Path) -> dict[str, str]:
|
|
"""The changed and gone paths of a diff file, by path: "changed" or "gone". RunError when it is not a K4 diff."""
|
|
doc = load_json(path)
|
|
if not isinstance(doc, dict) or doc.get("schema") != DIFF_SCHEMA:
|
|
raise RunError(f"{path} is not a {DIFF_SCHEMA} file")
|
|
out: dict[str, str] = {}
|
|
for change in ("changed", "gone"):
|
|
entries = doc.get(change)
|
|
if not isinstance(entries, list) or not all(isinstance(e, dict) and isinstance(e.get("path"), str)
|
|
for e in entries):
|
|
raise RunError(f"{path}: {change} is not a list of entries with a path")
|
|
out.update({e["path"]: change for e in entries})
|
|
return out
|
|
|
|
|
|
def page_resources(path: Path, rel: str) -> list[str]:
|
|
"""Every sources[].resource string in the frontmatter of one page; [] for a page without frontmatter."""
|
|
lines = path.read_text(encoding="utf-8").split("\n")
|
|
if lines[0].strip() != "---":
|
|
return []
|
|
end = next((i for i in range(1, len(lines)) if lines[i].strip() == "---"), None)
|
|
if end is None:
|
|
raise RunError(f"{rel}: the frontmatter is not closed with ---")
|
|
try:
|
|
data = yaml.safe_load("\n".join(lines[1:end]))
|
|
except yaml.YAMLError as exc:
|
|
raise RunError(f"{rel}: the frontmatter does not parse: {str(exc).splitlines()[0]}") from None
|
|
if data is None:
|
|
return []
|
|
if not isinstance(data, dict):
|
|
raise RunError(f"{rel}: the frontmatter is not a mapping")
|
|
cited = data.get("sources") or []
|
|
if not isinstance(cited, list):
|
|
raise RunError(f"{rel}: sources is not a list")
|
|
return [s["resource"] for s in cited if isinstance(s, dict) and isinstance(s.get("resource"), str)]
|
|
|
|
|
|
def marks(collection: Path, diff_path: Path) -> dict[str, Any]:
|
|
"""K4 item 2: the pages built on a changed or gone source, sorted on page, their sources sorted on path."""
|
|
changes = read_diff(diff_path)
|
|
if not collection.is_dir():
|
|
raise RunError(f"{collection} is not a directory")
|
|
pages: list[dict[str, Any]] = []
|
|
for p in sorted(collection.rglob("*.md")):
|
|
if not p.is_file():
|
|
continue
|
|
rel = p.relative_to(collection).as_posix()
|
|
hit = {r[1:] for r in page_resources(p, rel) if r.startswith("/") and r[1:] in changes}
|
|
if hit:
|
|
pages.append({"page": rel, "sources": [{"change": changes[s], "path": s} for s in sorted(hit)]})
|
|
pages.sort(key=lambda e: e["page"])
|
|
return {"pages": pages, "schema": MARKS_SCHEMA}
|
|
|
|
|
|
WATCH_SCHEMA = "okf-wiki/folder-watch/1"
|
|
WATCH_STATE = "watch-state.json"
|
|
LATEST = "latest"
|
|
|
|
|
|
def source_folder(doc: dict[str, Any]) -> Path:
|
|
"""The folder a copy was read from: its `folder`, or for a copy frozen before that field, the one folder its
|
|
entries name (each `url` less its `original` path). RunError when the copy was not read from a folder."""
|
|
if nonempty_str(doc.get("folder")):
|
|
return Path(doc["folder"])
|
|
found: set[Path] = set()
|
|
for e in doc["sources"]:
|
|
url, original = e.get("url"), e.get("original")
|
|
if not (isinstance(url, str) and url.startswith("file://") and nonempty_str(original)):
|
|
continue
|
|
path = Path(urllib.parse.unquote(urllib.parse.urlparse(url).path))
|
|
depth = len(Path(original).parts)
|
|
if len(path.parts) <= depth:
|
|
continue
|
|
found.add(Path(*path.parts[:-depth]))
|
|
if not found:
|
|
raise RunError("the copy was not read from a folder (no folder, and no entry with a file:// url and an "
|
|
"original path); only a folder copy can be read again")
|
|
if len(found) > 1:
|
|
raise RunError(f"the entries name more than one folder ({', '.join(sorted(map(str, found)))}); "
|
|
"read the folder again with the folder command")
|
|
return found.pop()
|
|
|
|
|
|
def sound_root(root: Path) -> dict[str, Any]:
|
|
problems = check(root)
|
|
if problems:
|
|
raise RunError(f"{root} fails check: {'; '.join(problems[:5])}")
|
|
return load_manifest(root)
|
|
|
|
|
|
def reread(root: Path, out: Path, frozen_at: str | None = None) -> tuple[dict[str, Any], list[str], Path]:
|
|
"""The reread command: the folder ROOT was read from, read again as the same collection into OUT."""
|
|
doc = sound_root(root)
|
|
folder = source_folder(doc)
|
|
if not folder.is_dir():
|
|
raise RunError(f"the folder {folder} that {root} was read from is not there")
|
|
try:
|
|
new, skipped = freeze_folder(folder, doc["collection"], out, frozen_at)
|
|
except Refused as exc:
|
|
raise RunError(f"{folder} cannot be read again: {'; '.join(exc.problems)}") from exc
|
|
return new, skipped, folder
|
|
|
|
|
|
def watch_lines(report: dict[str, Any]) -> list[str]:
|
|
"""One line per difference, "change path [new sha256]", so a later change of the same file is a new line."""
|
|
lines = [f"new {e['path']}" for e in report["new_sources"]]
|
|
lines += [f"changed {e['path']} {e['new_sha256']}" for e in report["changed"]]
|
|
lines += [f"gone {e['path']}" for e in report["gone"]]
|
|
return sorted(lines)
|
|
|
|
|
|
def watch(root: Path, into: Path, bundle: Path | None, frozen_at: str | None = None) -> tuple[str, bool]:
|
|
"""The watch command: (the report, whether something is new since the last run). See the module doc."""
|
|
doc = sound_root(root)
|
|
folder = source_folder(doc).resolve()
|
|
target = into.resolve()
|
|
if target == folder or folder in target.parents:
|
|
raise RunError(f"{into} is inside the folder {folder}; the next reading would take the watch's files for "
|
|
"documents. Put it beside the folder")
|
|
state_path = into / WATCH_STATE
|
|
state = load_json(state_path) if state_path.is_file() else None
|
|
if state is not None and not (isinstance(state, dict) and state.get("schema") == WATCH_SCHEMA
|
|
and isinstance(state.get("entries"), list)):
|
|
raise RunError(f"{state_path} is not a watch state ({WATCH_SCHEMA})")
|
|
into.mkdir(parents=True, exist_ok=True)
|
|
work = Path(tempfile.mkdtemp(prefix=".watch-", dir=into))
|
|
try:
|
|
_, skipped, _ = reread(root, work / LATEST, frozen_at)
|
|
report = diff(root, work / LATEST)
|
|
(work / LATEST / "diff.json").write_text(dump(report), encoding="utf-8")
|
|
marked: dict[str, int] = {}
|
|
pages = 0
|
|
if bundle is not None:
|
|
found = marks(bundle, work / LATEST / "diff.json")
|
|
(work / LATEST / "marks.json").write_text(dump(found), encoding="utf-8")
|
|
pages = len(found["pages"])
|
|
for pg in found["pages"]:
|
|
for src in pg["sources"]:
|
|
marked[src["path"]] = marked.get(src["path"], 0) + 1
|
|
latest = into / LATEST
|
|
if latest.exists():
|
|
latest.rename(work / "replaced")
|
|
(work / LATEST).rename(latest)
|
|
finally:
|
|
shutil.rmtree(work, ignore_errors=True)
|
|
|
|
entries = watch_lines(report)
|
|
same_copy = state is not None and state.get("frozen") == report["old"]
|
|
seen = {str(e) for e in state["entries"]} if state is not None and same_copy else set()
|
|
news = [e for e in entries if e not in seen]
|
|
resolved = sorted(seen - set(entries))
|
|
state_path.write_text(dump({"entries": entries, "frozen": report["old"], "schema": WATCH_SCHEMA}),
|
|
encoding="utf-8")
|
|
by_path = {e["path"]: e for k in ("new_sources", "changed", "gone") for e in report[k]}
|
|
counts = (f"{len(report['new_sources'])} new, {len(report['changed'])} changed, {len(report['gone'])} gone, "
|
|
f"{report['unchanged']} unchanged")
|
|
lines = [f"watch: {folder} read again into {latest}, against {root} (frozen {report['old']['frozen_at']}): "
|
|
f"{counts}"]
|
|
if not news:
|
|
lines.append(" nothing differs" if not entries else " nothing new since the last run")
|
|
else:
|
|
lines.append("New since the last run:" if seen or same_copy else "Every difference (first run against this copy):")
|
|
for e in news:
|
|
change, path = e.split(" ")[:2]
|
|
known = by_path[path].get("original", path)
|
|
n = marked.get(path)
|
|
lines.append(f" {change:<8} {known} ({path})" + (f", {n} page(s) marked" if n else ""))
|
|
if resolved:
|
|
lines.append("No longer differs from the frozen copy:")
|
|
lines += [f" {e.split(' ')[1]}" for e in resolved]
|
|
if skipped:
|
|
lines.append("Skipped in the folder:")
|
|
lines += [f" {line}" for line in skipped]
|
|
if bundle is not None:
|
|
lines.append(f"{pages} page(s) marked in {bundle}; the list: {latest / 'marks.json'}")
|
|
lines.append(f"Every difference: {latest / 'diff.json'}. This only warns; rebuild from {latest} to update.")
|
|
return "\n".join(lines) + "\n", bool(news)
|
|
|
|
|
|
def build_parser() -> argparse.ArgumentParser:
|
|
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
sub = p.add_subparsers(dest="command", required=True)
|
|
c = sub.add_parser("check")
|
|
c.add_argument("root")
|
|
f = sub.add_parser("freeze")
|
|
f.add_argument("staging")
|
|
f.add_argument("--meta", required=True)
|
|
f.add_argument("--out", required=True)
|
|
f.add_argument("--frozen-at")
|
|
fo = sub.add_parser("folder")
|
|
fo.add_argument("folder")
|
|
fo.add_argument("--collection", required=True)
|
|
fo.add_argument("--out", required=True)
|
|
fo.add_argument("--frozen-at")
|
|
s = sub.add_parser("subset")
|
|
s.add_argument("root")
|
|
s.add_argument("--paths", required=True)
|
|
s.add_argument("--out", required=True)
|
|
s.add_argument("--frozen-at")
|
|
d = sub.add_parser("diff")
|
|
d.add_argument("old")
|
|
d.add_argument("new")
|
|
m = sub.add_parser("marks")
|
|
m.add_argument("collection")
|
|
m.add_argument("--diff", required=True)
|
|
r = sub.add_parser("reread")
|
|
r.add_argument("root")
|
|
r.add_argument("--out", required=True)
|
|
r.add_argument("--frozen-at")
|
|
w = sub.add_parser("watch")
|
|
w.add_argument("root")
|
|
w.add_argument("--into", required=True)
|
|
w.add_argument("--bundle")
|
|
w.add_argument("--frozen-at")
|
|
return p
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
args = build_parser().parse_args(argv)
|
|
try:
|
|
if args.command == "check":
|
|
root = Path(args.root)
|
|
problems = check(root)
|
|
n = len(load_manifest(root)["sources"])
|
|
if problems:
|
|
print(f"check: {len(problems)} problems in {root} ({n} sources listed)")
|
|
for p in problems:
|
|
print(f" {p}")
|
|
return 1
|
|
print(f"check: {n} sources in {root}, every file present and listed")
|
|
return 0
|
|
if args.command == "diff":
|
|
report = diff(Path(args.old), Path(args.new))
|
|
sys.stdout.write(dump(report))
|
|
return 1 if has_change(report) else 0
|
|
if args.command == "marks":
|
|
marked = marks(Path(args.collection), Path(args.diff))
|
|
sys.stdout.write(dump(marked))
|
|
return 1 if marked["pages"] else 0
|
|
if args.command == "reread":
|
|
new, skipped, folder = reread(Path(args.root), Path(args.out), args.frozen_at)
|
|
print(f"reread: {len(new['sources'])} sources from {folder} -> {args.out}; {len(skipped)} skipped")
|
|
for line in skipped:
|
|
print(f" {line}")
|
|
return 0
|
|
if args.command == "watch":
|
|
text, news = watch(Path(args.root), Path(args.into), Path(args.bundle) if args.bundle else None,
|
|
args.frozen_at)
|
|
sys.stdout.write(text)
|
|
return 1 if news else 0
|
|
if args.command == "folder":
|
|
doc, skipped = freeze_folder(Path(args.folder), args.collection, Path(args.out), args.frozen_at)
|
|
print(f"folder: {len(doc['sources'])} sources -> {args.out}; {len(skipped)} skipped")
|
|
for line in skipped:
|
|
print(f" {line}")
|
|
return 0
|
|
if args.command == "freeze":
|
|
doc = freeze(Path(args.staging), Path(args.meta), Path(args.out), args.frozen_at)
|
|
else:
|
|
doc = subset(Path(args.root), Path(args.paths), Path(args.out), args.frozen_at)
|
|
print(f"{args.command}: {len(doc['sources'])} sources -> {args.out}")
|
|
return 0
|
|
except Refused as exc:
|
|
print(f"{args.command} refused: {len(exc.problems)} problems, nothing written", file=sys.stderr)
|
|
for p in exc.problems:
|
|
print(f" {p}", file=sys.stderr)
|
|
return 1
|
|
except (RunError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
|
|
print(f"{args.command} cannot run: {exc}", file=sys.stderr)
|
|
return 2
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|