First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
1153 lines
53 KiB
Python
1153 lines
53 KiB
Python
#!/usr/bin/env python3
|
|
"""Daily watch on the canonical Open Knowledge Format repository, and on the sources of collection 1.
|
|
|
|
Answers four questions every run: **has canonical moved past our pin in a way that touches the contract?**, **which
|
|
issues and pull requests are new or changed since the last run?**, **has the announced new home (`open-knowledge-format`)
|
|
got a public repository, or has the move itself (#63, canonical archived) changed?** and, when it is switched on, **which documentation
|
|
sources are new, changed or gone since the frozen copy, that the last message did not already say?** On a hit it
|
|
sends one coord message. On a miss it prints one line per check and exits 0.
|
|
|
|
The commit check is moved here from llm-ingestion-okf (`tools/okf_watch.py`, last changed in 3233b19), where it was
|
|
written and measured; the three rules below and the reasoning behind them are that file's. This file is not part of
|
|
the package: it lives in `tools/`, outside `okfwiki/`, because it needs `git` and the network. It imports the fetcher
|
|
(`tools/fetch_docs.py`) and `okfwiki.sources`, so the job's interpreter needs PyYAML (measured 09.10.2026: 6.0.3 in
|
|
`/usr/local/bin/python3`).
|
|
|
|
## Why this file is built the way it is
|
|
|
|
Almost every run of this watch produces a NEGATIVE result. That is the hazard it is designed around, and the three
|
|
rules that follow are not stylistic:
|
|
|
|
1. **A failed call is never an empty result.** `git()` raises on a non-zero exit and carries stderr, and an API answer
|
|
that is not 200 with the expected shape raises too (a 403 or 429, a page that is not JSON, an item without a
|
|
number). A helper that returned nothing for a failed query would manufacture "upstream is quiet" out of a broken
|
|
query, which is the exact defect this watch exists to prevent.
|
|
2. **It proves it can find, every run.** `prove_can_find()` re-runs the whole detect-and-classify path over a range
|
|
known to have changed `SPEC.md` (`ad30107^1..ad30107`, the ISO-datetime tightening). The issue check fetches a
|
|
known issue (#8, the JSON schema issue the plan follows) with its own call, and asks the list from that issue's own
|
|
time, where it must stand. If a known positive comes back empty, the query is broken and the run aborts LOUDLY
|
|
rather than reporting a clean sweep.
|
|
3. **The network failing is loud.** `git fetch` failing raises, and so does an unreachable API. Neither degrades to
|
|
"no change since last time".
|
|
|
|
## The checks are a list
|
|
|
|
`CHECKS` names each check, and the state file keeps one section per name. A check reads its own section and returns
|
|
a summary line, a report (empty when there is nothing to say) and its next section. `run()` runs every check first:
|
|
if one fails, the run fails, nothing is sent and no state is written, so a half-measured run never looks whole. The
|
|
one exception is `docs` (FAILS_ALONE): it fetches for minutes with hundreds of calls, so a failure there fails only
|
|
that check, with its own FAILED line and exit 2, while the others report and keep their state.
|
|
|
|
## The docs check (order 8 of wave 2)
|
|
|
|
Off unless `OKF_WIKI_DOCS_CHECK` is `on`: the job runs the code on `v1` in the main checkout, so nothing new happens
|
|
before the operator has said yes and the setting is put in the job. On, it fetches the documentation tree with
|
|
`tools/fetch_docs.py`, freezes it into `<OKF_WIKI_DOCS_DAILY>/<YYYYMMDD>/` (a K1 root; a second fetch the same day
|
|
replaces that day's folder; the day is the local one), runs `okfwiki.sources diff` against the frozen copy (`OKF_WIKI_DOCS_FROZEN`), writes
|
|
`diff.json` there, and `marks.json` when `OKF_WIKI_DOCS_BUNDLE` names a built collection. It never rebuilds: the
|
|
message says to build from the fetch in a session (the operator 07.11.2026, C19). Its state section keeps the
|
|
sha256 of the differences it last saw and one line per difference, so the message says only what is new since the
|
|
last one. The oldest dated folders beyond `OKF_WIKI_DOCS_KEEP` (7) are removed, and so are the work folders
|
|
(`.YYYYMMDD-*`) a killed run left; never a folder that is or holds the frozen copy, never a link, never anything but
|
|
those two names directly under the daily folder. Any error in the check is a WatchError, so it fails alone, and a
|
|
failure is sent even on a morning with no other news.
|
|
|
|
## Why quiet is the enumerated list, not signal
|
|
|
|
Listing what counts as normative can only match what upstream has already invented. Anything new falls outside the
|
|
list and the watch goes silent about it, failing in the direction nobody notices. So the enumeration runs the other
|
|
way: a small list of paths measured not to bear the contract, and everything else reports. Over-firing is visible and
|
|
fixable by widening QUIET_PREFIXES. Under-firing is neither.
|
|
|
|
`README.md` is deliberately NOT quiet. Upstream announced the repository move in a README commit, and that move is the
|
|
change with the longest reach the old library saw.
|
|
|
|
## What this watch does NOT catch
|
|
|
|
It sees commits, issue numbers and times. It cannot see *meaning*. When upstream tightened v0.2 in place without a
|
|
version bump, this watch would have fired correctly (SPEC.md changed), but nothing in a commit list says "a value that
|
|
conformed last month does not conform now". The watch's output is therefore "run the upgrade runbook" (rule R32 in
|
|
`docs/rule-decisions.md`) or "read these issues upstream", never "here is your exposure".
|
|
|
|
It also cannot see upstream abandoning this repository *silently*. A relocation announced somewhere other than this
|
|
git history and its issues is invisible here. The one move announced so far (PR #63, read 08.11.2026: the work goes to
|
|
the `open-knowledge-format` organisation with SPEC v0.3) has its own check, `home`: the organisation's public
|
|
repositories, #63's state and whether canonical is archived, reported when one of them changes.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import ssl
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
import time
|
|
import urllib.error
|
|
import urllib.request
|
|
from collections.abc import Callable, Mapping
|
|
from dataclasses import dataclass, field
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
from typing import Any, Protocol
|
|
|
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
|
|
|
import fetch_docs # noqa: E402 (it puts the repository root on the path, so okfwiki imports after it)
|
|
from okfwiki import sources # noqa: E402
|
|
from okfwiki.common import RunError # noqa: E402
|
|
|
|
CANONICAL_REPO = "GoogleCloudPlatform/open-knowledge-format"
|
|
CANONICAL_URL = f"https://github.com/{CANONICAL_REPO}.git"
|
|
|
|
#: Our pin. Moves only when the upgrade runbook has been run (rule R32).
|
|
PIN = "ad30107c31c06aec8a7d5636e0d1058118604e6f"
|
|
|
|
#: A range in canonical that really did change SPEC.md, used every run to show the query can find. `ad30107` is the
|
|
#: merge of `okf-iso-datetimes`; its first-parent range carries three SPEC.md commits.
|
|
KNOWN_POSITIVE = (
|
|
"ad30107c31c06aec8a7d5636e0d1058118604e6f^1",
|
|
"ad30107c31c06aec8a7d5636e0d1058118604e6f",
|
|
)
|
|
|
|
#: Paths measured not to bear the contract. Everything else is normative.
|
|
QUIET_PREFIXES = (
|
|
".github/",
|
|
".gitignore",
|
|
"CONTRIBUTING.md",
|
|
"CODE_OF_CONDUCT.md",
|
|
"LICENSE.md",
|
|
"src/reference_agent/viewer/",
|
|
)
|
|
#: Generated viewer output, committed alongside bundles it does not define.
|
|
QUIET_SUFFIXES = ("viz.html",)
|
|
|
|
DEFAULT_MIRROR = Path.home() / "repos" / "_okf-canonical"
|
|
DEFAULT_STATE = Path.home() / ".claude" / "okf-wiki-watch-state.json"
|
|
STATE_SCHEMA = "okf-wiki/watch-state/1"
|
|
SELF_REPO = "okf-wiki"
|
|
COORDINATOR_REPO = ".claude"
|
|
|
|
#: GitHub's public REST API, read without a key: 60 calls an hour (measured 08.10.2026); a run makes seven or more.
|
|
API_ROOT = "https://api.github.com/"
|
|
ISSUES_URL = f"{API_ROOT}repos/{CANONICAL_REPO}/issues"
|
|
#: Issue and pull request pages, newest change first. Read oldest first, an item updated mid-read moves to the end and
|
|
#: can push an unread one back onto a page already read; newest first it can only repeat one, and anything changed
|
|
#: during the read is newer than the `since` we keep, so the next run sees it.
|
|
LIST_URL = ISSUES_URL + "?state=all&sort=updated&direction=desc&per_page=100"
|
|
#: The list asked from the known issue's own time, oldest first, one page: the known issue must be on it. GitHub's
|
|
#: `since` includes its boundary (measured 08.10.2026: #8 came back for its own updated_at).
|
|
PROBE_URL = ISSUES_URL + "?state=all&sort=updated&direction=asc&per_page=10&since="
|
|
#: An issue known to exist upstream (the JSON schema issue, plan format choice (g)); fetched by its own call each run.
|
|
KNOWN_ISSUE = 8
|
|
#: More pages than this is an answer we do not understand, not a long list to cut short.
|
|
MAX_PAGES = 50
|
|
#: The system's CA bundle on macOS. The python.org build the daily job runs with has no CA certificates of its own
|
|
#: (measured 08.10.2026: 0 in its default store), so this bundle is added; verification is never turned off.
|
|
SYSTEM_CA_BUNDLE = Path("/etc/ssl/cert.pem")
|
|
TIME_RE = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z\Z")
|
|
|
|
#: One answer: status, headers, body.
|
|
Response = tuple[int, Mapping[str, str], bytes]
|
|
Fetch = Callable[[str], Response]
|
|
|
|
|
|
class WatchError(RuntimeError):
|
|
"""Anything that stops this watch from measuring. Never swallowed.
|
|
|
|
Raised for a failed git call, an unreachable mirror, a failed fetch, an unreadable state file, or a known-positive
|
|
that finds nothing. Each of those would otherwise surface as "no change": a negative nobody measured.
|
|
"""
|
|
|
|
|
|
def git(repo: Path, *args: str) -> str:
|
|
"""Run git, or raise. Returns stripped stdout; an empty return means empty.
|
|
|
|
The distinction the whole file rests on: a non-zero exit raises, so a caller reading `""` knows the query ran and
|
|
found nothing.
|
|
"""
|
|
try:
|
|
proc = subprocess.run(["git", *args], cwd=repo, capture_output=True, text=True, timeout=300)
|
|
except FileNotFoundError as exc: # pragma: no cover - git absent
|
|
raise WatchError(f"git not found on PATH: {exc}") from exc
|
|
except subprocess.TimeoutExpired as exc:
|
|
raise WatchError(f"git {' '.join(args)} timed out after 300s") from exc
|
|
if proc.returncode != 0:
|
|
raise WatchError(
|
|
f"git {' '.join(args)} failed with exit {proc.returncode} in {repo}: "
|
|
f"{proc.stderr.strip() or '(no stderr)'}"
|
|
)
|
|
return proc.stdout.strip()
|
|
|
|
|
|
def classify(paths: list[str]) -> tuple[list[str], list[str]]:
|
|
"""Split changed paths into (normative, quiet), preserving input order."""
|
|
normative: list[str] = []
|
|
quiet: list[str] = []
|
|
for p in paths:
|
|
if p.startswith(QUIET_PREFIXES) or p.endswith(QUIET_SUFFIXES):
|
|
quiet.append(p)
|
|
else:
|
|
normative.append(p)
|
|
return normative, quiet
|
|
|
|
|
|
def changed_paths(mirror: Path, base: str, head: str) -> list[str]:
|
|
"""Paths touched between two refs, de-duplicated, order preserved."""
|
|
out = git(mirror, "log", "--first-parent", "--name-only", "--format=", f"{base}..{head}")
|
|
seen: dict[str, None] = {}
|
|
for line in out.splitlines():
|
|
line = line.strip()
|
|
if line:
|
|
seen.setdefault(line, None)
|
|
return list(seen)
|
|
|
|
|
|
def commits_between(mirror: Path, base: str, head: str) -> list[tuple[str, str]]:
|
|
out = git(mirror, "log", "--first-parent", "--format=%h%x1f%s", f"{base}..{head}")
|
|
rows: list[tuple[str, str]] = []
|
|
for line in out.splitlines():
|
|
if "\x1f" in line:
|
|
sha, subject = line.split("\x1f", 1)
|
|
rows.append((sha, subject))
|
|
return rows
|
|
|
|
|
|
def prove_can_find(mirror: Path, base: str = KNOWN_POSITIVE[0], head: str = KNOWN_POSITIVE[1]) -> None:
|
|
"""Run the real query over a range known to be positive, or abort.
|
|
|
|
A negative result is only a measurement once the instrument has been shown able to produce a positive.
|
|
"""
|
|
paths = changed_paths(mirror, base, head)
|
|
normative, _ = classify(paths)
|
|
if not normative:
|
|
raise WatchError(
|
|
f"known-positive range {base}..{head} returned no normative paths. "
|
|
"The query is broken, not upstream quiet. Refusing to report a clean sweep."
|
|
)
|
|
|
|
|
|
# --- the state file: one section per check ---------------------------------------------------------------------------
|
|
|
|
|
|
def load_state(path: Path) -> dict[str, Any]:
|
|
"""The sections of the state file, by check name. A missing file is a first run; an unreadable one is an error.
|
|
|
|
Reading a damaged file as empty would turn the next run into a first run, which registers everything and reports
|
|
nothing: a silent negative.
|
|
"""
|
|
try:
|
|
text = path.read_text(encoding="utf-8")
|
|
except FileNotFoundError:
|
|
return {}
|
|
except OSError as exc:
|
|
raise WatchError(f"state file {path} cannot be read: {exc}") from exc
|
|
try:
|
|
raw = json.loads(text)
|
|
except ValueError as exc:
|
|
raise WatchError(f"state file {path} is not JSON: {exc}") from exc
|
|
if not isinstance(raw, dict) or raw.get("schema") != STATE_SCHEMA or not isinstance(raw.get("checks"), dict):
|
|
raise WatchError(f"state file {path} is not {STATE_SCHEMA}")
|
|
return dict(raw["checks"])
|
|
|
|
|
|
def save_state(path: Path, checks: dict[str, Any]) -> None:
|
|
path.parent.mkdir(parents=True, exist_ok=True)
|
|
data = {"checks": checks, "schema": STATE_SCHEMA}
|
|
tmp = path.with_name(path.name + ".tmp")
|
|
tmp.write_text(json.dumps(data, indent=1, sort_keys=True, ensure_ascii=False) + "\n", encoding="utf-8")
|
|
tmp.replace(path)
|
|
|
|
|
|
@dataclass
|
|
class CommitState:
|
|
"""What has already been announced, so a hit is announced exactly once.
|
|
|
|
Keyed on the pin: moving the pin means we absorbed everything behind it, so nothing before the new pin is news.
|
|
"""
|
|
|
|
pin: str
|
|
reported: list[str] = field(default_factory=list)
|
|
|
|
@classmethod
|
|
def from_section(cls, section: object, pin: str) -> CommitState:
|
|
if section is None:
|
|
return cls(pin=pin)
|
|
if not isinstance(section, dict) or not isinstance(section.get("reported"), list):
|
|
raise WatchError("state section 'commits' is malformed")
|
|
if section.get("pin") != pin:
|
|
return cls(pin=pin)
|
|
return cls(pin=pin, reported=[str(s) for s in section["reported"]])
|
|
|
|
def to_section(self) -> dict[str, Any]:
|
|
return {"pin": self.pin, "reported": list(self.reported)}
|
|
|
|
def record(self, shas: list[str]) -> None:
|
|
for s in shas:
|
|
if s not in self.reported:
|
|
self.reported.append(s)
|
|
|
|
|
|
def decide(state: CommitState, normative_shas: list[str]) -> list[str]:
|
|
"""The commits worth a message: normative, and not announced before."""
|
|
return [s for s in normative_shas if s not in state.reported]
|
|
|
|
|
|
def render_body(
|
|
*,
|
|
pin: str,
|
|
head: str,
|
|
commits: list[tuple[str, str]],
|
|
normative: list[str],
|
|
quiet: list[str],
|
|
) -> str:
|
|
"""The coord message. ASCII only - coord-send requires it."""
|
|
total = len(normative) + len(quiet)
|
|
lines = [
|
|
"Canonical OKF has moved past our pin.",
|
|
"",
|
|
f" repository : {CANONICAL_REPO}",
|
|
f" our pin : {pin[:7]}",
|
|
f" upstream : {head[:7]}",
|
|
f" paths : {len(normative)} of {total} changed paths are contract-bearing",
|
|
"",
|
|
"New commits (first-parent):",
|
|
]
|
|
lines += [f" {sha} {_ascii(subject)}" for sha, subject in commits] or [" (none)"]
|
|
lines += ["", "Contract-bearing paths:"]
|
|
lines += [f" {_ascii(p)}" for p in normative] or [" (none)"]
|
|
if quiet:
|
|
lines += ["", f"Not counted (quiet list): {len(quiet)} path(s)."]
|
|
lines += [
|
|
"",
|
|
"This watch sees commits, not meaning. It cannot tell you whether a",
|
|
"value that conformed last month still conforms. Next step is the",
|
|
"procedure, not a conclusion: the upgrade runbook,",
|
|
"docs/upstream-runbook.md in okf-wiki (rule R32 in",
|
|
"docs/rule-decisions.md).",
|
|
]
|
|
return "\n".join(lines) + "\n"
|
|
|
|
|
|
def _ascii(text: str) -> str:
|
|
"""Upstream text in a coord body: anything outside ASCII becomes '?', so the send never fails on it."""
|
|
return text.encode("ascii", "replace").decode("ascii")
|
|
|
|
|
|
def _coord_send_script() -> Path:
|
|
"""Newest repo-mailbox scripts dir, resolved at runtime.
|
|
|
|
Hard-coding a plugin version is a rot hazard: the path moves every plugin release and a stale one fails as
|
|
"command not found", which reads like the watch having nothing to say.
|
|
|
|
The plugin is looked up under every marketplace in the cache: it moved from ktg-plugin-marketplace to ktg-privat,
|
|
and a fixed marketplace then failed every morning.
|
|
"""
|
|
base = Path.home() / ".claude" / "plugins" / "cache"
|
|
|
|
def key(p: Path) -> tuple[int, ...]:
|
|
try:
|
|
return tuple(int(x) for x in p.name.split("."))
|
|
except ValueError:
|
|
return (-1,)
|
|
|
|
candidates = [d for d in base.glob("*/repo-mailbox/*") if (d / "scripts" / "coord-send.sh").is_file()]
|
|
if not candidates:
|
|
raise WatchError(f"no repo-mailbox coord-send.sh under any marketplace in {base}")
|
|
return max(candidates, key=key) / "scripts" / "coord-send.sh"
|
|
|
|
|
|
def send(to: str, subject: str, body: str, *, fyi: bool, dry_run: bool) -> None:
|
|
script = _coord_send_script()
|
|
args = ["bash", str(script), "--to", to, "--from", SELF_REPO, "--subject", subject]
|
|
if fyi:
|
|
args.append("--fyi")
|
|
if dry_run:
|
|
print(f"[dry-run] {' '.join(args)}\n{body}")
|
|
return
|
|
proc = subprocess.run(args, input=body, capture_output=True, text=True)
|
|
if proc.returncode != 0:
|
|
raise WatchError(
|
|
f"coord-send to {to} failed with exit {proc.returncode}: {proc.stderr.strip() or '(no stderr)'}"
|
|
)
|
|
|
|
|
|
# --- issues and pull requests ----------------------------------------------------------------------------------------
|
|
|
|
|
|
def tls_context(bundle: Path = SYSTEM_CA_BUNDLE) -> ssl.SSLContext:
|
|
"""The default verifying context, plus the system bundle when there is one. A bundle that will not load is loud."""
|
|
context = ssl.create_default_context()
|
|
if bundle.is_file():
|
|
try:
|
|
context.load_verify_locations(cafile=str(bundle))
|
|
except (ssl.SSLError, OSError) as exc:
|
|
raise WatchError(f"the CA bundle {bundle} cannot be loaded: {exc}") from exc
|
|
return context
|
|
|
|
|
|
def http_fetch(url: str, *, urlopen: Callable[..., Any] = urllib.request.urlopen) -> Response:
|
|
"""One GET against the API, without a key. A refusal comes back as its status; no answer at all raises."""
|
|
request = urllib.request.Request(
|
|
url,
|
|
headers={
|
|
"Accept": "application/vnd.github+json",
|
|
"User-Agent": "okf-wiki-upstream-watch",
|
|
"X-GitHub-Api-Version": "2022-11-28",
|
|
},
|
|
)
|
|
try:
|
|
with urlopen(request, timeout=60, context=tls_context()) as resp:
|
|
return int(resp.status), dict(resp.headers.items()), bytes(resp.read())
|
|
except urllib.error.HTTPError as exc:
|
|
with exc:
|
|
return int(exc.code), dict(exc.headers.items()) if exc.headers else {}, bytes(exc.read())
|
|
except (urllib.error.URLError, OSError) as exc:
|
|
raise WatchError(f"GET {url} got no answer: {exc}") from exc
|
|
|
|
|
|
def _header(headers: Mapping[str, str], name: str) -> str | None:
|
|
for key, value in headers.items():
|
|
if key.lower() == name:
|
|
return value
|
|
return None
|
|
|
|
|
|
def get_json(fetch: Fetch, url: str, what: str) -> tuple[Any, Mapping[str, str]]:
|
|
"""The parsed body of a 200 answer, or raise. 403 and 429 are refusals (often the hourly limit), not silence."""
|
|
status, headers, body = fetch(url)
|
|
if status != 200:
|
|
detail = ""
|
|
if status in (403, 429):
|
|
detail = f" (refused; rate limit remaining: {_header(headers, 'x-ratelimit-remaining') or 'not given'})"
|
|
raise WatchError(f"{what}: GET {url} answered HTTP {status}{detail}. This is not 'nothing new'.")
|
|
try:
|
|
return json.loads(body.decode("utf-8")), headers
|
|
except (UnicodeDecodeError, ValueError) as exc:
|
|
raise WatchError(f"{what}: GET {url} answered 200 with a body that is not JSON: {exc}") from exc
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Item:
|
|
"""One issue or pull request, as much as the watch keeps: its number, its kind and when it last changed."""
|
|
|
|
number: int
|
|
kind: str
|
|
updated_at: str
|
|
|
|
def url(self) -> str:
|
|
page = "pull" if self.kind == "pull request" else "issues"
|
|
return f"https://github.com/{CANONICAL_REPO}/{page}/{self.number}"
|
|
|
|
|
|
def parse_item(raw: object, what: str) -> Item:
|
|
if not isinstance(raw, dict):
|
|
raise WatchError(f"{what}: an item is not an object")
|
|
number = raw.get("number")
|
|
updated_at = raw.get("updated_at")
|
|
if not isinstance(number, int) or isinstance(number, bool) or number < 1:
|
|
raise WatchError(f"{what}: an item has no valid number")
|
|
if not isinstance(updated_at, str) or not TIME_RE.match(updated_at):
|
|
raise WatchError(f"{what}: item #{number} has no valid updated_at")
|
|
return Item(number=number, kind="pull request" if "pull_request" in raw else "issue", updated_at=updated_at)
|
|
|
|
|
|
def next_link(link: str | None) -> str | None:
|
|
"""The rel="next" target of a Link header, or None on the last page."""
|
|
for part in (link or "").split(","):
|
|
m = re.match(r'\s*<([^>]+)>\s*;\s*rel="([^"]+)"', part)
|
|
if m and m.group(2) == "next":
|
|
return m.group(1)
|
|
return None
|
|
|
|
|
|
def list_items(fetch: Fetch, since: str | None) -> dict[int, Item]:
|
|
"""Every issue and pull request updated at or after `since` (all of them when None), by number, every page read."""
|
|
url: str | None = LIST_URL + (f"&since={since}" if since else "")
|
|
found: dict[int, Item] = {}
|
|
pages = 0
|
|
while url is not None:
|
|
pages += 1
|
|
if pages > MAX_PAGES:
|
|
raise WatchError(f"issue list: more than {MAX_PAGES} pages; refusing to cut the list short")
|
|
what = f"issue list page {pages}"
|
|
data, headers = get_json(fetch, url, what)
|
|
if not isinstance(data, list):
|
|
raise WatchError(f"{what}: GET {url} answered with something other than a list")
|
|
for raw in data:
|
|
it = parse_item(raw, what)
|
|
# An item changed while the pages are read can come back twice; the later time is the one that holds.
|
|
if it.number not in found or it.updated_at > found[it.number].updated_at:
|
|
found[it.number] = it
|
|
url = next_link(_header(headers, "link"))
|
|
if url is not None and not url.startswith(API_ROOT):
|
|
raise WatchError(f"{what}: the next page points off the API ({url}); refusing to follow it")
|
|
return found
|
|
|
|
|
|
def prove_can_find_issue(fetch: Fetch) -> Item:
|
|
"""Fetch the known issue by its own call, or abort: a list that cannot be shown to find proves nothing."""
|
|
what = f"known-positive issue #{KNOWN_ISSUE}"
|
|
known = parse_item(get_json(fetch, f"{ISSUES_URL}/{KNOWN_ISSUE}", what)[0], what)
|
|
if known.number != KNOWN_ISSUE:
|
|
raise WatchError(f"{what}: the answer is #{known.number}. The query is broken, not upstream quiet.")
|
|
return known
|
|
|
|
|
|
def prove_list_can_find(fetch: Fetch, known: Item) -> None:
|
|
"""Ask the list from the known issue's own time and require it there, every run.
|
|
|
|
The list since the last run almost never holds the known issue, so without this call the list query itself would
|
|
never be shown able to find anything.
|
|
"""
|
|
what = f"known-positive issue #{KNOWN_ISSUE} in the list"
|
|
url = PROBE_URL + known.updated_at
|
|
data, _ = get_json(fetch, url, what)
|
|
if not isinstance(data, list):
|
|
raise WatchError(f"{what}: GET {url} answered with something other than a list")
|
|
if KNOWN_ISSUE not in {parse_item(raw, what).number for raw in data}:
|
|
raise WatchError(f"{what}: the list from {known.updated_at} does not hold it. The list is broken, not quiet.")
|
|
|
|
|
|
@dataclass
|
|
class IssueState:
|
|
"""The newest change time seen (the next list starts there) and every item seen, by number."""
|
|
|
|
since: str
|
|
seen: dict[int, Item]
|
|
|
|
@classmethod
|
|
def from_section(cls, section: object) -> IssueState | None:
|
|
"""None is a first run. A section that cannot be read fails: starting over would report nothing."""
|
|
if section is None:
|
|
return None
|
|
bad = WatchError("state section 'issues' is malformed")
|
|
if not isinstance(section, dict) or not isinstance(section.get("seen"), dict):
|
|
raise bad
|
|
since = section.get("since")
|
|
if not isinstance(since, str) or not TIME_RE.match(since):
|
|
raise bad
|
|
seen: dict[int, Item] = {}
|
|
for key, entry in section["seen"].items():
|
|
if not isinstance(key, str) or not key.isdigit() or not isinstance(entry, dict):
|
|
raise bad
|
|
kind, updated_at = entry.get("kind"), entry.get("updated_at")
|
|
if kind not in ("issue", "pull request"):
|
|
raise bad
|
|
if not isinstance(updated_at, str) or not TIME_RE.match(updated_at):
|
|
raise bad
|
|
seen[int(key)] = Item(number=int(key), kind=str(kind), updated_at=updated_at)
|
|
return cls(since=since, seen=seen)
|
|
|
|
def to_section(self) -> dict[str, Any]:
|
|
return {
|
|
"seen": {str(n): {"kind": i.kind, "updated_at": i.updated_at} for n, i in sorted(self.seen.items())},
|
|
"since": self.since,
|
|
}
|
|
|
|
|
|
def render_issue_report(since: str, returned: int, new: list[Item], changed: list[Item]) -> str:
|
|
"""The issue part of the coord message: numbers, kinds, times and links only. ASCII by construction."""
|
|
lines = [
|
|
f"Issues and pull requests in {CANONICAL_REPO} updated since {since}:",
|
|
f" {len(new)} new and {len(changed)} changed, of {returned} returned.",
|
|
"",
|
|
]
|
|
for label, items in (("new", new), ("changed", changed)):
|
|
for i in items:
|
|
lines.append(f" {label:<8} {i.kind:<13} #{i.number:<5} updated {i.updated_at} {i.url()}")
|
|
lines += ["", "This watch sees numbers and times, not content: read them upstream."]
|
|
return "\n".join(lines) + "\n"
|
|
|
|
|
|
# --- the home check: the announced new home --------------------------------------------------------------------------
|
|
|
|
#: Upstream PR #63 (open, read 08.11.2026) marks canonical as deprecated and names the new home: the organisation
|
|
#: below, with SPEC v0.3. On 08.11.2026 the organisation existed (created 28.09.2026) with 0 public repositories.
|
|
HOME_ORG = "open-knowledge-format"
|
|
HOME_PR = 63
|
|
ORG_URL = f"{API_ROOT}orgs/{HOME_ORG}"
|
|
#: One page holds 100; more pages are read by the Link header like the issue list.
|
|
ORG_REPOS_URL = f"{ORG_URL}/repos?type=public&per_page=100"
|
|
PR_URL = f"{API_ROOT}repos/{CANONICAL_REPO}/pulls/{HOME_PR}"
|
|
CANON_API_URL = f"{API_ROOT}repos/{CANONICAL_REPO}"
|
|
#: What a first run compares against: the state measured on 08.11.2026, so a move that happened before the first run
|
|
#: is still reported.
|
|
HOME_MEASURED: dict[str, Any] = {"archived": False, "pr": "open", "repos": []}
|
|
|
|
|
|
def _pr_state(raw: object) -> str:
|
|
if not isinstance(raw, dict) or raw.get("number") != HOME_PR:
|
|
raise WatchError(f"pull request #{HOME_PR}: the answer is not #{HOME_PR}. The query is broken, not quiet.")
|
|
state, merged = raw.get("state"), raw.get("merged")
|
|
if state not in ("open", "closed") or not isinstance(merged, bool):
|
|
raise WatchError(f"pull request #{HOME_PR}: no valid state or merged flag")
|
|
return "merged" if merged else str(state)
|
|
|
|
|
|
def home_section(section: object) -> dict[str, Any]:
|
|
"""The last section, or the 08.11 measurement on a first run. A section that cannot be read fails."""
|
|
if section is None:
|
|
return dict(HOME_MEASURED)
|
|
if (not isinstance(section, dict) or set(section) != set(HOME_MEASURED)
|
|
or not isinstance(section["archived"], bool) or section["pr"] not in ("open", "closed", "merged")
|
|
or not isinstance(section["repos"], list) or not all(isinstance(r, str) for r in section["repos"])):
|
|
raise WatchError("state section 'home' is malformed")
|
|
return section
|
|
|
|
|
|
def check_home(ctx: Context, section: object) -> Result:
|
|
"""The organisation's public repositories, #63's state and whether canonical is archived; a change reports."""
|
|
before = home_section(section)
|
|
what = f"organisation {HOME_ORG}"
|
|
data, _ = get_json(ctx.fetch, ORG_URL, what)
|
|
if not isinstance(data, dict) or data.get("login") != HOME_ORG:
|
|
raise WatchError(f"{what}: the answer is not {HOME_ORG}. The query is broken, not quiet.")
|
|
count = data.get("public_repos")
|
|
if not isinstance(count, int) or isinstance(count, bool) or count < 0:
|
|
raise WatchError(f"{what}: no valid public_repos count")
|
|
|
|
repos: list[str] = []
|
|
url: str | None = ORG_REPOS_URL
|
|
pages = 0
|
|
while url is not None:
|
|
pages += 1
|
|
if pages > MAX_PAGES:
|
|
raise WatchError(f"{what}: more than {MAX_PAGES} pages of repositories; refusing to cut the list short")
|
|
page, headers = get_json(ctx.fetch, url, f"{what} repositories page {pages}")
|
|
if not isinstance(page, list):
|
|
raise WatchError(f"{what}: GET {url} answered with something other than a list")
|
|
for raw in page:
|
|
name = raw.get("full_name") if isinstance(raw, dict) else None
|
|
if not isinstance(name, str) or not name.startswith(f"{HOME_ORG}/"):
|
|
raise WatchError(f"{what}: a repository without a valid full_name")
|
|
repos.append(name)
|
|
url = next_link(_header(headers, "link"))
|
|
if url is not None and not url.startswith(API_ROOT):
|
|
raise WatchError(f"{what}: the next page points off the API ({url}); refusing to follow it")
|
|
repos = sorted(set(repos))
|
|
# With no public repository the list cannot be shown to find one; the organisation's own count is the check.
|
|
if len(repos) != count:
|
|
raise WatchError(f"{what}: the list holds {len(repos)} repositories, the organisation's count says {count}. "
|
|
"The list is broken, not quiet.")
|
|
|
|
pr = _pr_state(get_json(ctx.fetch, PR_URL, f"pull request #{HOME_PR}")[0])
|
|
canon, _ = get_json(ctx.fetch, CANON_API_URL, f"repository {CANONICAL_REPO}")
|
|
if not isinstance(canon, dict) or canon.get("full_name") != CANONICAL_REPO or not isinstance(
|
|
canon.get("archived"), bool):
|
|
raise WatchError(f"repository {CANONICAL_REPO}: no valid full_name or archived flag")
|
|
after: dict[str, Any] = {"archived": bool(canon["archived"]), "pr": pr, "repos": repos}
|
|
|
|
summary = (f"{HOME_ORG}: {len(repos)} public repositories; #{HOME_PR} {pr}; canonical "
|
|
f"{'archived' if after['archived'] else 'not archived'}.")
|
|
lines: list[str] = []
|
|
for name in repos:
|
|
if name not in before["repos"]:
|
|
lines.append(f" new public repository {name} https://github.com/{name}")
|
|
for name in before["repos"]:
|
|
if name not in repos:
|
|
lines.append(f" gone public repository {name} (no longer listed)")
|
|
if pr != before["pr"]:
|
|
word = {"merged": "merged", "closed": "closed without merge", "open": "open again"}[pr]
|
|
lines.append(f" #{HOME_PR} (mark canonical as deprecated) is now {word}: "
|
|
f"https://github.com/{CANONICAL_REPO}/pull/{HOME_PR}")
|
|
if after["archived"] != before["archived"]:
|
|
lines.append(f" {CANONICAL_REPO} is now {'archived' if after['archived'] else 'no longer archived'}.")
|
|
if not lines:
|
|
return Result(summary=summary + " Unchanged since the last run.", report="", section=after)
|
|
report = [f"The announced new home of OKF ({HOME_ORG}) or the move itself changed:", "", *lines, "",
|
|
"When the new repository with SPEC v0.3 can be read, run the upgrade runbook against it in its own "
|
|
"session (rule R32), and move the address in CLAUDE.md there. The pin stays until then."]
|
|
return Result(summary=summary + f" {len(lines)} change(s).", report="\n".join(report) + "\n", section=after)
|
|
|
|
|
|
# --- the docs check: settings ----------------------------------------------------------------------------------------
|
|
|
|
DOCS_SWITCH = "OKF_WIKI_DOCS_CHECK"
|
|
SAMLING1 = Path.home() / "okf" / "benk" / "samling1"
|
|
DEFAULT_DAILY = SAMLING1 / "daglig"
|
|
DEFAULT_FROZEN = SAMLING1 / "20261008"
|
|
#: The preamble rule of the fetch ({"lines": N, "sha256": ...}), kept outside git (wave 1 order 2).
|
|
DEFAULT_PREAMBLE = SAMLING1 / "innledning.json"
|
|
DEFAULT_KEEP = 7
|
|
DAY_RE = re.compile(r"\d{8}\Z")
|
|
WORK_RE = re.compile(r"\.\d{8}-")
|
|
|
|
|
|
def _utc_now() -> datetime:
|
|
return datetime.now(timezone.utc)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class DocsConfig:
|
|
"""The docs check's settings. Off unless enabled; tests replace the site, the fetcher and the clock."""
|
|
|
|
enabled: bool = False
|
|
setting: str = ""
|
|
daily: Path = DEFAULT_DAILY
|
|
frozen: Path = DEFAULT_FROZEN
|
|
preamble: Path = DEFAULT_PREAMBLE
|
|
bundle: Path | None = None
|
|
keep: int = DEFAULT_KEEP
|
|
base: str = fetch_docs.BASE
|
|
index_url: str = fetch_docs.INDEX_URL
|
|
changelog: str = fetch_docs.CHANGELOG
|
|
changelog_page: str = fetch_docs.CHANGELOG_PAGE
|
|
pause: float = 0.5
|
|
fetcher: fetch_docs.Fetcher | None = None
|
|
clock: Callable[[], datetime] = _utc_now
|
|
|
|
|
|
def docs_config(env: Mapping[str, str]) -> DocsConfig:
|
|
"""The settings from the environment (the job's plist sets them). Only the word `on` turns the check on."""
|
|
setting = env.get(DOCS_SWITCH, "")
|
|
if setting != "on":
|
|
return DocsConfig(setting=setting)
|
|
keep = env.get("OKF_WIKI_DOCS_KEEP", str(DEFAULT_KEEP))
|
|
if not keep.isdigit() or int(keep) < 1:
|
|
raise WatchError(f"OKF_WIKI_DOCS_KEEP is {keep!r}, not a whole number of at least 1")
|
|
bundle = env.get("OKF_WIKI_DOCS_BUNDLE", "")
|
|
|
|
def path(name: str, default: Path) -> Path:
|
|
return Path(env[name]).expanduser() if env.get(name) else default
|
|
|
|
return DocsConfig(enabled=True, setting=setting, daily=path("OKF_WIKI_DOCS_DAILY", DEFAULT_DAILY),
|
|
frozen=path("OKF_WIKI_DOCS_FROZEN", DEFAULT_FROZEN),
|
|
preamble=path("OKF_WIKI_DOCS_PREAMBLE", DEFAULT_PREAMBLE),
|
|
bundle=Path(bundle).expanduser() if bundle else None, keep=int(keep))
|
|
|
|
|
|
# --- the checks and the run ------------------------------------------------------------------------------------------
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Context:
|
|
"""What every check may use. Tests replace `fetch`, `known_positive` and `docs`; `main()` passes the real ones."""
|
|
|
|
mirror: Path
|
|
pin: str
|
|
no_fetch: bool
|
|
fetch: Fetch
|
|
known_positive: tuple[str, str] = KNOWN_POSITIVE
|
|
docs: DocsConfig = field(default_factory=DocsConfig)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Result:
|
|
"""One check's outcome: the line for the log, the report for the message ("" = nothing to say), its next state."""
|
|
|
|
summary: str
|
|
report: str
|
|
section: dict[str, Any]
|
|
|
|
|
|
Check = Callable[[Context, object], Result]
|
|
|
|
|
|
class Send(Protocol):
|
|
def __call__(self, to: str, subject: str, body: str, *, fyi: bool, dry_run: bool) -> None: ...
|
|
|
|
|
|
def check_commits(ctx: Context, section: object) -> Result:
|
|
if not (ctx.mirror / ".git").is_dir():
|
|
raise WatchError(f"mirror {ctx.mirror} is not a git repository - cannot measure anything")
|
|
if not ctx.no_fetch:
|
|
git(ctx.mirror, "fetch", "--quiet", "origin", "main")
|
|
prove_can_find(ctx.mirror, *ctx.known_positive)
|
|
|
|
head = git(ctx.mirror, "rev-parse", "origin/main")
|
|
commits = commits_between(ctx.mirror, ctx.pin, head)
|
|
paths = changed_paths(ctx.mirror, ctx.pin, head)
|
|
normative, quiet = classify(paths)
|
|
state = CommitState.from_section(section, pin=ctx.pin)
|
|
|
|
if not normative:
|
|
return Result(
|
|
summary=(
|
|
f"no contract-bearing change. {CANONICAL_REPO} at {head[:7]}, our pin {ctx.pin[:7]}, "
|
|
f"{len(commits)} commit(s) ahead, {len(quiet)} of {len(paths)} changed paths on the quiet list. "
|
|
"Known-positive passed, so this zero is a measurement."
|
|
),
|
|
report="",
|
|
section=state.to_section(),
|
|
)
|
|
fresh = decide(state, [sha for sha, _ in commits])
|
|
if not fresh:
|
|
return Result(
|
|
summary=(
|
|
f"{len(normative)} contract-bearing path(s) at {head[:7]}, already announced. "
|
|
"Silent by design - report on change, not on state."
|
|
),
|
|
report="",
|
|
section=state.to_section(),
|
|
)
|
|
state.record(fresh)
|
|
return Result(
|
|
summary=f"{len(fresh)} new commit(s) at {head[:7]}, {len(normative)} contract-bearing path(s).",
|
|
report=render_body(pin=ctx.pin, head=head, commits=commits, normative=normative, quiet=quiet),
|
|
section=state.to_section(),
|
|
)
|
|
|
|
|
|
def check_issues(ctx: Context, section: object) -> Result:
|
|
known = prove_can_find_issue(ctx.fetch)
|
|
prove_list_can_find(ctx.fetch, known)
|
|
state = IssueState.from_section(section)
|
|
listed = list_items(ctx.fetch, state.since if state else None)
|
|
|
|
if state is None:
|
|
if KNOWN_ISSUE not in listed:
|
|
raise WatchError(f"known-positive issue #{KNOWN_ISSUE} is missing from the full list. The list is broken.")
|
|
issues = sum(1 for i in listed.values() if i.kind == "issue")
|
|
return Result(
|
|
summary=(
|
|
f"first run: {issues} issues and {len(listed) - issues} pull requests registered. "
|
|
f"Known-positive #{KNOWN_ISSUE} found by its own call and by the list."
|
|
),
|
|
report="",
|
|
section=IssueState(since=max(i.updated_at for i in listed.values()), seen=dict(listed)).to_section(),
|
|
)
|
|
|
|
ordered = sorted(listed.values(), key=lambda i: i.number)
|
|
new = [i for i in ordered if i.number not in state.seen]
|
|
changed = [i for i in ordered if i.number in state.seen and state.seen[i.number].updated_at != i.updated_at]
|
|
after = IssueState(since=max([state.since, *(i.updated_at for i in ordered)]), seen={**state.seen, **listed})
|
|
|
|
if not new and not changed:
|
|
return Result(
|
|
summary=(
|
|
f"nothing new or changed since {state.since}: {len(listed)} returned, all seen before. "
|
|
f"Known-positive #{KNOWN_ISSUE} found by its own call and by the list, so this zero is a measurement."
|
|
),
|
|
report="",
|
|
section=after.to_section(),
|
|
)
|
|
return Result(
|
|
summary=f"{len(new)} new and {len(changed)} changed since {state.since}, of {len(listed)} returned.",
|
|
report=render_issue_report(state.since, len(listed), new, changed),
|
|
section=after.to_section(),
|
|
)
|
|
|
|
|
|
# --- the docs check ---------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def fetch_root(cfg: DocsConfig, work: Path, frozen_at: str) -> tuple[Path, dict[str, Any]]:
|
|
"""Fetch the tree with the fetcher of tools/fetch_docs.py and freeze it as WORK/root; its facts are returned and
|
|
written beside the sources as fetch-report.json. Anything that stops the fetch is a WatchError."""
|
|
fetcher = cfg.fetcher or fetch_docs.net_fetcher(None)
|
|
calls = received = 0
|
|
|
|
def counted(url: str) -> fetch_docs.Fetched:
|
|
nonlocal calls, received
|
|
calls += 1
|
|
answer = fetcher(url)
|
|
received += len(answer.body)
|
|
return answer
|
|
|
|
start = time.monotonic()
|
|
root = work / "root"
|
|
try:
|
|
got = fetch_docs.collect(counted, fetch_docs.load_preamble(cfg.preamble), base=cfg.base,
|
|
index_url=cfg.index_url, changelog=cfg.changelog, changelog_page=cfg.changelog_page,
|
|
pause=cfg.pause)
|
|
if got.problems:
|
|
raise WatchError(f"the fetch broke {len(got.problems)} rule(s): {'; '.join(got.problems[:5])}")
|
|
staging = work / "staging"
|
|
staging.mkdir(parents=True)
|
|
for name, data in got.files.items():
|
|
(staging / name).write_bytes(data)
|
|
meta = work / "meta.json"
|
|
meta.write_text(sources.dump(got.meta), encoding="utf-8")
|
|
sources.freeze(staging, meta, root, frozen_at)
|
|
facts = {"bytes": received, "calls": calls, "docs": got.report["docs"]["count"],
|
|
"moved_out": len(got.report["moved_out"]), "release_notes": got.report["release_notes"]["count"],
|
|
"seconds": round(time.monotonic() - start, 1)}
|
|
(root / "fetch-report.json").write_text(sources.dump({**got.report, "fetch": facts}), encoding="utf-8")
|
|
except (RunError, fetch_docs.FetchError, sources.Refused, OSError, UnicodeDecodeError, ValueError) as exc:
|
|
raise WatchError(f"the fetch of {cfg.index_url} failed after {calls} call(s): {exc}") from exc
|
|
return root, facts
|
|
|
|
|
|
def holds_frozen(folder: Path, frozen: Path) -> bool:
|
|
"""True when FOLDER is the frozen copy or holds it, after links are resolved."""
|
|
return frozen == folder.resolve() or folder.resolve() in frozen.parents
|
|
|
|
|
|
def prune(cfg: DocsConfig) -> list[str]:
|
|
"""Remove the oldest dated folders beyond the limit and the work folders of killed runs: only those names directly
|
|
under the daily folder, never a link and never a folder that is or holds the frozen copy."""
|
|
frozen = cfg.frozen.resolve()
|
|
folders = [p for p in cfg.daily.iterdir() if p.is_dir() and not p.is_symlink() and not holds_frozen(p, frozen)]
|
|
days = sorted(p for p in folders if DAY_RE.match(p.name))
|
|
removed = (days[:-cfg.keep] if len(days) > cfg.keep else []) + [p for p in folders if WORK_RE.match(p.name)]
|
|
for p in removed:
|
|
try:
|
|
shutil.rmtree(p)
|
|
except OSError as exc:
|
|
raise WatchError(f"the old fetch {p} cannot be removed: {exc}") from exc
|
|
return [p.name for p in removed]
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class DocsState:
|
|
"""The differences the last run saw (one line each) and the sha256 of them, so a message says only what is new."""
|
|
|
|
diff_sha256: str
|
|
entries: list[str]
|
|
|
|
@classmethod
|
|
def from_section(cls, section: object) -> DocsState | None:
|
|
"""None is a first run (no section, or the empty one the check writes while off). Anything else unreadable
|
|
fails: starting over would report the same differences again."""
|
|
if section is None or section == {}:
|
|
return None
|
|
bad = WatchError("state section 'docs' is malformed")
|
|
if not isinstance(section, dict):
|
|
raise bad
|
|
digest, entries = section.get("diff_sha256"), section.get("entries")
|
|
if not isinstance(digest, str) or not re.fullmatch(r"[0-9a-f]{64}", digest):
|
|
raise bad
|
|
if not isinstance(entries, list) or not all(isinstance(e, str) for e in entries):
|
|
raise bad
|
|
return cls(diff_sha256=digest, entries=list(entries))
|
|
|
|
|
|
def diff_entries(report: dict[str, Any]) -> list[str]:
|
|
"""One line per difference: "new PATH", "changed PATH NEW_SHA256" (a later change is a new line), "gone PATH"."""
|
|
lines = [f"new {e['path']}" for e in report["new_sources"]]
|
|
lines += [f"changed {e['path']} {e['new_sha256']}" for e in report["changed"]]
|
|
lines += [f"gone {e['path']}" for e in report["gone"]]
|
|
return sorted(lines)
|
|
|
|
|
|
def check_docs(ctx: Context, section: object) -> Result:
|
|
cfg = ctx.docs
|
|
if not cfg.enabled:
|
|
return Result(
|
|
summary=f"off, nothing fetched ({DOCS_SWITCH} is {cfg.setting!r}; only 'on' turns it on).",
|
|
report="",
|
|
section=section if isinstance(section, dict) else {},
|
|
)
|
|
try:
|
|
return _check_docs_on(cfg, DocsState.from_section(section))
|
|
except WatchError:
|
|
raise
|
|
except Exception as exc: # anything else would crash the whole run; docs fails alone (FAILS_ALONE)
|
|
raise WatchError(f"{type(exc).__name__}: {exc}") from exc
|
|
|
|
|
|
def _check_docs_on(cfg: DocsConfig, state: DocsState | None) -> Result:
|
|
now = cfg.clock()
|
|
day, frozen_at = now.astimezone().strftime("%Y%m%d"), now.strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
try:
|
|
problems = sources.check(cfg.frozen)
|
|
except RunError as exc:
|
|
raise WatchError(f"the frozen copy cannot be checked: {exc}") from exc
|
|
if problems:
|
|
raise WatchError(f"the frozen copy {cfg.frozen} fails the K1 check: {'; '.join(problems[:5])}")
|
|
target = cfg.daily / day
|
|
if holds_frozen(target, cfg.frozen.resolve()):
|
|
raise WatchError(f"today's folder {target} is or holds the frozen copy; refusing to replace it")
|
|
|
|
cfg.daily.mkdir(parents=True, exist_ok=True)
|
|
work = Path(tempfile.mkdtemp(prefix=f".{day}-", dir=cfg.daily))
|
|
try:
|
|
fresh, facts = fetch_root(cfg, work, frozen_at)
|
|
if target.exists():
|
|
target.rename(work / "replaced")
|
|
try:
|
|
fresh.rename(target)
|
|
except OSError:
|
|
if (work / "replaced").exists():
|
|
(work / "replaced").rename(target) # keep the day's earlier fetch
|
|
raise
|
|
finally:
|
|
shutil.rmtree(work, ignore_errors=True)
|
|
|
|
try:
|
|
report = sources.diff(cfg.frozen, target)
|
|
(target / "diff.json").write_text(sources.dump(report), encoding="utf-8")
|
|
marked: int | None = None
|
|
if cfg.bundle is not None:
|
|
marks = sources.marks(cfg.bundle, target / "diff.json")
|
|
(target / "marks.json").write_text(sources.dump(marks), encoding="utf-8")
|
|
marked = len(marks["pages"])
|
|
except (RunError, OSError, UnicodeDecodeError) as exc:
|
|
raise WatchError(f"the diff against the frozen copy failed: {exc}") from exc
|
|
removed = prune(cfg)
|
|
|
|
entries = diff_entries(report)
|
|
digest = hashlib.sha256(sources.dump({k: report[k] for k in ("changed", "gone", "new_sources", "old")})
|
|
.encode("utf-8")).hexdigest()
|
|
seen = set(state.entries) if state else set()
|
|
news = [e for e in entries if e not in seen]
|
|
resolved = sorted(seen - set(entries))
|
|
counts = (f"{len(report['new_sources'])} new, {len(report['changed'])} changed, {len(report['gone'])} gone, "
|
|
f"{report['unchanged']} unchanged")
|
|
fetched = (f"fetched {facts['docs']} docs and {facts['release_notes']} release notes ({facts['calls']} calls, "
|
|
f"{facts['bytes']} bytes, {facts['seconds']} s) into {target}")
|
|
marked_line = "" if marked is None else f"; {marked} page(s) marked in {cfg.bundle}"
|
|
kept = f"; removed {len(removed)} old fetch(es)" if removed else ""
|
|
after = {"diff_sha256": digest, "entries": entries, "fetched": day, "frozen": report["old"]}
|
|
|
|
if (state is not None and state.diff_sha256 == digest) or (not news and not resolved):
|
|
quiet = "nothing new since the last message" if state or entries else "nothing differs, nothing to report"
|
|
return Result(
|
|
summary=f"{fetched}; against the frozen copy: {counts}{marked_line}; {quiet}{kept}.",
|
|
report="",
|
|
section=after,
|
|
)
|
|
lines = [
|
|
f"The documentation fetched {day}, against the frozen copy {cfg.frozen} (frozen {report['old']['frozen_at']}):",
|
|
f" {counts}.",
|
|
f" {fetched}.",
|
|
"",
|
|
"New since the last message:" if state else "First run of this check; every difference is new:",
|
|
]
|
|
for e in news:
|
|
change, path = e.split(" ")[:2]
|
|
lines.append(f" {change:<8} {path}")
|
|
if not news:
|
|
lines.append(" (none)")
|
|
if resolved:
|
|
lines += ["", "Since the last message:"]
|
|
lines += [f" {e.split(' ')[1]} no longer differs from the frozen copy" for e in resolved]
|
|
if marked is not None:
|
|
lines += ["", f"{marked} page(s) marked in {cfg.bundle} (the list: {target / 'marks.json'})."]
|
|
lines += [
|
|
"",
|
|
f"Every difference: {target / 'diff.json'}.",
|
|
"This check only warns, and no tool calls a model. To update, build from "
|
|
f"{target} in a session with the okf-wiki skill.",
|
|
]
|
|
return Result(
|
|
summary=f"{fetched}; against the frozen copy: {counts}{marked_line}; {len(news)} new and {len(resolved)} "
|
|
f"resolved since the last message{kept}.",
|
|
report="\n".join(lines) + "\n",
|
|
section=after,
|
|
)
|
|
|
|
|
|
#: The checks, in run order. A later check joins by name here and gets its own section in the state file.
|
|
CHECKS: tuple[tuple[str, Check], ...] = (
|
|
("commits", check_commits),
|
|
("issues", check_issues),
|
|
("home", check_home),
|
|
("docs", check_docs),
|
|
)
|
|
#: Checks whose failure fails only themselves (see the module docstring).
|
|
FAILS_ALONE = frozenset({"docs"})
|
|
|
|
|
|
def chosen(names: list[str] | None) -> tuple[tuple[str, Check], ...]:
|
|
"""The checks named (all of them for None), in run order. The state of the others is kept as it is."""
|
|
return CHECKS if names is None else tuple((n, c) for n, c in CHECKS if n in names)
|
|
|
|
|
|
def file_send(path: Path) -> Send:
|
|
"""A sender that appends each message to PATH instead of sending it (trial runs that keep their state)."""
|
|
|
|
def write(to: str, subject: str, body: str, *, fyi: bool, dry_run: bool) -> None:
|
|
with path.open("a", encoding="utf-8") as f:
|
|
f.write(f"to: {to}\nfyi: {fyi}\nsubject: {subject}\n\n{body}\n")
|
|
|
|
return write
|
|
|
|
|
|
def run(ctx: Context, state_path: Path, *, dry_run: bool, send: Send, checks: tuple[tuple[str, Check], ...] = CHECKS
|
|
) -> int:
|
|
"""Run every check, then send one message for what they found, then save the state. Any failure raises first."""
|
|
sections = load_state(state_path)
|
|
results: list[tuple[str, Result]] = []
|
|
failures: list[str] = []
|
|
alone: list[tuple[str, str]] = []
|
|
for name, check in checks:
|
|
try:
|
|
results.append((name, check(ctx, sections.get(name))))
|
|
except WatchError as exc:
|
|
if name in FAILS_ALONE:
|
|
alone.append((name, str(exc)))
|
|
else:
|
|
failures.append(f"{name}: {exc}")
|
|
if failures:
|
|
raise WatchError("; ".join(failures + [f"{name}: {why}" for name, why in alone]))
|
|
|
|
for name, result in results:
|
|
print(f"okf-wiki-watch: {name}: {result.summary}")
|
|
for name, why in alone:
|
|
print(f"okf-wiki-watch: {name}: FAILED - {why}. This is NOT 'no change': nothing was measured for {name}; "
|
|
"the other checks ran and keep their state.")
|
|
news = [(name, result) for name, result in results if result.report]
|
|
if news or alone:
|
|
subject = f"OKF upstream: news from {', '.join([name for name, _ in news] + [f'{n} FAILED' for n, _ in alone])}"
|
|
parts = [f"== {name} ==\n\n{result.report}" for name, result in news]
|
|
parts += [f"== {name} ==\n\nFAILED: {why}\nThis is not 'no change'.\n" for name, why in alone]
|
|
body = _ascii("\n".join(parts) + "\nSent by tools/upstream_watch.py in okf-wiki.\n")
|
|
send(SELF_REPO, subject, body, fyi=False, dry_run=dry_run)
|
|
send(COORDINATOR_REPO, subject, body, fyi=True, dry_run=dry_run)
|
|
|
|
code = 2 if alone else 0
|
|
if dry_run:
|
|
print("okf-wiki-watch: dry run - nothing sent, state not written.")
|
|
return code
|
|
save_state(state_path, {**sections, **{name: result.section for name, result in results}})
|
|
if news:
|
|
print(f"okf-wiki-watch: reported news from {len(news)} check(s).")
|
|
return code
|
|
|
|
|
|
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--mirror", type=Path, default=Path(os.environ.get("OKF_MIRROR", DEFAULT_MIRROR)))
|
|
ap.add_argument("--state", type=Path, default=Path(os.environ.get("OKF_WIKI_WATCH_STATE", DEFAULT_STATE)))
|
|
ap.add_argument("--dry-run", action="store_true",
|
|
help="print the coord messages instead of sending; no state is written, but a docs check that is "
|
|
"on still fetches into its dated folder")
|
|
ap.add_argument(
|
|
"--no-fetch", action="store_true", help="skip the git fetch of the mirror (the issue check still calls the API)"
|
|
)
|
|
ap.add_argument(
|
|
"--check", action="append", choices=[name for name, _ in CHECKS],
|
|
help="run only this check (repeatable); the state of the others is kept as it is",
|
|
)
|
|
ap.add_argument(
|
|
"--message-file", type=Path,
|
|
help="append the message to this file instead of sending it; unlike --dry-run the state IS written",
|
|
)
|
|
ap.add_argument(
|
|
"--pin",
|
|
default=PIN,
|
|
help=(
|
|
"override the pin for ONE run. Exists so the hit path can be demonstrated on demand - a watch whose "
|
|
"reporting branch has never run is a watch nobody has seen work. IMPLIES --dry-run: it does not move the "
|
|
"pin, and it must not be able to page anyone. On 2026-08-23 this flag without --dry-run delivered two "
|
|
"false alarms in the old watch, so the pairing is enforced here rather than remembered."
|
|
),
|
|
)
|
|
args = ap.parse_args(argv)
|
|
if args.pin != PIN:
|
|
args.dry_run = True
|
|
return args
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
args = parse_args(argv)
|
|
try:
|
|
ctx = Context(mirror=args.mirror, pin=args.pin, no_fetch=args.no_fetch, fetch=http_fetch,
|
|
docs=docs_config(os.environ))
|
|
sender: Send = file_send(args.message_file) if args.message_file else send
|
|
return run(ctx, args.state, dry_run=args.dry_run, send=sender, checks=chosen(args.check))
|
|
except WatchError as exc:
|
|
print(f"okf-wiki-watch: FAILED - {exc}", file=sys.stderr)
|
|
print("okf-wiki-watch: this is NOT 'no change'. Nothing was measured this run.", file=sys.stderr)
|
|
return 2
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|