okf-wiki/okfwiki/kpack.py
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

1992 lines
107 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""K, the fragment compiler's code side: skeleton, packs, input parts, the one hard check at writing, concept identity,
main pages and homes, one source's contribution replaced or removed on its own, and the steps without a model around
the compiler. No model call (CLAUDE.md rule 1).
Usage:
python3 -m okfwiki.kpack skeleton --sources DIR --work DIR [--extend]
python3 -m okfwiki.kpack pack N --blocks FILE --sources DIR --work DIR [--size S] [--max-blocks B] [--groups-apart]
python3 -m okfwiki.kpack accept N --blocks FILE --sources DIR --work DIR [--size S] [--max-blocks B] [--groups-apart]
python3 -m okfwiki.kpack replace SOURCE --fragments FILE --blocks FILE --sources DIR --work DIR [--size S]
python3 -m okfwiki.kpack remove SOURCE --work DIR
python3 -m okfwiki.kpack run-before N [--to M] --blocks FILE --sources DIR --work DIR [--size S] [--max-blocks B]
[--groups-apart]
python3 -m okfwiki.kpack run-after N [--to M] --blocks FILE --sources DIR --work DIR --out WIKI [--links FILE]
[--at ISO] --transcripts DIR --okter FILE --kost FILE --cap C --total T [--name NAME] [--size S]
[--max-blocks B] [--groups-apart] [--floor FILE]
python3 -m okfwiki.kpack cited --work DIR --blocks FILE --out FILE
python3 -m okfwiki.kpack floor --cited FILE --blocks-then FILE --sources-then DIR --blocks FILE --sources DIR
[--history-rate R] --out FILE
SOURCES holds kilde/ and, in a frozen copy, source-manifest.json (contract K1). K1 decides each source's kind and main
topic; without K1 the file name does, as before (release-vX.Y.Z.md is a release note, every other file a page with a main
topic). A source without a main topic (a release note, or a page K1 marks as history, such as the weekly What's New
pages) is history: it has no main page, its fragments attach to existing concepts first and go under "## History", and
a new concept from it needs a home like any page. --blocks is kblocks' blocks file. The work folder holds K-oppgaver/
(task parts), K-fragments/ (the compiler's JSON), K-work/ (index.json, accepted.json, packs.json, and backup/ with the
files as they were before the last write), K-audit/ (the auditor's JSON and the subset root it ran on) and K-guard/
(the guard's tasks and answers).
skeleton the concept index from the titles and level 2-3 headings of every docs page with a main topic that pass
concept test 2, one entry per normalised name (the first page in name order wins); prints how many headings
collide with a heading of the same name in another file. An index, not pages. With --extend, the index
of a built collection gains the docs pages with a main topic that are in no pack packs.json registers and
were not extended before: a name of theirs is added by the same rule, an existing entry never changes and
gets no new alias, and a name equal to an existing one is not added and is counted.
pack N pack N's input for the compiler: the sources, each marked "name its main page" or as history; every block of
its sources with its anchor (file › section › span) and numbered lines; the index lines whose normalised name
or alias occurs as a phrase in the normalised text of those sources (a phrase filter, not a word filter),
each marked "main page" or "has a home" when it is; the main pages a home may be chosen from, filtered for
the pack (main_catalogue); and the output contract, in parts of at most PART_CHARS characters (the Read
tool's cap). Prints the part paths and the outputs: one for a pack of OUTPUT_PARTS parts or fewer, else
part 1 alone and then one for every OUTPUT_PARTS parts, each with the parts it holds, which the compiler
writes one per turn (one answer holds only so much: round 1's pack 1 hit the limit with seven parts in one
output, and the first answer also holds the thinking).
accept N checks pack N's fragments (validate: the one hard check at writing), resolves each to a concept by code
(existing, merge candidate, new or rejected), updates the index and appends the accepted fragments, each
with the context quotes it names placed on a heading of its path or a line of its section (context_of; at
most MAX_CONTEXT, the rest dropped and counted, never a reject); a merged fragment that keeps none cites
the other lines of its anchor (anchor_context, at most MERGED_LINES, nearest the quote). Then it
records the main page the compiler names for each docs source of the pack (kept only when that page has a
fragment from the source) and a home for every page of the pack that has neither a main role nor a home
(kept only when it names a main page). A page without a home is accepted and listed, never rejected; the
guard asks for it after the auditor has run. Prints "accepted X of N", the rejections by reason, the id
mismatches (counted, never rejected), how many accepted fragments resolved to a concept whose index line was
not sent with the pack, the main pages and the pages without a home.
replace S checks a new set of fragments for source S against S alone (its blocks in --blocks), then puts them in the
place of S's accepted fragments; every other source's records stay as they were, in place. S keeps the
pack and position it had when its pack was accepted, even when the order has changed since; a source in
no accepted pack cannot be replaced (it joins the build through pack and accept). S's main page records
are swapped with its fragments.
remove S takes S's accepted fragments and main page records out; every other record stays as it was.
A home belongs to its page, one per page, and the model's newest choice wins: replace takes the homes the
new fragments give S's pages, and a page S wrote to that gets none keeps the home it had. After replace,
remove, accept and the guard's answer, a main page record that points to a page now gone is dropped, and
so is every home record no longer valid (its page or its home gone, its home no longer a main page, its
page now a main page, or a newer home of the page); a dropped record never comes back, and the pages left
without a home are named. A home record names the source that brings its page and passes on when that
source goes (counted), so a page another source keeps never loses its home with S.
run-before N [--to M]
pack for N..M, every pack rendered before any is written. An interval of more than one pack gives the later
packs the index of now, without the concepts and main pages of the packs before them in the interval, and
the cost is checked only after the interval: a build runs one pack at a time.
run-after N [--to M]
the steps after the compiler. For each pack not yet accepted, its outputs read together and the hard
check: a fragment rejected at writing is dropped and counted at accept, and the pack does not run again for
it; an output left unwritten sets the pack's outputs aside and exits 3 (run the compiler once more, same
parts, same outputs); an output left unwritten again, exit 6 (STOP). Then accept. Then assemble (kassemble build, --links placed), run the auditor on a
subset root of the sources of packs 1..M (K1 rule 9), so sources not yet built never count, and act on its
findings (link-rule is retired in Step 7 and left out): none, go on; only pages without a link from a main
page and sources without a main page, the guard's task is written (exit 4: run it, then run-after again,
which takes its answer, assembles and audits again); any finding after the guard, or one it cannot mend,
exit 5 (STOP, listed: K2 point 9, there is no exception list). Last the stop line: cost harvest, then
cost stop --lines build,link,orchestration-build --unit blocks --blocks K-work/packs.json with --done the
blocks of packs 1..M, --next the blocks of pack M+1 (0 after the last) and --next-chars their characters
(when the blocks file has them; each pack's record keeps its own as block_chars), so measured cost and the next
pack stop and the projection is reported only (the operator 12.10.2026); a non-zero exit of either is STOP, each with its own reason (exit 1 over the cost
stop or for a run the harvest cannot attribute, 2 when the harvest or the stop line cannot run). Prints
counts, and the ids of pages and sources only where a missing home, main page or a stop needs them; never
text from a source. With --floor (a file cited wrote for an earlier build of the same sources), after the
stop line every source of the packs that the floor names must cite at least its floor's anchor blocks;
one that cites fewer is named and exit 7 (STOP: condition b of wave 4's goal, checked after every pack).
A floor file may also give history_rate: the history sources of each pack (release notes, weekly pages),
which have no floor of their own, cite at least floor(their blocks x rate / 2) anchor blocks together, or
exit 7 (order 28 A4). A page without a home that the operator has approved as a reported exception, on
the list HOMELESS_APPROVED in the work folder (page, pack, date and his words; no code or model writes
it), no longer stops run-after and is not asked of the guard; the auditor still reports it, every other
finding stops as before, and it never gets a forced link (order 28 A5). A main-topic source no page cites
(a document with no readable content, so no fragment and no main page) that he has approved goes on the
same list under sources (source, pack, date and his words) and is treated the same way: reported, never
given a forced fragment or main page (the operator 13.11.2026). After such a stop and his approval,
run-after N again goes on from where the pack stopped.
cited per source, the distinct blocks that hold the anchor of an accepted fragment, from the work folder's
accepted.json and the blocks file of that build: the floor for a later build of the same sources.
floor the floor for a later build of sources that changed since (a new fetch), from the file cited wrote:
floor(cited x the source's blocks now / its blocks then / 2), counting only blocks that can carry a claim
(a thematic break or a paragraph that is only bold text carries none); a history source gets none of its
own, and --history-rate R is written as the file's history_rate.
The packs are --size sources each (5 unless the work folder already records a size). --max-blocks B closes a new pack
before the source that would take it over B blocks (a source over B alone is a pack of its own); --max-chars C does
the same by the characters of the blocks (B17; no cap unless given or recorded), and run-before names a source
larger than LARGEST_HELD, the largest measured to hold in one compiler run ("large: ...", it stops nothing); --groups-apart
keeps docs pages and release notes in separate new packs (order 28 A2); both are registered in packs.json like the
size, so later commands lay out the same packs without them. A pack number outside 1..N cannot run.
A built collection is continued (order 27 A2): run kblocks order and blocks for the larger root to new paths, then
skeleton --extend. The packs packs.json registers keep their sources, their place and their size; the sources of the
order in no registered pack follow in new packs after the last, docs pages before release notes, each group in K's
order, and --size sets the size of a new pack (registered with it). The first new pack starts with run-before alone:
the links of the pack before it are placed already, so its chain leaves out that pack's pass-after. Every write first copies the K-work files it replaces to K-work/backup/ and sets a
marker K-work/<command>.pending that names the files to copy back, the new files to delete and the files to move
back (a rejected pack set aside), then moves and writes (a pack's task parts too), then deletes the marker; a command
that finds a marker cannot run, and says what to undo. Exit 0 =
done, 1 = STOP by the stop line, 2 = could not run (also for an input the code did not foresee, with its traceback),
3 = run the compiler for the pack once more, 4 = run the guard, 5 = STOP: a finding stands after the guard or cannot be
mended by it, 6 = STOP: an output of the pack was left unwritten twice, 7 = STOP: a source cites fewer anchor blocks
than its floor.
"""
from __future__ import annotations
import argparse
import io
import json
import os
import re
import shutil
import sys
import traceback
from collections import Counter
from collections.abc import Callable, Mapping
from contextlib import redirect_stderr, redirect_stdout
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from okfwiki import audit, cost, klink
from okfwiki import sources as k1
from okfwiki.audit import sentence_spans
from okfwiki.common import RunError, jaccard, slug
from okfwiki.kblocks import RELEASE, docs_headings, headings_of, is_meta, kilde_of, read_lines, source_files
from okfwiki.kblocks import order as kblocks_order
PART_CHARS = 60_000
MERGE_JACCARD = 0.6
DEFAULT_SIZE = 5
LARGEST_HELD = 387_767 # the largest source measured to hold in one compiler run: one PDF of a real folder, 07.11
# (7 task parts, 4 outputs, 342 of 345 accepted, auditor 0, context at most 281k tokens). A
# measurement, not a limit: run-before names a larger source and stops nothing (B17)
PROVENANCE = ("documented", "inferred", "merged")
MAX_CONTEXT = 2 # context quotes kept per fragment (order 22: O or L is one missing piece; both together are two)
MERGED_LINES = 8 # lines of its anchor a merged fragment without context cites (order 25: all 20 merged claims of round 1
# unsupported, all L, 19 without context; the whole anchor reached 95 of 118 missing words, the nearest
# eight 88, and an anchor of up to 47 lines would hang 46 footnotes on one sentence)
FENCE_LINE = re.compile(r"^\s*(```|~~~)")
OUTPUT_PARTS = 2 # task parts per output after the first (order 22 A2: one answer holds only so much; round 1's pack 1,
# seven parts in one output, hit the limit and was written again with 84 fragments where wave 3 wrote
# 447). The first output holds part 1 alone: the first answer also holds the thinking (610 s before
# the cut, 49 s after it).
CITED_SCHEMA = "okf-wiki/cited-blocks/1"
REASONS = ("malformed", "anchor-outside-pack", "anchor-crosses-blocks", "sentence-count", "source-missing",
"quote-missing", "quote-not-in-span", "provenance-missing", "provenance-unknown", "tests-missing",
"release-note-new", "meta")
HOMELESS_APPROVED = "homeless-approved.json" # in the work folder; only the operator changes it (order 28 A5)
HOMELESS_SCHEMA = "okf-wiki/homeless-approved/1"
RESERVED_IDS = frozenset({"index", "log"}) # index.md and log.md are the bundle's own files (SPEC 3.1)
INDEX_SCHEMA = "okf-wiki/concept-index/1"
ACCEPTED_SCHEMA = "okf-wiki/accepted/1"
PACKS_SCHEMA = "okf-wiki/packs/1"
MANIFEST = "source-manifest.json"
MANIFEST_SCHEMA = "okf-wiki/source-manifest/1"
WEEK = re.compile(r"^docs-.*-(\d{4})-w(\d{2})\.md$")
STOP_LINES = "build,link,orchestration-build"
RETIRED = frozenset({"link-rule"}) # inbound from any page; Step 7 retires it (K2 point 5)
EXIT_STOP = 1 # the stop line, as okfwiki.cost stop gives it
EXIT_RETRY = 3
EXIT_GUARD = 4
EXIT_BREACH = 5
EXIT_REJECTED_TWICE = 6
EXIT_THIN = 7 # a source cites fewer anchor blocks than its floor (order 22 A2, condition b of wave 4's goal)
CONTRACT = """## Output contract
Write one JSON object `{"fragments": [...], "main_pages": [...], "homes": [...]}` to each output path named in your
prompt. When it names more than one, each names the parts it holds: write them one at a time, each in its own turn and
in the order given, each with the fragments of the blocks in the parts it names; `main_pages` and `homes` may stand in
any of them. One answer holds only so much, and an output left unwritten makes the whole pack run again. Each
fragment has:
- `concept` and `status`: "existing" with the index `id`, or "new" with `concept` its title as a reader would write it,
in the language and letters of the source (never an id), and `tests`: {"topic_shape", "citation"}.
- `text`: one to three sentences.
- `anchor`: {"file", "start", "end"} inside one block above.
- `quote`: one piece copied character for character from one line inside the anchor. Every fragment has a quote.
Code cites the whole line it stands on, and each sentence of `text` says only what that line and its context say.
- `context`: the other lines the text draws on, at most two pieces, each copied character for character from a
heading of the anchor's path (shown after the file name) or from another line of the same section: the subject a
heading names, or the condition, step or setting another line states. Leave it out only when the text says nothing
that its quote's line does not. Code places each on that heading or line and cites it too; a piece it cannot place
is dropped.
- `provenance`, one of three values, and the quote is required with each of them:
"documented" when the text says what its quote says;
"inferred" when the text draws a conclusion from lines of its anchor that no single line states;
"merged" when the text combines what several lines of its anchor say; name those lines in `context`, or code
cites the other lines of the anchor, the eight nearest the quote.
`main_pages`: one {"source", "concept"} for each source marked "name its main page": the concept that is that source's
main topic, named as your fragments from that source name it. A history source has no main page.
`homes`: one {"concept", "home"} for each concept you write a fragment for that is "new", or whose index line is
marked neither "main page" nor "has a home", unless you name it in `main_pages`. The `home` is an id under
"Main pages", or a concept you name in `main_pages`: the main page under which a reader would look for this concept.
A new concept from a history source needs a home like any other page; without one it is asked for again, and
the build stops when it has none.
"""
# --- the sources (contract K1) ---------------------------------------------------------------------------------
@dataclass(frozen=True)
class Fact:
"""What K1, or without it the file name, says about one source."""
kind: str
main_topic: bool
version: str | None = None
original: str | None = None # the path the document has in its folder (sources folder), when K1 keeps it
places: tuple[dict[str, Any] | None, ...] | None = None # the page, paragraph, cell or slide of each line (PDF, Office)
def place(self, line: int) -> dict[str, Any]:
"""The place of line (1-based) in the original document; {} for a text source or a line without one."""
if self.places is None or not 0 < line <= len(self.places):
return {}
return dict(self.places[line - 1] or {})
@property
def history(self) -> bool:
"""A source without a main topic adds history to concepts and has no main page (K1 rule 4)."""
return not self.main_topic
def name_fact(name: str) -> Fact:
m = RELEASE.match(name)
return Fact("release-notes", False, m.group(1)) if m else Fact("docs", True)
def read_k1(sources: Path) -> dict[str, Fact] | None:
"""The facts of every source K1 lists, by file name under kilde/; None when the root has no K1."""
path = sources / MANIFEST
if not path.is_file():
return None
try:
doc = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
raise RunError(f"{path} cannot be read as K1: {exc}") from exc
if not (isinstance(doc, dict) and doc.get("schema") == MANIFEST_SCHEMA and isinstance(doc.get("sources"), list)):
raise RunError(f"{path} is not a K1 source manifest ({MANIFEST_SCHEMA} with a list of sources)")
facts: dict[str, Fact] = {}
for e in doc["sources"]:
if not (isinstance(e, dict) and isinstance(e.get("path"), str) and e["path"].startswith("kilde/")
and isinstance(e.get("kind"), str) and isinstance(e.get("main_topic"), bool)):
raise RunError(f"{path}: an entry without a kilde/ path, a kind or a boolean main_topic: {e!r}")
version, original, places = e.get("version"), e.get("original"), e.get("places")
facts[e["path"][len("kilde/"):]] = Fact(e["kind"], e["main_topic"], version if isinstance(version, str) else None,
original if isinstance(original, str) else None,
tuple(places) if isinstance(places, list) else None)
return facts
def source_facts(sources: Path, names: list[str]) -> dict[str, Fact]:
"""K1 decides kind and main topic when the sources root has it; the file name decides otherwise, as before."""
listed = read_k1(sources)
if listed is None:
return {n: name_fact(n) for n in names}
missing = [n for n in names if n not in listed]
if missing:
raise RunError(f"{sources / MANIFEST} does not list {', '.join(missing[:5])}")
return {n: listed[n] for n in names}
def week_of(name: str) -> str | None:
"""The week of a weekly page from its file name (docs-...-YYYY-wNN.md -> YYYY-wNN)."""
m = WEEK.match(name)
return f"{m.group(1)}-w{m.group(2)}" if m else None
def source_line(name: str, fact: Fact) -> str:
"""A source as the pack's task lists it: its name, the name it has in its folder when that differs (the safe name
of a folder's document says less, and a text without a heading has no other title), and what it is."""
shown = f"{name} (original name: {fact.original})" if fact.original and fact.original != name else name
return f"{shown}: {describe(name, fact)}"
def describe(name: str, fact: Fact) -> str:
if fact.main_topic:
return f"{fact.kind} page; name its main page"
week, version = week_of(name), fact.version or name_fact(name).version
label = f"week {week}" if week else f"version {version}" if version else fact.kind
return f"history ({label}); no main page"
# --- names and identity ----------------------------------------------------------------------------------------
def norm(name: str) -> str:
text = name.replace("`", "").lower()
text = re.sub(r"[^\w\s/.\-]", " ", text)
words = text.split()
if words and words[0] == "the":
words = words[1:]
words = [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
return " ".join(w.strip(".") or w for w in words)
def unique_id(index: dict[str, Any], title: str) -> str:
taken = {e["id"] for e in index["entries"]} | RESERVED_IDS
base = slug(title)
eid, k = base, 2
while eid in taken:
eid, k = f"{base}-{k}", k + 1
return eid
def add_headings(index: dict[str, Any], kilde: Path, files: set[str]) -> tuple[int, int, int]:
"""Add an entry to the index for each name of the title and level 2-3 headings of `files` that passes concept test
2, one per normalised name (the first page in name order wins, a later spelling becomes an alias of the new entry).
A name equal to a name of an entry already in the index is not added, and that entry is never changed. (entries
added, names equal to an existing entry, headings that collide with a heading of the same name in another file)."""
existing = {m for e in index["entries"] for m in _names(e)}
seen: dict[str, dict[str, Any]] = {}
added = equal = collisions = 0
for h in docs_headings(kilde):
if is_meta(h.text) or h.file not in files:
continue
n = norm(h.text)
if not n:
continue
if n in existing:
equal += 1
continue
if n in seen:
if seen[n]["origin"]["file"] != h.file:
collisions += 1
if h.text not in seen[n]["aliases"]:
seen[n]["aliases"].append(h.text)
continue
e = {"id": unique_id(index, h.text), "title": h.text, "aliases": [h.text], "kind": "skeleton",
"merge_of": None, "origin": {"file": h.file, "line": h.line}}
index["entries"].append(e)
seen[n] = e
added += 1
return added, equal, collisions
def skeleton(kilde: Path, history: frozenset[str] = frozenset()) -> tuple[dict[str, Any], int]:
"""(the index, the number of headings that share a normalised name with a heading in another file). The files in
`history` (pages K1 marks as history) are left out: history adds to concepts and sets no structure."""
index: dict[str, Any] = {"schema": INDEX_SCHEMA, "entries": []}
pages = {p.name for p in source_files(kilde) if not RELEASE.match(p.name)}
_, _, collisions = add_headings(index, kilde, pages - history)
return index, collisions
def _names(e: dict[str, Any]) -> set[str]:
return {norm(a) for a in [e["title"], *e.get("aliases", [])]} - {""}
def _similar(a: str, b: str) -> float:
return jaccard(set(a.split()), set(b.split()))
def resolve(fragment: dict[str, Any], index: dict[str, Any], history: bool | None = None) -> tuple[str, str | None]:
"""existing <id> | merge-candidate <id> | new | rejected <reason> (D6, D3b). A fragment from a history source
(`history`; by default a release-* file) that would make a new concept is rejected as release-note-new here;
accept resolves it again after the pack's other fragments and keeps it, homeless when it has no home."""
by_id = {e["id"]: e for e in index["entries"]}
if fragment.get("status") == "existing" and fragment.get("id") in by_id:
return "existing", str(fragment["id"])
if fragment.get("concept") in by_id: # an index id given as the concept, as main_pages and homes read it (lookup)
return "existing", str(fragment["concept"])
n = norm(str(fragment.get("concept", "")))
for e in index["entries"]:
if n in _names(e):
return "existing", str(e["id"])
if history is None:
history = RELEASE.match(str(fragment.get("anchor", {}).get("file", ""))) is not None
if history:
return "rejected", "release-note-new"
if is_meta(str(fragment.get("concept", ""))):
return "rejected", "meta"
best = max(((max(_similar(n, m) for m in _names(e)), str(e["id"])) for e in index["entries"] if _names(e)),
default=(0.0, ""))
if best[0] >= MERGE_JACCARD:
return "merge-candidate", best[1]
return "new", None
def lookup(index: dict[str, Any], name: Any) -> str | None:
"""The concept a name stands for: an index id as it is, else the entry whose normalised names hold it."""
if not isinstance(name, str) or not name.strip():
return None
if any(e["id"] == name for e in index["entries"]):
return name
n = norm(name)
return next((str(e["id"]) for e in index["entries"] if n and n in _names(e)), None)
def id_mismatch(fragment: dict[str, Any], index: dict[str, Any]) -> bool:
"""An "existing" fragment whose id names no entry, or an entry whose names match its concept neither by normalised
form nor by Jaccard >= MERGE_JACCARD (rev 4 Step 4: 0 of 650 in P1). Counted at accept, never a reject."""
if fragment.get("status") != "existing":
return False
if "id" not in fragment and any(e["id"] == fragment.get("concept") for e in index["entries"]):
return False # the concept is the index id itself (resolve)
entry = next((e for e in index["entries"] if e["id"] == fragment.get("id")), None)
if entry is None:
return True
concept = norm(str(fragment.get("concept", "")))
return concept not in _names(entry) and not any(_similar(concept, m) >= MERGE_JACCARD for m in _names(entry))
def relevant_lines(index: dict[str, Any], text: str) -> list[dict[str, Any]]:
"""The index entries whose normalised name or alias occurs as a whole phrase in the normalised text (D7)."""
hay = f" {norm(text)} "
return [e for e in index["entries"] if any(f" {m} " in hay for m in _names(e))]
# --- main pages and homes in the accepted state ----------------------------------------------------------------
def pages_of(acc: dict[str, Any]) -> set[str]:
"""The concepts that have a page: those with at least one accepted fragment."""
return {str(r["concept_id"]) for r in acc["fragments"]}
def main_ids(acc: dict[str, Any]) -> list[str]:
"""The main pages: concepts named for a source in main_pages that have a page, in record order, each once."""
pages = pages_of(acc)
out: list[str] = []
for m in acc.get("main_pages", []):
cid = str(m["concept_id"])
if cid in pages and cid not in out:
out.append(cid)
return out
def homed_ids(acc: dict[str, Any], mains: set[str], pages: set[str]) -> set[str]:
"""The concepts with a home record whose home is a main page, both with a page."""
return {str(h["concept_id"]) for h in acc.get("homes", [])
if str(h["home"]) in mains and str(h["home"]) in pages and str(h["concept_id"]) in pages}
def tags_for(acc: dict[str, Any]) -> dict[str, str]:
mains = set(main_ids(acc))
tags = {cid: "has a home" for cid in homed_ids(acc, mains, pages_of(acc))}
tags.update({cid: "main page" for cid in mains})
return tags
def first_files(acc: dict[str, Any]) -> dict[str, str]:
"""Per concept, the source of its first fragment in build order (pack, position, line): the source that brought
the page, which a home record given for it by the guard goes with."""
out: dict[str, str] = {}
for r in sorted(acc["fragments"], key=lambda r: (int(r["pack"]), int(r["position"]), int(r["start"]))):
out.setdefault(str(r["concept_id"]), str(r["file"]))
return out
def pass_homes(acc: dict[str, Any]) -> int:
"""A home belongs to its page; its record names the source that brings the page (first_files). After replace or
remove, a record whose page stays passes to the page's source now; the number passed."""
firsts = first_files(acc)
passed = 0
for h in acc["homes"]:
src = firsts.get(str(h["concept_id"]))
if src is not None and h["source"] != src:
h["source"] = src
passed += 1
return passed
def drop_dangling(acc: dict[str, Any]) -> tuple[int, int, list[str]]:
"""Drop the main page records that point to a page now gone, and every home record that is no longer valid: its
page or its home is gone, its home is no longer a main page (or is the page itself), its page is now a main page,
or a newer record of the same page follows it (the model's newest choice wins, one home per page). A record that
was dropped never comes back. Run after every write. (main page records dropped, home records dropped, the pages
a dropped home leaves without one, sorted)."""
pages = pages_of(acc)
mains = [m for m in acc["main_pages"] if str(m["concept_id"]) in pages]
acc_mains = {**acc, "main_pages": mains}
main_set = set(main_ids(acc_mains))
seen: set[str] = set()
homes: list[dict[str, Any]] = []
for h in reversed(acc["homes"]):
c, t = str(h["concept_id"]), str(h["home"])
if c in pages and t in main_set and t != c and c not in main_set and c not in seen:
homes.append(h)
seen.add(c)
homes.reverse()
left = sorted({str(h["concept_id"]) for h in acc["homes"]} & (pages - main_set - seen))
dropped = (len(acc["main_pages"]) - len(mains), len(acc["homes"]) - len(homes), left)
acc["main_pages"], acc["homes"] = mains, homes
return dropped
def left_without(acc: dict[str, Any], mains_before: set[str], dropped: list[str]) -> list[str]:
"""The pages left without a home: those a dropped home left (drop_dangling), and those that lost their main role
and have no home; each still a page and no main page, sorted."""
pages, mains = pages_of(acc), set(main_ids(acc))
homed = {str(h["concept_id"]) for h in acc["homes"]}
lost = {c for c in mains_before if c in pages and c not in mains and c not in homed}
return sorted(set(dropped) | lost)
def catalogue_line(cid: str, title: str) -> str:
return f"- {cid} | {' '.join(title.split())}\n"
def main_catalogue(index: dict[str, Any], acc: dict[str, Any], pack: int, budget: int = klink.CATALOGUE_CHARS
) -> list[dict[str, Any]]:
"""The main pages a home may be chosen from for pack N (Retning point 4: the catalogue filtered per pack), ranked
as the link pass ranks its catalogue and cut at the first that does not fit the character budget: (0) written to
in this pack, (1) in the RECENT_PACKS packs before it, latest first, (2) the rest by how many distinct source
files they cite, most first; ties by id. Structure only, never names."""
titles = {str(e["id"]): str(e.get("title") or e["id"]) for e in index["entries"]}
last: dict[str, int] = {}
files: dict[str, set[str]] = {}
for r in acc["fragments"]:
cid, p = str(r["concept_id"]), int(r["pack"])
files.setdefault(cid, set()).add(str(r["file"]))
if p <= pack:
last[cid] = max(last.get(cid, 0), p)
def rank(cid: str) -> tuple[int, int, str]:
lp = last.get(cid)
if lp == pack:
return 0, 0, cid
if lp is not None and pack - lp <= klink.RECENT_PACKS:
return 1, -lp, cid
return 2, -len(files.get(cid, set())), cid
out: list[dict[str, Any]] = []
used = 0
for cid in sorted(main_ids(acc), key=rank):
size = len(catalogue_line(cid, titles.get(cid, cid)))
if used + size > budget:
break
out.append({"id": cid, "title": titles.get(cid, cid)})
used += size
return out
# --- packs and the compiler's input ----------------------------------------------------------------------------
def packs(names: list[str], size: int = DEFAULT_SIZE) -> list[list[str]]:
return [names[i:i + size] for i in range(0, len(names), size)]
def group(name: str) -> str:
"""A source's group in K's order: release notes, or docs pages (the weekly pages among them)."""
return "release" if RELEASE.match(name) else "docs"
def new_packs(rest: list[str], size: int, cap: int | None, apart: bool, blocks: Mapping[str, int],
char_cap: int | None = None, chars: Mapping[str, int] | None = None) -> list[list[str]]:
"""The sources in no registered pack, in packs (order 28 A2): a pack takes the next source while it holds fewer
than `size`, the source is of its group when `apart`, its blocks stay within `cap` and the characters of its
blocks within `char_cap` (B17); a source over a cap alone is a pack of its own."""
out: list[list[str]] = []
pack: list[str] = []
held = held_chars = 0
for name in rest:
w, wc = blocks.get(name, 0), (chars or {}).get(name, 0)
if pack and (len(pack) >= size or (apart and group(name) != group(pack[0]))
or (cap is not None and held + w > cap)
or (char_cap is not None and held_chars + wc > char_cap)):
out.append(pack)
pack, held, held_chars = [], 0, 0
pack.append(name)
held, held_chars = held + w, held_chars + wc
return out + ([pack] if pack else [])
def layout(order: list[str], log: dict[str, Any], size: int, cap: int | None = None, apart: bool = False,
blocks: Mapping[str, int] | None = None, char_cap: int | None = None,
chars: Mapping[str, int] | None = None) -> list[list[str]]:
"""Every pack's sources (order 27 A2: a built collection is continued, never packed again). A pack packs.json
registers (a record with its sources, written by run-before or accept) keeps its sources and its place; the sources
of the order in no registered pack fill the other pack numbers in turn, `size` at a time, in K's order (docs pages
before release notes, kblocks.order), so after the last registered pack when packs 1..K are registered. Without a
registered pack this is packs(order, size) for K's order. With `apart` a new pack never holds docs pages and
release notes together, and with `cap` (and the blocks per source) it closes before the source that would take it
over the cap (order 28 A2). A registered source the order lacks, a source registered twice, or a pack number left
without sources cannot run."""
if cap is not None and blocks is None:
raise RunError("a cap of blocks per pack needs the blocks of each source")
in_order = set(order)
registered: dict[int, list[str]] = {}
seen: set[str] = set()
for r in log.get("packs", []):
names = r.get("sources")
if not isinstance(names, list):
continue
for name in names:
if name not in in_order:
raise RunError(f"pack {r['pack']}'s source {name} is not in the order")
if name in seen:
raise RunError(f"{name} is registered in two packs")
seen.add(name)
registered[int(r["pack"])] = [str(x) for x in names]
rest = new_packs(kblocks_order([n for n in order if n not in seen]), size, cap, apart, blocks or {}, char_cap,
chars)
out: list[list[str]] = []
last = max(registered, default=0)
while rest or len(out) < last:
n = len(out) + 1
if n in registered:
out.append(registered[n])
elif rest:
out.append(rest.pop(0))
else:
raise RunError(f"pack {n} has no sources: the order holds none that is not registered in a pack")
return out
def blocks_per_source(doc: dict[str, Any]) -> Counter[str]:
return Counter(str(b["file"]) for b in doc["blocks"])
def chars_per_source(doc: dict[str, Any]) -> Counter[str]:
"""The characters of each source's blocks (kblocks `chars`; 0 in a blocks file written before blocks had them)."""
out: Counter[str] = Counter()
for b in doc["blocks"]:
out[str(b["file"])] += int(b.get("chars") or 0)
return out
def next_pack(doc: dict[str, Any], log: dict[str, Any], last: int, size: int, done: int, total: int,
cap: int | None = None, apart: bool = False, char_cap: int | None = None,
chars: Mapping[str, int] | None = None) -> list[str]:
"""The sources of the pack after `last`, which the stop line counts on before the next check; none when the
run's blocks are done (done >= total) or no pack follows. A root may hold sources after the run's last pack
(order 27: variants S and R do not build the release notes of their root)."""
if done >= total:
return []
all_packs = layout([str(x) for x in doc["order"]], log, size, cap, apart, blocks_per_source(doc), char_cap,
chars_per_source(doc) if chars is None else chars)
return all_packs[last] if last < len(all_packs) else []
def next_blocks(doc: dict[str, Any], log: dict[str, Any], last: int, size: int, done: int, total: int,
cap: int | None = None, apart: bool = False, char_cap: int | None = None) -> int:
names = next_pack(doc, log, last, size, done, total, cap, apart, char_cap)
return sum(b["file"] in names for b in doc["blocks"])
def pack_chars(doc: dict[str, Any], names: list[str]) -> int | None:
"""The characters of the blocks of `names` (kblocks `chars`), or None for a blocks file written before blocks
had characters: the unit the stop line prices the next pack by (motor B17)."""
mine = [b for b in doc["blocks"] if b["file"] in names]
return sum(int(b["chars"]) for b in mine) if all(isinstance(b.get("chars"), int) for b in mine) else None
def render_input(pack_no: int, names: list[str], blocks_doc: dict[str, Any], index_lines: list[dict[str, Any]],
kilde: Path, facts: dict[str, Fact] | None = None, tags: dict[str, str] | None = None,
catalogue: list[dict[str, Any]] | None = None) -> list[str]:
facts = facts or {n: name_fact(n) for n in names}
lines = {n: (kilde / n).read_text(encoding="utf-8").split("\n") for n in names}
pieces: list[str] = [f"# K pack {pack_no}: {len(names)} sources\n\nSources in order: {', '.join(names)}\n\n"
+ "## Sources\n\n" + "".join(f"- {source_line(n, facts[n])}\n" for n in names) + "\n"
+ CONTRACT + "\n## Blocks\n\n"]
for b in blocks_doc["blocks"]:
if b["file"] not in names:
continue
head = f"### {b['file']} › {b['section'] or '-'} › {b['start']}–{b['end']}"
if b.get("version"):
head += f" (version {b['version']})"
body = [f"{n}: {lines[b['file']][n - 1]}" for n in range(b["start"], b["end"] + 1)]
chunk = head + "\n"
for row in body:
if len(chunk) + len(row) + 1 > PART_CHARS // 2:
pieces.append(chunk + "\n")
chunk = head + " (continued)\n"
chunk += row + "\n"
pieces.append(chunk + "\n")
pieces.append("## Index (concepts named in these sources)\n\n")
for e in index_lines:
aliases = "; ".join(a for a in e["aliases"] if a != e["title"])
tag = (tags or {}).get(str(e["id"]))
pieces.append(f"- id: {e['id']} | {e['title']}" + (f" | aliases: {aliases}" if aliases else "")
+ (f" | {tag}" if tag else "") + "\n")
pieces.append("\n## Main pages (homes you may choose)\n\n")
if catalogue:
pieces += [catalogue_line(str(e["id"]), str(e["title"])) for e in catalogue]
else:
pieces.append("(none yet: the main pages you name in this pack are the homes you may choose)\n")
room = PART_CHARS - len("<!-- part 99 of 99 -->\n")
parts: list[str] = []
current = ""
for piece in pieces:
if current and len(current) + len(piece) > room:
parts.append(current)
current = ""
current += piece
if current:
parts.append(current)
total = len(parts)
return [f"<!-- part {i} of {total} -->\n" + p for i, p in enumerate(parts, 1)]
# --- the one hard check at writing -----------------------------------------------------------------------------
def validate(fragments: list[Any], names: list[str], blocks_doc: dict[str, Any], kilde: Path
) -> tuple[list[dict[str, Any]], Counter[str]]:
"""The fragments that pass, each with its quote line, version, section and position in `names`; and the rest
counted by the first reason that rejects them. Rule E: every fragment carries a quote that stands verbatim on one
line of its anchor span, and a provenance mark from PROVENANCE; the mark never exempts the quote."""
by_file: dict[str, list[dict[str, Any]]] = {}
for b in blocks_doc["blocks"]:
by_file.setdefault(b["file"], []).append(b)
texts: dict[str, list[str] | None] = {}
accepted: list[dict[str, Any]] = []
rejected: Counter[str] = Counter()
for f in fragments:
a = f.get("anchor") if isinstance(f, dict) else None
if not (isinstance(f, dict) and isinstance(f.get("concept"), str) and f["concept"].strip()
and f.get("status") in ("new", "existing") and isinstance(f.get("text"), str)
and isinstance(f.get("id", ""), str)
and isinstance(a, dict) and isinstance(a.get("file"), str)
and type(a.get("start")) is int and type(a.get("end")) is int and a["start"] <= a["end"]):
rejected["malformed"] += 1
continue
if a["file"] not in names:
rejected["anchor-outside-pack"] += 1
continue
block = next((b for b in by_file.get(a["file"], []) if b["start"] <= a["start"] and a["end"] <= b["end"]), None)
if block is None:
rejected["anchor-crosses-blocks"] += 1
continue
if not 1 <= len(sentence_spans(f["text"].strip())) <= 3:
rejected["sentence-count"] += 1
continue
if a["file"] not in texts:
try:
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
except (OSError, UnicodeDecodeError):
texts[a["file"]] = None
lines = texts[a["file"]]
if lines is None:
rejected["source-missing"] += 1
continue
quote = f.get("quote")
if not (isinstance(quote, str) and quote.strip()):
rejected["quote-missing"] += 1
continue
hit = next((a["start"] + i for i, line in enumerate(lines[a["start"] - 1:a["end"]]) if quote in line), None)
if hit is None:
rejected["quote-not-in-span"] += 1
continue
if "provenance" not in f:
rejected["provenance-missing"] += 1
continue
if not (isinstance(f["provenance"], str) and f["provenance"] in PROVENANCE):
rejected["provenance-unknown"] += 1
continue
tests = f.get("tests")
if f["status"] == "new" and not (isinstance(tests, dict) and str(tests.get("topic_shape", "")).strip()
and str(tests.get("citation", "")).strip()):
rejected["tests-missing"] += 1
continue
accepted.append({**f, "quote": quote, "quote_line": hit, "version": block.get("version"),
"section": block["section"], "position": names.index(a["file"]) + 1})
return accepted, rejected
def context_of(lines: list[str], block: dict[str, Any], quote_line: int, wanted: Any) -> tuple[list[dict[str, Any]], int]:
"""(the context quotes a fragment keeps, the number dropped). Order 22, correction O+L: a sentence may draw on a
heading above its quote or on another line of its section; each context quote the compiler names is placed on the
nearest line of the anchor block's section that holds it (the lines between the heading before the block and the
heading after it, never the quote's own line), else on the innermost heading of the block's heading path whose
text holds it. One that is not text, is found nowhere, repeats a line or comes after MAX_CONTEXT kept is dropped
and counted, never a reject: the fragment keeps its own quote."""
if wanted is None:
return [], 0
if not isinstance(wanted, list):
return [], 1
heads = headings_of(lines)
before = [h for h in heads if h[0] < block["start"]]
after = [h for h in heads if h[0] > block["end"]]
first = before[-1][0] + 1 if before else 1
last = after[0][0] - 1 if after else len(lines)
stack: list[tuple[int, int, str]] = []
for h in before:
stack = [x for x in stack if x[1] < h[1]] + [h]
near = sorted((n for n in range(first, last + 1) if n != quote_line and lines[n - 1].strip()),
key=lambda n: (abs(n - quote_line), n))
kept: list[dict[str, Any]] = []
dropped = 0
for q in wanted:
hit: dict[str, Any] | None = None
if isinstance(q, str) and q.strip() and "\n" not in q and len(kept) < MAX_CONTEXT:
n = next((n for n in near if q in lines[n - 1]), None)
if n is not None:
hit = {"line": n, "quote": q, "section": str(block["section"])}
else:
i = next((i for i in range(len(stack) - 1, -1, -1) if q in stack[i][2]), None)
if i is not None:
hit = {"line": stack[i][0], "quote": q, "section": " › ".join(t for _, _, t in stack[:i + 1])}
if hit is None or any(k["line"] == hit["line"] for k in kept):
dropped += 1
else:
kept.append(hit)
return kept, dropped
def anchor_context(lines: list[str], anchor: dict[str, Any], quote_line: int, section: str) -> list[dict[str, Any]]:
"""The context a merged fragment that names none cites (order 25): the other lines of its anchor that hold a word,
fence lines left out, the MERGED_LINES nearest the quote line, in line order."""
near = sorted((n for n in range(anchor["start"], anchor["end"] + 1)
if n != quote_line and 0 < n <= len(lines) and re.search(r"\w", lines[n - 1])
and not FENCE_LINE.match(lines[n - 1])),
key=lambda n: (abs(n - quote_line), n))[:MERGED_LINES]
return [{"line": n, "quote": lines[n - 1].strip(), "section": section} for n in sorted(near)]
@dataclass
class Context:
kept: int = 0
dropped: int = 0
anchored: int = 0
anchor_lines: int = 0
def record(self) -> dict[str, int]:
"""packs.json's context: kept and dropped, and the anchor counts only when a merged fragment was widened."""
rec = {"kept": self.kept, "dropped": self.dropped}
if self.anchored:
rec.update({"anchored": self.anchored, "anchor_lines": self.anchor_lines})
return rec
def with_context(valid: list[dict[str, Any]], blocks_doc: dict[str, Any], kilde: Path
) -> tuple[list[dict[str, Any]], Context]:
"""The fragments that passed validate with their context quotes placed by context_of (the key left out when none
is kept), a merged fragment that keeps none citing its anchor (anchor_context), and the counts."""
texts: dict[str, list[str]] = {}
out: list[dict[str, Any]] = []
counts = Context()
for given in valid:
f = {k: v for k, v in given.items() if k != "context"}
a = f["anchor"]
block = next(b for b in blocks_doc["blocks"]
if b["file"] == a["file"] and b["start"] <= a["start"] and a["end"] <= b["end"])
if a["file"] not in texts:
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
kept, dropped = context_of(texts[a["file"]], block, int(f["quote_line"]), given.get("context"))
counts.kept, counts.dropped = counts.kept + len(kept), counts.dropped + dropped
if not kept and f["provenance"] == "merged":
kept = anchor_context(texts[a["file"]], a, int(f["quote_line"]), str(block["section"]))
counts.anchored, counts.anchor_lines = counts.anchored + bool(kept), counts.anchor_lines + len(kept)
if kept:
f["context"] = kept
out.append(f)
return out, counts
def new_record(f: dict[str, Any], outcome: str, ref: str | None, index: dict[str, Any], pack: int,
position: int | None) -> dict[str, Any]:
"""The accepted record of one fragment; a new concept or merge candidate is added to the index."""
if outcome == "existing":
cid = str(ref)
else:
cid = unique_id(index, f["concept"])
index["entries"].append({"id": cid, "title": f["concept"], "aliases": [f["concept"]], "kind": outcome,
"merge_of": ref, "origin": {"file": f["anchor"]["file"], "line": f["anchor"]["start"]},
"tests": f.get("tests")})
return {"pack": pack, "position": f["position"] if position is None else position,
"file": f["anchor"]["file"], "start": f["anchor"]["start"], "end": f["anchor"]["end"],
"section": f["section"], "text": f["text"].strip(), "quote": f["quote"],
"quote_line": f["quote_line"], "version": f["version"], "provenance": f["provenance"],
"concept_id": cid, "outcome": outcome, **({"context": f["context"]} if f.get("context") else {})}
@dataclass
class Admitted:
records: list[dict[str, Any]]
rejected: Counter[str]
main_pages: list[dict[str, Any]]
homes: list[dict[str, Any]]
main_stats: dict[str, Any]
home_stats: dict[str, Any]
def admit_pack(valid: list[dict[str, Any]], index: dict[str, Any], pack: int, names: list[str],
facts: dict[str, Fact], declared_main: list[Any], declared_homes: list[Any], before: dict[str, Any],
position: int | None = None) -> Admitted:
"""The fragments that passed validate, resolved to concepts, with the pack's main pages and homes.
`before` is the accepted state the pack joins (for replace, without the replaced source's records). A main page
is kept for a source of the pack with a main topic when the named concept has a fragment from that source (K2
point 1). A history fragment that would make a new concept is resolved after the others and needs a home like any
page. A home is recorded for every concept of the pack that has neither a main role nor a home, when the declared
home is a main page other than itself; the last such home declared for a page wins, and every declared home of
that page that fails is counted invalid. The rest are homeless (listed, never rejected: the guard asks for them).
`position` overrides the position in the pack."""
slots: list[tuple[int, dict[str, Any]]] = []
rejected: Counter[str] = Counter()
deferred: list[tuple[int, dict[str, Any]]] = []
for i, f in enumerate(valid):
outcome, ref = resolve(f, index, history=facts[f["anchor"]["file"]].history)
if outcome == "rejected" and ref == "release-note-new":
deferred.append((i, f))
elif outcome == "rejected":
rejected[str(ref)] += 1
else:
slots.append((i, new_record(f, outcome, ref, index, pack, position)))
by_source: dict[str, set[str]] = {}
for _, r in slots:
by_source.setdefault(r["file"], set()).add(r["concept_id"])
mains: list[dict[str, Any]] = []
for m in declared_main:
if not isinstance(m, dict):
continue
s = source_name(m["source"]) if isinstance(m.get("source"), str) else ""
cid = lookup(index, m.get("id")) or lookup(index, m.get("concept"))
if (s in names and facts[s].main_topic and all(x["source"] != s for x in mains) and cid is not None
and cid in by_source.get(s, set())):
mains.append({"source": s, "concept_id": cid, "pack": pack, "by": "compiler"})
main_now = set(main_ids(before)) | {m["concept_id"] for m in mains}
homes: list[tuple[str, Any, Any]] = [(str(h.get("concept", "")), h.get("id"), h.get("home"))
for h in declared_homes if isinstance(h, dict)]
for i, f in deferred: # after the docs fragments, so a concept they made in this pack is found first
outcome, ref = resolve(f, index, history=False)
if outcome == "rejected":
rejected[str(ref)] += 1
else:
slots.append((i, new_record(f, outcome, ref, index, pack, position)))
records = [r for _, r in sorted(slots, key=lambda s: s[0])]
pages = pages_of(before) | {r["concept_id"] for r in records}
homed = homed_ids(before, main_now, pages)
first: dict[str, str] = {}
for r in records:
first.setdefault(r["concept_id"], r["file"])
need = [c for c in first if c not in main_now and c not in homed]
owners = [lookup(index, eid) or lookup(index, concept) for concept, eid, _ in homes]
home_records: list[dict[str, Any]] = []
homeless: list[str] = []
used: set[int] = set()
invalid = 0
for c in need: # the model's newest choice wins: the last declared home that passes; each that fails is invalid
mine = [j for j, owner in enumerate(owners) if owner == c]
used.update(mine)
chosen = None
for j in mine:
hid = lookup(index, homes[j][2])
if hid is not None and hid in main_now and hid != c and hid in pages:
chosen = hid
else:
invalid += 1
if chosen is None:
homeless.append(c)
else:
home_records.append({"source": first[c], "concept_id": c, "home": chosen, "pack": pack, "by": "compiler"})
not_needed = sum(1 for j, owner in enumerate(owners) if j not in used and owner in first)
unknown = sum(1 for j, owner in enumerate(owners) if j not in used and owner not in first)
main_stats = {"named": len(declared_main), "valid": len(mains),
"missing": [s for s in names if facts[s].main_topic and all(m["source"] != s for m in mains)]}
home_stats = {"needed": len(need), "given": len(home_records), "homeless": homeless, "not_needed": not_needed,
"invalid": invalid, "unknown": unknown}
return Admitted(records, rejected, mains, home_records, main_stats, home_stats)
# --- the work folder -------------------------------------------------------------------------------------------
def read_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise RunError(f"cannot read {path}: {exc}") from exc
def dump(data: Any) -> str:
return json.dumps(data, ensure_ascii=False, indent=1) + "\n"
def k_work(work: Path) -> Path:
return work / "K-work"
def check_pending(work: Path) -> None:
"""A marker left by a write that did not finish stops every command until the operator has dealt with it."""
markers = sorted(k_work(work).glob("*.pending"))
if not markers:
return
marker = markers[0]
try:
info = json.loads(marker.read_text(encoding="utf-8"))
how = (f"copy {', '.join(info['restore']) or 'nothing'} back from {k_work(work) / 'backup'}, and delete "
f"{', '.join(info['delete']) or 'nothing'} (they did not exist before)"
+ "".join(f", and move {dst} back to {src} (if it is there)" for src, dst in info.get("move_back", [])))
except (OSError, ValueError, KeyError, TypeError):
how = f"read {marker} and undo the write by hand"
raise RunError(f"{marker} is set: an earlier write did not finish. To undo it: {how}; then delete the marker")
def write_state(work: Path, label: str, files: dict[str, Any], outside: dict[Path, str] | None = None,
moves: dict[Path, Path] | None = None) -> None:
"""Write K-work files (and new files outside K-work) under a marker: back up the K-work files about to be
replaced, set the marker with what to restore, what to delete and which files to move back, move (each source to
a destination that does not exist), write, clear the marker."""
kw = k_work(work)
kw.mkdir(parents=True, exist_ok=True)
backup = kw / "backup"
if backup.exists():
shutil.rmtree(backup)
backup.mkdir()
existed = [name for name in files if (kw / name).is_file()]
for name in existed:
shutil.copy2(kw / name, backup / name)
marker = kw / f"{label}.pending"
new = sorted(name for name in files if name not in existed) + [str(path) for path in outside or {}]
marker.write_text(dump({"command": label, "restore": existed, "delete": new,
"move_back": [[str(a), str(b)] for a, b in (moves or {}).items()]}), encoding="utf-8")
for a, b in (moves or {}).items():
os.replace(a, b)
for path, text in (outside or {}).items():
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
for name, data in files.items():
tmp = kw / f".{name}.tmp"
tmp.write_text(dump(data), encoding="utf-8")
os.replace(tmp, kw / name)
marker.unlink()
def read_index(work: Path) -> dict[str, Any]:
path = k_work(work) / "index.json"
if not path.is_file():
raise RunError(f"{path} does not exist; run skeleton first")
index: dict[str, Any] = read_json(path)
return index
def read_accepted(work: Path) -> dict[str, Any]:
path = k_work(work) / "accepted.json"
acc: dict[str, Any] = read_json(path) if path.is_file() else {"schema": ACCEPTED_SCHEMA, "fragments": []}
acc.setdefault("main_pages", [])
acc.setdefault("homes", [])
return acc
def pack_log(work: Path) -> dict[str, Any]:
path = k_work(work) / "packs.json"
log: dict[str, Any] = read_json(path) if path.is_file() else {"schema": PACKS_SCHEMA, "packs": []}
return log
def record(log: dict[str, Any], n: int) -> dict[str, Any]:
for r in log["packs"]:
if r["pack"] == n:
return r # type: ignore[no-any-return]
r = {"pack": n}
log["packs"].append(r)
return r
def pack_size(log: dict[str, Any], given: int | None) -> int:
"""The size of a new pack: the one given, else the work folder's first size, else DEFAULT_SIZE. A registered pack
keeps its sources whatever size is given (layout; order 27 A2: the size is set per new pack)."""
if given is not None and given < 1:
raise RunError(f"--size {given} is not a pack size")
if given is not None:
return given
stored = log.get("size")
return DEFAULT_SIZE if stored is None else int(stored)
def pack_cap(log: dict[str, Any], given: int | None) -> int | None:
"""The cap of blocks for a new pack: the one given, else the work folder's, else none (order 28 A2)."""
if given is not None and given < 1:
raise RunError(f"--max-blocks {given} is not a cap of blocks")
if given is not None:
return given
stored = log.get("max_blocks")
return None if stored is None else int(stored)
def groups_apart(log: dict[str, Any], given: bool) -> bool:
"""Whether a new pack keeps docs pages and release notes apart: given once, the work folder keeps it."""
return given or bool(log.get("groups_apart"))
def shape(args: argparse.Namespace, log: dict[str, Any]) -> tuple[int, int | None, bool, int | None]:
"""(the size, the cap of blocks, groups apart, the cap of characters) for a new pack."""
return (pack_size(log, args.size), pack_cap(log, getattr(args, "max_blocks", None)),
groups_apart(log, bool(getattr(args, "groups_apart", False))),
pack_char_cap(log, getattr(args, "max_chars", None)))
def pack_char_cap(log: dict[str, Any], given: int | None) -> int | None:
"""The cap of characters for a new pack: the one given, else the work folder's, else none (B17; the operator
07.11: no artificial limits)."""
if given is not None and given < 1:
raise RunError(f"--max-chars {given} is not a cap of characters")
if given is not None:
return given
stored = log.get("max_chars")
return None if stored is None else int(stored)
def source_name(name: str) -> str:
"""A source as accepted.json names it: the file under kilde/; a K1 path (kilde/x.md, /kilde/x.md) is taken too."""
return name.removeprefix("/").removeprefix("kilde/")
def quiet(fn: Callable[[list[str]], int], argv: list[str]) -> tuple[int, str]:
out = io.StringIO()
with redirect_stdout(out), redirect_stderr(out):
code = fn(argv)
return code, out.getvalue()
# --- commands --------------------------------------------------------------------------------------------------
def cmd_skeleton(args: argparse.Namespace) -> int:
work, sources = Path(args.work), Path(args.sources)
check_pending(work)
out = k_work(work) / "index.json"
kilde = kilde_of(sources)
facts = source_facts(sources, sorted(p.name for p in kilde.iterdir() if p.is_file()))
if args.extend:
return extend_skeleton(work, kilde, facts)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
index, collisions = skeleton(kilde, frozenset(n for n, f in facts.items() if f.history))
write_state(work, "skeleton", {"index.json": index})
print(f"skeleton: {len(index['entries'])} entries -> {out}; {collisions} headings collide across files")
return 0
def extend_skeleton(work: Path, kilde: Path, facts: dict[str, Fact]) -> int:
"""skeleton --extend: the index of a built collection gains the docs pages with a main topic that are in no
registered pack and were not extended before (recorded in packs.json, so a page is extended once)."""
index, log = read_index(work), pack_log(work)
registered = {str(x) for r in log["packs"] for x in r.get("sources", [])}
extended = {str(x) for x in log.get("skeleton_extended", [])}
new = sorted(n for n, f in facts.items() if not f.history
and n not in registered and n not in extended)
added, equal, collisions = add_headings(index, kilde, set(new))
if new:
log["skeleton_extended"] = sorted(extended | set(new))
write_state(work, "skeleton-extend", {"index.json": index, "packs.json": log})
print(f"skeleton extended: {len(new)} new docs pages, {added} entries added, {equal} names equal to an existing "
f"entry, not added; {collisions} headings collide across the new pages -> {k_work(work) / 'index.json'}")
return 0
def interval(args: argparse.Namespace, log: dict[str, Any]) -> tuple[dict[str, Any], list[list[str]], int, int]:
"""(the blocks document, every pack's sources, the pack size, the last pack of the interval N..M)."""
doc: dict[str, Any] = read_json(Path(args.blocks))
size, cap, apart, char_cap = shape(args, log)
all_packs = layout([str(x) for x in doc["order"]], log, size, cap, apart, blocks_per_source(doc), char_cap,
chars_per_source(doc))
last = args.n if getattr(args, "to", None) is None else args.to
if last < args.n:
raise RunError(f"--to {last} is before pack {args.n}")
for n in (args.n, last):
if not 1 <= n <= len(all_packs):
raise RunError(f"pack {n} does not exist; there are {len(all_packs)} (1..{len(all_packs)})")
return doc, all_packs, size, last
def pack_names(args: argparse.Namespace, log: dict[str, Any]) -> tuple[dict[str, Any], list[str], int]:
"""(the blocks document, pack N's sources, the pack size)."""
doc, all_packs, size, _ = interval(args, log)
return doc, all_packs[args.n - 1], size
def cmd_run_before(args: argparse.Namespace) -> int:
work = Path(args.work)
check_pending(work)
log = pack_log(work)
doc, all_packs, size, last = interval(args, log)
sources = Path(args.sources)
kilde = kilde_of(sources)
index, acc = read_index(work), read_accepted(work)
tags = tags_for(acc)
out_dir = work / "K-oppgaver"
jobs = []
for n in range(args.n, last + 1): # render every pack before writing any, so a failure leaves nothing behind
names = all_packs[n - 1]
facts = source_facts(sources, names)
lines = relevant_lines(index, "\n".join((kilde / name).read_text(encoding="utf-8") for name in names))
catalogue = main_catalogue(index, acc, n)
parts = render_input(n, names, doc, lines, kilde, facts, tags, catalogue)
paths = [out_dir / f"pack-{n}-part-{i}.md" for i in range(1, len(parts) + 1)]
if any(p.exists() for p in paths):
raise RunError(f"pack {n}'s task parts exist in {out_dir}; refusing to overwrite")
jobs.append((n, names, lines, catalogue, parts, paths))
(work / "K-fragments").mkdir(parents=True, exist_ok=True) # where the compiler writes its output
_, cap, apart, char_cap = shape(args, log)
for n, names, lines, catalogue, parts, paths in jobs:
outputs = output_paths(work, n, len(parts))
rec = record(log, n)
rec.setdefault("size", size) # a registered pack keeps the size it was made with
if cap is not None: # and the cap; the work folder keeps the first cap and groups apart as its default
rec.setdefault("max_blocks", cap)
log.setdefault("max_blocks", cap)
if char_cap is not None:
rec.setdefault("max_chars", char_cap)
log.setdefault("max_chars", char_cap)
held = chars_per_source(doc)
for name in names:
if held[name] > LARGEST_HELD:
print(f"large: {name} holds {held[name]} characters of blocks, more than the {LARGEST_HELD} of the "
"largest source measured to hold in one compiler run; it runs as it is, and the stop line prices "
"it by its characters")
if apart:
log["groups_apart"] = True
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]),
**({} if (bc := pack_chars(doc, names)) is None else {"block_chars": bc}),
index_lines=len(lines), index_total=len(index["entries"]),
index_ids=[str(e["id"]) for e in lines], main_pages_sent=len(catalogue),
parts=len(parts), chars=sum(len(p) for p in parts), outputs=len(outputs))
log.setdefault("size", size) # the work folder's first size, the default for a new pack
write_state(work, f"pack-{n}", {"packs.json": log}, dict(zip(paths, parts)))
for p in paths:
print(f"part: {p}")
for path, label in outputs:
print(f"output: {path}" + (f" ({label})" if label else ""))
print(f"index lines sent: {len(lines)} of {len(index['entries'])}; main pages sent: {len(catalogue)}")
return 0
def output_paths(work: Path, n: int, parts: int) -> list[tuple[Path, str]]:
"""Pack N's outputs, each with the parts it holds: one output for a pack of OUTPUT_PARTS parts or fewer (the label
left empty), else part 1 alone and then one for every OUTPUT_PARTS parts, which the compiler writes one per turn."""
base = work / "K-fragments"
if parts <= OUTPUT_PARTS:
return [(base / f"pack-{n}.json", "")]
out: list[tuple[Path, str]] = []
for j, first in enumerate([1, *range(2, parts + 1, OUTPUT_PARTS)], 1):
last = 1 if first == 1 else min(first + OUTPUT_PARTS - 1, parts)
out.append((base / f"pack-{n}-out-{j}.json", f"part {first}" if first == last else f"parts {first}-{last}"))
return out
def pack_outputs(work: Path, n: int, log: dict[str, Any]) -> list[Path]:
"""The outputs run-before named for pack N; one, pack-N.json, for a pack rendered before outputs were counted."""
rec = record(log, n)
if "outputs" not in rec:
return [work / "K-fragments" / f"pack-{n}.json"]
paths = [p for p, _ in output_paths(work, n, int(rec.get("parts", 1)))]
if len(paths) != int(rec["outputs"]):
raise RunError(f"pack {n}'s log names {rec['outputs']} outputs for {rec.get('parts')} parts; "
f"this code gives {len(paths)}")
return paths
def read_outputs(paths: list[Path]) -> tuple[tuple[list[Any], list[Any], list[Any]], list[Path]]:
"""(the compiler's outputs read together, in order: fragments, main_pages, homes; the outputs not written)."""
fragments: list[Any] = []
mains: list[Any] = []
homes: list[Any] = []
missing: list[Path] = []
for path in paths:
if not path.is_file():
missing.append(path)
continue
f, m, h = read_compiled(path)
fragments += f
mains += m
homes += h
return (fragments, mains, homes), missing
def read_compiled(path: Path) -> tuple[list[Any], list[Any], list[Any]]:
"""(fragments, main_pages, homes) of the compiler's JSON; main_pages and homes may be left out."""
raw = read_json(path)
fragments = raw.get("fragments") if isinstance(raw, dict) else None
if not isinstance(fragments, list):
raise RunError(f"{path} has no fragments list")
lists: list[list[Any]] = []
for key in ("main_pages", "homes"):
value = raw.get(key, [])
if not isinstance(value, list):
raise RunError(f"{path}: {key} is not a list")
lists.append(value)
return fragments, lists[0], lists[1]
def print_rejected(rejected: Counter[str]) -> None:
for reason, count in sorted(rejected.items()):
print(f" rejected {reason}: {count}")
def print_context(c: Context) -> None:
if c.kept or c.dropped or c.anchored:
print(f" context quotes: kept {c.kept}, dropped {c.dropped}")
if c.anchored:
print(f" merged fragments cited by their anchor: {c.anchored} ({c.anchor_lines} lines)")
def print_roles(main_stats: dict[str, Any], home_stats: dict[str, Any]) -> None:
docs = main_stats["valid"] + len(main_stats["missing"])
without = f"; without one: {', '.join(main_stats['missing'])}" if main_stats["missing"] else ""
print(f" main pages: {main_stats['valid']} of {docs} docs sources (named {main_stats['named']}){without}")
homeless = f"; without a home: {', '.join(home_stats['homeless'])}" if home_stats["homeless"] else ""
kept = f"; kept the home they had: {home_stats['kept']}" if home_stats.get("kept") else ""
print(f" homes: {home_stats['given']} of {home_stats['needed']} pages that need one{kept}{homeless}")
def accept_pack(work: Path, n: int, doc: dict[str, Any], names: list[str], size: int, sources: Path,
compiled: tuple[list[Any], list[Any], list[Any]], log: dict[str, Any]) -> None:
fragments, declared_main, declared_homes = compiled
index = read_index(work)
acc = read_accepted(work)
rec = record(log, n)
if "accepted" in rec or any(a["pack"] == n for a in acc["fragments"]):
raise RunError(f"pack {n} is already accepted")
facts = source_facts(sources, names)
valid, rejected = validate(fragments, names, doc, kilde_of(sources))
valid, context = with_context(valid, doc, kilde_of(sources))
mismatches = sum(id_mismatch(f, index) for f in valid)
got = admit_pack(valid, index, n, names, facts, declared_main, declared_homes, acc)
rejected += got.rejected
sent = set(rec.get("index_ids", []))
not_sent = Counter(r["outcome"] for r in got.records if r["concept_id"] not in sent)
acc["fragments"].extend(got.records)
acc["main_pages"].extend(got.main_pages)
acc["homes"].extend(got.homes)
pass_homes(acc) # a home given in this pack to a page an earlier pack brought names that page's first source
dropped = drop_dangling(acc) # a page this pack makes a main page needs no home any more
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]),
**({} if (bc := pack_chars(doc, names)) is None else {"block_chars": bc}), proposed=len(fragments),
accepted=len(got.records), rejected=dict(sorted(rejected.items())), id_mismatch=mismatches,
not_sent={"total": sum(not_sent.values()), "by_outcome": dict(sorted(not_sent.items()))},
main_pages=got.main_stats, homes=got.home_stats, cited=cited_blocks(got.records, doc, names),
**nonzero(dropped=dropped[0] + dropped[1]))
if context.kept or context.dropped or context.anchored:
rec["context"] = context.record()
log.setdefault("size", size)
write_state(work, f"accept-{n}", {"index.json": index, "accepted.json": acc, "packs.json": log})
print(f"pack {n}: accepted {len(got.records)} of {len(fragments)}")
print(f" id mismatches (counted, not rejected): {mismatches}")
print(f" accepted with a concept not among the index lines sent: {sum(not_sent.values())}")
print_context(context)
print_roles(got.main_stats, got.home_stats)
print_dropped(*dropped, named=got.home_stats["homeless"])
print_rejected(rejected)
def cmd_accept(args: argparse.Namespace) -> int:
work, n = Path(args.work), args.n
check_pending(work)
log = pack_log(work)
doc, names, size = pack_names(args, log)
compiled, missing = read_outputs(pack_outputs(work, n, log))
if missing:
raise RunError(f"pack {n}'s outputs are not all written: {', '.join(str(p) for p in missing)}")
accept_pack(work, n, doc, names, size, Path(args.sources), compiled, log)
return 0
def cited_blocks(records: list[dict[str, Any]], doc: dict[str, Any], names: list[str] | None = None
) -> dict[str, int]:
"""Per source, the distinct blocks that hold the anchor of an accepted record (condition b of wave 4's goal);
each of `names` is counted, with 0 when it has none."""
spans: dict[str, list[tuple[int, int]]] = {}
for b in doc["blocks"]:
spans.setdefault(str(b["file"]), []).append((int(b["start"]), int(b["end"])))
seen: dict[str, set[tuple[int, int]]] = {n: set() for n in names or []}
for r in records:
span = next((s for s in spans.get(str(r["file"]), []) if s[0] <= int(r["start"]) <= s[1]), None)
if span is None:
raise RunError(f"no block of {r['file']} holds line {r['start']}: the blocks file is not this build's")
seen.setdefault(str(r["file"]), set()).add(span)
return {s: len(v) for s, v in sorted(seen.items())}
def cmd_cited(args: argparse.Namespace) -> int:
work, out = Path(args.work), Path(args.out)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
doc: dict[str, Any] = read_json(Path(args.blocks))
counts = cited_blocks(read_accepted(work)["fragments"], doc)
out.write_text(dump({"schema": CITED_SCHEMA, "sources": counts}), encoding="utf-8")
print(f"cited: {sum(counts.values())} anchor blocks over {len(counts)} sources -> {out}")
return 0
NO_CLAIM = re.compile(r"(?:\*\s*){3,}|(?:-\s*){3,}|(?:_\s*){3,}|\*\*[^*]+\*\*:?")
def claim_blocks(doc: dict[str, Any], kilde: Path) -> Counter[str]:
"""Per source, the blocks that can carry a claim: all but a thematic break and a paragraph that is only bold text
(a step heading), which no build has cited (wiki-38: 0 of 433 and 0 of 23; c1 packs 1-16: 0 of 261 and 0 of 10).
A code block always counts: its fence line never matches."""
lines: dict[str, list[str]] = {}
out: Counter[str] = Counter()
for b in doc["blocks"]:
f = str(b["file"])
if f not in lines:
lines[f] = read_lines(kilde / f)
text = " ".join(x.strip() for x in lines[f][int(b["start"]) - 1:int(b["end"])]).strip()
if not NO_CLAIM.fullmatch(text):
out[f] += 1
return out
def cmd_floor(args: argparse.Namespace) -> int:
"""The floor for a later build of sources that changed since (a new fetch): per source the earlier build cited
and that still has blocks, floor(cited x its claim blocks now / its claim blocks then / 2); a history source gets
none of its own (history_rate, when given, holds a pack's together)."""
out = Path(args.out)
if out.exists():
raise RunError(f"{out} exists; refusing to overwrite")
cited = read_floor(Path(args.cited))
then_doc, now_doc = read_json(Path(args.blocks_then)), read_json(Path(args.blocks))
then = claim_blocks(then_doc, kilde_of(Path(args.sources_then)))
now = claim_blocks(now_doc, kilde_of(Path(args.sources)))
gone = sorted(s for s in cited if not now[s])
names = [s for s in cited if now[s]]
facts = source_facts(Path(args.sources), names)
floor = {s: int(cited[s] * now[s] / then[s] / 2) for s in sorted(names) if not facts[s].history and then[s]}
left = (sum(blocks_per_source(then_doc)[s] - then[s] for s in floor),
sum(blocks_per_source(now_doc)[s] - now[s] for s in floor))
doc: dict[str, Any] = {"schema": CITED_SCHEMA, "sources": floor}
if args.history_rate is not None:
if args.history_rate < 0:
raise RunError(f"--history-rate {args.history_rate} is not a rate of cited blocks per block")
doc["history_rate"] = args.history_rate
out.write_text(dump(doc), encoding="utf-8")
print(f"floor: {len(floor)} sources, {sum(floor.values())} anchor blocks; blocks that carry no claim left out: "
f"then {left[0]}, now {left[1]} -> {out}")
if gone:
print(f" no blocks now (no floor): {', '.join(gone)}")
return 0
def read_floor(path: Path) -> dict[str, int]:
"""The floor of cited anchor blocks per source, as cited writes it (an earlier build of the same sources)."""
raw = read_json(path)
counts = raw.get("sources") if isinstance(raw, dict) and raw.get("schema") == CITED_SCHEMA else None
if not (isinstance(counts, dict) and all(isinstance(k, str) and type(v) is int for k, v in counts.items())):
raise RunError(f"{path} is not a floor of cited blocks ({CITED_SCHEMA})")
return {k: int(v) for k, v in counts.items()}
def read_history_rate(path: Path) -> float | None:
"""The floor file's rate for the history sources of a pack (order 28 A4), or None when it gives none."""
raw = read_json(path)
rate = raw.get("history_rate") if isinstance(raw, dict) else None
if rate is None:
return None
if isinstance(rate, bool) or not isinstance(rate, (int, float)) or rate < 0:
raise RunError(f"{path}: history_rate {rate!r} is not a rate of cited blocks per block")
return float(rate)
def cited_now(work: Path, doc: dict[str, Any], names: list[str]) -> dict[str, int]:
"""Per source of NAMES, the anchor blocks its records in accepted.json cite now, so a replace or remove since the
pack was accepted counts (the per-pack count packs.json records is from accept and is never updated)."""
records = [r for r in read_accepted(work)["fragments"] if r["file"] in names]
return cited_blocks(records, doc, names)
def history_floor_check(rate: float, cited_by_source: dict[str, int], doc: dict[str, Any], sources: Path,
all_packs: list[list[str]], first: int, last: int) -> int:
"""0 when the history sources (release notes, weekly pages) of each of packs FIRST..LAST that has any cite at
least floor(their blocks x rate / 2) anchor blocks together; otherwise EXIT_THIN (STOP). Order 28 A4: a release
note has no floor of its own (14 of 20 gave nothing in order 27); a cut answer shows in the pack's total."""
blocks = blocks_per_source(doc)
short = 0
for n in range(first, last + 1):
names = all_packs[n - 1]
facts = source_facts(sources, names)
hist = [s for s in names if facts[s].history]
if not hist:
continue
held = sum(blocks[s] for s in hist)
cited = sum(cited_by_source.get(s, 0) for s in hist)
need = int(held * rate / 2)
print(f"floor: pack {n}'s {len(hist)} history sources cite {cited} anchor blocks of {held}; the floor is {need}")
short += cited < need
if short:
print("STOP: the history sources of a pack cite fewer anchor blocks than their floor; the build stops and "
"reports, it never continues")
return EXIT_THIN
return 0
def approved_source(name: str) -> str | None:
"""The K1 path (kilde/<name>) a source entry of HOMELESS_APPROVED names, written as kilde/<name>, /kilde/<name> or
<name>; None when it names no K1 path whole."""
name = name.strip().lstrip("/")
name = name if name.startswith("kilde/") else "kilde/" + name
return name if klink.kilde_path(name) == name else None
def read_approved(work: Path) -> set[str]:
"""The operator's reported exceptions from HOMELESS_APPROVED in the work folder; no code writes it. Under `pages`,
the pages approved without a home (order 28 A5), each as its page file; under `sources`, the main-topic sources
no page cites (a document with no readable content) approved without a main page (the operator 13.11.2026), each
as its K1 path kilde/<name>. Each entry names the page or source, the pack, the date and the operator's words; a
file may hold either list or both."""
path = work / HOMELESS_APPROVED
if not path.is_file():
return set()
raw = read_json(path)
lists = {k: raw[k] for k in ("pages", "sources") if k in raw} \
if isinstance(raw, dict) and raw.get("schema") == HOMELESS_SCHEMA else {}
if not lists or not all(isinstance(v, list) for v in lists.values()):
raise RunError(f"{path} is not the operator's list of pages without a home and sources without a main page "
f"({HOMELESS_SCHEMA}: a list `pages`, a list `sources` or both)")
out: set[str] = set()
for key, what in (("pages", "page"), ("sources", "source")):
for e in lists.get(key, []):
if not (isinstance(e, dict) and type(e.get("pack")) is int
and all(isinstance(e.get(k), str) and str(e[k]).strip() for k in (what, "date", "operator"))):
raise RunError(f"{path}: an entry under {key} needs the {what}, the pack, the date and the operator's "
f"words: {e!r}")
name = klink.page_rel(str(e["page"])) if what == "page" else approved_source(str(e["source"]))
if name is None:
raise RunError(f"{path}: an entry under sources names its source by its K1 path kilde/<name>: {e!r}")
out.add(name)
return out
def missing_source(f: dict[str, Any]) -> str | None:
"""The K1 path a main-page-missing finding names, in `file` or failing that in `message` (as klink.homeless)."""
return klink.kilde_path(str(f.get("file") or "")) or klink.kilde_path(str(f.get("message") or ""))
def is_approved(f: dict[str, Any], approved: set[str]) -> bool:
"""True for a finding on the operator's list (read_approved): a no-main-page-link finding on a page under `pages`,
or a main-page-missing finding on a source under `sources`. The auditor still reports it."""
if f.get("check") == "no-main-page-link":
return klink.page_rel(str(f.get("file") or "")) in approved
if f.get("check") == "main-page-missing":
return missing_source(f) in approved
return False
def floor_check(floor: dict[str, int], cited: dict[str, int]) -> int:
"""0 when every source of the packs checked (CITED, from cited_now) that the floor names cites at least its floor's
anchor blocks; otherwise each short source is named and EXIT_THIN (STOP)."""
held = {s: c for s, c in sorted(cited.items()) if s in floor}
short = [s for s, c in held.items() if c < floor[s]]
print(f"floor: {len(held) - len(short)} of {len(held)} sources at or over")
for s in short:
print(f"floor: {s} cites {held[s]} anchor blocks of {floor[s]} in the floor")
if short:
print("STOP: a source cites fewer anchor blocks than its floor; the build stops and reports, it never "
"continues")
return EXIT_THIN
return 0
def nonzero(**counts: int) -> dict[str, int]:
return {k: v for k, v in counts.items() if v}
def print_dropped(mains: int, homes: int, left: list[str], passed: int = 0, named: list[str] | None = None) -> None:
"""The records drop_dangling dropped, and the pages it left without a home that `named` does not list already."""
if mains:
print(f" dropped {mains} main page record{'s' if mains != 1 else ''} pointing to a page that is gone")
if homes:
print(f" dropped {homes} home record{'s' if homes != 1 else ''} no longer valid (its page or its home is gone, "
f"its home is no longer a main page, its page is now one, or a newer home of the page stands)")
rest = [c for c in left if c not in (named or [])]
if rest:
print(f" without a home now: {', '.join(rest)}")
if passed:
print(f" passed {passed} home record{'s' if passed != 1 else ''} to the source that now brings its page")
def cmd_replace(args: argparse.Namespace) -> int:
work, source = Path(args.work), source_name(args.source)
check_pending(work)
doc: dict[str, Any] = read_json(Path(args.blocks))
if source not in doc["order"]:
raise RunError(f"{args.blocks} has no blocks for {source}")
log = pack_log(work)
size = pack_size(log, args.size)
place = next(((int(r["pack"]), r["sources"].index(source) + 1) for r in log["packs"]
if "accepted" in r and source in r.get("sources", [])), None)
if place is None:
raise RunError(f"{source} is in no accepted pack; a source joins the build through pack and accept")
pack, position = place # the place the source had when its pack was accepted, whatever the order is now
rec = record(log, pack)
sources = Path(args.sources)
fragments, declared_main, declared_homes = read_compiled(Path(args.fragments))
index = read_index(work)
acc = read_accepted(work)
valid, rejected = validate(fragments, [source], doc, kilde_of(sources))
valid, context = with_context(valid, doc, kilde_of(sources))
own = {str(r["concept_id"]) for r in acc["fragments"] if r["file"] == source} # the pages S wrote to
mains_before = set(main_ids(acc))
before = {"fragments": [r for r in acc["fragments"] if r["file"] != source],
"main_pages": [m for m in acc["main_pages"] if m["source"] != source],
"homes": [h for h in acc["homes"] if str(h["concept_id"]) not in own]}
got = admit_pack(valid, index, pack, [source], source_facts(sources, [source]), declared_main, declared_homes,
before, position)
rejected += got.rejected
old = [r for r in acc["fragments"] if r["file"] == source]
kept: list[dict[str, Any]] = []
for r in acc["fragments"]:
if r["file"] != source:
kept.append(r)
elif r is old[0]:
kept.extend(got.records) # the new records take the place of the old ones
old_mains = len(acc["main_pages"]) - len(before["main_pages"])
old_homes = list(reversed(acc["homes"])) # newest first, so a page's carried home is its newest valid one
acc["fragments"] = kept if old else kept + got.records
acc["main_pages"] = before["main_pages"] + got.main_pages
acc["homes"] = before["homes"] + got.homes
# a page S wrote to keeps its home when the new fragments give it none: the home belongs to the page; a home the
# model chose anew wins, and drop_dangling keeps a carried home only while it is valid
homed = {str(h["concept_id"]) for h in acc["homes"]}
carried: list[dict[str, Any]] = []
for h in old_homes:
if str(h["concept_id"]) in own and str(h["concept_id"]) not in homed:
carried.append(h)
homed.add(str(h["concept_id"]))
acc["homes"] += carried
dropped = drop_dangling(acc)
kept_homes = [str(h["concept_id"]) for h in carried if any(h is x for x in acc["homes"])]
left = left_without(acc, mains_before, dropped[2]) # print_dropped leaves out those the pack's homes name
got.home_stats["homeless"] = [c for c in got.home_stats["homeless"] if c not in kept_homes]
got.home_stats["kept"] = len(kept_homes)
passed = pass_homes(acc)
rec.setdefault("replaced", []).append({"source": source, "removed": len(old), "proposed": len(fragments),
"accepted": len(got.records), "rejected": dict(sorted(rejected.items())),
**nonzero(main_pages=old_mains, kept_homes=len(kept_homes),
dropped=dropped[0] + dropped[1], passed=passed)})
log.setdefault("size", size)
write_state(work, f"replace-{slug(source)}", {"index.json": index, "accepted.json": acc, "packs.json": log})
print(f"replace {source}: removed {len(old)}, accepted {len(got.records)} of {len(fragments)}")
print_context(context)
print_roles(got.main_stats, got.home_stats)
print_dropped(dropped[0], dropped[1], left, passed, got.home_stats["homeless"])
print_rejected(rejected)
return 0
def cmd_remove(args: argparse.Namespace) -> int:
work, source = Path(args.work), source_name(args.source)
check_pending(work)
acc = read_accepted(work)
old = [r for r in acc["fragments"] if r["file"] == source]
if not old:
raise RunError(f"no accepted fragment comes from {source}")
mains = sum(m["source"] == source for m in acc["main_pages"])
mains_before = set(main_ids(acc))
acc["fragments"] = [r for r in acc["fragments"] if r["file"] != source]
acc["main_pages"] = [m for m in acc["main_pages"] if m["source"] != source]
dropped = drop_dangling(acc) # a home goes with its page or its target, never with the source alone
passed = pass_homes(acc)
log = pack_log(work)
record(log, int(old[0]["pack"])).setdefault("removed", []).append(
{"source": source, "fragments": len(old), **nonzero(main_pages=mains, dropped=dropped[0] + dropped[1],
passed=passed)})
write_state(work, f"remove-{slug(source)}", {"accepted.json": acc, "packs.json": log})
print(f"remove {source}: removed {len(old)} fragments")
print_dropped(dropped[0], dropped[1], left_without(acc, mains_before, dropped[2]), passed)
return 0
# --- run-after: the steps after the compiler -------------------------------------------------------------------
def hard_check(work: Path, n: int, paths: list[Path], fragments: list[Any], names: list[str], doc: dict[str, Any],
kilde: Path, log: dict[str, Any], missing: int = 0) -> int:
"""0 when every output of pack N is written. A fragment that fails the hard check at writing is dropped and
counted when the pack is accepted, and the pack does not run again: the check stops the sentence, not the pack
(order 28, B2: pack 26 ran once more, 4.7 M, for 1 fragment of 574). An output left unwritten sets the outputs
written aside and logs the attempt: the first time EXIT_RETRY (the pack runs once more), then EXIT_REJECTED_TWICE
(STOP)."""
if not missing:
return 0
_, rejected = validate(fragments, names, doc, kilde)
rejected["output-missing"] += missing
attempts = record(log, n).setdefault("attempts", [])
attempts.append({"proposed": len(fragments), "rejected": dict(sorted(rejected.items()))})
moves: dict[Path, Path] = {}
for path in paths:
if path.is_file():
kept = path.with_name(f"{path.stem}.attempt-{len(attempts)}.json")
if kept.exists():
raise RunError(f"{kept} exists; refusing to overwrite")
moves[path] = kept
write_state(work, f"attempt-{n}", {"packs.json": log}, moves=moves)
reasons = ", ".join(f"{r} {c}" for r, c in sorted(rejected.items()))
bad = sum(rejected.values()) - missing
what = [f"{bad} of {len(fragments)} fragments rejected at writing"] if bad else []
what += [f"{missing} of {len(paths)} outputs not written"] if missing else []
print(f"pack {n}: {'; '.join(what)} ({reasons}); set aside as {', '.join(k.name for k in moves.values())}")
if len(attempts) == 1:
print(f"RETRY: run the compiler for pack {n} once more with the same parts; it writes "
f"{', '.join(str(p) for p in paths)}")
return EXIT_RETRY
print(f"STOP: pack {n} left an output unwritten twice; the operator decides")
return EXIT_REJECTED_TWICE
def run_audit(bundle: Path, root: Path) -> dict[str, Any]:
"""The auditor's JSON for BUNDLE against ROOT, run in this process (no subprocess in the package)."""
out, err = io.StringIO(), io.StringIO()
with redirect_stdout(out), redirect_stderr(err):
code = audit.main([str(bundle), str(root)])
if code not in (0, 1):
raise RunError(f"the auditor could not run (exit {code}): {err.getvalue().strip()}")
try:
doc = json.loads(out.getvalue())
except json.JSONDecodeError as exc:
raise RunError(f"the auditor's output is not JSON: {exc}") from exc
if not (isinstance(doc, dict) and isinstance(doc.get("findings"), list)):
raise RunError("the auditor's output has no list of findings")
return doc
def audit_root(sources: Path, work: Path, log: dict[str, Any], last: int) -> Path:
"""A subset root (K1 rule 9) of the sources of the accepted packs 1..last; only the newest is kept."""
names = sorted({str(s) for r in log["packs"] if "accepted" in r and int(r["pack"]) <= last
for s in r.get("sources", [])})
base = work / "K-audit"
base.mkdir(parents=True, exist_ok=True)
out, paths = base / f"sources-p{last}", base / f"sources-p{last}.paths"
if out.exists():
shutil.rmtree(out)
paths.write_text("".join(f"kilde/{n}\n" for n in names), encoding="utf-8")
code, printed = quiet(k1.main, ["subset", str(sources), "--paths", str(paths), "--out", str(out)])
if code != 0:
raise RunError(f"the subset root of packs 1..{last} could not be made (exit {code}): {printed.strip()}")
for old in base.iterdir():
if old.name.startswith("sources-p") and old.name not in (out.name, paths.name):
if old.is_dir():
shutil.rmtree(old)
else:
old.unlink()
return out
def save_audit(work: Path, last: int, doc: dict[str, Any]) -> Path:
base = work / "K-audit"
k = 1
while (base / f"p{last}-audit-{k}.json").exists():
k += 1
path = base / f"p{last}-audit-{k}.json"
path.write_text(dump(doc), encoding="utf-8")
return path
def ask_guard(work: Path, out: Path, last: int, audit_doc: dict[str, Any], log: dict[str, Any]) -> int:
pages, missing = klink.homeless(audit_doc)
catalogue = [str(e["id"]) for e in main_catalogue(read_index(work), read_accepted(work), last)]
parts = klink.guard_parts(out, pages, missing, catalogue, f"pack {last}")
gdir = work / "K-guard" / f"p{last}"
paths = [gdir / f"part-{i}.md" for i in range(1, len(parts) + 1)]
answer = gdir / "answer.json"
if any(p.exists() for p in [*paths, answer]):
raise RunError(f"{gdir} holds an earlier home task or answer; refusing to overwrite")
record(log, last)["guard"] = {"asked": {"pages": pages, "sources": missing, "catalogue": catalogue},
"parts": len(parts)}
write_state(work, f"guard-{last}", {"packs.json": log}, dict(zip(paths, parts)))
print(f"guard: {len(pages)} pages without a home, {len(missing)} sources without a main page")
for p in paths:
print(f" part: {p}")
print(f" answer: {answer}")
return EXIT_GUARD
def apply_guard(work: Path, out: Path, last: int, guard: dict[str, Any], log: dict[str, Any]) -> None:
answer = work / "K-guard" / f"p{last}" / "answer.json"
if not answer.is_file():
raise RunError(f"the guard's answer {answer} does not exist; run the home task, then run-after again")
asked = guard["asked"]
got = klink.read_guard(read_json(answer), asked["pages"], asked["sources"], asked["catalogue"],
klink.guard_candidates(out, asked["sources"]))
acc = read_accepted(work)
for src, pid in got.main_pages:
name = source_name(src)
acc["main_pages"] = [m for m in acc["main_pages"] if m["source"] != name] + [
{"source": name, "concept_id": pid, "pack": last, "by": "guard"}]
firsts = first_files(acc)
for pid, hid in got.homes:
if pid not in firsts:
raise RunError(f"the guard homed {pid}, which has no accepted fragment")
acc["homes"].append({"source": firsts[pid], "concept_id": pid, "home": hid, "pack": last, "by": "guard"})
dropped = drop_dangling(acc) # a main page the guard named in a source's place may take a home's target away
guard["answered"] = {"homes": len(got.homes), "main_pages": len(got.main_pages), "none": got.none,
"unanswered": got.unanswered, "rejected": dict(sorted(got.rejected.items())),
**nonzero(dropped=dropped[0] + dropped[1])}
write_state(work, f"guard-answer-{last}", {"accepted.json": acc, "packs.json": log})
print(f"guard answer: homes {len(got.homes)}, main pages {len(got.main_pages)}, none {len(got.none)}, "
f"unanswered {len(got.unanswered)}, rejected {sum(got.rejected.values())}")
print_dropped(*dropped)
def stop_line(args: argparse.Namespace, work: Path, log: dict[str, Any], last: int) -> int:
done = sum(int(r.get("blocks", 0)) for r in log["packs"] if "accepted" in r and int(r["pack"]) <= last)
print(f"done: {done} of {args.total} blocks")
code = cost.main(["harvest", "--transcripts", args.transcripts, "--okter", args.okter, "--work", args.work,
"--out", args.kost])
print(f"cost harvest: exit {code}")
if code:
print("STOP: the cost could not be measured (a run the harvest cannot attribute, or it could not run); "
"the build stops and reports, it never continues")
return code # 1 as the stop line gives it; 2 when the harvest could not run
doc, _, size, _ = interval(args, log)
_, cap, apart, char_cap = shape(args, log)
names = next_pack(doc, log, last, size, done, args.total, cap, apart, char_cap)
following = sum(b["file"] in names for b in doc["blocks"])
chars = pack_chars(doc, names)
stop = ["stop", "--lines", STOP_LINES, "--cap", str(args.cap), "--unit", "blocks", "--done", str(done),
"--total", str(args.total), "--kost", args.kost, "--blocks", str(k_work(work) / "packs.json"),
"--next", str(following), *([] if chars is None else ["--next-chars", str(chars)])]
code = cost.main(stop + (["--name", args.name] if args.name else []))
print(f"cost stop: exit {code}")
if code == EXIT_STOP:
print("STOP: the stop line; over the cost stop the build stops and reports, it never continues")
elif code:
print("STOP: the stop line could not run; the build stops and reports, it never continues")
return code
def cmd_run_after(args: argparse.Namespace) -> int:
from okfwiki import kassemble # kassemble imports this module, so it is imported here, not at the top
work, sources, out = Path(args.work), Path(args.sources), Path(args.out)
check_pending(work)
if read_k1(sources) is None:
raise RunError(f"{sources / MANIFEST} does not exist: run-after needs K1 at the sources root, because the "
f"auditor cannot check the main-page rule without it (K2 point 6)")
floor = read_floor(Path(args.floor)) if args.floor else None
rate = read_history_rate(Path(args.floor)) if args.floor else None
approved = read_approved(work)
log = pack_log(work)
doc, all_packs, size, last = interval(args, log)
kilde = kilde_of(sources)
for n in range(args.n, last + 1):
if "accepted" in record(log, n):
continue
paths = pack_outputs(work, n, log)
compiled, missing = read_outputs(paths)
if len(missing) == len(paths):
raise RunError(f"{', '.join(str(p) for p in paths)} {'does' if len(paths) == 1 else 'do'} not exist; "
f"run the compiler for pack {n} first")
code = hard_check(work, n, paths, compiled[0], all_packs[n - 1], doc, kilde, log, len(missing))
if code:
return code
accept_pack(work, n, doc, all_packs[n - 1], size, sources, compiled, log)
guard = record(log, last).get("guard")
if isinstance(guard, dict) and "answered" not in guard:
apply_guard(work, out, last, guard, log)
links = Path(args.links) if args.links else None
r = kassemble.build(work, sources, out, links, replace=out.exists(), at=args.at)
print(f"assemble: {r.pages} pages from {r.fragments} fragments: new {len(r.new)}, changed {len(r.changed)}, "
f"unchanged {len(r.unchanged)}, gone {len(r.gone)}; main pages {r.main_pages}, related links {r.related}"
+ (f"; links placed {r.placed}, stale {r.stale}" if links else ""))
if r.dropped:
print(f" dropped links to pages that are gone: {sum(len(t) for t in r.dropped.values())} "
f"({', '.join(sorted(r.dropped))})")
if r.unlinked:
print(f" removed from the links file: {r.unlinked} records into or on pages that are gone")
root = audit_root(sources, work, log, last)
audit_doc = run_audit(out, root)
saved = save_audit(work, last, audit_doc)
findings = [f for f in audit_doc["findings"] if isinstance(f, dict) and f.get("check") not in RETIRED]
classes = Counter(str(f.get("check")) for f in findings)
line = audit_doc.get("link_rule", {}).get("line") if isinstance(audit_doc.get("link_rule"), dict) else None
print(f"audit: {len(findings)} findings ({', '.join(f'{c} {k}' for c, k in sorted(classes.items())) or 'none'}) "
f"on packs 1..{last}, {len(list((root / 'kilde').iterdir()))} sources -> {saved}" + (f"; {line}" if line else ""))
excepted = [f for f in findings if is_approved(f, approved)]
if excepted: # the auditor still reports them; the operator approved each as a reported exception (order 28 A5)
homeless = sorted({klink.page_rel(str(f.get("file") or "")) for f in excepted
if f.get("check") == "no-main-page-link"})
mainless = sorted({str(missing_source(f)) for f in excepted if f.get("check") == "main-page-missing"})
if homeless:
print(f"approved without a home (the operator's list): {', '.join(homeless)}")
if mainless: # a document with no readable content: never a forced fragment or main page (13.11.2026)
print(f"approved without a main page (the operator's list): {', '.join(mainless)}")
findings = [f for f in findings if not is_approved(f, approved)]
audit_doc = {**audit_doc, "findings": [f for f in audit_doc["findings"]
if not (isinstance(f, dict) and is_approved(f, approved))]}
if findings:
cannot = [f for f in findings if f.get("check") not in klink.GUARDED]
if cannot or guard is not None:
why = "a finding the guard cannot mend" if cannot else "findings stand after the guard"
print(f"STOP: {why}; the operator decides (K2 point 9):")
for f in findings:
where = str(f.get("file") or "")
if f.get("check") == "main-page-missing": # its source, read as the guard reads it
where = missing_source(f) or where
print(f" {f.get('check')} {where}")
return EXIT_BREACH
return ask_guard(work, out, last, audit_doc, log)
code = stop_line(args, work, log, last)
if code or floor is None:
return code
cited = cited_now(work, doc, [s for n in range(args.n, last + 1) for s in all_packs[n - 1]])
thin = floor_check(floor, cited)
if rate is not None:
thin = history_floor_check(rate, cited, doc, sources, all_packs, args.n, last) or thin
return thin
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="python3 -m okfwiki.kpack", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
sub = p.add_subparsers(dest="command", required=True)
s = sub.add_parser("skeleton")
s.add_argument("--sources", required=True)
s.add_argument("--work", required=True)
s.add_argument("--extend", action="store_true")
for name in ("pack", "accept", "replace", "run-before", "run-after"):
c = sub.add_parser(name)
if name == "replace":
c.add_argument("source")
c.add_argument("--fragments", required=True)
else:
c.add_argument("n", type=int)
if name.startswith("run-"):
c.add_argument("--to", type=int)
c.add_argument("--blocks", required=True)
c.add_argument("--sources", required=True)
c.add_argument("--work", required=True)
c.add_argument("--size", type=int)
c.add_argument("--max-blocks", type=int)
c.add_argument("--max-chars", type=int)
c.add_argument("--groups-apart", action="store_true")
if name == "run-after":
c.add_argument("--out", required=True)
c.add_argument("--links")
c.add_argument("--at")
for flag in ("--transcripts", "--okter", "--kost"):
c.add_argument(flag, required=True)
c.add_argument("--cap", type=int, required=True)
c.add_argument("--total", type=int, required=True)
c.add_argument("--name")
c.add_argument("--floor")
r = sub.add_parser("remove")
r.add_argument("source")
r.add_argument("--work", required=True)
c = sub.add_parser("cited")
for flag in ("--work", "--blocks", "--out"):
c.add_argument(flag, required=True)
f = sub.add_parser("floor")
for flag in ("--cited", "--blocks-then", "--sources-then", "--blocks", "--sources", "--out"):
f.add_argument(flag, required=True)
f.add_argument("--history-rate", type=float)
return p
def cmd_pack(args: argparse.Namespace) -> int:
args.to = None
return cmd_run_before(args)
COMMANDS = {"skeleton": cmd_skeleton, "pack": cmd_pack, "accept": cmd_accept, "replace": cmd_replace,
"remove": cmd_remove, "run-before": cmd_run_before, "run-after": cmd_run_after, "cited": cmd_cited,
"floor": cmd_floor}
def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
return COMMANDS[args.command](args)
except (RunError, OSError, UnicodeDecodeError) as exc:
print(f"kpack cannot run: {exc}", file=sys.stderr)
return 2
except Exception: # an input this code did not foresee: still "could not run", with the traceback
traceback.print_exc()
return 2
if __name__ == "__main__":
sys.exit(main())