First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
1992 lines
107 KiB
Python
1992 lines
107 KiB
Python
"""K, the fragment compiler's code side: skeleton, packs, input parts, the one hard check at writing, concept identity,
|
||
main pages and homes, one source's contribution replaced or removed on its own, and the steps without a model around
|
||
the compiler. No model call (CLAUDE.md rule 1).
|
||
|
||
Usage:
|
||
python3 -m okfwiki.kpack skeleton --sources DIR --work DIR [--extend]
|
||
python3 -m okfwiki.kpack pack N --blocks FILE --sources DIR --work DIR [--size S] [--max-blocks B] [--groups-apart]
|
||
python3 -m okfwiki.kpack accept N --blocks FILE --sources DIR --work DIR [--size S] [--max-blocks B] [--groups-apart]
|
||
python3 -m okfwiki.kpack replace SOURCE --fragments FILE --blocks FILE --sources DIR --work DIR [--size S]
|
||
python3 -m okfwiki.kpack remove SOURCE --work DIR
|
||
python3 -m okfwiki.kpack run-before N [--to M] --blocks FILE --sources DIR --work DIR [--size S] [--max-blocks B]
|
||
[--groups-apart]
|
||
python3 -m okfwiki.kpack run-after N [--to M] --blocks FILE --sources DIR --work DIR --out WIKI [--links FILE]
|
||
[--at ISO] --transcripts DIR --okter FILE --kost FILE --cap C --total T [--name NAME] [--size S]
|
||
[--max-blocks B] [--groups-apart] [--floor FILE]
|
||
python3 -m okfwiki.kpack cited --work DIR --blocks FILE --out FILE
|
||
python3 -m okfwiki.kpack floor --cited FILE --blocks-then FILE --sources-then DIR --blocks FILE --sources DIR
|
||
[--history-rate R] --out FILE
|
||
|
||
SOURCES holds kilde/ and, in a frozen copy, source-manifest.json (contract K1). K1 decides each source's kind and main
|
||
topic; without K1 the file name does, as before (release-vX.Y.Z.md is a release note, every other file a page with a main
|
||
topic). A source without a main topic (a release note, or a page K1 marks as history, such as the weekly What's New
|
||
pages) is history: it has no main page, its fragments attach to existing concepts first and go under "## History", and
|
||
a new concept from it needs a home like any page. --blocks is kblocks' blocks file. The work folder holds K-oppgaver/
|
||
(task parts), K-fragments/ (the compiler's JSON), K-work/ (index.json, accepted.json, packs.json, and backup/ with the
|
||
files as they were before the last write), K-audit/ (the auditor's JSON and the subset root it ran on) and K-guard/
|
||
(the guard's tasks and answers).
|
||
|
||
skeleton the concept index from the titles and level 2-3 headings of every docs page with a main topic that pass
|
||
concept test 2, one entry per normalised name (the first page in name order wins); prints how many headings
|
||
collide with a heading of the same name in another file. An index, not pages. With --extend, the index
|
||
of a built collection gains the docs pages with a main topic that are in no pack packs.json registers and
|
||
were not extended before: a name of theirs is added by the same rule, an existing entry never changes and
|
||
gets no new alias, and a name equal to an existing one is not added and is counted.
|
||
pack N pack N's input for the compiler: the sources, each marked "name its main page" or as history; every block of
|
||
its sources with its anchor (file › section › span) and numbered lines; the index lines whose normalised name
|
||
or alias occurs as a phrase in the normalised text of those sources (a phrase filter, not a word filter),
|
||
each marked "main page" or "has a home" when it is; the main pages a home may be chosen from, filtered for
|
||
the pack (main_catalogue); and the output contract, in parts of at most PART_CHARS characters (the Read
|
||
tool's cap). Prints the part paths and the outputs: one for a pack of OUTPUT_PARTS parts or fewer, else
|
||
part 1 alone and then one for every OUTPUT_PARTS parts, each with the parts it holds, which the compiler
|
||
writes one per turn (one answer holds only so much: round 1's pack 1 hit the limit with seven parts in one
|
||
output, and the first answer also holds the thinking).
|
||
accept N checks pack N's fragments (validate: the one hard check at writing), resolves each to a concept by code
|
||
(existing, merge candidate, new or rejected), updates the index and appends the accepted fragments, each
|
||
with the context quotes it names placed on a heading of its path or a line of its section (context_of; at
|
||
most MAX_CONTEXT, the rest dropped and counted, never a reject); a merged fragment that keeps none cites
|
||
the other lines of its anchor (anchor_context, at most MERGED_LINES, nearest the quote). Then it
|
||
records the main page the compiler names for each docs source of the pack (kept only when that page has a
|
||
fragment from the source) and a home for every page of the pack that has neither a main role nor a home
|
||
(kept only when it names a main page). A page without a home is accepted and listed, never rejected; the
|
||
guard asks for it after the auditor has run. Prints "accepted X of N", the rejections by reason, the id
|
||
mismatches (counted, never rejected), how many accepted fragments resolved to a concept whose index line was
|
||
not sent with the pack, the main pages and the pages without a home.
|
||
replace S checks a new set of fragments for source S against S alone (its blocks in --blocks), then puts them in the
|
||
place of S's accepted fragments; every other source's records stay as they were, in place. S keeps the
|
||
pack and position it had when its pack was accepted, even when the order has changed since; a source in
|
||
no accepted pack cannot be replaced (it joins the build through pack and accept). S's main page records
|
||
are swapped with its fragments.
|
||
remove S takes S's accepted fragments and main page records out; every other record stays as it was.
|
||
A home belongs to its page, one per page, and the model's newest choice wins: replace takes the homes the
|
||
new fragments give S's pages, and a page S wrote to that gets none keeps the home it had. After replace,
|
||
remove, accept and the guard's answer, a main page record that points to a page now gone is dropped, and
|
||
so is every home record no longer valid (its page or its home gone, its home no longer a main page, its
|
||
page now a main page, or a newer home of the page); a dropped record never comes back, and the pages left
|
||
without a home are named. A home record names the source that brings its page and passes on when that
|
||
source goes (counted), so a page another source keeps never loses its home with S.
|
||
run-before N [--to M]
|
||
pack for N..M, every pack rendered before any is written. An interval of more than one pack gives the later
|
||
packs the index of now, without the concepts and main pages of the packs before them in the interval, and
|
||
the cost is checked only after the interval: a build runs one pack at a time.
|
||
run-after N [--to M]
|
||
the steps after the compiler. For each pack not yet accepted, its outputs read together and the hard
|
||
check: a fragment rejected at writing is dropped and counted at accept, and the pack does not run again for
|
||
it; an output left unwritten sets the pack's outputs aside and exits 3 (run the compiler once more, same
|
||
parts, same outputs); an output left unwritten again, exit 6 (STOP). Then accept. Then assemble (kassemble build, --links placed), run the auditor on a
|
||
subset root of the sources of packs 1..M (K1 rule 9), so sources not yet built never count, and act on its
|
||
findings (link-rule is retired in Step 7 and left out): none, go on; only pages without a link from a main
|
||
page and sources without a main page, the guard's task is written (exit 4: run it, then run-after again,
|
||
which takes its answer, assembles and audits again); any finding after the guard, or one it cannot mend,
|
||
exit 5 (STOP, listed: K2 point 9, there is no exception list). Last the stop line: cost harvest, then
|
||
cost stop --lines build,link,orchestration-build --unit blocks --blocks K-work/packs.json with --done the
|
||
blocks of packs 1..M, --next the blocks of pack M+1 (0 after the last) and --next-chars their characters
|
||
(when the blocks file has them; each pack's record keeps its own as block_chars), so measured cost and the next
|
||
pack stop and the projection is reported only (the operator 12.10.2026); a non-zero exit of either is STOP, each with its own reason (exit 1 over the cost
|
||
stop or for a run the harvest cannot attribute, 2 when the harvest or the stop line cannot run). Prints
|
||
counts, and the ids of pages and sources only where a missing home, main page or a stop needs them; never
|
||
text from a source. With --floor (a file cited wrote for an earlier build of the same sources), after the
|
||
stop line every source of the packs that the floor names must cite at least its floor's anchor blocks;
|
||
one that cites fewer is named and exit 7 (STOP: condition b of wave 4's goal, checked after every pack).
|
||
A floor file may also give history_rate: the history sources of each pack (release notes, weekly pages),
|
||
which have no floor of their own, cite at least floor(their blocks x rate / 2) anchor blocks together, or
|
||
exit 7 (order 28 A4). A page without a home that the operator has approved as a reported exception, on
|
||
the list HOMELESS_APPROVED in the work folder (page, pack, date and his words; no code or model writes
|
||
it), no longer stops run-after and is not asked of the guard; the auditor still reports it, every other
|
||
finding stops as before, and it never gets a forced link (order 28 A5). A main-topic source no page cites
|
||
(a document with no readable content, so no fragment and no main page) that he has approved goes on the
|
||
same list under sources (source, pack, date and his words) and is treated the same way: reported, never
|
||
given a forced fragment or main page (the operator 13.11.2026). After such a stop and his approval,
|
||
run-after N again goes on from where the pack stopped.
|
||
cited per source, the distinct blocks that hold the anchor of an accepted fragment, from the work folder's
|
||
accepted.json and the blocks file of that build: the floor for a later build of the same sources.
|
||
floor the floor for a later build of sources that changed since (a new fetch), from the file cited wrote:
|
||
floor(cited x the source's blocks now / its blocks then / 2), counting only blocks that can carry a claim
|
||
(a thematic break or a paragraph that is only bold text carries none); a history source gets none of its
|
||
own, and --history-rate R is written as the file's history_rate.
|
||
|
||
The packs are --size sources each (5 unless the work folder already records a size). --max-blocks B closes a new pack
|
||
before the source that would take it over B blocks (a source over B alone is a pack of its own); --max-chars C does
|
||
the same by the characters of the blocks (B17; no cap unless given or recorded), and run-before names a source
|
||
larger than LARGEST_HELD, the largest measured to hold in one compiler run ("large: ...", it stops nothing); --groups-apart
|
||
keeps docs pages and release notes in separate new packs (order 28 A2); both are registered in packs.json like the
|
||
size, so later commands lay out the same packs without them. A pack number outside 1..N cannot run.
|
||
|
||
A built collection is continued (order 27 A2): run kblocks order and blocks for the larger root to new paths, then
|
||
skeleton --extend. The packs packs.json registers keep their sources, their place and their size; the sources of the
|
||
order in no registered pack follow in new packs after the last, docs pages before release notes, each group in K's
|
||
order, and --size sets the size of a new pack (registered with it). The first new pack starts with run-before alone:
|
||
the links of the pack before it are placed already, so its chain leaves out that pack's pass-after. Every write first copies the K-work files it replaces to K-work/backup/ and sets a
|
||
marker K-work/<command>.pending that names the files to copy back, the new files to delete and the files to move
|
||
back (a rejected pack set aside), then moves and writes (a pack's task parts too), then deletes the marker; a command
|
||
that finds a marker cannot run, and says what to undo. Exit 0 =
|
||
done, 1 = STOP by the stop line, 2 = could not run (also for an input the code did not foresee, with its traceback),
|
||
3 = run the compiler for the pack once more, 4 = run the guard, 5 = STOP: a finding stands after the guard or cannot be
|
||
mended by it, 6 = STOP: an output of the pack was left unwritten twice, 7 = STOP: a source cites fewer anchor blocks
|
||
than its floor.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import argparse
|
||
import io
|
||
import json
|
||
import os
|
||
import re
|
||
import shutil
|
||
import sys
|
||
import traceback
|
||
from collections import Counter
|
||
from collections.abc import Callable, Mapping
|
||
from contextlib import redirect_stderr, redirect_stdout
|
||
from dataclasses import dataclass
|
||
from pathlib import Path
|
||
from typing import Any
|
||
|
||
from okfwiki import audit, cost, klink
|
||
from okfwiki import sources as k1
|
||
from okfwiki.audit import sentence_spans
|
||
from okfwiki.common import RunError, jaccard, slug
|
||
from okfwiki.kblocks import RELEASE, docs_headings, headings_of, is_meta, kilde_of, read_lines, source_files
|
||
from okfwiki.kblocks import order as kblocks_order
|
||
|
||
PART_CHARS = 60_000
|
||
MERGE_JACCARD = 0.6
|
||
DEFAULT_SIZE = 5
|
||
LARGEST_HELD = 387_767 # the largest source measured to hold in one compiler run: one PDF of a real folder, 07.11
|
||
# (7 task parts, 4 outputs, 342 of 345 accepted, auditor 0, context at most 281k tokens). A
|
||
# measurement, not a limit: run-before names a larger source and stops nothing (B17)
|
||
PROVENANCE = ("documented", "inferred", "merged")
|
||
MAX_CONTEXT = 2 # context quotes kept per fragment (order 22: O or L is one missing piece; both together are two)
|
||
MERGED_LINES = 8 # lines of its anchor a merged fragment without context cites (order 25: all 20 merged claims of round 1
|
||
# unsupported, all L, 19 without context; the whole anchor reached 95 of 118 missing words, the nearest
|
||
# eight 88, and an anchor of up to 47 lines would hang 46 footnotes on one sentence)
|
||
FENCE_LINE = re.compile(r"^\s*(```|~~~)")
|
||
OUTPUT_PARTS = 2 # task parts per output after the first (order 22 A2: one answer holds only so much; round 1's pack 1,
|
||
# seven parts in one output, hit the limit and was written again with 84 fragments where wave 3 wrote
|
||
# 447). The first output holds part 1 alone: the first answer also holds the thinking (610 s before
|
||
# the cut, 49 s after it).
|
||
CITED_SCHEMA = "okf-wiki/cited-blocks/1"
|
||
REASONS = ("malformed", "anchor-outside-pack", "anchor-crosses-blocks", "sentence-count", "source-missing",
|
||
"quote-missing", "quote-not-in-span", "provenance-missing", "provenance-unknown", "tests-missing",
|
||
"release-note-new", "meta")
|
||
HOMELESS_APPROVED = "homeless-approved.json" # in the work folder; only the operator changes it (order 28 A5)
|
||
HOMELESS_SCHEMA = "okf-wiki/homeless-approved/1"
|
||
RESERVED_IDS = frozenset({"index", "log"}) # index.md and log.md are the bundle's own files (SPEC 3.1)
|
||
INDEX_SCHEMA = "okf-wiki/concept-index/1"
|
||
ACCEPTED_SCHEMA = "okf-wiki/accepted/1"
|
||
PACKS_SCHEMA = "okf-wiki/packs/1"
|
||
MANIFEST = "source-manifest.json"
|
||
MANIFEST_SCHEMA = "okf-wiki/source-manifest/1"
|
||
WEEK = re.compile(r"^docs-.*-(\d{4})-w(\d{2})\.md$")
|
||
STOP_LINES = "build,link,orchestration-build"
|
||
RETIRED = frozenset({"link-rule"}) # inbound from any page; Step 7 retires it (K2 point 5)
|
||
EXIT_STOP = 1 # the stop line, as okfwiki.cost stop gives it
|
||
EXIT_RETRY = 3
|
||
EXIT_GUARD = 4
|
||
EXIT_BREACH = 5
|
||
EXIT_REJECTED_TWICE = 6
|
||
EXIT_THIN = 7 # a source cites fewer anchor blocks than its floor (order 22 A2, condition b of wave 4's goal)
|
||
|
||
CONTRACT = """## Output contract
|
||
|
||
Write one JSON object `{"fragments": [...], "main_pages": [...], "homes": [...]}` to each output path named in your
|
||
prompt. When it names more than one, each names the parts it holds: write them one at a time, each in its own turn and
|
||
in the order given, each with the fragments of the blocks in the parts it names; `main_pages` and `homes` may stand in
|
||
any of them. One answer holds only so much, and an output left unwritten makes the whole pack run again. Each
|
||
fragment has:
|
||
|
||
- `concept` and `status`: "existing" with the index `id`, or "new" with `concept` its title as a reader would write it,
|
||
in the language and letters of the source (never an id), and `tests`: {"topic_shape", "citation"}.
|
||
- `text`: one to three sentences.
|
||
- `anchor`: {"file", "start", "end"} inside one block above.
|
||
- `quote`: one piece copied character for character from one line inside the anchor. Every fragment has a quote.
|
||
Code cites the whole line it stands on, and each sentence of `text` says only what that line and its context say.
|
||
- `context`: the other lines the text draws on, at most two pieces, each copied character for character from a
|
||
heading of the anchor's path (shown after the file name) or from another line of the same section: the subject a
|
||
heading names, or the condition, step or setting another line states. Leave it out only when the text says nothing
|
||
that its quote's line does not. Code places each on that heading or line and cites it too; a piece it cannot place
|
||
is dropped.
|
||
- `provenance`, one of three values, and the quote is required with each of them:
|
||
"documented" when the text says what its quote says;
|
||
"inferred" when the text draws a conclusion from lines of its anchor that no single line states;
|
||
"merged" when the text combines what several lines of its anchor say; name those lines in `context`, or code
|
||
cites the other lines of the anchor, the eight nearest the quote.
|
||
|
||
`main_pages`: one {"source", "concept"} for each source marked "name its main page": the concept that is that source's
|
||
main topic, named as your fragments from that source name it. A history source has no main page.
|
||
|
||
`homes`: one {"concept", "home"} for each concept you write a fragment for that is "new", or whose index line is
|
||
marked neither "main page" nor "has a home", unless you name it in `main_pages`. The `home` is an id under
|
||
"Main pages", or a concept you name in `main_pages`: the main page under which a reader would look for this concept.
|
||
A new concept from a history source needs a home like any other page; without one it is asked for again, and
|
||
the build stops when it has none.
|
||
"""
|
||
|
||
|
||
# --- the sources (contract K1) ---------------------------------------------------------------------------------
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class Fact:
|
||
"""What K1, or without it the file name, says about one source."""
|
||
|
||
kind: str
|
||
main_topic: bool
|
||
version: str | None = None
|
||
original: str | None = None # the path the document has in its folder (sources folder), when K1 keeps it
|
||
places: tuple[dict[str, Any] | None, ...] | None = None # the page, paragraph, cell or slide of each line (PDF, Office)
|
||
|
||
def place(self, line: int) -> dict[str, Any]:
|
||
"""The place of line (1-based) in the original document; {} for a text source or a line without one."""
|
||
if self.places is None or not 0 < line <= len(self.places):
|
||
return {}
|
||
return dict(self.places[line - 1] or {})
|
||
|
||
@property
|
||
def history(self) -> bool:
|
||
"""A source without a main topic adds history to concepts and has no main page (K1 rule 4)."""
|
||
return not self.main_topic
|
||
|
||
|
||
def name_fact(name: str) -> Fact:
|
||
m = RELEASE.match(name)
|
||
return Fact("release-notes", False, m.group(1)) if m else Fact("docs", True)
|
||
|
||
|
||
def read_k1(sources: Path) -> dict[str, Fact] | None:
|
||
"""The facts of every source K1 lists, by file name under kilde/; None when the root has no K1."""
|
||
path = sources / MANIFEST
|
||
if not path.is_file():
|
||
return None
|
||
try:
|
||
doc = json.loads(path.read_text(encoding="utf-8"))
|
||
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||
raise RunError(f"{path} cannot be read as K1: {exc}") from exc
|
||
if not (isinstance(doc, dict) and doc.get("schema") == MANIFEST_SCHEMA and isinstance(doc.get("sources"), list)):
|
||
raise RunError(f"{path} is not a K1 source manifest ({MANIFEST_SCHEMA} with a list of sources)")
|
||
facts: dict[str, Fact] = {}
|
||
for e in doc["sources"]:
|
||
if not (isinstance(e, dict) and isinstance(e.get("path"), str) and e["path"].startswith("kilde/")
|
||
and isinstance(e.get("kind"), str) and isinstance(e.get("main_topic"), bool)):
|
||
raise RunError(f"{path}: an entry without a kilde/ path, a kind or a boolean main_topic: {e!r}")
|
||
version, original, places = e.get("version"), e.get("original"), e.get("places")
|
||
facts[e["path"][len("kilde/"):]] = Fact(e["kind"], e["main_topic"], version if isinstance(version, str) else None,
|
||
original if isinstance(original, str) else None,
|
||
tuple(places) if isinstance(places, list) else None)
|
||
return facts
|
||
|
||
|
||
def source_facts(sources: Path, names: list[str]) -> dict[str, Fact]:
|
||
"""K1 decides kind and main topic when the sources root has it; the file name decides otherwise, as before."""
|
||
listed = read_k1(sources)
|
||
if listed is None:
|
||
return {n: name_fact(n) for n in names}
|
||
missing = [n for n in names if n not in listed]
|
||
if missing:
|
||
raise RunError(f"{sources / MANIFEST} does not list {', '.join(missing[:5])}")
|
||
return {n: listed[n] for n in names}
|
||
|
||
|
||
def week_of(name: str) -> str | None:
|
||
"""The week of a weekly page from its file name (docs-...-YYYY-wNN.md -> YYYY-wNN)."""
|
||
m = WEEK.match(name)
|
||
return f"{m.group(1)}-w{m.group(2)}" if m else None
|
||
|
||
|
||
def source_line(name: str, fact: Fact) -> str:
|
||
"""A source as the pack's task lists it: its name, the name it has in its folder when that differs (the safe name
|
||
of a folder's document says less, and a text without a heading has no other title), and what it is."""
|
||
shown = f"{name} (original name: {fact.original})" if fact.original and fact.original != name else name
|
||
return f"{shown}: {describe(name, fact)}"
|
||
|
||
|
||
def describe(name: str, fact: Fact) -> str:
|
||
if fact.main_topic:
|
||
return f"{fact.kind} page; name its main page"
|
||
week, version = week_of(name), fact.version or name_fact(name).version
|
||
label = f"week {week}" if week else f"version {version}" if version else fact.kind
|
||
return f"history ({label}); no main page"
|
||
|
||
|
||
# --- names and identity ----------------------------------------------------------------------------------------
|
||
|
||
|
||
def norm(name: str) -> str:
|
||
text = name.replace("`", "").lower()
|
||
text = re.sub(r"[^\w\s/.\-]", " ", text)
|
||
words = text.split()
|
||
if words and words[0] == "the":
|
||
words = words[1:]
|
||
words = [w[:-1] if len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
|
||
return " ".join(w.strip(".") or w for w in words)
|
||
|
||
|
||
def unique_id(index: dict[str, Any], title: str) -> str:
|
||
taken = {e["id"] for e in index["entries"]} | RESERVED_IDS
|
||
base = slug(title)
|
||
eid, k = base, 2
|
||
while eid in taken:
|
||
eid, k = f"{base}-{k}", k + 1
|
||
return eid
|
||
|
||
|
||
def add_headings(index: dict[str, Any], kilde: Path, files: set[str]) -> tuple[int, int, int]:
|
||
"""Add an entry to the index for each name of the title and level 2-3 headings of `files` that passes concept test
|
||
2, one per normalised name (the first page in name order wins, a later spelling becomes an alias of the new entry).
|
||
A name equal to a name of an entry already in the index is not added, and that entry is never changed. (entries
|
||
added, names equal to an existing entry, headings that collide with a heading of the same name in another file)."""
|
||
existing = {m for e in index["entries"] for m in _names(e)}
|
||
seen: dict[str, dict[str, Any]] = {}
|
||
added = equal = collisions = 0
|
||
for h in docs_headings(kilde):
|
||
if is_meta(h.text) or h.file not in files:
|
||
continue
|
||
n = norm(h.text)
|
||
if not n:
|
||
continue
|
||
if n in existing:
|
||
equal += 1
|
||
continue
|
||
if n in seen:
|
||
if seen[n]["origin"]["file"] != h.file:
|
||
collisions += 1
|
||
if h.text not in seen[n]["aliases"]:
|
||
seen[n]["aliases"].append(h.text)
|
||
continue
|
||
e = {"id": unique_id(index, h.text), "title": h.text, "aliases": [h.text], "kind": "skeleton",
|
||
"merge_of": None, "origin": {"file": h.file, "line": h.line}}
|
||
index["entries"].append(e)
|
||
seen[n] = e
|
||
added += 1
|
||
return added, equal, collisions
|
||
|
||
|
||
def skeleton(kilde: Path, history: frozenset[str] = frozenset()) -> tuple[dict[str, Any], int]:
|
||
"""(the index, the number of headings that share a normalised name with a heading in another file). The files in
|
||
`history` (pages K1 marks as history) are left out: history adds to concepts and sets no structure."""
|
||
index: dict[str, Any] = {"schema": INDEX_SCHEMA, "entries": []}
|
||
pages = {p.name for p in source_files(kilde) if not RELEASE.match(p.name)}
|
||
_, _, collisions = add_headings(index, kilde, pages - history)
|
||
return index, collisions
|
||
|
||
|
||
def _names(e: dict[str, Any]) -> set[str]:
|
||
return {norm(a) for a in [e["title"], *e.get("aliases", [])]} - {""}
|
||
|
||
|
||
def _similar(a: str, b: str) -> float:
|
||
return jaccard(set(a.split()), set(b.split()))
|
||
|
||
|
||
def resolve(fragment: dict[str, Any], index: dict[str, Any], history: bool | None = None) -> tuple[str, str | None]:
|
||
"""existing <id> | merge-candidate <id> | new | rejected <reason> (D6, D3b). A fragment from a history source
|
||
(`history`; by default a release-* file) that would make a new concept is rejected as release-note-new here;
|
||
accept resolves it again after the pack's other fragments and keeps it, homeless when it has no home."""
|
||
by_id = {e["id"]: e for e in index["entries"]}
|
||
if fragment.get("status") == "existing" and fragment.get("id") in by_id:
|
||
return "existing", str(fragment["id"])
|
||
if fragment.get("concept") in by_id: # an index id given as the concept, as main_pages and homes read it (lookup)
|
||
return "existing", str(fragment["concept"])
|
||
n = norm(str(fragment.get("concept", "")))
|
||
for e in index["entries"]:
|
||
if n in _names(e):
|
||
return "existing", str(e["id"])
|
||
if history is None:
|
||
history = RELEASE.match(str(fragment.get("anchor", {}).get("file", ""))) is not None
|
||
if history:
|
||
return "rejected", "release-note-new"
|
||
if is_meta(str(fragment.get("concept", ""))):
|
||
return "rejected", "meta"
|
||
best = max(((max(_similar(n, m) for m in _names(e)), str(e["id"])) for e in index["entries"] if _names(e)),
|
||
default=(0.0, ""))
|
||
if best[0] >= MERGE_JACCARD:
|
||
return "merge-candidate", best[1]
|
||
return "new", None
|
||
|
||
|
||
def lookup(index: dict[str, Any], name: Any) -> str | None:
|
||
"""The concept a name stands for: an index id as it is, else the entry whose normalised names hold it."""
|
||
if not isinstance(name, str) or not name.strip():
|
||
return None
|
||
if any(e["id"] == name for e in index["entries"]):
|
||
return name
|
||
n = norm(name)
|
||
return next((str(e["id"]) for e in index["entries"] if n and n in _names(e)), None)
|
||
|
||
|
||
def id_mismatch(fragment: dict[str, Any], index: dict[str, Any]) -> bool:
|
||
"""An "existing" fragment whose id names no entry, or an entry whose names match its concept neither by normalised
|
||
form nor by Jaccard >= MERGE_JACCARD (rev 4 Step 4: 0 of 650 in P1). Counted at accept, never a reject."""
|
||
if fragment.get("status") != "existing":
|
||
return False
|
||
if "id" not in fragment and any(e["id"] == fragment.get("concept") for e in index["entries"]):
|
||
return False # the concept is the index id itself (resolve)
|
||
entry = next((e for e in index["entries"] if e["id"] == fragment.get("id")), None)
|
||
if entry is None:
|
||
return True
|
||
concept = norm(str(fragment.get("concept", "")))
|
||
return concept not in _names(entry) and not any(_similar(concept, m) >= MERGE_JACCARD for m in _names(entry))
|
||
|
||
|
||
def relevant_lines(index: dict[str, Any], text: str) -> list[dict[str, Any]]:
|
||
"""The index entries whose normalised name or alias occurs as a whole phrase in the normalised text (D7)."""
|
||
hay = f" {norm(text)} "
|
||
return [e for e in index["entries"] if any(f" {m} " in hay for m in _names(e))]
|
||
|
||
|
||
# --- main pages and homes in the accepted state ----------------------------------------------------------------
|
||
|
||
|
||
def pages_of(acc: dict[str, Any]) -> set[str]:
|
||
"""The concepts that have a page: those with at least one accepted fragment."""
|
||
return {str(r["concept_id"]) for r in acc["fragments"]}
|
||
|
||
|
||
def main_ids(acc: dict[str, Any]) -> list[str]:
|
||
"""The main pages: concepts named for a source in main_pages that have a page, in record order, each once."""
|
||
pages = pages_of(acc)
|
||
out: list[str] = []
|
||
for m in acc.get("main_pages", []):
|
||
cid = str(m["concept_id"])
|
||
if cid in pages and cid not in out:
|
||
out.append(cid)
|
||
return out
|
||
|
||
|
||
def homed_ids(acc: dict[str, Any], mains: set[str], pages: set[str]) -> set[str]:
|
||
"""The concepts with a home record whose home is a main page, both with a page."""
|
||
return {str(h["concept_id"]) for h in acc.get("homes", [])
|
||
if str(h["home"]) in mains and str(h["home"]) in pages and str(h["concept_id"]) in pages}
|
||
|
||
|
||
def tags_for(acc: dict[str, Any]) -> dict[str, str]:
|
||
mains = set(main_ids(acc))
|
||
tags = {cid: "has a home" for cid in homed_ids(acc, mains, pages_of(acc))}
|
||
tags.update({cid: "main page" for cid in mains})
|
||
return tags
|
||
|
||
|
||
def first_files(acc: dict[str, Any]) -> dict[str, str]:
|
||
"""Per concept, the source of its first fragment in build order (pack, position, line): the source that brought
|
||
the page, which a home record given for it by the guard goes with."""
|
||
out: dict[str, str] = {}
|
||
for r in sorted(acc["fragments"], key=lambda r: (int(r["pack"]), int(r["position"]), int(r["start"]))):
|
||
out.setdefault(str(r["concept_id"]), str(r["file"]))
|
||
return out
|
||
|
||
|
||
def pass_homes(acc: dict[str, Any]) -> int:
|
||
"""A home belongs to its page; its record names the source that brings the page (first_files). After replace or
|
||
remove, a record whose page stays passes to the page's source now; the number passed."""
|
||
firsts = first_files(acc)
|
||
passed = 0
|
||
for h in acc["homes"]:
|
||
src = firsts.get(str(h["concept_id"]))
|
||
if src is not None and h["source"] != src:
|
||
h["source"] = src
|
||
passed += 1
|
||
return passed
|
||
|
||
|
||
def drop_dangling(acc: dict[str, Any]) -> tuple[int, int, list[str]]:
|
||
"""Drop the main page records that point to a page now gone, and every home record that is no longer valid: its
|
||
page or its home is gone, its home is no longer a main page (or is the page itself), its page is now a main page,
|
||
or a newer record of the same page follows it (the model's newest choice wins, one home per page). A record that
|
||
was dropped never comes back. Run after every write. (main page records dropped, home records dropped, the pages
|
||
a dropped home leaves without one, sorted)."""
|
||
pages = pages_of(acc)
|
||
mains = [m for m in acc["main_pages"] if str(m["concept_id"]) in pages]
|
||
acc_mains = {**acc, "main_pages": mains}
|
||
main_set = set(main_ids(acc_mains))
|
||
seen: set[str] = set()
|
||
homes: list[dict[str, Any]] = []
|
||
for h in reversed(acc["homes"]):
|
||
c, t = str(h["concept_id"]), str(h["home"])
|
||
if c in pages and t in main_set and t != c and c not in main_set and c not in seen:
|
||
homes.append(h)
|
||
seen.add(c)
|
||
homes.reverse()
|
||
left = sorted({str(h["concept_id"]) for h in acc["homes"]} & (pages - main_set - seen))
|
||
dropped = (len(acc["main_pages"]) - len(mains), len(acc["homes"]) - len(homes), left)
|
||
acc["main_pages"], acc["homes"] = mains, homes
|
||
return dropped
|
||
|
||
|
||
def left_without(acc: dict[str, Any], mains_before: set[str], dropped: list[str]) -> list[str]:
|
||
"""The pages left without a home: those a dropped home left (drop_dangling), and those that lost their main role
|
||
and have no home; each still a page and no main page, sorted."""
|
||
pages, mains = pages_of(acc), set(main_ids(acc))
|
||
homed = {str(h["concept_id"]) for h in acc["homes"]}
|
||
lost = {c for c in mains_before if c in pages and c not in mains and c not in homed}
|
||
return sorted(set(dropped) | lost)
|
||
|
||
|
||
def catalogue_line(cid: str, title: str) -> str:
|
||
return f"- {cid} | {' '.join(title.split())}\n"
|
||
|
||
|
||
def main_catalogue(index: dict[str, Any], acc: dict[str, Any], pack: int, budget: int = klink.CATALOGUE_CHARS
|
||
) -> list[dict[str, Any]]:
|
||
"""The main pages a home may be chosen from for pack N (Retning point 4: the catalogue filtered per pack), ranked
|
||
as the link pass ranks its catalogue and cut at the first that does not fit the character budget: (0) written to
|
||
in this pack, (1) in the RECENT_PACKS packs before it, latest first, (2) the rest by how many distinct source
|
||
files they cite, most first; ties by id. Structure only, never names."""
|
||
titles = {str(e["id"]): str(e.get("title") or e["id"]) for e in index["entries"]}
|
||
last: dict[str, int] = {}
|
||
files: dict[str, set[str]] = {}
|
||
for r in acc["fragments"]:
|
||
cid, p = str(r["concept_id"]), int(r["pack"])
|
||
files.setdefault(cid, set()).add(str(r["file"]))
|
||
if p <= pack:
|
||
last[cid] = max(last.get(cid, 0), p)
|
||
|
||
def rank(cid: str) -> tuple[int, int, str]:
|
||
lp = last.get(cid)
|
||
if lp == pack:
|
||
return 0, 0, cid
|
||
if lp is not None and pack - lp <= klink.RECENT_PACKS:
|
||
return 1, -lp, cid
|
||
return 2, -len(files.get(cid, set())), cid
|
||
|
||
out: list[dict[str, Any]] = []
|
||
used = 0
|
||
for cid in sorted(main_ids(acc), key=rank):
|
||
size = len(catalogue_line(cid, titles.get(cid, cid)))
|
||
if used + size > budget:
|
||
break
|
||
out.append({"id": cid, "title": titles.get(cid, cid)})
|
||
used += size
|
||
return out
|
||
|
||
|
||
# --- packs and the compiler's input ----------------------------------------------------------------------------
|
||
|
||
|
||
def packs(names: list[str], size: int = DEFAULT_SIZE) -> list[list[str]]:
|
||
return [names[i:i + size] for i in range(0, len(names), size)]
|
||
|
||
|
||
def group(name: str) -> str:
|
||
"""A source's group in K's order: release notes, or docs pages (the weekly pages among them)."""
|
||
return "release" if RELEASE.match(name) else "docs"
|
||
|
||
|
||
def new_packs(rest: list[str], size: int, cap: int | None, apart: bool, blocks: Mapping[str, int],
|
||
char_cap: int | None = None, chars: Mapping[str, int] | None = None) -> list[list[str]]:
|
||
"""The sources in no registered pack, in packs (order 28 A2): a pack takes the next source while it holds fewer
|
||
than `size`, the source is of its group when `apart`, its blocks stay within `cap` and the characters of its
|
||
blocks within `char_cap` (B17); a source over a cap alone is a pack of its own."""
|
||
out: list[list[str]] = []
|
||
pack: list[str] = []
|
||
held = held_chars = 0
|
||
for name in rest:
|
||
w, wc = blocks.get(name, 0), (chars or {}).get(name, 0)
|
||
if pack and (len(pack) >= size or (apart and group(name) != group(pack[0]))
|
||
or (cap is not None and held + w > cap)
|
||
or (char_cap is not None and held_chars + wc > char_cap)):
|
||
out.append(pack)
|
||
pack, held, held_chars = [], 0, 0
|
||
pack.append(name)
|
||
held, held_chars = held + w, held_chars + wc
|
||
return out + ([pack] if pack else [])
|
||
|
||
|
||
def layout(order: list[str], log: dict[str, Any], size: int, cap: int | None = None, apart: bool = False,
|
||
blocks: Mapping[str, int] | None = None, char_cap: int | None = None,
|
||
chars: Mapping[str, int] | None = None) -> list[list[str]]:
|
||
"""Every pack's sources (order 27 A2: a built collection is continued, never packed again). A pack packs.json
|
||
registers (a record with its sources, written by run-before or accept) keeps its sources and its place; the sources
|
||
of the order in no registered pack fill the other pack numbers in turn, `size` at a time, in K's order (docs pages
|
||
before release notes, kblocks.order), so after the last registered pack when packs 1..K are registered. Without a
|
||
registered pack this is packs(order, size) for K's order. With `apart` a new pack never holds docs pages and
|
||
release notes together, and with `cap` (and the blocks per source) it closes before the source that would take it
|
||
over the cap (order 28 A2). A registered source the order lacks, a source registered twice, or a pack number left
|
||
without sources cannot run."""
|
||
if cap is not None and blocks is None:
|
||
raise RunError("a cap of blocks per pack needs the blocks of each source")
|
||
in_order = set(order)
|
||
registered: dict[int, list[str]] = {}
|
||
seen: set[str] = set()
|
||
for r in log.get("packs", []):
|
||
names = r.get("sources")
|
||
if not isinstance(names, list):
|
||
continue
|
||
for name in names:
|
||
if name not in in_order:
|
||
raise RunError(f"pack {r['pack']}'s source {name} is not in the order")
|
||
if name in seen:
|
||
raise RunError(f"{name} is registered in two packs")
|
||
seen.add(name)
|
||
registered[int(r["pack"])] = [str(x) for x in names]
|
||
rest = new_packs(kblocks_order([n for n in order if n not in seen]), size, cap, apart, blocks or {}, char_cap,
|
||
chars)
|
||
out: list[list[str]] = []
|
||
last = max(registered, default=0)
|
||
while rest or len(out) < last:
|
||
n = len(out) + 1
|
||
if n in registered:
|
||
out.append(registered[n])
|
||
elif rest:
|
||
out.append(rest.pop(0))
|
||
else:
|
||
raise RunError(f"pack {n} has no sources: the order holds none that is not registered in a pack")
|
||
return out
|
||
|
||
|
||
def blocks_per_source(doc: dict[str, Any]) -> Counter[str]:
|
||
return Counter(str(b["file"]) for b in doc["blocks"])
|
||
|
||
|
||
def chars_per_source(doc: dict[str, Any]) -> Counter[str]:
|
||
"""The characters of each source's blocks (kblocks `chars`; 0 in a blocks file written before blocks had them)."""
|
||
out: Counter[str] = Counter()
|
||
for b in doc["blocks"]:
|
||
out[str(b["file"])] += int(b.get("chars") or 0)
|
||
return out
|
||
|
||
|
||
def next_pack(doc: dict[str, Any], log: dict[str, Any], last: int, size: int, done: int, total: int,
|
||
cap: int | None = None, apart: bool = False, char_cap: int | None = None,
|
||
chars: Mapping[str, int] | None = None) -> list[str]:
|
||
"""The sources of the pack after `last`, which the stop line counts on before the next check; none when the
|
||
run's blocks are done (done >= total) or no pack follows. A root may hold sources after the run's last pack
|
||
(order 27: variants S and R do not build the release notes of their root)."""
|
||
if done >= total:
|
||
return []
|
||
all_packs = layout([str(x) for x in doc["order"]], log, size, cap, apart, blocks_per_source(doc), char_cap,
|
||
chars_per_source(doc) if chars is None else chars)
|
||
return all_packs[last] if last < len(all_packs) else []
|
||
|
||
|
||
def next_blocks(doc: dict[str, Any], log: dict[str, Any], last: int, size: int, done: int, total: int,
|
||
cap: int | None = None, apart: bool = False, char_cap: int | None = None) -> int:
|
||
names = next_pack(doc, log, last, size, done, total, cap, apart, char_cap)
|
||
return sum(b["file"] in names for b in doc["blocks"])
|
||
|
||
|
||
def pack_chars(doc: dict[str, Any], names: list[str]) -> int | None:
|
||
"""The characters of the blocks of `names` (kblocks `chars`), or None for a blocks file written before blocks
|
||
had characters: the unit the stop line prices the next pack by (motor B17)."""
|
||
mine = [b for b in doc["blocks"] if b["file"] in names]
|
||
return sum(int(b["chars"]) for b in mine) if all(isinstance(b.get("chars"), int) for b in mine) else None
|
||
|
||
|
||
def render_input(pack_no: int, names: list[str], blocks_doc: dict[str, Any], index_lines: list[dict[str, Any]],
|
||
kilde: Path, facts: dict[str, Fact] | None = None, tags: dict[str, str] | None = None,
|
||
catalogue: list[dict[str, Any]] | None = None) -> list[str]:
|
||
facts = facts or {n: name_fact(n) for n in names}
|
||
lines = {n: (kilde / n).read_text(encoding="utf-8").split("\n") for n in names}
|
||
pieces: list[str] = [f"# K pack {pack_no}: {len(names)} sources\n\nSources in order: {', '.join(names)}\n\n"
|
||
+ "## Sources\n\n" + "".join(f"- {source_line(n, facts[n])}\n" for n in names) + "\n"
|
||
+ CONTRACT + "\n## Blocks\n\n"]
|
||
for b in blocks_doc["blocks"]:
|
||
if b["file"] not in names:
|
||
continue
|
||
head = f"### {b['file']} › {b['section'] or '-'} › {b['start']}–{b['end']}"
|
||
if b.get("version"):
|
||
head += f" (version {b['version']})"
|
||
body = [f"{n}: {lines[b['file']][n - 1]}" for n in range(b["start"], b["end"] + 1)]
|
||
chunk = head + "\n"
|
||
for row in body:
|
||
if len(chunk) + len(row) + 1 > PART_CHARS // 2:
|
||
pieces.append(chunk + "\n")
|
||
chunk = head + " (continued)\n"
|
||
chunk += row + "\n"
|
||
pieces.append(chunk + "\n")
|
||
pieces.append("## Index (concepts named in these sources)\n\n")
|
||
for e in index_lines:
|
||
aliases = "; ".join(a for a in e["aliases"] if a != e["title"])
|
||
tag = (tags or {}).get(str(e["id"]))
|
||
pieces.append(f"- id: {e['id']} | {e['title']}" + (f" | aliases: {aliases}" if aliases else "")
|
||
+ (f" | {tag}" if tag else "") + "\n")
|
||
pieces.append("\n## Main pages (homes you may choose)\n\n")
|
||
if catalogue:
|
||
pieces += [catalogue_line(str(e["id"]), str(e["title"])) for e in catalogue]
|
||
else:
|
||
pieces.append("(none yet: the main pages you name in this pack are the homes you may choose)\n")
|
||
room = PART_CHARS - len("<!-- part 99 of 99 -->\n")
|
||
parts: list[str] = []
|
||
current = ""
|
||
for piece in pieces:
|
||
if current and len(current) + len(piece) > room:
|
||
parts.append(current)
|
||
current = ""
|
||
current += piece
|
||
if current:
|
||
parts.append(current)
|
||
total = len(parts)
|
||
return [f"<!-- part {i} of {total} -->\n" + p for i, p in enumerate(parts, 1)]
|
||
|
||
|
||
# --- the one hard check at writing -----------------------------------------------------------------------------
|
||
|
||
|
||
def validate(fragments: list[Any], names: list[str], blocks_doc: dict[str, Any], kilde: Path
|
||
) -> tuple[list[dict[str, Any]], Counter[str]]:
|
||
"""The fragments that pass, each with its quote line, version, section and position in `names`; and the rest
|
||
counted by the first reason that rejects them. Rule E: every fragment carries a quote that stands verbatim on one
|
||
line of its anchor span, and a provenance mark from PROVENANCE; the mark never exempts the quote."""
|
||
by_file: dict[str, list[dict[str, Any]]] = {}
|
||
for b in blocks_doc["blocks"]:
|
||
by_file.setdefault(b["file"], []).append(b)
|
||
texts: dict[str, list[str] | None] = {}
|
||
accepted: list[dict[str, Any]] = []
|
||
rejected: Counter[str] = Counter()
|
||
for f in fragments:
|
||
a = f.get("anchor") if isinstance(f, dict) else None
|
||
if not (isinstance(f, dict) and isinstance(f.get("concept"), str) and f["concept"].strip()
|
||
and f.get("status") in ("new", "existing") and isinstance(f.get("text"), str)
|
||
and isinstance(f.get("id", ""), str)
|
||
and isinstance(a, dict) and isinstance(a.get("file"), str)
|
||
and type(a.get("start")) is int and type(a.get("end")) is int and a["start"] <= a["end"]):
|
||
rejected["malformed"] += 1
|
||
continue
|
||
if a["file"] not in names:
|
||
rejected["anchor-outside-pack"] += 1
|
||
continue
|
||
block = next((b for b in by_file.get(a["file"], []) if b["start"] <= a["start"] and a["end"] <= b["end"]), None)
|
||
if block is None:
|
||
rejected["anchor-crosses-blocks"] += 1
|
||
continue
|
||
if not 1 <= len(sentence_spans(f["text"].strip())) <= 3:
|
||
rejected["sentence-count"] += 1
|
||
continue
|
||
if a["file"] not in texts:
|
||
try:
|
||
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
|
||
except (OSError, UnicodeDecodeError):
|
||
texts[a["file"]] = None
|
||
lines = texts[a["file"]]
|
||
if lines is None:
|
||
rejected["source-missing"] += 1
|
||
continue
|
||
quote = f.get("quote")
|
||
if not (isinstance(quote, str) and quote.strip()):
|
||
rejected["quote-missing"] += 1
|
||
continue
|
||
hit = next((a["start"] + i for i, line in enumerate(lines[a["start"] - 1:a["end"]]) if quote in line), None)
|
||
if hit is None:
|
||
rejected["quote-not-in-span"] += 1
|
||
continue
|
||
if "provenance" not in f:
|
||
rejected["provenance-missing"] += 1
|
||
continue
|
||
if not (isinstance(f["provenance"], str) and f["provenance"] in PROVENANCE):
|
||
rejected["provenance-unknown"] += 1
|
||
continue
|
||
tests = f.get("tests")
|
||
if f["status"] == "new" and not (isinstance(tests, dict) and str(tests.get("topic_shape", "")).strip()
|
||
and str(tests.get("citation", "")).strip()):
|
||
rejected["tests-missing"] += 1
|
||
continue
|
||
accepted.append({**f, "quote": quote, "quote_line": hit, "version": block.get("version"),
|
||
"section": block["section"], "position": names.index(a["file"]) + 1})
|
||
return accepted, rejected
|
||
|
||
|
||
def context_of(lines: list[str], block: dict[str, Any], quote_line: int, wanted: Any) -> tuple[list[dict[str, Any]], int]:
|
||
"""(the context quotes a fragment keeps, the number dropped). Order 22, correction O+L: a sentence may draw on a
|
||
heading above its quote or on another line of its section; each context quote the compiler names is placed on the
|
||
nearest line of the anchor block's section that holds it (the lines between the heading before the block and the
|
||
heading after it, never the quote's own line), else on the innermost heading of the block's heading path whose
|
||
text holds it. One that is not text, is found nowhere, repeats a line or comes after MAX_CONTEXT kept is dropped
|
||
and counted, never a reject: the fragment keeps its own quote."""
|
||
if wanted is None:
|
||
return [], 0
|
||
if not isinstance(wanted, list):
|
||
return [], 1
|
||
heads = headings_of(lines)
|
||
before = [h for h in heads if h[0] < block["start"]]
|
||
after = [h for h in heads if h[0] > block["end"]]
|
||
first = before[-1][0] + 1 if before else 1
|
||
last = after[0][0] - 1 if after else len(lines)
|
||
stack: list[tuple[int, int, str]] = []
|
||
for h in before:
|
||
stack = [x for x in stack if x[1] < h[1]] + [h]
|
||
near = sorted((n for n in range(first, last + 1) if n != quote_line and lines[n - 1].strip()),
|
||
key=lambda n: (abs(n - quote_line), n))
|
||
kept: list[dict[str, Any]] = []
|
||
dropped = 0
|
||
for q in wanted:
|
||
hit: dict[str, Any] | None = None
|
||
if isinstance(q, str) and q.strip() and "\n" not in q and len(kept) < MAX_CONTEXT:
|
||
n = next((n for n in near if q in lines[n - 1]), None)
|
||
if n is not None:
|
||
hit = {"line": n, "quote": q, "section": str(block["section"])}
|
||
else:
|
||
i = next((i for i in range(len(stack) - 1, -1, -1) if q in stack[i][2]), None)
|
||
if i is not None:
|
||
hit = {"line": stack[i][0], "quote": q, "section": " › ".join(t for _, _, t in stack[:i + 1])}
|
||
if hit is None or any(k["line"] == hit["line"] for k in kept):
|
||
dropped += 1
|
||
else:
|
||
kept.append(hit)
|
||
return kept, dropped
|
||
|
||
|
||
def anchor_context(lines: list[str], anchor: dict[str, Any], quote_line: int, section: str) -> list[dict[str, Any]]:
|
||
"""The context a merged fragment that names none cites (order 25): the other lines of its anchor that hold a word,
|
||
fence lines left out, the MERGED_LINES nearest the quote line, in line order."""
|
||
near = sorted((n for n in range(anchor["start"], anchor["end"] + 1)
|
||
if n != quote_line and 0 < n <= len(lines) and re.search(r"\w", lines[n - 1])
|
||
and not FENCE_LINE.match(lines[n - 1])),
|
||
key=lambda n: (abs(n - quote_line), n))[:MERGED_LINES]
|
||
return [{"line": n, "quote": lines[n - 1].strip(), "section": section} for n in sorted(near)]
|
||
|
||
|
||
@dataclass
|
||
class Context:
|
||
kept: int = 0
|
||
dropped: int = 0
|
||
anchored: int = 0
|
||
anchor_lines: int = 0
|
||
|
||
def record(self) -> dict[str, int]:
|
||
"""packs.json's context: kept and dropped, and the anchor counts only when a merged fragment was widened."""
|
||
rec = {"kept": self.kept, "dropped": self.dropped}
|
||
if self.anchored:
|
||
rec.update({"anchored": self.anchored, "anchor_lines": self.anchor_lines})
|
||
return rec
|
||
|
||
|
||
def with_context(valid: list[dict[str, Any]], blocks_doc: dict[str, Any], kilde: Path
|
||
) -> tuple[list[dict[str, Any]], Context]:
|
||
"""The fragments that passed validate with their context quotes placed by context_of (the key left out when none
|
||
is kept), a merged fragment that keeps none citing its anchor (anchor_context), and the counts."""
|
||
texts: dict[str, list[str]] = {}
|
||
out: list[dict[str, Any]] = []
|
||
counts = Context()
|
||
for given in valid:
|
||
f = {k: v for k, v in given.items() if k != "context"}
|
||
a = f["anchor"]
|
||
block = next(b for b in blocks_doc["blocks"]
|
||
if b["file"] == a["file"] and b["start"] <= a["start"] and a["end"] <= b["end"])
|
||
if a["file"] not in texts:
|
||
texts[a["file"]] = (kilde / a["file"]).read_text(encoding="utf-8").split("\n")
|
||
kept, dropped = context_of(texts[a["file"]], block, int(f["quote_line"]), given.get("context"))
|
||
counts.kept, counts.dropped = counts.kept + len(kept), counts.dropped + dropped
|
||
if not kept and f["provenance"] == "merged":
|
||
kept = anchor_context(texts[a["file"]], a, int(f["quote_line"]), str(block["section"]))
|
||
counts.anchored, counts.anchor_lines = counts.anchored + bool(kept), counts.anchor_lines + len(kept)
|
||
if kept:
|
||
f["context"] = kept
|
||
out.append(f)
|
||
return out, counts
|
||
|
||
|
||
def new_record(f: dict[str, Any], outcome: str, ref: str | None, index: dict[str, Any], pack: int,
|
||
position: int | None) -> dict[str, Any]:
|
||
"""The accepted record of one fragment; a new concept or merge candidate is added to the index."""
|
||
if outcome == "existing":
|
||
cid = str(ref)
|
||
else:
|
||
cid = unique_id(index, f["concept"])
|
||
index["entries"].append({"id": cid, "title": f["concept"], "aliases": [f["concept"]], "kind": outcome,
|
||
"merge_of": ref, "origin": {"file": f["anchor"]["file"], "line": f["anchor"]["start"]},
|
||
"tests": f.get("tests")})
|
||
return {"pack": pack, "position": f["position"] if position is None else position,
|
||
"file": f["anchor"]["file"], "start": f["anchor"]["start"], "end": f["anchor"]["end"],
|
||
"section": f["section"], "text": f["text"].strip(), "quote": f["quote"],
|
||
"quote_line": f["quote_line"], "version": f["version"], "provenance": f["provenance"],
|
||
"concept_id": cid, "outcome": outcome, **({"context": f["context"]} if f.get("context") else {})}
|
||
|
||
|
||
@dataclass
|
||
class Admitted:
|
||
records: list[dict[str, Any]]
|
||
rejected: Counter[str]
|
||
main_pages: list[dict[str, Any]]
|
||
homes: list[dict[str, Any]]
|
||
main_stats: dict[str, Any]
|
||
home_stats: dict[str, Any]
|
||
|
||
|
||
def admit_pack(valid: list[dict[str, Any]], index: dict[str, Any], pack: int, names: list[str],
|
||
facts: dict[str, Fact], declared_main: list[Any], declared_homes: list[Any], before: dict[str, Any],
|
||
position: int | None = None) -> Admitted:
|
||
"""The fragments that passed validate, resolved to concepts, with the pack's main pages and homes.
|
||
|
||
`before` is the accepted state the pack joins (for replace, without the replaced source's records). A main page
|
||
is kept for a source of the pack with a main topic when the named concept has a fragment from that source (K2
|
||
point 1). A history fragment that would make a new concept is resolved after the others and needs a home like any
|
||
page. A home is recorded for every concept of the pack that has neither a main role nor a home, when the declared
|
||
home is a main page other than itself; the last such home declared for a page wins, and every declared home of
|
||
that page that fails is counted invalid. The rest are homeless (listed, never rejected: the guard asks for them).
|
||
`position` overrides the position in the pack."""
|
||
slots: list[tuple[int, dict[str, Any]]] = []
|
||
rejected: Counter[str] = Counter()
|
||
deferred: list[tuple[int, dict[str, Any]]] = []
|
||
for i, f in enumerate(valid):
|
||
outcome, ref = resolve(f, index, history=facts[f["anchor"]["file"]].history)
|
||
if outcome == "rejected" and ref == "release-note-new":
|
||
deferred.append((i, f))
|
||
elif outcome == "rejected":
|
||
rejected[str(ref)] += 1
|
||
else:
|
||
slots.append((i, new_record(f, outcome, ref, index, pack, position)))
|
||
by_source: dict[str, set[str]] = {}
|
||
for _, r in slots:
|
||
by_source.setdefault(r["file"], set()).add(r["concept_id"])
|
||
mains: list[dict[str, Any]] = []
|
||
for m in declared_main:
|
||
if not isinstance(m, dict):
|
||
continue
|
||
s = source_name(m["source"]) if isinstance(m.get("source"), str) else ""
|
||
cid = lookup(index, m.get("id")) or lookup(index, m.get("concept"))
|
||
if (s in names and facts[s].main_topic and all(x["source"] != s for x in mains) and cid is not None
|
||
and cid in by_source.get(s, set())):
|
||
mains.append({"source": s, "concept_id": cid, "pack": pack, "by": "compiler"})
|
||
main_now = set(main_ids(before)) | {m["concept_id"] for m in mains}
|
||
homes: list[tuple[str, Any, Any]] = [(str(h.get("concept", "")), h.get("id"), h.get("home"))
|
||
for h in declared_homes if isinstance(h, dict)]
|
||
for i, f in deferred: # after the docs fragments, so a concept they made in this pack is found first
|
||
outcome, ref = resolve(f, index, history=False)
|
||
if outcome == "rejected":
|
||
rejected[str(ref)] += 1
|
||
else:
|
||
slots.append((i, new_record(f, outcome, ref, index, pack, position)))
|
||
records = [r for _, r in sorted(slots, key=lambda s: s[0])]
|
||
pages = pages_of(before) | {r["concept_id"] for r in records}
|
||
homed = homed_ids(before, main_now, pages)
|
||
first: dict[str, str] = {}
|
||
for r in records:
|
||
first.setdefault(r["concept_id"], r["file"])
|
||
need = [c for c in first if c not in main_now and c not in homed]
|
||
owners = [lookup(index, eid) or lookup(index, concept) for concept, eid, _ in homes]
|
||
home_records: list[dict[str, Any]] = []
|
||
homeless: list[str] = []
|
||
used: set[int] = set()
|
||
invalid = 0
|
||
for c in need: # the model's newest choice wins: the last declared home that passes; each that fails is invalid
|
||
mine = [j for j, owner in enumerate(owners) if owner == c]
|
||
used.update(mine)
|
||
chosen = None
|
||
for j in mine:
|
||
hid = lookup(index, homes[j][2])
|
||
if hid is not None and hid in main_now and hid != c and hid in pages:
|
||
chosen = hid
|
||
else:
|
||
invalid += 1
|
||
if chosen is None:
|
||
homeless.append(c)
|
||
else:
|
||
home_records.append({"source": first[c], "concept_id": c, "home": chosen, "pack": pack, "by": "compiler"})
|
||
not_needed = sum(1 for j, owner in enumerate(owners) if j not in used and owner in first)
|
||
unknown = sum(1 for j, owner in enumerate(owners) if j not in used and owner not in first)
|
||
main_stats = {"named": len(declared_main), "valid": len(mains),
|
||
"missing": [s for s in names if facts[s].main_topic and all(m["source"] != s for m in mains)]}
|
||
home_stats = {"needed": len(need), "given": len(home_records), "homeless": homeless, "not_needed": not_needed,
|
||
"invalid": invalid, "unknown": unknown}
|
||
return Admitted(records, rejected, mains, home_records, main_stats, home_stats)
|
||
|
||
|
||
# --- the work folder -------------------------------------------------------------------------------------------
|
||
|
||
|
||
def read_json(path: Path) -> Any:
|
||
try:
|
||
return json.loads(path.read_text(encoding="utf-8"))
|
||
except (OSError, json.JSONDecodeError) as exc:
|
||
raise RunError(f"cannot read {path}: {exc}") from exc
|
||
|
||
|
||
def dump(data: Any) -> str:
|
||
return json.dumps(data, ensure_ascii=False, indent=1) + "\n"
|
||
|
||
|
||
def k_work(work: Path) -> Path:
|
||
return work / "K-work"
|
||
|
||
|
||
def check_pending(work: Path) -> None:
|
||
"""A marker left by a write that did not finish stops every command until the operator has dealt with it."""
|
||
markers = sorted(k_work(work).glob("*.pending"))
|
||
if not markers:
|
||
return
|
||
marker = markers[0]
|
||
try:
|
||
info = json.loads(marker.read_text(encoding="utf-8"))
|
||
how = (f"copy {', '.join(info['restore']) or 'nothing'} back from {k_work(work) / 'backup'}, and delete "
|
||
f"{', '.join(info['delete']) or 'nothing'} (they did not exist before)"
|
||
+ "".join(f", and move {dst} back to {src} (if it is there)" for src, dst in info.get("move_back", [])))
|
||
except (OSError, ValueError, KeyError, TypeError):
|
||
how = f"read {marker} and undo the write by hand"
|
||
raise RunError(f"{marker} is set: an earlier write did not finish. To undo it: {how}; then delete the marker")
|
||
|
||
|
||
def write_state(work: Path, label: str, files: dict[str, Any], outside: dict[Path, str] | None = None,
|
||
moves: dict[Path, Path] | None = None) -> None:
|
||
"""Write K-work files (and new files outside K-work) under a marker: back up the K-work files about to be
|
||
replaced, set the marker with what to restore, what to delete and which files to move back, move (each source to
|
||
a destination that does not exist), write, clear the marker."""
|
||
kw = k_work(work)
|
||
kw.mkdir(parents=True, exist_ok=True)
|
||
backup = kw / "backup"
|
||
if backup.exists():
|
||
shutil.rmtree(backup)
|
||
backup.mkdir()
|
||
existed = [name for name in files if (kw / name).is_file()]
|
||
for name in existed:
|
||
shutil.copy2(kw / name, backup / name)
|
||
marker = kw / f"{label}.pending"
|
||
new = sorted(name for name in files if name not in existed) + [str(path) for path in outside or {}]
|
||
marker.write_text(dump({"command": label, "restore": existed, "delete": new,
|
||
"move_back": [[str(a), str(b)] for a, b in (moves or {}).items()]}), encoding="utf-8")
|
||
for a, b in (moves or {}).items():
|
||
os.replace(a, b)
|
||
for path, text in (outside or {}).items():
|
||
path.parent.mkdir(parents=True, exist_ok=True)
|
||
path.write_text(text, encoding="utf-8")
|
||
for name, data in files.items():
|
||
tmp = kw / f".{name}.tmp"
|
||
tmp.write_text(dump(data), encoding="utf-8")
|
||
os.replace(tmp, kw / name)
|
||
marker.unlink()
|
||
|
||
|
||
def read_index(work: Path) -> dict[str, Any]:
|
||
path = k_work(work) / "index.json"
|
||
if not path.is_file():
|
||
raise RunError(f"{path} does not exist; run skeleton first")
|
||
index: dict[str, Any] = read_json(path)
|
||
return index
|
||
|
||
|
||
def read_accepted(work: Path) -> dict[str, Any]:
|
||
path = k_work(work) / "accepted.json"
|
||
acc: dict[str, Any] = read_json(path) if path.is_file() else {"schema": ACCEPTED_SCHEMA, "fragments": []}
|
||
acc.setdefault("main_pages", [])
|
||
acc.setdefault("homes", [])
|
||
return acc
|
||
|
||
|
||
def pack_log(work: Path) -> dict[str, Any]:
|
||
path = k_work(work) / "packs.json"
|
||
log: dict[str, Any] = read_json(path) if path.is_file() else {"schema": PACKS_SCHEMA, "packs": []}
|
||
return log
|
||
|
||
|
||
def record(log: dict[str, Any], n: int) -> dict[str, Any]:
|
||
for r in log["packs"]:
|
||
if r["pack"] == n:
|
||
return r # type: ignore[no-any-return]
|
||
r = {"pack": n}
|
||
log["packs"].append(r)
|
||
return r
|
||
|
||
|
||
def pack_size(log: dict[str, Any], given: int | None) -> int:
|
||
"""The size of a new pack: the one given, else the work folder's first size, else DEFAULT_SIZE. A registered pack
|
||
keeps its sources whatever size is given (layout; order 27 A2: the size is set per new pack)."""
|
||
if given is not None and given < 1:
|
||
raise RunError(f"--size {given} is not a pack size")
|
||
if given is not None:
|
||
return given
|
||
stored = log.get("size")
|
||
return DEFAULT_SIZE if stored is None else int(stored)
|
||
|
||
|
||
def pack_cap(log: dict[str, Any], given: int | None) -> int | None:
|
||
"""The cap of blocks for a new pack: the one given, else the work folder's, else none (order 28 A2)."""
|
||
if given is not None and given < 1:
|
||
raise RunError(f"--max-blocks {given} is not a cap of blocks")
|
||
if given is not None:
|
||
return given
|
||
stored = log.get("max_blocks")
|
||
return None if stored is None else int(stored)
|
||
|
||
|
||
def groups_apart(log: dict[str, Any], given: bool) -> bool:
|
||
"""Whether a new pack keeps docs pages and release notes apart: given once, the work folder keeps it."""
|
||
return given or bool(log.get("groups_apart"))
|
||
|
||
|
||
def shape(args: argparse.Namespace, log: dict[str, Any]) -> tuple[int, int | None, bool, int | None]:
|
||
"""(the size, the cap of blocks, groups apart, the cap of characters) for a new pack."""
|
||
return (pack_size(log, args.size), pack_cap(log, getattr(args, "max_blocks", None)),
|
||
groups_apart(log, bool(getattr(args, "groups_apart", False))),
|
||
pack_char_cap(log, getattr(args, "max_chars", None)))
|
||
|
||
|
||
def pack_char_cap(log: dict[str, Any], given: int | None) -> int | None:
|
||
"""The cap of characters for a new pack: the one given, else the work folder's, else none (B17; the operator
|
||
07.11: no artificial limits)."""
|
||
if given is not None and given < 1:
|
||
raise RunError(f"--max-chars {given} is not a cap of characters")
|
||
if given is not None:
|
||
return given
|
||
stored = log.get("max_chars")
|
||
return None if stored is None else int(stored)
|
||
|
||
|
||
def source_name(name: str) -> str:
|
||
"""A source as accepted.json names it: the file under kilde/; a K1 path (kilde/x.md, /kilde/x.md) is taken too."""
|
||
return name.removeprefix("/").removeprefix("kilde/")
|
||
|
||
|
||
def quiet(fn: Callable[[list[str]], int], argv: list[str]) -> tuple[int, str]:
|
||
out = io.StringIO()
|
||
with redirect_stdout(out), redirect_stderr(out):
|
||
code = fn(argv)
|
||
return code, out.getvalue()
|
||
|
||
|
||
# --- commands --------------------------------------------------------------------------------------------------
|
||
|
||
|
||
def cmd_skeleton(args: argparse.Namespace) -> int:
|
||
work, sources = Path(args.work), Path(args.sources)
|
||
check_pending(work)
|
||
out = k_work(work) / "index.json"
|
||
kilde = kilde_of(sources)
|
||
facts = source_facts(sources, sorted(p.name for p in kilde.iterdir() if p.is_file()))
|
||
if args.extend:
|
||
return extend_skeleton(work, kilde, facts)
|
||
if out.exists():
|
||
raise RunError(f"{out} exists; refusing to overwrite")
|
||
index, collisions = skeleton(kilde, frozenset(n for n, f in facts.items() if f.history))
|
||
write_state(work, "skeleton", {"index.json": index})
|
||
print(f"skeleton: {len(index['entries'])} entries -> {out}; {collisions} headings collide across files")
|
||
return 0
|
||
|
||
|
||
def extend_skeleton(work: Path, kilde: Path, facts: dict[str, Fact]) -> int:
|
||
"""skeleton --extend: the index of a built collection gains the docs pages with a main topic that are in no
|
||
registered pack and were not extended before (recorded in packs.json, so a page is extended once)."""
|
||
index, log = read_index(work), pack_log(work)
|
||
registered = {str(x) for r in log["packs"] for x in r.get("sources", [])}
|
||
extended = {str(x) for x in log.get("skeleton_extended", [])}
|
||
new = sorted(n for n, f in facts.items() if not f.history
|
||
and n not in registered and n not in extended)
|
||
added, equal, collisions = add_headings(index, kilde, set(new))
|
||
if new:
|
||
log["skeleton_extended"] = sorted(extended | set(new))
|
||
write_state(work, "skeleton-extend", {"index.json": index, "packs.json": log})
|
||
print(f"skeleton extended: {len(new)} new docs pages, {added} entries added, {equal} names equal to an existing "
|
||
f"entry, not added; {collisions} headings collide across the new pages -> {k_work(work) / 'index.json'}")
|
||
return 0
|
||
|
||
|
||
def interval(args: argparse.Namespace, log: dict[str, Any]) -> tuple[dict[str, Any], list[list[str]], int, int]:
|
||
"""(the blocks document, every pack's sources, the pack size, the last pack of the interval N..M)."""
|
||
doc: dict[str, Any] = read_json(Path(args.blocks))
|
||
size, cap, apart, char_cap = shape(args, log)
|
||
all_packs = layout([str(x) for x in doc["order"]], log, size, cap, apart, blocks_per_source(doc), char_cap,
|
||
chars_per_source(doc))
|
||
last = args.n if getattr(args, "to", None) is None else args.to
|
||
if last < args.n:
|
||
raise RunError(f"--to {last} is before pack {args.n}")
|
||
for n in (args.n, last):
|
||
if not 1 <= n <= len(all_packs):
|
||
raise RunError(f"pack {n} does not exist; there are {len(all_packs)} (1..{len(all_packs)})")
|
||
return doc, all_packs, size, last
|
||
|
||
|
||
def pack_names(args: argparse.Namespace, log: dict[str, Any]) -> tuple[dict[str, Any], list[str], int]:
|
||
"""(the blocks document, pack N's sources, the pack size)."""
|
||
doc, all_packs, size, _ = interval(args, log)
|
||
return doc, all_packs[args.n - 1], size
|
||
|
||
|
||
def cmd_run_before(args: argparse.Namespace) -> int:
|
||
work = Path(args.work)
|
||
check_pending(work)
|
||
log = pack_log(work)
|
||
doc, all_packs, size, last = interval(args, log)
|
||
sources = Path(args.sources)
|
||
kilde = kilde_of(sources)
|
||
index, acc = read_index(work), read_accepted(work)
|
||
tags = tags_for(acc)
|
||
out_dir = work / "K-oppgaver"
|
||
jobs = []
|
||
for n in range(args.n, last + 1): # render every pack before writing any, so a failure leaves nothing behind
|
||
names = all_packs[n - 1]
|
||
facts = source_facts(sources, names)
|
||
lines = relevant_lines(index, "\n".join((kilde / name).read_text(encoding="utf-8") for name in names))
|
||
catalogue = main_catalogue(index, acc, n)
|
||
parts = render_input(n, names, doc, lines, kilde, facts, tags, catalogue)
|
||
paths = [out_dir / f"pack-{n}-part-{i}.md" for i in range(1, len(parts) + 1)]
|
||
if any(p.exists() for p in paths):
|
||
raise RunError(f"pack {n}'s task parts exist in {out_dir}; refusing to overwrite")
|
||
jobs.append((n, names, lines, catalogue, parts, paths))
|
||
(work / "K-fragments").mkdir(parents=True, exist_ok=True) # where the compiler writes its output
|
||
_, cap, apart, char_cap = shape(args, log)
|
||
for n, names, lines, catalogue, parts, paths in jobs:
|
||
outputs = output_paths(work, n, len(parts))
|
||
rec = record(log, n)
|
||
rec.setdefault("size", size) # a registered pack keeps the size it was made with
|
||
if cap is not None: # and the cap; the work folder keeps the first cap and groups apart as its default
|
||
rec.setdefault("max_blocks", cap)
|
||
log.setdefault("max_blocks", cap)
|
||
if char_cap is not None:
|
||
rec.setdefault("max_chars", char_cap)
|
||
log.setdefault("max_chars", char_cap)
|
||
held = chars_per_source(doc)
|
||
for name in names:
|
||
if held[name] > LARGEST_HELD:
|
||
print(f"large: {name} holds {held[name]} characters of blocks, more than the {LARGEST_HELD} of the "
|
||
"largest source measured to hold in one compiler run; it runs as it is, and the stop line prices "
|
||
"it by its characters")
|
||
if apart:
|
||
log["groups_apart"] = True
|
||
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]),
|
||
**({} if (bc := pack_chars(doc, names)) is None else {"block_chars": bc}),
|
||
index_lines=len(lines), index_total=len(index["entries"]),
|
||
index_ids=[str(e["id"]) for e in lines], main_pages_sent=len(catalogue),
|
||
parts=len(parts), chars=sum(len(p) for p in parts), outputs=len(outputs))
|
||
log.setdefault("size", size) # the work folder's first size, the default for a new pack
|
||
write_state(work, f"pack-{n}", {"packs.json": log}, dict(zip(paths, parts)))
|
||
for p in paths:
|
||
print(f"part: {p}")
|
||
for path, label in outputs:
|
||
print(f"output: {path}" + (f" ({label})" if label else ""))
|
||
print(f"index lines sent: {len(lines)} of {len(index['entries'])}; main pages sent: {len(catalogue)}")
|
||
return 0
|
||
|
||
|
||
def output_paths(work: Path, n: int, parts: int) -> list[tuple[Path, str]]:
|
||
"""Pack N's outputs, each with the parts it holds: one output for a pack of OUTPUT_PARTS parts or fewer (the label
|
||
left empty), else part 1 alone and then one for every OUTPUT_PARTS parts, which the compiler writes one per turn."""
|
||
base = work / "K-fragments"
|
||
if parts <= OUTPUT_PARTS:
|
||
return [(base / f"pack-{n}.json", "")]
|
||
out: list[tuple[Path, str]] = []
|
||
for j, first in enumerate([1, *range(2, parts + 1, OUTPUT_PARTS)], 1):
|
||
last = 1 if first == 1 else min(first + OUTPUT_PARTS - 1, parts)
|
||
out.append((base / f"pack-{n}-out-{j}.json", f"part {first}" if first == last else f"parts {first}-{last}"))
|
||
return out
|
||
|
||
|
||
def pack_outputs(work: Path, n: int, log: dict[str, Any]) -> list[Path]:
|
||
"""The outputs run-before named for pack N; one, pack-N.json, for a pack rendered before outputs were counted."""
|
||
rec = record(log, n)
|
||
if "outputs" not in rec:
|
||
return [work / "K-fragments" / f"pack-{n}.json"]
|
||
paths = [p for p, _ in output_paths(work, n, int(rec.get("parts", 1)))]
|
||
if len(paths) != int(rec["outputs"]):
|
||
raise RunError(f"pack {n}'s log names {rec['outputs']} outputs for {rec.get('parts')} parts; "
|
||
f"this code gives {len(paths)}")
|
||
return paths
|
||
|
||
|
||
def read_outputs(paths: list[Path]) -> tuple[tuple[list[Any], list[Any], list[Any]], list[Path]]:
|
||
"""(the compiler's outputs read together, in order: fragments, main_pages, homes; the outputs not written)."""
|
||
fragments: list[Any] = []
|
||
mains: list[Any] = []
|
||
homes: list[Any] = []
|
||
missing: list[Path] = []
|
||
for path in paths:
|
||
if not path.is_file():
|
||
missing.append(path)
|
||
continue
|
||
f, m, h = read_compiled(path)
|
||
fragments += f
|
||
mains += m
|
||
homes += h
|
||
return (fragments, mains, homes), missing
|
||
|
||
|
||
def read_compiled(path: Path) -> tuple[list[Any], list[Any], list[Any]]:
|
||
"""(fragments, main_pages, homes) of the compiler's JSON; main_pages and homes may be left out."""
|
||
raw = read_json(path)
|
||
fragments = raw.get("fragments") if isinstance(raw, dict) else None
|
||
if not isinstance(fragments, list):
|
||
raise RunError(f"{path} has no fragments list")
|
||
lists: list[list[Any]] = []
|
||
for key in ("main_pages", "homes"):
|
||
value = raw.get(key, [])
|
||
if not isinstance(value, list):
|
||
raise RunError(f"{path}: {key} is not a list")
|
||
lists.append(value)
|
||
return fragments, lists[0], lists[1]
|
||
|
||
|
||
def print_rejected(rejected: Counter[str]) -> None:
|
||
for reason, count in sorted(rejected.items()):
|
||
print(f" rejected {reason}: {count}")
|
||
|
||
|
||
def print_context(c: Context) -> None:
|
||
if c.kept or c.dropped or c.anchored:
|
||
print(f" context quotes: kept {c.kept}, dropped {c.dropped}")
|
||
if c.anchored:
|
||
print(f" merged fragments cited by their anchor: {c.anchored} ({c.anchor_lines} lines)")
|
||
|
||
|
||
def print_roles(main_stats: dict[str, Any], home_stats: dict[str, Any]) -> None:
|
||
docs = main_stats["valid"] + len(main_stats["missing"])
|
||
without = f"; without one: {', '.join(main_stats['missing'])}" if main_stats["missing"] else ""
|
||
print(f" main pages: {main_stats['valid']} of {docs} docs sources (named {main_stats['named']}){without}")
|
||
homeless = f"; without a home: {', '.join(home_stats['homeless'])}" if home_stats["homeless"] else ""
|
||
kept = f"; kept the home they had: {home_stats['kept']}" if home_stats.get("kept") else ""
|
||
print(f" homes: {home_stats['given']} of {home_stats['needed']} pages that need one{kept}{homeless}")
|
||
|
||
|
||
def accept_pack(work: Path, n: int, doc: dict[str, Any], names: list[str], size: int, sources: Path,
|
||
compiled: tuple[list[Any], list[Any], list[Any]], log: dict[str, Any]) -> None:
|
||
fragments, declared_main, declared_homes = compiled
|
||
index = read_index(work)
|
||
acc = read_accepted(work)
|
||
rec = record(log, n)
|
||
if "accepted" in rec or any(a["pack"] == n for a in acc["fragments"]):
|
||
raise RunError(f"pack {n} is already accepted")
|
||
facts = source_facts(sources, names)
|
||
valid, rejected = validate(fragments, names, doc, kilde_of(sources))
|
||
valid, context = with_context(valid, doc, kilde_of(sources))
|
||
mismatches = sum(id_mismatch(f, index) for f in valid)
|
||
got = admit_pack(valid, index, n, names, facts, declared_main, declared_homes, acc)
|
||
rejected += got.rejected
|
||
sent = set(rec.get("index_ids", []))
|
||
not_sent = Counter(r["outcome"] for r in got.records if r["concept_id"] not in sent)
|
||
acc["fragments"].extend(got.records)
|
||
acc["main_pages"].extend(got.main_pages)
|
||
acc["homes"].extend(got.homes)
|
||
pass_homes(acc) # a home given in this pack to a page an earlier pack brought names that page's first source
|
||
dropped = drop_dangling(acc) # a page this pack makes a main page needs no home any more
|
||
rec.update(sources=names, blocks=sum(b["file"] in names for b in doc["blocks"]),
|
||
**({} if (bc := pack_chars(doc, names)) is None else {"block_chars": bc}), proposed=len(fragments),
|
||
accepted=len(got.records), rejected=dict(sorted(rejected.items())), id_mismatch=mismatches,
|
||
not_sent={"total": sum(not_sent.values()), "by_outcome": dict(sorted(not_sent.items()))},
|
||
main_pages=got.main_stats, homes=got.home_stats, cited=cited_blocks(got.records, doc, names),
|
||
**nonzero(dropped=dropped[0] + dropped[1]))
|
||
if context.kept or context.dropped or context.anchored:
|
||
rec["context"] = context.record()
|
||
log.setdefault("size", size)
|
||
write_state(work, f"accept-{n}", {"index.json": index, "accepted.json": acc, "packs.json": log})
|
||
print(f"pack {n}: accepted {len(got.records)} of {len(fragments)}")
|
||
print(f" id mismatches (counted, not rejected): {mismatches}")
|
||
print(f" accepted with a concept not among the index lines sent: {sum(not_sent.values())}")
|
||
print_context(context)
|
||
print_roles(got.main_stats, got.home_stats)
|
||
print_dropped(*dropped, named=got.home_stats["homeless"])
|
||
print_rejected(rejected)
|
||
|
||
|
||
def cmd_accept(args: argparse.Namespace) -> int:
|
||
work, n = Path(args.work), args.n
|
||
check_pending(work)
|
||
log = pack_log(work)
|
||
doc, names, size = pack_names(args, log)
|
||
compiled, missing = read_outputs(pack_outputs(work, n, log))
|
||
if missing:
|
||
raise RunError(f"pack {n}'s outputs are not all written: {', '.join(str(p) for p in missing)}")
|
||
accept_pack(work, n, doc, names, size, Path(args.sources), compiled, log)
|
||
return 0
|
||
|
||
|
||
def cited_blocks(records: list[dict[str, Any]], doc: dict[str, Any], names: list[str] | None = None
|
||
) -> dict[str, int]:
|
||
"""Per source, the distinct blocks that hold the anchor of an accepted record (condition b of wave 4's goal);
|
||
each of `names` is counted, with 0 when it has none."""
|
||
spans: dict[str, list[tuple[int, int]]] = {}
|
||
for b in doc["blocks"]:
|
||
spans.setdefault(str(b["file"]), []).append((int(b["start"]), int(b["end"])))
|
||
seen: dict[str, set[tuple[int, int]]] = {n: set() for n in names or []}
|
||
for r in records:
|
||
span = next((s for s in spans.get(str(r["file"]), []) if s[0] <= int(r["start"]) <= s[1]), None)
|
||
if span is None:
|
||
raise RunError(f"no block of {r['file']} holds line {r['start']}: the blocks file is not this build's")
|
||
seen.setdefault(str(r["file"]), set()).add(span)
|
||
return {s: len(v) for s, v in sorted(seen.items())}
|
||
|
||
|
||
def cmd_cited(args: argparse.Namespace) -> int:
|
||
work, out = Path(args.work), Path(args.out)
|
||
if out.exists():
|
||
raise RunError(f"{out} exists; refusing to overwrite")
|
||
doc: dict[str, Any] = read_json(Path(args.blocks))
|
||
counts = cited_blocks(read_accepted(work)["fragments"], doc)
|
||
out.write_text(dump({"schema": CITED_SCHEMA, "sources": counts}), encoding="utf-8")
|
||
print(f"cited: {sum(counts.values())} anchor blocks over {len(counts)} sources -> {out}")
|
||
return 0
|
||
|
||
|
||
NO_CLAIM = re.compile(r"(?:\*\s*){3,}|(?:-\s*){3,}|(?:_\s*){3,}|\*\*[^*]+\*\*:?")
|
||
|
||
|
||
def claim_blocks(doc: dict[str, Any], kilde: Path) -> Counter[str]:
|
||
"""Per source, the blocks that can carry a claim: all but a thematic break and a paragraph that is only bold text
|
||
(a step heading), which no build has cited (wiki-38: 0 of 433 and 0 of 23; c1 packs 1-16: 0 of 261 and 0 of 10).
|
||
A code block always counts: its fence line never matches."""
|
||
lines: dict[str, list[str]] = {}
|
||
out: Counter[str] = Counter()
|
||
for b in doc["blocks"]:
|
||
f = str(b["file"])
|
||
if f not in lines:
|
||
lines[f] = read_lines(kilde / f)
|
||
text = " ".join(x.strip() for x in lines[f][int(b["start"]) - 1:int(b["end"])]).strip()
|
||
if not NO_CLAIM.fullmatch(text):
|
||
out[f] += 1
|
||
return out
|
||
|
||
|
||
def cmd_floor(args: argparse.Namespace) -> int:
|
||
"""The floor for a later build of sources that changed since (a new fetch): per source the earlier build cited
|
||
and that still has blocks, floor(cited x its claim blocks now / its claim blocks then / 2); a history source gets
|
||
none of its own (history_rate, when given, holds a pack's together)."""
|
||
out = Path(args.out)
|
||
if out.exists():
|
||
raise RunError(f"{out} exists; refusing to overwrite")
|
||
cited = read_floor(Path(args.cited))
|
||
then_doc, now_doc = read_json(Path(args.blocks_then)), read_json(Path(args.blocks))
|
||
then = claim_blocks(then_doc, kilde_of(Path(args.sources_then)))
|
||
now = claim_blocks(now_doc, kilde_of(Path(args.sources)))
|
||
gone = sorted(s for s in cited if not now[s])
|
||
names = [s for s in cited if now[s]]
|
||
facts = source_facts(Path(args.sources), names)
|
||
floor = {s: int(cited[s] * now[s] / then[s] / 2) for s in sorted(names) if not facts[s].history and then[s]}
|
||
left = (sum(blocks_per_source(then_doc)[s] - then[s] for s in floor),
|
||
sum(blocks_per_source(now_doc)[s] - now[s] for s in floor))
|
||
doc: dict[str, Any] = {"schema": CITED_SCHEMA, "sources": floor}
|
||
if args.history_rate is not None:
|
||
if args.history_rate < 0:
|
||
raise RunError(f"--history-rate {args.history_rate} is not a rate of cited blocks per block")
|
||
doc["history_rate"] = args.history_rate
|
||
out.write_text(dump(doc), encoding="utf-8")
|
||
print(f"floor: {len(floor)} sources, {sum(floor.values())} anchor blocks; blocks that carry no claim left out: "
|
||
f"then {left[0]}, now {left[1]} -> {out}")
|
||
if gone:
|
||
print(f" no blocks now (no floor): {', '.join(gone)}")
|
||
return 0
|
||
|
||
|
||
def read_floor(path: Path) -> dict[str, int]:
|
||
"""The floor of cited anchor blocks per source, as cited writes it (an earlier build of the same sources)."""
|
||
raw = read_json(path)
|
||
counts = raw.get("sources") if isinstance(raw, dict) and raw.get("schema") == CITED_SCHEMA else None
|
||
if not (isinstance(counts, dict) and all(isinstance(k, str) and type(v) is int for k, v in counts.items())):
|
||
raise RunError(f"{path} is not a floor of cited blocks ({CITED_SCHEMA})")
|
||
return {k: int(v) for k, v in counts.items()}
|
||
|
||
|
||
def read_history_rate(path: Path) -> float | None:
|
||
"""The floor file's rate for the history sources of a pack (order 28 A4), or None when it gives none."""
|
||
raw = read_json(path)
|
||
rate = raw.get("history_rate") if isinstance(raw, dict) else None
|
||
if rate is None:
|
||
return None
|
||
if isinstance(rate, bool) or not isinstance(rate, (int, float)) or rate < 0:
|
||
raise RunError(f"{path}: history_rate {rate!r} is not a rate of cited blocks per block")
|
||
return float(rate)
|
||
|
||
|
||
def cited_now(work: Path, doc: dict[str, Any], names: list[str]) -> dict[str, int]:
|
||
"""Per source of NAMES, the anchor blocks its records in accepted.json cite now, so a replace or remove since the
|
||
pack was accepted counts (the per-pack count packs.json records is from accept and is never updated)."""
|
||
records = [r for r in read_accepted(work)["fragments"] if r["file"] in names]
|
||
return cited_blocks(records, doc, names)
|
||
|
||
|
||
def history_floor_check(rate: float, cited_by_source: dict[str, int], doc: dict[str, Any], sources: Path,
|
||
all_packs: list[list[str]], first: int, last: int) -> int:
|
||
"""0 when the history sources (release notes, weekly pages) of each of packs FIRST..LAST that has any cite at
|
||
least floor(their blocks x rate / 2) anchor blocks together; otherwise EXIT_THIN (STOP). Order 28 A4: a release
|
||
note has no floor of its own (14 of 20 gave nothing in order 27); a cut answer shows in the pack's total."""
|
||
blocks = blocks_per_source(doc)
|
||
short = 0
|
||
for n in range(first, last + 1):
|
||
names = all_packs[n - 1]
|
||
facts = source_facts(sources, names)
|
||
hist = [s for s in names if facts[s].history]
|
||
if not hist:
|
||
continue
|
||
held = sum(blocks[s] for s in hist)
|
||
cited = sum(cited_by_source.get(s, 0) for s in hist)
|
||
need = int(held * rate / 2)
|
||
print(f"floor: pack {n}'s {len(hist)} history sources cite {cited} anchor blocks of {held}; the floor is {need}")
|
||
short += cited < need
|
||
if short:
|
||
print("STOP: the history sources of a pack cite fewer anchor blocks than their floor; the build stops and "
|
||
"reports, it never continues")
|
||
return EXIT_THIN
|
||
return 0
|
||
|
||
|
||
def approved_source(name: str) -> str | None:
|
||
"""The K1 path (kilde/<name>) a source entry of HOMELESS_APPROVED names, written as kilde/<name>, /kilde/<name> or
|
||
<name>; None when it names no K1 path whole."""
|
||
name = name.strip().lstrip("/")
|
||
name = name if name.startswith("kilde/") else "kilde/" + name
|
||
return name if klink.kilde_path(name) == name else None
|
||
|
||
|
||
def read_approved(work: Path) -> set[str]:
|
||
"""The operator's reported exceptions from HOMELESS_APPROVED in the work folder; no code writes it. Under `pages`,
|
||
the pages approved without a home (order 28 A5), each as its page file; under `sources`, the main-topic sources
|
||
no page cites (a document with no readable content) approved without a main page (the operator 13.11.2026), each
|
||
as its K1 path kilde/<name>. Each entry names the page or source, the pack, the date and the operator's words; a
|
||
file may hold either list or both."""
|
||
path = work / HOMELESS_APPROVED
|
||
if not path.is_file():
|
||
return set()
|
||
raw = read_json(path)
|
||
lists = {k: raw[k] for k in ("pages", "sources") if k in raw} \
|
||
if isinstance(raw, dict) and raw.get("schema") == HOMELESS_SCHEMA else {}
|
||
if not lists or not all(isinstance(v, list) for v in lists.values()):
|
||
raise RunError(f"{path} is not the operator's list of pages without a home and sources without a main page "
|
||
f"({HOMELESS_SCHEMA}: a list `pages`, a list `sources` or both)")
|
||
out: set[str] = set()
|
||
for key, what in (("pages", "page"), ("sources", "source")):
|
||
for e in lists.get(key, []):
|
||
if not (isinstance(e, dict) and type(e.get("pack")) is int
|
||
and all(isinstance(e.get(k), str) and str(e[k]).strip() for k in (what, "date", "operator"))):
|
||
raise RunError(f"{path}: an entry under {key} needs the {what}, the pack, the date and the operator's "
|
||
f"words: {e!r}")
|
||
name = klink.page_rel(str(e["page"])) if what == "page" else approved_source(str(e["source"]))
|
||
if name is None:
|
||
raise RunError(f"{path}: an entry under sources names its source by its K1 path kilde/<name>: {e!r}")
|
||
out.add(name)
|
||
return out
|
||
|
||
|
||
def missing_source(f: dict[str, Any]) -> str | None:
|
||
"""The K1 path a main-page-missing finding names, in `file` or failing that in `message` (as klink.homeless)."""
|
||
return klink.kilde_path(str(f.get("file") or "")) or klink.kilde_path(str(f.get("message") or ""))
|
||
|
||
|
||
def is_approved(f: dict[str, Any], approved: set[str]) -> bool:
|
||
"""True for a finding on the operator's list (read_approved): a no-main-page-link finding on a page under `pages`,
|
||
or a main-page-missing finding on a source under `sources`. The auditor still reports it."""
|
||
if f.get("check") == "no-main-page-link":
|
||
return klink.page_rel(str(f.get("file") or "")) in approved
|
||
if f.get("check") == "main-page-missing":
|
||
return missing_source(f) in approved
|
||
return False
|
||
|
||
|
||
def floor_check(floor: dict[str, int], cited: dict[str, int]) -> int:
|
||
"""0 when every source of the packs checked (CITED, from cited_now) that the floor names cites at least its floor's
|
||
anchor blocks; otherwise each short source is named and EXIT_THIN (STOP)."""
|
||
held = {s: c for s, c in sorted(cited.items()) if s in floor}
|
||
short = [s for s, c in held.items() if c < floor[s]]
|
||
print(f"floor: {len(held) - len(short)} of {len(held)} sources at or over")
|
||
for s in short:
|
||
print(f"floor: {s} cites {held[s]} anchor blocks of {floor[s]} in the floor")
|
||
if short:
|
||
print("STOP: a source cites fewer anchor blocks than its floor; the build stops and reports, it never "
|
||
"continues")
|
||
return EXIT_THIN
|
||
return 0
|
||
|
||
|
||
def nonzero(**counts: int) -> dict[str, int]:
|
||
return {k: v for k, v in counts.items() if v}
|
||
|
||
|
||
def print_dropped(mains: int, homes: int, left: list[str], passed: int = 0, named: list[str] | None = None) -> None:
|
||
"""The records drop_dangling dropped, and the pages it left without a home that `named` does not list already."""
|
||
if mains:
|
||
print(f" dropped {mains} main page record{'s' if mains != 1 else ''} pointing to a page that is gone")
|
||
if homes:
|
||
print(f" dropped {homes} home record{'s' if homes != 1 else ''} no longer valid (its page or its home is gone, "
|
||
f"its home is no longer a main page, its page is now one, or a newer home of the page stands)")
|
||
rest = [c for c in left if c not in (named or [])]
|
||
if rest:
|
||
print(f" without a home now: {', '.join(rest)}")
|
||
if passed:
|
||
print(f" passed {passed} home record{'s' if passed != 1 else ''} to the source that now brings its page")
|
||
|
||
|
||
def cmd_replace(args: argparse.Namespace) -> int:
|
||
work, source = Path(args.work), source_name(args.source)
|
||
check_pending(work)
|
||
doc: dict[str, Any] = read_json(Path(args.blocks))
|
||
if source not in doc["order"]:
|
||
raise RunError(f"{args.blocks} has no blocks for {source}")
|
||
log = pack_log(work)
|
||
size = pack_size(log, args.size)
|
||
place = next(((int(r["pack"]), r["sources"].index(source) + 1) for r in log["packs"]
|
||
if "accepted" in r and source in r.get("sources", [])), None)
|
||
if place is None:
|
||
raise RunError(f"{source} is in no accepted pack; a source joins the build through pack and accept")
|
||
pack, position = place # the place the source had when its pack was accepted, whatever the order is now
|
||
rec = record(log, pack)
|
||
sources = Path(args.sources)
|
||
fragments, declared_main, declared_homes = read_compiled(Path(args.fragments))
|
||
index = read_index(work)
|
||
acc = read_accepted(work)
|
||
valid, rejected = validate(fragments, [source], doc, kilde_of(sources))
|
||
valid, context = with_context(valid, doc, kilde_of(sources))
|
||
own = {str(r["concept_id"]) for r in acc["fragments"] if r["file"] == source} # the pages S wrote to
|
||
mains_before = set(main_ids(acc))
|
||
before = {"fragments": [r for r in acc["fragments"] if r["file"] != source],
|
||
"main_pages": [m for m in acc["main_pages"] if m["source"] != source],
|
||
"homes": [h for h in acc["homes"] if str(h["concept_id"]) not in own]}
|
||
got = admit_pack(valid, index, pack, [source], source_facts(sources, [source]), declared_main, declared_homes,
|
||
before, position)
|
||
rejected += got.rejected
|
||
old = [r for r in acc["fragments"] if r["file"] == source]
|
||
kept: list[dict[str, Any]] = []
|
||
for r in acc["fragments"]:
|
||
if r["file"] != source:
|
||
kept.append(r)
|
||
elif r is old[0]:
|
||
kept.extend(got.records) # the new records take the place of the old ones
|
||
old_mains = len(acc["main_pages"]) - len(before["main_pages"])
|
||
old_homes = list(reversed(acc["homes"])) # newest first, so a page's carried home is its newest valid one
|
||
acc["fragments"] = kept if old else kept + got.records
|
||
acc["main_pages"] = before["main_pages"] + got.main_pages
|
||
acc["homes"] = before["homes"] + got.homes
|
||
# a page S wrote to keeps its home when the new fragments give it none: the home belongs to the page; a home the
|
||
# model chose anew wins, and drop_dangling keeps a carried home only while it is valid
|
||
homed = {str(h["concept_id"]) for h in acc["homes"]}
|
||
carried: list[dict[str, Any]] = []
|
||
for h in old_homes:
|
||
if str(h["concept_id"]) in own and str(h["concept_id"]) not in homed:
|
||
carried.append(h)
|
||
homed.add(str(h["concept_id"]))
|
||
acc["homes"] += carried
|
||
dropped = drop_dangling(acc)
|
||
kept_homes = [str(h["concept_id"]) for h in carried if any(h is x for x in acc["homes"])]
|
||
left = left_without(acc, mains_before, dropped[2]) # print_dropped leaves out those the pack's homes name
|
||
got.home_stats["homeless"] = [c for c in got.home_stats["homeless"] if c not in kept_homes]
|
||
got.home_stats["kept"] = len(kept_homes)
|
||
passed = pass_homes(acc)
|
||
rec.setdefault("replaced", []).append({"source": source, "removed": len(old), "proposed": len(fragments),
|
||
"accepted": len(got.records), "rejected": dict(sorted(rejected.items())),
|
||
**nonzero(main_pages=old_mains, kept_homes=len(kept_homes),
|
||
dropped=dropped[0] + dropped[1], passed=passed)})
|
||
log.setdefault("size", size)
|
||
write_state(work, f"replace-{slug(source)}", {"index.json": index, "accepted.json": acc, "packs.json": log})
|
||
print(f"replace {source}: removed {len(old)}, accepted {len(got.records)} of {len(fragments)}")
|
||
print_context(context)
|
||
print_roles(got.main_stats, got.home_stats)
|
||
print_dropped(dropped[0], dropped[1], left, passed, got.home_stats["homeless"])
|
||
print_rejected(rejected)
|
||
return 0
|
||
|
||
|
||
def cmd_remove(args: argparse.Namespace) -> int:
|
||
work, source = Path(args.work), source_name(args.source)
|
||
check_pending(work)
|
||
acc = read_accepted(work)
|
||
old = [r for r in acc["fragments"] if r["file"] == source]
|
||
if not old:
|
||
raise RunError(f"no accepted fragment comes from {source}")
|
||
mains = sum(m["source"] == source for m in acc["main_pages"])
|
||
mains_before = set(main_ids(acc))
|
||
acc["fragments"] = [r for r in acc["fragments"] if r["file"] != source]
|
||
acc["main_pages"] = [m for m in acc["main_pages"] if m["source"] != source]
|
||
dropped = drop_dangling(acc) # a home goes with its page or its target, never with the source alone
|
||
passed = pass_homes(acc)
|
||
log = pack_log(work)
|
||
record(log, int(old[0]["pack"])).setdefault("removed", []).append(
|
||
{"source": source, "fragments": len(old), **nonzero(main_pages=mains, dropped=dropped[0] + dropped[1],
|
||
passed=passed)})
|
||
write_state(work, f"remove-{slug(source)}", {"accepted.json": acc, "packs.json": log})
|
||
print(f"remove {source}: removed {len(old)} fragments")
|
||
print_dropped(dropped[0], dropped[1], left_without(acc, mains_before, dropped[2]), passed)
|
||
return 0
|
||
|
||
|
||
# --- run-after: the steps after the compiler -------------------------------------------------------------------
|
||
|
||
|
||
def hard_check(work: Path, n: int, paths: list[Path], fragments: list[Any], names: list[str], doc: dict[str, Any],
|
||
kilde: Path, log: dict[str, Any], missing: int = 0) -> int:
|
||
"""0 when every output of pack N is written. A fragment that fails the hard check at writing is dropped and
|
||
counted when the pack is accepted, and the pack does not run again: the check stops the sentence, not the pack
|
||
(order 28, B2: pack 26 ran once more, 4.7 M, for 1 fragment of 574). An output left unwritten sets the outputs
|
||
written aside and logs the attempt: the first time EXIT_RETRY (the pack runs once more), then EXIT_REJECTED_TWICE
|
||
(STOP)."""
|
||
if not missing:
|
||
return 0
|
||
_, rejected = validate(fragments, names, doc, kilde)
|
||
rejected["output-missing"] += missing
|
||
attempts = record(log, n).setdefault("attempts", [])
|
||
attempts.append({"proposed": len(fragments), "rejected": dict(sorted(rejected.items()))})
|
||
moves: dict[Path, Path] = {}
|
||
for path in paths:
|
||
if path.is_file():
|
||
kept = path.with_name(f"{path.stem}.attempt-{len(attempts)}.json")
|
||
if kept.exists():
|
||
raise RunError(f"{kept} exists; refusing to overwrite")
|
||
moves[path] = kept
|
||
write_state(work, f"attempt-{n}", {"packs.json": log}, moves=moves)
|
||
reasons = ", ".join(f"{r} {c}" for r, c in sorted(rejected.items()))
|
||
bad = sum(rejected.values()) - missing
|
||
what = [f"{bad} of {len(fragments)} fragments rejected at writing"] if bad else []
|
||
what += [f"{missing} of {len(paths)} outputs not written"] if missing else []
|
||
print(f"pack {n}: {'; '.join(what)} ({reasons}); set aside as {', '.join(k.name for k in moves.values())}")
|
||
if len(attempts) == 1:
|
||
print(f"RETRY: run the compiler for pack {n} once more with the same parts; it writes "
|
||
f"{', '.join(str(p) for p in paths)}")
|
||
return EXIT_RETRY
|
||
print(f"STOP: pack {n} left an output unwritten twice; the operator decides")
|
||
return EXIT_REJECTED_TWICE
|
||
|
||
|
||
def run_audit(bundle: Path, root: Path) -> dict[str, Any]:
|
||
"""The auditor's JSON for BUNDLE against ROOT, run in this process (no subprocess in the package)."""
|
||
out, err = io.StringIO(), io.StringIO()
|
||
with redirect_stdout(out), redirect_stderr(err):
|
||
code = audit.main([str(bundle), str(root)])
|
||
if code not in (0, 1):
|
||
raise RunError(f"the auditor could not run (exit {code}): {err.getvalue().strip()}")
|
||
try:
|
||
doc = json.loads(out.getvalue())
|
||
except json.JSONDecodeError as exc:
|
||
raise RunError(f"the auditor's output is not JSON: {exc}") from exc
|
||
if not (isinstance(doc, dict) and isinstance(doc.get("findings"), list)):
|
||
raise RunError("the auditor's output has no list of findings")
|
||
return doc
|
||
|
||
|
||
def audit_root(sources: Path, work: Path, log: dict[str, Any], last: int) -> Path:
|
||
"""A subset root (K1 rule 9) of the sources of the accepted packs 1..last; only the newest is kept."""
|
||
names = sorted({str(s) for r in log["packs"] if "accepted" in r and int(r["pack"]) <= last
|
||
for s in r.get("sources", [])})
|
||
base = work / "K-audit"
|
||
base.mkdir(parents=True, exist_ok=True)
|
||
out, paths = base / f"sources-p{last}", base / f"sources-p{last}.paths"
|
||
if out.exists():
|
||
shutil.rmtree(out)
|
||
paths.write_text("".join(f"kilde/{n}\n" for n in names), encoding="utf-8")
|
||
code, printed = quiet(k1.main, ["subset", str(sources), "--paths", str(paths), "--out", str(out)])
|
||
if code != 0:
|
||
raise RunError(f"the subset root of packs 1..{last} could not be made (exit {code}): {printed.strip()}")
|
||
for old in base.iterdir():
|
||
if old.name.startswith("sources-p") and old.name not in (out.name, paths.name):
|
||
if old.is_dir():
|
||
shutil.rmtree(old)
|
||
else:
|
||
old.unlink()
|
||
return out
|
||
|
||
|
||
def save_audit(work: Path, last: int, doc: dict[str, Any]) -> Path:
|
||
base = work / "K-audit"
|
||
k = 1
|
||
while (base / f"p{last}-audit-{k}.json").exists():
|
||
k += 1
|
||
path = base / f"p{last}-audit-{k}.json"
|
||
path.write_text(dump(doc), encoding="utf-8")
|
||
return path
|
||
|
||
|
||
def ask_guard(work: Path, out: Path, last: int, audit_doc: dict[str, Any], log: dict[str, Any]) -> int:
|
||
pages, missing = klink.homeless(audit_doc)
|
||
catalogue = [str(e["id"]) for e in main_catalogue(read_index(work), read_accepted(work), last)]
|
||
parts = klink.guard_parts(out, pages, missing, catalogue, f"pack {last}")
|
||
gdir = work / "K-guard" / f"p{last}"
|
||
paths = [gdir / f"part-{i}.md" for i in range(1, len(parts) + 1)]
|
||
answer = gdir / "answer.json"
|
||
if any(p.exists() for p in [*paths, answer]):
|
||
raise RunError(f"{gdir} holds an earlier home task or answer; refusing to overwrite")
|
||
record(log, last)["guard"] = {"asked": {"pages": pages, "sources": missing, "catalogue": catalogue},
|
||
"parts": len(parts)}
|
||
write_state(work, f"guard-{last}", {"packs.json": log}, dict(zip(paths, parts)))
|
||
print(f"guard: {len(pages)} pages without a home, {len(missing)} sources without a main page")
|
||
for p in paths:
|
||
print(f" part: {p}")
|
||
print(f" answer: {answer}")
|
||
return EXIT_GUARD
|
||
|
||
|
||
def apply_guard(work: Path, out: Path, last: int, guard: dict[str, Any], log: dict[str, Any]) -> None:
|
||
answer = work / "K-guard" / f"p{last}" / "answer.json"
|
||
if not answer.is_file():
|
||
raise RunError(f"the guard's answer {answer} does not exist; run the home task, then run-after again")
|
||
asked = guard["asked"]
|
||
got = klink.read_guard(read_json(answer), asked["pages"], asked["sources"], asked["catalogue"],
|
||
klink.guard_candidates(out, asked["sources"]))
|
||
acc = read_accepted(work)
|
||
for src, pid in got.main_pages:
|
||
name = source_name(src)
|
||
acc["main_pages"] = [m for m in acc["main_pages"] if m["source"] != name] + [
|
||
{"source": name, "concept_id": pid, "pack": last, "by": "guard"}]
|
||
firsts = first_files(acc)
|
||
for pid, hid in got.homes:
|
||
if pid not in firsts:
|
||
raise RunError(f"the guard homed {pid}, which has no accepted fragment")
|
||
acc["homes"].append({"source": firsts[pid], "concept_id": pid, "home": hid, "pack": last, "by": "guard"})
|
||
dropped = drop_dangling(acc) # a main page the guard named in a source's place may take a home's target away
|
||
guard["answered"] = {"homes": len(got.homes), "main_pages": len(got.main_pages), "none": got.none,
|
||
"unanswered": got.unanswered, "rejected": dict(sorted(got.rejected.items())),
|
||
**nonzero(dropped=dropped[0] + dropped[1])}
|
||
write_state(work, f"guard-answer-{last}", {"accepted.json": acc, "packs.json": log})
|
||
print(f"guard answer: homes {len(got.homes)}, main pages {len(got.main_pages)}, none {len(got.none)}, "
|
||
f"unanswered {len(got.unanswered)}, rejected {sum(got.rejected.values())}")
|
||
print_dropped(*dropped)
|
||
|
||
|
||
def stop_line(args: argparse.Namespace, work: Path, log: dict[str, Any], last: int) -> int:
|
||
done = sum(int(r.get("blocks", 0)) for r in log["packs"] if "accepted" in r and int(r["pack"]) <= last)
|
||
print(f"done: {done} of {args.total} blocks")
|
||
code = cost.main(["harvest", "--transcripts", args.transcripts, "--okter", args.okter, "--work", args.work,
|
||
"--out", args.kost])
|
||
print(f"cost harvest: exit {code}")
|
||
if code:
|
||
print("STOP: the cost could not be measured (a run the harvest cannot attribute, or it could not run); "
|
||
"the build stops and reports, it never continues")
|
||
return code # 1 as the stop line gives it; 2 when the harvest could not run
|
||
doc, _, size, _ = interval(args, log)
|
||
_, cap, apart, char_cap = shape(args, log)
|
||
names = next_pack(doc, log, last, size, done, args.total, cap, apart, char_cap)
|
||
following = sum(b["file"] in names for b in doc["blocks"])
|
||
chars = pack_chars(doc, names)
|
||
stop = ["stop", "--lines", STOP_LINES, "--cap", str(args.cap), "--unit", "blocks", "--done", str(done),
|
||
"--total", str(args.total), "--kost", args.kost, "--blocks", str(k_work(work) / "packs.json"),
|
||
"--next", str(following), *([] if chars is None else ["--next-chars", str(chars)])]
|
||
code = cost.main(stop + (["--name", args.name] if args.name else []))
|
||
print(f"cost stop: exit {code}")
|
||
if code == EXIT_STOP:
|
||
print("STOP: the stop line; over the cost stop the build stops and reports, it never continues")
|
||
elif code:
|
||
print("STOP: the stop line could not run; the build stops and reports, it never continues")
|
||
return code
|
||
|
||
|
||
def cmd_run_after(args: argparse.Namespace) -> int:
|
||
from okfwiki import kassemble # kassemble imports this module, so it is imported here, not at the top
|
||
|
||
work, sources, out = Path(args.work), Path(args.sources), Path(args.out)
|
||
check_pending(work)
|
||
if read_k1(sources) is None:
|
||
raise RunError(f"{sources / MANIFEST} does not exist: run-after needs K1 at the sources root, because the "
|
||
f"auditor cannot check the main-page rule without it (K2 point 6)")
|
||
floor = read_floor(Path(args.floor)) if args.floor else None
|
||
rate = read_history_rate(Path(args.floor)) if args.floor else None
|
||
approved = read_approved(work)
|
||
log = pack_log(work)
|
||
doc, all_packs, size, last = interval(args, log)
|
||
kilde = kilde_of(sources)
|
||
for n in range(args.n, last + 1):
|
||
if "accepted" in record(log, n):
|
||
continue
|
||
paths = pack_outputs(work, n, log)
|
||
compiled, missing = read_outputs(paths)
|
||
if len(missing) == len(paths):
|
||
raise RunError(f"{', '.join(str(p) for p in paths)} {'does' if len(paths) == 1 else 'do'} not exist; "
|
||
f"run the compiler for pack {n} first")
|
||
code = hard_check(work, n, paths, compiled[0], all_packs[n - 1], doc, kilde, log, len(missing))
|
||
if code:
|
||
return code
|
||
accept_pack(work, n, doc, all_packs[n - 1], size, sources, compiled, log)
|
||
guard = record(log, last).get("guard")
|
||
if isinstance(guard, dict) and "answered" not in guard:
|
||
apply_guard(work, out, last, guard, log)
|
||
links = Path(args.links) if args.links else None
|
||
r = kassemble.build(work, sources, out, links, replace=out.exists(), at=args.at)
|
||
print(f"assemble: {r.pages} pages from {r.fragments} fragments: new {len(r.new)}, changed {len(r.changed)}, "
|
||
f"unchanged {len(r.unchanged)}, gone {len(r.gone)}; main pages {r.main_pages}, related links {r.related}"
|
||
+ (f"; links placed {r.placed}, stale {r.stale}" if links else ""))
|
||
if r.dropped:
|
||
print(f" dropped links to pages that are gone: {sum(len(t) for t in r.dropped.values())} "
|
||
f"({', '.join(sorted(r.dropped))})")
|
||
if r.unlinked:
|
||
print(f" removed from the links file: {r.unlinked} records into or on pages that are gone")
|
||
root = audit_root(sources, work, log, last)
|
||
audit_doc = run_audit(out, root)
|
||
saved = save_audit(work, last, audit_doc)
|
||
findings = [f for f in audit_doc["findings"] if isinstance(f, dict) and f.get("check") not in RETIRED]
|
||
classes = Counter(str(f.get("check")) for f in findings)
|
||
line = audit_doc.get("link_rule", {}).get("line") if isinstance(audit_doc.get("link_rule"), dict) else None
|
||
print(f"audit: {len(findings)} findings ({', '.join(f'{c} {k}' for c, k in sorted(classes.items())) or 'none'}) "
|
||
f"on packs 1..{last}, {len(list((root / 'kilde').iterdir()))} sources -> {saved}" + (f"; {line}" if line else ""))
|
||
excepted = [f for f in findings if is_approved(f, approved)]
|
||
if excepted: # the auditor still reports them; the operator approved each as a reported exception (order 28 A5)
|
||
homeless = sorted({klink.page_rel(str(f.get("file") or "")) for f in excepted
|
||
if f.get("check") == "no-main-page-link"})
|
||
mainless = sorted({str(missing_source(f)) for f in excepted if f.get("check") == "main-page-missing"})
|
||
if homeless:
|
||
print(f"approved without a home (the operator's list): {', '.join(homeless)}")
|
||
if mainless: # a document with no readable content: never a forced fragment or main page (13.11.2026)
|
||
print(f"approved without a main page (the operator's list): {', '.join(mainless)}")
|
||
findings = [f for f in findings if not is_approved(f, approved)]
|
||
audit_doc = {**audit_doc, "findings": [f for f in audit_doc["findings"]
|
||
if not (isinstance(f, dict) and is_approved(f, approved))]}
|
||
if findings:
|
||
cannot = [f for f in findings if f.get("check") not in klink.GUARDED]
|
||
if cannot or guard is not None:
|
||
why = "a finding the guard cannot mend" if cannot else "findings stand after the guard"
|
||
print(f"STOP: {why}; the operator decides (K2 point 9):")
|
||
for f in findings:
|
||
where = str(f.get("file") or "")
|
||
if f.get("check") == "main-page-missing": # its source, read as the guard reads it
|
||
where = missing_source(f) or where
|
||
print(f" {f.get('check')} {where}")
|
||
return EXIT_BREACH
|
||
return ask_guard(work, out, last, audit_doc, log)
|
||
code = stop_line(args, work, log, last)
|
||
if code or floor is None:
|
||
return code
|
||
cited = cited_now(work, doc, [s for n in range(args.n, last + 1) for s in all_packs[n - 1]])
|
||
thin = floor_check(floor, cited)
|
||
if rate is not None:
|
||
thin = history_floor_check(rate, cited, doc, sources, all_packs, args.n, last) or thin
|
||
return thin
|
||
|
||
|
||
def build_parser() -> argparse.ArgumentParser:
|
||
p = argparse.ArgumentParser(prog="python3 -m okfwiki.kpack", description=__doc__,
|
||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||
sub = p.add_subparsers(dest="command", required=True)
|
||
s = sub.add_parser("skeleton")
|
||
s.add_argument("--sources", required=True)
|
||
s.add_argument("--work", required=True)
|
||
s.add_argument("--extend", action="store_true")
|
||
for name in ("pack", "accept", "replace", "run-before", "run-after"):
|
||
c = sub.add_parser(name)
|
||
if name == "replace":
|
||
c.add_argument("source")
|
||
c.add_argument("--fragments", required=True)
|
||
else:
|
||
c.add_argument("n", type=int)
|
||
if name.startswith("run-"):
|
||
c.add_argument("--to", type=int)
|
||
c.add_argument("--blocks", required=True)
|
||
c.add_argument("--sources", required=True)
|
||
c.add_argument("--work", required=True)
|
||
c.add_argument("--size", type=int)
|
||
c.add_argument("--max-blocks", type=int)
|
||
c.add_argument("--max-chars", type=int)
|
||
c.add_argument("--groups-apart", action="store_true")
|
||
if name == "run-after":
|
||
c.add_argument("--out", required=True)
|
||
c.add_argument("--links")
|
||
c.add_argument("--at")
|
||
for flag in ("--transcripts", "--okter", "--kost"):
|
||
c.add_argument(flag, required=True)
|
||
c.add_argument("--cap", type=int, required=True)
|
||
c.add_argument("--total", type=int, required=True)
|
||
c.add_argument("--name")
|
||
c.add_argument("--floor")
|
||
r = sub.add_parser("remove")
|
||
r.add_argument("source")
|
||
r.add_argument("--work", required=True)
|
||
c = sub.add_parser("cited")
|
||
for flag in ("--work", "--blocks", "--out"):
|
||
c.add_argument(flag, required=True)
|
||
f = sub.add_parser("floor")
|
||
for flag in ("--cited", "--blocks-then", "--sources-then", "--blocks", "--sources", "--out"):
|
||
f.add_argument(flag, required=True)
|
||
f.add_argument("--history-rate", type=float)
|
||
return p
|
||
|
||
|
||
def cmd_pack(args: argparse.Namespace) -> int:
|
||
args.to = None
|
||
return cmd_run_before(args)
|
||
|
||
|
||
COMMANDS = {"skeleton": cmd_skeleton, "pack": cmd_pack, "accept": cmd_accept, "replace": cmd_replace,
|
||
"remove": cmd_remove, "run-before": cmd_run_before, "run-after": cmd_run_after, "cited": cmd_cited,
|
||
"floor": cmd_floor}
|
||
|
||
|
||
def main(argv: list[str] | None = None) -> int:
|
||
args = build_parser().parse_args(argv)
|
||
try:
|
||
return COMMANDS[args.command](args)
|
||
except (RunError, OSError, UnicodeDecodeError) as exc:
|
||
print(f"kpack cannot run: {exc}", file=sys.stderr)
|
||
return 2
|
||
except Exception: # an input this code did not foresee: still "could not run", with the traceback
|
||
traceback.print_exc()
|
||
return 2
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|