One rule explains every remaining `pdf` miss on the twelve-position reference: where a document DECLARES headings, Arm D's RECOVERED headings are the whole of the excess, and every declared one is a unit the reference wants. `--outline-gate` admits recovery only where the document declares none of its own, plus any one recovered heading covering OUTLINE_SHARE (0.20) of the text. It is `fold_units` clause 2's own principle moved from voting to admission, and it filters at ADMISSION so the text a removed mark opened is carried by the mark above it -- the post-filter form scores identically on all twelve positions and loses that text, which is why only one of them shipped. `--outline-gate` and `--drop-wrapped-outline` become the package default, one decision because neither carries the reference alone: `pdf` 2 of 8 -> 5 of 8 alone, 7 of 8 together; the sheet 5 of 12 -> 10 of 12; `docx` unchanged at 3 of 3. Each keeps an explicit opt-out. The bar the move had to clear was not the reference: hit@8 on a K2 bundle built with it holds 5 of 6 at ranks 1,1,1,1,1,-, no row losing rank 1. `--sheet-section-rows --keep-table-heading` reaches 11 of 12 and does NOT ship, because on a bundle built with it row 1 falls rank 1 -> 2. Cost to a consumer is a re-run: 492 concepts / 944 files -> 425 / 810. DOCUMENT_PRIOR_EXPONENT makes the document prior sublinear (total/n**0.5). A sum measures size and a density is diluted by every unit carrying none of the question, so a document split 1 -> 12 lost its prior by 12. Swept over five values on 18 rows it is at least as good as the delivered density everywhere and strictly better on three. Stated plainly: end to end it moved NOT ONE hit@8 row on any of four bundles, so it did not solve the knot it was adopted for -- what did is that the `pdf` gain never needed `--sheet-section-rows`. `--first-span-from-zero` is off and repairs a measured loss found while chasing one position's 940 characters: 32 of the 32 documents that get a plan leave the text above their first concept in no segment -- 159 704 characters, 9.18 % of the corpus, 45 841 from one document. It changes nothing on the reference. Off because it moves the first span of essentially every bundle with no hit@8 number behind it yet. vegnormal-okf FUNN 2: SPEC section 8's own star row parsed as prose, so every concept behind one was unreachable to the section 9.2 walk. `IndexPolicy.also_reads` carries it for the SEGMENTED profiles, read-only, after the emitted pattern misses -- the asymmetry `sources` already has. DEFAULT and STRICT_V1 untouched (O2). vegnormal-okf FUNN 1: Door C's own outcome was refused at exit 1, `bundle_id_missing`. `import_bundle` now takes `root_frontmatter_values`, keyword-only, rendered before any disk mutation, written only when the index is created -- Door B's mechanism and ordering. Report: docs/2026-09-09-k3-runde6-outline-gaten-og-prioren.md. Suite 1478 passed (1449 before), ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1401 lines
65 KiB
Python
1401 lines
65 KiB
Python
"""The bundle contract as configuration (Phase 3).
|
|
|
|
What a valid bundle looks like — which concept types exist, which frontmatter
|
|
keys are emitted and in which order, which filename namespaces the doors own,
|
|
and what an index line looks like — is a profile, not a set of constants
|
|
scattered across the doors. `DEFAULT` is exactly the ingest-spec v1 + Phase 2
|
|
contract, so nothing observable changes for a caller that never mentions a
|
|
profile; the golden fixtures are the byte-level proof.
|
|
|
|
Two things deliberately do NOT live here. Security is the guard's, always: no
|
|
disposition, origin or channel vocabulary belongs on a profile. And the
|
|
reserved `verdict` layer is a spec invariant rather than profile config — it is
|
|
refused at construction, so a profile admitting it cannot be built, let alone
|
|
passed to a door. The `timestamp`/`generated` pair is refused the same way, and
|
|
for the same reason: both are things a profile must not be able to express.
|
|
|
|
Profiles are constructed in code. Config-file loading and inheritance chains
|
|
are extension points, not v1 (settled with the operator at phase start).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from collections.abc import Collection, Iterable, Mapping, Sequence
|
|
from dataclasses import dataclass, field, replace
|
|
|
|
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
|
|
# the only path into it. Compared case-insensitively, as both doors already do.
|
|
RESERVED_OKF_TYPE = "verdict"
|
|
|
|
# The one key PAIR no profile may name (OKF §13.1): `timestamp` is readable as
|
|
# a legacy stand-in for `generated.at` only while `generated` is ABSENT, so a
|
|
# schema able to name both can describe a document that has neither. Stated as
|
|
# key names rather than as a judgement on values because every `generated` a
|
|
# schema can express today is a scalar (`_is_legal_value`) and therefore
|
|
# malformed as a v0.2 mapping — naming both IS the hazard here. When a value
|
|
# model can express a well-formed `generated`, this narrows with it.
|
|
# "Naming" spans EVERY field that puts a key in the schema's namespace — `order`,
|
|
# `required`, `allowed` and `nullable` alike. A field left out of that union is a
|
|
# hole in the gate, not a narrower gate: the namespace is open by default, so a
|
|
# key named only by `nullable` is admitted just as surely as an emitted one.
|
|
_TIMESTAMP_FALLBACK_PAIR = frozenset({"timestamp", "generated"})
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class TypeRejection:
|
|
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
|
|
|
|
The policy does not raise: Door A refuses with `ManifestError` and Door B
|
|
with `MaterializationError`, so the refusal has to be reported rather than
|
|
thrown. `reason` completes the sentence "<label>okf_type ..." and `code` is
|
|
the stable `IngestError.code` the caller asserts on.
|
|
"""
|
|
|
|
reason: str
|
|
code: str
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class TypePolicy:
|
|
"""Which `okf_type` values a bundle admits.
|
|
|
|
`allowed` is `None` for an open set — Phase 1/2 accept any type the
|
|
operator names — or a closed enum. Either way the reserved layer is
|
|
excluded, and a closed set that names it fails at construction.
|
|
"""
|
|
|
|
allowed: frozenset[str] | None = None
|
|
|
|
def __post_init__(self) -> None:
|
|
if self.allowed is None:
|
|
return
|
|
named = sorted(value for value in self.allowed if value.lower() == RESERVED_OKF_TYPE)
|
|
if named:
|
|
raise ValueError(
|
|
f"a profile must not admit the reserved {RESERVED_OKF_TYPE!r} layer "
|
|
f"(got {', '.join(repr(value) for value in named)}) — the promotion "
|
|
"gate is the only path into it (ingest-spec §3)"
|
|
)
|
|
|
|
def rejection(self, okf_type: str) -> TypeRejection | None:
|
|
"""The refusal for `okf_type`, or `None` when the profile admits it."""
|
|
if okf_type.lower() == RESERVED_OKF_TYPE:
|
|
return TypeRejection(
|
|
reason=f"must not be {RESERVED_OKF_TYPE!r} (reserved layer)",
|
|
code="okf_type_reserved",
|
|
)
|
|
if self.allowed is not None and okf_type not in self.allowed:
|
|
return TypeRejection(
|
|
reason=f"must be one of {', '.join(sorted(self.allowed))}, got {okf_type!r}",
|
|
code="okf_type_not_allowed",
|
|
)
|
|
return None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SchemaViolation:
|
|
"""One way a document's frontmatter departs from the schema.
|
|
|
|
Reported rather than raised, like `TypeRejection`: a document can depart in
|
|
several ways at once, and the caller decides whether that is a refusal, a
|
|
report line, or a migration to-do. `code` is the stable machine-readable
|
|
part; `reason` completes the sentence "<key> ...".
|
|
"""
|
|
|
|
key: str
|
|
reason: str
|
|
code: str
|
|
|
|
|
|
def _is_legal_value(value: object) -> bool:
|
|
"""A string scalar, or a non-empty list of strings — nothing else.
|
|
|
|
The list shape is real, not hypothetical: the proving consumer's `Release`
|
|
pages carry block lists. A schema that assumed scalars-only would reject
|
|
pages they ship today.
|
|
"""
|
|
if isinstance(value, str):
|
|
return True
|
|
if isinstance(value, list):
|
|
return bool(value) and all(isinstance(item, str) for item in value)
|
|
return False
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class FrontmatterSchema:
|
|
"""The frontmatter key namespace, what it admits, and how it is emitted.
|
|
|
|
`order` is the canonical emission order and `collapsed_keys` names the keys
|
|
whose whitespace runs collapse to single spaces on the way out. The DEFAULT
|
|
order spans both doors' key sets: Door A emits seven of these keys and Door
|
|
B six, and each door's subset comes out in exactly the order it wrote by
|
|
hand in Phases 1 and 2.
|
|
|
|
The judging fields — `required`, `allowed`, `nullable`, `key_pattern` — are
|
|
read by `violations`. `allowed` is `None` for an open namespace, which is
|
|
what DEFAULT keeps: Door A emits seven keys, Door B six, and Door C writes
|
|
an imported concept verbatim with whatever the sender wrote, so a closed
|
|
namespace or a required set on DEFAULT would declare invalid the very
|
|
bundles this library produces.
|
|
|
|
`required_by_type` is the one judging field that keys off a frontmatter
|
|
*value* rather than a key: `{<type value>: {<keys that type must carry>}}`.
|
|
OKF v0.2 §10.2 introduces exactly one such rule — `runtime` is REQUIRED for
|
|
`Attested Computation` and for no other type — and it cannot be expressed
|
|
through `required`, which would demand the key of every document. A type
|
|
the mapping does not name carries no extra requirement, which is what keeps
|
|
the field inside §14: a consumer must not reject on an unknown `type`, so a
|
|
conditional keyed on a type we do not know stays silent rather than guesses.
|
|
"""
|
|
|
|
order: tuple[str, ...]
|
|
collapsed_keys: frozenset[str] = field(default_factory=frozenset)
|
|
required: frozenset[str] = field(default_factory=frozenset)
|
|
allowed: frozenset[str] | None = None
|
|
nullable: frozenset[str] = field(default_factory=frozenset)
|
|
key_pattern: re.Pattern[str] | None = None
|
|
required_by_type: Mapping[str, frozenset[str]] = field(default_factory=dict)
|
|
|
|
def __post_init__(self) -> None:
|
|
conditional: frozenset[str] = frozenset().union(*self.required_by_type.values())
|
|
named = set(self.order) | set(self.required) | set(self.nullable) | conditional
|
|
if self.allowed is not None:
|
|
named |= set(self.allowed)
|
|
if _TIMESTAMP_FALLBACK_PAIR <= named:
|
|
raise ValueError(
|
|
"a profile must not name both 'timestamp' and 'generated' (OKF "
|
|
"§13.1 grants the timestamp fallback only while `generated` is "
|
|
"ABSENT, so a schema naming both can describe a document with "
|
|
"neither a valid `generated.at` nor an eligible fallback)"
|
|
)
|
|
if self.allowed is None:
|
|
return
|
|
for label, keys in (
|
|
("required", self.required),
|
|
("nullable", self.nullable),
|
|
("type-conditional required", conditional),
|
|
):
|
|
stray = sorted(keys - self.allowed)
|
|
if stray:
|
|
raise ValueError(
|
|
f"{label} keys must be inside the allowlist "
|
|
f"(got {', '.join(repr(key) for key in stray)}) — a schema that "
|
|
"demands a key it also forbids can never be satisfied"
|
|
)
|
|
|
|
def violations(self, values: Mapping[str, object]) -> tuple[SchemaViolation, ...]:
|
|
"""Every way `values` departs from this schema, deterministically ordered.
|
|
|
|
Sorted by `(key, code)` rather than reported in mapping order: two runs
|
|
over the same document must produce the same report, and mapping order
|
|
is an accident of how the document was parsed.
|
|
|
|
Takes an already-parsed mapping. Parsing strict frontmatter is a
|
|
separate concern — this library's line-oriented parser cannot represent
|
|
the block lists and nulls a strict schema admits, which is why Door C
|
|
writes imported concepts verbatim rather than round-tripping them.
|
|
"""
|
|
found: list[SchemaViolation] = []
|
|
|
|
for key in self.required - set(values):
|
|
found.append(SchemaViolation(key, "is required and absent", "frontmatter_key_missing"))
|
|
|
|
# The type-conditional rules (§10.2 today). Keyed off the VALUE, so it
|
|
# fires only on a type this schema names: an absent, non-scalar or
|
|
# unknown `type` carries no extra requirement. A separate code from the
|
|
# unconditional case because the two are different claims — one says the
|
|
# document is malformed, the other that it is malformed *for what it
|
|
# says it is* — and a caller may well treat them differently.
|
|
declared = values.get("type")
|
|
if isinstance(declared, str):
|
|
for key in self.required_by_type.get(declared, frozenset()) - set(values):
|
|
found.append(
|
|
SchemaViolation(
|
|
key,
|
|
f"is required for type {declared!r} and absent",
|
|
"frontmatter_key_missing_for_type",
|
|
)
|
|
)
|
|
|
|
for key, value in values.items():
|
|
if self.key_pattern is not None and not self.key_pattern.fullmatch(key):
|
|
found.append(
|
|
SchemaViolation(
|
|
key,
|
|
f"is not a legal key name (must match {self.key_pattern.pattern})",
|
|
"frontmatter_key_malformed",
|
|
)
|
|
)
|
|
continue
|
|
if self.allowed is not None and key not in self.allowed:
|
|
found.append(
|
|
SchemaViolation(
|
|
key, "is not on the key allowlist", "frontmatter_key_not_allowed"
|
|
)
|
|
)
|
|
continue
|
|
if value is None:
|
|
if key not in self.nullable:
|
|
found.append(SchemaViolation(key, "must not be null", "frontmatter_value_null"))
|
|
continue
|
|
if not _is_legal_value(value):
|
|
found.append(
|
|
SchemaViolation(
|
|
key,
|
|
"must be a string or a non-empty list of strings",
|
|
"frontmatter_value_shape",
|
|
)
|
|
)
|
|
|
|
return tuple(sorted(found, key=lambda violation: (violation.key, violation.code)))
|
|
|
|
def emit(self, values: Mapping[str, str]) -> str:
|
|
"""Render `values` as line-oriented `key: value`, one line per key.
|
|
|
|
Commons decision D1: the keys named in `order` come first, in that
|
|
order, followed by any remaining keys SORTED. Ordering the tail rather
|
|
than trusting insertion order is what makes a regeneration over the
|
|
same data byte-identical.
|
|
|
|
Returns the lines only — the caller owns the `---` fences.
|
|
"""
|
|
named = [key for key in self.order if key in values]
|
|
tail = sorted(key for key in values if key not in self.order)
|
|
return "\n".join(f"{key}: {self._render(key, values[key])}" for key in [*named, *tail])
|
|
|
|
def _render(self, key: str, value: str) -> str:
|
|
# §5 mandates whitespace-run collapse for `source_query` only
|
|
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
|
|
# must render on one line. Every other value is validated single-line
|
|
# at load and emitted verbatim — validation, not repair — so
|
|
# operator-supplied bytes (e.g. a title's internal double space)
|
|
# survive.
|
|
return " ".join(value.split()) if key in self.collapsed_keys else value
|
|
|
|
|
|
# The v0.1 ingest stamp. A literal rather than a configurable value: it is what
|
|
# every bundle this library has already written carries, and recognising it is
|
|
# what keeps those bundles re-runnable under a later profile.
|
|
_V0_1_STAMP = "true"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class OwnershipPolicy:
|
|
"""The `generated` value this profile writes, and the values it owns back.
|
|
|
|
Ownership is the §3 collision gate's question — may this run replace the
|
|
file already sitting at a generated name? — and the answer is a profile's,
|
|
because the stamp differs per profile. The literal `true` is the older
|
|
form; the O2 form is `generated: { by: <actor>, at: <ingested_at> }` (§5),
|
|
where the actor takes §7's `process:<id>` form.
|
|
|
|
Which form a profile writes is NOT read off an upstream version. `DEFAULT`
|
|
states commons' ingest-spec layer and writes the O2 form because commons
|
|
ratified it (V1, `54e0ec7`), while remaining v0.1 on every axis upstream
|
|
owns; `STRICT_V1` names no `generated` at all. The stamp tracks whichever
|
|
contract the profile states, and those contracts move independently.
|
|
|
|
`actor` is `None` for the literal stamp. Where it is set it carries no version,
|
|
deliberately: the value sits inside a byte-compared golden, so a producer
|
|
version there would fire golden regression on every release without any
|
|
contract having changed, and would make a shared cross-implementation
|
|
fixture impossible by construction (plan V1(d), operator 2026-07-27).
|
|
|
|
Recognition is ONE-WAY, and both directions are decisions rather than
|
|
accidents. A profile with an actor owns the literal stamp as well, so a
|
|
bundle written before V1 re-runs IN PLACE — the black-box promise is that a
|
|
spec release costs a consumer a re-run and nothing more. The reverse is
|
|
refused: a profile without an actor fails the run rather than replacing a
|
|
file whose shape it does not read (V-A3).
|
|
|
|
"Owns the O2 form" is never "owns any mapping". The prefix binds the
|
|
profile's OWN actor, so a mapping naming a different one — another
|
|
implementation's, or §7's `human:` actor on curated content — stays
|
|
unowned. That is what keeps the key's mere presence from proving authorship,
|
|
which upstream v0.2 makes load-bearing by writing `generated` on
|
|
hand-authored files too.
|
|
|
|
The actor test is a PREFIX rather than an equality, because the value carries
|
|
`ingested_at` and therefore differs on every run by design. It works because
|
|
`parse_frontmatter` returns the whole flow mapping as one opaque string
|
|
(V-A2) — no structure this library cannot yet read is parsed here.
|
|
"""
|
|
|
|
actor: str | None = None
|
|
|
|
def stamp(self, ingested_at: str) -> str:
|
|
"""The `generated` value a run at `ingested_at` writes."""
|
|
if self.actor is None:
|
|
return _V0_1_STAMP
|
|
return f"{{ by: {self.actor}, at: {ingested_at} }}"
|
|
|
|
def owns(self, value: str) -> bool:
|
|
"""Whether a `generated` value read back marks this library's output."""
|
|
if value == _V0_1_STAMP:
|
|
return True
|
|
if self.actor is None:
|
|
return False
|
|
return value.startswith(f"{{ by: {self.actor},")
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class PathPolicy:
|
|
"""The filename namespaces the three doors write into.
|
|
|
|
Each prefix keeps its door's generated names disjoint from `index.md`, from
|
|
the other doors, and from `promoted-verdict-*`, for every id the grammar
|
|
admits. The id grammar itself stays in `materialize`: it is a pattern the
|
|
slugger derives a separator class from, not a name to configure.
|
|
"""
|
|
|
|
concept_suffix: str
|
|
ingest_prefix: str
|
|
inbox_prefix: str
|
|
import_prefix: str
|
|
|
|
|
|
def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
|
|
"""The leading `---` block as ordered keys, and the body lines after it.
|
|
|
|
Line-oriented, the same shape `materialize.parse_frontmatter` reads —
|
|
duplicated rather than imported because `materialize` imports this module,
|
|
and because that one takes a path where an index reader has only text.
|
|
"""
|
|
lines = text.splitlines()
|
|
if not lines or lines[0].strip() != "---":
|
|
return {}, lines
|
|
head: dict[str, str] = {}
|
|
for offset, line in enumerate(lines[1:], start=1):
|
|
if line.strip() == "---":
|
|
return head, lines[offset + 1 :]
|
|
# An INDENTED key belongs to the block above it, not to the document.
|
|
# Without this, `key.strip()` would flatten it into the same namespace
|
|
# as the top-level keys and, arriving later, SUBSTITUTE for one of them
|
|
# -- a `sources:` entry's own `title:` silently becoming the document's,
|
|
# carrying `number` and `parent` with it. Skipping is deliberately not
|
|
# parsing: the nested value is not read, only refused. The structured
|
|
# reader is D1b.
|
|
if line[:1] in (" ", "\t"):
|
|
continue
|
|
key, sep, value = line.partition(":")
|
|
if sep:
|
|
head[key.strip()] = value.strip()
|
|
return head, []
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class FacetPolicy:
|
|
"""The metadata an index ENTRY carries beside its label and target.
|
|
|
|
A flat index is a table of contents. A faceted one is a table a consumer
|
|
can reason over without opening a single document - which is the whole
|
|
difference the 2026-08-26 bake-off measured: every arm retrieved 40/40, and
|
|
the only axis that separated them was trap exposure, where the flat index
|
|
lost precisely because it carried title/date/status/supersedes 0 times
|
|
while its own documents carried them 55/55/55/5.
|
|
|
|
`keys` is the ordered, closed set. Ordering by the POLICY rather than by
|
|
the caller's mapping is what keeps two callers passing the same facts from
|
|
emitting different bytes, exactly as the root frontmatter does; closing the
|
|
set is what keeps a value out of a file no reader of this contract looks
|
|
at.
|
|
|
|
The grammar is deliberately thin - `separator` once, then `key: value`
|
|
joined by `joiner` - because an index line is read by regex on both sides
|
|
of this library. A value carrying either delimiter is REFUSED rather than
|
|
escaped or repaired: escaping would make the line unreadable to a consumer
|
|
that splits naively, and this library validates rather than repairs
|
|
everywhere else.
|
|
"""
|
|
|
|
keys: tuple[str, ...]
|
|
separator: str = " \u2014 "
|
|
joiner: str = "; "
|
|
|
|
def __post_init__(self) -> None:
|
|
if not self.keys:
|
|
raise ValueError(
|
|
"a facet policy must name at least one key - a policy naming "
|
|
"none would render a separator with nothing after it"
|
|
)
|
|
|
|
def render(self, values: Mapping[str, str]) -> str:
|
|
"""The facet tail for `values`, or "" when none of them are present."""
|
|
unknown = sorted(set(values) - set(self.keys))
|
|
if unknown:
|
|
raise ValueError(
|
|
f"facet key(s) {', '.join(repr(key) for key in unknown)} are not named "
|
|
f"by this index policy, which pins {self.keys} - rendering an unnamed "
|
|
"key would put a value in a file no reader of this contract looks at"
|
|
)
|
|
present = [(key, values[key]) for key in self.keys if values.get(key)]
|
|
for key, value in present:
|
|
if "\n" in value or "\r" in value:
|
|
raise ValueError(f"facet {key!r} must be single-line, got {value!r}")
|
|
for label, delimiter in (("separator", self.separator), ("joiner", self.joiner)):
|
|
if delimiter in value:
|
|
raise ValueError(
|
|
f"facet {key!r} contains this policy's {label} {delimiter!r} "
|
|
f"({value!r}) - refusing to escape or repair it, which would "
|
|
"make the line parse one way here and another way downstream"
|
|
)
|
|
if not present:
|
|
return ""
|
|
return self.separator + self.joiner.join(f"{key}: {value}" for key, value in present)
|
|
|
|
def parse(self, tail: str) -> dict[str, str]:
|
|
"""The facet tail read back. The inverse of `render` by construction."""
|
|
parsed: dict[str, str] = {}
|
|
for chunk in tail.split(self.joiner):
|
|
key, sep, value = chunk.partition(":")
|
|
if sep:
|
|
parsed[key.strip()] = value.strip()
|
|
return parsed
|
|
|
|
|
|
#: The closed set `IndexPolicy.sort_order` draws from. A CALLER-SUPPLIED
|
|
#: CALLABLE is deliberately not an option here: it cannot be serialised into
|
|
#: the bundle, cannot be reproduced from it, and cannot be audited by anyone
|
|
#: reading it back — which is the whole of what a deterministic bundle claims.
|
|
SORT_ASCENDING = "ascending"
|
|
SORT_DESCENDING = "descending"
|
|
SORT_ORDERS = (SORT_ASCENDING, SORT_DESCENDING)
|
|
|
|
#: The closed set `IndexPolicy.sort_missing` draws from: where the entries that
|
|
#: do not carry the key at all are put. Closed for the same reason.
|
|
SORT_MISSING_FIRST = "first"
|
|
SORT_MISSING_LAST = "last"
|
|
SORT_MISSING = (SORT_MISSING_FIRST, SORT_MISSING_LAST)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class IndexEntry:
|
|
"""One managed index line, read back into its parts.
|
|
|
|
`concept_path` is the bundle-relative path of the concept the entry is
|
|
ABOUT, and it is the ordering tie-break. It is not part of the rendered
|
|
line and is therefore only ever populated on the write path — `parse_entry`
|
|
leaves it `None`, and the ordering then falls back to the link target.
|
|
That costs nothing today because no caller sorts entries it read back off
|
|
disk; every ordering happens where the entry is being built.
|
|
|
|
It exists because the two doors disagree about what the target IS. Door B's
|
|
target is the concept's own name, but Door C reduces a sender's concept
|
|
path to a generated filename, and the two do not order alike:
|
|
`notes-beta.md` precedes `notes/alpha.md` by concept path and follows it by
|
|
generated name. Ordering on the target would silently re-order every
|
|
existing Door C bundle.
|
|
"""
|
|
|
|
label: str
|
|
target: str
|
|
description: str | None = None
|
|
facets: Mapping[str, str] = field(default_factory=dict)
|
|
concept_path: str | None = None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class IndexViolation:
|
|
"""One way an index file departs from the policy.
|
|
|
|
Reported rather than raised, like `SchemaViolation`. `subject` is whatever
|
|
the finding is about — a line, a link target, or a frontmatter key — and
|
|
`code` is the stable machine-readable part.
|
|
"""
|
|
|
|
subject: str
|
|
reason: str
|
|
code: str
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class IndexPolicy:
|
|
"""The index file, the shape of its entries, and where indexes must exist.
|
|
|
|
`link_template` renders a managed line and `link_pattern` recognises one.
|
|
Both are carried because §6 does both — append on write, rewrite on
|
|
maintenance — and a round-trip test is what keeps the pair honest. The
|
|
pattern is anchored to the whole line by construction: removal keys on this
|
|
exact shape, never a bare substring, so curated prose that mentions a
|
|
target inline survives verbatim.
|
|
|
|
The judging fields are read by `violations` and `required_indexes`, and
|
|
every one of them is off under DEFAULT. They exist because two consumers
|
|
disagree about this file in ways no single shape can hold:
|
|
|
|
- `per_directory` — an index at every level, or only at the bundle root.
|
|
Upstream OKF binds `index.md` to the root ALONE, so a default demanding
|
|
one per level would declare upstream-conforming bundles invalid. Policy,
|
|
never an OKF rule.
|
|
- `heading_required` / `allows_prose` — the wiki's index is a generated
|
|
heading plus entries and nothing else; the catalog's *requires*
|
|
progressive-disclosure prose. Opposite requirements, both expressible.
|
|
- `entries_match_directory` — whether the index must match the directory
|
|
exactly, in both directions. The wiki enforces this at ERROR
|
|
(`BUNDLE_INDEX_COMPLETE`); the catalog holds that an index is an
|
|
AUTHORED count of a directory's children and that a validator
|
|
enumerating the directory has implemented the wrong contract. Neither
|
|
posture is baked in, and this library never enumerates anything: the
|
|
caller supplies the listing, and a policy that does not judge it refuses
|
|
to be handed one at all — so code written to the wrong contract fails at
|
|
the call rather than passing every test one would think to write.
|
|
- `root_frontmatter` — the ordered keys the ROOT index may carry, where
|
|
nested indexes carry none. Confirmed independently in both consumers, so
|
|
the asymmetry is shape rather than one repo's preference. Naming a key
|
|
here PERMITS it and fixes its position; it does not demand it.
|
|
- `root_frontmatter_required` — the subset that must actually be present.
|
|
Separate from the tuple above because the two are different claims, and
|
|
collapsing them was P1-F1: `OKF_V0_2` names `okf_version` to fix its
|
|
position, but upstream §8/§12 grant it as a MAY, and judging the pair as
|
|
one field failed 14 of 17 real bundles — upstream's own four included —
|
|
each with exactly this one violation, while our emitter treated the same
|
|
key as optional. A profile whose consumer really does demand the keys
|
|
says so here, which is what `STRICT_V1` does.
|
|
"""
|
|
|
|
name: str
|
|
link_template: str
|
|
link_pattern: re.Pattern[str]
|
|
per_directory: bool = False
|
|
heading_required: bool = False
|
|
allows_prose: bool = True
|
|
entries_match_directory: bool = False
|
|
root_frontmatter: tuple[str, ...] = ()
|
|
root_frontmatter_required: frozenset[str] = field(default_factory=frozenset)
|
|
facets: FacetPolicy | None = None
|
|
#: Row forms this policy READS but never writes. Consulted by `parse_entry`
|
|
#: only, after `entry_pattern` misses, so no emitted byte can move: every
|
|
#: line this library writes still comes from `link_template`.
|
|
#:
|
|
#: WHY IT EXISTS (vegnormal-okf, FUNN 1/2, 2026-09-08). OKF SPEC section 8
|
|
#: shows `* [Title](file.md) - description` in its own example and Google's
|
|
#: generator writes it, while this library's segmented profiles write
|
|
#: `- [Title](file.md)`. Measured, the star row parsed as `None` -- curated
|
|
#: prose -- so the section 9.2 index walk could not reach a single concept
|
|
#: behind one. A bundle we cannot walk is the silent loss the "arbitrary
|
|
#: bundle" direction forbids.
|
|
#:
|
|
#: Reading a form is NOT a licence to emit it. That asymmetry is this
|
|
#: repository's existing posture, not a new one: `sources` is read in both
|
|
#: YAML forms and written in one, for the same reason -- the emitted shape
|
|
#: is what our own parser must round-trip.
|
|
#:
|
|
#: Every member must be anchored at both ends, for the reason
|
|
#: `_faceted_pattern` states: an unanchored alternative would match a
|
|
#: target mentioned inside curated prose.
|
|
also_reads: tuple[re.Pattern[str], ...] = ()
|
|
sort_key: str | None = None
|
|
sort_order: str = SORT_ASCENDING
|
|
sort_missing: str = SORT_MISSING_LAST
|
|
_faceted_pattern: re.Pattern[str] | None = field(
|
|
init=False, repr=False, compare=False, default=None
|
|
)
|
|
|
|
def __post_init__(self) -> None:
|
|
if self.facets is not None:
|
|
# The faceted pattern is BUILT from the base one, so an unanchored
|
|
# base would silently produce an unanchored faceted pattern - and
|
|
# index maintenance keys on this pattern to decide which lines it
|
|
# may rewrite. A substring match there edits curated prose.
|
|
if not self.link_pattern.pattern.endswith("$"):
|
|
raise ValueError(
|
|
"a link pattern carrying facets must be anchored to the end "
|
|
"of the line ('$'), because the faceted pattern is derived "
|
|
"from it and an unanchored match would rewrite curated prose"
|
|
)
|
|
object.__setattr__(
|
|
self,
|
|
"_faceted_pattern",
|
|
re.compile(
|
|
self.link_pattern.pattern[:-1]
|
|
+ f"(?:{re.escape(self.facets.separator)}(?P<facets>.+))?$"
|
|
),
|
|
)
|
|
for extra in self.also_reads:
|
|
if not extra.pattern.startswith("^") or not extra.pattern.endswith("$"):
|
|
raise ValueError(
|
|
f"an also-read pattern must be anchored at both ends, got "
|
|
f"{extra.pattern!r} — an unanchored alternative would read a "
|
|
"target mentioned inside curated prose as an entry"
|
|
)
|
|
stray = sorted(self.root_frontmatter_required - set(self.root_frontmatter))
|
|
if stray:
|
|
raise ValueError(
|
|
f"required root keys must be named in the ordered set "
|
|
f"(got {', '.join(repr(key) for key in stray)}) — a key demanded "
|
|
"but never named could not be judged for position"
|
|
)
|
|
if self.sort_order not in SORT_ORDERS:
|
|
raise ValueError(
|
|
f"sort_order must be one of {SORT_ORDERS}, got {self.sort_order!r} — "
|
|
"the set is closed so that an index order is reproducible from the "
|
|
"bundle rather than from the caller that happened to write it"
|
|
)
|
|
if self.sort_missing not in SORT_MISSING:
|
|
raise ValueError(
|
|
f"sort_missing must be one of {SORT_MISSING}, got {self.sort_missing!r}"
|
|
)
|
|
if self.sort_key is not None and (
|
|
self.facets is None or self.sort_key not in self.facets.keys
|
|
):
|
|
named = self.facets.keys if self.facets is not None else ()
|
|
raise ValueError(
|
|
f"sort_key {self.sort_key!r} is not named by this index policy's facets, "
|
|
f"which pin {named or '()'} — every entry would be missing the key and "
|
|
"the ordering would silently do nothing"
|
|
)
|
|
if self.requires_description and "description" not in self.link_pattern.groupindex:
|
|
raise ValueError(
|
|
"link_template names {description} but link_pattern has no "
|
|
"'description' group — the pair would round-trip in one "
|
|
"direction only"
|
|
)
|
|
|
|
@property
|
|
def requires_description(self) -> bool:
|
|
"""Whether an entry carries a description alongside label and target."""
|
|
return "{description}" in self.link_template
|
|
|
|
@property
|
|
def entry_pattern(self) -> re.Pattern[str]:
|
|
"""The pattern that recognises a managed line, facets included.
|
|
|
|
IS `link_pattern` when this policy carries no facets, which is what
|
|
makes the feature additive: DEFAULT and STRICT_V1 match exactly the
|
|
lines they always matched, byte for byte.
|
|
"""
|
|
return self._faceted_pattern if self._faceted_pattern is not None else self.link_pattern
|
|
|
|
def render_link(
|
|
self,
|
|
label: str,
|
|
target: str,
|
|
description: str | None = None,
|
|
*,
|
|
facets: Mapping[str, str] | None = None,
|
|
) -> str:
|
|
if self.requires_description and description is None:
|
|
raise ValueError(
|
|
"this index policy's entries carry a description; rendering "
|
|
"without one emits a half-written entry that parses as nothing"
|
|
)
|
|
if not self.requires_description and description is not None:
|
|
raise ValueError(
|
|
"this index policy's entries carry no description; the value "
|
|
"offered would be dropped silently"
|
|
)
|
|
if self.facets is None and facets:
|
|
raise ValueError(
|
|
"this index policy carries no facets; the values offered would be dropped silently"
|
|
)
|
|
line = self.link_template.format(label=label, target=target, description=description)
|
|
if self.facets is None or facets is None:
|
|
return line
|
|
return line + self.facets.render(facets)
|
|
|
|
def parse_entry(self, line: str) -> IndexEntry | None:
|
|
"""One managed line read back into its parts, or `None` for anything else.
|
|
|
|
Anything this returns `None` for is curated content and survives
|
|
verbatim: the index is the one file where this library writes beside
|
|
somebody else's prose.
|
|
|
|
`also_reads` is tried only after the emitted form misses, and only
|
|
here: index MAINTENANCE keys on `entry_pattern` alone, so a foreign row
|
|
this reads is never a row this rewrites.
|
|
"""
|
|
stripped = line.rstrip("\r\n")
|
|
match = self.entry_pattern.match(stripped)
|
|
if match is None:
|
|
for extra in self.also_reads:
|
|
match = extra.match(stripped)
|
|
if match is not None:
|
|
break
|
|
if match is None:
|
|
return None
|
|
groups = match.groupdict()
|
|
tail = groups.get("facets")
|
|
return IndexEntry(
|
|
label=match.group("label"),
|
|
target=match.group("target"),
|
|
description=groups.get("description"),
|
|
facets=self.facets.parse(tail) if (self.facets is not None and tail) else {},
|
|
)
|
|
|
|
def sort_entries(self, entries: Iterable[IndexEntry]) -> tuple[IndexEntry, ...]:
|
|
"""The ONE ordering every index this library writes goes through.
|
|
|
|
Both doors call this. Not because sharing is tidy, but because the
|
|
alternative was measured: with Door B and Door C writing their indexes
|
|
through separate code, an ordering wired into one of them is a profile
|
|
field the other ignores in silence — nothing raises, and both files
|
|
still parse. Two implementations of one ordering ARE the drift.
|
|
|
|
Four passes, each a stable sort, so an earlier pass is exactly the
|
|
tie-break of a later one:
|
|
|
|
1. by concept path, which is the final tie-break and makes the order
|
|
total — two entries sharing a key never fall back to chance;
|
|
2. by the named key, reversed for `descending`;
|
|
3. the entries missing the key partitioned to whichever end
|
|
`sort_missing` says. Separate from pass 2 on purpose: folding the two
|
|
into one reversible key tuple would flip the missing group along with
|
|
the order, so `sort_missing="last"` would mean "first" under
|
|
`descending`;
|
|
4. navigation last. An outer GROUPING rather than a competitor to the
|
|
key: a link down to a child index is about a directory, carries no
|
|
facets, and would lead the file under `sort_missing="first"`.
|
|
|
|
A value that is present but empty counts as missing, because
|
|
`FacetPolicy.render` already drops it — an entry that renders without
|
|
the facet must not sort as though it carried one.
|
|
|
|
With no `sort_key` the whole of passes 2 and 3 is skipped and the result
|
|
is concepts before navigation, each group ascending by concept path.
|
|
That is what every profile shipped today already emitted.
|
|
"""
|
|
ordered = sorted(entries, key=lambda entry: entry.concept_path or entry.target)
|
|
if self.sort_key is not None:
|
|
key = self.sort_key
|
|
ordered = sorted(
|
|
ordered,
|
|
key=lambda entry: entry.facets.get(key) or "",
|
|
reverse=self.sort_order == SORT_DESCENDING,
|
|
)
|
|
missing = [entry for entry in ordered if not entry.facets.get(key)]
|
|
if missing:
|
|
present = [entry for entry in ordered if entry.facets.get(key)]
|
|
ordered = (
|
|
[*missing, *present]
|
|
if self.sort_missing == SORT_MISSING_FIRST
|
|
else [*present, *missing]
|
|
)
|
|
return tuple(
|
|
sorted(ordered, key=lambda entry: entry.target.rpartition("/")[2] == self.name)
|
|
)
|
|
|
|
def required_indexes(self, directories: Sequence[str]) -> tuple[str, ...]:
|
|
"""The index paths this policy requires, given the caller's directories.
|
|
|
|
Bundle-relative, with `""` for the root. The directories are an
|
|
argument precisely so that nothing here reaches the filesystem.
|
|
"""
|
|
if not self.per_directory:
|
|
return (self.name,)
|
|
return tuple(sorted(f"{d}/{self.name}" if d else self.name for d in directories))
|
|
|
|
def violations(
|
|
self,
|
|
text: str,
|
|
*,
|
|
is_root: bool = False,
|
|
expected_targets: Collection[str] | None = None,
|
|
) -> tuple[IndexViolation, ...]:
|
|
"""Every way `text` departs from this policy, deterministically ordered.
|
|
|
|
`expected_targets` is the directory's contents as the CALLER sees them,
|
|
required exactly when `entries_match_directory` is set and refused
|
|
otherwise. Sorted by `(subject, code)` so two runs over the same file
|
|
agree.
|
|
"""
|
|
if self.entries_match_directory and expected_targets is None:
|
|
raise ValueError(
|
|
"this index policy judges the index against the directory, so "
|
|
"a listing is required — skipping the check silently would "
|
|
"pass a gate whose input never arrived"
|
|
)
|
|
if not self.entries_match_directory and expected_targets is not None:
|
|
raise ValueError(
|
|
"this index policy's index is authored rather than derived "
|
|
"from the directory, so a listing has nothing to judge"
|
|
)
|
|
|
|
found: list[IndexViolation] = []
|
|
head, body = _split_frontmatter(text)
|
|
found.extend(self._frontmatter_violations(head, is_root=is_root))
|
|
|
|
headings: list[str] = []
|
|
listed: set[str] = set()
|
|
for line in body:
|
|
stripped = line.strip()
|
|
if not stripped:
|
|
continue
|
|
if line.startswith("# "):
|
|
headings.append(line)
|
|
continue
|
|
match = self.entry_pattern.match(line)
|
|
if match is not None:
|
|
listed.add(match.group("target"))
|
|
continue
|
|
if not self.allows_prose:
|
|
found.append(
|
|
IndexViolation(
|
|
stripped,
|
|
"is neither the heading nor an entry",
|
|
"index_prose_not_allowed",
|
|
)
|
|
)
|
|
|
|
if self.heading_required:
|
|
if not headings:
|
|
found.append(
|
|
IndexViolation("", "the index carries no `# ` heading", "index_heading_missing")
|
|
)
|
|
found.extend(
|
|
IndexViolation(extra, "is a second `# ` heading", "index_heading_extra")
|
|
for extra in headings[1:]
|
|
)
|
|
|
|
if expected_targets is not None:
|
|
expected = set(expected_targets)
|
|
found.extend(
|
|
IndexViolation(
|
|
target, "is in the directory but not in the index", "index_entry_missing"
|
|
)
|
|
for target in expected - listed
|
|
)
|
|
found.extend(
|
|
IndexViolation(
|
|
target, "is in the index but not in the directory", "index_entry_unexpected"
|
|
)
|
|
for target in listed - expected
|
|
)
|
|
|
|
return tuple(sorted(found, key=lambda violation: (violation.subject, violation.code)))
|
|
|
|
def _frontmatter_violations(
|
|
self, head: Mapping[str, str], *, is_root: bool
|
|
) -> list[IndexViolation]:
|
|
if not self.root_frontmatter:
|
|
return []
|
|
if not is_root:
|
|
# One finding about the block, not one per key: a nested index
|
|
# carrying frontmatter is a single structural fact.
|
|
first = next(iter(head), None)
|
|
if first is None:
|
|
return []
|
|
return [
|
|
IndexViolation(
|
|
first,
|
|
"is frontmatter on a nested index, which carries none",
|
|
"index_root_frontmatter_unexpected",
|
|
)
|
|
]
|
|
|
|
found = [
|
|
IndexViolation(key, "is pinned on the root index and absent", "index_root_key_missing")
|
|
for key in self.root_frontmatter
|
|
if key in self.root_frontmatter_required and key not in head
|
|
]
|
|
present = [key for key in head if key in self.root_frontmatter]
|
|
declared = [key for key in self.root_frontmatter if key in head]
|
|
if present != declared:
|
|
found.append(
|
|
IndexViolation(
|
|
present[0],
|
|
f"breaks the pinned key order {', '.join(self.root_frontmatter)}",
|
|
"index_root_key_order",
|
|
)
|
|
)
|
|
return found
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SegmentationPolicy:
|
|
"""Whether a bundle admits one document expanding into many concepts.
|
|
|
|
OKF v0.2 §2 calls a concept "a single unit of knowledge within a bundle"
|
|
and a concept ID "the path of the concept's file within the bundle" —
|
|
neither ties a concept to a source file. A door emitting exactly one flat
|
|
concept per dropped file therefore implements the shape Appendix A
|
|
presents v0.2 as migrating AWAY from, and §11 cannot notice: it checks
|
|
that every non-reserved `.md` parses with a non-empty `type`, so one giant
|
|
concept is fully conformant. Conformance is the floor, not the proof.
|
|
|
|
The presence of this object IS the capability. Every downstream branch
|
|
reads `profile.segmentation is not None` and never
|
|
`IndexPolicy.per_directory`: `STRICT_V1` already sets that field True
|
|
while Door B ignores it, so keying the 1-to-N path there would silently
|
|
change a shipped profile's output and break its byte-stability pin.
|
|
|
|
Every field NAMES a key and none supplies a value. The value of
|
|
`bundle_id` tracks the caller's own identity scheme, so it arrives through
|
|
`root_frontmatter_values` (decision D5): a constant here would claim a
|
|
decision this library does not own.
|
|
"""
|
|
|
|
hierarchical_paths: bool = True
|
|
bundle_id_key: str = "bundle_id"
|
|
segment_id_key: str = "segment_id"
|
|
offset_key: str = "source_offset"
|
|
nav_label: str = "index"
|
|
# The discriminator BETWEEN segmented profiles, and the reason it has to be
|
|
# a field with a value rather than a presence check: every 1-to-N branch
|
|
# keys on `profile.segmentation is not None`, which BOTH segmented profiles
|
|
# satisfy. A step that surfaced the adjudication state on that check would
|
|
# write it into `SEGMENTED_V1` as well and move a byte-pinned golden.
|
|
#
|
|
# `None` means this profile does not surface adjudication state at all,
|
|
# matching the shape used everywhere else here. When set, it NAMES the
|
|
# frontmatter key and index facet; the value written under it is the
|
|
# adjudicator's, never this module's.
|
|
adjudication_key: str | None = None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class ProvenancePolicy:
|
|
"""Whether a concept carries an address back to the document it came from.
|
|
|
|
Two layers, and the split is load-bearing rather than tidy.
|
|
|
|
The ADDRESS is SPEC's. §5.1:303-306 makes `sources[].resource` REQUIRED
|
|
within an entry and lets it be "an absolute URL, a bundle-relative path, or
|
|
a path into a `references/` subdirectory (§6)" -- which is exactly what a
|
|
dropped file's inbox-relative path is. No new key is invented where the
|
|
spec already has one.
|
|
|
|
The LOCATOR is OURS, and it has to be. §5.1 has no field for a page, a
|
|
sheet row or a line, and the guard's frontmatter grammar (1.3.0, measured)
|
|
refuses every route to putting one inside a `sources` entry: a key outside
|
|
its `sources` allowlist is rejected by name, and a nested flow list is
|
|
rejected as "a flow mapping admits scalar leaves only". So a locator inside
|
|
the entry would be a bundle we emit and could never read back through Door
|
|
C. Top-level keys, in the shape `source_offset` already uses.
|
|
|
|
Every field NAMES a key and none supplies a value, like every other policy
|
|
here. The presence of this object IS the capability: a profile that names
|
|
no provenance writes none, which is what keeps the five shipped profiles
|
|
that do not name it byte-identical.
|
|
"""
|
|
|
|
sources_key: str = "sources"
|
|
pages_key: str = "source_pages"
|
|
sheet_key: str = "source_sheet"
|
|
rows_key: str = "source_rows"
|
|
lines_key: str = "source_lines"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class BundleProfile:
|
|
"""One bundle contract: types, frontmatter, filenames, index."""
|
|
|
|
types: TypePolicy
|
|
frontmatter: FrontmatterSchema
|
|
paths: PathPolicy
|
|
index: IndexPolicy
|
|
ownership: OwnershipPolicy = field(default_factory=OwnershipPolicy)
|
|
# Defaulted, so all four existing constants construct unchanged and every
|
|
# positional call site stays source-compatible. `None` is not "segmentation
|
|
# off" as a setting — it is the profile not having the capability at all,
|
|
# which is what the downstream `is not None` checks read.
|
|
segmentation: SegmentationPolicy | None = None
|
|
# Arm E, capability only: a profile MAY name a renderer per suffix, applied
|
|
# to extracted text before it becomes a concept body. `None` reads the same
|
|
# way `segmentation` does -- the profile does not have the capability, not
|
|
# "the capability is switched off". No domain-aware renderer exists in this
|
|
# package; writing one is a Non-Goal and is named here as unassigned so the
|
|
# absence is deliberate rather than an oversight.
|
|
renderers: Mapping[str, str] | None = None
|
|
# Defaulted to `None` for the same reason `segmentation` is: `None` is not
|
|
# "provenance off", it is the profile not having the capability, which is
|
|
# what the door's `is not None` check reads. Five of the six shipped
|
|
# profiles leave it unset and keep their bytes.
|
|
provenance: ProvenancePolicy | None = None
|
|
|
|
|
|
# The ingest-spec + Phase 2 contract. Every value here was a constant in
|
|
# `manifest`, `materialize`, `inbox` or `importer` before this module existed;
|
|
# the golden suite is what proves that move changed no bytes.
|
|
#
|
|
# This profile tracks COMMONS' spec, not an upstream Google version — the two
|
|
# axes are independent, and `ownership` is where they visibly part company. Its
|
|
# `generated` is the O2 mapping because commons ratified that shape for their
|
|
# §5 layer, while the profile remains v0.1 on every axis upstream owns.
|
|
DEFAULT = BundleProfile(
|
|
types=TypePolicy(allowed=None),
|
|
frontmatter=FrontmatterSchema(
|
|
# Door A's seven keys and Door B's six, merged into one order that
|
|
# contains both as subsequences — neither door's output moves.
|
|
order=(
|
|
"type",
|
|
"title",
|
|
"source_system",
|
|
"source_query",
|
|
"source_file",
|
|
"source_sha256",
|
|
"ingested_at",
|
|
"ingest_manifest",
|
|
"generated",
|
|
),
|
|
collapsed_keys=frozenset({"source_query"}),
|
|
),
|
|
paths=PathPolicy(
|
|
concept_suffix=".md",
|
|
ingest_prefix="ingest-",
|
|
inbox_prefix="inbox-",
|
|
import_prefix="import-",
|
|
),
|
|
index=IndexPolicy(
|
|
name="index.md",
|
|
link_template="- [{label}]({target})",
|
|
link_pattern=re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
|
|
),
|
|
# V1, ratified 2026-08-02 and executed by commons 2026-08-09 (`54e0ec7`).
|
|
# The actor is the same constant `OKF_V0_2` carries, and that is commons'
|
|
# doing rather than a merge of the two profiles: ingest-spec §7 names
|
|
# `process:okf-ingest` as THE ingest actor, so any profile stating that
|
|
# spec's layer writes it. The profiles still differ everywhere else.
|
|
ownership=OwnershipPolicy(actor="process:okf-ingest"),
|
|
)
|
|
|
|
|
|
# `FRONTMATTER_KEY_ORDER` in the proving consumer's `bundle.py`: an eleven-key
|
|
# allowlist that doubles as canonical emission order. The first eight are the
|
|
# required set their operator ratified on 2026-07-25 (measured present on
|
|
# 522/522 documents); the last three are layer-specific and stay optional.
|
|
# Deriving both sets from this one tuple is what keeps them from drifting apart
|
|
# — their BUNDLE_HASH_REGISTRY gate depends on the order.
|
|
_STRICT_V1_KEY_ORDER = (
|
|
"type",
|
|
"title",
|
|
"description",
|
|
"timestamp",
|
|
"layer",
|
|
"source_tier",
|
|
"source_url",
|
|
"source_sha",
|
|
"version",
|
|
"date",
|
|
"summary",
|
|
)
|
|
|
|
# The `claude-code-llm-wiki` contract, from `docs/phase-3-split-table.md`.
|
|
#
|
|
# Deliberately stricter than the consumer's own validator: their REQUIRED_KEYS
|
|
# constant is the first four keys, which is their emit-path minimum rather than
|
|
# their contract. If they ever ship a document class that legitimately lacks
|
|
# `source_sha`, the agreed resolution is to bump this profile — they send the
|
|
# requirement before shipping the class, not after.
|
|
#
|
|
# `paths` is DEFAULT's and that is the whole truth for it: the filename
|
|
# namespaces are an ingest-spec invariant about what the doors write, not a
|
|
# consumer preference.
|
|
#
|
|
# The index shape is read out of their code rather than recalled:
|
|
# `bundle.py:498-524` renders `* [Title](target) - description` with
|
|
# subdirectories linking to their own index; `bundle.py:527-567` walks every
|
|
# level, writing frontmatter at the bundle root only and a bare `# ` heading
|
|
# below it; `validate.py:1081-1120` (gate BUNDLE_INDEX_COMPLETE, ERROR) demands
|
|
# the index and the directory match exactly in both directions.
|
|
STRICT_V1 = BundleProfile(
|
|
types=TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"})),
|
|
frontmatter=FrontmatterSchema(
|
|
order=_STRICT_V1_KEY_ORDER,
|
|
required=frozenset(_STRICT_V1_KEY_ORDER[:8]),
|
|
allowed=frozenset(_STRICT_V1_KEY_ORDER),
|
|
nullable=frozenset({"summary"}),
|
|
key_pattern=re.compile(r"^[a-z_]+$"),
|
|
),
|
|
paths=DEFAULT.paths,
|
|
index=IndexPolicy(
|
|
name="index.md",
|
|
link_template="* [{label}]({target}) - {description}",
|
|
link_pattern=re.compile(
|
|
r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)\s]+)\) - (?P<description>.+)$"
|
|
),
|
|
per_directory=True,
|
|
heading_required=True,
|
|
allows_prose=False,
|
|
entries_match_directory=True,
|
|
root_frontmatter=("okf_version", "bundle_profile", "okf_spec_commit"),
|
|
# The proving consumer demands all three, not merely permits them: their
|
|
# root index carries exactly these keys in exactly this order on every
|
|
# bundle measured (`c5141f8`). Stated explicitly so that separating
|
|
# "permitted" from "required" costs them nothing.
|
|
root_frontmatter_required=frozenset({"okf_version", "bundle_profile", "okf_spec_commit"}),
|
|
),
|
|
)
|
|
|
|
|
|
# The ordered structure keys: what `structure.py` derives, plus the two
|
|
# producer-declared keys the 2026-08-26 bake-off measured missing from the
|
|
# index (`status` 55/55 in the documents, 0/55 in the index; `date` likewise).
|
|
# One tuple feeds both the frontmatter order and the facet set, because two
|
|
# lists of the same keys drift.
|
|
_STRUCTURE_KEYS = (
|
|
"number",
|
|
"parent",
|
|
"status",
|
|
"date",
|
|
"version",
|
|
"supersedes",
|
|
"references",
|
|
# LAST, and load-bearing: it names which of the keys before it this library
|
|
# INFERRED rather than read. A consumer that trusts nothing derived can
|
|
# still use everything else, and one that accepts both knows which half it
|
|
# is betting on. An unmarked heuristic is worse than no heuristic.
|
|
"derived",
|
|
)
|
|
|
|
# DEFAULT plus structure. Additive in the strict sense: the namespaces, the
|
|
# type policy and the ownership stamp are DEFAULT's own objects, so a bundle
|
|
# written under either profile stays re-runnable under the other, and DEFAULT's
|
|
# bytes do not move.
|
|
#
|
|
# Why a new profile rather than facets on DEFAULT: DEFAULT states commons'
|
|
# ingest-spec 6 index layer. Changing its rendered bytes from here would be
|
|
# this repo editing another repo's contract (O2), and it would churn every
|
|
# golden fixture that door has ever written. The measured defect is real, but
|
|
# the fix belongs beside the contract it changes, not inside one we do not own.
|
|
#
|
|
# The cost is deliberate and small. Facets are rendered only where a value
|
|
# exists, so a bundle whose documents declare nothing pays nothing, and the
|
|
# 2026-08-26 arm that lost on trap exposure was 6 031 characters against
|
|
# 21 879 for the head-scan it lost to - the headroom for carrying the metadata
|
|
# back into the index is most of that gap.
|
|
STRUCTURED_V1 = BundleProfile(
|
|
types=DEFAULT.types,
|
|
frontmatter=FrontmatterSchema(
|
|
order=(*DEFAULT.frontmatter.order, *_STRUCTURE_KEYS),
|
|
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
|
|
),
|
|
paths=DEFAULT.paths,
|
|
index=replace(DEFAULT.index, facets=FacetPolicy(keys=_STRUCTURE_KEYS)),
|
|
ownership=DEFAULT.ownership,
|
|
)
|
|
|
|
# OKF v0.2, as an ADDITIVE profile: `DEFAULT` states commons' ingest-spec §5
|
|
# layer and keeps stating it, so nothing here migrates anything. The key order
|
|
# is DEFAULT's followed by the §5 families v0.2 adds, which is also the order
|
|
# upstream's own reference bundles emit them in (`generated` before `sources`).
|
|
#
|
|
# Naming a family is not writing it. `verified`, `status` and `stale_after` are
|
|
# expressible so a caller can emit them in canonical order and so the schema can
|
|
# judge a document that carries them; Door A writes none of them, because a
|
|
# field with no reader is not written. Their structured v0.2 values — block
|
|
# lists of mappings — are beyond this library's value model until D1's reader
|
|
# lands; that is a reader gap, not an ordering one.
|
|
# §10 `Attested Computation`: the type's contract fields, in §10.2's own listing
|
|
# order. FORMAT only — this library supports writing, judging and round-tripping
|
|
# the contract, and implements no execution: upstream defers the receipt and
|
|
# verdict wire formats, so there is nothing to build a runtime against, and
|
|
# "did this run produce the value the sanctioned way" sits on the far side of
|
|
# this library's boundary in any case.
|
|
ATTESTED_COMPUTATION_TYPE = "Attested Computation"
|
|
_ATTESTED_COMPUTATION_FIELDS = ("runtime", "parameters", "computation", "executor", "attester")
|
|
|
|
|
|
_OKF_V0_2_KEY_ORDER = (
|
|
*DEFAULT.frontmatter.order,
|
|
"sources",
|
|
"verified",
|
|
"status",
|
|
"stale_after",
|
|
# §10.2's contract fields, appended as one block and internally in the order
|
|
# that section enumerates them. Appended rather than interleaved because the
|
|
# keys before them are what the doors actually emit, and because `emit`
|
|
# writes only the keys present — so naming these costs no byte in any bundle
|
|
# that carries none of them. Without the block they would still be emitted,
|
|
# in `emit`'s sorted tail, where `attester` precedes `runtime`: alphabetical
|
|
# order standing in for the contract's own.
|
|
*_ATTESTED_COMPUTATION_FIELDS,
|
|
)
|
|
|
|
# PROVISIONAL. Shipped first as a pre-release (`v0.5.0a1`) to a named pilot set
|
|
# — `portfolio-optimiser-claude`, the plugin marketplace catalog, and
|
|
# `claude-code-llm-wiki` — and this surface may change on their feedback without
|
|
# a deprecation cycle. Saying so is what buys the freedom to act on the
|
|
# feedback; discovering it later is what would make the pilot a de-facto
|
|
# release. The versioned constants are the stable binding.
|
|
#
|
|
# Two things this profile deliberately does NOT do:
|
|
#
|
|
# - **It closes nothing.** §14 forbids a conformant consumer to reject on an
|
|
# unknown `type` value or on unknown additional keys, so an allowlist or a key
|
|
# pattern here would put the profile in violation of the version it is named
|
|
# for. `type` is required and is the only one (§4, §11).
|
|
# - **It NAMES `okf_version` but never carries its value.** The value tracks the
|
|
# upstream Google version and belongs to catalog (decision E1), so a constant
|
|
# here would be this repo claiming a decision it does not own — and the one
|
|
# that would have to be chased on every upstream release. The caller supplies
|
|
# it (`materialize_bundle(..., root_frontmatter_values=...)`); this policy
|
|
# fixes only the key and its position.
|
|
#
|
|
# WHERE it goes was open until 2026-07-31 between upstream's root-index
|
|
# frontmatter block and catalog's body-line convention. Catalog verified
|
|
# upstream themselves at the pinned commit `3fcbb9f` and reported §8:509-510
|
|
# ("Index files contain no frontmatter, with one exception: a bundle-root
|
|
# `index.md` MAY carry an `okf_version` key") and §12:773-775 ("in a
|
|
# bundle-root `index.md` frontmatter block (the only place frontmatter is
|
|
# permitted in an `index.md`)"). Frontmatter it is; their own spec diverges
|
|
# from upstream here, and that divergence is theirs to resolve.
|
|
#
|
|
# Declaring it stays a MAY: none of upstream's four reference bundles carries
|
|
# the key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits),
|
|
# so omitting `root_frontmatter_values` emits no block. The JUDGE says the same
|
|
# thing since P1-F1 — the key is named here and left out of
|
|
# `root_frontmatter_required` — because for one release it did not, and a
|
|
# bundle exercising the MAY was reported as violating by the very profile that
|
|
# had emitted it correctly.
|
|
#
|
|
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
|
|
# profile cannot be read back through a guard-gated import. The guard's T2
|
|
# frontmatter grammar admits scalars and flat lists of strings, and refuses every
|
|
# route to a mapping — flow on the disallowed-indicator set, block on the
|
|
# nested-mapping check, dotted keys on the key pattern. So `generated` as the
|
|
# mapping v0.2 specifies has no expressible form through that gate at all. This
|
|
# binds what can be IMPORTED (Door C), never what we emit: Door B's
|
|
# `screen_output` does not run that parser.
|
|
OKF_V0_2 = BundleProfile(
|
|
types=TypePolicy(allowed=None),
|
|
frontmatter=FrontmatterSchema(
|
|
order=_OKF_V0_2_KEY_ORDER,
|
|
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
|
|
required=frozenset({"type"}),
|
|
# §10.2's one type-conditional rule, and the whole of it: `runtime` is
|
|
# REQUIRED for this type because it is what says how to run the
|
|
# computation and therefore what `parameters` mean. The other four
|
|
# contract fields stay optional — `computation` absent means the body
|
|
# fence IS the computation (§10.3), which is a valid concept.
|
|
required_by_type={ATTESTED_COMPUTATION_TYPE: frozenset({"runtime"})},
|
|
),
|
|
paths=DEFAULT.paths,
|
|
# DEFAULT's index in every respect but one: the root MAY carry `okf_version`
|
|
# (§8, §12). Built with `replace` rather than restated so a later change to
|
|
# the shared shape cannot drift between the two.
|
|
index=replace(DEFAULT.index, root_frontmatter=("okf_version",)),
|
|
# Byte-identical to `DEFAULT.ownership` since V1, and deliberately NOT
|
|
# written as a reference to it. The two agree by coincidence of commons
|
|
# adopting §7's actor, not by dependency: this profile states UPSTREAM's
|
|
# v0.2, so if commons ever moves their actor again, this one must not
|
|
# follow. Restating it is what keeps that independence expressible.
|
|
ownership=OwnershipPolicy(actor="process:okf-ingest"),
|
|
)
|
|
|
|
|
|
# STRUCTURED_V1 plus the capability to split ONE document into MANY concepts.
|
|
#
|
|
# A new profile rather than a flag on an existing one, for the same reason
|
|
# STRUCTURED_V1 was: `DEFAULT` states commons' ingest-spec §6 layer and
|
|
# `STRICT_V1` mirrors a consumer's ratified contract, so moving either one's
|
|
# bytes from here would be this repo editing another repo's contract (O2).
|
|
# `OKF_V0_2` states upstream's. What stays byte-stable is therefore all four of
|
|
# them, proven by `tests/test_segmented_profile.py` at the profile level and by
|
|
# the golden suite at the byte level.
|
|
#
|
|
# What this profile adds on top of STRUCTURED_V1's facets and structure keys:
|
|
#
|
|
# - `segmentation` — the capability itself. Its presence is what every 1-to-N
|
|
# branch keys on.
|
|
# - `per_directory` — a segmented bundle has directories, and a bundle whose
|
|
# nested concepts are reachable only by guessing a path is a filing cabinet
|
|
# again. Set HERE rather than inherited, because STRUCTURED_V1 leaves it off.
|
|
# - `root_frontmatter=("bundle_id",)` — the identity carrier settled by order
|
|
# `…2527032751`: form (c), a root identifier consumers join on. Naming the
|
|
# key PERMITS it and fixes its position; the caller supplies the value (D5),
|
|
# because a bundle is a collection the caller delimits and only the caller
|
|
# knows what it is called. It is deliberately absent from
|
|
# `root_frontmatter_required`: Door B refuses a segmentation plan without an
|
|
# id, which is a rule about the door, not about every index this profile
|
|
# might ever write.
|
|
SEGMENTED_V1 = BundleProfile(
|
|
types=STRUCTURED_V1.types,
|
|
frontmatter=STRUCTURED_V1.frontmatter,
|
|
paths=STRUCTURED_V1.paths,
|
|
index=replace(
|
|
STRUCTURED_V1.index,
|
|
per_directory=True,
|
|
root_frontmatter=("bundle_id",),
|
|
# SPEC section 8's own row form, read and never written. See
|
|
# `IndexPolicy.also_reads`. Set on the SEGMENTED profiles alone:
|
|
# `DEFAULT` states commons' spec and `STRICT_V1` the wiki's ratified
|
|
# contract, and widening either from here would be this repository
|
|
# editing another repository's contract (O2).
|
|
also_reads=(
|
|
re.compile(
|
|
r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)\s]+)\)"
|
|
r"(?: - (?P<description>.+))?$"
|
|
),
|
|
),
|
|
),
|
|
ownership=STRUCTURED_V1.ownership,
|
|
segmentation=SegmentationPolicy(),
|
|
)
|
|
|
|
|
|
# Bound so the sixth profile's facet extension is typed: `IndexPolicy.facets`
|
|
# is `FacetPolicy | None`, and `SEGMENTED_V1` is known here to carry one.
|
|
_SEGMENTED_FACETS = SEGMENTED_V1.index.facets
|
|
assert _SEGMENTED_FACETS is not None
|
|
|
|
|
|
# The sixth profile. A segmented bundle could not declare which upstream spec
|
|
# it targets: `SEGMENTED_V1` names `bundle_id`, `OKF_V0_2` names `okf_version`,
|
|
# and the two never intersected. Additive, as upstream support always is here --
|
|
# a new profile, never a migration of an existing one.
|
|
#
|
|
# THE INDEX POLICY IS DECIDED HERE, NOT INHERITED, and that is the one thing in
|
|
# this construction that is easy to get wrong. Measured: `OKF_V0_2.index` has
|
|
# `facets=None` and `per_directory=False`, while `SEGMENTED_V1.index` has both.
|
|
# Building the sixth profile on `OKF_V0_2`'s index would have produced a
|
|
# segmented bundle with NO faceted index -- structurally valid, conformant, and
|
|
# missing the surface a consumer reads. So the index comes from `SEGMENTED_V1`
|
|
# with both root keys named, and the spec declaration comes from `OKF_V0_2`.
|
|
#
|
|
# `okf_version`'s VALUE is not here and must never be: a profile names a key,
|
|
# the caller owns its value (decision E1). The value tracks the upstream Google
|
|
# version and belongs to catalog; a constant here would claim a decision this
|
|
# library does not own, and would be the one thing to chase on every upstream
|
|
# release.
|
|
SEGMENTED_OKF_V0_2 = BundleProfile(
|
|
types=OKF_V0_2.types,
|
|
frontmatter=OKF_V0_2.frontmatter,
|
|
paths=SEGMENTED_V1.paths,
|
|
# The facet tuple is EXTENDED here rather than shared, and that is the
|
|
# discriminator doing its job: `SEGMENTED_V1.index.facets` is one object
|
|
# both profiles would otherwise point at, so appending `adjudication` to it
|
|
# would surface the state under the older profile too and move a
|
|
# byte-pinned golden. `FacetPolicy.render` refuses any key a policy does
|
|
# not name, which is why the key has to live here and cannot be added by
|
|
# the door at write time.
|
|
index=replace(
|
|
SEGMENTED_V1.index,
|
|
root_frontmatter=("okf_version", "bundle_id"),
|
|
facets=replace(
|
|
_SEGMENTED_FACETS,
|
|
keys=_SEGMENTED_FACETS.keys + ("adjudication",),
|
|
),
|
|
),
|
|
ownership=OKF_V0_2.ownership,
|
|
# Constructed rather than `replace`d off `SEGMENTED_V1.segmentation`: that
|
|
# attribute is typed `| None`, and the equality is asserted in the suite so
|
|
# this stays a fresh copy of the same policy plus the discriminator.
|
|
segmentation=SegmentationPolicy(adjudication_key="adjudication"),
|
|
# O3, and set on THIS profile alone. `sources` is a v0.2 key, so a profile
|
|
# stating v0.1 must not name it; `DEFAULT` and `STRICT_V1` state contracts
|
|
# owned in other repositories, so adding a key to either from here would be
|
|
# this repository editing someone else's contract (O2); and `OKF_V0_2` is
|
|
# Door A's, where `sources` is already written from the manifest. What is
|
|
# left is the segmented v0.2 profile -- the one whose concepts come from a
|
|
# dropped binary document and therefore the only one with an original to
|
|
# point at.
|
|
provenance=ProvenancePolicy(),
|
|
)
|
|
|
|
|
|
# "The latest version supported as STABLE", not the latest present in this
|
|
# module. It therefore keeps v0.1 UPSTREAM semantics for as long as v0.2 is
|
|
# provisional, and flipping it is the GA event — one auditable action rather
|
|
# than a side effect of a merge.
|
|
#
|
|
# "v0.1 semantics" is about upstream and has never covered commons' layer. V1
|
|
# moved `DEFAULT`'s stamp, so this alias's bytes moved with it, before GA and
|
|
# without the flip. That is not a leak in the alias: the two contracts are
|
|
# independent axes, and a consumer bound here tracks both by construction.
|
|
#
|
|
# The tradeoff is stated rather than hidden: an alias that moves means a consumer
|
|
# bound to it inherits upstream's breaking changes on a library upgrade. The
|
|
# versioned constants are the stable binding and are what a consumer should pin;
|
|
# this is for callers who have explicitly opted into tracking.
|
|
OKF_LATEST = DEFAULT
|