Commons ratified V1 2026-08-02 and executed it at `54e0ec7`; verified
against their tree rather than taken on report. ingest-spec.md:217 now
defines `generated` as `{ by: process:okf-ingest, at: <ingested_at> }`,
unquoted, `at` repeating `ingested_at` verbatim. `generated: true` no
longer appears in the spec.
`DEFAULT` states commons' §5 layer, so its stamp is theirs to decide.
`DEFAULT.ownership` gains the actor; the four goldens this repo's plan
named in advance were regenerated by RUNNING the materializer, each on
its own case's `ingested-at.txt`. The v0.2 golden was untouched, as
predicted -- it has carried the O2 form since D5.
Not a migration onto OKF v0.2: `DEFAULT` stays v0.1 on every axis
upstream owns and still emits no `sources`. Commons' spec and the Google
version are independent axes, and comments that narrated them as one
were rewritten rather than left to mislead. README and CLAUDE.md said
the additive rule without that boundary, which would have told a
consumer their DEFAULT bytes can never move; both now state it.
V-A3 is amended, not dropped. `DEFAULT` must OWN the mapping it now
writes -- a profile refusing its own output fires the collision gate on
files its own previous run wrote -- while a mapping naming a foreign
actor, or §7's `human:` actor on curated content, stays unowned. That
half is what carried the safety and it is asserted directly.
§11's stamp-integrity condition moved with the value: the forgeable
stamp was `true` and is now the mapping naming the ingest actor. The
defence was never the value -- the §3 scan globs `ingest-*.md`, so a
Door C import is unreachable however well it forges. Second spoof test
added; both were hand-mutated (glob widened to `*.md`) to confirm they
can fail.
The characterization test derived its foreign-stamp fixture from the
literal `generated: true`, which V1 leaves without a referent -- a
silent no-op waiting to happen. It now derives the needle from the
profile and asserts the substitution occurred.
Door B is deliberately untouched: not the ingest-spec's, marker is
`generated` + `source_file`, disjoint from Door A's `ingest_manifest`,
and the divergence predates V1.
Nothing released or notified. The pilot set pins `v0.5.0a2`, not `main`,
so this is invisible to portfolio-optimiser's freeze and demo; the
consumer exposure report is owed at the release that carries this.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VwcjUXbKySLbEG5WqTNkta
854 lines
39 KiB
Python
854 lines
39 KiB
Python
"""The bundle contract as configuration (Phase 3).
|
|
|
|
What a valid bundle looks like — which concept types exist, which frontmatter
|
|
keys are emitted and in which order, which filename namespaces the doors own,
|
|
and what an index line looks like — is a profile, not a set of constants
|
|
scattered across the doors. `DEFAULT` is exactly the ingest-spec v1 + Phase 2
|
|
contract, so nothing observable changes for a caller that never mentions a
|
|
profile; the golden fixtures are the byte-level proof.
|
|
|
|
Two things deliberately do NOT live here. Security is the guard's, always: no
|
|
disposition, origin or channel vocabulary belongs on a profile. And the
|
|
reserved `verdict` layer is a spec invariant rather than profile config — it is
|
|
refused at construction, so a profile admitting it cannot be built, let alone
|
|
passed to a door. The `timestamp`/`generated` pair is refused the same way, and
|
|
for the same reason: both are things a profile must not be able to express.
|
|
|
|
Profiles are constructed in code. Config-file loading and inheritance chains
|
|
are extension points, not v1 (settled with the operator at phase start).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from collections.abc import Collection, Mapping, Sequence
|
|
from dataclasses import dataclass, field, replace
|
|
|
|
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
|
|
# the only path into it. Compared case-insensitively, as both doors already do.
|
|
RESERVED_OKF_TYPE = "verdict"
|
|
|
|
# The one key PAIR no profile may name (OKF §13.1): `timestamp` is readable as
|
|
# a legacy stand-in for `generated.at` only while `generated` is ABSENT, so a
|
|
# schema able to name both can describe a document that has neither. Stated as
|
|
# key names rather than as a judgement on values because every `generated` a
|
|
# schema can express today is a scalar (`_is_legal_value`) and therefore
|
|
# malformed as a v0.2 mapping — naming both IS the hazard here. When a value
|
|
# model can express a well-formed `generated`, this narrows with it.
|
|
# "Naming" spans EVERY field that puts a key in the schema's namespace — `order`,
|
|
# `required`, `allowed` and `nullable` alike. A field left out of that union is a
|
|
# hole in the gate, not a narrower gate: the namespace is open by default, so a
|
|
# key named only by `nullable` is admitted just as surely as an emitted one.
|
|
_TIMESTAMP_FALLBACK_PAIR = frozenset({"timestamp", "generated"})
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class TypeRejection:
|
|
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
|
|
|
|
The policy does not raise: Door A refuses with `ManifestError` and Door B
|
|
with `MaterializationError`, so the refusal has to be reported rather than
|
|
thrown. `reason` completes the sentence "<label>okf_type ..." and `code` is
|
|
the stable `IngestError.code` the caller asserts on.
|
|
"""
|
|
|
|
reason: str
|
|
code: str
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class TypePolicy:
|
|
"""Which `okf_type` values a bundle admits.
|
|
|
|
`allowed` is `None` for an open set — Phase 1/2 accept any type the
|
|
operator names — or a closed enum. Either way the reserved layer is
|
|
excluded, and a closed set that names it fails at construction.
|
|
"""
|
|
|
|
allowed: frozenset[str] | None = None
|
|
|
|
def __post_init__(self) -> None:
|
|
if self.allowed is None:
|
|
return
|
|
named = sorted(value for value in self.allowed if value.lower() == RESERVED_OKF_TYPE)
|
|
if named:
|
|
raise ValueError(
|
|
f"a profile must not admit the reserved {RESERVED_OKF_TYPE!r} layer "
|
|
f"(got {', '.join(repr(value) for value in named)}) — the promotion "
|
|
"gate is the only path into it (ingest-spec §3)"
|
|
)
|
|
|
|
def rejection(self, okf_type: str) -> TypeRejection | None:
|
|
"""The refusal for `okf_type`, or `None` when the profile admits it."""
|
|
if okf_type.lower() == RESERVED_OKF_TYPE:
|
|
return TypeRejection(
|
|
reason=f"must not be {RESERVED_OKF_TYPE!r} (reserved layer)",
|
|
code="okf_type_reserved",
|
|
)
|
|
if self.allowed is not None and okf_type not in self.allowed:
|
|
return TypeRejection(
|
|
reason=f"must be one of {', '.join(sorted(self.allowed))}, got {okf_type!r}",
|
|
code="okf_type_not_allowed",
|
|
)
|
|
return None
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SchemaViolation:
|
|
"""One way a document's frontmatter departs from the schema.
|
|
|
|
Reported rather than raised, like `TypeRejection`: a document can depart in
|
|
several ways at once, and the caller decides whether that is a refusal, a
|
|
report line, or a migration to-do. `code` is the stable machine-readable
|
|
part; `reason` completes the sentence "<key> ...".
|
|
"""
|
|
|
|
key: str
|
|
reason: str
|
|
code: str
|
|
|
|
|
|
def _is_legal_value(value: object) -> bool:
|
|
"""A string scalar, or a non-empty list of strings — nothing else.
|
|
|
|
The list shape is real, not hypothetical: the proving consumer's `Release`
|
|
pages carry block lists. A schema that assumed scalars-only would reject
|
|
pages they ship today.
|
|
"""
|
|
if isinstance(value, str):
|
|
return True
|
|
if isinstance(value, list):
|
|
return bool(value) and all(isinstance(item, str) for item in value)
|
|
return False
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class FrontmatterSchema:
|
|
"""The frontmatter key namespace, what it admits, and how it is emitted.
|
|
|
|
`order` is the canonical emission order and `collapsed_keys` names the keys
|
|
whose whitespace runs collapse to single spaces on the way out. The DEFAULT
|
|
order spans both doors' key sets: Door A emits seven of these keys and Door
|
|
B six, and each door's subset comes out in exactly the order it wrote by
|
|
hand in Phases 1 and 2.
|
|
|
|
The judging fields — `required`, `allowed`, `nullable`, `key_pattern` — are
|
|
read by `violations`. `allowed` is `None` for an open namespace, which is
|
|
what DEFAULT keeps: Door A emits seven keys, Door B six, and Door C writes
|
|
an imported concept verbatim with whatever the sender wrote, so a closed
|
|
namespace or a required set on DEFAULT would declare invalid the very
|
|
bundles this library produces.
|
|
|
|
`required_by_type` is the one judging field that keys off a frontmatter
|
|
*value* rather than a key: `{<type value>: {<keys that type must carry>}}`.
|
|
OKF v0.2 §10.2 introduces exactly one such rule — `runtime` is REQUIRED for
|
|
`Attested Computation` and for no other type — and it cannot be expressed
|
|
through `required`, which would demand the key of every document. A type
|
|
the mapping does not name carries no extra requirement, which is what keeps
|
|
the field inside §14: a consumer must not reject on an unknown `type`, so a
|
|
conditional keyed on a type we do not know stays silent rather than guesses.
|
|
"""
|
|
|
|
order: tuple[str, ...]
|
|
collapsed_keys: frozenset[str] = field(default_factory=frozenset)
|
|
required: frozenset[str] = field(default_factory=frozenset)
|
|
allowed: frozenset[str] | None = None
|
|
nullable: frozenset[str] = field(default_factory=frozenset)
|
|
key_pattern: re.Pattern[str] | None = None
|
|
required_by_type: Mapping[str, frozenset[str]] = field(default_factory=dict)
|
|
|
|
def __post_init__(self) -> None:
|
|
conditional: frozenset[str] = frozenset().union(*self.required_by_type.values())
|
|
named = set(self.order) | set(self.required) | set(self.nullable) | conditional
|
|
if self.allowed is not None:
|
|
named |= set(self.allowed)
|
|
if _TIMESTAMP_FALLBACK_PAIR <= named:
|
|
raise ValueError(
|
|
"a profile must not name both 'timestamp' and 'generated' (OKF "
|
|
"§13.1 grants the timestamp fallback only while `generated` is "
|
|
"ABSENT, so a schema naming both can describe a document with "
|
|
"neither a valid `generated.at` nor an eligible fallback)"
|
|
)
|
|
if self.allowed is None:
|
|
return
|
|
for label, keys in (
|
|
("required", self.required),
|
|
("nullable", self.nullable),
|
|
("type-conditional required", conditional),
|
|
):
|
|
stray = sorted(keys - self.allowed)
|
|
if stray:
|
|
raise ValueError(
|
|
f"{label} keys must be inside the allowlist "
|
|
f"(got {', '.join(repr(key) for key in stray)}) — a schema that "
|
|
"demands a key it also forbids can never be satisfied"
|
|
)
|
|
|
|
def violations(self, values: Mapping[str, object]) -> tuple[SchemaViolation, ...]:
|
|
"""Every way `values` departs from this schema, deterministically ordered.
|
|
|
|
Sorted by `(key, code)` rather than reported in mapping order: two runs
|
|
over the same document must produce the same report, and mapping order
|
|
is an accident of how the document was parsed.
|
|
|
|
Takes an already-parsed mapping. Parsing strict frontmatter is a
|
|
separate concern — this library's line-oriented parser cannot represent
|
|
the block lists and nulls a strict schema admits, which is why Door C
|
|
writes imported concepts verbatim rather than round-tripping them.
|
|
"""
|
|
found: list[SchemaViolation] = []
|
|
|
|
for key in self.required - set(values):
|
|
found.append(SchemaViolation(key, "is required and absent", "frontmatter_key_missing"))
|
|
|
|
# The type-conditional rules (§10.2 today). Keyed off the VALUE, so it
|
|
# fires only on a type this schema names: an absent, non-scalar or
|
|
# unknown `type` carries no extra requirement. A separate code from the
|
|
# unconditional case because the two are different claims — one says the
|
|
# document is malformed, the other that it is malformed *for what it
|
|
# says it is* — and a caller may well treat them differently.
|
|
declared = values.get("type")
|
|
if isinstance(declared, str):
|
|
for key in self.required_by_type.get(declared, frozenset()) - set(values):
|
|
found.append(
|
|
SchemaViolation(
|
|
key,
|
|
f"is required for type {declared!r} and absent",
|
|
"frontmatter_key_missing_for_type",
|
|
)
|
|
)
|
|
|
|
for key, value in values.items():
|
|
if self.key_pattern is not None and not self.key_pattern.fullmatch(key):
|
|
found.append(
|
|
SchemaViolation(
|
|
key,
|
|
f"is not a legal key name (must match {self.key_pattern.pattern})",
|
|
"frontmatter_key_malformed",
|
|
)
|
|
)
|
|
continue
|
|
if self.allowed is not None and key not in self.allowed:
|
|
found.append(
|
|
SchemaViolation(
|
|
key, "is not on the key allowlist", "frontmatter_key_not_allowed"
|
|
)
|
|
)
|
|
continue
|
|
if value is None:
|
|
if key not in self.nullable:
|
|
found.append(SchemaViolation(key, "must not be null", "frontmatter_value_null"))
|
|
continue
|
|
if not _is_legal_value(value):
|
|
found.append(
|
|
SchemaViolation(
|
|
key,
|
|
"must be a string or a non-empty list of strings",
|
|
"frontmatter_value_shape",
|
|
)
|
|
)
|
|
|
|
return tuple(sorted(found, key=lambda violation: (violation.key, violation.code)))
|
|
|
|
def emit(self, values: Mapping[str, str]) -> str:
|
|
"""Render `values` as line-oriented `key: value`, one line per key.
|
|
|
|
Commons decision D1: the keys named in `order` come first, in that
|
|
order, followed by any remaining keys SORTED. Ordering the tail rather
|
|
than trusting insertion order is what makes a regeneration over the
|
|
same data byte-identical.
|
|
|
|
Returns the lines only — the caller owns the `---` fences.
|
|
"""
|
|
named = [key for key in self.order if key in values]
|
|
tail = sorted(key for key in values if key not in self.order)
|
|
return "\n".join(f"{key}: {self._render(key, values[key])}" for key in [*named, *tail])
|
|
|
|
def _render(self, key: str, value: str) -> str:
|
|
# §5 mandates whitespace-run collapse for `source_query` only
|
|
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
|
|
# must render on one line. Every other value is validated single-line
|
|
# at load and emitted verbatim — validation, not repair — so
|
|
# operator-supplied bytes (e.g. a title's internal double space)
|
|
# survive.
|
|
return " ".join(value.split()) if key in self.collapsed_keys else value
|
|
|
|
|
|
# The v0.1 ingest stamp. A literal rather than a configurable value: it is what
|
|
# every bundle this library has already written carries, and recognising it is
|
|
# what keeps those bundles re-runnable under a later profile.
|
|
_V0_1_STAMP = "true"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class OwnershipPolicy:
|
|
"""The `generated` value this profile writes, and the values it owns back.
|
|
|
|
Ownership is the §3 collision gate's question — may this run replace the
|
|
file already sitting at a generated name? — and the answer is a profile's,
|
|
because the stamp differs per profile. The literal `true` is the older
|
|
form; the O2 form is `generated: { by: <actor>, at: <ingested_at> }` (§5),
|
|
where the actor takes §7's `process:<id>` form.
|
|
|
|
Which form a profile writes is NOT read off an upstream version. `DEFAULT`
|
|
states commons' ingest-spec layer and writes the O2 form because commons
|
|
ratified it (V1, `54e0ec7`), while remaining v0.1 on every axis upstream
|
|
owns; `STRICT_V1` names no `generated` at all. The stamp tracks whichever
|
|
contract the profile states, and those contracts move independently.
|
|
|
|
`actor` is `None` for the literal stamp. Where it is set it carries no version,
|
|
deliberately: the value sits inside a byte-compared golden, so a producer
|
|
version there would fire golden regression on every release without any
|
|
contract having changed, and would make a shared cross-implementation
|
|
fixture impossible by construction (plan V1(d), operator 2026-07-27).
|
|
|
|
Recognition is ONE-WAY, and both directions are decisions rather than
|
|
accidents. A profile with an actor owns the literal stamp as well, so a
|
|
bundle written before V1 re-runs IN PLACE — the black-box promise is that a
|
|
spec release costs a consumer a re-run and nothing more. The reverse is
|
|
refused: a profile without an actor fails the run rather than replacing a
|
|
file whose shape it does not read (V-A3).
|
|
|
|
"Owns the O2 form" is never "owns any mapping". The prefix binds the
|
|
profile's OWN actor, so a mapping naming a different one — another
|
|
implementation's, or §7's `human:` actor on curated content — stays
|
|
unowned. That is what keeps the key's mere presence from proving authorship,
|
|
which upstream v0.2 makes load-bearing by writing `generated` on
|
|
hand-authored files too.
|
|
|
|
The actor test is a PREFIX rather than an equality, because the value carries
|
|
`ingested_at` and therefore differs on every run by design. It works because
|
|
`parse_frontmatter` returns the whole flow mapping as one opaque string
|
|
(V-A2) — no structure this library cannot yet read is parsed here.
|
|
"""
|
|
|
|
actor: str | None = None
|
|
|
|
def stamp(self, ingested_at: str) -> str:
|
|
"""The `generated` value a run at `ingested_at` writes."""
|
|
if self.actor is None:
|
|
return _V0_1_STAMP
|
|
return f"{{ by: {self.actor}, at: {ingested_at} }}"
|
|
|
|
def owns(self, value: str) -> bool:
|
|
"""Whether a `generated` value read back marks this library's output."""
|
|
if value == _V0_1_STAMP:
|
|
return True
|
|
if self.actor is None:
|
|
return False
|
|
return value.startswith(f"{{ by: {self.actor},")
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class PathPolicy:
|
|
"""The filename namespaces the three doors write into.
|
|
|
|
Each prefix keeps its door's generated names disjoint from `index.md`, from
|
|
the other doors, and from `promoted-verdict-*`, for every id the grammar
|
|
admits. The id grammar itself stays in `materialize`: it is a pattern the
|
|
slugger derives a separator class from, not a name to configure.
|
|
"""
|
|
|
|
concept_suffix: str
|
|
ingest_prefix: str
|
|
inbox_prefix: str
|
|
import_prefix: str
|
|
|
|
|
|
def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
|
|
"""The leading `---` block as ordered keys, and the body lines after it.
|
|
|
|
Line-oriented, the same shape `materialize.parse_frontmatter` reads —
|
|
duplicated rather than imported because `materialize` imports this module,
|
|
and because that one takes a path where an index reader has only text.
|
|
"""
|
|
lines = text.splitlines()
|
|
if not lines or lines[0].strip() != "---":
|
|
return {}, lines
|
|
head: dict[str, str] = {}
|
|
for offset, line in enumerate(lines[1:], start=1):
|
|
if line.strip() == "---":
|
|
return head, lines[offset + 1 :]
|
|
key, sep, value = line.partition(":")
|
|
if sep:
|
|
head[key.strip()] = value.strip()
|
|
return head, []
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class IndexViolation:
|
|
"""One way an index file departs from the policy.
|
|
|
|
Reported rather than raised, like `SchemaViolation`. `subject` is whatever
|
|
the finding is about — a line, a link target, or a frontmatter key — and
|
|
`code` is the stable machine-readable part.
|
|
"""
|
|
|
|
subject: str
|
|
reason: str
|
|
code: str
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class IndexPolicy:
|
|
"""The index file, the shape of its entries, and where indexes must exist.
|
|
|
|
`link_template` renders a managed line and `link_pattern` recognises one.
|
|
Both are carried because §6 does both — append on write, rewrite on
|
|
maintenance — and a round-trip test is what keeps the pair honest. The
|
|
pattern is anchored to the whole line by construction: removal keys on this
|
|
exact shape, never a bare substring, so curated prose that mentions a
|
|
target inline survives verbatim.
|
|
|
|
The judging fields are read by `violations` and `required_indexes`, and
|
|
every one of them is off under DEFAULT. They exist because two consumers
|
|
disagree about this file in ways no single shape can hold:
|
|
|
|
- `per_directory` — an index at every level, or only at the bundle root.
|
|
Upstream OKF binds `index.md` to the root ALONE, so a default demanding
|
|
one per level would declare upstream-conforming bundles invalid. Policy,
|
|
never an OKF rule.
|
|
- `heading_required` / `allows_prose` — the wiki's index is a generated
|
|
heading plus entries and nothing else; the catalog's *requires*
|
|
progressive-disclosure prose. Opposite requirements, both expressible.
|
|
- `entries_match_directory` — whether the index must match the directory
|
|
exactly, in both directions. The wiki enforces this at ERROR
|
|
(`BUNDLE_INDEX_COMPLETE`); the catalog holds that an index is an
|
|
AUTHORED count of a directory's children and that a validator
|
|
enumerating the directory has implemented the wrong contract. Neither
|
|
posture is baked in, and this library never enumerates anything: the
|
|
caller supplies the listing, and a policy that does not judge it refuses
|
|
to be handed one at all — so code written to the wrong contract fails at
|
|
the call rather than passing every test one would think to write.
|
|
- `root_frontmatter` — the ordered keys the ROOT index may carry, where
|
|
nested indexes carry none. Confirmed independently in both consumers, so
|
|
the asymmetry is shape rather than one repo's preference. Naming a key
|
|
here PERMITS it and fixes its position; it does not demand it.
|
|
- `root_frontmatter_required` — the subset that must actually be present.
|
|
Separate from the tuple above because the two are different claims, and
|
|
collapsing them was P1-F1: `OKF_V0_2` names `okf_version` to fix its
|
|
position, but upstream §8/§12 grant it as a MAY, and judging the pair as
|
|
one field failed 14 of 17 real bundles — upstream's own four included —
|
|
each with exactly this one violation, while our emitter treated the same
|
|
key as optional. A profile whose consumer really does demand the keys
|
|
says so here, which is what `STRICT_V1` does.
|
|
"""
|
|
|
|
name: str
|
|
link_template: str
|
|
link_pattern: re.Pattern[str]
|
|
per_directory: bool = False
|
|
heading_required: bool = False
|
|
allows_prose: bool = True
|
|
entries_match_directory: bool = False
|
|
root_frontmatter: tuple[str, ...] = ()
|
|
root_frontmatter_required: frozenset[str] = field(default_factory=frozenset)
|
|
|
|
def __post_init__(self) -> None:
|
|
stray = sorted(self.root_frontmatter_required - set(self.root_frontmatter))
|
|
if stray:
|
|
raise ValueError(
|
|
f"required root keys must be named in the ordered set "
|
|
f"(got {', '.join(repr(key) for key in stray)}) — a key demanded "
|
|
"but never named could not be judged for position"
|
|
)
|
|
if self.requires_description and "description" not in self.link_pattern.groupindex:
|
|
raise ValueError(
|
|
"link_template names {description} but link_pattern has no "
|
|
"'description' group — the pair would round-trip in one "
|
|
"direction only"
|
|
)
|
|
|
|
@property
|
|
def requires_description(self) -> bool:
|
|
"""Whether an entry carries a description alongside label and target."""
|
|
return "{description}" in self.link_template
|
|
|
|
def render_link(self, label: str, target: str, description: str | None = None) -> str:
|
|
if self.requires_description and description is None:
|
|
raise ValueError(
|
|
"this index policy's entries carry a description; rendering "
|
|
"without one emits a half-written entry that parses as nothing"
|
|
)
|
|
if not self.requires_description and description is not None:
|
|
raise ValueError(
|
|
"this index policy's entries carry no description; the value "
|
|
"offered would be dropped silently"
|
|
)
|
|
return self.link_template.format(label=label, target=target, description=description)
|
|
|
|
def required_indexes(self, directories: Sequence[str]) -> tuple[str, ...]:
|
|
"""The index paths this policy requires, given the caller's directories.
|
|
|
|
Bundle-relative, with `""` for the root. The directories are an
|
|
argument precisely so that nothing here reaches the filesystem.
|
|
"""
|
|
if not self.per_directory:
|
|
return (self.name,)
|
|
return tuple(sorted(f"{d}/{self.name}" if d else self.name for d in directories))
|
|
|
|
def violations(
|
|
self,
|
|
text: str,
|
|
*,
|
|
is_root: bool = False,
|
|
expected_targets: Collection[str] | None = None,
|
|
) -> tuple[IndexViolation, ...]:
|
|
"""Every way `text` departs from this policy, deterministically ordered.
|
|
|
|
`expected_targets` is the directory's contents as the CALLER sees them,
|
|
required exactly when `entries_match_directory` is set and refused
|
|
otherwise. Sorted by `(subject, code)` so two runs over the same file
|
|
agree.
|
|
"""
|
|
if self.entries_match_directory and expected_targets is None:
|
|
raise ValueError(
|
|
"this index policy judges the index against the directory, so "
|
|
"a listing is required — skipping the check silently would "
|
|
"pass a gate whose input never arrived"
|
|
)
|
|
if not self.entries_match_directory and expected_targets is not None:
|
|
raise ValueError(
|
|
"this index policy's index is authored rather than derived "
|
|
"from the directory, so a listing has nothing to judge"
|
|
)
|
|
|
|
found: list[IndexViolation] = []
|
|
head, body = _split_frontmatter(text)
|
|
found.extend(self._frontmatter_violations(head, is_root=is_root))
|
|
|
|
headings: list[str] = []
|
|
listed: set[str] = set()
|
|
for line in body:
|
|
stripped = line.strip()
|
|
if not stripped:
|
|
continue
|
|
if line.startswith("# "):
|
|
headings.append(line)
|
|
continue
|
|
match = self.link_pattern.match(line)
|
|
if match is not None:
|
|
listed.add(match.group("target"))
|
|
continue
|
|
if not self.allows_prose:
|
|
found.append(
|
|
IndexViolation(
|
|
stripped,
|
|
"is neither the heading nor an entry",
|
|
"index_prose_not_allowed",
|
|
)
|
|
)
|
|
|
|
if self.heading_required:
|
|
if not headings:
|
|
found.append(
|
|
IndexViolation("", "the index carries no `# ` heading", "index_heading_missing")
|
|
)
|
|
found.extend(
|
|
IndexViolation(extra, "is a second `# ` heading", "index_heading_extra")
|
|
for extra in headings[1:]
|
|
)
|
|
|
|
if expected_targets is not None:
|
|
expected = set(expected_targets)
|
|
found.extend(
|
|
IndexViolation(
|
|
target, "is in the directory but not in the index", "index_entry_missing"
|
|
)
|
|
for target in expected - listed
|
|
)
|
|
found.extend(
|
|
IndexViolation(
|
|
target, "is in the index but not in the directory", "index_entry_unexpected"
|
|
)
|
|
for target in listed - expected
|
|
)
|
|
|
|
return tuple(sorted(found, key=lambda violation: (violation.subject, violation.code)))
|
|
|
|
def _frontmatter_violations(
|
|
self, head: Mapping[str, str], *, is_root: bool
|
|
) -> list[IndexViolation]:
|
|
if not self.root_frontmatter:
|
|
return []
|
|
if not is_root:
|
|
# One finding about the block, not one per key: a nested index
|
|
# carrying frontmatter is a single structural fact.
|
|
first = next(iter(head), None)
|
|
if first is None:
|
|
return []
|
|
return [
|
|
IndexViolation(
|
|
first,
|
|
"is frontmatter on a nested index, which carries none",
|
|
"index_root_frontmatter_unexpected",
|
|
)
|
|
]
|
|
|
|
found = [
|
|
IndexViolation(key, "is pinned on the root index and absent", "index_root_key_missing")
|
|
for key in self.root_frontmatter
|
|
if key in self.root_frontmatter_required and key not in head
|
|
]
|
|
present = [key for key in head if key in self.root_frontmatter]
|
|
declared = [key for key in self.root_frontmatter if key in head]
|
|
if present != declared:
|
|
found.append(
|
|
IndexViolation(
|
|
present[0],
|
|
f"breaks the pinned key order {', '.join(self.root_frontmatter)}",
|
|
"index_root_key_order",
|
|
)
|
|
)
|
|
return found
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class BundleProfile:
|
|
"""One bundle contract: types, frontmatter, filenames, index."""
|
|
|
|
types: TypePolicy
|
|
frontmatter: FrontmatterSchema
|
|
paths: PathPolicy
|
|
index: IndexPolicy
|
|
ownership: OwnershipPolicy = field(default_factory=OwnershipPolicy)
|
|
|
|
|
|
# The ingest-spec + Phase 2 contract. Every value here was a constant in
|
|
# `manifest`, `materialize`, `inbox` or `importer` before this module existed;
|
|
# the golden suite is what proves that move changed no bytes.
|
|
#
|
|
# This profile tracks COMMONS' spec, not an upstream Google version — the two
|
|
# axes are independent, and `ownership` is where they visibly part company. Its
|
|
# `generated` is the O2 mapping because commons ratified that shape for their
|
|
# §5 layer, while the profile remains v0.1 on every axis upstream owns.
|
|
DEFAULT = BundleProfile(
|
|
types=TypePolicy(allowed=None),
|
|
frontmatter=FrontmatterSchema(
|
|
# Door A's seven keys and Door B's six, merged into one order that
|
|
# contains both as subsequences — neither door's output moves.
|
|
order=(
|
|
"type",
|
|
"title",
|
|
"source_system",
|
|
"source_query",
|
|
"source_file",
|
|
"source_sha256",
|
|
"ingested_at",
|
|
"ingest_manifest",
|
|
"generated",
|
|
),
|
|
collapsed_keys=frozenset({"source_query"}),
|
|
),
|
|
paths=PathPolicy(
|
|
concept_suffix=".md",
|
|
ingest_prefix="ingest-",
|
|
inbox_prefix="inbox-",
|
|
import_prefix="import-",
|
|
),
|
|
index=IndexPolicy(
|
|
name="index.md",
|
|
link_template="- [{label}]({target})",
|
|
link_pattern=re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
|
|
),
|
|
# V1, ratified 2026-08-02 and executed by commons 2026-08-09 (`54e0ec7`).
|
|
# The actor is the same constant `OKF_V0_2` carries, and that is commons'
|
|
# doing rather than a merge of the two profiles: ingest-spec §7 names
|
|
# `process:okf-ingest` as THE ingest actor, so any profile stating that
|
|
# spec's layer writes it. The profiles still differ everywhere else.
|
|
ownership=OwnershipPolicy(actor="process:okf-ingest"),
|
|
)
|
|
|
|
|
|
# `FRONTMATTER_KEY_ORDER` in the proving consumer's `bundle.py`: an eleven-key
|
|
# allowlist that doubles as canonical emission order. The first eight are the
|
|
# required set their operator ratified on 2026-07-25 (measured present on
|
|
# 522/522 documents); the last three are layer-specific and stay optional.
|
|
# Deriving both sets from this one tuple is what keeps them from drifting apart
|
|
# — their BUNDLE_HASH_REGISTRY gate depends on the order.
|
|
_STRICT_V1_KEY_ORDER = (
|
|
"type",
|
|
"title",
|
|
"description",
|
|
"timestamp",
|
|
"layer",
|
|
"source_tier",
|
|
"source_url",
|
|
"source_sha",
|
|
"version",
|
|
"date",
|
|
"summary",
|
|
)
|
|
|
|
# The `claude-code-llm-wiki` contract, from `docs/phase-3-split-table.md`.
|
|
#
|
|
# Deliberately stricter than the consumer's own validator: their REQUIRED_KEYS
|
|
# constant is the first four keys, which is their emit-path minimum rather than
|
|
# their contract. If they ever ship a document class that legitimately lacks
|
|
# `source_sha`, the agreed resolution is to bump this profile — they send the
|
|
# requirement before shipping the class, not after.
|
|
#
|
|
# `paths` is DEFAULT's and that is the whole truth for it: the filename
|
|
# namespaces are an ingest-spec invariant about what the doors write, not a
|
|
# consumer preference.
|
|
#
|
|
# The index shape is read out of their code rather than recalled:
|
|
# `bundle.py:498-524` renders `* [Title](target) - description` with
|
|
# subdirectories linking to their own index; `bundle.py:527-567` walks every
|
|
# level, writing frontmatter at the bundle root only and a bare `# ` heading
|
|
# below it; `validate.py:1081-1120` (gate BUNDLE_INDEX_COMPLETE, ERROR) demands
|
|
# the index and the directory match exactly in both directions.
|
|
STRICT_V1 = BundleProfile(
|
|
types=TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"})),
|
|
frontmatter=FrontmatterSchema(
|
|
order=_STRICT_V1_KEY_ORDER,
|
|
required=frozenset(_STRICT_V1_KEY_ORDER[:8]),
|
|
allowed=frozenset(_STRICT_V1_KEY_ORDER),
|
|
nullable=frozenset({"summary"}),
|
|
key_pattern=re.compile(r"^[a-z_]+$"),
|
|
),
|
|
paths=DEFAULT.paths,
|
|
index=IndexPolicy(
|
|
name="index.md",
|
|
link_template="* [{label}]({target}) - {description}",
|
|
link_pattern=re.compile(
|
|
r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)\s]+)\) - (?P<description>.+)$"
|
|
),
|
|
per_directory=True,
|
|
heading_required=True,
|
|
allows_prose=False,
|
|
entries_match_directory=True,
|
|
root_frontmatter=("okf_version", "bundle_profile", "okf_spec_commit"),
|
|
# The proving consumer demands all three, not merely permits them: their
|
|
# root index carries exactly these keys in exactly this order on every
|
|
# bundle measured (`c5141f8`). Stated explicitly so that separating
|
|
# "permitted" from "required" costs them nothing.
|
|
root_frontmatter_required=frozenset({"okf_version", "bundle_profile", "okf_spec_commit"}),
|
|
),
|
|
)
|
|
|
|
|
|
# OKF v0.2, as an ADDITIVE profile: `DEFAULT` states commons' ingest-spec §5
|
|
# layer and keeps stating it, so nothing here migrates anything. The key order
|
|
# is DEFAULT's followed by the §5 families v0.2 adds, which is also the order
|
|
# upstream's own reference bundles emit them in (`generated` before `sources`).
|
|
#
|
|
# Naming a family is not writing it. `verified`, `status` and `stale_after` are
|
|
# expressible so a caller can emit them in canonical order and so the schema can
|
|
# judge a document that carries them; Door A writes none of them, because a
|
|
# field with no reader is not written. Their structured v0.2 values — block
|
|
# lists of mappings — are beyond this library's value model until D1's reader
|
|
# lands; that is a reader gap, not an ordering one.
|
|
# §10 `Attested Computation`: the type's contract fields, in §10.2's own listing
|
|
# order. FORMAT only — this library supports writing, judging and round-tripping
|
|
# the contract, and implements no execution: upstream defers the receipt and
|
|
# verdict wire formats, so there is nothing to build a runtime against, and
|
|
# "did this run produce the value the sanctioned way" sits on the far side of
|
|
# this library's boundary in any case.
|
|
ATTESTED_COMPUTATION_TYPE = "Attested Computation"
|
|
_ATTESTED_COMPUTATION_FIELDS = ("runtime", "parameters", "computation", "executor", "attester")
|
|
|
|
|
|
_OKF_V0_2_KEY_ORDER = (
|
|
*DEFAULT.frontmatter.order,
|
|
"sources",
|
|
"verified",
|
|
"status",
|
|
"stale_after",
|
|
# §10.2's contract fields, appended as one block and internally in the order
|
|
# that section enumerates them. Appended rather than interleaved because the
|
|
# keys before them are what the doors actually emit, and because `emit`
|
|
# writes only the keys present — so naming these costs no byte in any bundle
|
|
# that carries none of them. Without the block they would still be emitted,
|
|
# in `emit`'s sorted tail, where `attester` precedes `runtime`: alphabetical
|
|
# order standing in for the contract's own.
|
|
*_ATTESTED_COMPUTATION_FIELDS,
|
|
)
|
|
|
|
# PROVISIONAL. Shipped first as a pre-release (`v0.5.0a1`) to a named pilot set
|
|
# — `portfolio-optimiser-claude`, the plugin marketplace catalog, and
|
|
# `claude-code-llm-wiki` — and this surface may change on their feedback without
|
|
# a deprecation cycle. Saying so is what buys the freedom to act on the
|
|
# feedback; discovering it later is what would make the pilot a de-facto
|
|
# release. The versioned constants are the stable binding.
|
|
#
|
|
# Two things this profile deliberately does NOT do:
|
|
#
|
|
# - **It closes nothing.** §14 forbids a conformant consumer to reject on an
|
|
# unknown `type` value or on unknown additional keys, so an allowlist or a key
|
|
# pattern here would put the profile in violation of the version it is named
|
|
# for. `type` is required and is the only one (§4, §11).
|
|
# - **It NAMES `okf_version` but never carries its value.** The value tracks the
|
|
# upstream Google version and belongs to catalog (decision E1), so a constant
|
|
# here would be this repo claiming a decision it does not own — and the one
|
|
# that would have to be chased on every upstream release. The caller supplies
|
|
# it (`materialize_bundle(..., root_frontmatter_values=...)`); this policy
|
|
# fixes only the key and its position.
|
|
#
|
|
# WHERE it goes was open until 2026-07-31 between upstream's root-index
|
|
# frontmatter block and catalog's body-line convention. Catalog verified
|
|
# upstream themselves at the pinned commit `3fcbb9f` and reported §8:509-510
|
|
# ("Index files contain no frontmatter, with one exception: a bundle-root
|
|
# `index.md` MAY carry an `okf_version` key") and §12:773-775 ("in a
|
|
# bundle-root `index.md` frontmatter block (the only place frontmatter is
|
|
# permitted in an `index.md`)"). Frontmatter it is; their own spec diverges
|
|
# from upstream here, and that divergence is theirs to resolve.
|
|
#
|
|
# Declaring it stays a MAY: none of upstream's four reference bundles carries
|
|
# the key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits),
|
|
# so omitting `root_frontmatter_values` emits no block. The JUDGE says the same
|
|
# thing since P1-F1 — the key is named here and left out of
|
|
# `root_frontmatter_required` — because for one release it did not, and a
|
|
# bundle exercising the MAY was reported as violating by the very profile that
|
|
# had emitted it correctly.
|
|
#
|
|
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
|
|
# profile cannot be read back through a guard-gated import. The guard's T2
|
|
# frontmatter grammar admits scalars and flat lists of strings, and refuses every
|
|
# route to a mapping — flow on the disallowed-indicator set, block on the
|
|
# nested-mapping check, dotted keys on the key pattern. So `generated` as the
|
|
# mapping v0.2 specifies has no expressible form through that gate at all. This
|
|
# binds what can be IMPORTED (Door C), never what we emit: Door B's
|
|
# `screen_output` does not run that parser.
|
|
OKF_V0_2 = BundleProfile(
|
|
types=TypePolicy(allowed=None),
|
|
frontmatter=FrontmatterSchema(
|
|
order=_OKF_V0_2_KEY_ORDER,
|
|
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
|
|
required=frozenset({"type"}),
|
|
# §10.2's one type-conditional rule, and the whole of it: `runtime` is
|
|
# REQUIRED for this type because it is what says how to run the
|
|
# computation and therefore what `parameters` mean. The other four
|
|
# contract fields stay optional — `computation` absent means the body
|
|
# fence IS the computation (§10.3), which is a valid concept.
|
|
required_by_type={ATTESTED_COMPUTATION_TYPE: frozenset({"runtime"})},
|
|
),
|
|
paths=DEFAULT.paths,
|
|
# DEFAULT's index in every respect but one: the root MAY carry `okf_version`
|
|
# (§8, §12). Built with `replace` rather than restated so a later change to
|
|
# the shared shape cannot drift between the two.
|
|
index=replace(DEFAULT.index, root_frontmatter=("okf_version",)),
|
|
# Byte-identical to `DEFAULT.ownership` since V1, and deliberately NOT
|
|
# written as a reference to it. The two agree by coincidence of commons
|
|
# adopting §7's actor, not by dependency: this profile states UPSTREAM's
|
|
# v0.2, so if commons ever moves their actor again, this one must not
|
|
# follow. Restating it is what keeps that independence expressible.
|
|
ownership=OwnershipPolicy(actor="process:okf-ingest"),
|
|
)
|
|
|
|
|
|
# "The latest version supported as STABLE", not the latest present in this
|
|
# module. It therefore keeps v0.1 UPSTREAM semantics for as long as v0.2 is
|
|
# provisional, and flipping it is the GA event — one auditable action rather
|
|
# than a side effect of a merge.
|
|
#
|
|
# "v0.1 semantics" is about upstream and has never covered commons' layer. V1
|
|
# moved `DEFAULT`'s stamp, so this alias's bytes moved with it, before GA and
|
|
# without the flip. That is not a leak in the alias: the two contracts are
|
|
# independent axes, and a consumer bound here tracks both by construction.
|
|
#
|
|
# The tradeoff is stated rather than hidden: an alias that moves means a consumer
|
|
# bound to it inherits upstream's breaking changes on a library upgrade. The
|
|
# versioned constants are the stable binding and are what a consumer should pin;
|
|
# this is for callers who have explicitly opted into tracking.
|
|
OKF_LATEST = DEFAULT
|