open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(phase-3): the index policy becomes configurable, with the reader that judges it

Browse source 
`IndexPolicy` grew five judging fields and `IndexPolicy.violations`, closing the
gap a test has been pinning since `848e395`. `STRICT_V1.index` is now the proving
consumer's shape rather than DEFAULT's.

The design was settled by a conflict rather than by a preference. The convention
owner reported that an index is an authored count of a directory's children and
that a validator enumerating the directory has implemented the wrong contract.
Checked against the proving consumer before adopting it, the two turn out to be
directly opposed: gate BUNDLE_INDEX_COMPLETE (validate.py:1081-1120, ERROR) builds
its expected set by enumerating the directory and demands an exact bidirectional
match, and bundle.py:498-567 writes every index from a tree walk. Neither is
incoherent inside its own spec, so authored-versus-derived is a policy field in
both directions and a library invariant in neither.

Consequences encoded rather than documented: nothing here enumerates a directory
— the caller supplies the listing, `violations` refuses one when the profile's
index is authored and refuses to run without one when it is derived, so code
written to the wrong contract fails at the call instead of passing every test one
would think to write. Root and nested indexes are asymmetric (confirmed in both
consumers, different key sets). A per-entry description is template-level, so
`render_link` refuses both a missing description and an unwanted one.

DEFAULT keeps upstream's root-only index and judges nothing, for the same measured
reason it carries no required frontmatter key set: upstream binds `index.md` to
the bundle root alone, so a judging default would condemn conforming bundles.

25 new tests, 493 green. C1 re-proven: `git diff --stat examples/` empty.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 20:30:54 +02:00
commit 9436427520
6 changed files with 626 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

249
src/llm_ingestion_okf/profiles.py
View file

@ -20,7 +20,7 @@ are extension points, not v1 (settled with the operator at phase start).
from __future__ import annotations
import re
from collections.abc import Mapping
from collections.abc import Collection, Mapping, Sequence
from dataclasses import dataclass, field
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
@ -234,9 +234,43 @@ class PathPolicy:
import_prefix: str
def _split_frontmatter(text: str) -> tuple[dict[str, str], list[str]]:
"""The leading `---` block as ordered keys, and the body lines after it.
Line-oriented, the same shape `materialize.parse_frontmatter` reads —
duplicated rather than imported because `materialize` imports this module,
and because that one takes a path where an index reader has only text.
"""
lines = text.splitlines()
if not lines or lines[0].strip() != "---":
return {}, lines
head: dict[str, str] = {}
for offset, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
return head, lines[offset + 1 :]
key, sep, value = line.partition(":")
if sep:
head[key.strip()] = value.strip()
return head, []
@dataclass(frozen=True)
class IndexViolation:
"""One way an index file departs from the policy.
Reported rather than raised, like `SchemaViolation`. `subject` is whatever
the finding is about — a line, a link target, or a frontmatter key — and
`code` is the stable machine-readable part.
"""
subject: str
reason: str
code: str
@dataclass(frozen=True)
class IndexPolicy:
"""The index file and the shape of the links this library manages in it.
"""The index file, the shape of its entries, and where indexes must exist.
`link_template` renders a managed line and `link_pattern` recognises one.
Both are carried because §6 does both — append on write, rewrite on
@ -244,14 +278,191 @@ class IndexPolicy:
pattern is anchored to the whole line by construction: removal keys on this
exact shape, never a bare substring, so curated prose that mentions a
target inline survives verbatim.
The judging fields are read by `violations` and `required_indexes`, and
every one of them is off under DEFAULT. They exist because two consumers
disagree about this file in ways no single shape can hold:
- `per_directory` — an index at every level, or only at the bundle root.
Upstream OKF binds `index.md` to the root ALONE, so a default demanding
one per level would declare upstream-conforming bundles invalid. Policy,
never an OKF rule.
- `heading_required` / `allows_prose` — the wiki's index is a generated
heading plus entries and nothing else; the catalog's *requires*
progressive-disclosure prose. Opposite requirements, both expressible.
- `entries_match_directory` — whether the index must match the directory
exactly, in both directions. The wiki enforces this at ERROR
(`BUNDLE_INDEX_COMPLETE`); the catalog holds that an index is an
AUTHORED count of a directory's children and that a validator
enumerating the directory has implemented the wrong contract. Neither
posture is baked in, and this library never enumerates anything: the
caller supplies the listing, and a policy that does not judge it refuses
to be handed one at all — so code written to the wrong contract fails at
the call rather than passing every test one would think to write.
- `root_frontmatter` — the ordered keys the ROOT index carries, where
nested indexes carry none. Confirmed independently in both consumers, so
the asymmetry is shape rather than one repo's preference.
"""
name: str
link_template: str
link_pattern: re.Pattern[str]
per_directory: bool = False
heading_required: bool = False
allows_prose: bool = True
entries_match_directory: bool = False
root_frontmatter: tuple[str, ...] = ()
def render_link(self, label: str, target: str) -> str:
return self.link_template.format(label=label, target=target)
def __post_init__(self) -> None:
if self.requires_description and "description" not in self.link_pattern.groupindex:
raise ValueError(
"link_template names {description} but link_pattern has no "
"'description' group — the pair would round-trip in one "
"direction only"
)
@property
def requires_description(self) -> bool:
"""Whether an entry carries a description alongside label and target."""
return "{description}" in self.link_template
def render_link(self, label: str, target: str, description: str | None = None) -> str:
if self.requires_description and description is None:
raise ValueError(
"this index policy's entries carry a description; rendering "
"without one emits a half-written entry that parses as nothing"
)
if not self.requires_description and description is not None:
raise ValueError(
"this index policy's entries carry no description; the value "
"offered would be dropped silently"
)
return self.link_template.format(label=label, target=target, description=description)
def required_indexes(self, directories: Sequence[str]) -> tuple[str, ...]:
"""The index paths this policy requires, given the caller's directories.
Bundle-relative, with `""` for the root. The directories are an
argument precisely so that nothing here reaches the filesystem.
"""
if not self.per_directory:
return (self.name,)
return tuple(sorted(f"{d}/{self.name}" if d else self.name for d in directories))
def violations(
self,
text: str,
*,
is_root: bool = False,
expected_targets: Collection[str] | None = None,
) -> tuple[IndexViolation, ...]:
"""Every way `text` departs from this policy, deterministically ordered.
`expected_targets` is the directory's contents as the CALLER sees them,
required exactly when `entries_match_directory` is set and refused
otherwise. Sorted by `(subject, code)` so two runs over the same file
agree.
"""
if self.entries_match_directory and expected_targets is None:
raise ValueError(
"this index policy judges the index against the directory, so "
"a listing is required — skipping the check silently would "
"pass a gate whose input never arrived"
)
if not self.entries_match_directory and expected_targets is not None:
raise ValueError(
"this index policy's index is authored rather than derived "
"from the directory, so a listing has nothing to judge"
)
found: list[IndexViolation] = []
head, body = _split_frontmatter(text)
found.extend(self._frontmatter_violations(head, is_root=is_root))
headings: list[str] = []
listed: set[str] = set()
for line in body:
stripped = line.strip()
if not stripped:
continue
if line.startswith("# "):
headings.append(line)
continue
match = self.link_pattern.match(line)
if match is not None:
listed.add(match.group("target"))
continue
if not self.allows_prose:
found.append(
IndexViolation(
stripped,
"is neither the heading nor an entry",
"index_prose_not_allowed",
)
)
if self.heading_required:
if not headings:
found.append(
IndexViolation("", "the index carries no `# ` heading", "index_heading_missing")
)
found.extend(
IndexViolation(extra, "is a second `# ` heading", "index_heading_extra")
for extra in headings[1:]
)
if expected_targets is not None:
expected = set(expected_targets)
found.extend(
IndexViolation(
target, "is in the directory but not in the index", "index_entry_missing"
)
for target in expected - listed
)
found.extend(
IndexViolation(
target, "is in the index but not in the directory", "index_entry_unexpected"
)
for target in listed - expected
)
return tuple(sorted(found, key=lambda violation: (violation.subject, violation.code)))
def _frontmatter_violations(
self, head: Mapping[str, str], *, is_root: bool
) -> list[IndexViolation]:
if not self.root_frontmatter:
return []
if not is_root:
# One finding about the block, not one per key: a nested index
# carrying frontmatter is a single structural fact.
first = next(iter(head), None)
if first is None:
return []
return [
IndexViolation(
first,
"is frontmatter on a nested index, which carries none",
"index_root_frontmatter_unexpected",
)
]
found = [
IndexViolation(key, "is pinned on the root index and absent", "index_root_key_missing")
for key in self.root_frontmatter
if key not in head
]
present = [key for key in head if key in self.root_frontmatter]
declared = [key for key in self.root_frontmatter if key in head]
if present != declared:
found.append(
IndexViolation(
present[0],
f"breaks the pinned key order {', '.join(self.root_frontmatter)}",
"index_root_key_order",
)
)
return found
@dataclass(frozen=True)
@ -327,13 +538,16 @@ _STRICT_V1_KEY_ORDER = (
# `source_sha`, the agreed resolution is to bump this profile — they send the
# requirement before shipping the class, not after.
#
# `paths` and `index` are DEFAULT's, and only `paths` is the whole truth: the
# filename namespaces are an ingest-spec invariant about what the doors write,
# not a consumer preference. The consumer's index shape — one index per
# directory, `* [Title](link) - description` entries, a root index carrying
# okf_version/bundle_profile/okf_spec_commit — is NOT expressible by IndexPolicy
# yet and lands with the reader that judges it. Nothing consumes a profile's
# index policy today, so the gap is pinned by a test rather than left to drift.
# `paths` is DEFAULT's and that is the whole truth for it: the filename
# namespaces are an ingest-spec invariant about what the doors write, not a
# consumer preference.
#
# The index shape is read out of their code rather than recalled:
# `bundle.py:498-524` renders `* [Title](target) - description` with
# subdirectories linking to their own index; `bundle.py:527-567` walks every
# level, writing frontmatter at the bundle root only and a bare `# ` heading
# below it; `validate.py:1081-1120` (gate BUNDLE_INDEX_COMPLETE, ERROR) demands
# the index and the directory match exactly in both directions.
STRICT_V1 = BundleProfile(
types=TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"})),
frontmatter=FrontmatterSchema(
@ -344,5 +558,16 @@ STRICT_V1 = BundleProfile(
key_pattern=re.compile(r"^[a-z_]+$"),
),
paths=DEFAULT.paths,
index=DEFAULT.index,
index=IndexPolicy(
name="index.md",
link_template="* [{label}]({target}) - {description}",
link_pattern=re.compile(
r"^\* \[(?P<label>[^\]]*)\]\((?P<target>[^)\s]+)\) - (?P<description>.+)$"
),
per_directory=True,
heading_required=True,
allows_prose=False,
entries_match_directory=True,
root_frontmatter=("okf_version", "bundle_profile", "okf_spec_commit"),
),
)