open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

refactor(phase-3): the bundle contract becomes a profile object

Browse source 
Phase 3 step 1. What Phases 1 and 2 hard-coded about a valid bundle now
lives on one frozen `BundleProfile`, and `DEFAULT` states exactly the
ingest-spec v1 + Phase 2 contract. Nothing observable changes: the 425
existing tests are unmodified and green, and `git diff --stat examples/`
is empty, which is assumption C1's whole proof. Both oracles are real
rather than nominal — the golden suite compares `read_bytes()`, and Door
B pins its frontmatter block as an exact string.

Moved onto the profile, each one previously a constant with a reader:
the index name and its managed-link shape (template plus pattern, kept
honest by a round-trip test), the three filename namespaces
(`ingest-`/`inbox-`/`import-` and `.md`), the reserved `verdict` layer
(duplicated in `manifest` and `inbox` before this), the frontmatter key
order, and the `source_query` whitespace collapse.

Two details are worth naming. The DEFAULT key order spans both doors:
Door A's seven keys and Door B's six are subsequences of one nine-key
order, so a single canonical order reproduces both doors byte-for-byte.
And emission follows commons decision D1 — ordered prefix, then any
remaining keys sorted — which is the mechanism the proving consumer's
hash registry needs; under DEFAULT the tail is always empty.

`TypePolicy` refuses the reserved layer at CONSTRUCTION (assumption C3),
so a profile admitting `verdict` cannot be built, let alone passed to a
door. It reports refusals rather than raising them, because Door A
refuses with `ManifestError` and Door B with `MaterializationError` for
the same type — the wording and the stable code come from the policy,
the exception class stays each door's own.

Deliberately NOT moved, each for a stated reason: the required and
allowlisted key sets the proving consumer needs (no code reads them
until the STRICT_V1 validator exists, and a field nothing reads is a
claim nothing tests), the id grammar (a pattern the slugger derives a
separator class from, not a flat name), `NAME_MAX_BYTES` (a filesystem
fact, not a contract choice), and every disposition/origin/channel
vocabulary (guard territory, always). The profile is also not exported
from the package root yet — the optional `profile` argument on the
flows is step 3's plumbing change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 15:09:14 +02:00
commit 6f42c10608
6 changed files with 470 additions and 77 deletions
Download patch file Download diff file Expand all files Collapse all files

21
src/llm_ingestion_okf/importer.py
View file

@ -34,20 +34,18 @@ from typing import Protocol
from .errors import IngestError, MaterializationError, SourceError from .errors import IngestError, MaterializationError, SourceError
from .extract import decode_text from .extract import decode_text
from .materialize import ( from .materialize import (
INDEX_NAME,
check_filename_length, check_filename_length,
link_in_index, link_in_index,
reduce_to_id_grammar, reduce_to_id_grammar,
validate_ingested_at, validate_ingested_at,
write_bytes, write_bytes,
) )
from .profiles import DEFAULT
# An OKF concept is a `.md` document by definition — the guard's path gate # An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a # rejects anything else outright — so nothing else in the source tree is a
# concept, and nothing else is this door's to merge. # concept, and nothing else is this door's to merge. The suffix and the
_CONCEPT_SUFFIX = ".md" # `import-` namespace are the profile's (`DEFAULT.paths`).
_FILENAME_PREFIX = "import-"
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition` # The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
# is a `str, Enum`, so the value is the stable thing to compare against across # is a `str, Enum`, so the value is the stable thing to compare against across
@ -176,7 +174,7 @@ def import_slug(concept_path: str) -> str:
collapse them onto one filename. A path that reduces to nothing fails fast collapse them onto one filename. A path that reduces to nothing fails fast
rather than being given an invented name. rather than being given an invented name.
""" """
concept_id = concept_path[: -len(_CONCEPT_SUFFIX)] concept_id = concept_path[: -len(DEFAULT.paths.concept_suffix)]
slug = reduce_to_id_grammar(concept_id) slug = reduce_to_id_grammar(concept_id)
if not slug: if not slug:
raise MaterializationError( raise MaterializationError(
@ -195,7 +193,8 @@ def import_filename(slug: str) -> str:
grammar admits. grammar admits.
""" """
return check_filename_length( return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_CONCEPT_SUFFIX}", code="import_path_too_long" f"{DEFAULT.paths.import_prefix}{slug}{DEFAULT.paths.concept_suffix}",
code="import_path_too_long",
) )
@ -206,7 +205,7 @@ def _index_label(concept_path: str) -> str:
does not. Fail-fast, never repair — the same rule Door A applies to a does not. Fail-fast, never repair — the same rule Door A applies to a
manifest title and Door B to a dropped filename. manifest title and Door B to a dropped filename.
""" """
label = concept_path[: -len(_CONCEPT_SUFFIX)] label = concept_path[: -len(DEFAULT.paths.concept_suffix)]
if any(char in label for char in "\n\r[]"): if any(char in label for char in "\n\r[]"):
raise MaterializationError( raise MaterializationError(
f"concept path {concept_path!r} contains '[' or ']', which would break " f"concept path {concept_path!r} contains '[' or ']', which would break "
@ -229,7 +228,7 @@ def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD` # glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
# would be a concept on APFS and not one on ext4 — the same bundle # would be a concept on APFS and not one on ext4 — the same bundle
# importing differently per platform. The guard folds case here too. # importing differently per platform. The guard folds case here too.
if not path.is_file() or path.suffix.lower() != _CONCEPT_SUFFIX: if not path.is_file() or path.suffix.lower() != DEFAULT.paths.concept_suffix:
continue continue
concept_path = path.relative_to(source).as_posix() concept_path = path.relative_to(source).as_posix()
try: try:
@ -409,9 +408,9 @@ def import_bundle(
# §6 index — the last disk mutation, and only when something merged. # §6 index — the last disk mutation, and only when something merged.
if merged: if merged:
index_path = bundle / INDEX_NAME index_path = bundle / DEFAULT.index.name
if not index_path.is_file(): if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, "") write_bytes(bundle, DEFAULT.index.name, "")
for entry in merged: for entry in merged:
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path)) link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))

42
src/llm_ingestion_okf/inbox.py
View file

@ -28,7 +28,6 @@ from pathlib import Path
from .errors import IngestError, MaterializationError, SourceError from .errors import IngestError, MaterializationError, SourceError
from .extract import extract_text from .extract import extract_text
from .materialize import ( from .materialize import (
INDEX_NAME,
check_filename_length, check_filename_length,
link_in_index, link_in_index,
parse_frontmatter, parse_frontmatter,
@ -36,11 +35,7 @@ from .materialize import (
validate_ingested_at, validate_ingested_at,
write_bytes, write_bytes,
) )
from .profiles import DEFAULT
_RESERVED_OKF_TYPE = "verdict"
_FILENAME_PREFIX = "inbox-"
_FILENAME_SUFFIX = ".md"
def inbox_slug(source_filename: str) -> str: def inbox_slug(source_filename: str) -> str:
@ -73,7 +68,8 @@ def inbox_filename(slug: str) -> str:
not give it. not give it.
""" """
return check_filename_length( return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}", code="inbox_slug_too_long" f"{DEFAULT.paths.inbox_prefix}{slug}{DEFAULT.paths.concept_suffix}",
code="inbox_slug_too_long",
) )
@ -104,12 +100,11 @@ def render_inbox_concept(
validate_ingested_at(ingested_at) validate_ingested_at(ingested_at)
# The verdict layer is RESERVED: the promotion gate is the only path into # The verdict layer is RESERVED: the promotion gate is the only path into
# it, at this door exactly as at Door A's manifest validation. # it, at this door exactly as at Door A's manifest validation — the same
if okf_type.lower() == _RESERVED_OKF_TYPE: # profile decides, each door raises its own typed error.
raise MaterializationError( rejection = DEFAULT.types.rejection(okf_type)
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)", if rejection is not None:
code="okf_type_reserved", raise MaterializationError(f"okf_type {rejection.reason}", code=rejection.code)
)
# The title is rendered verbatim into `- [title](target)` and into # The title is rendered verbatim into `- [title](target)` and into
# line-oriented frontmatter — met by fail-fast validation, never repair. # line-oriented frontmatter — met by fail-fast validation, never repair.
if any(char in title for char in "\n\r[]"): if any(char in title for char in "\n\r[]"):
@ -131,8 +126,7 @@ def render_inbox_concept(
"ingested_at": ingested_at, "ingested_at": ingested_at,
"generated": "true", "generated": "true",
} }
rendered = "\n".join(f"{key}: {value}" for key, value in frontmatter.items()) return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{_normalize_body(text)}"
return f"---\n{rendered}\n---\n\n{_normalize_body(text)}"
# --- the guard seam ------------------------------------------------------- # --- the guard seam -------------------------------------------------------
@ -248,11 +242,9 @@ def process_inbox(
a reserved `okf_type`, and a missing inbox directory. a reserved `okf_type`, and a missing inbox directory.
""" """
validate_ingested_at(ingested_at) validate_ingested_at(ingested_at)
if okf_type.lower() == _RESERVED_OKF_TYPE: run_rejection = DEFAULT.types.rejection(okf_type)
raise MaterializationError( if run_rejection is not None:
f"okf_type must not be {_RESERVED_OKF_TYPE!r} (reserved layer)", raise MaterializationError(f"okf_type {run_rejection.reason}", code=run_rejection.code)
code="okf_type_reserved",
)
inbox = Path(inbox_dir) inbox = Path(inbox_dir)
if not inbox.is_dir(): if not inbox.is_dir():
raise SourceError(f"inbox directory does not exist: {inbox}", code="source_root_missing") raise SourceError(f"inbox directory does not exist: {inbox}", code="source_root_missing")
@ -299,7 +291,11 @@ def process_inbox(
# pre-existing curated content by a later file's check. # pre-existing curated content by a later file's check.
bundle = Path(bundle_dir) bundle = Path(bundle_dir)
pre_existing = ( pre_existing = (
{path.name for path in bundle.glob("*.md") if path.name != INDEX_NAME} {
path.name
for path in bundle.glob(f"*{DEFAULT.paths.concept_suffix}")
if path.name != DEFAULT.index.name
}
if bundle.is_dir() if bundle.is_dir()
else set() else set()
) )
@ -369,9 +365,9 @@ def process_inbox(
# §6 index — the last disk mutation, and only when something was written. # §6 index — the last disk mutation, and only when something was written.
if persisted: if persisted:
index_path = bundle / INDEX_NAME index_path = bundle / DEFAULT.index.name
if not index_path.is_file(): if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, "") write_bytes(bundle, DEFAULT.index.name, "")
for entry in persisted: for entry in persisted:
link_in_index( link_in_index(
bundle, entry.path.name, unicodedata.normalize("NFC", Path(entry.source_file).stem) bundle, entry.path.name, unicodedata.normalize("NFC", Path(entry.source_file).stem)

16
src/llm_ingestion_okf/manifest.py
View file

@ -15,11 +15,10 @@ from pathlib import Path
from typing import Any, Union from typing import Any, Union
from .errors import ManifestError from .errors import ManifestError
from .profiles import DEFAULT
_ID_PATTERN = re.compile(r"[a-z0-9][a-z0-9-]*\Z") _ID_PATTERN = re.compile(r"[a-z0-9][a-z0-9-]*\Z")
_RESERVED_OKF_TYPE = "verdict"
@dataclass(frozen=True) @dataclass(frozen=True)
class FileSource: class FileSource:
@ -66,7 +65,7 @@ def generated_filename(extraction_id: str) -> str:
The `ingest-` prefix keeps the namespace disjoint from `index.md` and The `ingest-` prefix keeps the namespace disjoint from `index.md` and
`promoted-verdict-*` (spec §3) for every id the §4 grammar admits. `promoted-verdict-*` (spec §3) for every id the §4 grammar admits.
""" """
return f"ingest-{extraction_id}.md" return f"{DEFAULT.paths.ingest_prefix}{extraction_id}{DEFAULT.paths.concept_suffix}"
def load_manifest(path: Path) -> Manifest: def load_manifest(path: Path) -> Manifest:
@ -193,11 +192,12 @@ def _validate_extraction(data: object, index: int) -> Extraction:
okf_type = _require_str(obj["okf_type"], f"{label}.okf_type") okf_type = _require_str(obj["okf_type"], f"{label}.okf_type")
# The verdict layer is RESERVED (spec §3): the promotion gate is the only # The verdict layer is RESERVED (spec §3): the promotion gate is the only
# path into it — enforced here, fail-fast, before any source call. # path into it — enforced here, fail-fast, before any source call. The
if okf_type.lower() == _RESERVED_OKF_TYPE: # profile decides which types a bundle admits; the door raises its own
raise ManifestError( # typed error, since Door B refuses the same type as a MaterializationError.
f"{label}.okf_type must not be 'verdict' (reserved layer)", code="okf_type_reserved" rejection = DEFAULT.types.rejection(okf_type)
) if rejection is not None:
raise ManifestError(f"{label}.okf_type {rejection.reason}", code=rejection.code)
max_rows = obj["max_rows"] max_rows = obj["max_rows"]
if not _is_int(max_rows) or max_rows < 1: if not _is_int(max_rows) or max_rows < 1:

48
src/llm_ingestion_okf/materialize.py
View file

@ -27,19 +27,13 @@ from .manifest import (
generated_filename, generated_filename,
load_manifest_bytes, load_manifest_bytes,
) )
from .profiles import DEFAULT
from .render import render_fenced_block, render_table from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__) _LOGGER = logging.getLogger(__name__)
INDEX_NAME = "index.md"
_INGESTED_AT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$") _INGESTED_AT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
# One managed index line: `- [<label>](<target>)`. Anchored full-line —
# removal keys on this exact shape for specific ingest targets, never a bare
# substring (a promoted verdict's line has the same shape but a non-ingest
# target; curated prose mentioning a target inline does not match).
_MANAGED_LINE_RE = re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$")
@dataclass(frozen=True) @dataclass(frozen=True)
class IngestResult: class IngestResult:
@ -120,24 +114,6 @@ def check_filename_length(name: str, *, code: str) -> str:
return name return name
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line at
# manifest load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space) survive.
return " ".join(value.split())
def _render_frontmatter(frontmatter: dict[str, str]) -> str:
# Line-oriented `key: value`, insertion order. Only `source_query` is
# collapsed; all other values pass through verbatim.
return "\n".join(
f"{key}: {_collapse_whitespace(value) if key == 'source_query' else value}"
for key, value in frontmatter.items()
)
def parse_frontmatter(path: Path) -> dict[str, str]: def parse_frontmatter(path: Path) -> dict[str, str]:
lines = path.read_text(encoding="utf-8").splitlines() lines = path.read_text(encoding="utf-8").splitlines()
if not lines or lines[0].strip() != "---": if not lines or lines[0].strip() != "---":
@ -176,7 +152,8 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
def _render_concept_file( def _render_concept_file(
manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str
) -> str: ) -> str:
# §5 frontmatter: exactly these keys, in exactly this order. # §5 frontmatter: exactly these keys. The order and the `source_query`
# whitespace collapse are the profile's — DEFAULT states the §5 layer.
frontmatter = { frontmatter = {
"type": extraction.okf_type, "type": extraction.okf_type,
"title": extraction.title, "title": extraction.title,
@ -186,7 +163,7 @@ def _render_concept_file(
"ingest_manifest": stamp, "ingest_manifest": stamp,
"generated": "true", "generated": "true",
} }
return f"---\n{_render_frontmatter(frontmatter)}\n---\n\n{body}" return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{body}"
def write_bytes(bundle_dir: Path, name: str, content: str) -> Path: def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
@ -213,7 +190,7 @@ def _update_index_lines(
for line in lines: for line in lines:
content = line.rstrip("\r\n") content = line.rstrip("\r\n")
ending = line[len(content) :] ending = line[len(content) :]
match = _MANAGED_LINE_RE.match(content) match = DEFAULT.index.link_pattern.match(content)
if match is not None: if match is not None:
target = match.group("target") target = match.group("target")
if target in removed_targets: if target in removed_targets:
@ -221,7 +198,7 @@ def _update_index_lines(
continue continue
new_label = labels_by_target.get(target) new_label = labels_by_target.get(target)
if new_label is not None and match.group("label") != new_label: if new_label is not None and match.group("label") != new_label:
line = f"- [{new_label}]({target})" + ending line = DEFAULT.index.render_link(new_label, target) + ending
changed = True changed = True
updated.append(line) updated.append(line)
if changed: if changed:
@ -231,7 +208,7 @@ def _update_index_lines(
def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None: def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# §6: idempotent by target — a link whose target is already present in # §6: idempotent by target — a link whose target is already present in
# the index is never added twice. # the index is never added twice.
index_path = safe_resolve(bundle_dir, INDEX_NAME) index_path = safe_resolve(bundle_dir, DEFAULT.index.name)
body = index_path.read_bytes().decode("utf-8") body = index_path.read_bytes().decode("utf-8")
if f"]({target_name})" in body: if f"]({target_name})" in body:
return return
@ -239,7 +216,8 @@ def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# bundle_summary first, but Door B has no summary to invent, so its index # bundle_summary first, but Door B has no summary to invent, so its index
# starts empty and must not open with a blank line. # starts empty and must not open with a blank line.
prefix = body if (body == "" or body.endswith("\n")) else body + "\n" prefix = body if (body == "" or body.endswith("\n")) else body + "\n"
index_path.write_bytes(f"{prefix}- [{label}]({target_name})\n".encode()) line = DEFAULT.index.render_link(label, target_name)
index_path.write_bytes(f"{prefix}{line}\n".encode())
def materialize_bundle( def materialize_bundle(
@ -336,8 +314,8 @@ def materialize_bundle(
# ingest stamp are ours to replace. # ingest stamp are ours to replace.
owned = { owned = {
path.name path.name
for path in sorted(bundle.glob("*.md")) for path in sorted(bundle.glob(f"*{DEFAULT.paths.concept_suffix}"))
if path.name != INDEX_NAME and _is_ingest_owned(path, manifest_file.stem) if path.name != DEFAULT.index.name and _is_ingest_owned(path, manifest_file.stem)
} }
# §3 collision gate — BEFORE any mutation: a staged filename occupied by # §3 collision gate — BEFORE any mutation: a staged filename occupied by
# a file WITHOUT the stamp is curated content; never overwrite it. # a file WITHOUT the stamp is curated content; never overwrite it.
@ -356,12 +334,12 @@ def materialize_bundle(
# §6 index generation — the last disk mutation. A fresh index gets # §6 index generation — the last disk mutation. A fresh index gets
# bundle_summary as its body; links are appended in extraction order. # bundle_summary as its body; links are appended in extraction order.
index_path = bundle / INDEX_NAME index_path = bundle / DEFAULT.index.name
labels_by_target = { labels_by_target = {
generated_filename(extraction.id): extraction.title for extraction in manifest.extractions generated_filename(extraction.id): extraction.title for extraction in manifest.extractions
} }
if not index_path.is_file(): if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, manifest.bundle_summary + "\n") write_bytes(bundle, DEFAULT.index.name, manifest.bundle_summary + "\n")
else: else:
# Links whose target is an ingest-owned file removed this run MUST be # Links whose target is an ingest-owned file removed this run MUST be
# removed; all other links — curated and promoted — are preserved. # removed; all other links — curated and promoted — are preserved.

203
src/llm_ingestion_okf/profiles.py Normal file
View file

@ -0,0 +1,203 @@
"""The bundle contract as configuration (Phase 3).
What a valid bundle looks like — which concept types exist, which frontmatter
keys are emitted and in which order, which filename namespaces the doors own,
and what an index line looks like — is a profile, not a set of constants
scattered across the doors. `DEFAULT` is exactly the ingest-spec v1 + Phase 2
contract, so nothing observable changes for a caller that never mentions a
profile; the golden fixtures are the byte-level proof.
Two things deliberately do NOT live here. Security is the guard's, always: no
disposition, origin or channel vocabulary belongs on a profile. And the
reserved `verdict` layer is a spec invariant rather than profile config — it is
refused at construction, so a profile admitting it cannot be built, let alone
passed to a door.
Profiles are constructed in code. Config-file loading and inheritance chains
are extension points, not v1 (settled with the operator at phase start).
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from dataclasses import dataclass, field
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
# the only path into it. Compared case-insensitively, as both doors already do.
RESERVED_OKF_TYPE = "verdict"
@dataclass(frozen=True)
class TypeRejection:
"""Why a profile refuses an `okf_type`, for the door to frame and raise.
The policy does not raise: Door A refuses with `ManifestError` and Door B
with `MaterializationError`, so the refusal has to be reported rather than
thrown. `reason` completes the sentence "<label>okf_type ..." and `code` is
the stable `IngestError.code` the caller asserts on.
"""
reason: str
code: str
@dataclass(frozen=True)
class TypePolicy:
"""Which `okf_type` values a bundle admits.
`allowed` is `None` for an open set — Phase 1/2 accept any type the
operator names — or a closed enum. Either way the reserved layer is
excluded, and a closed set that names it fails at construction.
"""
allowed: frozenset[str] | None = None
def __post_init__(self) -> None:
if self.allowed is None:
return
named = sorted(value for value in self.allowed if value.lower() == RESERVED_OKF_TYPE)
if named:
raise ValueError(
f"a profile must not admit the reserved {RESERVED_OKF_TYPE!r} layer "
f"(got {', '.join(repr(value) for value in named)}) — the promotion "
"gate is the only path into it (ingest-spec §3)"
)
def rejection(self, okf_type: str) -> TypeRejection | None:
"""The refusal for `okf_type`, or `None` when the profile admits it."""
if okf_type.lower() == RESERVED_OKF_TYPE:
return TypeRejection(
reason=f"must not be {RESERVED_OKF_TYPE!r} (reserved layer)",
code="okf_type_reserved",
)
if self.allowed is not None and okf_type not in self.allowed:
return TypeRejection(
reason=f"must be one of {', '.join(sorted(self.allowed))}, got {okf_type!r}",
code="okf_type_not_allowed",
)
return None
@dataclass(frozen=True)
class FrontmatterSchema:
"""The frontmatter key namespace and how it is emitted.
`order` is the canonical emission order and `collapsed_keys` names the keys
whose whitespace runs collapse to single spaces on the way out. The DEFAULT
order spans both doors' key sets: Door A emits seven of these keys and Door
B six, and each door's subset comes out in exactly the order it wrote by
hand in Phases 1 and 2.
The required/allowlisted key sets the proving consumer needs (see
`docs/phase-3-split-table.md`) are not here yet: no code reads them until
the STRICT_V1 validator exists, and a field nothing reads is a claim
nothing tests.
"""
order: tuple[str, ...]
collapsed_keys: frozenset[str] = field(default_factory=frozenset)
def emit(self, values: Mapping[str, str]) -> str:
"""Render `values` as line-oriented `key: value`, one line per key.
Commons decision D1: the keys named in `order` come first, in that
order, followed by any remaining keys SORTED. Ordering the tail rather
than trusting insertion order is what makes a regeneration over the
same data byte-identical.
Returns the lines only — the caller owns the `---` fences.
"""
named = [key for key in self.order if key in values]
tail = sorted(key for key in values if key not in self.order)
return "\n".join(f"{key}: {self._render(key, values[key])}" for key in [*named, *tail])
def _render(self, key: str, value: str) -> str:
# §5 mandates whitespace-run collapse for `source_query` only
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line
# at load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space)
# survive.
return " ".join(value.split()) if key in self.collapsed_keys else value
@dataclass(frozen=True)
class PathPolicy:
"""The filename namespaces the three doors write into.
Each prefix keeps its door's generated names disjoint from `index.md`, from
the other doors, and from `promoted-verdict-*`, for every id the grammar
admits. The id grammar itself stays in `materialize`: it is a pattern the
slugger derives a separator class from, not a name to configure.
"""
concept_suffix: str
ingest_prefix: str
inbox_prefix: str
import_prefix: str
@dataclass(frozen=True)
class IndexPolicy:
"""The index file and the shape of the links this library manages in it.
`link_template` renders a managed line and `link_pattern` recognises one.
Both are carried because §6 does both — append on write, rewrite on
maintenance — and a round-trip test is what keeps the pair honest. The
pattern is anchored to the whole line by construction: removal keys on this
exact shape, never a bare substring, so curated prose that mentions a
target inline survives verbatim.
"""
name: str
link_template: str
link_pattern: re.Pattern[str]
def render_link(self, label: str, target: str) -> str:
return self.link_template.format(label=label, target=target)
@dataclass(frozen=True)
class BundleProfile:
"""One bundle contract: types, frontmatter, filenames, index."""
types: TypePolicy
frontmatter: FrontmatterSchema
paths: PathPolicy
index: IndexPolicy
# The ingest-spec v1 + Phase 2 contract, unchanged. Every value here was a
# constant in `manifest`, `materialize`, `inbox` or `importer` before this
# module existed; the golden suite is what proves the move changed no bytes.
DEFAULT = BundleProfile(
types=TypePolicy(allowed=None),
frontmatter=FrontmatterSchema(
# Door A's seven keys and Door B's six, merged into one order that
# contains both as subsequences — neither door's output moves.
order=(
"type",
"title",
"source_system",
"source_query",
"source_file",
"source_sha256",
"ingested_at",
"ingest_manifest",
"generated",
),
collapsed_keys=frozenset({"source_query"}),
),
paths=PathPolicy(
concept_suffix=".md",
ingest_prefix="ingest-",
inbox_prefix="inbox-",
import_prefix="import-",
),
index=IndexPolicy(
name="index.md",
link_template="- [{label}]({target})",
link_pattern=re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$"),
),
)

217
tests/test_profile.py Normal file
View file

@ -0,0 +1,217 @@
"""The bundle contract as configuration (Phase 3, step 1).
What Phases 1 and 2 hard-coded about a valid bundle — the index name and its
managed-link shape, the three filename namespaces, the reserved layer, and the
frontmatter key order — becomes a profile object here. `DEFAULT` must express
exactly the existing contract: the Phase 1/2 suite and the golden fixtures are
the byte-level proof (assumption C1), and these tests pin the profile's own
behavior.
"""
from __future__ import annotations
import dataclasses
import pytest
from llm_ingestion_okf.profiles import (
DEFAULT,
BundleProfile,
FrontmatterSchema,
TypePolicy,
)
def test_the_profile_and_every_policy_on_it_are_frozen() -> None:
"""A profile is a value, not a mutable registry.
A consumer holds DEFAULT while a run is in flight; a profile that could be
mutated mid-run would make the contract a moving target and the golden
determinism claim unprovable.
"""
for value in (DEFAULT, DEFAULT.types, DEFAULT.frontmatter, DEFAULT.paths, DEFAULT.index):
with pytest.raises(dataclasses.FrozenInstanceError):
value.name = "mutated" # type: ignore[misc]
def test_default_profile_pins_the_phase_1_2_names() -> None:
"""The names the two doors already write, now stated in one place."""
assert DEFAULT.index.name == "index.md"
assert DEFAULT.paths.concept_suffix == ".md"
assert DEFAULT.paths.ingest_prefix == "ingest-"
assert DEFAULT.paths.inbox_prefix == "inbox-"
assert DEFAULT.paths.import_prefix == "import-"
# --- the reserved layer (assumption C3) -----------------------------------
@pytest.mark.parametrize("reserved", ["verdict", "Verdict", "VERDICT"])
def test_no_type_policy_can_admit_the_reserved_verdict_layer(reserved: str) -> None:
"""C3: the verdict reservation is a spec invariant, not profile config.
Refused at CONSTRUCTION, not at use: a profile that admits `verdict` must
not exist at all, or the promotion gate stops being the only path into that
layer the moment someone passes the profile to a door.
"""
with pytest.raises(ValueError, match="verdict"):
TypePolicy(allowed=frozenset({"Concept", reserved}))
def test_a_closed_type_policy_without_the_reserved_layer_constructs() -> None:
policy = TypePolicy(allowed=frozenset({"Concept", "Guide", "Reference", "Release"}))
assert policy.rejection("Concept") is None
def test_default_type_policy_is_open_apart_from_the_reserved_layer() -> None:
"""Phase 1/2 accept any okf_type but `verdict`; DEFAULT must not narrow that."""
assert DEFAULT.types.allowed is None
for admitted in ("Concept", "note", "whatever-the-operator-wants"):
assert DEFAULT.types.rejection(admitted) is None
for reserved in ("verdict", "Verdict", "VERDICT"):
assert DEFAULT.types.rejection(reserved) is not None
def test_the_reserved_rejection_reproduces_both_doors_message_and_code() -> None:
"""The doors keep their own typed errors; the policy supplies the wording.
Door A raises ManifestError and Door B MaterializationError for the same
refusal, so the policy cannot raise — it hands back the reason and the
stable code, and each door frames it. These are the exact strings the
Phase 1/2 tests already assert on.
"""
rejection = DEFAULT.types.rejection("verdict")
assert rejection is not None
assert rejection.code == "okf_type_reserved"
assert f"okf_type {rejection.reason}" == "okf_type must not be 'verdict' (reserved layer)"
def test_a_closed_type_policy_rejects_an_off_enum_type_distinctly() -> None:
"""An off-enum refusal is not the reserved-layer refusal wearing its code."""
policy = TypePolicy(allowed=frozenset({"Concept", "Release"}))
rejection = policy.rejection("Guide")
assert rejection is not None
assert rejection.code == "okf_type_not_allowed"
assert "Concept, Release" in rejection.reason
# --- index policy ---------------------------------------------------------
def test_index_link_template_and_pattern_describe_the_same_shape() -> None:
"""Two fields that must agree, proven to agree rather than trusted to.
Rendering and parsing managed index links are separate operations (§6
appends, and maintenance rewrites), so the profile carries both a template
and a pattern. A round-trip is what keeps them from drifting apart.
"""
for label, target in [("Title", "ingest-x.md"), ("", "inbox-y.md")]:
line = DEFAULT.index.render_link(label, target)
match = DEFAULT.index.link_pattern.match(line)
assert match is not None, f"pattern does not match its own template output: {line!r}"
assert match.group("label") == label
assert match.group("target") == target
def test_default_index_link_is_the_phase_1_shape() -> None:
assert DEFAULT.index.render_link("Sales", "ingest-sales.md") == "- [Sales](ingest-sales.md)"
def test_index_link_pattern_is_anchored_to_the_whole_line() -> None:
"""§6 removal keys on the exact shape, never a bare substring — a curated
line that merely mentions a link must survive verbatim."""
assert DEFAULT.index.link_pattern.match("see - [Sales](ingest-sales.md) below") is None
# --- frontmatter emission -------------------------------------------------
def test_emission_is_the_ordered_prefix_then_the_sorted_tail() -> None:
"""Commons decision D1, carried over intact.
Keys named in the profile's order come first in that order; anything else
follows sorted. That is what makes a regeneration over the same data
byte-identical, which is what the proving consumer's hash registry needs.
"""
schema = FrontmatterSchema(order=("type", "title"))
emitted = schema.emit({"zeta": "3", "title": "T", "alpha": "1", "type": "Concept"})
assert emitted == "type: Concept\ntitle: T\nalpha: 1\nzeta: 3"
def test_emission_skips_ordered_keys_that_are_absent() -> None:
schema = FrontmatterSchema(order=("type", "title", "generated"))
assert schema.emit({"generated": "true", "type": "Concept"}) == "type: Concept\ngenerated: true"
def test_default_emission_reproduces_door_a_key_order() -> None:
"""Byte-identity with `_render_concept_file`'s §5 frontmatter."""
emitted = DEFAULT.frontmatter.emit(
{
"type": "Concept",
"title": "Sales",
"source_system": "crm",
"source_query": "sales.csv",
"ingested_at": "2026-07-03T12:00:00Z",
"ingest_manifest": "m@0123456789abcdef",
"generated": "true",
}
)
assert emitted.splitlines() == [
"type: Concept",
"title: Sales",
"source_system: crm",
"source_query: sales.csv",
"ingested_at: 2026-07-03T12:00:00Z",
"ingest_manifest: m@0123456789abcdef",
"generated: true",
]
def test_default_emission_reproduces_door_b_key_order() -> None:
"""Byte-identity with `render_inbox_concept`'s provenance layer.
Door B's six keys are a different subset of the same schema, and the
canonical order must reproduce BOTH doors — a single order that reordered
either one would change bytes already frozen in the fixtures.
"""
emitted = DEFAULT.frontmatter.emit(
{
"type": "Concept",
"title": "Notes",
"source_file": "notes.md",
"source_sha256": "abc",
"ingested_at": "2026-07-03T12:00:00Z",
"generated": "true",
}
)
assert emitted.splitlines() == [
"type: Concept",
"title: Notes",
"source_file: notes.md",
"source_sha256: abc",
"ingested_at: 2026-07-03T12:00:00Z",
"generated: true",
]
def test_source_query_is_the_only_collapsed_key() -> None:
"""§5 collapses whitespace runs for `source_query` alone (a multi-line
SELECT must render on one line); every other value is validated single-line
at load and emitted verbatim, so an operator's internal double space
survives."""
assert DEFAULT.frontmatter.collapsed_keys == frozenset({"source_query"})
emitted = DEFAULT.frontmatter.emit(
{"title": "two spaces", "source_query": "SELECT a\n FROM t"}
)
assert emitted == "title: two spaces\nsource_query: SELECT a FROM t"
def test_a_profile_is_assembled_from_its_policies() -> None:
"""The profile is the four policies and nothing else — no behavior branches
outside what the object expresses."""
assert {field.name for field in dataclasses.fields(BundleProfile)} == {
"types",
"frontmatter",
"paths",
"index",
}