open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

refactor(phase-3): the bundle contract becomes a profile object

Browse source 
Phase 3 step 1. What Phases 1 and 2 hard-coded about a valid bundle now
lives on one frozen `BundleProfile`, and `DEFAULT` states exactly the
ingest-spec v1 + Phase 2 contract. Nothing observable changes: the 425
existing tests are unmodified and green, and `git diff --stat examples/`
is empty, which is assumption C1's whole proof. Both oracles are real
rather than nominal — the golden suite compares `read_bytes()`, and Door
B pins its frontmatter block as an exact string.

Moved onto the profile, each one previously a constant with a reader:
the index name and its managed-link shape (template plus pattern, kept
honest by a round-trip test), the three filename namespaces
(`ingest-`/`inbox-`/`import-` and `.md`), the reserved `verdict` layer
(duplicated in `manifest` and `inbox` before this), the frontmatter key
order, and the `source_query` whitespace collapse.

Two details are worth naming. The DEFAULT key order spans both doors:
Door A's seven keys and Door B's six are subsequences of one nine-key
order, so a single canonical order reproduces both doors byte-for-byte.
And emission follows commons decision D1 — ordered prefix, then any
remaining keys sorted — which is the mechanism the proving consumer's
hash registry needs; under DEFAULT the tail is always empty.

`TypePolicy` refuses the reserved layer at CONSTRUCTION (assumption C3),
so a profile admitting `verdict` cannot be built, let alone passed to a
door. It reports refusals rather than raising them, because Door A
refuses with `ManifestError` and Door B with `MaterializationError` for
the same type — the wording and the stable code come from the policy,
the exception class stays each door's own.

Deliberately NOT moved, each for a stated reason: the required and
allowlisted key sets the proving consumer needs (no code reads them
until the STRICT_V1 validator exists, and a field nothing reads is a
claim nothing tests), the id grammar (a pattern the slugger derives a
separator class from, not a flat name), `NAME_MAX_BYTES` (a filesystem
fact, not a contract choice), and every disposition/origin/channel
vocabulary (guard territory, always). The profile is also not exported
from the package root yet — the optional `profile` argument on the
flows is step 3's plumbing change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 15:09:14 +02:00
commit 6f42c10608
6 changed files with 470 additions and 77 deletions
Download patch file Download diff file Expand all files Collapse all files

48
src/llm_ingestion_okf/materialize.py
View file

@ -27,19 +27,13 @@ from .manifest import (
generated_filename,
load_manifest_bytes,
)
from .profiles import DEFAULT
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
INDEX_NAME = "index.md"
_INGESTED_AT_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
# One managed index line: `- [<label>](<target>)`. Anchored full-line —
# removal keys on this exact shape for specific ingest targets, never a bare
# substring (a promoted verdict's line has the same shape but a non-ingest
# target; curated prose mentioning a target inline does not match).
_MANAGED_LINE_RE = re.compile(r"^- \[(?P<label>[^\]]*)\]\((?P<target>[^)]+)\)$")
@dataclass(frozen=True)
class IngestResult:
@ -120,24 +114,6 @@ def check_filename_length(name: str, *, code: str) -> str:
return name
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT
# must render on one line. Every other value is validated single-line at
# manifest load and emitted verbatim — validation, not repair — so
# operator-supplied bytes (e.g. a title's internal double space) survive.
return " ".join(value.split())
def _render_frontmatter(frontmatter: dict[str, str]) -> str:
# Line-oriented `key: value`, insertion order. Only `source_query` is
# collapsed; all other values pass through verbatim.
return "\n".join(
f"{key}: {_collapse_whitespace(value) if key == 'source_query' else value}"
for key, value in frontmatter.items()
)
def parse_frontmatter(path: Path) -> dict[str, str]:
lines = path.read_text(encoding="utf-8").splitlines()
if not lines or lines[0].strip() != "---":
@ -176,7 +152,8 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
def _render_concept_file(
manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str
) -> str:
# §5 frontmatter: exactly these keys, in exactly this order.
# §5 frontmatter: exactly these keys. The order and the `source_query`
# whitespace collapse are the profile's — DEFAULT states the §5 layer.
frontmatter = {
"type": extraction.okf_type,
"title": extraction.title,
@ -186,7 +163,7 @@ def _render_concept_file(
"ingest_manifest": stamp,
"generated": "true",
}
return f"---\n{_render_frontmatter(frontmatter)}\n---\n\n{body}"
return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{body}"
def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
@ -213,7 +190,7 @@ def _update_index_lines(
for line in lines:
content = line.rstrip("\r\n")
ending = line[len(content) :]
match = _MANAGED_LINE_RE.match(content)
match = DEFAULT.index.link_pattern.match(content)
if match is not None:
target = match.group("target")
if target in removed_targets:
@ -221,7 +198,7 @@ def _update_index_lines(
continue
new_label = labels_by_target.get(target)
if new_label is not None and match.group("label") != new_label:
line = f"- [{new_label}]({target})" + ending
line = DEFAULT.index.render_link(new_label, target) + ending
changed = True
updated.append(line)
if changed:
@ -231,7 +208,7 @@ def _update_index_lines(
def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# §6: idempotent by target — a link whose target is already present in
# the index is never added twice.
index_path = safe_resolve(bundle_dir, INDEX_NAME)
index_path = safe_resolve(bundle_dir, DEFAULT.index.name)
body = index_path.read_bytes().decode("utf-8")
if f"]({target_name})" in body:
return
@ -239,7 +216,8 @@ def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# bundle_summary first, but Door B has no summary to invent, so its index
# starts empty and must not open with a blank line.
prefix = body if (body == "" or body.endswith("\n")) else body + "\n"
index_path.write_bytes(f"{prefix}- [{label}]({target_name})\n".encode())
line = DEFAULT.index.render_link(label, target_name)
index_path.write_bytes(f"{prefix}{line}\n".encode())
def materialize_bundle(
@ -336,8 +314,8 @@ def materialize_bundle(
# ingest stamp are ours to replace.
owned = {
path.name
for path in sorted(bundle.glob("*.md"))
if path.name != INDEX_NAME and _is_ingest_owned(path, manifest_file.stem)
for path in sorted(bundle.glob(f"*{DEFAULT.paths.concept_suffix}"))
if path.name != DEFAULT.index.name and _is_ingest_owned(path, manifest_file.stem)
}
# §3 collision gate — BEFORE any mutation: a staged filename occupied by
# a file WITHOUT the stamp is curated content; never overwrite it.
@ -356,12 +334,12 @@ def materialize_bundle(
# §6 index generation — the last disk mutation. A fresh index gets
# bundle_summary as its body; links are appended in extraction order.
index_path = bundle / INDEX_NAME
index_path = bundle / DEFAULT.index.name
labels_by_target = {
generated_filename(extraction.id): extraction.title for extraction in manifest.extractions
}
if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, manifest.bundle_summary + "\n")
write_bytes(bundle, DEFAULT.index.name, manifest.bundle_summary + "\n")
else:
# Links whose target is an ingest-owned file removed this run MUST be
# removed; all other links — curated and promoted — are preserved.