open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(profiles): materialize_bundle takes a keyword-only profile (req 6)

Browse source 
`OKF_V0_2` landed in D2 but was unreachable from outside: no door took a
profile. This threads one through, keyword-only behind the `*` the signature
already carried, so every three-positional call site stays source-compatible —
which is what po-claude asked for, and what makes additivity a property of the
signature rather than something a consumer measures.

Nine sites, not the ~6 STATE claimed. The load-bearing one is the call at
materialize.py:378: the CONTENT phase has accepted `profile` since D2, but the
call site never passed one, so A-E3/A-E4/A-E5 were all unreachable. The other
eight are the disk phase (ownership glob, index name, index maintenance,
concept filenames) plus `generated_filename` in manifest.py.

`link_in_index` is public and called from all three doors, so it gets
`*, profile=DEFAULT` rather than having the lookup moved to the call site:
doors B and C keep exactly the behaviour they had, and which profile THEY own
stays an open question instead of being decided silently by a signature change.

Byte-neutrality is proven, not asserted: `OKF_V0_2.paths is DEFAULT.paths` and
`.index is DEFAULT.index`, and the golden suite is green. That identity is also
why six of the nine sites cannot be proven reachable by any shipped-profile
test — no assertion distinguishes two names for one object. A synthetic
test-only profile renaming the index and the concept files closes that gap, so
a site left on `DEFAULT` fails by name rather than passing quietly.

Scope stated rather than glossed: the profile does NOT reach manifest type
validation (`manifest.py:198` still reads `DEFAULT.types`; measured equal to
`OKF_V0_2.types`, so nothing is hidden today), and `STRICT_V1` is not supported
here — its index policy sets three judging fields the materializer does not
honour. Both are named in the docstring.

No `okf_version` anywhere: that lands once, at D5, when the §12 placement
question closes.

550 tests pass (was 542).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tf2BbC8uSRVU4ApQ9NL7QR
This commit is contained in:
Kjell Tore Guttormsen 2026-07-31 15:37:17 +02:00
commit ed08ac15e9
3 changed files with 270 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

61
src/llm_ingestion_okf/materialize.py
View file

@ -251,7 +251,11 @@ def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:
def _update_index_lines(
index_path: Path, removed_targets: set[str], labels_by_target: dict[str, str]
index_path: Path,
removed_targets: set[str],
labels_by_target: dict[str, str],
*,
profile: BundleProfile = DEFAULT,
) -> None:
"""§6 maintenance on an EXISTING index: drop managed lines whose target is
an ingest file removed in this run; refresh in place a managed label that
@ -265,7 +269,7 @@ def _update_index_lines(
for line in lines:
content = line.rstrip("\r\n")
ending = line[len(content) :]
match = DEFAULT.index.link_pattern.match(content)
match = profile.index.link_pattern.match(content)
if match is not None:
target = match.group("target")
if target in removed_targets:
@ -273,17 +277,25 @@ def _update_index_lines(
continue
new_label = labels_by_target.get(target)
if new_label is not None and match.group("label") != new_label:
line = DEFAULT.index.render_link(new_label, target) + ending
line = profile.index.render_link(new_label, target) + ending
changed = True
updated.append(line)
if changed:
index_path.write_bytes("".join(updated).encode("utf-8"))
def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
def link_in_index(
bundle_dir: Path, target_name: str, label: str, *, profile: BundleProfile = DEFAULT
) -> None:
# §6: idempotent by target — a link whose target is already present in
# the index is never added twice.
index_path = safe_resolve(bundle_dir, DEFAULT.index.name)
#
# `profile` is keyword-only with a default because this function is public
# and called from all three doors (A here, B in inbox.py, C in importer.py).
# Doors B and C keep the default, which is the behaviour they already had;
# which profile THEY should own is a separate question, and answering it by
# changing this signature would have decided it silently.
index_path = safe_resolve(bundle_dir, profile.index.name)
body = index_path.read_bytes().decode("utf-8")
if f"]({target_name})" in body:
return
@ -291,7 +303,7 @@ def link_in_index(bundle_dir: Path, target_name: str, label: str) -> None:
# bundle_summary first, but Door B has no summary to invent, so its index
# starts empty and must not open with a blank line.
prefix = body if (body == "" or body.endswith("\n")) else body + "\n"
line = DEFAULT.index.render_link(label, target_name)
line = profile.index.render_link(label, target_name)
index_path.write_bytes(f"{prefix}{line}\n".encode())
@ -302,6 +314,7 @@ def materialize_bundle(
*,
allow_network: bool = False,
http_get: HttpGet | None = None,
profile: BundleProfile = DEFAULT,
) -> IngestResult:
"""Materialize a manifest's extractions into an OKF bundle (§5).
@ -313,6 +326,17 @@ def materialize_bundle(
seam (default urllib_get, the only socket path; ignored for `file`/`sql`)
so tests run socket-free (§11). Source calls are logged per §8 (which
source, when, row count) — never cell contents, never secrets.
`profile` selects the bundle contract: `DEFAULT` (commons' ingest-spec §5)
or `OKF_V0_2`. It is keyword-only behind the `*` the signature already
carried, so every three-positional call site stays source-compatible —
support for a new upstream version is additive, never a migration. The
profile governs the emitted frontmatter, the ownership stamp the collision
gate recognises, the concept filenames, and the index; it does NOT reach
manifest type validation, which runs against `DEFAULT` (the two policies
compare equal today). `STRICT_V1` is not supported here: its index policy
sets `per_directory`, `entries_match_directory` and `root_frontmatter`,
none of which this materializer honours.
"""
validate_ingested_at(ingested_at)
manifest_file = Path(manifest_path)
@ -376,9 +400,9 @@ def materialize_bundle(
"source call: source=%s ingested_at=%s rows=%d", source.id, ingested_at, row_count
)
content = _render_concept_file(
manifest, extraction, body, ingested_at=ingested_at, stamp=stamp
manifest, extraction, body, ingested_at=ingested_at, stamp=stamp, profile=profile
)
staged.append((generated_filename(extraction.id), content))
staged.append((generated_filename(extraction.id, profile=profile), content))
# Disk phase.
bundle = Path(bundle_dir)
@ -389,8 +413,9 @@ def materialize_bundle(
# ingest stamp are ours to replace.
owned = {
path.name
for path in sorted(bundle.glob(f"*{DEFAULT.paths.concept_suffix}"))
if path.name != DEFAULT.index.name and _is_ingest_owned(path, manifest_file.stem)
for path in sorted(bundle.glob(f"*{profile.paths.concept_suffix}"))
if path.name != profile.index.name
and _is_ingest_owned(path, manifest_file.stem, profile=profile)
}
# §3 collision gate — BEFORE any mutation: a staged filename occupied by
# a file WITHOUT the stamp is curated content; never overwrite it.
@ -409,16 +434,22 @@ def materialize_bundle(
# §6 index generation — the last disk mutation. A fresh index gets
# bundle_summary as its body; links are appended in extraction order.
index_path = bundle / DEFAULT.index.name
index_path = bundle / profile.index.name
labels_by_target = {
generated_filename(extraction.id): extraction.title for extraction in manifest.extractions
generated_filename(extraction.id, profile=profile): extraction.title
for extraction in manifest.extractions
}
if not index_path.is_file():
write_bytes(bundle, DEFAULT.index.name, manifest.bundle_summary + "\n")
write_bytes(bundle, profile.index.name, manifest.bundle_summary + "\n")
else:
# Links whose target is an ingest-owned file removed this run MUST be
# removed; all other links — curated and promoted — are preserved.
_update_index_lines(index_path, owned - staged_names, labels_by_target)
_update_index_lines(index_path, owned - staged_names, labels_by_target, profile=profile)
for extraction in manifest.extractions:
link_in_index(bundle, generated_filename(extraction.id), extraction.title)
link_in_index(
bundle,
generated_filename(extraction.id, profile=profile),
extraction.title,
profile=profile,
)
return IngestResult(written=written, stamp=stamp)