open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(okf-v0.2): D5 — the v0.2 golden fixture, with okf_version in root frontmatter

Browse source 
Placement settled by catalog's own reading of upstream at the pinned commit
3fcbb9f: SS8:509-510 and SS12:773-775 both put `okf_version` in a bundle-root
`index.md` frontmatter block, and SS12 calls it the only place frontmatter is
permitted in an index. Catalog's spec says the opposite about the same file;
that divergence is theirs against upstream, and we conform to upstream.

The value never touches a profile. `OKF_V0_2.index.root_frontmatter` names the
key; the caller supplies the value through a new keyword-only
`root_frontmatter_values` mapping. That keeps V4/V-A5 intact - `okf_version`'s
value tracks the upstream Google version and belongs to catalog (E1), so a
constant here would claim a decision we do not own and would have to be chased
on every upstream release. In the fixture the value is fixture DATA
(`okf-version.txt`), not a literal in our source.

Ordering comes from the policy, not the caller's mapping: a dict preserves
insertion order, so two callers passing the same keys would otherwise emit
different bytes. A key the policy does not name is refused fail-fast, before
any disk mutation. Omitting the argument emits no block at all - SS12 is a MAY
and none of upstream's four reference bundles declares the key.

The block is written only when the index is CREATED, so a re-run into an
existing bundle stays byte-identical (A-E5).

Raw-byte assertions rather than parsed ones, on the committed fixture as well
as on fresh runs: catalog measured that a quoted value fails their shape regex
with exit 1 and that a BOM hides the marker while still exiting 0.
`yaml.safe_load` returns "0.2" either way and strips a BOM first, so a parsed
assertion masks exactly those two defects. Asserting the frozen fixture catches
what a self-comparison cannot - regenerating from a broken emitter moves both
sides together.

A-E6 is now placement-explicit (promised catalog in 99cf987), and separates the
two byte properties: BOM-free is a property of the file, unquoted is a property
of CATALOG'S GATE and not of OKF v0.2 - upstream's own SS12 example is quoted,
so their gate rejects the spec's canonical form.

README gains the upstream-version section it was missing; CLAUDE.md gains the
mechanism behind "no profile hard-codes an upstream version": a profile names a
key, a caller owns its value.

550 -> 559 tests. test_profile_threading's `OKF_V0_2.index is DEFAULT.index`
assertion is replaced rather than deleted: object identity was a proxy for "the
shipped profiles differ in no NAME-bearing field", which is what makes the
synthetic test profile necessary, so the guard now asserts that directly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dgkSPjkLpACjMayd9R5jx
This commit is contained in:
Kjell Tore Guttormsen 2026-07-31 17:27:48 +02:00
commit 2504011010
14 changed files with 470 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

53
src/llm_ingestion_okf/materialize.py
View file

@ -13,6 +13,7 @@ import hashlib
import logging
import re
import unicodedata
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
@ -307,6 +308,36 @@ def link_in_index(
index_path.write_bytes(f"{prefix}{line}\n".encode())
def _render_root_frontmatter(values: Mapping[str, str], *, profile: BundleProfile) -> str:
"""The root index's frontmatter block (§8, §12), or "" when nothing is
declared.
The policy names the keys and fixes their order; the caller supplies the
values. Ordering by the POLICY rather than by the mapping is what keeps two
callers passing the same keys from emitting different bytes — a dict
preserves insertion order, and a golden fixture would then depend on the
order a caller happened to build its argument in.
Values are written verbatim. `okf_version` must reach catalog's shape gate
unquoted, so nothing here may add quoting; the golden fixture asserts that
on raw bytes.
"""
unknown = sorted(set(values) - set(profile.index.root_frontmatter))
if unknown:
raise MaterializationError(
f"root frontmatter key(s) {', '.join(repr(key) for key in unknown)} are not "
f"named by the {profile.index.name} policy, which pins "
f"{profile.index.root_frontmatter or '()'} — writing an unnamed key would "
"put a value in a file no reader of this contract looks at",
code="index_root_frontmatter_unexpected",
)
declared = [key for key in profile.index.root_frontmatter if key in values]
if not declared:
return ""
lines = "".join(f"{key}: {values[key]}\n" for key in declared)
return f"---\n{lines}---\n\n"
def materialize_bundle(
manifest_path: Path,
bundle_dir: Path,
@ -315,6 +346,7 @@ def materialize_bundle(
allow_network: bool = False,
http_get: HttpGet | None = None,
profile: BundleProfile = DEFAULT,
root_frontmatter_values: Mapping[str, str] | None = None,
) -> IngestResult:
"""Materialize a manifest's extractions into an OKF bundle (§5).
@ -335,10 +367,25 @@ def materialize_bundle(
gate recognises, the concept filenames, and the index; it does NOT reach
manifest type validation, which runs against `DEFAULT` (the two policies
compare equal today). `STRICT_V1` is not supported here: its index policy
sets `per_directory`, `entries_match_directory` and `root_frontmatter`,
none of which this materializer honours.
sets `per_directory` and `entries_match_directory`, neither of which this
materializer honours.
`root_frontmatter_values` supplies the values for the keys the profile's
index policy names — `okf_version` under `OKF_V0_2` (§8, §12). The split is
deliberate: the profile names the key, the caller owns the value, because
`okf_version`'s value tracks the upstream Google version and belongs to
catalog (E1). Offering a key the policy does not name is refused fail-fast,
before any disk mutation. Omitting the argument emits no block at all — §12
is a MAY, and none of upstream's reference bundles declares it.
The block is written only when the index is CREATED. A re-run into an
existing bundle leaves it untouched, which is what makes the second run
byte-identical to the first (A-E5).
"""
validate_ingested_at(ingested_at)
# Before any source access or disk mutation: a caller error here must not
# leave a partially written bundle behind.
root_frontmatter = _render_root_frontmatter(root_frontmatter_values or {}, profile=profile)
manifest_file = Path(manifest_path)
try:
raw = manifest_file.read_bytes()
@ -440,7 +487,7 @@ def materialize_bundle(
for extraction in manifest.extractions
}
if not index_path.is_file():
write_bytes(bundle, profile.index.name, manifest.bundle_summary + "\n")
write_bytes(bundle, profile.index.name, root_frontmatter + manifest.bundle_summary + "\n")
else:
# Links whose target is an ingest-owned file removed this run MUST be
# removed; all other links — curated and promoted — are preserved.

33
src/llm_ingestion_okf/profiles.py
View file

@ -22,7 +22,7 @@ from __future__ import annotations
import re
from collections.abc import Collection, Mapping, Sequence
from dataclasses import dataclass, field
from dataclasses import dataclass, field, replace
# The one layer no profile may admit (ingest-spec §3): the promotion gate is
# the only path into it. Compared case-insensitively, as both doors already do.
@ -682,13 +682,25 @@ _OKF_V0_2_KEY_ORDER = (
# unknown `type` value or on unknown additional keys, so an allowlist or a key
# pattern here would put the profile in violation of the version it is named
# for. `type` is required and is the only one (§4, §11).
# - **It does not declare `okf_version`.** The index policy is DEFAULT's, which
# binds `index.md` to the bundle root alone — upstream's shape (§8), and none
# of upstream's four reference bundles declares the version at all (§12 makes
# it a MAY). Declaring it is D5's, once, in the fixture: the value belongs to
# catalog (E1), and WHERE it goes is open between upstream's root-index
# frontmatter block and catalog's body-line convention. A profile that pinned
# one of those today would be pinning the wrong one half the time.
# - **It NAMES `okf_version` but never carries its value.** The value tracks the
# upstream Google version and belongs to catalog (decision E1), so a constant
# here would be this repo claiming a decision it does not own — and the one
# that would have to be chased on every upstream release. The caller supplies
# it (`materialize_bundle(..., root_frontmatter_values=...)`); this policy
# fixes only the key and its position.
#
# WHERE it goes was open until 2026-07-31 between upstream's root-index
# frontmatter block and catalog's body-line convention. Catalog verified
# upstream themselves at the pinned commit `3fcbb9f` and reported §8:509-510
# ("Index files contain no frontmatter, with one exception: a bundle-root
# `index.md` MAY carry an `okf_version` key") and §12:773-775 ("in a
# bundle-root `index.md` frontmatter block (the only place frontmatter is
# permitted in an `index.md`)"). Frontmatter it is; their own spec diverges
# from upstream here, and that divergence is theirs to resolve.
#
# Declaring it stays a MAY: none of upstream's four reference bundles carries
# the key at all (catalog grepped `okf/bundles` and `okf/samples`: zero hits),
# so omitting `root_frontmatter_values` emits no block.
#
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
# profile cannot be read back through a guard-gated import. The guard's T2
@ -706,7 +718,10 @@ OKF_V0_2 = BundleProfile(
required=frozenset({"type"}),
),
paths=DEFAULT.paths,
index=DEFAULT.index,
# DEFAULT's index in every respect but one: the root MAY carry `okf_version`
# (§8, §12). Built with `replace` rather than restated so a later change to
# the shared shape cannot drift between the two.
index=replace(DEFAULT.index, root_frontmatter=("okf_version",)),
ownership=OwnershipPolicy(actor="process:llm-ingestion-okf"),
)