open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(okf-v0.2): D2 — the profile, the ownership policy, and sources

Browse source 
The v0.2 profile lands additively: DEFAULT keeps stating commons' §5 layer
byte for byte (the golden suite is the proof, not the claim), and OKF_V0_2
adds the families v0.2 introduces on top of DEFAULT's key order.

Two questions the plan left open were the operator's, and both were decided
before code rather than discovered during it:

- `generated.by` is `process:llm-ingestion-okf` — plan V1's option (d), the
  §7 process actor form. The value carries no version, which is what keeps a
  byte-compared fixture stable across releases and leaves a shared
  cross-implementation fixture possible. A-E3 was written against the
  `<producer>/<version>` form and is now STALE in the pilot specification; the
  correction is owed to portfolio-optimiser-claude before they run.
- `sources[].resource` is the manifest source's locator verbatim: the file
  root, the sql connection_ref (an env-var NAME, never its value), or the http
  base_url. `credential_ref` is not a locator and is never emitted.

Ownership becomes a policy on the profile rather than a literal in the gate.
The emitter and `_is_ingest_owned` are coupled through the stamp value, so
OwnershipPolicy is where they meet and can only change together. Recognition
is one-way by decision: OKF_V0_2 owns the v0.1 stamp too, so a DEFAULT-written
bundle re-runs IN PLACE (operator, 2026-07-27), while DEFAULT still refuses a
v0.2 file rather than replacing it — V-A3's fail-safe is preserved.

An unquotable locator is refused rather than emitted. Measured with PyYAML
rather than reasoned: `[{ id: a, resource: data, backup }]` raises nothing and
parses to a mapping with a `backup` key nobody wrote, so the failure mode is a
silently wrong provenance record. Validation, not repair.

Deliberately NOT here: the `okf_version` declaration. §12 makes it a MAY and
none of upstream's four reference bundles exercises it; WHERE it goes is open
between upstream's root-index frontmatter and catalog's body-line convention,
and catalog verifies against upstream first. It is declared once, at D5.

V-A5 is extended to the new profile — the one whose NAME is the place a
version literal would look natural, and it carries none.

542 tests, mypy --strict clean.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 09:11:12 +02:00
commit c90171dad0
5 changed files with 562 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

4
src/llm_ingestion_okf/errors.py
View file

@ -96,6 +96,10 @@ class MaterializationError(IngestError):
- `ingested_at_invalid` — ingested_at is not ISO-8601 UTC with a Z suffix
- `collision_unstamped` — the §3 collision gate: a generated name is
occupied by a file without the ingest stamp
- `source_reference_unquotable` — a manifest source's id or locator
contains a character that would restructure the `sources` flow mapping
(Door A, v0.2 profiles); refused rather than emitted, because the
resulting document parses cleanly into a record no one wrote
- `inbox_slug_empty` — a dropped file's name reduces to an empty slug
under the id grammar (Door B; never an invented fallback name)
- `inbox_slug_too_long` — the generated inbox filename would exceed the

93
src/llm_ingestion_okf/materialize.py
View file

@ -23,11 +23,12 @@ from .manifest import (
FileSource,
HttpSource,
Manifest,
Source,
SqlSource,
generated_filename,
load_manifest_bytes,
)
from .profiles import DEFAULT
from .profiles import DEFAULT, BundleProfile
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
@ -128,10 +129,18 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
return frontmatter
def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# §3/§5 ownership: the ingest stamp is `generated: true` AND an
# `ingest_manifest` reference. Promoted verdict files carry neither key,
# so they can never classify as ingest-owned.
def _is_ingest_owned(path: Path, manifest_stem: str, *, profile: BundleProfile = DEFAULT) -> bool:
# §3/§5 ownership: the profile's ingest stamp AND an `ingest_manifest`
# reference. Promoted verdict files carry neither key, so they can never
# classify as ingest-owned.
#
# The stamp is the PROFILE's because it differs per profile (v0.1 writes
# `generated: true`, v0.2 a `{ by: ..., at: ... }` mapping) — and because the
# emitter and this predicate are coupled through that value. Changing the
# emitted form without the predicate is what makes the library stop
# recognising its own output, firing the collision gate on the files its own
# previous run wrote. `OwnershipPolicy` is where the two meet, so they can
# only be changed together.
#
# §10.2 per-manifest ownership: a file is THIS manifest's to replace only
# when the reference names it by stem. The stamp is `{stem}@{sha256[:16]}`;
@ -141,7 +150,8 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# keeps its own. rsplit strips the trailing `@{sha}`, so a stem that itself
# contains `@` still compares correctly.
frontmatter = parse_frontmatter(path)
if frontmatter.get("generated") != "true":
generated = frontmatter.get("generated")
if generated is None or not profile.ownership.owns(generated):
return False
reference = frontmatter.get("ingest_manifest")
if reference is None:
@ -149,8 +159,67 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
return reference.rsplit("@", 1)[0] == manifest_stem
# The characters that terminate or restructure a YAML flow mapping. `:\s`
# catches a colon that would open a nested key; a colon inside `https://host`
# does not, and stays a plain scalar.
_FLOW_UNSAFE_RE = re.compile(r"[,\[\]{}]|:\s")
def _source_locator(source: Source) -> str:
"""Where a manifest source points, per source type.
A filesystem root, the NAME of the environment variable holding the DSN, or
the base URL. `credential_ref` is not a locator and is never returned here:
a credential reference has no reader in a bundle.
"""
if isinstance(source, FileSource):
return source.root
if isinstance(source, SqlSource):
return source.connection_ref
return source.base_url
def _render_sources(source: Source) -> str:
"""§5 `sources` as an inline flow sequence of one flow mapping.
Two keys, not upstream's five: a manifest source has no `author`, no
`last_modified`, and no bundle-internal `resource` in upstream's sense, and
inventing them would be writing fields with no reader.
The flow form rather than upstream's block list, measured and chosen: a
block list read back through this library's line-oriented parser turns each
item line into a KEY nobody wrote — and `_is_ingest_owned` reads through
that same parser. The flow form also satisfies commons' §5 "all values MUST
be single-line", and §11 requires parseable YAML rather than block YAML.
Refusing an unquotable locator is the point of the check rather than a
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a
mapping with a `backup` key nobody wrote. A silently wrong provenance record
is worse than a refused run, and repairing the value by quoting it would
change bytes the operator supplied. Validation, not repair — the same
posture as the filename-length gate.
"""
locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value):
raise MaterializationError(
f"the source {label} {value!r} contains a character that would "
"restructure the `sources` flow mapping (one of `,[]{}` or a "
"colon followed by whitespace) — refusing to emit a provenance "
"record that parses cleanly into something no one wrote",
code="source_reference_unquotable",
)
return f"[{{ id: {source.id}, resource: {locator} }}]"
def _render_concept_file(
manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str
manifest: Manifest,
extraction: Extraction,
body: str,
*,
ingested_at: str,
stamp: str,
profile: BundleProfile = DEFAULT,
) -> str:
# §5 frontmatter: exactly these keys. The order and the `source_query`
# whitespace collapse are the profile's — DEFAULT states the §5 layer.
@ -161,9 +230,15 @@ def _render_concept_file(
"source_query": extraction.query,
"ingested_at": ingested_at,
"ingest_manifest": stamp,
"generated": "true",
"generated": profile.ownership.stamp(ingested_at),
}
return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{body}"
# Written only by a profile that NAMES it. `emit` sorts an unnamed key into
# the tail rather than dropping it, so building one mapping for both
# profiles would append `sources` to every v0.1 bundle — additivity is a
# property of what is constructed here, not of the emitter.
if "sources" in profile.frontmatter.order:
frontmatter["sources"] = _render_sources(manifest.source)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"
def write_bytes(bundle_dir: Path, name: str, content: str) -> Path:

126
src/llm_ingestion_okf/profiles.py
View file

@ -242,6 +242,58 @@ class FrontmatterSchema:
return " ".join(value.split()) if key in self.collapsed_keys else value
# The v0.1 ingest stamp. A literal rather than a configurable value: it is what
# every bundle this library has already written carries, and recognising it is
# what keeps those bundles re-runnable under a later profile.
_V0_1_STAMP = "true"
@dataclass(frozen=True)
class OwnershipPolicy:
"""The `generated` value this profile writes, and the values it owns back.
Ownership is the §3 collision gate's question — may this run replace the
file already sitting at a generated name? — and the answer is a profile's,
because the stamp differs per profile. v0.1 writes the literal `true`; v0.2
writes `generated: { by: <actor>, at: <ingested_at> }` (§5), where the actor
takes §7's `process:<id>` form.
`actor` is `None` for the v0.1 stamp. Where it is set it carries no version,
deliberately: the value sits inside a byte-compared golden, so a producer
version there would fire golden regression on every release without any
contract having changed, and would make a shared cross-implementation
fixture impossible by construction (plan V1(d), operator 2026-07-27).
Recognition is ONE-WAY, and both directions are decisions rather than
accidents. A v0.2 profile owns the v0.1 stamp as well, so a bundle written
under `DEFAULT` re-runs IN PLACE — the black-box promise is that an upstream
release costs a consumer a re-run and nothing more. The reverse is refused:
`DEFAULT` meeting a v0.2 file fails the run rather than replacing a file
whose shape it does not read (V-A3).
The v0.2 test is a PREFIX rather than an equality, because the value carries
`ingested_at` and therefore differs on every run by design. It works because
`parse_frontmatter` returns the whole flow mapping as one opaque string
(V-A2) — no structure this library cannot yet read is parsed here.
"""
actor: str | None = None
def stamp(self, ingested_at: str) -> str:
"""The `generated` value a run at `ingested_at` writes."""
if self.actor is None:
return _V0_1_STAMP
return f"{{ by: {self.actor}, at: {ingested_at} }}"
def owns(self, value: str) -> bool:
"""Whether a `generated` value read back marks this library's output."""
if value == _V0_1_STAMP:
return True
if self.actor is None:
return False
return value.startswith(f"{{ by: {self.actor},")
@dataclass(frozen=True)
class PathPolicy:
"""The filename namespaces the three doors write into.
@ -497,6 +549,7 @@ class BundleProfile:
frontmatter: FrontmatterSchema
paths: PathPolicy
index: IndexPolicy
ownership: OwnershipPolicy = field(default_factory=OwnershipPolicy)
# The ingest-spec v1 + Phase 2 contract, unchanged. Every value here was a
@ -595,3 +648,76 @@ STRICT_V1 = BundleProfile(
root_frontmatter=("okf_version", "bundle_profile", "okf_spec_commit"),
),
)
# OKF v0.2, as an ADDITIVE profile: `DEFAULT` states commons' ingest-spec §5
# layer and keeps stating it, so nothing here migrates anything. The key order
# is DEFAULT's followed by the §5 families v0.2 adds, which is also the order
# upstream's own reference bundles emit them in (`generated` before `sources`).
#
# Naming a family is not writing it. `verified`, `status` and `stale_after` are
# expressible so a caller can emit them in canonical order and so the schema can
# judge a document that carries them; Door A writes none of them, because a
# field with no reader is not written. Their structured v0.2 values — block
# lists of mappings — are beyond this library's value model until D1's reader
# lands; that is a reader gap, not an ordering one.
_OKF_V0_2_KEY_ORDER = (
*DEFAULT.frontmatter.order,
"sources",
"verified",
"status",
"stale_after",
)
# PROVISIONAL. Shipped first as a pre-release (`v0.5.0a1`) to a named pilot set
# — `portfolio-optimiser-claude`, the plugin marketplace catalog, and
# `claude-code-llm-wiki` — and this surface may change on their feedback without
# a deprecation cycle. Saying so is what buys the freedom to act on the
# feedback; discovering it later is what would make the pilot a de-facto
# release. The versioned constants are the stable binding.
#
# Two things this profile deliberately does NOT do:
#
# - **It closes nothing.** §14 forbids a conformant consumer to reject on an
# unknown `type` value or on unknown additional keys, so an allowlist or a key
# pattern here would put the profile in violation of the version it is named
# for. `type` is required and is the only one (§4, §11).
# - **It does not declare `okf_version`.** The index policy is DEFAULT's, which
# binds `index.md` to the bundle root alone — upstream's shape (§8), and none
# of upstream's four reference bundles declares the version at all (§12 makes
# it a MAY). Declaring it is D5's, once, in the fixture: the value belongs to
# catalog (E1), and WHERE it goes is open between upstream's root-index
# frontmatter block and catalog's body-line convention. A profile that pinned
# one of those today would be pinning the wrong one half the time.
#
# **Measured limitation (guard 0.2.0, 2026-07-26):** a bundle emitted under this
# profile cannot be read back through a guard-gated import. The guard's T2
# frontmatter grammar admits scalars and flat lists of strings, and refuses every
# route to a mapping — flow on the disallowed-indicator set, block on the
# nested-mapping check, dotted keys on the key pattern. So `generated` as the
# mapping v0.2 specifies has no expressible form through that gate at all. This
# binds what can be IMPORTED (Door C), never what we emit: Door B's
# `screen_output` does not run that parser.
OKF_V0_2 = BundleProfile(
types=TypePolicy(allowed=None),
frontmatter=FrontmatterSchema(
order=_OKF_V0_2_KEY_ORDER,
collapsed_keys=DEFAULT.frontmatter.collapsed_keys,
required=frozenset({"type"}),
),
paths=DEFAULT.paths,
index=DEFAULT.index,
ownership=OwnershipPolicy(actor="process:llm-ingestion-okf"),
)
# "The latest version supported as STABLE", not the latest present in this
# module. It therefore keeps v0.1 semantics for as long as v0.2 is provisional,
# and flipping it is the GA event — one auditable action rather than a side
# effect of a merge.
#
# The tradeoff is stated rather than hidden: an alias that moves means a consumer
# bound to it inherits upstream's breaking changes on a library upgrade. The
# versioned constants are the stable binding and are what a consumer should pin;
# this is for callers who have explicitly opted into tracking.
OKF_LATEST = DEFAULT