open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(okf-v0.2): D2 — the profile, the ownership policy, and sources

Browse source 
The v0.2 profile lands additively: DEFAULT keeps stating commons' §5 layer
byte for byte (the golden suite is the proof, not the claim), and OKF_V0_2
adds the families v0.2 introduces on top of DEFAULT's key order.

Two questions the plan left open were the operator's, and both were decided
before code rather than discovered during it:

- `generated.by` is `process:llm-ingestion-okf` — plan V1's option (d), the
  §7 process actor form. The value carries no version, which is what keeps a
  byte-compared fixture stable across releases and leaves a shared
  cross-implementation fixture possible. A-E3 was written against the
  `<producer>/<version>` form and is now STALE in the pilot specification; the
  correction is owed to portfolio-optimiser-claude before they run.
- `sources[].resource` is the manifest source's locator verbatim: the file
  root, the sql connection_ref (an env-var NAME, never its value), or the http
  base_url. `credential_ref` is not a locator and is never emitted.

Ownership becomes a policy on the profile rather than a literal in the gate.
The emitter and `_is_ingest_owned` are coupled through the stamp value, so
OwnershipPolicy is where they meet and can only change together. Recognition
is one-way by decision: OKF_V0_2 owns the v0.1 stamp too, so a DEFAULT-written
bundle re-runs IN PLACE (operator, 2026-07-27), while DEFAULT still refuses a
v0.2 file rather than replacing it — V-A3's fail-safe is preserved.

An unquotable locator is refused rather than emitted. Measured with PyYAML
rather than reasoned: `[{ id: a, resource: data, backup }]` raises nothing and
parses to a mapping with a `backup` key nobody wrote, so the failure mode is a
silently wrong provenance record. Validation, not repair.

Deliberately NOT here: the `okf_version` declaration. §12 makes it a MAY and
none of upstream's four reference bundles exercises it; WHERE it goes is open
between upstream's root-index frontmatter and catalog's body-line convention,
and catalog verifies against upstream first. It is declared once, at D5.

V-A5 is extended to the new profile — the one whose NAME is the place a
version literal would look natural, and it carries none.

542 tests, mypy --strict clean.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 09:11:12 +02:00
commit c90171dad0
5 changed files with 562 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

93
src/llm_ingestion_okf/materialize.py
View file

@ -23,11 +23,12 @@ from .manifest import (
FileSource,
HttpSource,
Manifest,
Source,
SqlSource,
generated_filename,
load_manifest_bytes,
)
from .profiles import DEFAULT
from .profiles import DEFAULT, BundleProfile
from .render import render_fenced_block, render_table
_LOGGER = logging.getLogger(__name__)
@ -128,10 +129,18 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
return frontmatter
def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# §3/§5 ownership: the ingest stamp is `generated: true` AND an
# `ingest_manifest` reference. Promoted verdict files carry neither key,
# so they can never classify as ingest-owned.
def _is_ingest_owned(path: Path, manifest_stem: str, *, profile: BundleProfile = DEFAULT) -> bool:
# §3/§5 ownership: the profile's ingest stamp AND an `ingest_manifest`
# reference. Promoted verdict files carry neither key, so they can never
# classify as ingest-owned.
#
# The stamp is the PROFILE's because it differs per profile (v0.1 writes
# `generated: true`, v0.2 a `{ by: ..., at: ... }` mapping) — and because the
# emitter and this predicate are coupled through that value. Changing the
# emitted form without the predicate is what makes the library stop
# recognising its own output, firing the collision gate on the files its own
# previous run wrote. `OwnershipPolicy` is where the two meet, so they can
# only be changed together.
#
# §10.2 per-manifest ownership: a file is THIS manifest's to replace only
# when the reference names it by stem. The stamp is `{stem}@{sha256[:16]}`;
@ -141,7 +150,8 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
# keeps its own. rsplit strips the trailing `@{sha}`, so a stem that itself
# contains `@` still compares correctly.
frontmatter = parse_frontmatter(path)
if frontmatter.get("generated") != "true":
generated = frontmatter.get("generated")
if generated is None or not profile.ownership.owns(generated):
return False
reference = frontmatter.get("ingest_manifest")
if reference is None:
@ -149,8 +159,67 @@ def _is_ingest_owned(path: Path, manifest_stem: str) -> bool:
return reference.rsplit("@", 1)[0] == manifest_stem
# The characters that terminate or restructure a YAML flow mapping. `:\s`
# catches a colon that would open a nested key; a colon inside `https://host`
# does not, and stays a plain scalar.
_FLOW_UNSAFE_RE = re.compile(r"[,\[\]{}]|:\s")
def _source_locator(source: Source) -> str:
"""Where a manifest source points, per source type.
A filesystem root, the NAME of the environment variable holding the DSN, or
the base URL. `credential_ref` is not a locator and is never returned here:
a credential reference has no reader in a bundle.
"""
if isinstance(source, FileSource):
return source.root
if isinstance(source, SqlSource):
return source.connection_ref
return source.base_url
def _render_sources(source: Source) -> str:
"""§5 `sources` as an inline flow sequence of one flow mapping.
Two keys, not upstream's five: a manifest source has no `author`, no
`last_modified`, and no bundle-internal `resource` in upstream's sense, and
inventing them would be writing fields with no reader.
The flow form rather than upstream's block list, measured and chosen: a
block list read back through this library's line-oriented parser turns each
item line into a KEY nobody wrote — and `_is_ingest_owned` reads through
that same parser. The flow form also satisfies commons' §5 "all values MUST
be single-line", and §11 requires parseable YAML rather than block YAML.
Refusing an unquotable locator is the point of the check rather than a
nicety: `[{ id: x, resource: data, backup }]` is not a parse ERROR, it is a
mapping with a `backup` key nobody wrote. A silently wrong provenance record
is worse than a refused run, and repairing the value by quoting it would
change bytes the operator supplied. Validation, not repair — the same
posture as the filename-length gate.
"""
locator = _source_locator(source)
for label, value in (("id", source.id), ("resource", locator)):
if _FLOW_UNSAFE_RE.search(value):
raise MaterializationError(
f"the source {label} {value!r} contains a character that would "
"restructure the `sources` flow mapping (one of `,[]{}` or a "
"colon followed by whitespace) — refusing to emit a provenance "
"record that parses cleanly into something no one wrote",
code="source_reference_unquotable",
)
return f"[{{ id: {source.id}, resource: {locator} }}]"
def _render_concept_file(
manifest: Manifest, extraction: Extraction, body: str, *, ingested_at: str, stamp: str
manifest: Manifest,
extraction: Extraction,
body: str,
*,
ingested_at: str,
stamp: str,
profile: BundleProfile = DEFAULT,
) -> str:
# §5 frontmatter: exactly these keys. The order and the `source_query`
# whitespace collapse are the profile's — DEFAULT states the §5 layer.
@ -161,9 +230,15 @@ def _render_concept_file(
"source_query": extraction.query,
"ingested_at": ingested_at,
"ingest_manifest": stamp,
"generated": "true",
"generated": profile.ownership.stamp(ingested_at),
}
return f"---\n{DEFAULT.frontmatter.emit(frontmatter)}\n---\n\n{body}"
# Written only by a profile that NAMES it. `emit` sorts an unnamed key into
# the tail rather than dropping it, so building one mapping for both
# profiles would append `sources` to every v0.1 bundle — additivity is a
# property of what is constructed here, not of the emitter.
if "sources" in profile.frontmatter.order:
frontmatter["sources"] = _render_sources(manifest.source)
return f"---\n{profile.frontmatter.emit(frontmatter)}\n---\n\n{body}"
def write_bytes(bundle_dir: Path, name: str, content: str) -> Path: