open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(importer): Door C projects the sender's own facets into the index

Browse source 
vegnormal-okf measured the gap on 2026-08-27: the arm reading DEFAULT's
index.md scored 0 hits of 8, the arm reading a faceted index of the same
frontmatter scored 25 of 29. Same bundle, same concepts, same model. The
DEFAULT arm did not answer wrongly, it abstained -- the metadata is in the
bundle and the index throws it away (30 974 characters over 269
requirements, 0 occurrences of any of the eight facts).

FacetPolicy and STRUCTURED_V1 already did this. They did not reach Door C.

`import_bundle` now takes a keyword-only `profile` defaulting to DEFAULT, so
every existing call site emits the bytes it always did, and `link_in_index`
takes the facets to render.

Door C PROJECTS and never DERIVES, which is the answer to the objection this
work opened with: deriving structure for a document a third party wrote would
put our inference into an index entry ABOUT their bytes, where it reads as
their claim. The concept file was already verbatim; the entry describing it
now is too. Where the sender carries `derived`, THEIR list travels unchanged,
so a reader can still see which of the sender's facts the sender inferred.

The projection asks the policy which keys to carry and never what a key
means. That is what makes it work for a meeting note as well as a numbered
norm -- nothing in it can key off a numbering scheme -- and it is why a
consumer whose concepts are named by UUID can get `title` into the index by
naming the key, with no change here.

Two things measured during the work rather than assumed:

- A value carrying the policy's own joiner cannot be rendered. Door C's
  tolerance is structural and it refuses no sender on form, so the FACET is
  dropped and the concept still merges -- reported per concept and key in
  `ImportResult.unrendered_facets`, never dropped silently.
- A faceted entry can go stale where a flat one never could: the collision
  gate refuses an updated concept, so the operator's only route is to remove
  the merged file and re-import, after which the file said `gjeldende` while
  the index still said `utkast`. A faceted entry for a present target is now
  refreshed in place instead of skipped. Unfaceted callers keep the early
  return byte for byte.

Suite 695 -> 707; ruff and mypy --strict clean.

Order 20260826T224500Z-873805419-from-vegnormal-okf.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-27 10:58:33 +02:00
commit 1f7d3502b8
3 changed files with 491 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

131
src/llm_ingestion_okf/importer.py
View file

@ -27,6 +27,7 @@ obeys the verdict it returns.
from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
@ -41,7 +42,7 @@ from .materialize import (
validate_ingested_at,
write_bytes,
)
from .profiles import DEFAULT
from .profiles import DEFAULT, BundleProfile, FacetPolicy
# An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a
@ -183,6 +184,30 @@ class UnverifiedReference:
key: str
@dataclass(frozen=True)
class UnrenderedFacet:
"""A merged concept declares a facet value the index policy cannot render.
The policy refuses a value carrying its own separator or joiner rather than
escaping it (escaping makes the line parse one way here and another way
downstream). At Door B that refuses the DOCUMENT, because the value is one
this library derived and the operator can fix the source. At Door C it must
not: this door judges no shape and refuses no sender on form — the whole
reason it writes concepts verbatim — so refusing a merge over a semicolon in
someone else's frontmatter is precisely the failure the module docstring
names.
So the FACET is dropped and the CONCEPT is merged. Dropping it silently is
the other failure: the sender made a claim our index does not show, and a
reader comparing the two would find no trace of why. Reported, like an
unverified pointer — an advisory over the merged set, never a fifth bucket.
"""
concept_path: str
key: str
reason: str
@dataclass(frozen=True)
class ImportResult:
"""Every concept's outcome, in sorted concept-path order.
@ -193,9 +218,9 @@ class ImportResult:
the guard's log body and `ingested_at` the run's explicit timestamp — the
caller persists them if their reserved-file policy says to.
`unverified_references` is not a fifth bucket and does not partition
anything: every concept it names has already merged. It is an advisory over
the merged set.
`unverified_references` and `unrendered_facets` are not fifth and sixth
buckets and partition nothing: every concept they name has already merged.
Both are advisories over the merged set.
"""
merged: tuple[MergedConcept, ...]
@ -205,9 +230,10 @@ class ImportResult:
log: str
ingested_at: str
unverified_references: tuple[UnverifiedReference, ...] = ()
unrendered_facets: tuple[UnrenderedFacet, ...] = ()
def import_slug(concept_path: str) -> str:
def import_slug(concept_path: str, *, profile: BundleProfile = DEFAULT) -> str:
"""Reduce a bundle-relative concept path to the Phase 1 id grammar.
The whole path reduces, not just its final segment: `tables/users.md` and
@ -215,7 +241,7 @@ def import_slug(concept_path: str) -> str:
collapse them onto one filename. A path that reduces to nothing fails fast
rather than being given an invented name.
"""
concept_id = concept_path[: -len(DEFAULT.paths.concept_suffix)]
concept_id = concept_path[: -len(profile.paths.concept_suffix)]
slug = reduce_to_id_grammar(concept_id)
if not slug:
raise MaterializationError(
@ -226,7 +252,7 @@ def import_slug(concept_path: str) -> str:
return slug
def import_filename(slug: str) -> str:
def import_filename(slug: str, *, profile: BundleProfile = DEFAULT) -> str:
"""The bundle filename for an imported concept.
The `import-` prefix keeps the namespace disjoint from `index.md`, Door A's
@ -234,19 +260,19 @@ def import_filename(slug: str) -> str:
grammar admits.
"""
return check_filename_length(
f"{DEFAULT.paths.import_prefix}{slug}{DEFAULT.paths.concept_suffix}",
f"{profile.paths.import_prefix}{slug}{profile.paths.concept_suffix}",
code="import_path_too_long",
)
def _index_label(concept_path: str) -> str:
def _index_label(concept_path: str, *, profile: BundleProfile = DEFAULT) -> str:
"""The concept-ID, validated as an index link label.
The guard's path gate permits brackets in a concept path; `- [label](target)`
does not. Fail-fast, never repair — the same rule Door A applies to a
manifest title and Door B to a dropped filename.
"""
label = concept_path[: -len(DEFAULT.paths.concept_suffix)]
label = concept_path[: -len(profile.paths.concept_suffix)]
if any(char in label for char in "\n\r[]"):
raise MaterializationError(
f"concept path {concept_path!r} contains '[' or ']', which would break "
@ -256,7 +282,47 @@ def _index_label(concept_path: str) -> str:
return label
def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
def _project_facets(
frontmatter: Mapping[str, str], policy: FacetPolicy
) -> tuple[dict[str, str], list[tuple[str, str]]]:
"""The sender's declared values for the keys the policy names, and the drops.
A PROJECTION and never a derivation. Every value here is one the sender
wrote in their own frontmatter; nothing is inferred from their body, their
filename, or their neighbours in the bundle. That is the ownership answer at
this door: the concept file is verbatim, and so is the index entry's account
of what the concept claims. Where the sender carries `derived`, THEIR list
travels unchanged, so a reader can still tell which of the sender's facts
the sender inferred — a distinction this library would erase by adding
inferences of its own beside them.
The loop asks the policy which keys to carry and never what a key means.
That is what makes the door work for a meeting note as well as a numbered
norm: nothing here can key off a numbering scheme, because nothing here
reads a value at all except to check the policy can render it.
Each key is rendered ALONE to find the offender, because the policy reports
a refusal for the entry rather than for one field, and dropping the whole
tail over one bad value would lose the other facts the sender declared.
"""
values: dict[str, str] = {}
dropped: list[tuple[str, str]] = []
for key in policy.keys:
value = frontmatter.get(key)
if not value:
continue
try:
policy.render({key: value})
except ValueError as exc:
dropped.append((key, str(exc)))
continue
values[key] = value
return values, dropped
def _read_bundle(
source: Path, *, profile: BundleProfile = DEFAULT
) -> tuple[dict[str, str], list[FailedConcept]]:
"""Read every concept document under `source`, keyed by POSIX-relative path.
Unreadable and undecodable concepts never reach the gate — they are per-
@ -269,7 +335,7 @@ def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
# would be a concept on APFS and not one on ext4 — the same bundle
# importing differently per platform. The guard folds case here too.
if not path.is_file() or path.suffix.lower() != DEFAULT.paths.concept_suffix:
if not path.is_file() or path.suffix.lower() != profile.paths.concept_suffix:
continue
concept_path = path.relative_to(source).as_posix()
try:
@ -296,6 +362,7 @@ def import_bundle(
origin: str,
channel: str,
gate: ImportGate,
profile: BundleProfile = DEFAULT,
) -> ImportResult:
"""Merge the accepted concepts of an external OKF bundle (Door C).
@ -307,6 +374,14 @@ def import_bundle(
INCLUDING a disposition this library does not recognise and a concept the
gate returned no verdict for, fails closed.
`profile` names the filename namespace this door writes into and the shape
of the index it maintains. It is keyword-only and defaults to `DEFAULT`, so
every existing call site emits the bytes it always did — a consumer with
branch bases built through this door is not asked to rebuild them. Where the
profile's index carries facets, each merged concept's OWN frontmatter is
projected onto them; see :func:`_project_facets` for why this door projects
and never derives.
One bad concept never aborts the run. Unreadable concepts, unusable names
and collisions are reported per concept in :class:`ImportResult` while the
rest still merge. Only three conditions fail the whole run, and all three
@ -328,12 +403,13 @@ def import_bundle(
f"source bundle directory does not exist: {source}", code="source_root_missing"
)
documents, failed = _read_bundle(source)
documents, failed = _read_bundle(source, profile=profile)
merged: list[MergedConcept] = []
quarantined: list[RefusedConcept] = []
rejected: list[RefusedConcept] = []
unverified: list[UnverifiedReference] = []
unrendered: list[UnrenderedFacet] = []
log = ""
if documents:
@ -378,8 +454,8 @@ def import_bundle(
slug_owners: dict[str, list[str]] = {}
for concept_path, text in accepted:
try:
name = import_filename(import_slug(concept_path))
_index_label(concept_path)
name = import_filename(import_slug(concept_path, profile=profile), profile=profile)
_index_label(concept_path, profile=profile)
except IngestError as exc:
failed.append(FailedConcept(concept_path=concept_path, error=exc))
continue
@ -463,11 +539,29 @@ def import_bundle(
# §6 index — the last disk mutation, and only when something merged.
if merged:
index_path = bundle / DEFAULT.index.name
index_path = bundle / profile.index.name
if not index_path.is_file():
write_bytes(bundle, DEFAULT.index.name, "")
write_bytes(bundle, profile.index.name, "")
for entry in merged:
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))
facets: dict[str, str] | None = None
if profile.index.facets is not None:
# Read from the WRITTEN file, like the pointer scan above,
# so the entry describes the concept that exists rather than
# the text that was staged.
facets, dropped = _project_facets(
parse_frontmatter(entry.path), profile.index.facets
)
unrendered.extend(
UnrenderedFacet(concept_path=entry.concept_path, key=key, reason=reason)
for key, reason in dropped
)
link_in_index(
bundle,
entry.path.name,
_index_label(entry.concept_path, profile=profile),
profile=profile,
facets=facets,
)
return ImportResult(
merged=tuple(merged),
@ -477,4 +571,5 @@ def import_bundle(
log=log,
ingested_at=ingested_at,
unverified_references=tuple(unverified),
unrendered_facets=tuple(unrendered),
)

62
src/llm_ingestion_okf/materialize.py
View file

@ -286,26 +286,78 @@ def _update_index_lines(
index_path.write_bytes("".join(updated).encode("utf-8"))
def _refresh_index_entry(
index_path: Path,
target_name: str,
label: str,
facets: Mapping[str, str],
*,
profile: BundleProfile,
) -> None:
"""Re-render the managed line for `target_name`, in place and alone.
Keyed on the policy's entry pattern and on the parsed target, never on a
substring: the index is the one file this library writes beside somebody
else's prose, so a curated line that merely MENTIONS the target has to
survive verbatim, and so does its own line ending.
"""
lines = index_path.read_bytes().decode("utf-8").splitlines(keepends=True)
updated: list[str] = []
changed = False
for line in lines:
content = line.rstrip("\r\n")
ending = line[len(content) :]
match = profile.index.entry_pattern.match(content)
if match is not None and match.group("target") == target_name:
refreshed = profile.index.render_link(label, target_name, facets=facets)
if refreshed != content:
line = refreshed + ending
changed = True
updated.append(line)
if changed:
index_path.write_bytes("".join(updated).encode("utf-8"))
def link_in_index(
bundle_dir: Path, target_name: str, label: str, *, profile: BundleProfile = DEFAULT
bundle_dir: Path,
target_name: str,
label: str,
*,
profile: BundleProfile = DEFAULT,
facets: Mapping[str, str] | None = None,
) -> None:
# §6: idempotent by target — a link whose target is already present in
# the index is never added twice.
#
# `profile` is keyword-only with a default because this function is public
# and called from all three doors (A here, B in inbox.py, C in importer.py).
# Doors B and C keep the default, which is the behaviour they already had;
# which profile THEY should own is a separate question, and answering it by
# changing this signature would have decided it silently.
# Door A and Door B's unfaceted path keep the default; which profile a
# caller should own is a separate question, and answering it by changing
# this signature would have decided it silently.
#
# `facets` also decides what "already present" MEANS, and the split is not a
# convenience. A flat entry carries a label and a target, both stable, so it
# can never disagree with the file it points at and returning early is
# exactly right — a hand-edited label survives. An entry carrying the
# concept's FACTS can go stale, and an index that contradicts the bundle it
# indexes is worse than one that says nothing: the consumer reads the index
# and stops there. So a faceted entry for a target already present is
# REFRESHED in place rather than skipped.
#
# Additive by construction: with `facets=None` nothing below the early
# return runs, so every unfaceted caller emits the bytes it always did.
index_path = safe_resolve(bundle_dir, profile.index.name)
body = index_path.read_bytes().decode("utf-8")
if f"]({target_name})" in body:
if facets is None:
return
_refresh_index_entry(index_path, target_name, label, facets, profile=profile)
return
# An empty index needs no separator: Door A always seeds its index with
# bundle_summary first, but Door B has no summary to invent, so its index
# starts empty and must not open with a blank line.
prefix = body if (body == "" or body.endswith("\n")) else body + "\n"
line = profile.index.render_link(label, target_name)
line = profile.index.render_link(label, target_name, facets=facets)
index_path.write_bytes(f"{prefix}{line}\n".encode())