open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(import): Door C flow against an injected import gate (Phase 2 step 5)

Browse source 
Reads an external OKF bundle as {bundle-relative path -> document text},
hands it WHOLE to an injected gate over the guard's okf.import_bundle (a
bundle-level call: it resolves the cross-link graph across concepts), and
merges only concepts clearing the non-blocking floor. Same injection pattern
as Door B, so the core stays dependency-free while the CI channel for the
real guard is settled.

Two constraints shaped the design and are pinned by tests:

- A merged concept is written VERBATIM. Stamping provenance into it would
  require round-tripping its frontmatter through this library's line-oriented
  parser, which cannot represent the block lists the guard's parser accepts --
  silent data loss -- and would persist bytes the guard never screened.
- Ownership is therefore proven by content identity: identical bytes at the
  target name are a no-op re-merge (re-import of an unchanged bundle is
  idempotent), and anything else at the name is refused. Curated content and
  an updated concept are refused alike; refusing is what never destroys.

The floor is fail-closed beyond the plan's "no error" wording: an error, an
unrecognised disposition, and a concept the gate returned no verdict for are
all refusals. quarantine_review stays its own bucket, as at Door B.
origin/channel are validated against the guard's pinned vocabulary -- it
derives trust from origin by enum identity, so an unrecognised string would be
silently downgraded rather than caught.

Three primitives promoted for reuse rather than duplicated:
reduce_to_id_grammar and check_filename_length to materialize.py, and
extract.decode_text. Door C slugs the WHOLE concept path, so tables/users.md
and views/users.md stay distinct. Concept discovery folds case explicitly
rather than globbing *.md, whose case-sensitivity follows the filesystem and
would import the same bundle differently on APFS and ext4.

README's "what is gated today" section corrected: it claimed nothing is gated,
which is no longer true, but the honest statement is narrower than "the doors
are gated" -- the library cannot verify that an injected adapter is a real
guard, and a permissive stub is believed.

405 tests green; ruff, ruff format and mypy --strict clean.
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 06:57:25 +02:00
commit f10fc60de2
11 changed files with 1250 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

26
src/llm_ingestion_okf/__init__.py
View file

@ -20,6 +20,14 @@ Door B (bundle inbox) public surface: process_inbox plus its result types.
Its persist gate is INJECTED -- the caller supplies a Gate adapter over
llm-ingestion-guard and process_inbox obeys the verdict; the library imports
no guard function itself and makes no security decision of its own.
Door C (external bundle import) public surface: import_bundle plus its result
types. Its gate is injected the same way, over the guard's okf.import_bundle:
the caller declares origin and channel at the door, the gate assesses every
concept, and only concepts clearing the non-blocking floor are merged. Merged
concepts are written verbatim -- ownership is proven by content identity
rather than by a stamp, so an occupied name is only ever re-used when the
bytes already there are identical.
"""
from .errors import (
@ -41,6 +49,16 @@ from .inbox import (
PersistedFile,
process_inbox,
)
from .importer import (
BundleDecision,
FailedConcept,
ImportDecision,
ImportGate,
ImportResult,
MergedConcept,
RefusedConcept,
import_bundle,
)
from .manifest import (
Extraction,
FileSource,
@ -55,25 +73,33 @@ __version__ = "0.3.2"
__all__ = [
"BlockedFile",
"BundleDecision",
"Extraction",
"ExtractionError",
"FailedConcept",
"FailedFile",
"FileSource",
"Gate",
"GateDecision",
"HttpSource",
"ImportDecision",
"ImportGate",
"ImportResult",
"InboxResult",
"IngestError",
"IngestResult",
"Manifest",
"ManifestError",
"MaterializationError",
"MergedConcept",
"NetworkGateError",
"PersistedFile",
"RefusedConcept",
"RenderError",
"SourceError",
"SqlSource",
"extract_text",
"import_bundle",
"load_manifest",
"materialize_bundle",
"process_inbox",

13
src/llm_ingestion_okf/errors.py
View file

@ -110,6 +110,19 @@ class MaterializationError(IngestError):
would inject frontmatter lines
- `okf_type_reserved` — an inbox concept claims the reserved 'verdict'
layer (the same reservation ManifestError enforces at Door A)
- `import_path_empty` — an external concept path reduces to an empty slug
under the id grammar (Door C; never an invented fallback name)
- `import_path_too_long` — the generated import filename would exceed the
255-byte filesystem limit (Door C; never a truncated name)
- `import_slug_collision` — two concepts in one external bundle reduce to
one generated filename (Door C); both are refused rather than letting
iteration order decide which one survives
- `import_label_invalid` — an external concept path contains `[`/`]`, which
would break its index link (the guard's path gate permits them)
- `import_provenance_invalid` — an `origin`/`channel` outside the guard's
pinned vocabulary (Door C); refused rather than carried, because the
guard derives trust from `origin` by enum identity and an unrecognised
value would be silently downgraded
"""

10
src/llm_ingestion_okf/extract.py
View file

@ -33,7 +33,7 @@ _OPTIONAL_EXTENSIONS = frozenset({".pdf", ".docx", ".xlsx"})
_SKIP_TAGS = frozenset({"script", "style"})
def _decode(data: bytes) -> str:
def decode_text(data: bytes) -> str:
"""Decode file bytes as UTF-8 (BOM-stripping), typed on failure.
utf-8-sig so a byte-order mark never leaks into the first character
@ -50,12 +50,12 @@ def _decode(data: bytes) -> str:
def _extract_passthrough(data: bytes) -> str:
"""`md`/`txt`: the decoded text verbatim."""
return _decode(data)
return decode_text(data)
def _extract_csv(data: bytes) -> str:
"""`csv`: parse with the stdlib reader, render the Phase 1 markdown table."""
reader = csv.reader(io.StringIO(_decode(data)))
reader = csv.reader(io.StringIO(decode_text(data)))
header = next(reader, None)
if header is None:
raise ExtractionError("CSV has no header row", code="extractor_empty_csv")
@ -65,7 +65,7 @@ def _extract_csv(data: bytes) -> str:
def _extract_json(data: bytes) -> str:
"""`json`: the decoded text verbatim inside a fenced block (Phase 1 renderer)."""
return render_fenced_block(_decode(data))
return render_fenced_block(decode_text(data))
class _HTMLTextExtractor(HTMLParser):
@ -105,7 +105,7 @@ class _HTMLTextExtractor(HTMLParser):
def _extract_html(data: bytes) -> str:
"""`html`/`htm`: text via `html.parser`, script/style stripped (spec B3)."""
parser = _HTMLTextExtractor()
parser.feed(_decode(data))
parser.feed(decode_text(data))
parser.close()
return parser.text()

425
src/llm_ingestion_okf/importer.py Normal file
View file

@ -0,0 +1,425 @@
"""Door C: external bundle import — read, gate per concept, merge the accepted.
A third-party OKF bundle is read as `{bundle-relative path -> document text}`,
handed WHOLE to the guard's `okf.import_bundle` (a bundle-level call: it
resolves the cross-link graph across concepts, so gating them one at a time
would throw that half of the gate away), and only concepts whose verdict clears
the non-blocking floor are merged.
A merged concept is written VERBATIM — exactly the text the gate saw. Nothing
of this library's is merged into its frontmatter, and that is a correctness
constraint, not a preference: the guard's frontmatter parser accepts block
lists, which this library's line-oriented :func:`parse_frontmatter` cannot
round-trip, so re-rendering a concept would silently drop data the sender
supplied. It would also persist bytes the guard never screened.
That leaves ownership to be proven by content identity instead of by a stamp:
a target name held by byte-identical content is a no-op re-merge (so re-import
of an unchanged bundle is idempotent), and a target name held by anything else
is refused. Curated content and an UPDATED external concept are refused alike —
the library cannot tell them apart without a marker it has no safe place to
write, and refusing is the answer that never destroys. A configurable
reserved-file/frontmatter policy is Phase 3's; this is the v1 floor.
No security decision is taken here: the gate is injected, and this module only
obeys the verdict it returns.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol
from .errors import IngestError, MaterializationError, SourceError
from .extract import decode_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
reduce_to_id_grammar,
validate_ingested_at,
write_bytes,
)
# An OKF concept is a `.md` document by definition — the guard's path gate
# rejects anything else outright — so nothing else in the source tree is a
# concept, and nothing else is this door's to merge.
_CONCEPT_SUFFIX = ".md"
_FILENAME_PREFIX = "import-"
# The guard's non-blocking floor and its review queue, by VALUE (`Disposition`
# is a `str, Enum`, so the value is the stable thing to compare against across
# the pinned `>=0.2,<0.3` range). Pinned as constants here rather than imported
# because the dependency is injected — deliberately restated independently of
# Door B's copy in `inbox.py`, so drift in either door is visible rather than
# silently shared. The step-4 adapter's signature smoke test is what catches a
# rename in the guard itself.
_DISPOSITION_MERGE = "warn"
_DISPOSITION_QUARANTINE = "quarantine_review"
# The guard's Origin/Channel vocabularies, likewise by value. Validated here
# because `trust_for` compares by enum IDENTITY: a value outside these sets
# would reach the guard as a plain string, miss the identity check, and be
# silently classified untrusted. That failure is safe but silent, and a
# provenance declaration the library cannot recognise is not one it should
# carry — refusing is not a trust decision, it is refusing to guess at one.
_ORIGINS = frozenset({"external", "internal"})
_CHANNELS = frozenset({"automatic", "manual"})
# --- the guard seam -------------------------------------------------------
@dataclass(frozen=True)
class ImportDecision:
"""One concept's verdict, carried verbatim from `okf.import_bundle`.
`disposition` is the guard's `Disposition` VALUE and `error` its
`ConceptResult.error` — non-`None` on a hard reject (bad path, unsafe
frontmatter, non-https `resource`), in which case the concept must not be
merged whatever the disposition says. `reasons` is the audit trail the
adapter flattened out of the concept's scan report.
"""
path: str
disposition: str
error: str | None = None
reasons: tuple[str, ...] = ()
@dataclass(frozen=True)
class BundleDecision:
"""The gate's verdict on a whole bundle: per-concept results plus the log.
`log` is the guard's `BundleResult.log()` body, returned to the caller and
never written here.
"""
concepts: tuple[ImportDecision, ...]
log: str = ""
class ImportGate(Protocol):
"""The persist gate, injected. The library never imports the guard itself.
`origin` and `channel` are keyword-only by design: both are plain strings
at this seam, and a positional call site that transposed them would move a
concept between trust tiers without any type error to catch it.
"""
def __call__(self, bundle: dict[str, str], *, origin: str, channel: str) -> BundleDecision: ...
# --- per-concept outcomes -------------------------------------------------
@dataclass(frozen=True)
class MergedConcept:
"""An external concept that cleared the gate and is present in the bundle.
Also covers the no-op re-merge: identical bytes already at the target name
are the concept being present, not a second write.
"""
concept_path: str
path: Path
reasons: tuple[str, ...] = ()
@dataclass(frozen=True)
class RefusedConcept:
"""A concept the guard did not clear. Quarantine and rejection are reported
separately: quarantine means "hold for human review" and is the operator's
queue, while a fail-secure verdict is a decision, not a queue."""
concept_path: str
disposition: str
error: str | None
reasons: tuple[str, ...]
@dataclass(frozen=True)
class FailedConcept:
"""A concept this library could not process — unreadable, unusable name, or
a collision. Always a typed error, never a leaked stdlib exception."""
concept_path: str
error: IngestError
@dataclass(frozen=True)
class ImportResult:
"""Every concept's outcome, in sorted concept-path order.
Four disjoint buckets, and every concept lands in exactly one: a run's
report is complete by construction, so a concept that silently vanished
would show up as a missing entry rather than as nothing at all. `log` is
the guard's log body and `ingested_at` the run's explicit timestamp — the
caller persists them if their reserved-file policy says to.
"""
merged: tuple[MergedConcept, ...]
quarantined: tuple[RefusedConcept, ...]
rejected: tuple[RefusedConcept, ...]
failed: tuple[FailedConcept, ...]
log: str
ingested_at: str
def import_slug(concept_path: str) -> str:
"""Reduce a bundle-relative concept path to the Phase 1 id grammar.
The whole path reduces, not just its final segment: `tables/users.md` and
`views/users.md` are distinct concepts, and slugging the stem alone would
collapse them onto one filename. A path that reduces to nothing fails fast
rather than being given an invented name.
"""
concept_id = concept_path[: -len(_CONCEPT_SUFFIX)]
slug = reduce_to_id_grammar(concept_id)
if not slug:
raise MaterializationError(
f"concept path {concept_path!r} reduces to an empty slug under the "
"id grammar ([a-z0-9][a-z0-9-]*) — refusing to invent a filename",
code="import_path_empty",
)
return slug
def import_filename(slug: str) -> str:
"""The bundle filename for an imported concept.
The `import-` prefix keeps the namespace disjoint from `index.md`, Door A's
`ingest-*`, Door B's `inbox-*`, and `promoted-verdict-*` for every slug the
grammar admits.
"""
return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_CONCEPT_SUFFIX}", code="import_path_too_long"
)
def _index_label(concept_path: str) -> str:
"""The concept-ID, validated as an index link label.
The guard's path gate permits brackets in a concept path; `- [label](target)`
does not. Fail-fast, never repair — the same rule Door A applies to a
manifest title and Door B to a dropped filename.
"""
label = concept_path[: -len(_CONCEPT_SUFFIX)]
if any(char in label for char in "\n\r[]"):
raise MaterializationError(
f"concept path {concept_path!r} contains '[' or ']', which would break "
"its index link — rename it at the sender",
code="import_label_invalid",
)
return label
def _read_bundle(source: Path) -> tuple[dict[str, str], list[FailedConcept]]:
"""Read every concept document under `source`, keyed by POSIX-relative path.
Unreadable and undecodable concepts never reach the gate — they are per-
concept failures, and a concept the gate never saw is never merged.
"""
documents: dict[str, str] = {}
failed: list[FailedConcept] = []
for path in sorted(source.rglob("*")):
# The suffix test is explicit and case-folded rather than a `*.md`
# glob: glob case-sensitivity follows the FILESYSTEM, so `NOTE.MD`
# would be a concept on APFS and not one on ext4 — the same bundle
# importing differently per platform. The guard folds case here too.
if not path.is_file() or path.suffix.lower() != _CONCEPT_SUFFIX:
continue
concept_path = path.relative_to(source).as_posix()
try:
documents[concept_path] = decode_text(path.read_bytes())
except OSError as exc:
failed.append(
FailedConcept(
concept_path=concept_path,
error=SourceError(
f"cannot read concept {concept_path}: {exc}", code="source_file_missing"
),
)
)
except IngestError as exc:
failed.append(FailedConcept(concept_path=concept_path, error=exc))
return documents, failed
def import_bundle(
source_dir: Path,
bundle_dir: Path,
ingested_at: str,
*,
origin: str,
channel: str,
gate: ImportGate,
) -> ImportResult:
"""Merge the accepted concepts of an external OKF bundle (Door C).
An explicit operator command, never a watcher or a scheduler. `origin` and
`channel` are required with no defaults — trust follows origin, never
channel, and the caller is the one who knows both. `gate` is the guard
adapter over `okf.import_bundle`: every concept is assessed before anything
is written, and only the guard's non-blocking floor merges — anything else,
INCLUDING a disposition this library does not recognise and a concept the
gate returned no verdict for, fails closed.
One bad concept never aborts the run. Unreadable concepts, unusable names
and collisions are reported per concept in :class:`ImportResult` while the
rest still merge. Only three conditions fail the whole run, and all three
are wrong for every concept at once: an invalid `ingested_at`, an
unrecognised `origin`/`channel`, and a missing source directory.
"""
validate_ingested_at(ingested_at)
if origin not in _ORIGINS or channel not in _CHANNELS:
raise MaterializationError(
f"origin must be one of {sorted(_ORIGINS)} and channel one of "
f"{sorted(_CHANNELS)}, got origin={origin!r} channel={channel!r} — "
"refusing to carry a provenance declaration the guard would not "
"recognise (it decides trust from these values)",
code="import_provenance_invalid",
)
source = Path(source_dir)
if not source.is_dir():
raise SourceError(
f"source bundle directory does not exist: {source}", code="source_root_missing"
)
documents, failed = _read_bundle(source)
merged: list[MergedConcept] = []
quarantined: list[RefusedConcept] = []
rejected: list[RefusedConcept] = []
log = ""
if documents:
decision = gate(dict(documents), origin=origin, channel=channel)
log = decision.log
by_path = {entry.path: entry for entry in decision.concepts}
accepted: list[tuple[str, str]] = []
for concept_path in sorted(documents):
verdict = by_path.get(concept_path)
if verdict is None:
# No verdict is not consent: a concept the gate dropped from
# its result is refused, never read as approval by omission.
rejected.append(
RefusedConcept(
concept_path=concept_path,
disposition="",
error="the gate returned no verdict for this concept",
reasons=(),
)
)
continue
# An error is a refusal on its own terms: the guard pairs one with
# FAIL_SECURE today, but the floor must not depend on that pairing.
if verdict.error is not None or verdict.disposition != _DISPOSITION_MERGE:
refused = RefusedConcept(
concept_path=concept_path,
disposition=verdict.disposition,
error=verdict.error,
reasons=verdict.reasons,
)
if verdict.error is None and verdict.disposition == _DISPOSITION_QUARANTINE:
quarantined.append(refused)
else:
rejected.append(refused)
continue
accepted.append((concept_path, documents[concept_path]))
# Name every accepted concept BEFORE any write, so an intra-run slug
# collision is caught while both concepts can still be refused together.
named: list[tuple[str, str, str]] = []
slug_owners: dict[str, list[str]] = {}
for concept_path, text in accepted:
try:
name = import_filename(import_slug(concept_path))
_index_label(concept_path)
except IngestError as exc:
failed.append(FailedConcept(concept_path=concept_path, error=exc))
continue
named.append((concept_path, name, text))
slug_owners.setdefault(name, []).append(concept_path)
colliding = {name for name, owners in slug_owners.items() if len(owners) > 1}
for name in sorted(colliding):
for concept_path in slug_owners[name]:
others = ", ".join(
repr(other) for other in slug_owners[name] if other != concept_path
)
failed.append(
FailedConcept(
concept_path=concept_path,
error=MaterializationError(
f"{concept_path!r} and {others} both reduce to {name!r} — "
"rename one at the sender; refusing to pick a winner",
code="import_slug_collision",
),
)
)
# The occupancy gate, evaluated against the bundle as it was BEFORE
# this run: a file written below must never be judged by a later
# concept's check. Byte-identity is the only ownership proof available
# at this door, so anything else at the name is curated content or an
# update — refused either way, never overwritten.
bundle = Path(bundle_dir)
# `None` marks a no-op re-merge — an explicit sentinel, because an
# empty concept document is legitimate and must still be written.
staged: list[tuple[str, str, str | None]] = []
for concept_path, name, text in named:
if name in colliding:
continue
content = text.encode("utf-8")
existing = bundle / name
if existing.is_file():
if existing.read_bytes() == content:
staged.append((concept_path, name, None))
continue
failed.append(
FailedConcept(
concept_path=concept_path,
error=MaterializationError(
f"generated filename {name!r} is occupied by different content — "
"it is either curated or an earlier version of this concept, and "
"without a stamp the two cannot be told apart; remove it to accept "
"the update (§3)",
code="collision_unstamped",
),
)
)
continue
staged.append((concept_path, name, text))
# Disk phase.
for concept_path, name, pending in staged:
if pending is None:
path = bundle / name
else:
bundle.mkdir(parents=True, exist_ok=True)
path = write_bytes(bundle, name, pending)
verdict = by_path[concept_path]
merged.append(
MergedConcept(concept_path=concept_path, path=path, reasons=verdict.reasons)
)
# §6 index — the last disk mutation, and only when something merged.
if merged:
index_path = bundle / INDEX_NAME
if not index_path.is_file():
write_bytes(bundle, INDEX_NAME, "")
for entry in merged:
link_in_index(bundle, entry.path.name, _index_label(entry.concept_path))
return ImportResult(
merged=tuple(merged),
quarantined=tuple(quarantined),
rejected=tuple(rejected),
failed=tuple(sorted(failed, key=lambda entry: entry.concept_path)),
log=log,
ingested_at=ingested_at,
)

54
src/llm_ingestion_okf/inbox.py
View file

@ -20,7 +20,6 @@ supplies the verdict and this module only obeys it.
from __future__ import annotations
import hashlib
import re
import unicodedata
from collections.abc import Callable
from dataclasses import dataclass
@ -30,28 +29,16 @@ from .errors import IngestError, MaterializationError, SourceError
from .extract import extract_text
from .materialize import (
INDEX_NAME,
check_filename_length,
link_in_index,
parse_frontmatter,
reduce_to_id_grammar,
validate_ingested_at,
write_bytes,
)
_RESERVED_OKF_TYPE = "verdict"
# Every character outside the Phase 1 id grammar (`[a-z0-9][a-z0-9-]*`) is a
# separator. Deliberately NOT a transliteration: mapping non-ASCII letters to
# ASCII ones would be a semantic claim the slugger cannot make — Norwegian
# `møte` (meeting) would become `mote` (fashion). The readable name survives
# verbatim in the `title` field; the slug is an identifier, not a label.
_SEPARATOR_RUN_RE = re.compile(r"[^a-z0-9]+")
# NAME_MAX: the per-component limit on every filesystem this library targets
# (APFS, ext4, NTFS all cap at 255). Checked here rather than caught at the
# write, because the OS signals it as an OSError whose errno differs per
# platform (63 on macOS, 36 on Linux) — an untyped, unportable failure at the
# very moment the caller needs a typed per-file outcome. Verified empirically
# on APFS 2026-07-25: a 255-byte name writes, a 258-byte one raises errno 63.
_MAX_FILENAME_BYTES = 255
_FILENAME_PREFIX = "inbox-"
_FILENAME_SUFFIX = ".md"
@ -64,15 +51,7 @@ def inbox_slug(source_filename: str) -> str:
A name that reduces to nothing fails fast: never an invented fallback like
`untitled`, which would silently collide across unrelated files.
"""
# NFC first: macOS (APFS/HFS+) hands filenames over DECOMPOSED, so an
# `é` arrives as `e` + combining acute. Without normalising, the same
# visual filename slugs differently depending on where it came from — the
# combining mark alone becomes a separator and the base letter survives
# (`cafe`), where a composed `é` is one non-grammar character (`caf`).
# Composing first makes the whole letter one unit, so non-ASCII is
# uniformly a separator and the slug is stable across both forms.
stem = unicodedata.normalize("NFC", Path(source_filename).stem)
slug = _SEPARATOR_RUN_RE.sub("-", stem.lower()).strip("-")
slug = reduce_to_id_grammar(Path(source_filename).stem)
if not slug:
raise MaterializationError(
f"inbox filename {source_filename!r} reduces to an empty slug under the "
@ -88,27 +67,14 @@ def inbox_filename(slug: str) -> str:
The `inbox-` prefix keeps the namespace disjoint from `index.md`, Door A's
`ingest-*`, and `promoted-verdict-*` for every slug the grammar admits.
A name the filesystem cannot hold fails fast rather than being truncated:
truncation is lossy AND collision-prone (two long names sharing a prefix
would reduce to one filename, and the second write would silently claim
the first file). Refusing keeps the same posture as `inbox_slug_empty` —
the library never invents a filename the operator did not give it. The
operator's fix is to rename the dropped file, so the message carries both
the actual size and the limit.
A name the filesystem cannot hold fails fast rather than being truncated
(see :func:`check_filename_length`). Refusing keeps the same posture as
`inbox_slug_empty` — the library never invents a filename the operator did
not give it.
"""
name = f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}"
# The slug is ASCII by construction (the id grammar admits nothing else),
# so len() in characters and in bytes agree — encoding here anyway keeps
# the check honest if the grammar is ever widened.
size = len(name.encode("utf-8"))
if size > _MAX_FILENAME_BYTES:
raise MaterializationError(
f"inbox filename for slug {slug!r} would be {size} bytes, over the "
f"{_MAX_FILENAME_BYTES}-byte filesystem limit — rename the dropped "
"file; refusing to truncate (lossy and collision-prone)",
code="inbox_slug_too_long",
)
return name
return check_filename_length(
f"{_FILENAME_PREFIX}{slug}{_FILENAME_SUFFIX}", code="inbox_slug_too_long"
)
def _normalize_body(text: str) -> str:

56
src/llm_ingestion_okf/materialize.py
View file

@ -12,6 +12,7 @@ from __future__ import annotations
import hashlib
import logging
import re
import unicodedata
from dataclasses import dataclass
from pathlib import Path
@ -64,6 +65,61 @@ def validate_ingested_at(ingested_at: str) -> None:
)
# --- generated-name primitives (shared by Doors B and C) ------------------
# Every character outside the Phase 1 id grammar (`[a-z0-9][a-z0-9-]*`) is a
# separator. Deliberately NOT a transliteration: mapping non-ASCII letters to
# ASCII ones would be a semantic claim the slugger cannot make — Norwegian
# `møte` (meeting) would become `mote` (fashion). The readable name survives
# verbatim in the concept's title or index label; the slug is an identifier,
# not a label.
_SEPARATOR_RUN_RE = re.compile(r"[^a-z0-9]+")
# NAME_MAX: the per-component limit on every filesystem this library targets
# (APFS, ext4, NTFS all cap at 255). Checked before the write rather than
# caught at it, because the OS signals it as an OSError whose errno differs per
# platform (63 on macOS, 36 on Linux) — an untyped, unportable failure at the
# very moment the caller needs a typed per-file outcome. Verified empirically
# on APFS 2026-07-25: a 255-byte name writes, a 258-byte one raises errno 63.
NAME_MAX_BYTES = 255
def reduce_to_id_grammar(text: str) -> str:
"""Reduce arbitrary text to the Phase 1 id grammar, or to the empty string.
Lowercased, with every run of non-grammar characters collapsed to a single
`-` and both ends stripped. The caller decides what an empty result means:
both doors refuse it rather than invent a fallback name.
"""
# NFC first: macOS (APFS/HFS+) hands filenames over DECOMPOSED, so an `é`
# arrives as `e` + combining acute. Without normalising, the same visual
# name reduces differently depending on where it came from — the combining
# mark alone becomes a separator and the base letter survives (`cafe`),
# where a composed `é` is one non-grammar character (`caf`). Composing
# first makes the whole letter one unit, so non-ASCII is uniformly a
# separator and the result is stable across both forms.
return _SEPARATOR_RUN_RE.sub("-", unicodedata.normalize("NFC", text).lower()).strip("-")
def check_filename_length(name: str, *, code: str) -> str:
"""Refuse a generated filename the filesystem cannot hold.
Never truncated: truncation is lossy AND collision-prone (two long names
sharing a prefix would reduce to one filename, and the second write would
silently claim the first file). The message carries both the actual size
and the limit, because the operator's fix is to shorten the source name.
"""
size = len(name.encode("utf-8"))
if size > NAME_MAX_BYTES:
raise MaterializationError(
f"the generated filename would be {size} bytes, over the "
f"{NAME_MAX_BYTES}-byte filesystem limit — shorten the source name; "
"refusing to truncate (lossy and collision-prone)",
code=code,
)
return name
def _collapse_whitespace(value: str) -> str:
# §5 mandates whitespace-run collapse for ONE field only: `source_query`
# (ingest-spec.md:140-141), where a legitimately multi-line SQL SELECT